betaFeedback

WorksComputer Science

Stochastic learning flow: moment equations for stochastic gradient descent at finite step size

Abstract

Analytical theories of learning in deep networks often assume gradient flow, discarding the finite step size and batch noise of stochastic gradient descent (SGD). Starting from the discrete update, we derive continuous-time equations for the mean and covariance of the parameter distribution under a Gaussian closure for a general stochastic learning, to first order in the learning rate ηη. Applying these stochastic learning flow (SLF) equations to SGD, we show that the mean descends an effective loss regularized by the covariance and loss curvature, while the covariance, unlike in stochastic differential equation approximations, becomes unstable at curvature 2/η2/η. In SGD ensembles of linear and nonlinear networks, SLF predicts learning curves and mean trajectories more accurately than gradient flow and captures the covariance dynamics, failing where the parameter distribution becomes strongly non-Gaussian. In the full-batch limit, the edge of stability corresponds to a stationary point of the coupled sharpness-covariance dynamics, recovering self-stabilization and central flow, and batch-to-batch curvature fluctuations lower this threshold under SGD.

Introduction

Many of the most successful analytical theories of learning in deep networks, including exact solutions for deep linear networks [1, 2] and the neural tangent kernel [3], are formulated in the continuous-time limit of gradient flow. This limit makes the tools of dynamical systems theory available, but at the cost of discarding the finite step size and the sampling noise of stochastic, iterative algorithms such as stochastic gradient descent (SGD). These are not small corrections in practice. The learning rate and batch size change which solutions training finds and how well they generalize [4, 5, 6, 7], large learning rates produce qualitatively new dynamics [8, 9], and networks trained with gradient descent typically operate at the edge of stability, where the product of the learning rate and the largest Hessian eigenvalue is close to 2, far from the regime in which gradient flow applies [10].

A widely used approach approximates SGD by a stochastic differential equation (SDE), in which gradient flow is augmented by a diffusion term matched to the covariance of the minibatch gradient noise [11, 5, 12, 13]. Higher-order versions add deterministic finite-step corrections to the drift [13, 14, 15, 16], but as expansions in the product of the learning rate and the curvature, which break down where that product is of order one. In particular, these descriptions contain no instability at curvature 2/η2/\eta, and SDE surrogates of SGD do not stabilize where SGD does [17].

Conversely, the edge of stability has been analyzed with largely deterministic theories [18, 19], which recent work extends to SGD through local stability analyses [20, 21] or additive noise [22]. In high-dimensional linear [23, 24, 25] and generalized-linear [26] models, exact deterministic equations for the risk under SGD at finite step size and batch size are available, but they rely on the quadratic structure of the loss. Ensembles of stochastic learners have also been described directly at the level of the parameter distribution.

Kramers–Moyal expansions of the master equation for on-line learning at constant learning rate have long been used to describe the parameter distribution of stochastic learners, focusing on Fokker–Planck dynamics and their equilibria [27] or on moment equations with random update times [28]. Working instead from the discrete update, Yaida Yaida [29] derived exact fluctuation–dissipation relations at stationarity, and Liu et al. Liu et al. [30] obtained the exact stationary covariance of SGD near a quadratic minimum. What is missing is a continuous-time description of the parameter distribution that retains the deterministic effects of finite step size and the stochastic effects of minibatch sampling together, without expanding in the product of learning rate and curvature.

To derive such a description, we start from the discrete update rule and obtain, for general architectures and update rules, evolution equations for the mean and covariance of the parameter distribution p(θ,t)p(\theta,t) that retain all terms at first order in the learning rate. In these stochastic learning flow (SLF) equations, the parameter covariance is a dynamical variable on the same footing as the mean. Its noise-driven part coincides with the diffusion of SDE models, but additional finite-step terms qualitatively change its dynamics. For SGD, the mean descends an effective loss that includes an implicit regularization term set by the covariance and the loss curvature, which provides a mechanism for escape from saddle points and sharp minima.

We test the theory against ensembles of networks trained with SGD. In a deep linear network, SLF predicts the learning curves and parameter updates far more accurately than gradient flow at large learning rates and small batch sizes. In a nonlinear network trained on binary MNIST, the covariance correction improves the prediction of the mean by an amount that grows with the batch noise, and the integrated covariance matches the measured one. In both cases the description fails where the parameter distribution becomes strongly non-Gaussian. Finally, the SLF equations contain the edge-of-stability threshold and central flow as limiting cases, and show how fluctuations of the curvature across batches lower the threshold under SGD.

Results

Consider an artificial neural network with parameters θ\boldsymbol{\theta} and update equation

θk(t+δt)=θk(t)+ηfk(θ(t)),(1)\theta_k(t+\delta t)=\theta_k(t)+\eta f_k(\boldsymbol{\theta}(t)), \tag*{(1)}

where η\eta is a learning rate, and fkf_k is stochastic. For example, in stochastic gradient descent, we have f=−∇L\mathbf{f}=-\nabla L, where LL is the loss function computed over a batch of data, so that the stochasticity arises from the random draws of data in each batch. In the case of reinforcement learning, f\mathbf{f} may depend on exploratory noise as well.

In the sections that follow, we will first derive continuous-time moment equations describing the noisy evolution of the parameters during learning for general update equations (1), then specialize and interpret the equations in the particular case of SGD. To validate the theory and establish its limitations, we will test these equations against empirical ensembles of SGD learners in linear and nonlinear networks. Finally, we will show how these generalized equations account for EoS dynamics.

General dynamical equations for noisy parameter evolution

Beginning from (1), our goal will be to derive an equation characterizing the time evolution of the distribution p(θ,t)p(\theta,t) of network weights during training. Our approach is based on the Kramers–Moyal expansion of the master equation for the discrete update [31, 32, 33]. It follows earlier work that described ensembles of on-line learning processes at constant learning rate with master and Fokker–Planck equations [28, 27], and that we previously applied at leading order to learning in single-layer networks [34, 35]. Related moment relations, obtained from the stationarity of the discrete-time update, yield fluctuation–dissipation relations for SGD [29]. To do this, we begin with the following “master equation", which describes the evolution of this distribution over a single learning step:

p(θ,t+δt)=∫dδθ p(θ,t+δt∣θ−δθ,t)p(θ−δθ,t).(2)p(\theta,t+\delta t)=\int d\delta\theta\ p(\theta,t+\delta t\mid\theta-\delta\theta,t)p(\theta-\delta\theta,t). \tag*{(2)}

Assuming the step size to be small, the left-hand side of this equation can be expanded as

p(θ,t+δt)=p(θ,t)+δt∂tp(θ,t)+12δt2∂t2p(θ,t)+…,(3)p(\theta,t+\delta t)=p(\theta,t)+\delta t\partial_t p(\theta,t)+\frac{1}{2}\delta t^2\partial_t^2p(\theta,t)+\ldots, \tag*{(3)}

and the integrand from the right-hand side as

p(θ+δθ−δθ,t+δt∣θ−δθ,t)p(θ−δθ,t)=p(θ+δθ,t+δt∣θ,t)p(θ,t)−∑iδθi∂∂θi[p(θ+δθ,t+δt∣θ,t)p(θ,t)]+12∑ijδθiδθj∂2∂θi∂θj[p(θ+δθ,t+δt∣θ,t)p(θ,t)]+…(4)\begin{aligned} &p(\theta+\delta\theta-\delta\theta,t+\delta t\mid\theta-\delta\theta,t)p(\theta-\delta\theta,t)\\ &=p(\theta+\delta\theta,t+\delta t\mid\theta,t)p(\theta,t) -\sum_i\delta\theta_i\frac{\partial}{\partial\theta_i}[p(\theta+\delta\theta,t+\delta t\mid\theta,t)p(\theta,t)]\\ &\quad+\frac{1}{2}\sum_{ij}\delta\theta_i\delta\theta_j\frac{\partial^2}{\partial\theta_i\partial\theta_j}[p(\theta+\delta\theta,t+\delta t\mid\theta,t)p(\theta,t)]+\ldots \tag*{(4)} \end{aligned}

Setting the two sides of the equation equal to one another then gives

(δt∂t+12δt2∂t2)p(θ,t)=−∑i∂∂θi[αi(θ)p(θ,t)]+12∑ij∂2∂θi∂θj[βij(θ)p(θ,t)]+O(δt3,η3),(5)\left(\delta t\partial_t+\frac{1}{2}\delta t^2\partial_t^2\right)p(\theta,t)=-\sum_i\frac{\partial}{\partial\theta_i}[\alpha_i(\theta)p(\theta,t)]+\frac{1}{2}\sum_{ij}\frac{\partial^2}{\partial\theta_i\partial\theta_j}[\beta_{ij}(\theta)p(\theta,t)]+O(\delta t^3,\eta^3), \tag*{(5)}

where

αi(θ)=∫dδθ δθip(θ+δθ,t+δt∣θ,t)(6)\alpha_i(\theta)=\int d\delta\theta\ \delta\theta_i p(\theta+\delta\theta,t+\delta t\mid\theta,t) \tag*{(6)}

and

βij(θ)=∫dδθ δθiδθjp(θ+δθ,t+δt∣θ,t).(7)\beta_{ij}(\theta)=\int d\delta\theta\ \delta\theta_i\delta\theta_jp(\theta+\delta\theta,t+\delta t\mid\theta,t). \tag*{(7)}

Using (1), these last two equations are

αi(θ)=η⟨fi(θ)⟩,(8)\alpha_i(\theta)=\eta\langle f_i(\theta)\rangle, \tag*{(8)}
βij(θ)=η2⟨fi(θ)fj(θ)⟩,(9)\beta_{ij}(\theta)=\eta^2\langle f_i(\theta)f_j(\theta)\rangle, \tag*{(9)}

where ⟨⋅⟩\langle\cdot\rangle denotes an average over the stochasticity of a single update at fixed θ\theta (for SGD, over the draw of a batch). (Note that, in the case of reinforcement learning, this would additionally include averaging over exploratory noise in the network.) Below we write ⟨⋅⟩θ\langle\cdot\rangle_\theta for an average over p(θ,t)p(\theta,t), and Cov⁡(⋅,⋅)\operatorname{Cov}(\cdot,\cdot) and Var⁡(⋅)\operatorname{Var}(\cdot) for covariances and variances with respect to ⟨⋅⟩\langle\cdot\rangle; for example, Cov⁡(fk,fl)=⟨fkfl⟩−⟨fk⟩⟨fl⟩\operatorname{Cov}(f_k,f_l)=\langle f_kf_l\rangle-\langle f_k\rangle\langle f_l\rangle. For finite ensembles of trained networks, E[θ]\mathbb{E}[\theta] and Cov⁡(θ)\operatorname{Cov}(\theta) denote the sample mean and covariance over replicas.

Thus, dropping the higher-order terms, (5) gives the following Fokker-Planck-like description of the dynamics of the parameter vector:

(δt∂t+12δt2∂t2)p(θ,t)=−η∑i∂∂θi[⟨fi(θ)⟩p(θ,t)]+η22∑ij∂2∂θi∂θj[⟨fi(θ)fj(θ)⟩p(θ,t)].(10)\left(\delta t\partial_t+\frac{1}{2}\delta t^2\partial_t^2\right)p(\theta,t)=-\eta\sum_i\frac{\partial}{\partial\theta_i}[\langle f_i(\theta)\rangle p(\theta,t)]+\frac{\eta^2}{2}\sum_{ij}\frac{\partial^2}{\partial\theta_i\partial\theta_j}[\langle f_i(\theta)f_j(\theta)\rangle p(\theta,t)]. \tag*{(10)}

However, note that we have kept the O(δt2)O(\delta t^2) term in the expansion, which is nonstandard in the ordinary Fokker-Planck derivation but, as we will see below, is crucial for accounting for the effects induced by finite step size.

Related to this, we can see that the two terms on the right-hand side of (10) scale with different powers of the small parameter η\eta. This arises from the fact that both the deterministic and stochastic parts of the update equation scale ∼η\sim\eta in (1), which is in contrast to classical theories such as Brownian motion, where the two terms are taken to scale as ∼δt\sim\delta t and ∼δt\sim\sqrt{\delta t}, respectively, leading to an equation in which the drift and diffusion terms are both of the same order in δt\delta t. Hence, given the update equation (1) that defines iterative learning, the next-to-leading order expansion in (10) is the simplest consistent evolution equation for p(θ,t)p(\boldsymbol{\theta},t) that accounts for noise-related effects arising from input (or output) noise and finite step size.

Rather than attempting to characterize the evolution of the full distribution p(θ,t)p(\boldsymbol{\theta},t), we can gain more insight by analyzing the time evolution of the first two moments. To do this, we multiply both sides of (10) by powers of θ\boldsymbol{\theta} and integrate. For the first moment, this gives

(δt∂t+12δt2∂t2)∫dθ θkp(θ,t)=−η∑i∫dθ θk∂∂θi[⟨fi(θ)⟩p(θ,t)]+η22∑ij∫dθ θk∂2∂θi∂θj[⟨fi(θ)fj(θ)⟩p(θ,t)].(11)\left(\delta t\partial_t+\frac{1}{2}\delta t^2\partial_t^2\right)\int d\boldsymbol{\theta}\,\theta_k p(\boldsymbol{\theta},t)=-\eta\sum_i\int d\boldsymbol{\theta}\,\theta_k\frac{\partial}{\partial\theta_i}\left[\langle f_i(\boldsymbol{\theta})\rangle p(\boldsymbol{\theta},t)\right]+\frac{\eta^2}{2}\sum_{ij}\int d\boldsymbol{\theta}\,\theta_k\frac{\partial^2}{\partial\theta_i\partial\theta_j}\left[\langle f_i(\boldsymbol{\theta})f_j(\boldsymbol{\theta})\rangle p(\boldsymbol{\theta},t)\right]. \tag*{(11)}

Integrating this by parts then gives

(δt∂t+12δt2∂t2)⟨θk⟩θ=η⟨⟨fk(θ)⟩⟩θ.(12)\left(\delta t\partial_t+\frac{1}{2}\delta t^2\partial_t^2\right)\langle\theta_k\rangle_\theta=\eta\langle\langle f_k(\boldsymbol{\theta})\rangle\rangle_\theta. \tag*{(12)}

Similarly, we can multiply by θkθl\theta_k\theta_l and integrate by parts to obtain

(δt∂t+12δt2∂t2)⟨θkθl⟩θ=η⟨θk⟨fl(θ)⟩⟩θ+η⟨θl⟨fk(θ)⟩⟩θ+η2⟨⟨fk(θ)fl(θ)⟩⟩θ.(13)\left(\delta t\partial_t+\frac{1}{2}\delta t^2\partial_t^2\right)\langle\theta_k\theta_l\rangle_\theta=\eta\langle\theta_k\langle f_l(\boldsymbol{\theta})\rangle\rangle_\theta+\eta\langle\theta_l\langle f_k(\boldsymbol{\theta})\rangle\rangle_\theta+\eta^2\langle\langle f_k(\boldsymbol{\theta})f_l(\boldsymbol{\theta})\rangle\rangle_\theta. \tag*{(13)}

We will comment on the limitations associated with this choice in discussing our empirical results below.

As is typically done in continuous-time formulations of gradient descent, we can now let δt=η\delta t=\eta. Doing this and defining

Dt=∂t+η2∂t2,(14)D_t=\partial_t+\frac{\eta}{2}\partial_t^2, \tag*{(14)}

Equations (12) and (13) become

Dt⟨θk⟩θ=⟨⟨fk(θ)⟩⟩θ(15)D_t\langle\theta_k\rangle_\theta=\langle\langle f_k(\boldsymbol{\theta})\rangle\rangle_\theta \tag*{(15)}

and

Dt⟨θkθl⟩θ=⟨θk⟨fl(θ)⟩⟩θ+⟨θl⟨fk(θ)⟩⟩θ+η⟨⟨fk(θ)fl(θ)⟩⟩θ.(16)D_t\langle\theta_k\theta_l\rangle_\theta=\langle\theta_k\langle f_l(\boldsymbol{\theta})\rangle\rangle_\theta+\langle\theta_l\langle f_k(\boldsymbol{\theta})\rangle\rangle_\theta+\eta\langle\langle f_k(\boldsymbol{\theta})f_l(\boldsymbol{\theta})\rangle\rangle_\theta. \tag*{(16)}

While, by setting δt=η\delta t=\eta in our equations, we do assume that η\eta is small, we do not take the limit η→0\eta\to0 as in the commonly studied case of gradient flow. Rather, we assume only that η≪1\eta\ll1, such that terms ∼O(η)\sim O(\eta) in the above equations are interesting, while terms ∼O(η2)\sim O(\eta^2) can be neglected. At this point we can see the justification for keeping the O(δt2)O(\delta t^2) term on the left-hand side of (10): given the definition of the update equation, this scheme keeps terms up to the same order on both sides of our dynamics equations.

Combining (15) and (16), we can obtain an equation for the parameter covariance, which gives the following closed pair of dynamical equations for the first two moments:

Dt⟨θk⟩θ=⟨⟨fk(θ)⟩⟩θ,D_t\langle\theta_k\rangle_\theta=\langle\langle f_k(\boldsymbol{\theta})\rangle\rangle_\theta,
Dt(⟨θkθl⟩θ−⟨θk⟩θ⟨θl⟩θ)=⟨(θk−⟨θk⟩θ)⟨fl(θ)⟩⟩θ+⟨(θl−⟨θl⟩θ)⟨fk(θ)⟩⟩θ+η[⟨⟨fk(θ)fl(θ)⟩⟩θ−⟨⟨fk(θ)⟩⟩θ⟨⟨fl(θ)⟩⟩θ].(17)\begin{aligned} D_t\left(\langle\theta_k\theta_l\rangle_\theta-\langle\theta_k\rangle_\theta\langle\theta_l\rangle_\theta\right) ={}&\langle(\theta_k-\langle\theta_k\rangle_\theta)\langle f_l(\boldsymbol{\theta})\rangle\rangle_\theta+\langle(\theta_l-\langle\theta_l\rangle_\theta)\langle f_k(\boldsymbol{\theta})\rangle\rangle_\theta\\ &+\eta\left[\langle\langle f_k(\boldsymbol{\theta})f_l(\boldsymbol{\theta})\rangle\rangle_\theta-\langle\langle f_k(\boldsymbol{\theta})\rangle\rangle_\theta\langle\langle f_l(\boldsymbol{\theta})\rangle\rangle_\theta\right]. \tag*{(17)} \end{aligned}

Next, we can expand in the fluctuations of the parameters about their mean values:

fk(θ)=fk(⟨θ⟩θ+θ−⟨θ⟩θ)=fk(⟨θ⟩θ)+∑i(θi−⟨θi⟩θ)∂ifk(⟨θ⟩θ)+12∑ij(θi−⟨θi⟩θ)(θj−⟨θj⟩θ)∂i∂jfk(⟨θ⟩θ)+O([θ−⟨θ⟩θ]3),(18)\begin{aligned} f_k(\boldsymbol{\theta})&=f_k(\langle\boldsymbol{\theta}\rangle_\theta+\boldsymbol{\theta}-\langle\boldsymbol{\theta}\rangle_\theta)\\ &=f_k(\langle\boldsymbol{\theta}\rangle_\theta)+\sum_i(\theta_i-\langle\theta_i\rangle_\theta)\partial_i f_k(\langle\boldsymbol{\theta}\rangle_\theta)+\frac{1}{2}\sum_{ij}(\theta_i-\langle\theta_i\rangle_\theta)(\theta_j-\langle\theta_j\rangle_\theta)\partial_i\partial_j f_k(\langle\boldsymbol{\theta}\rangle_\theta)+O([\boldsymbol{\theta}-\langle\boldsymbol{\theta}\rangle_\theta]^3), \tag*{(18)} \end{aligned}

where we have defined ∂i=∂∂⟨θ⟩θ\partial_i = \frac{\partial}{\partial\langle\theta\rangle_\theta}. Truncating at second order in δθ\delta\theta amounts to a Gaussian closure, whereby third and higher cumulants of p(θ,t)p(\theta,t) are neglected, as are terms of O(C2)O(C^2). This requires fluctuations to be small on the scale over which ff varies. § 2.3 and § 2.4 examine where this fails.

If we also define fk=fk(⟨θ⟩θ)f_k = f_k(\langle\theta\rangle_\theta), mk=⟨θk⟩θm_k = \langle\theta_k\rangle_\theta, and Ckl=⟨θkθl⟩θ−⟨θk⟩θ⟨θl⟩θC_{kl} = \langle\theta_k\theta_l\rangle_\theta- \langle\theta_k\rangle_\theta\langle\theta_l\rangle_\theta, the evolution equations (17) become

Dtmk=⟨fk⟩+12∑ijCij∂i∂j⟨fk⟩(19)D_t m_k = \langle f_k\rangle+ \frac{1}{2}\sum_{ij} C_{ij}\partial_i\partial_j\langle f_k\rangle \tag*{(19)}

and

DtCkl=∑iCki∂i⟨fl⟩+∑iCli∂i⟨fk⟩+η{⟨fkfl⟩−⟨fk⟩⟨fl⟩+12∑ijCij[⟨fk∂i∂jfl⟩−⟨fk⟩∂i∂j⟨fl⟩+⟨fl∂i∂jfk⟩−⟨fl⟩∂i∂j⟨fk⟩+2⟨(∂ifk)(∂jfl)⟩]}.(20)\begin{aligned} D_t C_{kl} ={}& \sum_i C_{ki}\partial_i\langle f_l\rangle+ \sum_i C_{li}\partial_i\langle f_k\rangle+ \eta\left\{\langle f_kf_l\rangle- \langle f_k\rangle\langle f_l\rangle\right.\\ &\left. + \frac{1}{2}\sum_{ij} C_{ij}\left[\langle f_k\partial_i\partial_j f_l\rangle- \langle f_k\rangle\partial_i\partial_j\langle f_l\rangle+ \langle f_l\partial_i\partial_j f_k\rangle- \langle f_l\rangle\partial_i\partial_j\langle f_k\rangle+ 2\langle(\partial_i f_k)(\partial_j f_l)\rangle\right]\right\}. \tag*{(20)} \end{aligned}

These two equations describe the coupled flow of the parameter mean and covariance for a general stochastic learning rule.

At leading order, near a fixed point and with Poisson-distributed update times, (19)–(20) reduce to the moment equations of Ref. [28]. Two features distinguish the results here from that earlier work, however. First, the O(η)O(\eta) terms in (20), including η⟨(∂ifk)Cij(∂jfl)⟩\eta\langle(\partial_i f_k)C_{ij}(\partial_j f_l)\rangle, are retained. Second, rather than being random, the update times are fixed, which enters through DtD_t: its ∂t2\partial_t^2 term converts η⟨⟨fkfl⟩⟩θ\eta\langle\langle f_kf_l\rangle\rangle_\theta into the covariance η[⟨⟨fkfl⟩⟩θ−⟨⟨fk⟩⟩θ⟨⟨fl⟩⟩θ]\eta[\langle\langle f_kf_l\rangle\rangle_\theta-\langle\langle f_k\rangle\rangle_\theta\langle\langle f_l\rangle\rangle_\theta] in (17). With Poisson timing, by contrast, even full-batch gradient descent would acquire covariance from the randomness of the update times.

The general results (19) and (20) enable us to analyze the dynamics of learning for arbitrary architectures (without assuming infinite-width networks) and learning rules, where the noise arises both from finite step size and from noise in the update rule itself due to sampling noise and/or exploratory noise. In the language of physics, (19) and (20) describe the motion of very light particles subject to large viscous drag forces as well as externally applied forces (the right-hand sides of the two equations). The first effect of η>0\eta> 0 is to endow these particles with a small mass, η2\frac{\eta}{2}. As we will analyze in detail in the following section, the other, more consequential effect is to subject the particles to additional forces that can qualitatively change the ways in which their trajectories evolve.

Evolution equations for stochastic gradient descent

In stochastic gradient descent (SGD), the update rule is given by fk=−∂kLf_k=-\partial_kL, where LL is the loss function defined from a batch of data (and hence is a stochastic variable). If we additionally define Hij=∂i∂jLH_{ij}=\partial_i\partial_jL as the Hessian of the loss function, the dynamical equations (19) and (20) become

Dtmk=⟨fk⟩−12∑ijCij∂k⟨Hij⟩(21)D_t m_k = \langle f_k\rangle- \frac{1}{2}\sum_{ij}C_{ij}\partial_k\langle H_{ij}\rangle \tag*{(21)}

and

DtCkl=−∑i(Cki⟨Hil⟩+⟨Hki⟩Cil)+η∑ijCij⟨HikHjl⟩+η{Cov⁡(fk,fl)−12∑ijCij[Cov⁡(fk,∂lHij)+Cov⁡(fl,∂kHij)]}.(22)\begin{aligned} D_t C_{kl} ={}& -\sum_i\left(C_{ki}\langle H_{il}\rangle+ \langle H_{ki}\rangle C_{il}\right) + \eta\sum_{ij}C_{ij}\langle H_{ik}H_{jl}\rangle\\ &+ \eta\left\{\operatorname{Cov}(f_k,f_l) - \frac{1}{2}\sum_{ij}C_{ij}\left[\operatorname{Cov}(f_k,\partial_lH_{ij})+\operatorname{Cov}(f_l,\partial_kH_{ij})\right]\right\}. \tag*{(22)} \end{aligned}

Equations (21) and (22) show that nontrivial parameter dynamics during learning arise not only from the gradient of the loss function, as captured by the gradient-flow equation ∂tmk=⟨fk⟩\partial_t m_k=\langle f_k\rangle, but also through its local curvature, which is encoded in the Hessian H\mathbf{H}. To understand the dynamics of C\mathbf{C}, we can separately consider the effects of each of the terms on the right hand side of (22).

Because C\mathbf{C} is a dynamical variable independent of m\mathbf{m}, ∑ijCij∂k⟨Hij⟩=∂kTr⁡(C⟨H⟩)\sum_{ij} C_{ij}\partial_k\langle H_{ij}\rangle= \partial_k\operatorname{Tr}(\mathbf{C}\langle\mathbf{H}\rangle), with the gradient taken at fixed C\mathbf{C}, so that (21) and (22) can be written more compactly as

Dtm=⟨f⟩−12∇Tr⁡(C⟨H⟩)(23)D_t\mathbf{m}=\langle\mathbf{f}\rangle-\frac{1}{2}\nabla\operatorname{Tr}(\mathbf{C}\langle\mathbf{H}\rangle) \tag*{(23)}

and

DtC=−(C⟨H⟩+⟨H⟩C)+η⟨HCH⟩+η{Cov⁡(f)−12[Cov⁡(f,∇Tr⁡(CH))+Cov⁡(∇Tr⁡(CH),f)]}(24)\begin{aligned} D_t\mathbf{C}={}&-(\mathbf{C}\langle\mathbf{H}\rangle+\langle\mathbf{H}\rangle\mathbf{C})+\eta\langle\mathbf{HCH}\rangle\\ &+\eta\left\{\operatorname{Cov}(\mathbf{f})-\frac{1}{2}\left[\operatorname{Cov}(\mathbf{f},\nabla\operatorname{Tr}(\mathbf{CH}))+\operatorname{Cov}(\nabla\operatorname{Tr}(\mathbf{CH}),\mathbf{f})\right]\right\} \tag*{(24)} \end{aligned}

Equation (23) shows that the parameters evolve under SGD via gradient descent on an effective loss function:

Leff(m)=⟨L(m)⟩+12Tr⁡[C(t)⟨H(m)⟩].(25)L^{\mathrm{eff}}(\mathbf{m})=\langle L(\mathbf{m})\rangle+\frac{1}{2}\operatorname{Tr}[\mathbf{C}(t)\langle\mathbf{H}(\mathbf{m})\rangle]. \tag*{(25)}

The dynamic regularization term in this loss function modifies gradient descent by pushing the solution away from regions of large positive curvature and toward flat valleys in the loss landscape. The details of its effects, however, depend crucially on the dynamics of C(t)\mathbf{C}(t) arising from (24), where these dynamics are themselves driven by the curvature of the loss landscape.

The first line of (24) is driven by the geometry of the loss landscape. To see this clearly, consider the full-batch limit, in which ⟨H⟩=H\langle\mathbf{H}\rangle=\mathbf{H}, and the second line of (24) vanishes. Written in the basis in which H\mathbf{H} is diagonal, we have H=SΛST\mathbf{H}=\mathbf{S}\boldsymbol{\Lambda}\mathbf{S}^{\mathsf{T}}, where Λ=diag⁡(λ1,λ2,…)\boldsymbol{\Lambda}=\operatorname{diag}(\lambda_1,\lambda_2,\ldots), and C~=STCS\widetilde{\mathbf{C}}=\mathbf{S}^{\mathsf{T}}\mathbf{C}\mathbf{S}. Equation (24) then becomes

DtC~kl=(ηλkλl−λk−λl)C~kl.(26)D_t\widetilde{C}_{kl}=(\eta\lambda_k\lambda_l-\lambda_k-\lambda_l)\widetilde{C}_{kl}. \tag*{(26)}

For a quadratic loss, (26) is exact rather than truncated: its right-hand side equals [(1−ηλk)(1−ηλl)−1]C~kl/η[(1-\eta\lambda_k)(1-\eta\lambda_l)-1]\widetilde{C}_{kl}/\eta, the one-step change of C~kl\widetilde{C}_{kl} under gradient descent divided by the step size. The only approximation is replacing this finite difference with DtD_t. The instability at ηλ=2\eta\lambda=2 is therefore located exactly, while growth and decay rates near it are approximate when ηλ\eta\lambda is not small. In the limit η→0\eta\to0, (26) shows that modes of C\mathbf{C} aligned with directions of positive loss curvature are pushed to exponentially shrink, while those aligned with directions of negative loss curvature are pushed to exponentially grow, with the eigenvalues of H\mathbf{H} controlling the rate of convergence or divergence. In the case where η→0\eta\to0 and the loss is convex, we can see that C→0\mathbf{C}\to0, and our equations reduce to gradient flow, with ∂tmk=⟨fk⟩\partial_t m_k=\langle f_k\rangle. However, if the loss is non-convex, we see that C\mathbf{C} is amplified in regions of negative curvature even in the limit η→0\eta\to0, which, as noted in previous work [36, 1], means that nearby trajectories diverge transiently along negative-curvature directions encountered during training.

While the O(η0)O(\eta^0) term pulls the parameters down into valleys, (26) shows that introducing finite step size makes these valleys unstable once they become too narrow, providing a destabilizing mechanism of escape from saddle points and local minima. In particular, (26) shows that the dynamics near a local minimum become unstable if η>2λmax⁡\eta>\frac{2}{\lambda_{\max}}, providing a mechanism for the network to avoid converging in sharp local minima. This is the edge of stability criterion [37, 10]: the parameter covariance along any particular direction is stable if and only if the corresponding Hessian eigenvalue satisfies λ<2η\lambda<\frac{2}{\eta}. We analyze this more fully in § 2.5 below. Together, the terms in the first line of (24) describe a competition between terms that push and pull the parameters toward and away from loss minima.

These dynamics are illustrated in a one-dimensional example in Figure 1. In this one-dimensional loss landscape for an ensemble of one-parameter networks with nearly identical initialization, gradient descent with a small learning rate causes the parameter to converge to a sharp local minimum, while larger learning rates enable the parameter to escape the local minimum and converge to a broader, globally optimal solution (Figure 1A-C). By relating the learning dynamics to the curvature of the loss landscape, the SLF equations characterize the mechanism for this escape. Because the curvature at the local minimum exceeds the critical value 2/η2/\eta when η\eta is sufficiently large (Figure 1D), the parameter covariance diverges upon approach to the local minimum, providing a mechanism for escape (Figure 1E) (whereupon the SLF solution, which diverges as the ensemble escapes, is restarted from the simulated moments—Methods). This escape corresponds to a smoothing out of the effective loss function, which includes the implicit regularization term that is dynamically activated in regions of high curvature (Figure 1F). Together, this simple example illustrates that, even in the absence of batch noise, the finite-step-size corrections captured by SLF provide a mechanism of escape from sharp local minima during learning.

Figure 1. Curvature and parameter covariance determine effective loss landscape. (A) A one-dimensional loss landscape with initial parameter value m(0)=−4m(0)=-4. (B) Loss converges to a local or global minimum under gradient descent (dotted lines: simulations; solid lines: theory) with two different learning rates. (C) Parameters converge to a local or global minimum. (D) Curvature of the loss function; dashed lines are critical values hcrit=2ηh_{\mathrm{crit}}=\frac{2}{\eta} for each of the learning rates. (E) Covariance of the parameter ensemble under gradient descent, given initial variance c(0)=0.082c(0)=0.08^2. (F) Effective loss L+12chL+\frac{1}{2}ch along the trajectory of the ensemble mean (dotted), compared with LL (gray).

The effects of batch noise, i.e. the random draws of data in each batch, are captured by the terms in the second line of (24), as well as by fluctuations in the term η⟨HCH⟩\eta\langle HCH\rangle. To begin, consider the term Cov⁡(f)\operatorname{Cov}(f), which characterizes the variance of the gradients computed from different batches of data. Previous work has shown that, early in training, this covariance is relatively isotropic, with gradients from different batches pushing CC in different directions [38]. Later in training, if the loss is a negative log-likelihood and the model approximately matches the data distribution, the Fisher identity (Appendix § 4) implies Cov⁡(f)≈⟨H⟩/B\operatorname{Cov}(f)\approx\langle H\rangle/B [38, 39, 40]. This condition does not always hold, however. With noiseless targets, for example, Cov⁡(f)\operatorname{Cov}(f) vanishes at an interpolating minimum. For this reason we evaluate Cov⁡(f)\operatorname{Cov}(f) directly in all numerical experiments. Unlike the curvature term described above, this term always has the effect of pushing the parameters out of regions of large positive curvature. The appearance of the coefficient η/B\eta/B is consistent with previous work that have found this ratio to be a key determinant of SGD dynamics, with higher values leading the parameters to wider final minima [5].

The last term on the right hand side of (22), −η2[Cov⁡(f,∇Tr⁡(CH))+Cov⁡(∇Tr⁡(CH),f)]-\frac{\eta}{2}[\operatorname{Cov}(f,\nabla\operatorname{Tr}(CH))+\operatorname{Cov}(\nabla\operatorname{Tr}(CH),f)], shows that, in the presence of batch noise and finite step size, higher derivatives of the loss function can influence the parameter dynamics. To gain insight into this force arising from finite step size, we can consider it along just one dimension:

DtC=−η2C[Cov⁡(f,∂3L)+Cov⁡(∂3L,f)].(27)D_t C=-\frac{\eta}{2}C\left[\operatorname{Cov}(f,\partial^3 L)+\operatorname{Cov}(\partial^3 L,f)\right]. \tag*{(27)}

The sign of this term is set by how the batch gradient covaries with the batch third derivative across batches, not by the sign of ⟨∂3L⟩\langle\partial^3 L\rangle itself, so it can either amplify or damp the covariance. Its counterpart in the mean dynamics (22) has a definite effect, however. In one dimension, Dtm=−12C⟨∂3L⟩=−12C∂m⟨h⟩D_t\mathbf{m} = -\frac{1}{2}\mathbf{C}\langle\partial^3 L\rangle= -\frac{1}{2}\mathbf{C}\partial_{\mathbf{m}}\langle h\rangle, where h=∂2Lh = \partial^2 L is the curvature, so the covariance drives the mean toward lower curvature at a rate proportional to CC. This is the mechanism by which the covariance feeds back on the sharpness, which we analyze in § 2.5.

Together, the dynamical equation for the parameter covariance gives a rich picture of SGD dynamics that is entirely absent from the gradient-flow approximation. In particular, it describes a tug of war between the O(η0)O(\eta^0) terms, which push the parameters down into valleys of the loss function; the SGD noise term, which provides exploratory noise early in training followed by a mechanism for escaping local minima later in training; and the remaining O(η)O(\eta) terms, in which batch noise provides mechanisms for escape from sharp minima.

Stochastic Learning Flow in a two-layer linear network

To test whether the SLF equations capture the noisy dynamics of learning in a setting where the loss landscape is understood analytically, we first applied them to a two-layer deep linear network (DLN) trained with SGD. Although its input–output map is linear, the DLN loss is nonconvex, and its critical points and gradient-flow trajectories are known in closed form [1, 2, 41]. From small initialization, the network learns the singular modes of the task one at a time, with each mode learned in a rapid transition between a saddle point and the next critical point. We used the smallest network that exhibits all of these features: Nx=Nh=Ny=2N_x = N_h = N_y = 2 units per layer, trained on a task with two distinct singular values, so that the network passes through two saddle points before reaching a minimum. With P=8P = 8 parameters, the full Hessian and parameter covariance could be computed and followed mode by mode, and the averages over inputs in {(xref)}(23)–(24) could be evaluated exactly (Methods). The network mapped isotropic inputs x∼N(0,I)x \sim\mathcal{N}(0, I) to outputs y^=W2W1x\hat{y} = W^2W^1x, which were trained to match targets y=Σyxxy = \Sigma^{yx}x by minimizing the mean squared error ⟨L⟩=12∫dx p(x)∣y(x)−y^∣2\langle L\rangle= \frac{1}{2}\int dx\,p(x)|y(x) - \hat{y}|^2.

We computed the parameter mean m(t)\mathbf{m}(t) and covariance C(t)\mathbf{C}(t) from the SLF equations {(xref)}(23)–(24) as well as m(t)\mathbf{m}(t) in the η→0\eta\to0 gradient-flow (GF) limit. The results were compared against an ensemble of identically initialized simulated networks trained with SGD all starting from the same initial condition, where stochastic noise during learning led to a distribution of parameters θ\boldsymbol{\theta} across this ensemble of Ns=800N_s = 800 networks, characterized by the empirical mean E[θ(t)]=1Ns∑n=1Nsθ(n)(t)\mathbb{E}[\boldsymbol{\theta}(t)] = \frac{1}{N_s}\sum_{n=1}^{N_s}\boldsymbol{\theta}^{(n)}(t) and covariance Cov⁡(θ(t))\operatorname{Cov}(\boldsymbol{\theta}(t)).

While the expected SGD loss curve was described well by either SLF or GF in DLNs trained with small η\eta and/or large batch size BB, SLF captured the SGD learning curve much better than GF as η\eta was increased and/or BB was decreased (Figure 2A-C). In addition, while the parameter update direction dm(t)dt\frac{d\mathbf{m}(t)}{dt} under either SLF or GF was well aligned with the SGD update direction E[Δθ(t)]\mathbb{E}[\Delta\boldsymbol{\theta}(t)] for small η\eta and/or large BB, only SLF was able to capture the SGD parameter updates throughout learning in the noisy learning regime (Figure 2D-F). Together, these results show that SLF captures the loss curves and mean parameter dynamics of SGD much more faithfully than GF, particularly in the high-noise regime of large η\eta and small BB.

Figure 2. Stochastic Learning Flow captures learning curves and parameter changes in a two-layer deep linear network. (A)-(B) Expected loss during training of an ensemble of networks trained with SGD from a single initial condition, as well as corresponding gradient-flow (GF) and Stochastic Learning Flow (SLF) curves in the low-noise (A) and high-noise (B) regimes. (C) The normalized root mean square error (NRMSE) of the SLF vs. SGD and GF vs. SGD loss curves. (D)-(E) Alignment of the ensemble-averaged SGD parameter update direction with the update direction predicted by either GF or SLF in the low-noise (D) and high-noise (E) regimes. (F) The minimum update alignment values during training for various learning rates and batch sizes. Points in (C) and (F) are mean ±\pm std across n=5n = 5 identically initialized networks. Open circles denote (η,B)(\eta, B) settings for which p(θ,t)p(\theta, t) is strongly non-Gaussian, limiting the applicability of the SLF theory (cf. Figure 3D).

To understand the stochastic mechanisms responsible for these learning dynamics, we next investigated the behavior of the parameter covariance matrix C(t)\mathbf{C}(t) as the DLN learned each of the two singular modes of the task matrix Cov⁡(y,x)\operatorname{Cov}(y,x) (Figure 3A). As in previous work, we found that learning each of these modes corresponded with a change in sign of one or more Hessian eigenvalues from negative to positive, indicative of transitions through a sequence of saddle points and ultimately to a minimum [1, 2, 41, 42, 43] (Figure 3B). Projecting the parameter covariance along these Hessian modes revealed a sequence of transient expansions along negative-curvature directions, followed by contractions along these directions as the curvature changed sign (Figure 3C). While the timings of these transient increases precisely matched those in an ensemble of SGD-trained networks, we found that the peak covariance amplitudes from the SLF equations exceeded the SGD ensemble covariance, and that this overshoot grew with the third moment of the SGD ensemble, which itself increased as batch size was reduced (Figure 3C-D). Given C(t)\mathbf{C}(t) and the time-varying Hessian, we were able to construct the implicit loss regularization from (25), which exhibited transient peaks as the network passed through each saddle point (Figure 3E). Across a range of learning rates and batch sizes, the dynamics of the implicit regularization term in the integrated flow equations quantitatively matched the SGD ensemble (Figure 3F). Comparing Figures 3 and 2 shows that, in the noisy regime of large η\eta and small BB, the magnitude of the regularization term comes within an order of magnitude of the MSE loss. Together, these results show that the dynamics of the parameter covariance can be predicted by the SLF theory for small and intermediate levels of batch noise and learning rates, and that is accompanied by growing non-Gaussianity of the parameter distribution.

Curvature and covariance dynamics in a deep linear network

Figure 3. Curvature and covariance dynamics in a deep linear network. (A) An ensemble of identically initialized two-layer DLNs learn two task modes with SGD (dotted); learning dynamics are characterized by SLF (solid) and GF (dashed), with vertical lines denoting times of mode learning. (B) Eigenvalues of the Hessian during training (colored: output-potent; gray: output-null). (C) Parameter covariance from the SGD ensemble projected onto the Hessian eigenmodes (dotted), together with the SLF covariance (colored: mode-projected; gray: total). (D) Relative overshoot in the SLF estimate of the covariance peak in (C) increases with the non-Gaussian third moment of the parameter distribution as batch size decreases (colors as in panel F). (E) The implicit regularization term from the effective loss function during training. (F) The peak values of the implicit regularization during training. Points in (D,F) are means ±\pm std across 5 identically initialized networks. Open circles denote (η,B)(\eta, B) settings for which p(θ,t)p(\theta, t) is strongly non-Gaussian, limiting the applicability of the SLF theory.

Stochastic Learning Flow in a nonlinear network

To test the ability of the SLF equations to characterize learning dynamics in a nonlinear network, where less is known analytically about the loss landscape and learning dynamics, we next trained a network to perform a binary classification task. Because validation of the theory requires a large number of replicas to be trained for each setting of parameters and hyperparameters, we implemented a small-scale two-layer classifier on two digits (0 vs. 1) from the MNIST dataset. Each test began from a common parameter vector reached partway through training, so that C(0)=0C(0) = 0 and the covariance was generated entirely by minibatch noise, and followed the ensemble for 200 steps, beyond which the Gaussian approximation began to break down at intermediate batch sizes (Methods).

We first asked whether including the covariance in the parameter flow equations increased the ability of the model to predict the evolution of the parameters over time. The error during training was computed by comparing the mean parameter vector m\mathbf{m} from the integrated SLF equations to the ensemble mean from a set of SGD-trained networks with identical initializations (Figure 4A). Integrating the coupled SLF equations (23)-(24) yielded parameters that were within the estimation noise floor from the finite SGD ensemble (Figure 4A, black line). Further, computing m(t)\mathbf{m}(t) from (23) using the measured C(t)\mathbf{C}(t) from the SGD ensemble rather than the integrated C(t)\mathbf{C}(t) from (24) led to equally good prediction (Figure 4A, blue line). In contrast, setting C(t)=0\mathbf{C}(t)=0 in the flow equation impaired the prediction (Figure 4A, red line), with the size of this effect growing with decreasing batch size (Figure 4B), showing that, in the regimes we tested, C(t)\mathbf{C}(t) arising from batch noise plays a small but significant role in determining the parameter evolution.

Figure 4. Stochastic Learning Flow in a nonlinear network. (A) In a two-layer network trained on binary MNIST, the integrated SLF equations (black) predict the mean parameter trajectory of an ensemble of n=256n = 256 SGD-trained replicas to within estimation noise (gray). The prediction is unchanged if SLF uses the replica covariance rather than the integrated covariance (blue), but degrades if the parameter covariance is neglected in the flow equations (red). (B) Ratio of the parameter error at step k=200k = 200 assuming C=0C = 0 (cf. red line in (A)) to the error of the full SLF equations (cf. black line in (A)) (6 initial conditions at each batch size; bars are medians). (C) Ratio of predicted covariance trace to measured covariance trace as a function of the measured covariance trace, which increases with decreasing batch size (for each BB, points correspond to optimizer steps and network initializations). (D) Relative covariance error vs. Hill index of the distribution of losses across replicas, quantifying heaviness of the distribution tail (points are 10 timesteps over 0<k≤500 < k \le50 from a single network for each BB; dashed line: index 2, where the 2nd moment of the distribution diverges; colored lines: median sampling precision of Tr⁡Cmeas\operatorname{Tr} C_{\mathrm{meas}} in cases where they can be estimated reliably).

We next quantified the extent to which the parameter covariance C(t)\mathbf{C}(t) predicted by the SLF equations matched the empirical parameter covariance measured across the SGD ensemble. Because of the high dimensionality of the parameter space, which makes comparing covariance matrices using a finite number of observations difficult, we quantified the similarity of the overall variance by computing the traces of the covariance matrices. At low and intermediate levels of batch noise, we found that the SLF equations successfully predicted the overall variance to within 10% across two orders of magnitude (Figure 4C).

We hypothesized that increased error in the variance prediction relative to measured values at higher levels of batch noise might be, as in the DLN studied in § 2.3, associated with non-Gaussian features of the parameter distribution emerging through training. Because of the higher dimensionality of the parameter space in this nonlinear network, higher moments of the parameter distribution itself could not be accurately computed from the finite number of SGD replicates used in our simulation. Hence, we considered the one-dimensional distribution of losses over the first 50 steps of training and used the Hill index to quantify the heaviness of the distribution’s tail, where an index value of <2< 2 indicates divergence of the distribution’s second moment. Heavy tails are expected because multiplication by random batch Hessians makes SGD a random-coefficient recursion, whose stationary distributions are generically power-law [44, 45, 46]. The error of the predicted variance grew as the Hill index fell (D). In cases where the 2nd moment existed and the sampling precision of Tr⁡C\operatorname{Tr} C could be reliably estimated, we found that this sampling precision closely tracked the variance prediction error, indicating that the Gaussian closure and the variance estimator fail together because both are second-moment objects, and consistent with both being driven by heavy tails.

Together, these results show that there exists an intermediate regime of batch noise in which SLF successfully predicts parameters and their covariances during training, and that the breakdown of these predictions as batch noise increases further is associated with non-Gaussian effects.

Edge of Stability as a dynamical stationary point

Previous work [10] found empirically that deep networks trained with full-batch gradient descent operate at the Edge of Stability (EoS), with the maximum eigenvalue of the loss Hessian rising under training and then hovering near 2/η2/\eta. Two opposing forces contribute to this phenomenon. The first, progressive sharpening, is the tendency of loss curvature to increase under gradient descent, pushing the solution toward the threshold from below [10, 40]. The second arises from finite step size and forces the solution out of regions of high curvature [8, 10, 18], as seen in Figure 1D-E. In this section, we show that the EoS emerges as a stationary point of the coupled dynamics (23)–(24) in the full-batch limit, and that it is shifted downward by noise arising from finite batch size.

To begin, consider full-batch gradient descent and project along the direction v1\mathbf{v}_1 of sharpest curvature, for which Hv1=λ1v1H\mathbf{v}_1=\lambda_1\mathbf{v}_1, where λ1\lambda_1 is the largest eigenvalue of HH. Let σ12=v1TCv1\sigma_1^2=\mathbf{v}_1^{\mathsf{T}}C\mathbf{v}_1 be the covariance, where covariance along other directions is suppressed by (26). Then

dσ12dt=v1TCv1,(28)\frac{\mathrm{d}\sigma_1^2}{\mathrm{d}t}=\mathbf{v}_1^{\mathsf{T}}C\mathbf{v}_1, \tag*{(28)}

where, because v1\mathbf{v}_1 is normalized, we used ddt(v1Tv1)=0\frac{\mathrm{d}}{\mathrm{d}t}(\mathbf{v}_1^{\mathsf{T}}\mathbf{v}_1)=0. In this case, (24) gives

Dtσ12=λ1(ηλ1−2)σ12.(29)D_t\sigma_1^2=\lambda_1(\eta\lambda_1-2)\sigma_1^2. \tag*{(29)}

This equation shows that λ1=2η\lambda_1=\frac{2}{\eta} defines the EoS, with σ12(t)\sigma_1^2(t) increasing (decreasing) for sharpness above (below) this value. In particular, the existence of this critical value depends on the finite-step-size term ηλ12\eta\lambda_1^2, which arises from the term η⟨H⟩C⟨H⟩\eta\langle H\rangle C\langle H\rangle in the covariance flow and is absent in first-order SDE approaches. Coupled to these dynamics are the dynamics of λ1\lambda_1 itself, which evolves during learning as ∂tλ1=(∇λ1)Tm˙\partial_t\lambda_1=(\nabla\lambda_1)^{\mathsf{T}}\dot{\mathbf{m}}. Using the fact that m˙=F+O(η)\dot{\mathbf{m}}=\mathbf{F}+O(\eta), where F=−∇⟨L⟩−12∇Tr⁡(C⟨H⟩)\mathbf{F}=-\nabla\langle L\rangle-\frac{1}{2}\nabla\operatorname{Tr}(C\langle H\rangle), this gives

Dtλ1=(∇λ1)TDtm+η2(Dtm)T(∇∇λ1)Dtm.(30)D_t\lambda_1=(\nabla\lambda_1)^{\mathsf{T}}D_t\mathbf{m}+\frac{\eta}{2}(D_t\mathbf{m})^{\mathsf{T}}(\nabla\nabla\lambda_1)D_t\mathbf{m}. \tag*{(30)}

Dropping the terms of O(η2)O(\eta^2), this is

Dtλ1=S1−12g1σ12+η2Q1,(31)D_t\lambda_1=S_1-\frac{1}{2}g_1\sigma_1^2+\frac{\eta}{2}Q_1, \tag*{(31)}

where we have defined

S1=−(∇λ1)T(∇⟨L⟩),g1=∣∇λ1∣2,Q1=FT(∇∇λ1)F.(32)\begin{aligned} S_1&=-(\nabla\lambda_1)^{\mathsf{T}}(\nabla\langle L\rangle),\\ g_1&=|\nabla\lambda_1|^2,\\ Q_1&=\mathbf{F}^{\mathsf{T}}(\nabla\nabla\lambda_1)\mathbf{F}. \tag*{(32)} \end{aligned}

Here S1S_1 is the rate of progressive sharpening under gradient flow, which previous empirical and theoretical work has demonstrated tends to be positive in multilayer networks [10, 40]. The second term in (31) shows that covariance back-reaction (which, in full-batch learning, corresponds to period-2 oscillations) tends to counteract progressive sharpening, providing the mechanism for escape from sharp minima; this is the self-stabilization mechanism of Ref. [18], expressed through the covariance. The third term is the curvature of the sharpness landscape along the direction of parameter drift and arises due to finite step size.

The dynamical equations (29) and (31) exhibit a stationary point at

λ1∗=2η,σ12∗=2S1+ηQ1g1.(33)\lambda_1^*=\frac{2}{\eta},\qquad\sigma_1^{2*}=\frac{2S_1+\eta Q_1}{g_1}. \tag*{(33)}

In full-batch gradient descent, σ12\sigma_1^2 is the variance of an ensemble of trajectories or, for a single trajectory, the squared amplitude of its period-2 oscillation about the two-step average 12(θk−1+θk)\frac{1}{2}(\theta_{k-1}+\theta_k). Because σ12=0\sigma_1^2=0 is unstable for λ1>2/η\lambda_1>2/\eta by (29), any initial spread, however small, grows until it reaches σ12∗\sigma_1^{2*}. (33) shows that the EoS threshold corresponds to the unique stationary state of the covariance dynamics.

Further, we note that gradient flow corresponds to the σ12=0\sigma_1^2=0 limit of this theory, in which case σ12\sigma_1^2 remains 0 by (29), and the curvature evolves according to Dtλ1=S1+12Q1D_t\lambda_1=S_1+\frac{1}{2}Q_1, which possesses no mechanism to arrest progressive sharpening.

For ηQ1→0\eta Q_1\to0 this is central flow [19], in which the mean descends on L+12σ12v1v1Tλ1L+\frac{1}{2}\sigma_1^2v_1v_1^{\mathsf{T}}\lambda_1, the effective loss (25) with C=σ12v1v1T\mathbf{C}=\sigma_1^2\mathbf{v}_1\mathbf{v}_1^{\mathsf{T}}, and σ12=2S1/g1\sigma_1^2=2S_1/g_1 is fixed by imposing λ˙1=0\dot{\lambda}_1=0 at λ1=2/η\lambda_1=2/\eta. Here the same values follow from the stationary state of the covariance dynamics rather than being imposed, and (29)–(30) also describe the approach to the edge. Central flow corresponds to the limit in which the relaxation of σ12\sigma_1^2 is replaced by a constraint.

In the case where the batch size BB is finite, the dynamical equations for the parameter covariance and loss curvature contain corrections due to the effects of batch noise. The same projection that was used above in this case gives

Dtσ12=−ρ(λ1)σ12+ηcB,(34)D_t\sigma_1^2=-\rho(\lambda_1)\sigma_1^2+\frac{\eta c}{B}, \tag*{(34)}

where

ρ(λ1)=−η(1+κB)λ12+2λ1−ηBd.(35)\rho(\lambda_1)=-\eta\left(1+\frac{\kappa}{B}\right)\lambda_1^2+2\lambda_1-\frac{\eta}{B}d. \tag*{(35)}

Here we have defined

c=Var⁡i(v1T∇ℓi),d=Cov⁡i(v1T∇ℓi,v1T∇hi),(36)\begin{aligned} c&=\operatorname{Var}_i(\mathbf{v}_1^{\mathsf{T}}\nabla\ell_i),\\ d&=\operatorname{Cov}_i(\mathbf{v}_1^{\mathsf{T}}\nabla\ell_i,\mathbf{v}_1^{\mathsf{T}}\nabla h_i), \tag*{(36)} \end{aligned}

where hb=1B∑i∈bhih_b=\frac{1}{B}\sum_{i\in b}h_i is the batch curvature, and the sample curvature hih_i satisfies Ei[hi]=λ1\mathbb{E}_i[h_i]=\lambda_1. Letting Var⁡i[hi]=κλ12\operatorname{Var}_i[h_i]=\kappa\lambda_1^2, the statistics of hbh_b are then defined by Eb[hb]=λ1\mathbb{E}_b[h_b]=\lambda_1 and Var⁡b[hb]=κλ12B\operatorname{Var}_b[h_b]=\frac{\kappa\lambda_1^2}{B}. Here Ei\mathbb{E}_i, Var⁡i\operatorname{Var}_i and Cov⁡i\operatorname{Cov}_i denote averages over the NN training samples, and Eb\mathbb{E}_b and Var⁡b\operatorname{Var}_b averages over batches, so that Eb[⋅]=⟨⋅⟩\mathbb{E}_b[\cdot]=\langle\cdot\rangle.

The variance along v1\mathbf{v}_1 remains finite only while ρ>0\rho>0, so batch noise lowers the threshold at which it diverges to the point where ρ(λ0)=0\rho(\lambda_0)=0, which to first order in dd is

λ0=2η(1+κB)−ηd2B,(37)\lambda_0=\frac{2}{\eta\left(1+\frac{\kappa}{B}\right)}-\frac{\eta d}{2B}, \tag*{(37)}

consistent with the observation that SGD hovers below 2η\frac{2}{\eta} [10, 17]. The first term in (37) is the mean-square linear stability threshold of SGD projected onto a single direction. That batch-to-batch curvature fluctuations lower this threshold was identified in Ref. [20], and the exact mean-square condition was derived (assuming batches sampled without replacement) in Ref. [21]. At each step, the parameter variance along v1\mathbf{v}_1 is multiplied by (1−ηhb)2(1-\eta h_b)^2, which is convex in hbh_b, so that fluctuations in hbh_b cause marginal stability to arrive earlier. The second term arises from the coupling of gradient noise to the third derivative of the loss; it vanishes at interpolating minima, where every per-sample gradient is zero, which is why it is absent from the linear stability analysis of Ref. [21]. Supplementary Figure S1 shows that, while (37) successfully captures the reduction in λ0\lambda_0 due to batch noise in a nonlinear network using a probe test, versions of (37) with vs. without the second term were identical to within <1%<1\%, indicating that the dd term plays a comparatively small role in determining the EoS in the network we tested.

Figure S1. The edge of stability is shifted by batch noise. Mean-square EoS threshold λ0\lambda_0 calculated in a two-layer network from (37), either with (solid line) or without (dashed line) the second term from the equation. Points show critical curvature based on a one-step probe estimator in the same network. Lower panel shows the relative error between the measured and calculated values. Error bars in the top panel and shading in the lower panel show ±2\pm2 std of the probe measurement.

Two limitations apply to the result in (37). First, λ0\lambda_0 is a mean-square threshold—beyond it the variance diverges, while typical trajectories can remain bounded up to a Lyapunov threshold lying, to leading order, as far above 2/η2/\eta as λ0\lambda_0 lies below it [44, 47]. Second, because the back-reaction in (30) is the rank-one contraction of (∇λ1)T∇tr⁡(C⟨H⟩)(\nabla\lambda_1)^{\mathsf{T}}\nabla\operatorname{tr}(\mathbf{C}\langle\mathbf{H}\rangle), which depends on the whole covariance once batch noise populates its other modes, the noise amplitude in (33) does not carry over to finite BB.

Together, the results in this section show that the EoS can be understood as a dynamical stationary point of the SLF equations in the full-batch limit, and that the EoS stability threshold is shifted downward by noise arising from finite batch size.

Discussion

We have derived a set of coupled dynamical equations describing the evolution of the first two moments of the parameter distribution over an ensemble of networks, where these moments include noise effects arising from random sampling of inputs and finite step size. Applying these equations to SGD, we found that the theory describes gradient flow on an effective loss function that couples the parameter mean and covariance, the evolution of which depends on the loss curvature together with noise effects arising from finite step size and batch noise. Applying the flow equations to deep linear and nonlinear networks, we found that they enable successful prediction of loss curves, parameter updates, and parameter covariance at low and intermediate levels of noise, and we identified the regime in which the predictions begin to break down as non-Gaussian features of the parameter distribution become more pronounced. Finally, in the full-batch limit of gradient descent, the theory recovers the edge of stability as a stationary point of a pair of coupled equations describing the dynamics along the sharpest direction in the loss landscape, while, for finite batch size, the theory predicts the lowering of the EoS stability threshold as batch noise increases.

An important limitation of the SLF theory is the truncation of the moment equations at second order. As shown in Figure 3 and Figure 4, the emergence of non-Gaussian features in the parameter and loss distributions coincides with a decrease in the ability of SLF to predict the dynamics of the distribution. While addressing this by extending the approach of (11) and (13) to include higher moments would be possible in principle and could extend the ability of the theory to capture dynamics as the distribution becomes non-Gaussian, the complexity of the resulting equations would make interpretation more difficult, and the theory would still be unable to capture the possible fragmentation of the parameter distribution into multiple modes as replicas settle into different minima later in training. Future work could address this by treating the few directions along which non-Gaussian structure develops separately, for example with a reduced Fokker–Planck description or a description of the tails in the spirit of random-coefficient recursions [44], while retaining the second-moment description elsewhere. For the EoS theory, an additional limitation is the projection of the parameter dynamics along a single direction. While this is justified in the case of full-batch gradient descent, our results for the case of SGD show that the sharp mode becomes coupled to other modes due to the injection of variance along additional directions arising from batch noise. Developing a dynamical theory to fully capture EoS phenomena under SGD by including this mode coupling remains an important direction for future work. Finally, due to the high computational cost of solving the flow equations, which requires P×PP \times P covariance matrices and dense Hessian computations, the networks tested here have at most a few thousand parameters, so whether the regime of validity identified in Sections 2.3-2.4 persists in networks with more than a few thousand parameters remains to be determined.

A widely used approach models SGD by an SDE whose drift is the full-batch gradient and whose diffusion is the minibatch gradient covariance [11, 5, 12, 48, 49]. Several of these analyses fix the diffusion to be constant near an optimum [11] or proportional to the Hessian [5], which removes the batch-curvature fluctuation and cross terms of (24). Other approaches retain its dependence on θ\theta [12, 48, 49], which generates these terms. Li et al. [13] showed that this SDE is a first-order weak approximation of SGD and derived a second-order one in which the drift descends on L+η4∣∇L∣2L + \frac{\eta}{4} \lvert\nabla L \rvert^2, which was also derived by backward error analysis and analyzed numerically in Ref. [14] and which can be obtained in SLF by eliminating the ∂t2\partial_t^2 term of (23) perturbatively at C=0C = 0. These approximations have since been refined [50].

What all of these SDE descriptions lack is the deterministic finite-step part of the covariance dynamics: the term η⟨H⟩C⟨H⟩\eta\langle H \rangle C \langle H \rangle and the operator DtD_t. The difference matters where ηλ\eta\lambda is not small. The second-order SDE expands the one-step amplification of a perturbation in powers of ηλ\eta\lambda, a series that converges only for ηλ<1\eta\lambda< 1, whereas the SLF covariance equation retains the one-step change of the second moment, which for a quadratic loss is exact in ηλ\eta\lambda. Accordingly, near a quadratic minimum the stationary solution of (24) coincides with the exact stationary covariance of discrete-time SGD [30], including its divergence at ηλ=2\eta\lambda= 2. Moment relations obtained directly from the discrete update have also been used to derive fluctuation–dissipation relations at stationarity [29]; SLF extends this approach to the full time-dependent dynamics. This is why the SLF equations contain the edge-of-stability threshold, and may explain why SDE surrogates of SGD, even simulated at finite step size, do not stabilize where SGD does [17]. Kunin et al. [16] describe the late phase of SGD with an underdamped equation whose inertial term, for plain SGD, coincides with that of DtD_t. Because they apply it to individual trajectories rather than to the parameter distribution, however, their covariance dynamics lack η⟨H⟩C⟨H⟩\eta\langle H\rangle C\langle H\rangle and the instability at ηλ=2\eta\lambda= 2.

The SLF equations bring together several existing descriptions of the edge of stability. The back-reaction of the covariance on the sharpness is the self-stabilization mechanism of Damian et al. [18], expressed through the covariance; central flow [19] is the full-batch limit in which the relaxation of the covariance is replaced by a constraint; and the batch-size shift of the threshold coincides with the mean-square linear stability analysis of SGD [21], with an additional term from the coupling of gradient noise to the third derivative of the loss. Liao et al. [22] extended self-stabilization to SGD and derived a closed-form displacement of the sharpness below 2/η2/\eta from additive gradient noise, with the threshold held at 2/η2/\eta. In our framework that displacement depends on the whole covariance once batch noise populates modes other than the sharpest, so a description in terms of the sharp direction alone is incomplete at finite batch size. Two further questions remain open: where between the mean-square and Lyapunov thresholds trajectories settle [47], and how additional modes are recruited to the edge, as observed empirically [10].

While we have analyzed SLF for the particular case of SGD, the general flow equations (19)–(20) apply to any stochastic update rule. Characterizing how the flow of the parameter distribution depends on the learning rule is a potentially fruitful direction for future work. This includes momentum and adaptive methods, in which momentum is known to change the inertial term [16], and reinforcement learning, in which exploratory noise adds to sampling noise. It also includes regularization terms added to the loss, and learning-rate schedules, for which the identification δt=η\delta t = \eta used here would have to be generalized. In addition, future work can use the theory to characterize the evolution of hidden-layer representations in multilayer networks, and how these learned representations depend on SGD noise, the learning rule and the loss function. The implicit regularization term in (25) also connects the covariance dynamics to the long-standing question of how SGD noise biases training toward flat regions of the loss landscape, which the theory could help address quantitatively. Together, the framework presented here treats the parameter covariance as a dynamical variable on the same footing as the mean, and so places the effects of finite step size and sampling noise within reach of the tools of dynamical systems theory that have been used to analyze gradient flow.

Geometry of the gradient noise

To see the relationship between the SGD noise term Cov⁡(fk,fl)\operatorname{Cov}(f_k,f_l) in (22) and the Hessian, we can make use of the Fisher identity [51]:

Cov⁡p(y∣x,θ)(∂kℓ,∂lℓ)=Ep(y∣x,θ)[Hkl],(38)\operatorname{Cov}_{p(y|x,\theta)}(\partial_k\ell,\partial_l\ell)=\mathbb{E}_{p(y|x,\theta)}[H_{kl}], \tag*{(38)}

where we have assumed that the model loss can be expressed as a negative log likelihood ℓ(x)=−ln⁡p(y∣x,θ)\ell(x)=-\ln p(y|x,\theta). To see this, we write

0=∂l∂k(1)=∂l∂k∫dy p(y∣x,θ)=∂l∫dy p(y∣x,θ)∂kln⁡p(y∣x,θ)=∫dy p(y∣x,θ)[(∂kln⁡p(y∣x,θ))(∂lln⁡p(y∣x,θ))+∂k∂lln⁡p(y∣x,θ)].(39)\begin{aligned} 0 &= \partial_l\partial_k(1) \\ &= \partial_l\partial_k\int dy\ p(y|x,\theta) \\ &= \partial_l\int dy\ p(y|x,\theta)\partial_k\ln p(y|x,\theta) \\ &= \int dy\ p(y|x,\theta)\left[(\partial_k\ln p(y|x,\theta))(\partial_l\ln p(y|x,\theta))+\partial_k\partial_l\ln p(y|x,\theta)\right]. \tag*{(39)} \end{aligned}

which, using ℓ(x)=−ln⁡p(y∣x,θ)\ell(x)=-\ln p(y|x,\theta), is exactly the Fisher identity (38).

Next, if we further assume that the model is close to convergence, such that pdata(x,y)≈pmodel(x,y)p_{\mathrm{data}}(x,y)\approx p_{\mathrm{model}}(x,y), where pmodel(x,y)=p(y∣x,θ)pdata(x)p_{\mathrm{model}}(x,y)=p(y|x,\theta)p_{\mathrm{data}}(x), we have Ex,y∼pdata(x,y)[…]≈Ex∼pdata(x)[Ey∼p(y∣x)[…]]\mathbb{E}_{x,y\sim p_{\mathrm{data}}(x,y)}[\ldots]\approx\mathbb{E}_{x\sim p_{\mathrm{data}}(x)}[\mathbb{E}_{y\sim p(y|x)}[\ldots]]. Applying this to fkflf_kf_l and making use of (38) then gives

Ex,y∼pdata(x,y)[fkfl]≈Ex,y∼pdata(x,y)[Hkl].(40)\mathbb{E}_{x,y\sim p_{\mathrm{data}}(x,y)}[f_kf_l]\approx\mathbb{E}_{x,y\sim p_{\mathrm{data}}(x,y)}[H_{kl}]. \tag*{(40)}

Finally, given the definition fk=1B∑b=1B∂kℓ(xb)f_k=\frac{1}{B}\sum_{b=1}^{B}\partial_k\ell(x^b), we have

Cov⁡(fk,fl)=1BCov⁡(∂kℓ,∂lℓ)=1BEx,y∼pdata(x,y)[fkfl]≈1BEx,y∼pdata(x,y)[Hkl].(41)\begin{aligned} \operatorname{Cov}(f_k,f_l)&=\frac{1}{B}\operatorname{Cov}(\partial_k\ell,\partial_l\ell)\\ &=\frac{1}{B}\mathbb{E}_{x,y\sim p_{\mathrm{data}}(x,y)}[f_kf_l]\\ &\approx\frac{1}{B}\mathbb{E}_{x,y\sim p_{\mathrm{data}}(x,y)}[H_{kl}]. \tag*{(41)} \end{aligned}

This shows that, when the model is close to convergence, the SGD noise is proportional to the Hessian.

Methods

One-dimensional example (Figure 1)

Loss and ensembles. The loss was

L(m)=cℓ(m+12)2[1−tanh⁡(sm)]1+m2+cr(m−12)2[1+tanh⁡(sm)]−γm,L(m) = \frac{c_\ell(m+\frac{1}{2})^2[1-\tanh(sm)]}{1+m^2}+c_r(m-\frac{1}{2})^2[1+\tanh(sm)]-\gamma m,

with cℓ=30c_\ell=30, cr=1c_r=1, s=3s=3 and γ=5\gamma=5. All derivatives of LL were evaluated by central differences with step 2×10−42\times10^{-4}. For each η\eta, an ensemble of 10410^4 replicas was initialized from N(−4,0.082)\mathcal{N}(-4,0.08^2) and trained by full-batch gradient descent for 50 steps.

SLF equations. With τ=t/η\tau=t/\eta, the one-dimensional, noise-free form of (23)–(24) becomes ∂τ2u=2ηRu−2∂τu\partial_\tau^2u=2\eta R_u-2\partial_\tau u for u∈{m,c}u\in\{m,c\}, with RuR_u the right-hand side. This system was integrated by fourth-order Runge–Kutta with step Δτ=5×10−4\Delta\tau=5\times10^{-4} from the ensemble’s initial sample mean and variance with zero initial velocities. Integration was halted when c>2c>2, left blank for three steps, and restarted once from the simulated moments and their finite-difference τ\tau-derivatives. At η=0.1\eta=0.1 this occurred at step 4, with the restart at step 7; at η=0.01\eta=0.01 it did not occur. Panel F shows L(E[θ])+12Var⁡(θ)h(E[θ])L(\mathbb{E}[\theta])+\frac{1}{2}\operatorname{Var}(\theta)h(\mathbb{E}[\theta]) from the simulated ensemble.

Deep linear network (Figures 2 and 3)

Network and task. The network had Nx=Nh=Ny=2N_x=N_h=N_y=2 units per layer, y^=W2W1x\hat{\mathbf{y}}=\mathbf{W}^2\mathbf{W}^1\mathbf{x}, so θ=(vec⁡W1,vec⁡W2)\theta=(\operatorname{vec}\mathbf{W}^1,\operatorname{vec}\mathbf{W}^2) had P=8P=8 components. Inputs were drawn as x∼N(0,I)\mathbf{x}\sim\mathcal{N}(0,\mathbf{I}) and targets were y=Σyxx\mathbf{y}=\mathbf{\Sigma}^{yx}\mathbf{x} with Σyx=Udiag⁡(s1,s2)VT\mathbf{\Sigma}^{yx}=\mathbf{U}\operatorname{diag}(s_1,s_2)\mathbf{V}^{\mathsf{T}}, (s1,s2)=(2.5,1)(s_1,s_2)=(2.5,1), and U,V\mathbf{U},\mathbf{V} random orthogonal matrices drawn once with a fixed seed. Training was online, and each step used a fresh batch of BB inputs, with batch loss L=12Tr⁡[(W−Σyx)Σ^(W−Σyx)T]L=\frac{1}{2}\operatorname{Tr}[(\mathbf{W}-\mathbf{\Sigma}^{yx})\hat{\mathbf{\Sigma}}(\mathbf{W}-\mathbf{\Sigma}^{yx})^{\mathsf{T}}], where W=W2W1\mathbf{W}=\mathbf{W}^2\mathbf{W}^1 and Σ^=B−1∑n=1BxnxnT\hat{\mathbf{\Sigma}}=B^{-1}\sum_{n=1}^{B}\mathbf{x}_n\mathbf{x}_n^{\mathsf{T}}. Every run started from the same balanced initialization, W1=ϵO1T\mathbf{W}^1=\sqrt{\epsilon}\mathbf{O}_1^{\mathsf{T}} and W2=ϵO2\mathbf{W}^2=\sqrt{\epsilon}\mathbf{O}_2, with ϵ=0.01\epsilon=0.01 and O1,O2\mathbf{O}_1,\mathbf{O}_2 random orthogonal matrices drawn once with a fixed seed. Runs lasted to t=kη=8t=k\eta=8.

SGD ensembles. Each ensemble comprised Ns=800N_s=800 replicas started from the initial condition above, each with an independent stream of batches. No replica diverged in any of the runs reported. The empirical mean E[θ(t)]\mathbb{E}[\theta(t)] and covariance Cov⁡(θ(t))\operatorname{Cov}(\theta(t)) were recorded at every step. The SGD expected loss was the mean over replicas of the population loss 12∥W−Σyx∥F2\frac{1}{2}\lVert\mathbf{W}-\mathbf{\Sigma}^{yx}\rVert_F^2, after discarding the largest 1% of values.

SLF and GF. The SLF equations (23)–(24) were integrated with population averages over x\mathbf{x} evaluated in closed form. Because fb\mathbf{f}_b and Hb\mathbf{H}_b are linear in Σ^\hat{\mathbf{\Sigma}}, the moments ⟨Σ^ab⟩=δab\langle\hat{\Sigma}_{ab}\rangle=\delta_{ab} and ⟨Σ^abΣ^cd⟩=δabδcd+B−1(δacδbd+δadδbc)\langle\hat{\Sigma}_{ab}\hat{\Sigma}_{cd}\rangle=\delta_{ab}\delta_{cd}+B^{-1}(\delta_{ac}\delta_{bd}+\delta_{ad}\delta_{bc}) give ⟨f⟩\langle\mathbf{f}\rangle, ⟨H⟩\langle\mathbf{H}\rangle, Cov⁡(f)\operatorname{Cov}(\mathbf{f}), ⟨HCH⟩\langle\mathbf{HCH}\rangle, ∇Tr⁡(C⟨H⟩)\nabla\operatorname{Tr}(\mathbf{C}\langle\mathbf{H}\rangle) and the cross term Cov⁡(f,∇Tr⁡(CH))\operatorname{Cov}(\mathbf{f},\nabla\operatorname{Tr}(\mathbf{CH})) exactly. With Dt=∂t+(η/2)∂t2D_t=\partial_t+(\eta/2)\partial_t^2, each equation was written as a first-order system, m˙=v\dot{\mathbf{m}}=\mathbf{v}, v˙=(2/η)(Rm−v)\dot{\mathbf{v}}=(2/\eta)(\mathbf{R}_m-\mathbf{v}), and likewise for C\mathbf{C}, where Rm\mathbf{R}_m and RC\mathbf{R}_C are the right-hand sides of (23)–(24). The system was integrated with an adaptive Runge–Kutta solver (RK45; relative and absolute tolerances 10−410^{-4} and 10−610^{-6}) over one interval [kη,(k+1)η][k\eta,(k+1)\eta] at a time, from m(0)=θ(0)\mathbf{m}(0)=\boldsymbol{\theta}(0) and C(0)=0\mathbf{C}(0)=0, with initial velocities v(0)=Rm\mathbf{v}(0)=\mathbf{R}_m and C˙(0)=RC\dot{\mathbf{C}}(0)=\mathbf{R}_C. The loss depends on (W1,W2)(\mathbf{W}^1,\mathbf{W}^2) only through W\mathbf{W}, so it is invariant under W1→G−1W1\mathbf{W}^1\to\mathbf{G}^{-1}\mathbf{W}^1, W2→W2G\mathbf{W}^2\to\mathbf{W}^2\mathbf{G} for any invertible G\mathbf{G}. At the end of each interval, C\mathbf{C} and C˙\dot{\mathbf{C}} were therefore projected onto the (P−Nh2)(P-N_h^2)-dimensional complement of the tangent space of this symmetry at m\mathbf{m}. The SLF expected loss was ⟨L(m)⟩+12Tr⁡(C⟨H⟩)\langle L(\mathbf{m})\rangle+\frac{1}{2}\operatorname{Tr}(\mathbf{C}\langle\mathbf{H}\rangle). GF solved m˙=⟨f⟩\dot{\mathbf{m}}=\langle\mathbf{f}\rangle with C=0\mathbf{C}=0 (RK45; tolerances 10−910^{-9} and 10−1110^{-11}). All three descriptions were compared on the grid tk=kηt_k=k\eta.

Figure 2. For each (η,B)(\eta,B), SLF and GF were run once and the SGD ensemble five times with independent batch streams. The normalized root mean square error was NRMSE⁡={⟨(⟨L⟩th−⟨L⟩SGD)2⟩1/2/σ(⟨L⟩SGD)}\operatorname{NRMSE}=\{\langle(\langle L\rangle_{\mathrm{th}}-\langle L\rangle_{\mathrm{SGD}})^2\rangle^{1/2}/\sigma(\langle L\rangle_{\mathrm{SGD}})\}, where ⟨L⟩th\langle L\rangle_{\mathrm{th}} is the expected loss predicted by SLF or GF, ⟨L⟩SGD\langle L\rangle_{\mathrm{SGD}} is the SGD expected loss, and the bar and σ\sigma denote the mean and standard deviation over all steps. For the alignment, consecutive updates Δmk=m(tk+1)−m(tk)\Delta\mathbf{m}_k = \mathbf{m}(t_{k+1}) - \mathbf{m}(t_k) and ΔE[θ]k\Delta\mathbb{E}[\boldsymbol{\theta}]_k were projected onto the physical subspace at E[θ(tk)]\mathbb{E}[\boldsymbol{\theta}(t_k)] before their cosine was taken. Steps at which the SGD update norm was below 10−310^{-3} of its maximum were discarded, and the cosine was averaged over a moving window of 0.05 time units. Panel F shows the minimum of this smoothed cosine over the run.

Figure 3. Panels A–C and E used (η,B)=(0.15,16)(\eta, B) = (0.15, 16). Mode amplitudes were aα=uαTWvαa_\alpha= \mathbf{u}_\alpha^{\mathsf{T}}\mathbf{W}\mathbf{v}_\alpha, with uα,vα\mathbf{u}_\alpha,\mathbf{v}_\alpha the columns of U,V\mathbf{U},\mathbf{V}. Vertical lines mark the times at which the SGD ensemble mean first reached aα=sα/2a_\alpha= s_\alpha/2. Eigenvalues and eigenvectors wi\mathbf{w}_i of ⟨H⟩\langle\mathbf{H}\rangle were computed within the physical subspace at each step, and branches were followed through crossings by maximizing the absolute overlap of eigenvectors between consecutive steps (Hungarian assignment). Panel C shows wiTCwi\mathbf{w}_i^{\mathsf{T}}\mathbf{C}\mathbf{w}_i for SLF and, with eigenvectors computed at the ensemble mean, for the gauge-projected Cov⁡(θ)\operatorname{Cov}(\theta). In panel E, the SGD value was 12Tr⁡(C⟨H⟩)\frac{1}{2}\operatorname{Tr}(\mathbf{C}\langle\mathbf{H}\rangle) with C\mathbf{C} the gauge-projected ensemble covariance and ⟨H⟩\langle\mathbf{H}\rangle evaluated at the ensemble mean. Panels D and F used the sweep of Figure 2. The non-Gaussianity of each SGD ensemble was measured from the mode amplitudes of its replicas, aα(j)=uαTW(j)vαa_\alpha^{(j)} = \mathbf{u}_\alpha^{\mathsf{T}}\mathbf{W}^{(j)}\mathbf{v}_\alpha. At time tt, we computed m3(t)=∑α∣(aα−aˉα)3∣m_3(t) = \sum_\alpha|(a_\alpha-\bar a_\alpha)^3|, where the bar denotes the average over replicas after discarding the 1% most extreme values of aαa_\alpha at each end. The value reported for an ensemble is the peak m3=max⁡tm3(t)m_3 = \max_t m_3(t), evaluated at every max⁡(1,⌊n/60⌋)\max(1,\lfloor n/60\rfloor)-th of the nn steps. The ordinate of panel D is max⁡tTr⁡CSLF/max⁡tTr⁡Cov⁡(θ)−1\max_t\operatorname{Tr}C_{\mathrm{SLF}}/\max_t\operatorname{Tr}\operatorname{Cov}(\theta)-1, with the ensemble covariance gauge-projected. Cells with ratio below 1 are omitted. The line is a least-squares fit over cells with m3≥10−2m_3 \ge10^{-2}. The gray band is an estimate of the finite-NsN_s floor, spanning 0.5 to 2 times the median ordinate over cells with m3<10−3m_3 < 10^{-3}. Open symbols in Figures 2C,F and 3D,F mark cells whose m3m_3 exceeds the value at which the fitted law gives an ordinate of 1, i.e. where the Gaussian-closure overshoot equals the quantity itself.

Nonlinear network (Figure 4)

Network and data. Digits 0 and 1 from the MNIST training set were used as inputs, with N=104N = 10^4 balanced examples, downsampled from 28×2828 \times28 to 14×1414 \times14 by 2×22 \times2 average pooling (d=196d = 196) and standardised per pixel, and targets were y=±1y = \pm1. The network output was y^=W⃗2GELU⁡(W⃗1x⃗+b⃗1)+b2\hat y = \vec W_2\operatorname{GELU}(\vec W_1\vec x+\vec b_1)+b_2 with h=24h = 24 hidden units, so P=hd+2h+1=4753P = hd+2h+1 = 4753. The GELU nonlinearity was chosen to ensure that second- and higher-order derivatives for theoretical analysis would be well defined. The per-sample loss was ℓi=12(y^i−yi)2\ell_i = \frac{1}{2}(\hat y_i-y_i)^2, and LL was its mean over a minibatch, with no weight decay. All computations were in float64.

Initial conditions and ensembles. Parameters were initialised at unit scale (seed 0) and trained by 150 steps of full-batch gradient descent at learning rate 0.05, giving a common warm start. For each seed ss and each BB, a parent ensemble of 256 SGD replicas was run from the warm start for 500 steps at the production (η,B)(\eta,B), and one of its replicas, chosen at random, became the initial condition θJ\theta_J. Each segment then started all 256 replicas at θJ\theta_J, so C(0)=0\mathbf{C}(0)=0 and the covariance was generated entirely by minibatch noise during the segment. Minibatches were drawn independently for each replica, with replacement. The learning rate was fixed at η=0.13345\eta=0.13345; one optimiser step advanced tt by η\eta, so a 200-step segment spanned t≈26.7t\approx26.7.

Integration of the SLF equations. With Dt=∂t+(η/2)∂t2D_t=\partial_t+(\eta/2)\partial_t^2, each equation was integrated as a first-order system, m˙=p\dot{\mathbf{m}}=\mathbf{p}, p=(2/η)(Rm−p)\mathbf{p}=(2/\eta)(\mathbf{R}_m-\mathbf{p}) with Rm\mathbf{R}_m the right-hand side of (23), and likewise for C\mathbf{C}, by fixed-step fourth-order Runge–Kutta with two substeps per optimiser step. ⟨H⟩\langle\mathbf{H}\rangle was computed exactly at every optimiser step and interpolated linearly within it, as were the source terms of (24); the covariance-dependent source terms used C\mathbf{C} from the previous optimiser step, and the correction 12∇Tr⁡(C⟨H⟩)\frac{1}{2}\nabla\operatorname{Tr}(\mathbf{C}\langle\mathbf{H}\rangle) was held fixed across each step. Initial velocities took their slow-manifold values, m˙(0)=Rm−(η/2)R˙m\dot{\mathbf{m}}(0)=\mathbf{R}_m-(\eta/2)\dot{\mathbf{R}}_m and its analogue for C\mathbf{C}, so that the spurious fast mode of DtD_t, at rate 2/η2/\eta, was not excited. In a 50-step segment at B=256B=256, halving the substep changed Tr⁡Cpred\operatorname{Tr}\mathbf{C}_{\mathrm{pred}} by less than one part in 10410^4.

Terms of (24). All terms were evaluated at m(t)\mathbf{m}(t). For minibatches drawn with replacement,

⟨HCH⟩=⟨H⟩C⟨H⟩+1BEi[δHiCδHi],\langle\mathbf{HCH}\rangle= \langle\mathbf{H}\rangle\mathbf{C}\langle\mathbf{H}\rangle+ \frac{1}{B}\mathbb{E}_{i}\left[\delta\mathbf{H}_{i}\mathbf{C}\delta\mathbf{H}_{i}\right],

where δHi=Hi−⟨H⟩\delta\mathbf{H}_{i}=\mathbf{H}_{i}-\langle\mathbf{H}\rangle,

and Cov⁡(f)=B−1Cov⁡i(∇ℓi)\operatorname{Cov}(\mathbf{f})=B^{-1}\operatorname{Cov}_{i}(\nabla\ell_{i}). The fluctuation term was computed exactly by using the structure of the per-sample Hessian. The output y^\hat{y} depended on θ\boldsymbol{\theta} only through the 2h+12h+1 scalars zj=W1j⋅x+b1jz_{j}=\mathbf{W}_{1j}\cdot\mathbf{x}+b_{1j}, W2j\mathbf{W}_{2j} and b2b_{2}, each affine in θ\boldsymbol{\theta}, so ∇2y^\nabla^{2}\hat{y} had rank at most 2h2h and Hi=gigiT+ri∇2y^i=UiDiUiT\mathbf{H}_{i}=\mathbf{g}_{i}\mathbf{g}_{i}^{\mathsf{T}}+r_{i}\nabla^{2}\hat{y}_{i}=\mathbf{U}_{i}\mathbf{D}_{i}\mathbf{U}_{i}^{\mathsf{T}}, with gi=∇y^i\mathbf{g}_{i}=\nabla\hat{y}_{i}, rir_{i} the residual, Ui∈RP×(2h+1)\mathbf{U}_{i}\in\mathbb{R}^{P\times(2h+1)} known in closed form and Di\mathbf{D}_{i} block diagonal. The same structure made the two terms involving ∇Tr⁡(CH)\nabla\operatorname{Tr}(\mathbf{CH}) exact. Because the 2h+12h+1 scalars were affine in θ\boldsymbol{\theta}, the directions along which they varied did not depend on θ\boldsymbol{\theta}; writing Tr⁡(CHi)\operatorname{Tr}(\mathbf{CH}_{i}) as a function of the scalars, with C\mathbf{C} contracted onto those fixed directions, gave qi=∇Tr⁡(CHi)\mathbf{q}_{i}=\nabla\operatorname{Tr}(\mathbf{CH}_{i}) by automatic differentiation. Then the correction in (23) was 12∇Tr⁡(C⟨H⟩)=12Ei[qi]\frac{1}{2}\nabla\operatorname{Tr}(\mathbf{C}\langle\mathbf{H}\rangle)=\frac{1}{2}\mathbb{E}_{i}[\mathbf{q}_{i}], and the cross covariance in (24) was Cov⁡(f,∇Tr⁡(CH))=−B−1Cov⁡i(∇ℓi,qi)\operatorname{Cov}(\mathbf{f},\nabla\operatorname{Tr}(\mathbf{CH}))=-B^{-1}\operatorname{Cov}_{i}(\nabla\ell_{i},\mathbf{q}_{i}), where H\mathbf{H} in the covariance was the minibatch Hessian. For the measured-covariance prediction the correction was instead averaged over the 256 replica displacements, which was exact for the sample covariance.

The segments in panels A-C were run with stochastic estimates of these two terms: 256 Gaussian draws z⃗∼N(0,C⃗)\vec{z}\sim N(0,\vec{C}) for the correction, using Ez⃗[z⃗T⟨H⟩z⃗]=Tr⁡(C⟨H⟩)\mathbb{E}_{\vec{z}}[\vec{z}^{\mathsf{T}}\langle\mathbf{H}\rangle\vec{z}]=\operatorname{Tr}(\mathbf{C}\langle\mathbf{H}\rangle), and 8 Gaussian probes for the cross term, with variances those of C⃗\vec{C} along the eigenvectors of ⟨H⟩\langle\mathbf{H}\rangle and retained on the 64 eigenvectors of largest variance. Repeating the 200-step B=256B=256 segment for one seed with the exact evaluation, on identical minibatch streams, changed eclosede_{\mathrm{closed}} by at most 0.18% at any step, Tr⁡Cpred/Tr⁡Cmeas\operatorname{Tr}C_{\mathrm{pred}}/\operatorname{Tr}C_{\mathrm{meas}} by at most 0.07%, and the panel-B ratio from 1.1376 to 1.1373. All points at B≤128B\le128 in panel D used the exact evaluation.

Error measures. Each prediction’s error was e(t)=∥m(t)−E[θ](t)∥/∥E[θ](t)∥e(t)=\lVert\mathbf{m}(t)-\mathbb{E}[\boldsymbol{\theta}](t)\rVert/\lVert\mathbb{E}[\boldsymbol{\theta}](t)\rVert, where E[θ]\mathbb{E}[\boldsymbol{\theta}] is the empirical ensemble mean, evaluated at every step. Panel B showed, for each segment, eno−C/eclosede_{\mathrm{no-C}}/e_{\mathrm{closed}} at step 200. This value lay within 2.5% of the segment’s median over steps 190-200 in every segment, so it was not a single noisy draw; the median over all 200 steps gave the same ordering, with medians 1.037, 1.077 and 1.134. The sampling floor was ∥EA[θ]−EB[θ]∥/(2∥E[θ]∥)\lVert\mathbb{E}_{A}[\boldsymbol{\theta}]-\mathbb{E}_{B}[\boldsymbol{\theta}]\rVert/(2\lVert\mathbb{E}[\boldsymbol{\theta}]\rVert) for disjoint halves AA and BB of the ensemble. The covariance traces were Tr⁡Cmeas=n−1∑j∥θj−E[θ]∥2\operatorname{Tr}C_{\mathrm{meas}}=n^{-1}\sum_{j}\lVert\boldsymbol{\theta}_{j}-\mathbb{E}[\boldsymbol{\theta}]\rVert^{2} and the trace of the integrated C\mathbf{C}; the unbiased normalisation 1/(n−1)1/(n-1) would lower every trace ratio in panels C and D by 0.39%. The sampling error of Tr⁡Cmeas\operatorname{Tr}C_{\mathrm{meas}} was the RMS over 200 random 128/128 partitions of 12∣Tr⁡CA−Tr⁡CB∣\frac{1}{2}\lvert\operatorname{Tr}C_{A}-\operatorname{Tr}C_{B}\rvert, relative to Tr⁡Cmeas\operatorname{Tr}C_{\mathrm{meas}}, computed from the replica ensembles stored for panel D (below); the 1.3% quoted for panel C was its median over the states at B≥256B\ge256. A replica whose parameters became non-finite was removed, and ensemble statistics were taken over the remainder.

Gauge. The loss was invariant under permutations of hidden units, so E[θ]\mathbb{E}[\boldsymbol{\theta}] was a meaningful network only if the replicas shared one assignment of units. Aligning each replica to a consensus reference required a non-identity permutation for none of the 2560 replicas examined at each B≥128B\ge128. GELU is not odd, so there was no sign symmetry to check.

Panel D and the breakdown at small BB. The segments at B∈{32,64,128}B\in\{32,64,128\} used the same protocol with seed 11. Replica ensembles were stored at 10 times during the first 50 steps at each of the six batch sizes. At each stored state, the tail index was the Hill estimator applied to the upper 10% of the distribution of L(θi)L(\boldsymbol{\theta}_{i}) across replicas, checked against a Gaussian surrogate with the same mean and covariance, and the trace error was read at the same step. At B≤128B\le128, Tr⁡Cpred\operatorname{Tr}C_{\mathrm{pred}} came from repeating the 50-step segment with the exact evaluation described above; the replicas, and hence Tr⁡Cmeas\operatorname{Tr}C_{\mathrm{meas}} and the tail index, were identical between the two runs. Seven states, all at B=32B=32, were excluded because Tr⁡Cmeas\operatorname{Tr}C_{\mathrm{meas}} or Tr⁡Cpred\operatorname{Tr}C_{\mathrm{pred}} was non-finite: the ensemble had diverged, and the eigendecomposition used in evaluating the fluctuation term failed. Indefiniteness of the integrated covariance was measured by its smallest eigenvalue divided by its largest. Where Tr⁡Cmeas\operatorname{Tr}C_{\mathrm{meas}} had diverged, Tr⁡Cpred/Tr⁡Cmeas\operatorname{Tr}C_{\mathrm{pred}}/\operatorname{Tr}C_{\mathrm{meas}} was negligible and the plotted error saturated at 1. The horizontal lines were the median, over each batch size’s states, of the sampling error of Tr⁡Cmeas\operatorname{Tr}C_{\mathrm{meas}} at that state; a state was counted as exceeding its sampling error by comparison with its own value. That sampling error required the loss to have finite variance across replicas, so it was not defined below a tail index of 2, and the lines at B=64B = 64 and B=32B = 32 indicated only its scale.

Edge-of-stability threshold test (Figure S1)

Network and task. The network was a two-layer perceptron, y^=W2tanh⁡(W1x+b1)+b2\hat{y} = \mathbf{W}_2 \tanh(\mathbf{W}_1\mathbf{x}+\mathbf{b}_1)+b_2, with dx=15d_x = 15 inputs, h=32h = 32 hidden units and a scalar output (P=545P = 545), trained on the loss L=1N∑i=1N12(y^i−yi)2L = \frac{1}{N}\sum_{i=1}^{N}\frac{1}{2}(\hat{y}_i-y_i)^2 over N=200N = 200 fixed training examples. The inputs were whitened exactly to a power-law covariance: with μk∝k−1\mu_k \propto k^{-1} normalized to unit mean, the input matrix was X=NQdiag⁡(μk1/2)\mathbf{X} = \sqrt{N}\mathbf{Q}\operatorname{diag}(\mu_k^{1/2}), where Q\mathbf{Q} is the orthonormal factor of a 200×15200 \times15 standard Gaussian matrix. Targets were produced by a random teacher of the same form with 12 hidden units (weights drawn from N(0,1/fan-in)\mathcal{N}(0,1/\text{fan-in}), hidden biases from N(0,0.32)\mathcal{N}(0,0.3^2), output bias zero), standardized to zero mean and unit variance and multiplied by 4. The student was initialized with weights drawn from N(0,0.25/fan-in)\mathcal{N}(0,0.25/\text{fan-in}) and zero biases. The learning rate was η=0.1\eta= 0.1, so 2/η=202/\eta= 20.

Base points. Full-batch gradient descent was run for 1500 steps from this initialization. At each step k≥1k \ge1 we formed the bounce average mk=12(θk−1+θk)\mathbf{m}_k = \frac{1}{2}(\boldsymbol{\theta}_{k-1}+\boldsymbol{\theta}_k) and computed λ1\lambda_1 and v1\mathbf{v}_1 of the full-batch Hessian by Lanczos iteration. The interval [0.63,1.10]×2/η[0.63, 1.10]\times2/\eta, truncated at the largest λ1\lambda_1 reached, was divided into 44 equal bins in λ1\lambda_1. From each non-empty bin, the middle (in time) of the steps falling in that bin was taken as a base point, giving 34 base points between steps 12 and 1377 with λ1\lambda_1 from 13.31 to 21.47. At each base point we computed the per-sample curvatures hi=v1THiv1h_i = \mathbf{v}_1^{\mathsf T}\mathbf{H}_i\mathbf{v}_1, κ=Var⁡i(hi)/λ12\kappa= \operatorname{Var}_i(h_i)/\lambda_1^2 and dd of (36), by automatic differentiation.

Probe. At each base point m\mathbf{m} and each BB, we displaced n=6×104n = 6 \times10^4 antithetic pairs of replicas to m±ϵv1\mathbf{m}\pm\epsilon\mathbf{v}_1, with ϵ=10−5∣m∣\epsilon= 10^{-5}|\mathbf{m}|. Both members of a pair took one SGD step with the same batch of BB examples, drawn uniformly with replacement, and different pairs used independent batches. The half difference of their displacements along v1\mathbf{v}_1, p=12v1T(θ′+−θ′−)\mathbf{p} = \frac{1}{2}\mathbf{v}_1^{\mathsf T}(\boldsymbol{\theta}^{\prime+}-\boldsymbol{\theta}^{\prime-}), cancels the gradient-noise source exactly and gives the mean-square growth rate r=η−1ln⁡(p2/ϵ2)r = \eta^{-1}\ln(p^2/\epsilon^2), whose zero is λ0\lambda_0 with d=0d = 0. The standard error of rr was propagated from the sample variance of p2p^2. The measured threshold at each BB was the largest λ1\lambda_1 at which rr, ordered by λ1\lambda_1, changed sign, located by linear interpolation between the two bracketing base points. Its error combined the standard errors of rr at those points, divided by the slope of the interpolant. This error was then multiplied by 1.710, the root mean square of the normalized differences between these thresholds and those from a repeat of the sweep at B=5B = 5–128 with different random draws.

Prediction and comparison. The predicted threshold was λ0(d=0)=2/[η(1+κ/B)]\lambda_0(d=0) = 2/[\eta(1+\kappa/B)], with κ\kappa linearly interpolated in λ1\lambda_1 between base points and evaluated at the measured threshold. The curve including the second term of (37) subtracts ηd/(2B)\eta d/(2B), with dd interpolated in the same way. Residuals are 100(λ0pred−λ0obs)/λ0obs100(\lambda_0^{\mathrm{pred}}-\lambda_0^{\mathrm{obs}})/\lambda_0^{\mathrm{obs}}. The reported χ2/dof\chi^2/\mathrm{dof} is the mean of the squared normalized residuals over the seven batch sizes, with no fitted parameters.

Use of AI tools

The theoretical framework, the derivation of the SLF equations, and the design of the study are the authors’ own. In developing the numerical experiments and the manuscript, we made use of AI assistants based on Anthropic’s Claude model Opus 5.5. AI coding agents wrote and ran simulation and analysis code for the DLN and nonlinear-network experiments under specifications made by the authors. An AI assistant also helped interpret results, check derivations, survey related literature, and edit parts of the text. The authors reviewed, verified, and revised all AI-generated code, analyses, and text, and take full responsibility for the content of the paper.

Supplemental Information

References

  1. [1]Andrew M Saxe, James L McClelland, and Surya Ganguli. “Exact solutions to the nonlinear dynamics of learning in deep linear neural networks”. arXiv preprint arXiv:1312.6120 (2013).arxiv.org/abs/1312.6120
  2. [2]Andrew M Saxe, James L McClelland, and Surya Ganguli. “A mathematical theory of semantic development in deep neural networks”. Proceedings of the National Academy of Sciences 116.23 (2019), pp. 11537–11546.DOI
  3. [3]Arthur Jacot, Franck Gabriel, and Clément Hongler. “Neural tangent kernel: Convergence and generalization in neural networks”. Advances in neural information processing systems 31 (2018).arxiv.org/abs/1806.07572
  4. [4]Nitish Shirish Keskar et al. “On large-batch training for deep learning: Generalization gap and sharp minima”. ICLR (2017).arxiv.org/abs/1609.04836
  5. [5]Stanisław Jastrzębski et al. “Three factors influencing minima in SGD”. arXiv preprint arXiv:1711.04623 (2017).arxiv.org/abs/1711.04623
  6. [6]Samuel L. Smith and Quoc V. Le. “A Bayesian Perspective on Generalization and Stochastic Gradient Descent”. International Conference on Learning Representations (ICLR). 2018. URL: https : //openreview.net/forum?id=BIji4yg0Z.arxiv.org/abs/1710.06451
  7. [7]Yuanzhi Li, Colin Wei, and Tengyu Ma. “Towards explaining the regularization effect of initial large learning rate in training neural networks”. Advances in neural information processing systems 32 (2019).arxiv.org/abs/1907.04595
  8. [8]Aitor Lewkowycz et al. “The large learning rate phase of deep learning: the catapult mechanism”. arXiv preprint arXiv:2003.02218 (2020).arxiv.org/abs/2003.02218
  9. [9]Mathieu Even et al. “(S) GD over Diagonal Linear Networks: Implicit bias, Large Stepsizes and Edge of Stability”. Advances in Neural Information Processing Systems 36 (2023), pp. 29406–29448.DOI
  10. [10]Jeremy Cohen et al. “Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability”. International Conference on Learning Representations. 2021.arxiv.org/abs/2103.00065
  11. [11]Stephan Mandt, Matthew Hoffman, and David Blei. “A variational analysis of stochastic gradient algorithms”. International conference on machine learning. PMLR. 2016, pp. 354–363.arxiv.org/abs/1602.02666
  12. [12]Pratik Chaudhari and Stefano Soatto. “Stochastic gradient descent performs variational inference, converges to limit cycles for deep networks”. 2018 Information Theory and Applications Workshop (ITA). IEEE. 2018, pp. 1–10.arxiv.org/abs/1710.11029
  13. [13]Qianxiao Li, Cheng Tai, et al. “Stochastic modified equations and adaptive stochastic gradient algorithms”. International Conference on Machine Learning. PMLR. 2017, pp. 2101–2110.
  14. [14]David GT Barrett and Benoit Dherin. “Implicit gradient regularization”. arXiv preprint arXiv:2009.11162 (2020).arxiv.org/abs/2009.11162
  15. [15]Samuel L Smith et al. “On the origin of implicit regularization in stochastic gradient descent”. ICLR (2021).arxiv.org/abs/2101.12176
  16. [16]Daniel Kunin et al. “The limiting dynamics of sgd: Modified loss, phase-space oscillations, and anomalous diffusion”. Neural Computation 36.1 (2023), pp. 151–174.
  17. [17]Arseniy Andreyev and Pierfrancesco Beneventano. “Edge of stochastic stability: Revisiting the edge of stability for sgd”. arXiv preprint arXiv:2412.20553 (2024).arxiv.org/abs/2412.20553
  18. [18]Alex Damian, Eshaan Nichani, and Jason D Lee. “Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stability.” ICLR 2023 (2023).arxiv.org/abs/2209.15594
  19. [19]Jeremy Cohen et al. “Understanding Optimization in Deep Learning with Central Flows”. The Thirteenth International Conference on Learning Representations. 2025.arxiv.org/abs/2410.24206
  20. [20]Lei Wu, Chao Ma, et al. “How sgd selects the global minima in over-parameterized learning: A dynamical stability perspective”. Advances in Neural Information Processing Systems 31 (2018).
  21. [21]Rotem Mulayoff and Tomer Michaeli. “Exact mean square linear stability analysis for SGD”. The Thirty Seventh Annual Conference on Learning Theory. PMLR. 2024, pp. 3915–3969.arxiv.org/abs/2306.07850
  22. [22]Fangshuo Liao, Afroditi Kolomvaki, and Anastasios Kyrillidis. “SGD at the Edge of Stability: The Stochastic Sharpness Gap”. arXiv preprint arXiv:2604.21016 (2026).arxiv.org/abs/2604.21016
  23. [23]Courtney Paquette et al. “SGD in the large: Average-case analysis, asymptotics, and stepsize criticality”. Conference on Learning Theory. PMLR. 2021, pp. 3548–3626.arxiv.org/abs/2102.04396
  24. [24]Courtney Paquette et al. “Homogenization of SGD in high-dimensions: exact dynamics and general-ization properties: C. Paquette et al.” Mathematical Programming 214.1 (2025), pp. 1–90.
  25. [25]Blake Bordelon and Cengiz Pehle van. “Learning curves for SGD on structured features”. ICLR (2022).arxiv.org/abs/2106.02713
  26. [26]Elizabeth Collins-Woodfin et al. “Hitting the high-dimensional notes: An ode for sgd learning dynamics on glms and multi-index models”. Information and Inference: A Journal of the IMA 13.4 (2024), iaae028.arxiv.org/abs/2308.08977
  27. [27]Todd Leen and John Moody. “Weight space probability densities in stochastic learning: I. Dynamics and equilibria”. Advances in neural information processing systems 5 (1992).
  28. [28]Tom M Heskes and Bert Kappen. “Learning processes in neural networks”. Physical Review A 44.4 (1991), p. 2718.pubmed.ncbi.nlm.nih.gov/9906254
  29. [29]Sho Yaida. “Fluctuation-dissipation relations for stochastic gradient descent”. ICLR (2019).arxiv.org/abs/1810.00004
  30. [30]Kangqiao Liu, Liu Ziyin, and Masahito Ueda. “Noise and fluctuation of finite learning rate stochastic gradient descent”. International Conference on Machine Learning. PMLR. 2021, pp. 7045–7056.arxiv.org/abs/2012.03636
  31. [31]Hannes Risken. “Fokker-planck equation”. The Fokker-Planck equation: methods of solution and applications. Springer, 1989, pp. 63–95.
  32. [32]Hendrik Anthony Kramers. “Brownian motion in a field of force and the diffusion model of chemical reactions”. physica 7.4 (1940), pp. 284–304.DOI
  33. [33]José Enrique Moyal. “Stochastic processes and statistical physics”. Journal of the Royal Statistical Society. Series B (Methodological) 11.2 (1949), pp. 150–210.DOI
  34. [34]James M Murray and G Sean Escola. “Remembrance of things practiced with fast and slow learning in cortical and subcortical pathways”. Nature Communications 11.1 (2020), p. 6441.
  35. [35]Christian Schmid and James M Murray. “Dynamics of supervised and reinforcement learning in the non-linear perceptron”. Advances in neural information processing systems 37 (2024), pp. 136424–136445.arxiv.org/abs/2409.03749
  36. [36]Yann N Dauphin et al. “Identifying and attacking the saddle point problem in high-dimensional non-convex optimization”. Advances in neural information processing systems 27 (2014).arxiv.org/abs/1406.2572
  37. [37]Yann Le Cun, Ido Kanter, and Sara A Solla. “Eigenvalues of covariance matrices: Application to neural-network learning”. Physical review letters 66.18 (1991), p. 2396.
  38. [38]Guy Gur-Ari, Daniel A Roberts, and Ethan Dyer. “Gradient descent happens in a tiny subspace”. arXiv preprint arXiv:1812.04754 (2018).
  39. [39]Zhanxing Zhu et al. “The Anisotropic Noise in Stochastic Gradient Descent: Its Behavior of Escaping from Sharp Minima and Regularization Effects”. International Conference on Machine Learning. PMLR. 2019, pp. 7654–7663.arxiv.org/abs/1803.00195
  40. [40]Valentin Thomas et al. “On the interplay between noise and curvature and its effect on optimization and generalization”. International Conference on Artificial Intelligence and Statistics. PMLR. 2020, pp. 3503–3513.arxiv.org/abs/1906.07774
  41. [41]Arthur Jacot et al. “Saddle-to-saddle dynamics in deep linear networks: Small initialization training, symmetry, and sparsity”. arXiv preprint arXiv:2106.15933 (2021).arxiv.org/abs/2106.15933
  42. [42]Scott Pesme and Nicolas Flammarion. “Saddle-to-saddle dynamics in diagonal linear networks”. Advances in Neural Information Processing Systems 36 (2023), pp. 7475–7505.
  43. [43]Yedi Zhang, Andrew Saxe, and Peter Latham. “Saddle-to-saddle dynamics explains a simplicity bias across neural network architectures”. International Conference on Learning Representations. Vol. 2026. 2026, pp. 57507–57551.arxiv.org/abs/2512.20607
  44. [44]Harry Kesten. “Random difference equations and renewal theory for products of random matrices”. Acta Mathematica 131 (1973).DOI
  45. [45]Mert Gurbuzbalaban, Umut Simsekli, and Lingjiong Zhu. “The heavy-tail phenomenon in SGD”. International Conference on Machine Learning. PMLR. 2021, pp. 3964–3975.
  46. [46]Liam Hodgkinson and Michael Mahoney. “Multiplicative noise and heavy tails in stochastic optimization”. International Conference on Machine Learning. PMLR. 2021, pp. 4262–4274.arxiv.org/abs/2006.06293
  47. [47]Liu Ziyin et al. “Type-ii saddles and probabilistic stability of stochastic gradient descent”. arXiv preprint arXiv:2303.13093 (2023).
  48. [48]Alnur Ali, Edgar Dobriban, and Ryan Tibshirani. “The implicit regularization of stochastic gradient flow for least squares”. International conference on machine learning. PMLR. 2020, pp. 233–244.arxiv.org/abs/2003.07802
  49. [49]Feng Chen et al. “Stochastic collapse: How gradient noise attracts sgd dynamics towards simpler subnetworks”. Advances in Neural Information Processing Systems 36 (2023), pp. 35027–35063.
  50. [50]Benjamin Gess, Sebastian Kassing, and Vitalii Konarovskyi. “Stochastic modified flows, mean-field limits and dynamics of stochastic gradient descent”. arXiv preprint arXiv:2302.07125 (2023).arxiv.org/abs/2302.07125
  51. [51]Erich Leo Lehmann and George Casella. Theory of point estimation. Springer, 1998.

Paper details

vlnc:W_tr9uvdc5
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 1

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents