WorksComputer Science
Stochastic learning flow: moment equations for stochastic gradient descent at finite step size
Abstract
Analytical theories of learning in deep networks often assume gradient flow, discarding the finite step size and batch noise of stochastic gradient descent (SGD). Starting from the discrete update, we derive continuous-time equations for the mean and covariance of the parameter distribution under a Gaussian closure for a general stochastic learning, to first order in the learning rate . Applying these stochastic learning flow (SLF) equations to SGD, we show that the mean descends an effective loss regularized by the covariance and loss curvature, while the covariance, unlike in stochastic differential equation approximations, becomes unstable at curvature . In SGD ensembles of linear and nonlinear networks, SLF predicts learning curves and mean trajectories more accurately than gradient flow and captures the covariance dynamics, failing where the parameter distribution becomes strongly non-Gaussian. In the full-batch limit, the edge of stability corresponds to a stationary point of the coupled sharpness-covariance dynamics, recovering self-stabilization and central flow, and batch-to-batch curvature fluctuations lower this threshold under SGD.
Introduction
Many of the most successful analytical theories of learning in deep networks, including exact solutions for deep linear networks [1, 2] and the neural tangent kernel [3], are formulated in the continuous-time limit of gradient flow. This limit makes the tools of dynamical systems theory available, but at the cost of discarding the finite step size and the sampling noise of stochastic, iterative algorithms such as stochastic gradient descent (SGD). These are not small corrections in practice. The learning rate and batch size change which solutions training finds and how well they generalize [4, 5, 6, 7], large learning rates produce qualitatively new dynamics [8, 9], and networks trained with gradient descent typically operate at the edge of stability, where the product of the learning rate and the largest Hessian eigenvalue is close to 2, far from the regime in which gradient flow applies [10].
A widely used approach approximates SGD by a stochastic differential equation (SDE), in which gradient flow is augmented by a diffusion term matched to the covariance of the minibatch gradient noise [11, 5, 12, 13]. Higher-order versions add deterministic finite-step corrections to the drift [13, 14, 15, 16], but as expansions in the product of the learning rate and the curvature, which break down where that product is of order one. In particular, these descriptions contain no instability at curvature , and SDE surrogates of SGD do not stabilize where SGD does [17].
Conversely, the edge of stability has been analyzed with largely deterministic theories [18, 19], which recent work extends to SGD through local stability analyses [20, 21] or additive noise [22]. In high-dimensional linear [23, 24, 25] and generalized-linear [26] models, exact deterministic equations for the risk under SGD at finite step size and batch size are available, but they rely on the quadratic structure of the loss. Ensembles of stochastic learners have also been described directly at the level of the parameter distribution.
Kramers–Moyal expansions of the master equation for on-line learning at constant learning rate have long been used to describe the parameter distribution of stochastic learners, focusing on Fokker–Planck dynamics and their equilibria [27] or on moment equations with random update times [28]. Working instead from the discrete update, Yaida Yaida [29] derived exact fluctuation–dissipation relations at stationarity, and Liu et al. Liu et al. [30] obtained the exact stationary covariance of SGD near a quadratic minimum. What is missing is a continuous-time description of the parameter distribution that retains the deterministic effects of finite step size and the stochastic effects of minibatch sampling together, without expanding in the product of learning rate and curvature.
To derive such a description, we start from the discrete update rule and obtain, for general architectures and update rules, evolution equations for the mean and covariance of the parameter distribution that retain all terms at first order in the learning rate. In these stochastic learning flow (SLF) equations, the parameter covariance is a dynamical variable on the same footing as the mean. Its noise-driven part coincides with the diffusion of SDE models, but additional finite-step terms qualitatively change its dynamics. For SGD, the mean descends an effective loss that includes an implicit regularization term set by the covariance and the loss curvature, which provides a mechanism for escape from saddle points and sharp minima.
We test the theory against ensembles of networks trained with SGD. In a deep linear network, SLF predicts the learning curves and parameter updates far more accurately than gradient flow at large learning rates and small batch sizes. In a nonlinear network trained on binary MNIST, the covariance correction improves the prediction of the mean by an amount that grows with the batch noise, and the integrated covariance matches the measured one. In both cases the description fails where the parameter distribution becomes strongly non-Gaussian. Finally, the SLF equations contain the edge-of-stability threshold and central flow as limiting cases, and show how fluctuations of the curvature across batches lower the threshold under SGD.
Results
Consider an artificial neural network with parameters and update equation
where is a learning rate, and is stochastic. For example, in stochastic gradient descent, we have , where is the loss function computed over a batch of data, so that the stochasticity arises from the random draws of data in each batch. In the case of reinforcement learning, may depend on exploratory noise as well.
In the sections that follow, we will first derive continuous-time moment equations describing the noisy evolution of the parameters during learning for general update equations (1), then specialize and interpret the equations in the particular case of SGD. To validate the theory and establish its limitations, we will test these equations against empirical ensembles of SGD learners in linear and nonlinear networks. Finally, we will show how these generalized equations account for EoS dynamics.
General dynamical equations for noisy parameter evolution
Beginning from (1), our goal will be to derive an equation characterizing the time evolution of the distribution of network weights during training. Our approach is based on the Kramers–Moyal expansion of the master equation for the discrete update [31, 32, 33]. It follows earlier work that described ensembles of on-line learning processes at constant learning rate with master and Fokker–Planck equations [28, 27], and that we previously applied at leading order to learning in single-layer networks [34, 35]. Related moment relations, obtained from the stationarity of the discrete-time update, yield fluctuation–dissipation relations for SGD [29]. To do this, we begin with the following “master equation", which describes the evolution of this distribution over a single learning step:
Assuming the step size to be small, the left-hand side of this equation can be expanded as
and the integrand from the right-hand side as
Setting the two sides of the equation equal to one another then gives
where
and
Using (1), these last two equations are
where denotes an average over the stochasticity of a single update at fixed (for SGD, over the draw of a batch). (Note that, in the case of reinforcement learning, this would additionally include averaging over exploratory noise in the network.) Below we write for an average over , and and for covariances and variances with respect to ; for example, . For finite ensembles of trained networks, and denote the sample mean and covariance over replicas.
Thus, dropping the higher-order terms, (5) gives the following Fokker-Planck-like description of the dynamics of the parameter vector:
However, note that we have kept the term in the expansion, which is nonstandard in the ordinary Fokker-Planck derivation but, as we will see below, is crucial for accounting for the effects induced by finite step size.
Related to this, we can see that the two terms on the right-hand side of (10) scale with different powers of the small parameter . This arises from the fact that both the deterministic and stochastic parts of the update equation scale in (1), which is in contrast to classical theories such as Brownian motion, where the two terms are taken to scale as and , respectively, leading to an equation in which the drift and diffusion terms are both of the same order in . Hence, given the update equation (1) that defines iterative learning, the next-to-leading order expansion in (10) is the simplest consistent evolution equation for that accounts for noise-related effects arising from input (or output) noise and finite step size.
Rather than attempting to characterize the evolution of the full distribution , we can gain more insight by analyzing the time evolution of the first two moments. To do this, we multiply both sides of (10) by powers of and integrate. For the first moment, this gives
Integrating this by parts then gives
Similarly, we can multiply by and integrate by parts to obtain
We will comment on the limitations associated with this choice in discussing our empirical results below.
As is typically done in continuous-time formulations of gradient descent, we can now let . Doing this and defining
Equations (12) and (13) become
and
While, by setting in our equations, we do assume that is small, we do not take the limit as in the commonly studied case of gradient flow. Rather, we assume only that , such that terms in the above equations are interesting, while terms can be neglected. At this point we can see the justification for keeping the term on the left-hand side of (10): given the definition of the update equation, this scheme keeps terms up to the same order on both sides of our dynamics equations.
Combining (15) and (16), we can obtain an equation for the parameter covariance, which gives the following closed pair of dynamical equations for the first two moments:
Next, we can expand in the fluctuations of the parameters about their mean values:
where we have defined . Truncating at second order in amounts to a Gaussian closure, whereby third and higher cumulants of are neglected, as are terms of . This requires fluctuations to be small on the scale over which varies. § 2.3 and § 2.4 examine where this fails.
If we also define , , and , the evolution equations (17) become
and
These two equations describe the coupled flow of the parameter mean and covariance for a general stochastic learning rule.
At leading order, near a fixed point and with Poisson-distributed update times, (19)–(20) reduce to the moment equations of Ref. [28]. Two features distinguish the results here from that earlier work, however. First, the terms in (20), including , are retained. Second, rather than being random, the update times are fixed, which enters through : its term converts into the covariance in (17). With Poisson timing, by contrast, even full-batch gradient descent would acquire covariance from the randomness of the update times.
The general results (19) and (20) enable us to analyze the dynamics of learning for arbitrary architectures (without assuming infinite-width networks) and learning rules, where the noise arises both from finite step size and from noise in the update rule itself due to sampling noise and/or exploratory noise. In the language of physics, (19) and (20) describe the motion of very light particles subject to large viscous drag forces as well as externally applied forces (the right-hand sides of the two equations). The first effect of is to endow these particles with a small mass, . As we will analyze in detail in the following section, the other, more consequential effect is to subject the particles to additional forces that can qualitatively change the ways in which their trajectories evolve.
Evolution equations for stochastic gradient descent
In stochastic gradient descent (SGD), the update rule is given by , where is the loss function defined from a batch of data (and hence is a stochastic variable). If we additionally define as the Hessian of the loss function, the dynamical equations (19) and (20) become
and
Equations (21) and (22) show that nontrivial parameter dynamics during learning arise not only from the gradient of the loss function, as captured by the gradient-flow equation , but also through its local curvature, which is encoded in the Hessian . To understand the dynamics of , we can separately consider the effects of each of the terms on the right hand side of (22).
Because is a dynamical variable independent of , , with the gradient taken at fixed , so that (21) and (22) can be written more compactly as
and
Equation (23) shows that the parameters evolve under SGD via gradient descent on an effective loss function:
The dynamic regularization term in this loss function modifies gradient descent by pushing the solution away from regions of large positive curvature and toward flat valleys in the loss landscape. The details of its effects, however, depend crucially on the dynamics of arising from (24), where these dynamics are themselves driven by the curvature of the loss landscape.
The first line of (24) is driven by the geometry of the loss landscape. To see this clearly, consider the full-batch limit, in which , and the second line of (24) vanishes. Written in the basis in which is diagonal, we have , where , and . Equation (24) then becomes
For a quadratic loss, (26) is exact rather than truncated: its right-hand side equals , the one-step change of under gradient descent divided by the step size. The only approximation is replacing this finite difference with . The instability at is therefore located exactly, while growth and decay rates near it are approximate when is not small. In the limit , (26) shows that modes of aligned with directions of positive loss curvature are pushed to exponentially shrink, while those aligned with directions of negative loss curvature are pushed to exponentially grow, with the eigenvalues of controlling the rate of convergence or divergence. In the case where and the loss is convex, we can see that , and our equations reduce to gradient flow, with . However, if the loss is non-convex, we see that is amplified in regions of negative curvature even in the limit , which, as noted in previous work [36, 1], means that nearby trajectories diverge transiently along negative-curvature directions encountered during training.
While the term pulls the parameters down into valleys, (26) shows that introducing finite step size makes these valleys unstable once they become too narrow, providing a destabilizing mechanism of escape from saddle points and local minima. In particular, (26) shows that the dynamics near a local minimum become unstable if , providing a mechanism for the network to avoid converging in sharp local minima. This is the edge of stability criterion [37, 10]: the parameter covariance along any particular direction is stable if and only if the corresponding Hessian eigenvalue satisfies . We analyze this more fully in § 2.5 below. Together, the terms in the first line of (24) describe a competition between terms that push and pull the parameters toward and away from loss minima.
These dynamics are illustrated in a one-dimensional example in Figure 1. In this one-dimensional loss landscape for an ensemble of one-parameter networks with nearly identical initialization, gradient descent with a small learning rate causes the parameter to converge to a sharp local minimum, while larger learning rates enable the parameter to escape the local minimum and converge to a broader, globally optimal solution (Figure 1A-C). By relating the learning dynamics to the curvature of the loss landscape, the SLF equations characterize the mechanism for this escape. Because the curvature at the local minimum exceeds the critical value when is sufficiently large (Figure 1D), the parameter covariance diverges upon approach to the local minimum, providing a mechanism for escape (Figure 1E) (whereupon the SLF solution, which diverges as the ensemble escapes, is restarted from the simulated moments—Methods). This escape corresponds to a smoothing out of the effective loss function, which includes the implicit regularization term that is dynamically activated in regions of high curvature (Figure 1F). Together, this simple example illustrates that, even in the absence of batch noise, the finite-step-size corrections captured by SLF provide a mechanism of escape from sharp local minima during learning.
Figure 1. Curvature and parameter covariance determine effective loss landscape. (A) A one-dimensional loss landscape with initial parameter value . (B) Loss converges to a local or global minimum under gradient descent (dotted lines: simulations; solid lines: theory) with two different learning rates. (C) Parameters converge to a local or global minimum. (D) Curvature of the loss function; dashed lines are critical values for each of the learning rates. (E) Covariance of the parameter ensemble under gradient descent, given initial variance . (F) Effective loss along the trajectory of the ensemble mean (dotted), compared with (gray).
The effects of batch noise, i.e. the random draws of data in each batch, are captured by the terms in the second line of (24), as well as by fluctuations in the term . To begin, consider the term , which characterizes the variance of the gradients computed from different batches of data. Previous work has shown that, early in training, this covariance is relatively isotropic, with gradients from different batches pushing in different directions [38]. Later in training, if the loss is a negative log-likelihood and the model approximately matches the data distribution, the Fisher identity (Appendix § 4) implies [38, 39, 40]. This condition does not always hold, however. With noiseless targets, for example, vanishes at an interpolating minimum. For this reason we evaluate directly in all numerical experiments. Unlike the curvature term described above, this term always has the effect of pushing the parameters out of regions of large positive curvature. The appearance of the coefficient is consistent with previous work that have found this ratio to be a key determinant of SGD dynamics, with higher values leading the parameters to wider final minima [5].
The last term on the right hand side of (22), , shows that, in the presence of batch noise and finite step size, higher derivatives of the loss function can influence the parameter dynamics. To gain insight into this force arising from finite step size, we can consider it along just one dimension:
The sign of this term is set by how the batch gradient covaries with the batch third derivative across batches, not by the sign of itself, so it can either amplify or damp the covariance. Its counterpart in the mean dynamics (22) has a definite effect, however. In one dimension, , where is the curvature, so the covariance drives the mean toward lower curvature at a rate proportional to . This is the mechanism by which the covariance feeds back on the sharpness, which we analyze in § 2.5.
Together, the dynamical equation for the parameter covariance gives a rich picture of SGD dynamics that is entirely absent from the gradient-flow approximation. In particular, it describes a tug of war between the terms, which push the parameters down into valleys of the loss function; the SGD noise term, which provides exploratory noise early in training followed by a mechanism for escaping local minima later in training; and the remaining terms, in which batch noise provides mechanisms for escape from sharp minima.
Stochastic Learning Flow in a two-layer linear network
To test whether the SLF equations capture the noisy dynamics of learning in a setting where the loss landscape is understood analytically, we first applied them to a two-layer deep linear network (DLN) trained with SGD. Although its input–output map is linear, the DLN loss is nonconvex, and its critical points and gradient-flow trajectories are known in closed form [1, 2, 41]. From small initialization, the network learns the singular modes of the task one at a time, with each mode learned in a rapid transition between a saddle point and the next critical point. We used the smallest network that exhibits all of these features: units per layer, trained on a task with two distinct singular values, so that the network passes through two saddle points before reaching a minimum. With parameters, the full Hessian and parameter covariance could be computed and followed mode by mode, and the averages over inputs in {(xref)}(23)–(24) could be evaluated exactly (Methods). The network mapped isotropic inputs to outputs , which were trained to match targets by minimizing the mean squared error .
We computed the parameter mean and covariance from the SLF equations {(xref)}(23)–(24) as well as in the gradient-flow (GF) limit. The results were compared against an ensemble of identically initialized simulated networks trained with SGD all starting from the same initial condition, where stochastic noise during learning led to a distribution of parameters across this ensemble of networks, characterized by the empirical mean and covariance .
While the expected SGD loss curve was described well by either SLF or GF in DLNs trained with small and/or large batch size , SLF captured the SGD learning curve much better than GF as was increased and/or was decreased (Figure 2A-C). In addition, while the parameter update direction under either SLF or GF was well aligned with the SGD update direction for small and/or large , only SLF was able to capture the SGD parameter updates throughout learning in the noisy learning regime (Figure 2D-F). Together, these results show that SLF captures the loss curves and mean parameter dynamics of SGD much more faithfully than GF, particularly in the high-noise regime of large and small .
Figure 2. Stochastic Learning Flow captures learning curves and parameter changes in a two-layer deep linear network. (A)-(B) Expected loss during training of an ensemble of networks trained with SGD from a single initial condition, as well as corresponding gradient-flow (GF) and Stochastic Learning Flow (SLF) curves in the low-noise (A) and high-noise (B) regimes. (C) The normalized root mean square error (NRMSE) of the SLF vs. SGD and GF vs. SGD loss curves. (D)-(E) Alignment of the ensemble-averaged SGD parameter update direction with the update direction predicted by either GF or SLF in the low-noise (D) and high-noise (E) regimes. (F) The minimum update alignment values during training for various learning rates and batch sizes. Points in (C) and (F) are mean std across identically initialized networks. Open circles denote settings for which is strongly non-Gaussian, limiting the applicability of the SLF theory (cf. Figure 3D).
To understand the stochastic mechanisms responsible for these learning dynamics, we next investigated the behavior of the parameter covariance matrix as the DLN learned each of the two singular modes of the task matrix (Figure 3A). As in previous work, we found that learning each of these modes corresponded with a change in sign of one or more Hessian eigenvalues from negative to positive, indicative of transitions through a sequence of saddle points and ultimately to a minimum [1, 2, 41, 42, 43] (Figure 3B). Projecting the parameter covariance along these Hessian modes revealed a sequence of transient expansions along negative-curvature directions, followed by contractions along these directions as the curvature changed sign (Figure 3C). While the timings of these transient increases precisely matched those in an ensemble of SGD-trained networks, we found that the peak covariance amplitudes from the SLF equations exceeded the SGD ensemble covariance, and that this overshoot grew with the third moment of the SGD ensemble, which itself increased as batch size was reduced (Figure 3C-D). Given and the time-varying Hessian, we were able to construct the implicit loss regularization from (25), which exhibited transient peaks as the network passed through each saddle point (Figure 3E). Across a range of learning rates and batch sizes, the dynamics of the implicit regularization term in the integrated flow equations quantitatively matched the SGD ensemble (Figure 3F). Comparing Figures 3 and 2 shows that, in the noisy regime of large and small , the magnitude of the regularization term comes within an order of magnitude of the MSE loss. Together, these results show that the dynamics of the parameter covariance can be predicted by the SLF theory for small and intermediate levels of batch noise and learning rates, and that is accompanied by growing non-Gaussianity of the parameter distribution.

Figure 3. Curvature and covariance dynamics in a deep linear network. (A) An ensemble of identically initialized two-layer DLNs learn two task modes with SGD (dotted); learning dynamics are characterized by SLF (solid) and GF (dashed), with vertical lines denoting times of mode learning. (B) Eigenvalues of the Hessian during training (colored: output-potent; gray: output-null). (C) Parameter covariance from the SGD ensemble projected onto the Hessian eigenmodes (dotted), together with the SLF covariance (colored: mode-projected; gray: total). (D) Relative overshoot in the SLF estimate of the covariance peak in (C) increases with the non-Gaussian third moment of the parameter distribution as batch size decreases (colors as in panel F). (E) The implicit regularization term from the effective loss function during training. (F) The peak values of the implicit regularization during training. Points in (D,F) are means std across 5 identically initialized networks. Open circles denote settings for which is strongly non-Gaussian, limiting the applicability of the SLF theory.
Stochastic Learning Flow in a nonlinear network
To test the ability of the SLF equations to characterize learning dynamics in a nonlinear network, where less is known analytically about the loss landscape and learning dynamics, we next trained a network to perform a binary classification task. Because validation of the theory requires a large number of replicas to be trained for each setting of parameters and hyperparameters, we implemented a small-scale two-layer classifier on two digits (0 vs. 1) from the MNIST dataset. Each test began from a common parameter vector reached partway through training, so that and the covariance was generated entirely by minibatch noise, and followed the ensemble for 200 steps, beyond which the Gaussian approximation began to break down at intermediate batch sizes (Methods).
We first asked whether including the covariance in the parameter flow equations increased the ability of the model to predict the evolution of the parameters over time. The error during training was computed by comparing the mean parameter vector from the integrated SLF equations to the ensemble mean from a set of SGD-trained networks with identical initializations (Figure 4A). Integrating the coupled SLF equations (23)-(24) yielded parameters that were within the estimation noise floor from the finite SGD ensemble (Figure 4A, black line). Further, computing from (23) using the measured from the SGD ensemble rather than the integrated from (24) led to equally good prediction (Figure 4A, blue line). In contrast, setting in the flow equation impaired the prediction (Figure 4A, red line), with the size of this effect growing with decreasing batch size (Figure 4B), showing that, in the regimes we tested, arising from batch noise plays a small but significant role in determining the parameter evolution.
Figure 4. Stochastic Learning Flow in a nonlinear network. (A) In a two-layer network trained on binary MNIST, the integrated SLF equations (black) predict the mean parameter trajectory of an ensemble of SGD-trained replicas to within estimation noise (gray). The prediction is unchanged if SLF uses the replica covariance rather than the integrated covariance (blue), but degrades if the parameter covariance is neglected in the flow equations (red). (B) Ratio of the parameter error at step assuming (cf. red line in (A)) to the error of the full SLF equations (cf. black line in (A)) (6 initial conditions at each batch size; bars are medians). (C) Ratio of predicted covariance trace to measured covariance trace as a function of the measured covariance trace, which increases with decreasing batch size (for each , points correspond to optimizer steps and network initializations). (D) Relative covariance error vs. Hill index of the distribution of losses across replicas, quantifying heaviness of the distribution tail (points are 10 timesteps over from a single network for each ; dashed line: index 2, where the 2nd moment of the distribution diverges; colored lines: median sampling precision of in cases where they can be estimated reliably).
We next quantified the extent to which the parameter covariance predicted by the SLF equations matched the empirical parameter covariance measured across the SGD ensemble. Because of the high dimensionality of the parameter space, which makes comparing covariance matrices using a finite number of observations difficult, we quantified the similarity of the overall variance by computing the traces of the covariance matrices. At low and intermediate levels of batch noise, we found that the SLF equations successfully predicted the overall variance to within 10% across two orders of magnitude (Figure 4C).
We hypothesized that increased error in the variance prediction relative to measured values at higher levels of batch noise might be, as in the DLN studied in § 2.3, associated with non-Gaussian features of the parameter distribution emerging through training. Because of the higher dimensionality of the parameter space in this nonlinear network, higher moments of the parameter distribution itself could not be accurately computed from the finite number of SGD replicates used in our simulation. Hence, we considered the one-dimensional distribution of losses over the first 50 steps of training and used the Hill index to quantify the heaviness of the distribution’s tail, where an index value of indicates divergence of the distribution’s second moment. Heavy tails are expected because multiplication by random batch Hessians makes SGD a random-coefficient recursion, whose stationary distributions are generically power-law [44, 45, 46]. The error of the predicted variance grew as the Hill index fell (D). In cases where the 2nd moment existed and the sampling precision of could be reliably estimated, we found that this sampling precision closely tracked the variance prediction error, indicating that the Gaussian closure and the variance estimator fail together because both are second-moment objects, and consistent with both being driven by heavy tails.
Together, these results show that there exists an intermediate regime of batch noise in which SLF successfully predicts parameters and their covariances during training, and that the breakdown of these predictions as batch noise increases further is associated with non-Gaussian effects.
Edge of Stability as a dynamical stationary point
Previous work [10] found empirically that deep networks trained with full-batch gradient descent operate at the Edge of Stability (EoS), with the maximum eigenvalue of the loss Hessian rising under training and then hovering near . Two opposing forces contribute to this phenomenon. The first, progressive sharpening, is the tendency of loss curvature to increase under gradient descent, pushing the solution toward the threshold from below [10, 40]. The second arises from finite step size and forces the solution out of regions of high curvature [8, 10, 18], as seen in Figure 1D-E. In this section, we show that the EoS emerges as a stationary point of the coupled dynamics (23)–(24) in the full-batch limit, and that it is shifted downward by noise arising from finite batch size.
To begin, consider full-batch gradient descent and project along the direction of sharpest curvature, for which , where is the largest eigenvalue of . Let be the covariance, where covariance along other directions is suppressed by (26). Then
where, because is normalized, we used . In this case, (24) gives
This equation shows that defines the EoS, with increasing (decreasing) for sharpness above (below) this value. In particular, the existence of this critical value depends on the finite-step-size term , which arises from the term in the covariance flow and is absent in first-order SDE approaches. Coupled to these dynamics are the dynamics of itself, which evolves during learning as . Using the fact that , where , this gives
Dropping the terms of , this is
where we have defined
Here is the rate of progressive sharpening under gradient flow, which previous empirical and theoretical work has demonstrated tends to be positive in multilayer networks [10, 40]. The second term in (31) shows that covariance back-reaction (which, in full-batch learning, corresponds to period-2 oscillations) tends to counteract progressive sharpening, providing the mechanism for escape from sharp minima; this is the self-stabilization mechanism of Ref. [18], expressed through the covariance. The third term is the curvature of the sharpness landscape along the direction of parameter drift and arises due to finite step size.
The dynamical equations (29) and (31) exhibit a stationary point at
In full-batch gradient descent, is the variance of an ensemble of trajectories or, for a single trajectory, the squared amplitude of its period-2 oscillation about the two-step average . Because is unstable for by (29), any initial spread, however small, grows until it reaches . (33) shows that the EoS threshold corresponds to the unique stationary state of the covariance dynamics.
Further, we note that gradient flow corresponds to the limit of this theory, in which case remains 0 by (29), and the curvature evolves according to , which possesses no mechanism to arrest progressive sharpening.
For this is central flow [19], in which the mean descends on , the effective loss (25) with , and is fixed by imposing at . Here the same values follow from the stationary state of the covariance dynamics rather than being imposed, and (29)–(30) also describe the approach to the edge. Central flow corresponds to the limit in which the relaxation of is replaced by a constraint.
In the case where the batch size is finite, the dynamical equations for the parameter covariance and loss curvature contain corrections due to the effects of batch noise. The same projection that was used above in this case gives
where
Here we have defined
where is the batch curvature, and the sample curvature satisfies . Letting , the statistics of are then defined by and . Here , and denote averages over the training samples, and and averages over batches, so that .
The variance along remains finite only while , so batch noise lowers the threshold at which it diverges to the point where , which to first order in is
consistent with the observation that SGD hovers below [10, 17]. The first term in (37) is the mean-square linear stability threshold of SGD projected onto a single direction. That batch-to-batch curvature fluctuations lower this threshold was identified in Ref. [20], and the exact mean-square condition was derived (assuming batches sampled without replacement) in Ref. [21]. At each step, the parameter variance along is multiplied by , which is convex in , so that fluctuations in cause marginal stability to arrive earlier. The second term arises from the coupling of gradient noise to the third derivative of the loss; it vanishes at interpolating minima, where every per-sample gradient is zero, which is why it is absent from the linear stability analysis of Ref. [21]. Supplementary Figure S1 shows that, while (37) successfully captures the reduction in due to batch noise in a nonlinear network using a probe test, versions of (37) with vs. without the second term were identical to within , indicating that the term plays a comparatively small role in determining the EoS in the network we tested.
Figure S1. The edge of stability is shifted by batch noise. Mean-square EoS threshold calculated in a two-layer network from (37), either with (solid line) or without (dashed line) the second term from the equation. Points show critical curvature based on a one-step probe estimator in the same network. Lower panel shows the relative error between the measured and calculated values. Error bars in the top panel and shading in the lower panel show std of the probe measurement.
Two limitations apply to the result in (37). First, is a mean-square threshold—beyond it the variance diverges, while typical trajectories can remain bounded up to a Lyapunov threshold lying, to leading order, as far above as lies below it [44, 47]. Second, because the back-reaction in (30) is the rank-one contraction of , which depends on the whole covariance once batch noise populates its other modes, the noise amplitude in (33) does not carry over to finite .
Together, the results in this section show that the EoS can be understood as a dynamical stationary point of the SLF equations in the full-batch limit, and that the EoS stability threshold is shifted downward by noise arising from finite batch size.
Discussion
We have derived a set of coupled dynamical equations describing the evolution of the first two moments of the parameter distribution over an ensemble of networks, where these moments include noise effects arising from random sampling of inputs and finite step size. Applying these equations to SGD, we found that the theory describes gradient flow on an effective loss function that couples the parameter mean and covariance, the evolution of which depends on the loss curvature together with noise effects arising from finite step size and batch noise. Applying the flow equations to deep linear and nonlinear networks, we found that they enable successful prediction of loss curves, parameter updates, and parameter covariance at low and intermediate levels of noise, and we identified the regime in which the predictions begin to break down as non-Gaussian features of the parameter distribution become more pronounced. Finally, in the full-batch limit of gradient descent, the theory recovers the edge of stability as a stationary point of a pair of coupled equations describing the dynamics along the sharpest direction in the loss landscape, while, for finite batch size, the theory predicts the lowering of the EoS stability threshold as batch noise increases.
An important limitation of the SLF theory is the truncation of the moment equations at second order. As shown in Figure 3 and Figure 4, the emergence of non-Gaussian features in the parameter and loss distributions coincides with a decrease in the ability of SLF to predict the dynamics of the distribution. While addressing this by extending the approach of (11) and (13) to include higher moments would be possible in principle and could extend the ability of the theory to capture dynamics as the distribution becomes non-Gaussian, the complexity of the resulting equations would make interpretation more difficult, and the theory would still be unable to capture the possible fragmentation of the parameter distribution into multiple modes as replicas settle into different minima later in training. Future work could address this by treating the few directions along which non-Gaussian structure develops separately, for example with a reduced Fokker–Planck description or a description of the tails in the spirit of random-coefficient recursions [44], while retaining the second-moment description elsewhere. For the EoS theory, an additional limitation is the projection of the parameter dynamics along a single direction. While this is justified in the case of full-batch gradient descent, our results for the case of SGD show that the sharp mode becomes coupled to other modes due to the injection of variance along additional directions arising from batch noise. Developing a dynamical theory to fully capture EoS phenomena under SGD by including this mode coupling remains an important direction for future work. Finally, due to the high computational cost of solving the flow equations, which requires covariance matrices and dense Hessian computations, the networks tested here have at most a few thousand parameters, so whether the regime of validity identified in Sections 2.3-2.4 persists in networks with more than a few thousand parameters remains to be determined.
A widely used approach models SGD by an SDE whose drift is the full-batch gradient and whose diffusion is the minibatch gradient covariance [11, 5, 12, 48, 49]. Several of these analyses fix the diffusion to be constant near an optimum [11] or proportional to the Hessian [5], which removes the batch-curvature fluctuation and cross terms of (24). Other approaches retain its dependence on [12, 48, 49], which generates these terms. Li et al. [13] showed that this SDE is a first-order weak approximation of SGD and derived a second-order one in which the drift descends on , which was also derived by backward error analysis and analyzed numerically in Ref. [14] and which can be obtained in SLF by eliminating the term of (23) perturbatively at . These approximations have since been refined [50].
What all of these SDE descriptions lack is the deterministic finite-step part of the covariance dynamics: the term and the operator . The difference matters where is not small. The second-order SDE expands the one-step amplification of a perturbation in powers of , a series that converges only for , whereas the SLF covariance equation retains the one-step change of the second moment, which for a quadratic loss is exact in . Accordingly, near a quadratic minimum the stationary solution of (24) coincides with the exact stationary covariance of discrete-time SGD [30], including its divergence at . Moment relations obtained directly from the discrete update have also been used to derive fluctuation–dissipation relations at stationarity [29]; SLF extends this approach to the full time-dependent dynamics. This is why the SLF equations contain the edge-of-stability threshold, and may explain why SDE surrogates of SGD, even simulated at finite step size, do not stabilize where SGD does [17]. Kunin et al. [16] describe the late phase of SGD with an underdamped equation whose inertial term, for plain SGD, coincides with that of . Because they apply it to individual trajectories rather than to the parameter distribution, however, their covariance dynamics lack and the instability at .
The SLF equations bring together several existing descriptions of the edge of stability. The back-reaction of the covariance on the sharpness is the self-stabilization mechanism of Damian et al. [18], expressed through the covariance; central flow [19] is the full-batch limit in which the relaxation of the covariance is replaced by a constraint; and the batch-size shift of the threshold coincides with the mean-square linear stability analysis of SGD [21], with an additional term from the coupling of gradient noise to the third derivative of the loss. Liao et al. [22] extended self-stabilization to SGD and derived a closed-form displacement of the sharpness below from additive gradient noise, with the threshold held at . In our framework that displacement depends on the whole covariance once batch noise populates modes other than the sharpest, so a description in terms of the sharp direction alone is incomplete at finite batch size. Two further questions remain open: where between the mean-square and Lyapunov thresholds trajectories settle [47], and how additional modes are recruited to the edge, as observed empirically [10].
While we have analyzed SLF for the particular case of SGD, the general flow equations (19)–(20) apply to any stochastic update rule. Characterizing how the flow of the parameter distribution depends on the learning rule is a potentially fruitful direction for future work. This includes momentum and adaptive methods, in which momentum is known to change the inertial term [16], and reinforcement learning, in which exploratory noise adds to sampling noise. It also includes regularization terms added to the loss, and learning-rate schedules, for which the identification used here would have to be generalized. In addition, future work can use the theory to characterize the evolution of hidden-layer representations in multilayer networks, and how these learned representations depend on SGD noise, the learning rule and the loss function. The implicit regularization term in (25) also connects the covariance dynamics to the long-standing question of how SGD noise biases training toward flat regions of the loss landscape, which the theory could help address quantitatively. Together, the framework presented here treats the parameter covariance as a dynamical variable on the same footing as the mean, and so places the effects of finite step size and sampling noise within reach of the tools of dynamical systems theory that have been used to analyze gradient flow.
Geometry of the gradient noise
To see the relationship between the SGD noise term in (22) and the Hessian, we can make use of the Fisher identity [51]:
where we have assumed that the model loss can be expressed as a negative log likelihood . To see this, we write
which, using , is exactly the Fisher identity (38).
Next, if we further assume that the model is close to convergence, such that , where , we have . Applying this to and making use of (38) then gives
Finally, given the definition , we have
This shows that, when the model is close to convergence, the SGD noise is proportional to the Hessian.
Methods
One-dimensional example (Figure 1)
Loss and ensembles. The loss was
with , , and . All derivatives of were evaluated by central differences with step . For each , an ensemble of replicas was initialized from and trained by full-batch gradient descent for 50 steps.
SLF equations. With , the one-dimensional, noise-free form of (23)–(24) becomes for , with the right-hand side. This system was integrated by fourth-order Runge–Kutta with step from the ensemble’s initial sample mean and variance with zero initial velocities. Integration was halted when , left blank for three steps, and restarted once from the simulated moments and their finite-difference -derivatives. At this occurred at step 4, with the restart at step 7; at it did not occur. Panel F shows from the simulated ensemble.
Deep linear network (Figures 2 and 3)
Network and task. The network had units per layer, , so had components. Inputs were drawn as and targets were with , , and random orthogonal matrices drawn once with a fixed seed. Training was online, and each step used a fresh batch of inputs, with batch loss , where and . Every run started from the same balanced initialization, and , with and random orthogonal matrices drawn once with a fixed seed. Runs lasted to .
SGD ensembles. Each ensemble comprised replicas started from the initial condition above, each with an independent stream of batches. No replica diverged in any of the runs reported. The empirical mean and covariance were recorded at every step. The SGD expected loss was the mean over replicas of the population loss , after discarding the largest 1% of values.
SLF and GF. The SLF equations (23)–(24) were integrated with population averages over evaluated in closed form. Because and are linear in , the moments and give , , , , and the cross term exactly. With , each equation was written as a first-order system, , , and likewise for , where and are the right-hand sides of (23)–(24). The system was integrated with an adaptive Runge–Kutta solver (RK45; relative and absolute tolerances and ) over one interval at a time, from and , with initial velocities and . The loss depends on only through , so it is invariant under , for any invertible . At the end of each interval, and were therefore projected onto the -dimensional complement of the tangent space of this symmetry at . The SLF expected loss was . GF solved with (RK45; tolerances and ). All three descriptions were compared on the grid .
Figure 2. For each , SLF and GF were run once and the SGD ensemble five times with independent batch streams. The normalized root mean square error was , where is the expected loss predicted by SLF or GF, is the SGD expected loss, and the bar and denote the mean and standard deviation over all steps. For the alignment, consecutive updates and were projected onto the physical subspace at before their cosine was taken. Steps at which the SGD update norm was below of its maximum were discarded, and the cosine was averaged over a moving window of 0.05 time units. Panel F shows the minimum of this smoothed cosine over the run.
Figure 3. Panels A–C and E used . Mode amplitudes were , with the columns of . Vertical lines mark the times at which the SGD ensemble mean first reached . Eigenvalues and eigenvectors of were computed within the physical subspace at each step, and branches were followed through crossings by maximizing the absolute overlap of eigenvectors between consecutive steps (Hungarian assignment). Panel C shows for SLF and, with eigenvectors computed at the ensemble mean, for the gauge-projected . In panel E, the SGD value was with the gauge-projected ensemble covariance and evaluated at the ensemble mean. Panels D and F used the sweep of Figure 2. The non-Gaussianity of each SGD ensemble was measured from the mode amplitudes of its replicas, . At time , we computed , where the bar denotes the average over replicas after discarding the 1% most extreme values of at each end. The value reported for an ensemble is the peak , evaluated at every -th of the steps. The ordinate of panel D is , with the ensemble covariance gauge-projected. Cells with ratio below 1 are omitted. The line is a least-squares fit over cells with . The gray band is an estimate of the finite- floor, spanning 0.5 to 2 times the median ordinate over cells with . Open symbols in Figures 2C,F and 3D,F mark cells whose exceeds the value at which the fitted law gives an ordinate of 1, i.e. where the Gaussian-closure overshoot equals the quantity itself.
Nonlinear network (Figure 4)
Network and data. Digits 0 and 1 from the MNIST training set were used as inputs, with balanced examples, downsampled from to by average pooling () and standardised per pixel, and targets were . The network output was with hidden units, so . The GELU nonlinearity was chosen to ensure that second- and higher-order derivatives for theoretical analysis would be well defined. The per-sample loss was , and was its mean over a minibatch, with no weight decay. All computations were in float64.
Initial conditions and ensembles. Parameters were initialised at unit scale (seed 0) and trained by 150 steps of full-batch gradient descent at learning rate 0.05, giving a common warm start. For each seed and each , a parent ensemble of 256 SGD replicas was run from the warm start for 500 steps at the production , and one of its replicas, chosen at random, became the initial condition . Each segment then started all 256 replicas at , so and the covariance was generated entirely by minibatch noise during the segment. Minibatches were drawn independently for each replica, with replacement. The learning rate was fixed at ; one optimiser step advanced by , so a 200-step segment spanned .
Integration of the SLF equations. With , each equation was integrated as a first-order system, , with the right-hand side of (23), and likewise for , by fixed-step fourth-order Runge–Kutta with two substeps per optimiser step. was computed exactly at every optimiser step and interpolated linearly within it, as were the source terms of (24); the covariance-dependent source terms used from the previous optimiser step, and the correction was held fixed across each step. Initial velocities took their slow-manifold values, and its analogue for , so that the spurious fast mode of , at rate , was not excited. In a 50-step segment at , halving the substep changed by less than one part in .
Terms of (24). All terms were evaluated at . For minibatches drawn with replacement,
where ,
and . The fluctuation term was computed exactly by using the structure of the per-sample Hessian. The output depended on only through the scalars , and , each affine in , so had rank at most and , with , the residual, known in closed form and block diagonal. The same structure made the two terms involving exact. Because the scalars were affine in , the directions along which they varied did not depend on ; writing as a function of the scalars, with contracted onto those fixed directions, gave by automatic differentiation. Then the correction in (23) was , and the cross covariance in (24) was , where in the covariance was the minibatch Hessian. For the measured-covariance prediction the correction was instead averaged over the 256 replica displacements, which was exact for the sample covariance.
The segments in panels A-C were run with stochastic estimates of these two terms: 256 Gaussian draws for the correction, using , and 8 Gaussian probes for the cross term, with variances those of along the eigenvectors of and retained on the 64 eigenvectors of largest variance. Repeating the 200-step segment for one seed with the exact evaluation, on identical minibatch streams, changed by at most 0.18% at any step, by at most 0.07%, and the panel-B ratio from 1.1376 to 1.1373. All points at in panel D used the exact evaluation.
Error measures. Each prediction’s error was , where is the empirical ensemble mean, evaluated at every step. Panel B showed, for each segment, at step 200. This value lay within 2.5% of the segment’s median over steps 190-200 in every segment, so it was not a single noisy draw; the median over all 200 steps gave the same ordering, with medians 1.037, 1.077 and 1.134. The sampling floor was for disjoint halves and of the ensemble. The covariance traces were and the trace of the integrated ; the unbiased normalisation would lower every trace ratio in panels C and D by 0.39%. The sampling error of was the RMS over 200 random 128/128 partitions of , relative to , computed from the replica ensembles stored for panel D (below); the 1.3% quoted for panel C was its median over the states at . A replica whose parameters became non-finite was removed, and ensemble statistics were taken over the remainder.
Gauge. The loss was invariant under permutations of hidden units, so was a meaningful network only if the replicas shared one assignment of units. Aligning each replica to a consensus reference required a non-identity permutation for none of the 2560 replicas examined at each . GELU is not odd, so there was no sign symmetry to check.
Panel D and the breakdown at small . The segments at used the same protocol with seed 11. Replica ensembles were stored at 10 times during the first 50 steps at each of the six batch sizes. At each stored state, the tail index was the Hill estimator applied to the upper 10% of the distribution of across replicas, checked against a Gaussian surrogate with the same mean and covariance, and the trace error was read at the same step. At , came from repeating the 50-step segment with the exact evaluation described above; the replicas, and hence and the tail index, were identical between the two runs. Seven states, all at , were excluded because or was non-finite: the ensemble had diverged, and the eigendecomposition used in evaluating the fluctuation term failed. Indefiniteness of the integrated covariance was measured by its smallest eigenvalue divided by its largest. Where had diverged, was negligible and the plotted error saturated at 1. The horizontal lines were the median, over each batch size’s states, of the sampling error of at that state; a state was counted as exceeding its sampling error by comparison with its own value. That sampling error required the loss to have finite variance across replicas, so it was not defined below a tail index of 2, and the lines at and indicated only its scale.
Edge-of-stability threshold test (Figure S1)
Network and task. The network was a two-layer perceptron, , with inputs, hidden units and a scalar output (), trained on the loss over fixed training examples. The inputs were whitened exactly to a power-law covariance: with normalized to unit mean, the input matrix was , where is the orthonormal factor of a standard Gaussian matrix. Targets were produced by a random teacher of the same form with 12 hidden units (weights drawn from , hidden biases from , output bias zero), standardized to zero mean and unit variance and multiplied by 4. The student was initialized with weights drawn from and zero biases. The learning rate was , so .
Base points. Full-batch gradient descent was run for 1500 steps from this initialization. At each step we formed the bounce average and computed and of the full-batch Hessian by Lanczos iteration. The interval , truncated at the largest reached, was divided into 44 equal bins in . From each non-empty bin, the middle (in time) of the steps falling in that bin was taken as a base point, giving 34 base points between steps 12 and 1377 with from 13.31 to 21.47. At each base point we computed the per-sample curvatures , and of (36), by automatic differentiation.
Probe. At each base point and each , we displaced antithetic pairs of replicas to , with . Both members of a pair took one SGD step with the same batch of examples, drawn uniformly with replacement, and different pairs used independent batches. The half difference of their displacements along , , cancels the gradient-noise source exactly and gives the mean-square growth rate , whose zero is with . The standard error of was propagated from the sample variance of . The measured threshold at each was the largest at which , ordered by , changed sign, located by linear interpolation between the two bracketing base points. Its error combined the standard errors of at those points, divided by the slope of the interpolant. This error was then multiplied by 1.710, the root mean square of the normalized differences between these thresholds and those from a repeat of the sweep at –128 with different random draws.
Prediction and comparison. The predicted threshold was , with linearly interpolated in between base points and evaluated at the measured threshold. The curve including the second term of (37) subtracts , with interpolated in the same way. Residuals are . The reported is the mean of the squared normalized residuals over the seven batch sizes, with no fitted parameters.
Use of AI tools
The theoretical framework, the derivation of the SLF equations, and the design of the study are the authors’ own. In developing the numerical experiments and the manuscript, we made use of AI assistants based on Anthropic’s Claude model Opus 5.5. AI coding agents wrote and ran simulation and analysis code for the DLN and nonlinear-network experiments under specifications made by the authors. An AI assistant also helped interpret results, check derivations, survey related literature, and edit parts of the text. The authors reviewed, verified, and revised all AI-generated code, analyses, and text, and take full responsibility for the content of the paper.
Supplemental Information
References
- [1]Andrew M Saxe, James L McClelland, and Surya Ganguli. “Exact solutions to the nonlinear dynamics of learning in deep linear neural networks”. arXiv preprint arXiv:1312.6120 (2013).arxiv.org/abs/1312.6120
- [2]Andrew M Saxe, James L McClelland, and Surya Ganguli. “A mathematical theory of semantic development in deep neural networks”. Proceedings of the National Academy of Sciences 116.23 (2019), pp. 11537–11546.DOI
- [3]Arthur Jacot, Franck Gabriel, and Clément Hongler. “Neural tangent kernel: Convergence and generalization in neural networks”. Advances in neural information processing systems 31 (2018).arxiv.org/abs/1806.07572
- [4]Nitish Shirish Keskar et al. “On large-batch training for deep learning: Generalization gap and sharp minima”. ICLR (2017).arxiv.org/abs/1609.04836
- [5]Stanisław Jastrzębski et al. “Three factors influencing minima in SGD”. arXiv preprint arXiv:1711.04623 (2017).arxiv.org/abs/1711.04623
- [6]Samuel L. Smith and Quoc V. Le. “A Bayesian Perspective on Generalization and Stochastic Gradient Descent”. International Conference on Learning Representations (ICLR). 2018. URL: https : //openreview.net/forum?id=BIji4yg0Z.arxiv.org/abs/1710.06451
- [7]Yuanzhi Li, Colin Wei, and Tengyu Ma. “Towards explaining the regularization effect of initial large learning rate in training neural networks”. Advances in neural information processing systems 32 (2019).arxiv.org/abs/1907.04595
- [8]Aitor Lewkowycz et al. “The large learning rate phase of deep learning: the catapult mechanism”. arXiv preprint arXiv:2003.02218 (2020).arxiv.org/abs/2003.02218
- [9]Mathieu Even et al. “(S) GD over Diagonal Linear Networks: Implicit bias, Large Stepsizes and Edge of Stability”. Advances in Neural Information Processing Systems 36 (2023), pp. 29406–29448.DOI
- [10]Jeremy Cohen et al. “Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability”. International Conference on Learning Representations. 2021.arxiv.org/abs/2103.00065
- [11]Stephan Mandt, Matthew Hoffman, and David Blei. “A variational analysis of stochastic gradient algorithms”. International conference on machine learning. PMLR. 2016, pp. 354–363.arxiv.org/abs/1602.02666
- [12]Pratik Chaudhari and Stefano Soatto. “Stochastic gradient descent performs variational inference, converges to limit cycles for deep networks”. 2018 Information Theory and Applications Workshop (ITA). IEEE. 2018, pp. 1–10.arxiv.org/abs/1710.11029
- [13]Qianxiao Li, Cheng Tai, et al. “Stochastic modified equations and adaptive stochastic gradient algorithms”. International Conference on Machine Learning. PMLR. 2017, pp. 2101–2110.
- [14]David GT Barrett and Benoit Dherin. “Implicit gradient regularization”. arXiv preprint arXiv:2009.11162 (2020).arxiv.org/abs/2009.11162
- [15]Samuel L Smith et al. “On the origin of implicit regularization in stochastic gradient descent”. ICLR (2021).arxiv.org/abs/2101.12176
- [16]Daniel Kunin et al. “The limiting dynamics of sgd: Modified loss, phase-space oscillations, and anomalous diffusion”. Neural Computation 36.1 (2023), pp. 151–174.
- [17]Arseniy Andreyev and Pierfrancesco Beneventano. “Edge of stochastic stability: Revisiting the edge of stability for sgd”. arXiv preprint arXiv:2412.20553 (2024).arxiv.org/abs/2412.20553
- [18]Alex Damian, Eshaan Nichani, and Jason D Lee. “Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stability.” ICLR 2023 (2023).arxiv.org/abs/2209.15594
- [19]Jeremy Cohen et al. “Understanding Optimization in Deep Learning with Central Flows”. The Thirteenth International Conference on Learning Representations. 2025.arxiv.org/abs/2410.24206
- [20]Lei Wu, Chao Ma, et al. “How sgd selects the global minima in over-parameterized learning: A dynamical stability perspective”. Advances in Neural Information Processing Systems 31 (2018).
- [21]Rotem Mulayoff and Tomer Michaeli. “Exact mean square linear stability analysis for SGD”. The Thirty Seventh Annual Conference on Learning Theory. PMLR. 2024, pp. 3915–3969.arxiv.org/abs/2306.07850
- [22]Fangshuo Liao, Afroditi Kolomvaki, and Anastasios Kyrillidis. “SGD at the Edge of Stability: The Stochastic Sharpness Gap”. arXiv preprint arXiv:2604.21016 (2026).arxiv.org/abs/2604.21016
- [23]Courtney Paquette et al. “SGD in the large: Average-case analysis, asymptotics, and stepsize criticality”. Conference on Learning Theory. PMLR. 2021, pp. 3548–3626.arxiv.org/abs/2102.04396
- [24]Courtney Paquette et al. “Homogenization of SGD in high-dimensions: exact dynamics and general-ization properties: C. Paquette et al.” Mathematical Programming 214.1 (2025), pp. 1–90.
- [25]Blake Bordelon and Cengiz Pehle van. “Learning curves for SGD on structured features”. ICLR (2022).arxiv.org/abs/2106.02713
- [26]Elizabeth Collins-Woodfin et al. “Hitting the high-dimensional notes: An ode for sgd learning dynamics on glms and multi-index models”. Information and Inference: A Journal of the IMA 13.4 (2024), iaae028.arxiv.org/abs/2308.08977
- [27]Todd Leen and John Moody. “Weight space probability densities in stochastic learning: I. Dynamics and equilibria”. Advances in neural information processing systems 5 (1992).
- [28]Tom M Heskes and Bert Kappen. “Learning processes in neural networks”. Physical Review A 44.4 (1991), p. 2718.pubmed.ncbi.nlm.nih.gov/9906254
- [29]Sho Yaida. “Fluctuation-dissipation relations for stochastic gradient descent”. ICLR (2019).arxiv.org/abs/1810.00004
- [30]Kangqiao Liu, Liu Ziyin, and Masahito Ueda. “Noise and fluctuation of finite learning rate stochastic gradient descent”. International Conference on Machine Learning. PMLR. 2021, pp. 7045–7056.arxiv.org/abs/2012.03636
- [31]Hannes Risken. “Fokker-planck equation”. The Fokker-Planck equation: methods of solution and applications. Springer, 1989, pp. 63–95.
- [32]Hendrik Anthony Kramers. “Brownian motion in a field of force and the diffusion model of chemical reactions”. physica 7.4 (1940), pp. 284–304.DOI
- [33]José Enrique Moyal. “Stochastic processes and statistical physics”. Journal of the Royal Statistical Society. Series B (Methodological) 11.2 (1949), pp. 150–210.DOI
- [34]James M Murray and G Sean Escola. “Remembrance of things practiced with fast and slow learning in cortical and subcortical pathways”. Nature Communications 11.1 (2020), p. 6441.
- [35]Christian Schmid and James M Murray. “Dynamics of supervised and reinforcement learning in the non-linear perceptron”. Advances in neural information processing systems 37 (2024), pp. 136424–136445.arxiv.org/abs/2409.03749
- [36]Yann N Dauphin et al. “Identifying and attacking the saddle point problem in high-dimensional non-convex optimization”. Advances in neural information processing systems 27 (2014).arxiv.org/abs/1406.2572
- [37]Yann Le Cun, Ido Kanter, and Sara A Solla. “Eigenvalues of covariance matrices: Application to neural-network learning”. Physical review letters 66.18 (1991), p. 2396.
- [38]Guy Gur-Ari, Daniel A Roberts, and Ethan Dyer. “Gradient descent happens in a tiny subspace”. arXiv preprint arXiv:1812.04754 (2018).
- [39]Zhanxing Zhu et al. “The Anisotropic Noise in Stochastic Gradient Descent: Its Behavior of Escaping from Sharp Minima and Regularization Effects”. International Conference on Machine Learning. PMLR. 2019, pp. 7654–7663.arxiv.org/abs/1803.00195
- [40]Valentin Thomas et al. “On the interplay between noise and curvature and its effect on optimization and generalization”. International Conference on Artificial Intelligence and Statistics. PMLR. 2020, pp. 3503–3513.arxiv.org/abs/1906.07774
- [41]Arthur Jacot et al. “Saddle-to-saddle dynamics in deep linear networks: Small initialization training, symmetry, and sparsity”. arXiv preprint arXiv:2106.15933 (2021).arxiv.org/abs/2106.15933
- [42]Scott Pesme and Nicolas Flammarion. “Saddle-to-saddle dynamics in diagonal linear networks”. Advances in Neural Information Processing Systems 36 (2023), pp. 7475–7505.
- [43]Yedi Zhang, Andrew Saxe, and Peter Latham. “Saddle-to-saddle dynamics explains a simplicity bias across neural network architectures”. International Conference on Learning Representations. Vol. 2026. 2026, pp. 57507–57551.arxiv.org/abs/2512.20607
- [44]Harry Kesten. “Random difference equations and renewal theory for products of random matrices”. Acta Mathematica 131 (1973).DOI
- [45]Mert Gurbuzbalaban, Umut Simsekli, and Lingjiong Zhu. “The heavy-tail phenomenon in SGD”. International Conference on Machine Learning. PMLR. 2021, pp. 3964–3975.
- [46]Liam Hodgkinson and Michael Mahoney. “Multiplicative noise and heavy tails in stochastic optimization”. International Conference on Machine Learning. PMLR. 2021, pp. 4262–4274.arxiv.org/abs/2006.06293
- [47]Liu Ziyin et al. “Type-ii saddles and probabilistic stability of stochastic gradient descent”. arXiv preprint arXiv:2303.13093 (2023).
- [48]Alnur Ali, Edgar Dobriban, and Ryan Tibshirani. “The implicit regularization of stochastic gradient flow for least squares”. International conference on machine learning. PMLR. 2020, pp. 233–244.arxiv.org/abs/2003.07802
- [49]Feng Chen et al. “Stochastic collapse: How gradient noise attracts sgd dynamics towards simpler subnetworks”. Advances in Neural Information Processing Systems 36 (2023), pp. 35027–35063.
- [50]Benjamin Gess, Sebastian Kassing, and Vitalii Konarovskyi. “Stochastic modified flows, mean-field limits and dynamics of stochastic gradient descent”. arXiv preprint arXiv:2302.07125 (2023).arxiv.org/abs/2302.07125
- [51]Erich Leo Lehmann and George Casella. Theory of point estimation. Springer, 1998.



