betaFeedback

WorksAgricultural and Biological Sciences

On the stability of learning

Abstract

Learning algorithms are usually analyzed under the assumption that informative gradients are continuously available. But this is not always true—learning signals in biological neural networks, for example, may be sparse, local, and only intermittently available at any given synapse. In this work, we show that if gradient information is noisy and only intermittently available, learning may be unstable. In particular, even if average learning dynamics converge to the correct optimum, variance in learning may be asymptotically infinite. We explore this point in a number of analytically tractable settings, including single-layer and multi-layer linear networks, and derive explicit conditions under which learning is stable. These conditions complement, but are conceptually distinct from, existing conditions related to the stability of stochastic gradient descent. Our work demonstrates that unbiased credit assignment is not enough for reliable learning; sufficiently frequent and informative feedback is also required, which yields an additional constraint on models of biologically plausible learning.

Introduction

The claim that the brain does not perform backpropagation [16] rests in part on the idea that gradient information may be difficult to communicate. In particular, the gradient of an objective function with respect to any given synaptic weight depends on the synaptic weights of a large number of downstream neurons, and it is not obvious in general how such non-local information could be reliably communicated [3, 2]. This issue has motivated a substantial body of work seeking alternative and more biologically plausible learning rules [1, 7], as well as schemes for approximately implementing backpropagation in specific contexts [4, 5, 6]. It remains unknown if and how this information is communicated, but it is generally agreed that communicating it accurately would be difficult given locality constraints.

In this paper, we consider a related but conceptually distinct issue. If gradient information can be communicated, it is not at all obvious that it can be communicated reliably. It is plausible that a scheme which delivers the correct updates on average only delivers them occasionally, to different neurons at different times, and perhaps somewhat noise-corrupted. How do learning dynamics change when gradients are noisy and only intermittently available? We argue that learning changes substantially, and in fact can become unstable if updates are not delivered sufficiently frequently.

This idea is interesting for a few reasons. First, it implies that credit assignment which is correct on average is insufficient, since there remains the additional obstacle that weight updates cannot be too variable. Second, instability is not a generic consequence of noisy weight dynamics, but rather depends crucially on the multiplicative, sparse nature of the noise introduced by assuming intermittent gradient arrivals.

We show that the severity of this issue can be reduced by a few obvious noise mitigation strategies, but that the presence of the issue (i.e., learning becoming unstable if feedback is sufficiently sparse) survives. This problem exists in both linear and nonlinear networks; in shallow and deep networks; and after averaging parameters or recent gradients. The details of the issue change, but the core finding remains: an informative signal must arrive sufficiently often for learning to remain stable.

We proceed as follows. We begin by motivating the simplest version of our model. Next, by analyzing it in detail, we show that there exist gradient-variance-dependent stability issues that are exacerbated as informative feedback becomes less frequent. We then discuss simple examples of this issue near the fixed points of more complex networks, and the extent to which basic noise mitigation strategies address it.

Sparse gradient arrivals can amplify errors

Figure 1. Sparse gradient arrivals: motivation and intuition. (a) A simple example of sparse gradient arrivals: a weight receives intermittent updates, in between which the regularization term produces relaxation toward zero. Three example stochastic simulations are shown. (b) Error as a function of number of arrivals in a toy model. If s/ρ<1s/\rho < 1, error always decreases (left); if 1<s/ρ<21 < s/\rho < 2, error alternates in sign but decreases in magnitude (center); and if s/ρ>2s/\rho > 2, learning becomes unstable, and error both alternates in sign and increases in magnitude (right).

Suppose we are interested in updating a set of weights w∈RN\bm{w} \in \mathbb{R}^N to optimize an objective

J~(w):=J(w)+γ2∥w∥22\begin{equation} \tilde{J}(\bm{w}) := J(\bm{w}) + \frac{\gamma}{2} \Vert \bm{w} \Vert_2^2 \tag*{(1)} \end{equation}

where JJ is a task-related objective and the second term regularizes w\bm{w}. We explicitly separate JJ and the regularization term because their biological interpretation in what follows is different. In the gradient-flow limit typical of many theoretical treatments of learning [8, 13, 10], gradient descent involves updates of the form

ddtw=η[−∂J∂w−γw]\begin{equation} \frac{d}{dt} \bm{w} = \eta \left[ - \frac{\partial J}{\partial \bm{w}} - \gamma \bm{w} \right] \tag*{(2)} \end{equation}

where η>0\eta > 0 is the learning rate and t∈[0,∞)t \in [0 ,\infty) is the amount of elapsed training time. We will investigate the consequences of assuming wiw_i only has intermittent access to the correct gradient. We do this as follows. We model sparse gradient arrivals via a Poisson process [11]: we assume that each synapse has a fixed probability per unit time of receiving a noise-corrupted version of the relevant gradient information. In doing things this way, we ignore potential temporal structure in gradient arrivals and correlations between the information arriving at different synapses; accounting for these factors will likely complexify the resulting model, and more importantly, make the associated stability issues worse rather than better.

Poisson gradient arrivals assumption.

To model temporally sparse updates, we assume gradients are coupled to a Poisson process with mean arrival rate ηρ\eta \rho, where ρ>0\rho > 0 is the non-dimensionalized arrival rate. As ρ\rho increases, neurons tend to receive more gradient arrivals per learning time scale 1/η1/\eta. If we denote a realization of this Poisson process by r(t)∈RN\bm{r}(t) \in \mathbb{R}^N, our model corresponds to the replacement

−∂J∂w→r(t)⊙1ρ[−∂J∂w+ξ]\begin{equation} - \frac{\partial J}{\partial \bm{w}} \to \bm{r}(t) \odot \frac{1}{\rho} \left[ - \frac{\partial J}{\partial \bm{w}} + \bm{\xi} \right] \tag*{(3)} \end{equation}

where ⊙\odot denotes the Hadamard (or elementwise) product. The quantity ξ∼N(0,σobs2I)\bm{\xi} \sim \mathcal{N}( \bm{0}, \sigma^2_{\text{obs}} \bm{I}) represents observation noise. Our main motivation for including it is to illustrate how, unlike sparse gradient arrivals, this kind of noise is comparatively benign.

According to this prescription, weight changes due to gradient arrivals have mean and covariance

E[Δw]=−ηΔt∂J∂wCov(Δwi,Δwj)=ηΔtρ δij[(∂J∂w)2+σobs2] .\begin{equation} \mathbb{E}[ \Delta \bm{w} ] = - \eta \Delta t \frac{\partial J}{\partial \bm{w}} \hspace{1in} \text{Cov}( \Delta w_i, \Delta w_j ) = \frac{\eta \Delta t}{\rho} \ \delta_{i j} \left[ \left( \frac{\partial J}{\partial \bm{w}} \right)^2 + \sigma^2_{\text{obs}} \right] \ . \tag*{(4)} \end{equation}
Useful intuition.

We can make a simple heuristic argument to explain why sparse gradient arrivals can destabilize learning. Consider gradient descent (Eq. (2)) with only one weight and no regularization (γ=0\gamma = 0) on a mean-squared error objective

J:=s2(w−w∗)2 .\begin{equation} J := \frac{s}{2} (w - w^*)^2 \ . \tag*{(5)} \end{equation}

For Poisson gradient arrivals, weights do not change until there is an arrival, so learning effectively occurs in discrete time. The error en:=wn−w∗e_n := w_n - w^* after nn gradient arrivals evolves according to the recurrence relation

en+1=(1−sρ)en .\begin{equation} e_{n + 1} = \left( 1 - \frac{s}{\rho} \right) e_n \ . \tag*{(6)} \end{equation}

These dynamics yield a geometric series with en=(1−s/ρ)ne0e_n = ( 1 - s/\rho)^n e_0 which has three regimes. For a sufficiently large arrival rate (s/ρ<1s/\rho < 1), error consistently decreases and asymptotically approaches zero (Figure 1b, left). For a moderately large arrival rate (1<s/ρ<21 < s/\rho < 2), weight dynamics overshoots its target and the error alternates between positive and negative; it still asymptotically approaches zero (Figure 1b, center). If the arrival rate is too small (s/ρ>2s/\rho > 2), learning becomes unstable, and error both alternates in sign and increases in magnitude (Figure 1b, right). This indicates that, at least for this toy model, the ratio s/ρs/\rho between input variance and the gradient arrival rate controls whether learning is stable. Hence, the arrival rate ρ\rho must be larger than

 ρ>ρc:=s2  .\begin{equation} \boxed{ \ \rho > \rho_c := \frac{s}{2} \ } \ . \tag*{(7)} \end{equation}

We will see that more complex models exhibit surprisingly similar behavior, and in particular lead to similar expressions for the critical arrival rate ρc\rho_c.

Instability in a linear model

Figure 2. Instability in a linear model. (a) Average dynamics are unconditionally stable and do not depend on the arrival rate ρ\rho. (b) Output variance converges more slowly as ρ\rho decreases and diverges to infinity when ρ<ρc\rho < \rho_c. (c) The steady state output variance diverges like 1/(ρ−ρc)1/(\rho - \rho_c). (d) ρc\rho_c decreases as γ\gamma increases, so one way to improve stability is to increase the regularization strength.

In this section, we study instability due to sparse gradient arrivals in a tractable linear model, in part because we think it reveals the essence of the phenomenon. Consider an objective

J~=Ex{∥wTx−(w∗)Tx∥222}+γ2∥w∥22=12(w−w∗)TS(w−w∗)+γ2∥w∥22\begin{equation} \tilde{J} = \mathbb{E}_{\bm{x}}\left\{ \frac{\Vert \bm{w}^T \bm{x} - (\bm{w}^*)^T \bm{x} \Vert_2^2}{2} \right\} + \frac{\gamma}{2} \Vert \bm{w} \Vert_2^2 = \frac{1}{2} (\bm{w} - \bm{w}^*)^T \bm{S} (\bm{w} - \bm{w}^*) + \frac{\gamma}{2} \Vert \bm{w} \Vert_2^2 \tag*{(8)} \end{equation}

where the input x\bm{x} is assumed to have mean zero and covariance S∈RN×N\bm{S} \in \mathbb{R}^{N \times N}. The negative gradient of JJ equals S(w∗−w)\bm{S} (\bm{w}^* - \bm{w}), so our prescription yields dynamics

ddtw=r⊙1ρ[S(w∗−w)+ξ]−ηγw .\begin{equation} \frac{d}{dt} \bm{w} = \bm{r} \odot \frac{1}{ \rho} [ \bm{S} (\bm{w}^* - \bm{w}) + \bm{\xi} ] - \eta \gamma \bm{w} \ . \tag*{(9)} \end{equation}

The average wˉ:=E[w]\bar{\bm{w}} := \mathbb{E}[ \bm{w} ] evolves according to standard-looking linear dynamics

ddtwˉ=η[S(w∗−wˉ)−γwˉ]\begin{equation} \frac{d}{dt} \bar{\bm{w}} = \eta \left[ \bm{S} (\bm{w}^* - \bar{\bm{w}}) - \gamma \bar{\bm{w}} \right] \tag*{(10)} \end{equation}

which approaches the fixed point at w′:=(S+γI)−1Sw∗\bm{w}' := (\bm{S} + \gamma \bm{I})^{-1} \bm{S} \bm{w}^* exponentially quickly. On the other hand, the dynamics of the covariance matrix C:=E[(w−wˉ)(w−wˉ)T]\bm{C} := \mathbb{E}[ (\bm{w} - \bar{\bm{w}}) (\bm{w} - \bar{\bm{w}})^T] are substantially more interesting (see Appendix § 9):

 1ηddtC=−2γC−CS−SC+1ρσobs2I+1ρ diag(gˉi2+[SCS]ii)  .\begin{equation} \boxed{ \ \frac{1}{\eta} \frac{d}{dt} \bm{C} = - 2 \gamma \bm{C} - \bm{C} \bm{S} - \bm{S} \bm{C} + \frac{1}{\rho} \sigma^2_{\text{obs}} \bm{I} + \frac{1}{\rho} \ \text{diag}( \bar{g}_i^2 + [ \bm{S} \bm{C} \bm{S} ]_{i i} ) \ } \ . \tag*{(11)} \end{equation}

We can assess stability by determining the conditions under which lim⁡t→∞tr[C(t)]\lim_{t \to \infty} \text{tr}[\bm{C}(t)] is finite. While this is difficult to do analytically in general, it is possible to do if we make at least one additional assumption about S\bm{S}: either that (i) S\bm{S} is diagonal, or that (ii) S\bm{S} is circulant.

Stability given a diagonal input covariance

If S\bm{S} is diagonal, Eq. (11) simplifies since [SCS]ii=Sii2Cii[ \bm{S} \bm{C} \bm{S} ]_{ii} = S_{i i}^2 C_{i i}. Let us simplify notation by writing ci:=Ciic_i := C_{ii} and si:=Siis_i := S_{ii}. The diagonal elements of C\bm{C} then evolve according to

1ηddtci=1ρ(σobs2+gˉi2)−(2γci+2si−si2ρ)ci .\begin{equation} \frac{1}{\eta} \frac{d}{dt} c_i = \frac{1}{\rho} ( \sigma^2_{\text{obs}} + \bar{g}_i^2 ) - \left( 2 \gamma c_i + 2 s_i - \frac{s_i^2}{\rho} \right) c_i \ . \tag*{(12)} \end{equation}

These dynamics are linear, and whether cic_i converges to a stable value or diverges to infinity depends on the coefficient of cic_i—that is, it must be negative. For learning to be stable, the coefficients of all cic_i must be negative; hence, it must be true that

 ρ>ρci:=si22(si+γ) \begin{equation} \boxed{ \ \rho > \rho_c^i := \frac{s_i^2}{2 (s_i + \gamma)} \ } \tag*{(13)} \end{equation}

for all ii. Equivalently, ρ>ρc:=maxiρci\rho > \rho_c := \text{max}_i \rho_c^i, which depends on the largest input variance sis_i. Note that this condition reduces to the critical arrival rate associated with our naive model (Eq. (7)) when we take γ→0\gamma \to 0.

As ρ\rho decreases, cic_i converges increasingly slowly, and below ρc\rho_c it becomes unstable (Figure 2b). The steady state diagonal covariance elements diverge like 1/(ρ−ρci)1/(\rho - \rho_c^i) (Figure 2c), or more quantitatively,

ci=1ρ−ρciσobs2+gˉi22(si+γ) .\begin{equation} c_{i} = \frac{1}{\rho - \rho_c^i} \frac{\sigma^2_{\text{obs}} + \bar{g}_i^2}{2 (s_i + \gamma)} \ . \tag*{(14)} \end{equation}

Our formula indicates that the critical arrival rate threshold ρc\rho_c can be reduced in two ways: one can either decrease input variance or increase the regularization strength γ\gamma (Figure 2d). The latter method recalls the standard bias-variance tradeoff [12]: we can decrease the (infinite) variance at the cost of increasing bias, since increasing γ\gamma makes w\bm{w} more different from the target weight w∗\bm{w}^*.

Stability given a circulant input covariance

Suppose instead that S\bm{S} is circulant. Then S\bm{S} can be diagonalized via a discrete Fourier basis, i.e., S=FΛF†\bm{S} = \bm{F} \bm{\Lambda} \bm{F}^{\dag} (see Appendix § 9). The arrival-related term can be written in terms of the covariance matrix in that basis, C~:=F†CF\tilde{\bm{C}} := \bm{F}^{\dag} \bm{C} \bm{F}, since

1N∑i[SCS]ii=1N∑qsq2C~qq .\begin{equation} \frac{1}{N} \sum_i [ \bm{S} \bm{C} \bm{S} ]_{ii} = \frac{1}{N} \sum_q s_q^2 \tilde{C}_{q q} \ . \tag*{(15)} \end{equation}

In this case, different C~qq\tilde{C}_{q q} do not decouple, but it is still possible to straightforwardly characterize dynamics. After some algebra, we obtain a critical arrival rate

 ρ>ρc:=12N∑qsq2sq+γ  .\begin{equation} \boxed{ \ \rho > \rho_c := \frac{1}{2 N} \sum_q \frac{s_q^2}{s_q + \gamma} \ } \ . \tag*{(16)} \end{equation}

Unlike the previous result, this one involves the entire spectrum of S\bm{S} (although it reduces to the previous one when all sqs_q are the same). Still, the result conveys the same qualitative message: ρc\rho_c increases as the input variance increases, and decreases as the regularization strength γ\gamma increases.

Instability in a nonlinear network

Figure 3. Instability in a two-layer network. (a) We trained a two-layer network to reproduce the output of an identical teacher network. To ensure our linear model provides a (locally) reasonable description of its dynamics, we initialized it near the teacher’s weights. (b) Normalized objective and output variance throughout learning. Note that for ρ<ρc\rho < \rho_c (dark red), JJ blows up. For ρ\rho somewhat above the threshold (red, orange), learning still looks unstable on this finite time scale. Learning only appears stable well beyond the threshold (green). (c) Instantaneous growth rate of the covariance dynamics near the fixed point. The black line indicates the linear model’s prediction and the red dots indicate numerical measurements. Below ρ=ρc\rho = \rho_c, this local growth rate can be positive, which indicates instability.

One virtue of our linear model analysis is that it implies sparse gradient arrivals pose problems for much more general networks. Near a stable fixed point w∗\bm{w}^* of any well-behaved objective JJ,

J(w)≈J(w∗)+12(w−w∗)TH(w−w∗)\begin{equation} J(\bm{w}) \approx J(\bm{w}^*) + \frac{1}{2} (\bm{w} - \bm{w}^*)^T \bm{H} (\bm{w} - \bm{w}^*) \tag*{(17)} \end{equation}

where H\bm{H} denotes the (positive definite) Hessian of JJ at the fixed point w∗\bm{w}^*. Hence, gradient descent dynamics are approximately linear near this fixed point, and if we make the identification S→H\bm{S} \to \bm{H}, they are well-described by our linear model (Eq. (9)).

Example: a two-layer network.

Although the above mathematical argument is fairly persuasive, it remains helpful to see an explicit nonlinear example where sparse gradient arrivals destabilize learning. For concreteness’ sake, we trained a simple two-layer network to reproduce the output of a teacher network with the same architecture (Figure 3a). We initialized the student network near the teacher’s weights and assessed how changing the arrival rate ρ\rho affected learning. For simplicity and to speed up computation, only the hidden layer’s biases b\bm{b} and the output parameters v\bm{v} and cc were trained; we found that this did not affect our conclusions, however.

We numerically computed the Hessian around the teacher network’s parameters in order to estimate the critical arrival rate ρc\rho_c below which learning ought to be unstable. For ρ<ρc\rho < \rho_c, we found that the loss function and output variance blew up as learning proceeded (Figure 3b, c; the dark red line indicates ρ=0.98ρc\rho = 0.98 \rho_c). In fact, we found that ρ\rho marginally above the critical arrival rate (ρ=1.2ρc\rho = 1.2 \rho_c, red; ρ=1.3ρc\rho = 1.3 \rho_c, orange) also exhibited unstable learning dynamics on the finite time scales we examined. Learning only became stable, with a smoothly-decreasing loss, when ρ\rho was much larger than ρc\rho_c (ρ=2ρc\rho = 2 \rho_c, green).

We also tested whether the instantaneous change in the output variance (or the “local growth rate”), which is predicted by Eq. (11), empirically describes small-time changes in output variance. We find that this is indeed the case, and that the local growth rate is positive below ρ=ρc\rho = \rho_c (Figure 3c).

Instability in a deep linear network

Figure 4. Depth benignly modulates stability in a deep linear network. (a) ρc\rho_c can increase, decrease, or remain constant with LL depending on the value of the target W∗W_*. In all cases, the difference between ρc(L)\rho_c(L) and its asymptotic value ρc(∞)\rho_c(\infty) is not substantial. (b) ρc\rho_c is non-monotonic as a function of γ\gamma (L=10L = 10 shown), but this is mainly because sufficiently high regularization destabilizes the nontrivial fixed point.

Are deeper networks less stable in the sense we study here? To partially address this question in an analytically tractable setting, we consider an extremely simple deep linear network with L≥1L \geq 1 layers whose hidden ‘layers’ are just scalars:

y=wL⋯w1x\begin{equation} y = w_L \cdots w_1 x \tag*{(18)} \end{equation}

where wiw_i is the scalar weight associated with the ii-th ‘layer’, xx is a scalar input, and yy is a scalar output. Although deep linear networks are admittedly much simpler than their nonlinear counterparts, they have the virtue that they are analytically tractable, and they recapitulate a surprising amount of the phenomenology associated with more realistic networks [8, 9, 10].

Consider a teacher-student setting in which our goal is to match yy to a target output y∗:=W∗xy_* := W_* x:

J=Ex{(y−y∗)22}=v2(w1⋯wL−W∗)2\begin{equation} J = \mathbb{E}_x\left\{ \frac{( y - y_*)^2}{2} \right\} = \frac{v}{2} ( w_1 \cdots w_L - W_* )^2 \tag*{(19)} \end{equation}

where we have assumed that xx has mean zero and variance v>0v > 0. Although the gradient descent learning dynamics of this network are generally nonlinear and complicated, we can analyze it around the symmetric fixed point with w1=⋯=wL=qw_1 = \cdots = w_L = q. We will study the impact of depth on the stability of dynamics near this fixed point. Note that we are studying the fixed point of the regularized objective J~\tilde{J} (see Eq. (1)), not JJ.

Define the quantity h:=vq2L−2h := v q^{2 L - 2}. The Hessian H\bm{H} of JJ about the symmetric fixed point has two eigenvalues (see Appendix § 10):

h⊥=γ(with multiplicity L−1)h∥=γ+L(h−γ)(with multiplicity 1) .\begin{equation} h_{\perp}= \gamma \hspace{0.2in} (\text{with multiplicity } L-1) \hspace{0.5in} h_{\parallel} = \gamma + L (h - \gamma) \hspace{0.2in} (\text{with multiplicity } 1) \ . \tag*{(20)} \end{equation}

Since H\bm{H} is circulant, we can immediately reuse the result from § 3 to find that

 ρc(L)=12L[h∥2h∥+γ+(L−1)h⊥2h⊥+γ]=12L[[γ+L(h−γ)]2L(h−γ)+2γ+(L−1)2γ]  .\begin{equation} \boxed{ \ \rho_c(L) = \frac{1}{2 L} \left[ \frac{h_{\parallel}^2}{h_{\parallel} + \gamma} + (L - 1) \frac{h_{\perp}^2}{h_{\perp} + \gamma} \right] = \frac{1}{2 L} \left[ \frac{[ \gamma + L (h - \gamma) ]^2}{ L (h - \gamma) + 2 \gamma} + \frac{(L - 1)}{2} \gamma \right] \ } \ . \tag*{(21)} \end{equation}

In the case of no regularization (γ=0\gamma = 0), we have the simpler critical arrival rate

 ρc(L)=h2=v2W∗2−2/L  .\begin{equation} \boxed{ \ \rho_c(L) = \frac{h}{2} = \frac{v}{2} W_*^{2 - 2/L} \ } \ . \tag*{(22)} \end{equation}

Interestingly, ρc\rho_c only depends benignly on LL, and moreover has a well-behaved infinite-depth (L→∞L \to \infty) limit (Figure 4a). There is an apparent instability as γ\gamma increases, since ρc\rho_c diverges to infinity at a finite value of γ\gamma (Figure 4b), but this is misleading; it occurs not because covariance dynamics become unstable, but because it is no longer optimal to match W∗W_* when there is too much regularization.

At least in this case, depth does not exacerbate stability issues. In some sense this should be unsurprising, since a result that suggests ρc\rho_c scales poorly with depth would seem inconsistent with biology, whose networks are often deep and highly recurrent; that is, learning is hard, but it should not be impossible.

Pooling parameters can improve stability

Figure 5. Pooling parameters can improve stability. (a) If nearby synapses store similar information, neurons may benefit from sharing information with their spatial neighbors. Pictured is a line of synaptic weights; arrows indicate bidirectional information sharing. (b) The eigenvalues dqd_q of D\bm{D} are indexed by spatial frequencies ∣q∣|q|, and dqd_q decreases as ∣q∣|q| increases. (c) The qq-dependence of dqd_q causes higher spatial frequencies of the fixed point to be damped, and the effect size increases as the amount of diffusion kk increases (more purple →\to higher kk). (d) One effect of this frequency-dependence is to smooth the fixed point of the average dynamics. In the pictured example, wi∗w_i^* is highly oscillatory (black), but dynamics converge to a smoothed version of it (shades of purple). (e) ρc\rho_c decreases as kk increases, so although parameter-pooling produces bias, it also improves stability.

One approach to improving stability is to average over parameters, since if nearby neurons have similar weights, it may make sense for them to share information (Figure 5a). Spatially proximal neurons are known to communicate in a variety of non-synaptic ways, including through gap junctions [15] and extracellular vesicles [14], and although the computational role of these features is unknown, it is plausible that they share computationally relevant information, including regarding synaptic weights. Parameter-pooling inevitably increases bias, but this tradeoff may be worth it if learning is otherwise unstable.

Assume that weights are arranged along a one-dimensional ring, so that wiw_i is ‘near’ wi+1w_{i+1} and wi−1w_{i-1}. Also assume that S\bm{S} is circulant. Consider a slight modification of Eq. (9) which incorporates parameter sharing via a ‘diffusion’ matrix D\bm{D} which satisfies [Dw]i=wi−1−2wi+wi+1[ \bm{D} \bm{w} ]_i = w_{i-1} - 2 w_i + w_{i+1}:

ddtw=r⊙1ρ[S(w∗−w)+ξ]−ηγw+ηkDw .\begin{equation} \frac{d}{dt} \bm{w} = \bm{r} \odot \frac{1}{ \rho} [ \bm{S} (\bm{w}^* - \bm{w}) + \bm{\xi} ] - \eta \gamma \bm{w} + \eta k \bm{D} \bm{w} \ . \tag*{(23)} \end{equation}

Here, k≥0k \geq 0 quantifies the strength of diffusion. This model reduces to the one studied in § 3 when k=0k = 0, but otherwise behaves differently. Its average dynamics read

1ηddtw=S(w∗−w)−γw+kDw\begin{equation} \frac{1}{\eta} \frac{d}{dt} \bm{w} = \bm{S} (\bm{w}^* - \bm{w}) - \gamma \bm{w} + k \bm{D} \bm{w} \tag*{(24)} \end{equation}

and, in the Fourier basis that diagonalizes D\bm{D} and S\bm{S}, the components of the fixed point are

wqFP:=sqsq+γ−kdqwq∗=Bqwq∗ .\begin{equation} w_q^{\text{FP}} := \frac{s_q}{s_q + \gamma - k d_q} w^*_q = B_q w^*_q \ . \tag*{(25)} \end{equation}

The prefactor BqB_q, which controls the asymptotic bias in learning, is highly spatial-frequency-dependent: because dqd_q is more negative for large frequencies ∣q∣|q| (Figure 5b), BqB_q is closer to zero for large ∣q∣|q|, and the effect size increases as the diffusion strength kk increases (Figure 5c). This implies that what is learned is a spatially smoothed version of the target (Figure 5d).

Meanwhile, following algebra similar to what we encountered in § 3 (see Appendix § 11), covariance dynamics are stable provided that a stability condition

 ρ>ρc:=12N∑qsq2sq+γ−kdq \begin{equation} \boxed{ \ \rho > \rho_c := \frac{1}{2 N} \sum_q \frac{s_q^2}{s_q + \gamma - k d_q} \ } \tag*{(26)} \end{equation}

holds. Since ρc\rho_c decreases as kk increases, pooling increases stability (Figure 5e).

Averaging over gradient arrivals can improve stability

Figure 6. Averaging over gradient arrivals can improve stability. (a) Intuitively, one can reduce variance by performing a temporal average over recent gradient arrivals. (b) Mean dynamics are linear but not monotonic. As the gradient-averaging time scale τz\tau_z increases, they converge more slowly. (black: τz=0\tau_z = 0; lightest green: τz=100\tau_z = 100) (c, d) Increasing τz\tau_z improves stability since ρc\rho_c decreases as long as γ>0\gamma > 0. But if γ=0\gamma = 0, there is no stability benefit.

Another plausible way to improve stability is to effectively reduce the variance of gradient arrivals by not learning directly from the available gradient signal, but from a temporal average of recent arrivals (Figure 6a). We can concretely instantiate this idea by assuming that each parameter wiw_i has a matching accumulator variable ziz_i which leaky-integrates recent gradients on a time scale τz>0\tau_z > 0:

ddtz=r⊙1ητzρ[S(w∗−w)+ξ]−1τzzddtw=η(z−γw) .\begin{equation} \begin{split} \frac{d}{dt} \bm{z} &= \bm{r} \odot \frac{1}{\eta \tau_z \rho} [ \bm{S} (\bm{w}^* - \bm{w}) + \bm{\xi} ] - \frac{1}{\tau_z} \bm{z} \\ \frac{d}{dt} \bm{w} &= \eta ( \bm{z} - \gamma \bm{w} ) \ . \end{split} \tag*{(27)} \end{equation}

By construction, the fixed point of average w\bm{w} dynamics is the same as before, although its transient behavior is slightly different (Figure 6b). Significantly, though, the addition of the accumulator variable z\bm{z} greatly changes covariance dynamics. The covariances Z:=Cov(z)\bm{Z} := \text{Cov}(\bm{z}), X:=Cov(z,w)\bm{X} := \text{Cov}(\bm{z}, \bm{w}), and W:=Cov(w)\bm{W} := \text{Cov}(\bm{w}) evolve according to the ODEs (see Appendix § 12)

ddtZ=−2τzZ−1τz(XS+SXT)+1ηρτz2diag[gi(wˉ)2+[SWS]ii+σobs2]ddtX=ηZ−(ηγ+1τz)X−1τzSWddtW=η[X+XT−2γW] .\begin{equation} \begin{split} \frac{d}{dt} \bm{Z} &= - \frac{2}{\tau_z} \bm{Z} - \frac{1}{\tau_z} ( \bm{X} \bm{S} + \bm{S} \bm{X}^T ) + \frac{1}{\eta \rho \tau_z^2} \text{diag}[ g_i(\bar{\bm{w}})^2 + [\bm{S} \bm{W} \bm{S}]_{ii} + \sigma^2_{\text{obs}} ] \\ \frac{d}{dt} \bm{X} &= \eta \bm{Z}- \left( \eta \gamma + \frac{1}{\tau_z} \right) \bm{X} - \frac{1}{\tau_z} \bm{S} \bm{W} \\ \frac{d}{dt} \bm{W} &= \eta \left[ \bm{X} + \bm{X}^T - 2 \gamma \bm{W} \right] \ . \end{split} \tag*{(28)} \end{equation}

As before, it is much easier to analyze these dynamics if we make specific assumptions about S\bm{S}. We will again assume that S\bm{S} is either diagonal or circulant.

Stability given a diagonal input covariance

If S\bm{S} is diagonal, the dynamics of (Zii,Xii,Wii)( Z_{ii}, X_{ii}, W_{ii}) for any fixed ii are independent of those for i′≠ii' \neq i, and are characterized by a 3×33 \times 3 matrix

M:=(−2τz−2siτzsi2ηρτz2η−(ηγ+1τz)−siτz02η−2ηγ) .\begin{equation} \bm{M} := \begin{pmatrix} - \frac{2}{\tau_z} & - \frac{2 s_i}{\tau_z} & \frac{s_i^2}{\eta \rho \tau_z^2}\\ \eta & - \left( \eta \gamma + \frac{1}{\tau_z} \right) & - \frac{s_i}{\tau_z} \\ 0 & 2 \eta & - 2 \eta \gamma \end{pmatrix} \ . \tag*{(29)} \end{equation}

Covariance dynamics are stable provided that M\bm{M} has only negative eigenvalues, and this holds unless ρ\rho is sufficiently small. We can find where the critical value ρci\rho_c^i is by computing where the determinant of M\bm{M} becomes zero. This yields a stability condition

 ρ>ρci:=si22(1+ηγτz)(si+γ) \begin{equation} \boxed{ \ \rho > \rho_c^i := \frac{s_i^2 }{2( 1 + \eta \gamma \tau_z) ( s_i + \gamma )} \ } \tag*{(30)} \end{equation}

analogous to what we discovered before. It reduces to our earlier result (Eq. (13)) in the τz→0\tau_z \to 0 limit, but otherwise produces a lower critical value of ρ\rho (Figure 6c, d). Relatedly, we obtain steady state weight variances

Wii=1ρ−ρciσobs2+gˉi22(si+γ)(1+γητz)\begin{equation} W_{i i} = \frac{1}{\rho - \rho_c^i} \frac{\sigma^2_{\text{obs}} + \bar{g}_i^2}{2 (s_i + \gamma) ( 1 + \gamma \eta \tau_z)} \tag*{(31)} \end{equation}

which again diverge like 1/(ρ−ρci)1/(\rho - \rho_c^i). It is important to emphasize that this gain in stability does not come for free: increases in τz\tau_z can also greatly slow down the convergence of average learning dynamics (Figure 6b, light green line).

Stability given a circulant input covariance

For circulant S\bm{S}, after some algebra, we obtain a result

 ρ>12N(1+ηγτz)∑qsq2sq+γ \begin{equation} \boxed{ \ \rho > \frac{1}{2 N (1 + \eta \gamma \tau_z)} \sum_q \frac{s_q^2}{s_q + \gamma} \ } \tag*{(32)} \end{equation}

which is again similar to what we found before (c.f. Eq. (16)), and reduces to it in the τz→0\tau_z \to 0 limit.

Discussion

Learning is difficult, and in this work we have identified and analyzed yet another reason why. It is not enough to perform accurate credit assignment on average if the relevant learning signal is highly variable, since this high variance can produce unstable learning. The source of this variance, sparse gradient arrivals, is distinct from other types of noise relevant to learning (e.g., noise due to using stochastic rather than full-batch gradient descent), and its effects are apparently more insidious.

Because any well-behaved network is approximately linear near a stable fixed point, one expects this issue to be generic rather than architecture-dependent. We considered three ways of addressing the problem—regularization, parameter-sharing, and gradient-averaging—and each reduces the severity of the issue but involves tradeoffs. Regularization and parameter-sharing reduce the (infinite) variance at the expense of bias; moreover, parameter-sharing may only be natural when there is specific (e.g., topographic) structure according to which spatial relationships correlate with relationships between synaptic weights. Gradient-averaging does not produce bias, but does slow down learning. It is plausible that all of these mechanisms, and perhaps others, are utilized by biological networks to avoid the kind of instability we study here.

Connection to analyses of stochastic gradient descent.

Standard analyses of stochastic gradient descent [17, 18] involve a mathematical idea similar to the one we study here, whose content is especially similar to the naive model we considered in § 2: if the learning rate or step size is taken to be too large, errors will accumulate and learning will become unstable. In other ways, the issue is qualitatively different. Our model is defined in continuous- rather than discrete-time, and moreover involves a nontrivial interaction between discrete and stochastic gradient arrivals on the one hand, and continuous relaxation (due to regularization) and accumulation (in the model considered in § 7) on the other. Moreover, the biological interpretation of sparse arrivals—which can be viewed as related to error-communication difficulties rather than simply an overall learning rate—is conceptually distinct. Finally, average dynamics are unconditionally stable.

Speculation regarding biological means of variance reduction.

If biological systems do indeed confront the problem of potential instability due to sparse gradient (or more generally, learning signal) arrivals, it is plausible that evolution has developed mechanisms to partially combat it. There is biological precedent for each of the three mechanisms—regularization, parameter-pooling, and gradient-averaging—we considered. It is known that the dendritic spines, which are sites of synaptic plasticity, turn over on relatively short time scales, e.g., on the order of one or two weeks in the hippocampus [19]. This continual ‘forgetting’ of synaptic weights can be modeled in the same way we have incorporated regularization. Our gradient-averaging model can be viewed as maintaining a kind of eligibility trace of learning-related information, and there is mounting evidence that such traces exist and play a major role in biological learning [21, 20]. Parameter-pooling has not been directly observed to our knowledge, but as we have mentioned, there are plausible means by which it could occur.

Limitations and future work.

For clarity and tractability, we have mainly analyzed simple linear models here. More interesting behavior may be possible in less linear regimes, e.g., away from a stable fixed point. Another major assumption we made is that sparse gradients arrive independently at each synapse according to a Poisson process. In reality, the delivery of gradient information to any given neuron is likely correlated with the delivery of gradients to other neurons (e.g., because the information is coming from roughly the same source). Although details associated with this issue likely differ for more general models, our observations suggest that it is worth further study. Putative methods of credit assignment must produce both good average performance and stable variance.

AI use statement

In this work, we used generative AI tools for brainstorming, literature search, to assist with checking the mathematical correctness of our results, and to assist with code generation. The author has checked all text and results and takes full responsibility for them.

Analysis of linear model

In this appendix, we analyze the behavior of the linear model discussed in § 3 of the main text. Recall that it has

ddtw=r⊙1ρ[S(w∗−w)+ξ]−ηγw\begin{equation} \frac{d}{dt} \bm{w} = \bm{r} \odot \frac{1}{ \rho} [ \bm{S} (\bm{w}^* - \bm{w}) + \bm{\xi} ] - \eta \gamma \bm{w} \tag*{(33)} \end{equation}

where r\bm{r} is distributed according to a Poisson arrival process with rate ηρ\eta \rho and ξ∼N(0,σobs2)\xi \sim \mathcal{N}(0, \sigma^2_{\text{obs}}) is observation noise associated with a gradient arrival. The master equation that defines how the distribution P(w∣t)P(\bm{w} | t) of weight values evolves in time is

∂P∂t=η∂∂w⋅[γwP]+ηρ∑i∫−∞∞dξ N(ξ;0,σobs2) [P(w−(1/ρ)[gi(w)+ξ]ei∣t)−P(w∣t)]\begin{equation} \frac{\partial P}{\partial t} = \eta \frac{\partial}{\partial \bm{w}} \cdot \left[ \gamma \bm{w} P \right] + \eta \rho \sum_i \int_{-\infty}^{\infty} d\xi \ \mathcal{N}(\xi; 0, \sigma^2_{\text{obs}}) \ \left[ P(\bm{w} - (1/\rho)[g_i(\bm{w})+\xi]\bm{e}_i | t) - P(\bm{w} | t) \right] \tag*{(34)} \end{equation}

where ei\bm{e}_i denotes the standard basis vector with a 11 as its ii-th component and zeros elsewhere. gi(w)g_i(\bm{w}) is shorthand notation for the ii-th component of S(w∗−w)\bm{S} (\bm{w}^* - \bm{w}), the true gradient.

Derivation of mean dynamics

Let us first derive the dynamics of the mean weight wˉ:=E[w]\bar{\bm{w}} := \mathbb{E}[ \bm{w} ]. The master equation (Eq. (34)) has the form of a Fokker-Planck equation plus the Poisson arrival term. The regularization term contributes the expected −ηγwˉ- \eta \gamma \bar{\bm{w}}. We can determine the contribution of the Poisson arrival term by computing its average over w\bm{w}. Note that

 ηρ∫w{∫−∞∞dξ N(ξ;0,σobs2) [P(w−(1/ρ)[gi(w)+ξ]ei∣t)−P(w∣t)]} dw= ηρ∫ N(ξ;0,σobs2) [wP(w−(1/ρ)[gi(w)+ξ]ei∣t)−wP(w∣t)] dw= ηρ∫ N(ξ;0,σobs2) [w+(1/ρ)(gi(w)+ξ)ei−w]P(w∣t) dw= η E[gi(w)ei] .\begin{equation} \begin{split} & \ \eta \rho \int \bm{w} \left\{ \int_{-\infty}^{\infty} d\xi \ \mathcal{N}(\xi; 0, \sigma^2_{\text{obs}}) \ \left[ P(\bm{w} - (1/\rho) [g_i(\bm{w})+\xi]\bm{e}_i | t) - P(\bm{w} | t) \right] \right\} \ d\bm{w} \\ =& \ \eta \rho \int \ \mathcal{N}(\xi; 0, \sigma^2_{\text{obs}}) \ \left[ \bm{w} P(\bm{w} - (1/\rho)[g_i(\bm{w})+\xi]\bm{e}_i | t) - \bm{w} P(\bm{w} | t) \right] \ d\bm{w} \\ =& \ \eta \rho \int \ \mathcal{N}(\xi; 0, \sigma^2_{\text{obs}}) \ \left[ \bm{w} + (1/\rho)(g_i(\bm{w})+\xi) \bm{e}_i - \bm{w} \right] P(\bm{w} | t) \ d\bm{w} \\ =& \ \eta \ \mathbb{E}[ g_i(\bm{w}) \bm{e}_i ] \ . \end{split} \tag*{(35)} \end{equation}

Summing over ii, we find that average dynamics are

 ddtwˉ=η[S(w∗−wˉ)−γwˉ]  ,\begin{equation} \boxed{ \ \frac{d}{dt} \bar{\bm{w}} = \eta \left[ \bm{S} (\bm{w}^* - \bar{\bm{w}}) - \gamma \bar{\bm{w}} \right] \ } \ , \tag*{(36)} \end{equation}

i.e., they are unbiased.

Derivation of covariance dynamics

Let us now derive the dynamics of the covariance matrix C:=Cov(w)\bm{C} := \text{Cov}(\bm{w}). Again, the regularization term contributes to its dynamics in the usual way; the hard part is to determine how the Poisson arrival term contributes.

The average of the Poisson arrival term times wwT\bm{w} \bm{w}^T equals

 ηρ∫wwT{∫−∞∞dξ N(ξ;0,σobs2) [P(w−(1/ρ)[gi(w)+ξ]ei∣t)−P(w∣t)]}dw= ηρ∫ N(ξ;0,σobs2) [wwTP(w−(1/ρ)[gi(w)+ξ]ei∣t)−wwTP(w∣t)] dw= ηρ∫ N(ξ;0,σobs2) { [w+(1/ρ)(gi(w)+ξ)ei][w+(1/ρ)(gi(w)+ξ)ei]T−wwT }P(w∣t)= η E[(weiT+eiwT)gi(w)+(1/ρ)(gi(w)2+σobs2)eieiT] .\begin{equation} \begin{split} & \ \eta \rho \int \bm{w} \bm{w}^T \left\{ \int_{-\infty}^{\infty} d\xi \ \mathcal{N}(\xi; 0, \sigma^2_{\text{obs}}) \ \left[ P(\bm{w} - (1/\rho) [g_i(\bm{w})+\xi]\bm{e}_i | t) - P(\bm{w} | t) \right] \right\} d\bm{w} \\ =& \ \eta \rho \int \ \mathcal{N}(\xi; 0, \sigma^2_{\text{obs}}) \ \left[ \bm{w} \bm{w}^T P(\bm{w} - (1/\rho) [g_i(\bm{w})+\xi]\bm{e}_i | t) - \bm{w} \bm{w}^T P(\bm{w} | t) \right] \ d\bm{w} \\ =& \ \eta \rho \int \ \mathcal{N}(\xi; 0, \sigma^2_{\text{obs}}) \ \left\{ \ [ \bm{w} + (1/\rho) (g_i(\bm{w})+\xi) \bm{e}_i ] [ \bm{w} +(1/\rho) (g_i(\bm{w})+\xi) \bm{e}_i ]^T - \bm{w} \bm{w}^T \ \right\} P(\bm{w} | t) \\ =& \ \eta \ \mathbb{E}[ ( \bm{w} \bm{e}_i^T + \bm{e}_i \bm{w}^T) g_i(\bm{w}) + (1/\rho) ( g_i(\bm{w})^2 + \sigma^2_{\text{obs}} )\bm{e}_i \bm{e}_i^T ] \ . \end{split} \tag*{(37)} \end{equation}

Summing over ii, this becomes

 η∑iE[(weiT+eiwT)gi(w)+(1/ρ)(gi(w)2+σobs2)eieiT]= η E[w(w∗−w)TS+S(w∗−w)wT+(1/ρ)diag(gi(w)2+σobs2)] .\begin{equation} \begin{split} & \ \eta \sum_i \mathbb{E}[ ( \bm{w} \bm{e}_i^T + \bm{e}_i \bm{w}^T) g_i(\bm{w}) + (1/\rho) ( g_i(\bm{w})^2 + \sigma^2_{\text{obs}} )\bm{e}_i \bm{e}_i^T ] \\ =& \ \eta \ \mathbb{E}[ \bm{w} (\bm{w}^* - \bm{w})^T \bm{S} + \bm{S} (\bm{w}^* - \bm{w}) \bm{w}^T + (1/\rho) \text{diag}( g_i(\bm{w})^2 + \sigma^2_{\text{obs}} ) ] \ . \end{split} \tag*{(38)} \end{equation}

Meanwhile, the time derivative of the mean product wˉwˉT\bar{\bm{w}} \bar{\bm{w}}^T equals

ddt(wˉwˉT)=(ddtwˉ)wˉT+wˉ(ddtwˉ)T=η[S(w∗−wˉ)−γwˉ]wˉT+ηwˉ[S(w∗−wˉ)−γwˉ]T .\begin{equation} \begin{split} \frac{d}{dt} ( \bar{\bm{w}} \bar{\bm{w}}^T ) &= \left( \frac{d}{dt} \bar{\bm{w}} \right) \bar{\bm{w}}^T + \bar{\bm{w}} \left( \frac{d}{dt} \bar{\bm{w}} \right)^T \\ &= \eta \left[ \bm{S} (\bm{w}^* - \bar{\bm{w}}) - \gamma \bar{\bm{w}} \right] \bar{\bm{w}}^T + \eta \bar{\bm{w}} \left[ \bm{S} (\bm{w}^* - \bar{\bm{w}}) - \gamma \bar{\bm{w}} \right]^T \ . \end{split} \tag*{(39)} \end{equation}

Putting these results together, we find that C=E[wwT]−wˉwˉT\bm{C} = \mathbb{E}[ \bm{w} \bm{w}^T] - \bar{\bm{w}} \bar{\bm{w}}^T evolves according to

1ηddtC=−2γC−CS−SC+(1/ρ) diag(E[gi(w)2]+σobs2) .\begin{equation} \frac{1}{\eta} \frac{d}{dt} \bm{C} = - 2 \gamma \bm{C} - \bm{C} \bm{S} - \bm{S} \bm{C} + (1/\rho) \ \text{diag}( \mathbb{E}[g_i(\bm{w})^2] + \sigma^2_{\text{obs}} ) \ . \tag*{(40)} \end{equation}

This result can be slightly simplified using the fact that

E[gi(w)2]=gˉi2+[SCS]ii ,\begin{equation} \mathbb{E}[ g_i(\bm{w})^2 ] = \bar{g}_i^2 + [ \bm{S} \bm{C} \bm{S} ]_{i i} \ , \tag*{(41)} \end{equation}

i.e., using the fact that the squared gradient can be written as a sum of ‘bias’ and ‘variance’ terms. Here gˉi\bar{g}_i is shorthand notation for gi(wˉ)g_i(\bar{\bm{w}}). We conclude that

 1ηddtC=−2γC−CS−SC+1ρσobs2I+1ρ diag(gˉi2+[SCS]ii)  .\begin{equation} \boxed{ \ \frac{1}{\eta} \frac{d}{dt} \bm{C} = - 2 \gamma \bm{C} - \bm{C} \bm{S} - \bm{S} \bm{C} + \frac{1}{\rho} \sigma^2_{\text{obs}} \bm{I} + \frac{1}{\rho} \ \text{diag}( \bar{g}_i^2 + [ \bm{S} \bm{C} \bm{S} ]_{i i} ) \ } \ . \tag*{(42)} \end{equation}

Analysis of Eq. (42) is cumbersome for general S\bm{S}, but greatly simplifies for specific choices of S\bm{S}. In the next two subsections, we perform a stability analysis of these covariance dynamics assuming S\bm{S} is either diagonal or circulant.

Stability analysis for diagonal SS

Assume S\bm{S} is diagonal. Let ci:=Ciic_i := C_{ii} and si:=Siis_i := S_{i i}, and collect both into the vectors c\bm{c} and s\bm{s}. Eq. (42) implies that the diagonal entries of C\bm{C} satisfy

1ηddtc=−2γc−2Sc+1ρσobs21+1ρ [g+S2c]=1ρ[σobs21+g]−[2γI+2S−1ρS2]c\begin{equation} \begin{split} \frac{1}{\eta} \frac{d}{dt} \bm{c} &= - 2 \gamma \bm{c} - 2 \bm{S} \bm{c} + \frac{1}{\rho} \sigma^2_{\text{obs}} \bm{1} + \frac{1}{\rho} \ [ \bm{g} + \bm{S}^2 \bm{c} ] \\ &= \frac{1}{\rho} [ \sigma^2_{\text{obs}} \bm{1} + \bm{g}] - [ 2 \gamma \bm{I} + 2 \bm{S} - \frac{1}{\rho} \bm{S}^2 ] \bm{c} \end{split} \tag*{(43)} \end{equation}

where we use g\bm{g} to denote the vector whose components are gˉi2\bar{g}_i^2 and 1\bm{1} to denote the vector whose components are all ones.

Stability.

Because this ODE is linear, it is clear that stability requires that

2γ+2si−1ρsi2>0\begin{equation} 2 \gamma + 2 s_i - \frac{1}{\rho} s_i^2 > 0 \tag*{(44)} \end{equation}

for all ii. Rearranging, this implies that we require

 ρ>ρci:=si22(si+γ) \begin{equation} \boxed{ \ \rho > \rho_c^i := \frac{s_i^2}{2 (s_i + \gamma)} \ } \tag*{(45)} \end{equation}

for all ii, or equivalently that

 ρ>ρc:=smax22(smax+γ) \begin{equation} \boxed{ \ \rho > \rho_c := \frac{s_{\text{max}}^2}{2 (s_{\text{max}} + \gamma)} \ } \tag*{(46)} \end{equation}

where smaxs_{\text{max}} is the largest input variance sis_i.

Steady state covariance.

Setting the left-hand side of Eq. (43) to zero, we find that the steady state covariance is

 Cii=1ρ−ρciσobs2+gi(wˉ∗)22(si+γ)  ,\begin{equation} \boxed{ \ C_{i i} = \frac{1}{\rho - \rho_c^i} \frac{\sigma^2_{\text{obs}} + g_i(\bar{\bm{w}}^*)^2}{2 (s_i + \gamma)} \ } \ , \tag*{(47)} \end{equation}

where wˉ∗\bar{\bm{w}}^{*} denotes the fixed point of mean dynamics, provided that ρ>ρc\rho > \rho_c.

Stability analysis for circulant SS

Assume S\bm{S} is circulant. Importantly, circulant matrices can be diagonalized via (discrete) Fourier basis functions. Concretely, let F\bm{F} denote the N×NN \times N matrix whose components are

Fjq=1Neiqj .\begin{equation} F_{j q} = \frac{1}{\sqrt{N}} e^{i q j} \ . \tag*{(48)} \end{equation}

Then we can write S=FΛF†\bm{S} = \bm{F} \bm{\Lambda} \bm{F}^{\dag}, where Λ\bm{\Lambda} is a diagonal matrix containing the eigenvalues of S\bm{S}. Moreover, if we define the transformed covariance C~:=F†CF\tilde{\bm{C}} := \bm{F}^{\dag} \bm{C} \bm{F}, then

1N∑i[SCS]ii=1N∑qsq2C~qq ,\begin{equation} \frac{1}{N} \sum_i [ \bm{S} \bm{C} \bm{S} ]_{ii} = \frac{1}{N} \sum_q s_q^2 \tilde{C}_{q q} \ , \tag*{(49)} \end{equation}

where sqs_q here denotes the qq-th eigenvalue of S\bm{S}. The virtue of this result, and hence the reason the circulant assumption is useful, is that the above quantity is independent of the index ii.

Combining this result with Eq. (42), we find that the diagonal entries C~qq\tilde{C}_{q q} of the transformed covariance satisfy

1ηddtC~qq=−2γC~qq−2sqC~qq+1ρσobs2+1ρ (Aqq+1N∑q′sq′2C~q′q′)\begin{equation} \frac{1}{\eta} \frac{d}{dt} \tilde{C}_{qq} = - 2 \gamma \tilde{C}_{qq} - 2 s_q \tilde{C}_{q q} + \frac{1}{\rho} \sigma^2_{\text{obs}} + \frac{1}{\rho} \ ( A_{q q} +\frac{1}{N} \sum_{q'} s_{q'}^2 \tilde{C}_{q' q'} ) \tag*{(50)} \end{equation}

where A~:=F†diag(gˉi2)F\tilde{\bm{A}} := \bm{F}^{\dag} \text{diag}( \bar{g}_i^2 ) \bm{F}.

Stability.

Define cq:=C~qqc_q := \tilde{C}_{q q} and collect the cqc_q into the vector c\bm{c}. Also, collect the A~qq\tilde{A}_{q q} into a\bm{a}. Eq. (50) can be written in the form

1ηddtc=−2γc−2Λc+1ρσobs21+1ρ (a+1N11TΛ2c)=1ρ(σobs21+a)−[2γI+2Λ−1ρN11TΛ2]c=1ρ(σobs21+a)−2γ[I+(1/γ)Λ−12ρNγ11TΛ2]c .\begin{equation} \begin{split} \frac{1}{\eta} \frac{d}{dt} \bm{c} &= - 2 \gamma \bm{c} - 2 \bm{\Lambda} \bm{c} + \frac{1}{\rho} \sigma^2_{\text{obs}} \bm{1} + \frac{1}{\rho} \ ( \bm{a} +\frac{1}{N} \bm{1} \bm{1}^T \bm{\Lambda}^2 \bm{c}) \\ &= \frac{1}{\rho} ( \sigma^2_{\text{obs}} \bm{1} + \bm{a}) - \left[ 2 \gamma \bm{I} + 2 \bm{\Lambda} - \frac{1}{\rho N} \bm{1} \bm{1}^T \bm{\Lambda}^2 \right] \bm{c} \\ &= \frac{1}{\rho} ( \sigma^2_{\text{obs}} \bm{1} + \bm{a}) - 2 \gamma \left[ \bm{I} + (1/\gamma) \bm{\Lambda} - \frac{1}{2 \rho N \gamma} \bm{1} \bm{1}^T \bm{\Lambda}^2 \right] \bm{c} \ . \end{split} \tag*{(51)} \end{equation}

From this result, it is clear that the stability of c\bm{c} dynamics depends on the matrix

M:=I+1γΛ−12ρNγ11TΛ2 .\begin{equation} \bm{M} := \bm{I} + \frac{1}{\gamma} \bm{\Lambda} - \frac{1}{2 \rho N \gamma} \bm{1} \bm{1}^T \bm{\Lambda}^2 \ . \tag*{(52)} \end{equation}

This a diagonal matrix plus a rank-one perturbation. We can use the matrix determinant lemma to compute its determinant:

det⁡(M)=det⁡(I+1γΛ)[1−12ρNγ1TΛ2(I+1γΛ)−11]=det⁡(I+1γΛ)[1−12ρNγ∑qsq2sq/γ+1] .\begin{equation} \begin{split} \det(\bm{M}) &= \det( \bm{I} + \frac{1}{\gamma} \bm{\Lambda} ) \left[ 1 - \frac{1}{2 \rho N \gamma} \bm{1}^T \bm{\Lambda}^2 ( \bm{I} + \frac{1}{\gamma} \bm{\Lambda} )^{-1} \bm{1} \right] \\ &= \det( \bm{I} + \frac{1}{\gamma} \bm{\Lambda} ) \left[ 1 - \frac{1}{2 \rho N \gamma} \sum_q \frac{s_q^2}{s_q/\gamma + 1} \right] \ . \end{split} \tag*{(53)} \end{equation}

For sufficiently high ρ\rho, it is clear that this determinant is positive, which indicates stability. We can identify the stability threshold by determining where the determinant is zero; since the eigenvalues of S\bm{S} are all positive, this can only happen when the bracketed factor equals zero.

We conclude that stability requires

 ρ>ρc:=12N∑qsq2sq+γ \begin{equation} \boxed{ \ \rho > \rho_c := \frac{1}{2 N} \sum_q \frac{s_q^2}{s_q + \gamma} \ } \tag*{(54)} \end{equation}

since beyond this value of ρ\rho, the determinant is negative.

Steady state covariance.

The above covariance dynamics can be rewritten as

1ηddtc=1ρ(σobs21+a)−2γMc ,\begin{equation} \begin{split} \frac{1}{\eta} \frac{d}{dt} \bm{c} &= \frac{1}{\rho} ( \sigma^2_{\text{obs}} \bm{1} + \bm{a}) - 2 \gamma \bm{M} \bm{c} \ , \end{split} \tag*{(55)} \end{equation}

so its steady state occurs at

c∗:=12ργM−1(σobs21+a) .\begin{equation} \bm{c}^* := \frac{1}{2 \rho \gamma} \bm{M}^{-1} ( \sigma^2_{\text{obs}} \bm{1} + \bm{a}) \ . \tag*{(56)} \end{equation}

This can be simplified if we write out M−1\bm{M}^{-1} explicitly. By the Sherman-Morrison formula,

M−1=(I+Λ/γ)−1+12ρNγ(I+Λ/γ)−111TΛ2(I+Λ/γ)−11−12ρNγ1TΛ2(I+Λ/γ)−11=(I+Λ/γ)−1+12Nγ(I+Λ/γ)−111TΛ2(I+Λ/γ)−1ρ−ρc .\begin{equation} \begin{split} \bm{M}^{-1} &= ( \bm{I} + \bm{\Lambda}/\gamma)^{-1} + \frac{1}{2 \rho N \gamma} \frac{( \bm{I} + \bm{\Lambda}/\gamma)^{-1} \bm{1} \bm{1}^T \bm{\Lambda}^2 ( \bm{I} + \bm{\Lambda}/\gamma)^{-1} }{1 - \frac{1}{2 \rho N \gamma} \bm{1}^T \bm{\Lambda}^2 ( \bm{I} + \bm{\Lambda}/\gamma)^{-1} \bm{1}} \\ &= ( \bm{I} + \bm{\Lambda}/\gamma)^{-1} + \frac{1}{2 N \gamma} \frac{( \bm{I} + \bm{\Lambda}/\gamma)^{-1} \bm{1} \bm{1}^T \bm{\Lambda}^2 ( \bm{I} + \bm{\Lambda}/\gamma)^{-1} }{\rho - \rho_c} \ . \end{split} \tag*{(57)} \end{equation}

Although the expression we obtain is more complicated than in the case of diagonal S\bm{S}, we still see a divergence that goes like 1/(ρ−ρc)1/(\rho - \rho_c) as ρ\rho approaches the critical arrival rate from above.

More details on the deep linear network

In this appendix, we provide a more thorough account of the deep linear network analysis described in § 5. We consider a scalar deep network with L≥2L \geq 2 ‘layers’:

y=wL⋯w1x .\begin{equation} y = w_L \cdots w_1 x \ . \tag*{(58)} \end{equation}

Here, xx is the input, wiw_i is the weight associated with the ii-th ‘layer’, and yy is the output. xx is assumed to have mean zero and variance vv.

We assume this network is trained via gradient descent to match a target output y∗:=W∗xy_* := W_* x which also depends linearly on its input. The corresponding objective equals

J~=J+∑iγ2wi2=v2(w1⋯wL−W∗)2+∑iγ2wi2 .\begin{equation} \tilde{J} = J + \sum_i \frac{\gamma}{2} w_i^2 = \frac{v}{2} ( w_1 \cdots w_L - W_* )^2 + \sum_i \frac{\gamma}{2} w_i^2 \ . \tag*{(59)} \end{equation}
Fixed point assumption.

Although learning dynamics are generally nonlinear, they are approximately linear near a fixed point. For simplicity, we will consider the symmetric fixed point with

w1=⋯=wL=q ,\begin{equation} w_1 = \cdots = w_L = q \ , \tag*{(60)} \end{equation}

where qq is chosen to minimize the objective, i.e.,

q:=argminqv2(qL−W∗)2+Lγ2q2 .\begin{equation} q := \text{argmin}_q \frac{v}{2} ( q^L - W_* )^2 + \frac{L \gamma}{2} q^2 \ . \tag*{(61)} \end{equation}

Since qq minimizes J~\tilde{J}, dJ~/dq=0d\tilde{J}/dq = 0, so the condition

 v(W∗−qL)qL−2=γ \begin{equation} \boxed{ \ v (W_* - q^L ) q^{L-2} = \gamma \ } \tag*{(62)} \end{equation}

holds.

Hessian near fixed point.

To characterize linear dynamics near the symmetric fixed point, we need to compute the corresponding L×LL \times L Hessian matrix. Note that the first derivatives at an arbitrary point are

∂J∂wi=v(w1⋯wL−W∗)Wwi\begin{equation} \frac{\partial J}{\partial w_i} = v ( w_1 \cdots w_L - W_*) \frac{W}{w_i} \tag*{(63)} \end{equation}

where we define W:=w1⋯wLW := w_1 \cdots w_L. The Hessian’s components at an arbitrary point are

Hii=vW2wi2Hij=vW2wiwj+v(W−W∗)Wwiwj .\begin{equation} \begin{split} H_{i i} &= v \frac{W^2}{w_i^2} \\ H_{i j} &= v\frac{W^2}{w_i w_j} + v ( W - W_*) \frac{W}{w_i w_j} \ . \end{split} \tag*{(64)} \end{equation}

At the symmetric fixed point,

Hii=vq2L−2Hij=vq2L−2+v(qL−W∗)qL−2 .\begin{equation} \begin{split} H_{i i} &= v q^{2 L-2} \\ H_{i j} &= v q^{2 L-2} + v ( q^L - W_*) q^{L-2} \ . \end{split} \tag*{(65)} \end{equation}

To ease notation, define h:=vq2L−2h := v q^{2 L - 2}. This result implies that we can write the Hessian matrix at the symmetric fixed point as

H=γI+(h−γ)11T ,\begin{equation} \bm{H} = \gamma \bm{I} + (h - \gamma) \bm{1} \bm{1}^T \ , \tag*{(66)} \end{equation}

i.e., it is a diagonal matrix plus a rank-one ‘correction’. We immediately learn that H\bm{H} has two eigenvalues: one associated with the correction, and another eigenvalue (with multiplicity L−1L-1) associated with the orthogonal directions:

h⊥=γh∥=γ+L(h−γ) .\begin{equation} h_{\perp} = \gamma \hspace{1in} h_{\parallel} = \gamma + L (h - \gamma) \ . \tag*{(67)} \end{equation}
Stability condition near symmetric fixed point.

This Hessian is circulant, so we can reuse a linear model result (see Appendix § 9) to immediately learn that

 ρc(L)=12L[h∥2h∥+γ+(L−1)h⊥2h⊥+γ]=12L[[γ+L(h−γ)]2L(h−γ)+2γ+(L−1)2γ]  .\begin{equation} \boxed{ \ \rho_c(L) = \frac{1}{2 L} \left[ \frac{h_{\parallel}^2}{h_{\parallel} + \gamma} + (L - 1) \frac{h_{\perp}^2}{h_{\perp} + \gamma} \right] = \frac{1}{2 L} \left[ \frac{[ \gamma + L (h - \gamma) ]^2}{ L (h - \gamma) + 2 \gamma} + \frac{(L - 1)}{2} \gamma \right] \ } \ . \tag*{(68)} \end{equation}

In the case of no regularization (γ=0\gamma = 0), we have the simpler critical arrival rate

 ρc(L)=h2=v2W∗2−2/L \begin{equation} \boxed{ \ \rho_c(L) = \frac{h}{2} = \frac{v}{2} W_*^{2 - 2/L} \ } \tag*{(69)} \end{equation}

which asymptotically approaches ρc(∞):=(v/2)W∗2\rho_c(\infty) := (v/2) W_*^2 as L→∞L \to \infty.

More details on pooling parameters

In this appendix, we derive results related to the parameter-pooling model discussed in § 6. It is defined in almost the same way as our linear model (see § 3 and Appendix § 9):

ddtw=r⊙1ρ[S(w∗−w)+ξ]−ηγw+ηkDw .\begin{equation} \frac{d}{dt} \bm{w} = \bm{r} \odot \frac{1}{ \rho} [ \bm{S} (\bm{w}^* - \bm{w}) + \bm{\xi} ] - \eta \gamma \bm{w} + \eta k \bm{D} \bm{w} \ . \tag*{(70)} \end{equation}

The main difference is that there is an additional term ηkDw\eta k \bm{D} \bm{w}, where D\bm{D} is a ‘diffusion’ matrix that implements a sort of spatial parameter pooling. In particular, D\bm{D} is defined such that

∑jDijwj:=wi+1−2wi+wi−1\begin{equation} \sum_j D_{i j} w_j := w_{i+1} - 2 w_i + w_{i - 1} \tag*{(71)} \end{equation}

for all ii. The interpretation is that information flows from nodes i+1i+1 and i−1i-1 to node ii, and from node ii to nodes i+1i+1 and i−1i-1.

For simplicity, we assume periodic boundary conditions, so that D\bm{D} is circulant rather than tridiagonal. Under this assumption, its eigenvalues are

dq:=−4sin⁡2(πqN)≤0\begin{equation} d_q := - 4 \sin^2\left( \frac{\pi q}{N} \right) \leq 0 \tag*{(72)} \end{equation}

where q=0,1,...,N−1q = 0, 1, ..., N-1, and it can be diagonalized using discrete Fourier basis functions.

The derivation of mean and covariance dynamics for this model is almost the same as it was for the linear model studied in Appendix § 9; everything is the same up to the identification

γI→γI−kD .\begin{equation} \gamma \bm{I} \to \gamma \bm{I} - k \bm{D} \ . \tag*{(73)} \end{equation}

For this reason, we omit the derivation and directly state the results. As in Appendix § 9, we assume that S\bm{S} is circulant for analytic tractability. This assumption is particularly important here because it implies that S\bm{S} and D\bm{D} commute.

Mean dynamics.

Mean dynamics follow

 ddtwˉ=η[S(w∗−wˉ)−(γI−kD)wˉ]  .\begin{equation} \boxed{ \ \frac{d}{dt} \bar{\bm{w}} = \eta \left[ \bm{S} (\bm{w}^* - \bar{\bm{w}}) - (\gamma \bm{I} - k \bm{D}) \bar{\bm{w}} \right] \ } \ . \tag*{(74)} \end{equation}

The corresponding fixed point is

wˉFP:=(S+γI−kD)−1Sw∗ .\begin{equation} \bar{\bm{w}}^{\text{FP}} := ( \bm{S} + \gamma \bm{I} - k \bm{D})^{-1} \bm{S} \bm{w}^* \ . \tag*{(75)} \end{equation}

In the discrete Fourier basis, this becomes

wqFP:=sqsq+γ−kdqwq∗=Bqwq∗ .\begin{equation} w_q^{\text{FP}} := \frac{s_q}{s_q + \gamma - k d_q} w^*_q = B_q w^*_q \ . \tag*{(76)} \end{equation}

where sqs_q denotes the qq-th eigenvalue of S\bm{S} and dqd_q denotes the qq-th eigenvalue of D\bm{D}. As noted in the main text, the specific form of dqd_q makes the overall associated prefactor BqB_q fairly spatial-frequency-dependent.

Covariance dynamics.

Covariance dynamics follow

 1ηddtC=−2γC−CS+kCD+kDC−SC+1ρσobs2I+1ρ diag(gˉi2+[SCS]ii)  .\begin{equation} \boxed{ \ \frac{1}{\eta} \frac{d}{dt} \bm{C} = - 2 \gamma \bm{C} - \bm{C} \bm{S} + k \bm{C} \bm{D} + k \bm{D} \bm{C} - \bm{S} \bm{C} + \frac{1}{\rho} \sigma^2_{\text{obs}} \bm{I} + \frac{1}{\rho} \ \text{diag}( \bar{g}_i^2 + [ \bm{S} \bm{C} \bm{S} ]_{i i} ) \ } \ . \tag*{(77)} \end{equation}

Moreover, the diagonal elements of the discrete-Fourier-transformed matrix C~:=F†CF\tilde{\bm{C}} := \bm{F}^{\dag} \bm{C} \bm{F} (see Appendix § 9 for more context) satisfy

1ηddtC~qq=−2γC~qq+2kdqC~qq−2sqC~qq+1ρσobs2+1ρ (Aqq+1N∑q′sq′2C~q′q′) .\begin{equation} \frac{1}{\eta} \frac{d}{dt} \tilde{C}_{qq} = - 2 \gamma \tilde{C}_{qq} + 2 k d_q \tilde{C}_{q q} - 2 s_q \tilde{C}_{q q} + \frac{1}{\rho} \sigma^2_{\text{obs}} + \frac{1}{\rho} \ ( A_{q q} +\frac{1}{N} \sum_{q'} s_{q'}^2 \tilde{C}_{q' q'} ) \ . \tag*{(78)} \end{equation}

Following a slight modification of the argument from Appendix § 9, we obtain the stability condition

 ρ>12N∑qsq2sq+γ−kdq  .\begin{equation} \boxed{ \ \rho > \frac{1}{2 N} \sum_q \frac{s_q^2}{s_q + \gamma - k d_q } \ } \ . \tag*{(79)} \end{equation}

More details on averaging over gradient arrivals

In this appendix, we analyze the mean and covariance dynamics of the gradient-accumulation model discussed in § 7 of the main text. It is defined similarly to the linear model (see § 3 and Appendix § 9), but with a ‘gradient accumulation’ variable ziz_i for each weight wiw_i:

ddtz=r⊙1ητzρ[S(w∗−w)+ξ]−1τzzddtw=η(z−γw) .\begin{equation} \begin{split} \frac{d}{dt} \bm{z} &= \bm{r} \odot \frac{1}{\eta \tau_z \rho} [ \bm{S} (\bm{w}^* - \bm{w}) + \bm{\xi} ] - \frac{1}{\tau_z} \bm{z} \\ \frac{d}{dt} \bm{w} &= \eta ( \bm{z} - \gamma \bm{w} ) \ . \end{split} \tag*{(80)} \end{equation}

This model’s master equation has the form

∂P∂t= −η∂∂w⋅[(z−γw)P]+1τz∂∂z⋅(zP)+ ηρ∑i∫−∞∞dξ N(ξ;0,σobs2) [P(z−1ηρτz[gi(w)+ξ]ei,w∣t)−P(z,w∣t)] .\begin{equation} \begin{split} \frac{\partial P}{\partial t} =& \ - \eta \frac{\partial}{\partial \bm{w}} \cdot \left[ ( \bm{z} - \gamma \bm{w} ) P \right] + \frac{1}{\tau_z} \frac{\partial}{\partial \bm{z}} \cdot ( \bm{z} P )\\ & + \ \eta \rho \sum_i \int_{-\infty}^{\infty} d\xi \ \mathcal{N}(\xi; 0, \sigma^2_{\text{obs}}) \ \left[ P(\bm{z} - \frac{1}{\eta \rho \tau_z} [g_i(\bm{w})+\xi]\bm{e}_i, \bm{w} | t) - P(\bm{z}, \bm{w} | t) \right] \ . \end{split} \tag*{(81)} \end{equation}

In this section, because there are multiple covariance matrices, we will use different notation than elsewhere. Let Z:=Cov(z)\bm{Z} := \text{Cov}(\bm{z}), X:=Cov(z,w)\bm{X} := \text{Cov}(\bm{z}, \bm{w}) denote the cross-covariance, and W:=Cov(w)\bm{W} := \text{Cov}(\bm{w}).

The calculations required to treat this model largely resemble those from Appendix § 9, but are lengthier, and hence more tedious. Because they are straightforward, we skip a number of intermediate steps and mostly present the final results.

Mean dynamics.

This model’s average dynamics are

ddtz=1τzS(w∗−wˉ)−1τzzˉddtwˉ=η(zˉ−γwˉ) .\begin{equation} \begin{split} \frac{d}{dt} \bm{z} &= \frac{1}{\tau_z} \bm{S} (\bm{w}^* - \bar{\bm{w}}) - \frac{1}{\tau_z} \bar{\bm{z}} \\ \frac{d}{dt} \bar{\bm{w}} &= \eta ( \bar{\bm{z}} - \gamma \bar{\bm{w}} ) \ . \end{split} \tag*{(82)} \end{equation}

In particular, the steady state value of wˉ\bar{\bm{w}} is the same as it is in the linear model; this kind of gradient accumulation does not produce bias, at least asymptotically.

Covariance dynamics.

After a tedious calculation, one can find that

ddtZ=−2τzZ−1τz(XS+SXT)+1ηρτz2diag[gi(wˉ)2+[SWS]ii+σobs2]ddtX=ηZ−(ηγ+1τz)X−1τzSWddtW=η[X+XT−2γW] .\begin{equation} \begin{split} \frac{d}{dt} \bm{Z} &= - \frac{2}{\tau_z} \bm{Z} - \frac{1}{\tau_z} ( \bm{X} \bm{S} + \bm{S} \bm{X}^T ) + \frac{1}{\eta \rho \tau_z^2} \text{diag}[ g_i(\bar{\bm{w}})^2 + [\bm{S} \bm{W} \bm{S}]_{ii} + \sigma^2_{\text{obs}} ] \\ \frac{d}{dt} \bm{X} &= \eta \bm{Z}- \left( \eta \gamma + \frac{1}{\tau_z} \right) \bm{X} - \frac{1}{\tau_z} \bm{S} \bm{W} \\ \frac{d}{dt} \bm{W} &= \eta \left[ \bm{X} + \bm{X}^T - 2 \gamma \bm{W} \right] \ . \end{split} \tag*{(83)} \end{equation}
Diagonal SS.

Assume S\bm{S} is diagonal and let ZiZ_i, XiX_i, and WiW_i denote the diagonals of these covariance matrices. The diagonal entries have dynamics

ddtZi=−2τzZi−2τzsiXi+1ηρτz2[gi(wˉ)2+si2Wi+σobs2]ddtXi=ηZi−(ηγ+1τz)Xi−1τzsiWiddtWi=2η[Xi−γWi] .\begin{equation} \begin{split} \frac{d}{dt} Z_i &= - \frac{2}{\tau_z} Z_i - \frac{2}{\tau_z} s_i X_i + \frac{1}{\eta \rho \tau_z^2} [ g_i(\bar{\bm{w}})^2 + s_i^2 W_i + \sigma^2_{\text{obs}} ] \\ \frac{d}{dt} X_i &= \eta Z_i - \left( \eta \gamma + \frac{1}{\tau_z} \right) X_i - \frac{1}{\tau_z} s_i W_i \\ \frac{d}{dt} W_i &= 2 \eta \left[ X_i - \gamma W_i \right] \ . \end{split} \tag*{(84)} \end{equation}

The stability of this three-dimensional system of ODEs is characterized by the 3×33 \times 3 matrix

M:=(−2τz−2siτzsi2ηρτz2η−(ηγ+1τz)−siτz02η−2ηγ) .\begin{equation} \bm{M} := \begin{pmatrix} - \frac{2}{\tau_z} & - \frac{2 s_i}{\tau_z} & \frac{s_i^2}{\eta \rho \tau_z^2}\\ \eta & - \left( \eta \gamma + \frac{1}{\tau_z} \right) & - \frac{s_i}{\tau_z} \\ 0 & 2 \eta & - 2 \eta \gamma \end{pmatrix} \ . \tag*{(85)} \end{equation}

We can directly compute that its determinant is

 −2τz[2ηγ(ηγ+1/τz)+2ηsi/τz]+2siτz[−2η2γ]+si2ηρτz2[2η2]= −2τz[2ηγ(ηγ+1/τz)+2ηsi(1/τz+ηγ)]+si2ηρτz2[2η2]= −4ητz(ηγ+1/τz)[γ+si]+si2ηρτz2[2η2] .\begin{equation} \begin{split} & \ - \frac{2}{\tau_z} \left[ 2 \eta \gamma ( \eta \gamma + 1/\tau_z) + 2 \eta s_i/\tau_z \right] + \frac{2 s_i}{\tau_z} \left[ - 2 \eta^2 \gamma \right] + \frac{s_i^2}{\eta \rho \tau_z^2} \left[ 2 \eta^2 \right] \\ =& \ - \frac{2}{\tau_z} \left[ 2 \eta \gamma ( \eta \gamma + 1/\tau_z) + 2 \eta s_i (1/\tau_z + \eta \gamma ) \right] + \frac{s_i^2}{\eta \rho \tau_z^2} \left[ 2 \eta^2 \right] \\ =& \ - \frac{4 \eta}{\tau_z} ( \eta \gamma + 1/\tau_z) \left[ \gamma + s_i \right] + \frac{s_i^2}{\eta \rho \tau_z^2} \left[ 2 \eta^2 \right] \ . \end{split} \tag*{(86)} \end{equation}

Stability requires that the determinant is negative. Hence, we need it to be true that

2(1+ηγτz)(γ+si)−si2ρ>0 .\begin{equation} 2( 1 + \eta \gamma \tau_z ) ( \gamma + s_i ) - \frac{s_i^2 }{\rho} > 0 \ . \tag*{(87)} \end{equation}

Equivalently,

 ρ>ρci:=si22(1+γητz)(si+γ)  .\begin{equation} \boxed{ \ \rho > \rho_c^i := \frac{s_i^2 }{2( 1 + \gamma \eta \tau_z) ( s_i + \gamma )} \ } \ . \tag*{(88)} \end{equation}

In the infinitely fast integration limit (τz→0\tau_z \to 0), this reduces to the linear model result (see Appendix § 9).

The corresponding steady state weight variances are

 Wii=1ρ−ρciσobs2+gˉi22(si+γ)(1+γητz)  .\begin{equation} \boxed{ \ W_{i i} = \frac{1}{\rho - \rho_c^i} \frac{\sigma^2_{\text{obs}} + \bar{g}_i^2}{2 (s_i + \gamma) ( 1 + \gamma \eta \tau_z)} \ } \ . \tag*{(89)} \end{equation}

Interestingly, this expression is the same as what we found for the linear model, except that ρci\rho_c^i has a different form.

Circulant SS.

If S\bm{S} is instead circulant, we must go through a lengthier version of the circulant algebra from Appendix § 9. In the end, we find almost the same results. Most importantly, we find a stability condition

 ρ>ρc:=12N(1+ηγτz)∑ksk2sk+γ \begin{equation} \boxed{ \ \rho > \rho_c := \frac{1}{2 N (1 + \eta \gamma \tau_z)} \sum_k \frac{s_k^2}{s_k + \gamma} \ } \tag*{(90)} \end{equation}

which reduces to the corresponding linear model condition in the τz→0\tau_z \to 0 limit.

References

  1. [1]Lillicrap, T. P., Cownden, D., Tweed, D. B., & Akerman, C. J. (2016). Random synaptic feedback weights support error backpropagation for deep learning. Nature Communications, 7(1), 13276. https://doi.org/10.1038/ncomms13276
  2. [2]Lillicrap, T. P., Santoro, A., Marris, L., Akerman, C. J., & Hinton, G. (2020). Backpropagation and the brain. Nature Reviews Neuroscience, 21(6), 335–346. https://doi.org/10.1038/s41583-020-0277-3
  3. [3]Crick, F. (1989). The recent excitement about neural networks. Nature, 337(6203), 129–132. https://doi.org/10.1038/337129a0
  4. [4]Sacramento, J., Ponte Costa, R., Bengio, Y., & Senn, W. (2018). Dendritic cortical microcircuits approximate the backpropagation algorithm. Advances in Neural Information Processing Systems, 31. https://proceedings.neurips.cc/paper_files/paper/2018/file/1dc3a89d0d440ba31729b0ba74b93a33-Paper.pdfproceedings.neurips.cc/paper_files/paper/2018/file/1dc3a89d0d440ba31729b0ba74b93a33-Paper.pdf
  5. [5]Whittington, J. C. R., & Bogacz, R. (2019). Theories of Error Back-Propagation in the Brain. Trends in Cognitive Sciences, 23(3), 235–250. https://doi.org/10.1016/j.tics.2018.12.005
  6. [6]Millidge, B., Tschantz, A., & Buckley, C. L. (2022). Predictive Coding Approximates Backprop Along Arbitrary Computation Graphs. Neural Computation, 34(6), 1329–1368. https://doi.org/10.1162/neco_a_01497
  7. [7]Scellier, B., & Bengio, Y. (2017). Equilibrium Propagation: Bridging the Gap between Energy-Based Models and Backpropagation. Frontiers in Computational Neuroscience, Volume 11 - 2017. https://doi.org/10.3389/fncom.2017.00024
  8. [8]Saxe, A. M., McClelland, J. L., & Ganguli, S. (2013). Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. arXiv E-prints, arXiv:1312.6120. https://doi.org/10.48550/arXiv.1312.6120
  9. [9]Saxe, A. M., McClelland, J. L., & Ganguli, S. (2019). A mathematical theory of semantic development in deep neural networks. Proceedings of the National Academy of Sciences, 116(23), 11537–11546. https://doi.org/10.1073/pnas.1820226116
  10. [10]Domine, C., Anguita, N., Proca, A. M., Braun, L., Kunin, D., Mediano, P., & Saxe, A. (2025). From Lazy to Rich: Exact Learning Dynamics in Deep Linear Networks. International Conference on Learning Representations, 2025, 102485–102536. https://proceedings.iclr.cc/paper_files/paper/2025/file/fde1a69a5b6e554b2f1f727197d2651d-Paper-Conference.pdfproceedings.iclr.cc/paper_files/paper/2025/file/fde1a69a5b6e554b2f1f727197d2651d-Paper-Conference.pdf
  11. [11]Kingman, J. F. C. (1992). Poisson processes (Vol. 3). Clarendon Press.
  12. [12]Bishop, C. M. (2006). Pattern recognition and machine learning. Springer.
  13. [13]Bordelon, B., Canatar, A., & Pehlevan, C. (2020). Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networks. Proceedings of the 37th International Conference on Machine Learning, 119, 1024–1034. https://proceedings.mlr.press/v119/bordelon20a.htmlproceedings.mlr.press/v119/bordelon20a.html
  14. [14]Nieves Torres, D., & Lee, S. H. (2023). Inter-neuronal signaling mediated by small extracellular vesicles: wireless communication?. Frontiers in Molecular Neuroscience, Volume 16 - 2023. https://doi.org/10.3389/fnmol.2023.1187300
  15. [15]Shimizu, K., & Stopfer, M. (2013). Gap junctions. Current Biology, 23(23), R1026–R1031. https://doi.org/10.1016/j.cub.2013.10.067
  16. [16]Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986). Learning representations by back-propagating errors. Nature, 323(6088), 533–536. https://doi.org/10.1038/323533a0
  17. [17]Wu, L., Ma, C., & E, W. (2018). How SGD Selects the Global Minima in Over-parameterized Learning: A Dynamical Stability Perspective. Advances in Neural Information Processing Systems, 31. https://proceedings.neurips.cc/paper_files/paper/2018/file/6651526b6fb8f29a00507de6a49ce30f-Paper.pdfproceedings.neurips.cc/paper_files/paper/2018/file/6651526b6fb8f29a00507de6a49ce30f-Paper.pdf
  18. [18]Ahn, K., Zhang, J., & Sra, S. (2022). Understanding the unstable convergence of gradient descent. Proceedings of the 39th International Conference on Machine Learning, 162, 247–257. https://proceedings.mlr.press/v162/ahn22a.htmlproceedings.mlr.press/v162/ahn22a.html
  19. [19]Attardo, A., Fitzgerald, J. E., & Schnitzer, M. J. (2015). Impermanence of dendritic spines in live adult CA1 hippocampus. Nature, 523(7562), 592–596. https://doi.org/10.1038/nature14467
  20. [20]Gerstner, W., Lehmann, M., Liakoni, V., Corneil, D., & Brea, J. (2018). Eligibility Traces and Plasticity on Behavioral Time Scales: Experimental Support of NeoHebbian Three-Factor Learning Rules. Frontiers in Neural Circuits, Volume 12 - 2018. https://doi.org/10.3389/fncir.2018.00053
  21. [21]He, K., Huertas, M., Hong, S. Z., Tie, X., Hell, J. W., Shouval, H., & Kirkwood, A. (2015). Distinct Eligibility Traces for LTP and LTD in Cortical Synapses. Neuron, 88(3), 528–538. https://doi.org/10.1016/j.neuron.2015.09.037

Paper details

vlnc:W_vk3g7t26
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Agricultural and Biological Sciences

generative AI tools

View statement
Looking for related papers…
Contents