betaFeedback

WorksComputer Science

A theory of noisy linear recurrent neural network learning dynamics

Abstract

Biological neural networks are noisy: their dynamics involve difficult-to-predict fluctuations which are likely due not just to chaos, but also to a variety of low-level biophysical processes. Despite the omnipresence of intrinsic noise, its theoretical consequences are incompletely understood, especially in the context of learning. For example, does noise meaningfully shape representation learning, or is it better viewed as a nuisance factor whose presence or absence changes little about what solution is found and when? Here, we argue that intrinsic noise can significantly modulate representation learning in recurrent neural networks (RNNs), since it can strongly break solution-space degeneracies and in some cases control whether a non-degenerate solution exists at all. We show that this is true for an analytically tractable model of noisy RNN learning with a linear activation function and white noise inputs. We find that, in the so-called rich regime (where weights are initialized with small values), noise strongly modulates the directions in which the read-in, readout, and recurrent weights align. Moreover, we find that noise produces a kind of phase transition: for sufficiently high noise, a non-degenerate solution is not even optimal if parameter scales are limited (e.g., via regularization). These results extend recent analytic work on RNN representation learning and show that noise can act as a fundamental determinant of representations.

Introduction

Recurrent neural networks (RNNs) are extraordinarily successful models of the brain [5, 9, 10, 14], and the past two decades of theory and experiment have produced a working understanding of what they learn and how. Their loss landscapes are generally more complex than those of feedforward networks, in part because recurrence can produce exploding and vanishing gradients [16, 17]; their solution space is generally highly degenerate [8]; and they can be biased towards learning low-rank solutions [15]. But a major feature of RNNs that remains poorly understood is the significance of intrinsic noise.

Noise is omnipresent in biological neural networks: spike trains and firing rate traces throughout the nervous system involve difficult-to-predict fluctuations [3], which may be a manifestation of chaotic behavior [1, 2] or low-level stochasticity associated with chemical reactions. Although from one perspective noise is a nuisance, from another it provides pressure for networks to find more robust solutions. Noise is routinely used in RNN training [12, 13], in part to improve generalization, and there is evidence that noise can drive networks into flatter loss landscape minima [11].

From a representation-learning perspective, it may be helpful to view noise as having two potential consequences. First, noise can provide geometric constraints on solutions: weights must be oriented in certain directions (i.e., the less noisy directions) and not others, and their overall scales must be chosen to minimize noise. Second, the goal of minimizing noise may compete with that of achieving good task performance; if there is too much noise, one expects that the system may be driven to a trivial solution at which task performance is poor, but there is little or no noise.

Although RNN learning dynamics are difficult to study, an interesting toy model of them has emerged in recent work: linear RNNs, especially linear RNNs trained to perform leaky integration of a white noise stimulus [7, 4]. As [4] show, when network parameters are initially small, learning has two phases: an alignment phase, in which parameter movement concentrates along a specific (initialization-dependent) direction of state space, followed by a dynamics phase, in which an eigenvalue ‘escapes’ from the bulk in order to solve the task.

In this work, we consider an extremely similar theoretical setup, whose primary difference is the presence of (generally anisotropic) intrinsic noise. We show that their theory must be substantially modified even if there is only a small amount of noise, since noise affects both the alignment and dynamics phases, and even whether a nontrivial solution exists at all. Throughout, we focus on characterizing learning in the rich regime, as opposed to the lazy regime, in part because the rich regime is thought to be more representative of learning in performant RNNs.

Our main contributions are as follows.

  • We develop an analytically tractable theory of representation learning in linear RNNs with intrinsic noise. We show that noise breaks degeneracies of the original objective, which yields preferred state space directions and read-in/readout scales.

  • We characterize how intrinsic noise modifies rich-regime learning dynamics. During the alignment phase, anisotropic noise changes the growth rates of different state-space modes, which biases learning toward less noisy directions. Noise also breaks the symmetry between the read-in and readout weights, and in the absence of regularization, drives the input weight scale to infinity and the output weight scale to zero.

  • We show that sufficiently strong noise can produce phase transitions beyond which only a trivial solution is learned. This is true both when all parameters are trainable and when some parameters are frozen.

Mathematical formulation

Figure 1. Theoretical setting. (a) We study a noisy linear RNN which receives a scalar white noise signal x(t)x(t) as input and produces an output y(t)y(t). The dynamics of individual rir_i are noisy (bottom). (b) On average, the network filters its input. (c) Output variance is input-independent and eventually saturates (assuming the network is stable). (d) We assume the network is trained to optimize an objective with three terms: a filter-matching term that incentivizes matching its average behavior to that of a target filter f∗(t)f_*(t), a noise reduction term, and a regularization term.

Our setup largely mirrors that of [4]. Throughout this paper, we study a linear RNN with N≥1N \geq 1 neurons whose instantaneous state is characterized by a vector r:=(r1,...,rN)T∈RN\bm{r} := (r_1, ..., r_N)^T \in \mathbb{R}^N. We assume dynamics

r˙=Ar+x(t)u+G η(t)=−γr+Wr+x(t)u+G η(t)\begin{equation} \dot{\bm{r}} = \bm{A} \bm{r} + x(t) \bm{u} + \bm{G} \ \bm{\eta}(t) = - \gamma \bm{r} + \bm{W} \bm{r} + x(t) \bm{u} + \bm{G} \ \bm{\eta}(t) \tag*{(1)} \end{equation}

where x(t)x(t) is a time-dependent input signal, u∈RN\bm{u} \in \mathbb{R}^N are the read-in weights, γ>0\gamma > 0 is the coefficient of the leak term, and W∈RN×N\bm{W} \in \mathbb{R}^{N \times N} are the recurrent weights (Figure 1a). To ease notation, we sometimes work with the combination A:=−γI+W\bm{A} := - \gamma \bm{I} + \bm{W}. The remaining term instantiates intrinsic noise: G∈RN×N\bm{G} \in \mathbb{R}^{N \times N} controls its amplitude, and η(t)∈RN\bm{\eta}(t) \in \mathbb{R}^N is a vector of Gaussian white noise processes. We are most interested in understanding how the structure of G\bm{G} shapes learning.

Because Eq. (1) is linear in r\bm{r} and involves only state-independent additive noise, we can immediately write down the distribution of r(t)\bm{r}(t) conditional on any initial condition, although we will assume r(0)=0\bm{r}(0) = \bm{0} for simplicity. We find that r(t)\bm{r}(t) is normally distributed with

p(r,t∣r0=0,0)=N(∫0teAt′u x(t−t′) dt′,∫0teAt′SeATt′ dt′)\begin{equation} p(\bm{r} , t | \bm{r}_0 = \bm{0}, 0) = \mathcal{N}\left( \int_0^t e^{\bm{A} t'} \bm{u} \ x(t-t') \ dt' , \int_0^t e^{\bm{A} t'} \bm{S} e^{\bm{A}^T t'} \ dt' \right) \tag*{(2)} \end{equation}

where S:=GGT\bm{S} := \bm{G} \bm{G}^T is, up to a factor of 22, the diffusion tensor associated with G\bm{G} (which we take to be positive definite). On average, the network linearly filters its input (Figure 1b). But because this integration is noisy, S\bm{S}-dependent noise accumulates until (provided the eigenvalues of A\bm{A} all have negative real parts) it saturates (Figure 1c).

Objective function

Following [4], assume a linear readout determined by readout weights v∈RN\bm{v} \in \mathbb{R}^N:

y(t):=vTr(t) .\begin{equation} y(t) := \bm{v}^T \bm{r}(t) \ . \tag*{(3)} \end{equation}

This readout defines a filter f(t):=e−γtvTeWtuf(t) := e^{- \gamma t} \bm{v}^T e^{\bm{W} t} \bm{u} since, on average,

Er[y(t)]=∫0tvTeAt′u x(t−t′) dt′=∫0tf(t′) x(t−t′) dt′ .\begin{equation} \mathbb{E}_{\bm{r}}[ y(t) ] = \int_0^t \bm{v}^T e^{\bm{A} t'} \bm{u} \ x(t-t') \ dt' = \int_0^t f(t') \ x(t-t') \ dt' \ . \tag*{(4)} \end{equation}

Assume we train some combination of u\bm{u}, v\bm{v}, and W\bm{W} so that y(t)y(t) matches a target output

y∗(t):=∫0te−c∗t′x(t−t′) dt′=∫0tf∗(t′)x(t−t′) dt′\begin{equation} y_*(t) := \int_0^t e^{- c_* t'} x(t - t') \ dt' = \int_0^t f_*(t') x(t - t') \ dt' \tag*{(5)} \end{equation}

which linearly filters its input x(t)x(t) using the target filter f∗(t):=e−c∗tf_*(t) := e^{- c_* t}. Here, c∗>0c_* > 0 controls the time scale on which input is leaky-integrated. Assume we do this using the supervised objective

J:=lim⁡T→∞1T∫0T12Ex,r{ [y(t)−y∗(t)]2 } dt+κ2∥u∥2+κ2∥v∥2 ,\begin{equation} J := \lim_{T \to \infty} \frac{1}{T} \int_0^T \frac{1}{2} \mathbb{E}_{x, \bm{r}}\left\{ \ [ y(t) - y_*(t) ]^2 \ \right\} \ dt + \frac{\kappa}{2} \Vert \bm{u} \Vert^2 + \frac{\kappa}{2} \Vert \bm{v} \Vert^2 \ , \tag*{(6)} \end{equation}

which demands that the network output y(t)y(t) agrees with y∗(t)y_*(t) on average. Note that we have also included a regularization term whose strength is determined by κ≥0\kappa \geq 0. This objective can be simplified if the input has white noise statistics [7, 4], i.e., if Ex[x(t)]=0\mathbb{E}_x[ x(t) ] = 0 and Ex[x(t)x(t−t′)]=δ(t−t′)\mathbb{E}_x[ x(t) x(t - t') ] = \delta(t - t'). We find (see Appendix § 9)

 J(u,v,W)=∫0∞12[f(t)−f∗(t)]2+e−2γt2vTeWtSeWTtv dt+κ2∥u∥2+κ2∥v∥2  .\begin{equation} \boxed{ \ J(\bm{u}, \bm{v}, \bm{W}) = \int_0^{\infty} \frac{1}{2} \left[ f(t) - f_*(t) \right]^2 + \frac{e^{- 2 \gamma t}}{2} \bm{v}^T e^{\bm{W} t} \bm{S} e^{\bm{W}^T t} \bm{v} \ dt + \frac{\kappa}{2} \Vert \bm{u} \Vert^2 + \frac{\kappa}{2} \Vert \bm{v} \Vert^2 \ } \ . \tag*{(7)} \end{equation}

Note that JJ has three terms: (i) a filter-matching term JfilterJ_{\text{filter}}, (ii) a noise term JnoiseJ_{\text{noise}}, and (iii) a regularization term JregJ_{\text{reg}} (Figure 1d). [4] only considered the filter-matching term; we will see that the other two terms strongly modulate learning dynamics.

Learning dynamics

RNNs are normally trained via backpropagation through time [18], FORCE [19], or real-time recurrent learning [20, 6]. One virtue of linear RNNs, at least for theory, is that gradients of JJ with respect to u\bm{u}, v\bm{v}, and W\bm{W} can be computed exactly, so we can ignore the extra structure associated with those learning rules. Assuming parameters are updated via gradient descent, in the gradient-flow limit in which the learning rate is taken infinitesimally small, we recover ordinary differential equations (ODEs) (see Appendix § 9)

References

  1. [1]Sompolinsky, H., Crisanti, A., & Sommers, H. J. (1988). Chaos in Random Neural Networks. Phys. Rev. Lett., 61, 259–262. https://doi.org/10.1103/PhysRevLett.61.259
  2. [2]Crisanti, A., & Sompolinsky, H. (2018). Path integral approach to random neural networks. Phys. Rev. E, 98, 062120. https://doi.org/10.1103/PhysRevE.98.062120
  3. [3]Faisal, A. A., Selen, L. P. J., & Wolpert, D. M. (2008). Noise in the nervous system. Nature Reviews Neuroscience, 9(4), 292–303. https://doi.org/10.1038/nrn2258
  4. [4]Bordelon, B., Cotler, J., Pehlevan, C., & Zavatone-Veth, J. A. (2025). Dynamics of learning to integrate in linear recurrent neural networks. arXiv E-prints, arXiv:2503.18754. https://doi.org/10.48550/arXiv.2503.18754
  5. [5]Vogels, T. P., Rajan, K., & Abbott, L. F. (2005). Neural network dynamics. Annual Review of Neuroscience, 28(Volume 28, 2005), 357–376. https://doi.org/10.1146/annurev.neuro.28.061604.135637
  6. [6]Irie, K., Gopalakrishnan, A., & Schmidhuber, J. (2024). Exploring the Promise and Limits of Real-Time Recurrent Learning. International Conference on Learning Representations, 2024, 27575–27598. https://proceedings.iclr.cc/paper_files/paper/2024/file/74aec30590e07dbe2e29879f9df14fb2-Paper-Conference.pdfproceedings.iclr.cc/paper_files/paper/2024/file/74aec30590e07dbe2e29879f9df14fb2-Paper-Conference.pdf
  7. [7]Proca, A. M., Dominé, C. C. J., Shanahan, M., & Mediano, P. A. M. (2025). Learning dynamics in linear recurrent neural networks. Proceedings of the 42nd International Conference on Machine Learning, 267, 49860–49902. https://proceedings.mlr.press/v267/proca25a.htmlproceedings.mlr.press/v267/proca25a.html
  8. [8]Huang, A., Singh, S. H., Martinelli, F., & Rajan, K. (2025). Measuring and Controlling Solution Degeneracy across Task-Trained Recurrent Neural Networks. Advances in Neural Information Processing Systems, 38, Main Conference, 105259–105304. https://doi.org/10.52202/085713-3517
  9. [9]Mante, V., Sussillo, D., Shenoy, K. V., & Newsome, W. T. (2013). Context-dependent computation by recurrent dynamics in prefrontal cortex. Nature, 503(7474), 78–84. https://doi.org/10.1038/nature12742
  10. [10]Rajan, K., Harvey, C. D., & Tank, D. W. (2016). Recurrent Network Models of Sequence Generation and Memory. Neuron, 90(1), 128–142. https://doi.org/10.1016/j.neuron.2016.02.009
  11. [11]Lim, S. H., Erichson, N. B., Hodgkinson, L., & Mahoney, M. (2021). Noisy Recurrent Neural Networks. Advances in Neural Information Processing Systems, 34, 5124–5137. https://proceedings.neurips.cc/paper_files/paper/2021/file/29301521774ff3cbd26652b2d5c95996-Paper.pdfproceedings.neurips.cc/paper_files/paper/2021/file/29301521774ff3cbd26652b2d5c95996-Paper.pdf
  12. [12]Yang, G. R., Joglekar, M. R., Song, H. F., Newsome, W. T., & Wang, X.-J. (2019). Task representations in neural networks trained to perform many cognitive tasks. Nature Neuroscience, 22(2), 297–306. https://doi.org/10.1038/s41593-018-0310-2
  13. [13]Driscoll, L. N., Shenoy, K., & Sussillo, D. (2024). Flexible multitask computation in recurrent networks utilizes shared dynamical motifs. Nature Neuroscience, 27(7), 1349–1363. https://doi.org/10.1038/s41593-024-01668-6
  14. [14]Vyas, S., Golub, M. D., Sussillo, D., & Shenoy, K. V. (2020). Computation Through Neural Population Dynamics. Annual Review of Neuroscience, 43(Volume 43, 2020), 249–275. https://doi.org/10.1146/annurev-neuro-092619-094115
  15. [15]Schuessler, F., Mastrogiuseppe, F., Dubreuil, A., Ostojic, S., & Barak, O. (2020). The interplay between randomness and structure during learning in RNNs. Advances in Neural Information Processing Systems, 33, 13352–13362. https://proceedings.neurips.cc/paper_files/paper/2020/file/9ac1382fd8fc4b631594aa135d16ad75-Paper.pdfproceedings.neurips.cc/paper_files/paper/2020/file/9ac1382fd8fc4b631594aa135d16ad75-Paper.pdf
  16. [16]Bengio, Y., Simard, P., & Frasconi, P. (1994). Learning long-term dependencies with gradient descent is difficult. IEEE Transactions on Neural Networks, 5(2), 157–166. https://doi.org/10.1109/72.279181
  17. [17]Pascanu, R., Mikolov, T., & Bengio, Y. (2013). On the difficulty of training recurrent neural networks. Proceedings of the 30th International Conference on Machine Learning, 28(3), 1310–1318. https://proceedings.mlr.press/v28/pascanu13.htmlproceedings.mlr.press/v28/pascanu13.html
  18. [18]Werbos, P. J. (1990). Backpropagation through time: what it does and how to do it. Proceedings of the IEEE, 78(10), 1550–1560. https://doi.org/10.1109/5.58337
  19. [19]Sussillo, D., & Abbott, L. F. (2009). Generating Coherent Patterns of Activity from Chaotic Neural Networks. Neuron, 63(4), 544–557. https://doi.org/10.1016/j.neuron.2009.07.018
  20. [20]Williams, R. J., & Zipser, D. (1989). A Learning Algorithm for Continually Running Fully Recurrent Neural Networks. Neural Computation, 1(2), 270–280. https://doi.org/10.1162/neco.1989.1.2.270
  21. [21]Lukoševičius, M., & Jaeger, H. (2009). Reservoir computing approaches to recurrent neural network training. Computer Science Review, 3(3), 127–149. https://doi.org/10.1016/j.cosrev.2009.03.005
  22. [22]Domine, C., Anguita, N., Proca, A. M., Braun, L., Kunin, D., Mediano, P., & Saxe, A. (2025). From Lazy to Rich: Exact Learning Dynamics in Deep Linear Networks. International Conference on Learning Representations, 2025, 102485–102536. https://proceedings.iclr.cc/paper_files/paper/2025/file/fde1a69a5b6e554b2f1f727197d2651d-Paper-Conference.pdfproceedings.iclr.cc/paper_files/paper/2025/file/fde1a69a5b6e554b2f1f727197d2651d-Paper-Conference.pdf
  23. [23]Smith, S., Elsen, E., & De, S. (2020). On the Generalization Benefit of Noise in Stochastic Gradient Descent. Proceedings of the 37th International Conference on Machine Learning, 119, 9058–9067. https://proceedings.mlr.press/v119/smith20a.htmlproceedings.mlr.press/v119/smith20a.html
  24. [24]Smith, S. L., & Le, Q. V. (2018). A Bayesian Perspective on Generalization and Stochastic Gradient Descent. International Conference on Learning Representations. https://openreview.net/forum?id=BJij4yg0Zopenreview.net/forum?id=BJij4yg0Z
  25. [25]Vastola, J. (2025). Generalization through variance: how noise shapes inductive biases in diffusion models. International Conference on Learning Representations, 2025, 34922–34960. https://proceedings.iclr.cc/paper_files/paper/2025/file/564ccf9de3a2a959d3d338889bad9d88-Paper-Conference.pdfproceedings.iclr.cc/paper_files/paper/2025/file/564ccf9de3a2a959d3d338889bad9d88-Paper-Conference.pdf
  26. [26]Bordelon, B., & Pehlevan, C. (2023). Self-consistent dynamical field theory of kernel evolution in wide neural networks*. Journal of Statistical Mechanics: Theory and Experiment, 2023(11), 114009. https://doi.org/10.1088/1742-5468/ad01b0
  27. [27]Vastola, J. J., & Rajan, K. (2026). Solvable models of learning to pursue a moving target. Reinforcement Learning Journal, 7.

Paper details

vlnc:W_4ktm6dn3
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

generative AI tools

View statement
Looking for related papers…
Contents