betaFeedback

WorksAgricultural and Biological Sciences

Dynamics of random recurrent neural networks: colored noise and sublinear activations

Abstract

We study the dynamics of recurrent neural networks with random couplings, sublinear odd activation functions, and temporally correlated noise. Using large N methods, we determine stationary correlations and distinguish the loss of local linear stability from the onset of chaos, characterized by a positive largest Lyapunov exponent. These thresholds delimit an intermediate regime in which local instability coexists with asymptotic stability under a common noise history. Their separation depends on both the noise correlation time and the asymptotic growth of the activation function. We show that internal noise with any finite correlation time cuts off the divergence of the network's intrinsic correlation time tau_corr at the edge of chaos; it exhibits a critical scaling tau_{corr,max} propto 1/sqrt{D_0}, where D_0 is the integrated noise strength. Static random input instead produces coincident thresholds and a divergent correlation time. We also calculate the frequency-dependent participation dimension from a four-point correlation function. Its zero-frequency value is inversely proportional to the fourth power of the network correlation time, predicting a smooth minimum at the weak-noise chaos threshold. Numerical simulations support the large-network predictions and reveal appreciable finite-size effects.

Introduction

Recurrent neural networks (RNNs) play a key role for various computational tasks in biological and artificial systems [1, 2]. An important goal is to understand their dynamics, given that the dynamical state of the system strongly influences its computational capabilities [3]. The seminal work of Sompolinsky, Crisanti and Sommers [4] established that large recurrent networks with random, asymmetric couplings and nonlinear activation can undergo a transition to chaos. Crucially, networks that operate near the transition to chaos are found to perform optimally [5, 6, 7]. The dynamical mean-field theory of [4] characterized the activity correlations and largest Lyapunov exponent. Subsequent path-integral work provided a systematic derivation of the mean-field description and its stability properties [8]. These results made random recurrent networks a tractable setting for studying how microscopic interactions generate complex temporal activity.

Fluctuating inputs add important modifications to this problem. Correlation theory for stochastic neural networks showed how network interactions shape temporal and cross-neuron correlations [9]. Additive noise was shown early to suppress chaos in random neural networks [10]. Deterministic periodic input can also suppress intrinsic chaos, with an effectiveness that depends nonmonotonically on the driving frequency [11]. For driven random networks, [12] found that time-dependent internal white noise can suppress chaos and shift its onset to stronger recurrent coupling. Their analysis also revealed the existence of an intermediate regime between the onset of local instability and the transition to chaos. Thus, the response to a small perturbation and the long-term growth of differences between trajectories describe distinct aspects of the dynamics. The intermediate regime is locally unstable but nonchaotic. These two transitions also provide a useful framework for interpreting changes in population covariance and effective recurrent coupling [13]. Related work on spiking circuits identified mechanisms for asynchronous chaotic rate fluctuations with distinct characteristic time scales [14].

In this work we study two aspects of RNNs that can qualitatively affect the dynamics: the temporal structure of disordered input and the activation function. First, prior works have mostly focused on the case of internal white noise, which corresponds to a vanishingly small correlation time. However, realistic internal fluctuations arising from neural and synaptic processes can persist over a finite interval. We will model this with an Ornstein–Uhlenbeck (OU) input that introduces a tunable correlation time τn\tau_n. Varying τn\tau_n allows us to ask how internal colored noise affects network correlations, local instability and the onset of chaos. Second, the activation function affects the dynamics in key ways. Many classical analyses use a bounded, saturating response, but other response shapes may be relevant over the range of activity sampled by the network. For example, type-I excitability is associated with a square-root dependence of firing rate on input near onset [15, 16, 17]. Near critical behavior generated by linear activation functions has been found to be relevant for describing brain activity [18, 19]. Previous work has shown that nonlinear response properties influence effective recurrent coupling and population correlations [13, 20]. In this work we use a smooth family of activation functions that spans saturating and sublinear (nonsaturating) responses and approaches a linear response as a limiting case.

Our goal is to determine, both theoretically and via numerical simulations, the dynamics of RNNs. In particular, we seek to understand how the colored noise and the form of the activation function affect the unstable dynamics and the development of chaos. We derive the large-network correlation equations, and a matched short-/long-time solution that reproduces the simulated correlators without measured input. We then examine two dynamical thresholds: local instability, from the local stability criterion, and chaos, from the sign of the largest Lyapunov exponent, which we obtain from a Schrödinger problem whose potential is built from the correlator of the derivative of the activation function. The two thresholds are distinct, with an intermediate regime that is locally unstable but not chaotic, and simulations map their separation as a function of the activation exponent and the noise correlation time. Near the edge of chaos, internal noise of any finite correlation time cuts off the divergence of the intrinsic correlation time with a universal exponent, τcorr,max∼D0−1/2\tau_{\mathrm{corr,max}} \sim D_0^{-1/2}, valid in the limit D0→0D_0 \to0 of integrated noise strength. For weak noise the maximum sits at the chaos threshold. The exponent is the same for white and colored noise. We also investigate whether these dynamical regimes differ in the geometric organization of population activity, as characterized by the participation dimension [21]. Its evaluation requires a four-point correlation function, which captures the contribution of correlations between neurons to the effective dimensionality of the activity. We find that the zero-frequency participation dimension is controlled by the same intrinsic time scale, DPR(0)/N=(τ/τcorr)4D_{\mathrm{PR}}(0)/N = (\tau/\tau_{\mathrm{corr}})^4, so that it has a minimum scaling as D02D_0^2 in the weak disorder limit. The simulations also allow us to determine the relevance of finite-size effects in the different regimes.

The paper is organized as follows. § 2 introduces the random recurrent network, the colored-noise process, the family of activation functions, and the numerical methods. § 3 develops the large-NN mean-field theory, analyzes the temporal correlations, and derives criteria for local linear instability and chaos. § 4 studies the dynamics in the weak disorder limit, obtaining explicit results for the instability and chaos thresholds and for the network correlation time. § 5 compares the theoretical predictions with simulations and presents phase diagrams showing how both transition thresholds depend on the noise correlation time and activation function. § 6 examines four-point correlations and the participation dimension of the network activity. § 7 contains a discussion of the results and conclusions. The appendices provide details of the collective-field and replica derivations and discuss finite-size effects.

The model

Dynamics, disorder and internal noise

We consider a network of NN recurrently coupled units with continuous time and a firing-rate dynamics. Denoting by ϕi(t)\phi_i(t) the total input current to unit ii, and by f(ϕ)f(\phi) a single-neuron activation function, the network obeys the Langevin equation

τdϕi(t)dt=−ϕi(t)+∑j=1NWijf(ϕj(t))+ξi(t),i=1,…,N,(1)\tau\frac{d\phi_i(t)}{dt}=-\phi_i(t)+\sum_{j=1}^{N}W_{ij}f\left(\phi_j(t)\right)+\xi_i(t),\qquad i=1,\ldots,N, \tag*{(1)}

where τ\tau is the single-neuron integration time constant. This is a standard model used throughout the random-network literature; in the linear case it reduces to a multivariate Ornstein-Uhlenbeck process. See e.g. [9, 18, 19, 4] for related rate-network dynamics and correlation theories. The model contains two independent sources of randomness, which play different roles: the WW-disorder and the internal noise.

The recurrent couplings WijW_{ij} are quenched (time-independent) random variables, drawn once per network realization from a Gaussian ensemble with

⟨Wij⟩=0,⟨WijWkl⟩=λ2Nδikδjl.(2)\langle W_{ij}\rangle=0,\qquad\langle W_{ij}W_{kl}\rangle=\frac{\lambda^2}{N}\delta_{ik}\delta_{jl}. \tag*{(2)}

The parameter λ\lambda controls the strength of the recurrent coupling and is the main control parameter of the theory: λ→0\lambda\to0 is the decoupled limit, while λ≫1\lambda\gg1 is the strong-coupling regime. Because WW does not evolve in time, its effect on the dynamics is entirely captured, at large NN, by disorder averages ⟨⋯ ⟩W\langle\cdots\rangle_W; this is the noise source responsible for generating an effective recurrent interaction in the single-neuron dynamics.

On the other hand, each neuron receives a time-dependent noise ξi(t)\xi_i(t), uncorrelated across neurons,

⟨ξi(t)⟩=0,⟨ξi(t)ξj(t′)⟩=δijGξ(t−t′).(3)\langle\xi_i(t)\rangle=0,\qquad\langle\xi_i(t)\xi_j(t')\rangle=\delta_{ij}G_{\xi}(t-t'). \tag*{(3)}

This is a dynamical (annealed) noise, and unlike most previous literature on random networks, which specializes to white noise, we keep Gξ(t−t′)G_{\xi}(t-t') general. This is what we will call colored noise. A convenient choice, used throughout this work, is an Ornstein-Uhlenbeck noise,

dξi(t)dt=−ξi(t)τn+σnηi(t),⟨ηi(t)ηj(t′)⟩=δijδ(t−t′),(4)\frac{d\xi_i(t)}{dt}=-\frac{\xi_i(t)}{\tau_n}+\sigma_n\eta_i(t),\qquad\langle\eta_i(t)\eta_j(t')\rangle=\delta_{ij}\delta(t-t'), \tag*{(4)}

whose stationary two-point function is

Gξ(t)=Dξe−∣t∣/τn,Dξ≡σn2τn2.(5)G_{\xi}(t)=D_{\xi}e^{-|t|/\tau_n},\qquad D_{\xi}\equiv\frac{\sigma_n^2\tau_n}{2}. \tag*{(5)}

We will denote

D0≡12∫−∞∞dt Gξ(t)=Dξτn.(6)D_0\equiv\frac{1}{2}\int_{-\infty}^{\infty}dt\,G_{\xi}(t)=D_{\xi}\tau_n. \tag*{(6)}

Two limits of (2.5) recover previously studied cases: the white-noise limit is τn→0\tau_n\to0, Dξ→∞D_{\xi}\to\infty with D0D_0 fixed, and reduces to the case considered in [12, 13]. The quenched limit is τn→∞\tau_n\to\infty at fixed DξD_{\xi}, in which each neuron sees an effectively static offset, Gξ(t)→DξG_{\xi}(t)\to D_{\xi}; this was studied in [20]. Away from these two limits, τn\tau_n defines a new time scale of the problem, to be compared with τ\tau and with the intrinsic correlation time generated self-consistently by the recurrent dynamics; one of our main goals below is to understand how this comparison controls the correlation structure and the onset of chaos. For a discussion of colored noise in the context of cortical neurons, and previous references, see [22].

Activation function

We take f(ϕ)f(\phi) to be odd, f(−ϕ)=−f(ϕ)f(-\phi)=-f(\phi), sublinear at large ∣ϕ∣|\phi|, and smooth. Building on our previous work [20], we use a Padé-type form

f(ϕ)=ϕ(1+β2ϕ2)1−p2,0≤p≤1, β>0,(7)f(\phi)=\frac{\phi}{\left(1+\beta^{2}\phi^{2}\right)^{\frac{1-p}{2}}}, \qquad0\le p\le1,\ \beta>0, \tag*{(7)}

which has the asymptotic behavior

f(ϕ)≈ϕ,for ∣ϕ∣≪1/βf(ϕ)≈βp−1sgn⁡(ϕ)∣ϕ∣p,for ∣ϕ∣≫1/β,(8)\begin{aligned} f(\phi)&\approx\phi,\quad&&\text{for }|\phi|\ll1/\beta\\ f(\phi)&\approx\beta^{p-1}\operatorname{sgn}(\phi)|\phi|^{p},\quad&&\text{for }|\phi|\gg1/\beta, \tag*{(8)} \end{aligned}

and is C∞(R)C^{\infty}(\mathbb{R}) everywhere. This smoothness will be convenient for perturbative expansions and for the numerical evaluation of ff and its derivatives. The gain parameter β\beta sets the scale at which the nonlinearity turns on.

When p→0p\to0 this gives a saturating activation function, which we expect will have similar properties than other saturating functions such as tanh⁡(x)\tanh(x). In the limit p→1p\to1, we obtain the linear case. In between both limits, we obtain a nonlinear and nonsaturating regime. In realistic situations the activation function will eventually saturate; but we will assume that the neural activation function is well approximated by (2.7) over a range of parameters. As a potential application, p=1/2p=1/2 corresponds to neural dynamics that undergo a type I bifurcation that leads to spike generation [15, 16, 17]. One of the goals of this work is to explore this new parameter space. In particular, we will find an interesting interplay between pp and the dynamical evolution of the network.

Numerical experiments

In this work we will use both numerical simulations and theoretical calculations to understand the dynamics of nonlinear RNNs. Before proceeding to a more detailed theoretical analysis, let us briefly discuss how (1) is numerically solved.

Eq. (1) together with the Ornstein-Uhlenbeck noise (4) is integrated with a simple Euler-Maruyama scheme at fixed step δt\delta t,

ξi(t+δt)=(1−δtτn)ξi(t)+δt σnηi(t),ηi(t)∼N(0,1) i.i.d.,(9)\xi_i(t+\delta t)=\left(1-\frac{\delta t}{\tau_n}\right)\xi_i(t)+\sqrt{\delta t}\,\sigma_n\eta_i(t),\qquad\eta_i(t)\sim\mathcal{N}(0,1)\ \text{i.i.d.}, \tag*{(9)}
ϕi(t+δt)=ϕi(t)+δtτ[−ϕi(t)+∑jWijf(ϕj(t))+ξi(t)].(10)\phi_i(t+\delta t)=\phi_i(t)+\frac{\delta t}{\tau}\left[-\phi_i(t)+\sum_j W_{ij}f\bigl(\phi_j(t)\bigr)+\xi_i(t)\right]. \tag*{(10)}

The first line is the Ornstein-Uhlenbeck process discretized at step δt\delta t up to O(δt)O(\delta t), with noise amplitude σn=2Dξ/τn\sigma_n = \sqrt{2D_\xi/\tau_n} fixed by (5); we verify this normalization directly, by simulating a large population of independent OU processes with no recurrent coupling and measuring ⟨ξi(t)ξi(0)⟩\langle\xi_i(t)\xi_i(0)\rangle, which reproduces Dξe−∣t∣/τnD_\xi e^{-|t|/\tau_n} to high accuracy. The couplings WijW_{ij} are drawn once per realization from (2). The activation function is (7) at fixed gain β\beta; the exponent pp is treated as an external parameter of the numerical experiment, and λ\lambda is scanned over a grid at fixed pp.

For each realization of WW (a “WW-seed”) the dynamics is run for a transient of δt ntran\delta t\,n_{\mathrm{tran}} time units to reach the stationary state, then for a further δt nmax\delta t\,n_{\mathrm{max}} time units during which the trajectory ϕi(t)\phi_i(t) is recorded. Repeating this for several independent noise realizations (“replicas”) at fixed WW, and averaging the population autocorrelation over neurons and replicas, gives numerical estimates of

Cϕ(t)=1N∑i⟨ϕi(t+t′)ϕi(t′)⟩,Cf(t)=1N∑i⟨f(ϕi(t+t′))f(ϕi(t′))⟩,(11)C_\phi(t) = \frac{1}{N}\sum_i \langle\phi_i(t+t')\phi_i(t')\rangle,\qquad C_f(t) = \frac{1}{N}\sum_i \langle f(\phi_i(t+t'))f(\phi_i(t'))\rangle, \tag*{(11)}

computed via FFT-based autocorrelation (and similarly for other correlators that will be needed). We have checked that after these averages and the transient, these correlators are independent of t′t', signaling a time-translation invariant state.

References

  1. [1]Z. C. Lipton, J. Berkowitz, and C. Elkan, “A critical review of recurrent neural networks for sequence learning,” 2015. https://arxiv.org/abs/1506.00019.
  2. [2]Y. Yu, X. Si, C. Hu, and J. Zhang, “A review of recurrent neural networks: Lstm cells and network architectures,” Neural Computation 31 no. 7, (07, 2019) 1235–1270. https://doi.org/10.1162/neco_a_01199.
  3. [3]M. I. Rabinovich, P. Varona, A. I. Selverston, and H. D. I. Abarbanel, “Dynamical principles in neuroscience,” Rev. Mod. Phys. 78 (Nov, 2006) 1213–1265. https://link.aps.org/doi/10.1103/RevModPhys.78.1213.DOI
  4. [4]H. Sompolinsky, A. Crisanti, and H. J. Sommers, “Chaos in random neural networks,” Phys. Rev. Lett. 61 (Jul, 1988) 259–262. https://link.aps.org/doi/10.1103/PhysRevLett.61.259.DOI
  5. [5]N. Bertschinger and T. Natschlager, “Real-time computation at the edge of chaos in recurrent neural networks,” Neural Computation 16 no. 7, (07, 2004) 1413–1436. https://doi.org/10.1162/089976604323057443.
  6. [6]D. Sussillo and L. F. Abbott, “Generating coherent patterns of activity from chaotic neural networks,” Neuron 63 no. 4, (2009) 544–557.DOI
  7. [7]T. Toyoizumi and L. F. Abbott, “Beyond the edge of chaos: Amplification and temporal integration by recurrent networks in the chaotic regime,” Phys. Rev. E 84 (Nov, 2011) 051908. https://link.aps.org/doi/10.1103/PhysRevE.84.051908.DOI
  8. [8]A. Crisanti and H. Sompolinsky, “Path integral approach to random neural networks,” Physical Review E 98 no. 6, (Dec., 2018) . http://dx.doi.org/10.1103/PhysRevE.98.062120.
  9. [9]I. Ginzburg and H. Sompolinsky, “Theory of correlations in stochastic neural networks,” Physical review E 50 no. 4, (1994) 3171.pubmed.ncbi.nlm.nih.gov/9962363
  10. [10]L. Molgedey, J. Schuchhardt, and H. G. Schuster, “Suppressing chaos in neural networks by noise,” Phys. Rev. Lett. 69 (1992) 3717–3719.pubmed.ncbi.nlm.nih.gov/10046895
  11. [11]K. Rajan, L. F. Abbott, and H. Sompolinsky, “Stimulus-dependent suppression of chaos in recurrent neural networks,” Phys. Rev. E 82 (2010) 011903.pubmed.ncbi.nlm.nih.gov/20866644
  12. [12]J. Schuecker, S. Goedeke, and M. Helias, “Optimal sequence memory in driven random networks,” Physical Review X 8 no. 4, (2018) 041029. https://doi.org/10.1103/PhysRevX.8.041029.
  13. [13]X. Shen and Y. Hu, “Covariance spectrum in nonlinear recurrent neural networks,” arXiv preprint arXiv:2508.05288 (2025).arxiv.org/abs/2508.05288
  14. [14]O. Harish and D. Hansel, “Asynchronous rate chaos in spiking neuronal circuits,” PLOS Computational Biology 11 no. 7, (2015) e1004266.DOI
  15. [15]B. Ermentrout, “Type i membranes, phase resetting curves, and synchrony,” Neural computation 8 no. 5, (1996) 979–1001.pubmed.ncbi.nlm.nih.gov/8697231
  16. [16]B. S. Gutkin and G. B. Ermentrout, “Dynamics of membrane excitability determine interspike interval variability: a link between spike generation mechanisms and cortical spike train statistics,” Neural computation 10 no. 5, (1998) 1047–1065.pubmed.ncbi.nlm.nih.gov/9654767
  17. [17]E. M. Izhikevich, Dynamical systems in neuroscience. MIT press, 2007.DOI
  18. [18]D. Dahmen, S. Recanatesi, X. Jia, G. K. Ocker, N. Lahiji, S. Musall, L. Campagnola, S. Seeman, T. Jarsky, M. Helias, and E. Shea-Brown, “Strong and localized recurrence controls the dimensionality of neural activity across brain areas,” Nature Neuroscience 29 no. 10, (2026) 2578–2587. https://doi.org/10.1038/s41593-026-02395-w.
  19. [19]D. Dahmen, S. Grün, M. Diesmann, and M. Helias, “Second type of criticality in the brain uncovers rich multiple-neuron dynamics,” Proceedings of the National Academy of Sciences 116 no. 26, (2019) 13051–13060. https://doi.org/10.1073/pnas.1818972116.
  20. [20]G. Mato, F. Rigatuso, and G. Torroba, “Statistics of correlations in nonlinear recurrent neural networks,” Phys. Rev. E 114 no. 1, (2026) 014226, arXiv:2510.21742 [q-bio.NC].DOI
  21. [21]P. Gao, E. Trautmann, B. Yu, G. Santhanam, S. Ryu, K. Shenoy, and S. Ganguli, “A theory of multineuronal dimensionality, dynamics and measurement,” BioRxiv (2017) 214262.DOI
  22. [22]N. Fourcaud and N. Brunel, “Dynamics of the firing probability of noisy integrate-and-fire neurons,” Neural Computation 14 no. 9, (09, 2002) 2057–2110, https://direct.mit.edu/neco/article-pdf/14/9/2057/815358/089976602320264015.pdf. https://doi.org/10.1162/089976602320264015.
  23. [23]G. Benettin, L. Galgani, A. Giorgilli, and J.-M. Strelcyn, “Lyapunov characteristic exponents for smooth dynamical systems and for hamiltonian systems; a method for computing all of them. part 1: Theory,” Meccanica 15 no. 1, (1980) 9–20.DOI
  24. [24]M. Helias and D. Dahmen, Statistical Field Theory for Neural Networks. Springer International Publishing, 2020. http://dx.doi.org/10.1007/978-3-030-46444-8.
  25. [25]K. T. Grosvenor and R. Jefferson, “The edge of chaos: quantum field theory and deep neural networks,” SciPost Physics 12 no. 3, (2022) 081, arXiv:2109.13247 [hep-th]. https://doi.org/10.21468/SciPostPhys.12.3.081.
  26. [26]D. G. Clark, L. Abbott, and A. Litwin-Kumar, “Dimension of activity in random neural networks,” Physical Review Letters 131 no. 11, (2023) 118401.
  27. [27]A. J. Wakhl...o, “Solution of a chaotic neural network at fixed connectivity,” 2026. https://arxiv.org/abs/2604.24141.
  28. [28]S. Song, P. J. Sjöström, M. Reigl, S. Nelson, and D. B. Chklovskii, “Highly nonrandom features of synaptic connectivity in local cortical circuits,” PLoS Biology 3 no. 3, (2005) e68.pubmed.ncbi.nlm.nih.gov/15737062
  29. [29]S. Recanatesi, S. Bradde, V. Balasubramanian, N. A. Steinmetz, and E. Shea-Brown, “A scale-dependent measure of system dimensionality,” Patterns 3 no. 8, (2022) .DOI

Paper details

vlnc:W_4suathsm
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 2

A view counts once per browser per day, a download each time the PDF is requested.

  • Agricultural and Biological Sciences

Not recorded for this paper

View statement
Looking for related papers…
Contents