betaFeedback

WorksPhysics and Astronomy

Towards Critical Branching Mechanism in Recurrent Neural Networks

Abstract

Criticality has been proposed as a key organizing principle in biological neural systems, yet its origin and relevance in artificial neural networks remain unclear. We analyze hidden-state dynamics in trained long short-term memory (LSTM) networks and show that small networks near their optimal training epochs (steps) exhibit scale-free avalanche statistics and branching parameters close to unity, indicative of near-critical dynamics, while larger models remain subcritical. To explain the coexistence of subcritical branching with robust 1/fβ1/f^β noise, we introduce a mixture branching process framework that links heterogeneous branching dynamics to long-range temporal correlations. These results identify critical-like behavior in LSTMs as an emergent, capacity-dependent dynamical regime.

Introduction

Scientists have long drawn inspiration from biological neural systems in the design of Artificial Neural Networks (ANNs), with an emphasis on mimicking their structural and functional characteristics. By investigating the dynamical behavior of biological brains, researchers seek to identify principles that can be transferred to ANN design Hopfield [ [1], Rumelhart et al. [ [2]. One key finding in this area is that both biological brains and ANNs tend to operate near the edge of chaos, a critical state that maximizes information-processing capacity Mora and Bialek [ [3], Bertschinger et al. [ [4], Boedecker et al. [ [5], Feng et al. [ [6], Wei et al. [ [7]. Motivated by the discovery of neuronal avalanches in biological systems Beggs and Plenz [ [8], scale-free avalanche-like activity propagating across layers has also been observed in deep feedforward neural networks such as ResNet Ghavasieh et al. [ [9], He et al. [ [10]. Furthermore, accumulating evidence suggests that both biological neural systems and Long Short-Term Memory (LSTM) networks Hochreiter and Schmidhuber [ [11] display 1/fβ1/f^{\beta} noise in their activity, a statistical hallmark often associated with self-organized criticality (SOC) Linkenkaer-Hansen et al. [ [12], Montez et al. [ [13], Pathania et al. [ [14], Allegrini et al. [ [15], Chong and Feng [ [16], Bak et al. [ [17], Per Bak and Wiesenfeld [ [18].

In the context of neuroscience, the SOC hypothesis has been extensively investigated, particularly in in-vitro cortical networks, where neural activity organizes into scale-free cascades known as neuronal avalanches Beggs and Plenz [ [8], Plenz et al. [ [19], Yu et al. [ [20], Petermann et al. [ [21]. These avalanches are characterized by power-law distributions of event sizes and durations, indicating the absence of characteristic scales. From a theoretical perspective, such dynamics can be modeled using a mean-field sandpile model, which is mathematically equivalent to a critical branching process Christensen and Olami [ [22], Zapperi et al. [ [23]. Within this framework, the branching parameter serves as a control variable that quantifies the balance between activity propagation and decay, with criticality emerging when the system is poised between quiescence and runaway excitation Harris et al. [ [24].

While the SOC paradigm and avalanche dynamics in biological neural systems are well established, the relevance to ANNs remains far less understood. In particular, although recent studies have reported the presence of 1/fβ1/f^{\beta} noise and long-range temporal correlations in LSTM networks Chong and Feng [ [16], a unifying theoretical framework explaining the emergence of such scale-free behavior in trained networks is still lacking. Unlike biological systems, ANNs are deterministic, parameterized models trained via gradient-based optimization, raising fundamental questions about how critical-like dynamics arise in the absence of explicit self-organizing mechanisms.

Here, we propose a framework to extend the concept of neuronal avalanches from neuroscience to the realm of ANNs, with a particular focus on LSTM networks. By defining avalanche-like events in LSTM hidden-state dynamics and analyzing their statistical properties, we establish a direct connection between LSTM avalanches and the branching process. Furthermore, we demonstrate that the emergence of 1/fβ1/f^{\beta} noise in LSTMs is closely linked to the underlying branching structure of network dynamics, providing a possible mechanistic interpretation of scale-free temporal correlations in LSTMs.

The remainder of this paper is organized as follows. In § 2, we introduce the LSTM networks, dataset, and training protocols in our analysis. In § 3, we define avalanche events in LSTM dynamics and demonstrate the emergence of the scale-free avalanche size distribution at specific stages of training and for particular network configurations. In § 4, we adopt a branching-process framework to account for the observed avalanche behaviors and apply an established method to estimate the branching parameter, which serves as a quantitative indicator of the dynamical state of the network. However, the branching-process analysis in § 4 primarily captures short-range temporal correlations and does not account for the long-range 1/fβ1/f^{\beta} noise observed in LSTM activity. To address this limitation, in § 5 we investigate the long-range temporal correlations in LSTMs and introduce an extended branching mechanism capable of reproducing the observed 1/fβ1/f^{\beta} behavior. This framework yields an ensemble-averaged branching parameter whose evolution mirrors that of the conventional branching parameter across training epochs and network configurations. Throughout the paper, an epoch denotes a discrete step in the training process corresponding to one full pass over the dataset.

LSTM networks

The LSTM architecture employed in this work consists of three components: an embedding layer, a single LSTM layer, and a linear classifier implemented as a dense layer followed by a sigmoid function, as shown in Figure 1. The embedding layer maps each discrete input token onto a continuous representation in an eight-dimensional (vector) latent space. The single-layer LSTM can be mathematically described as Hochreiter and Schmidhuber [ [11]:

Figure 1. Schematic of the LSTM architecture used for binary sentiment classification. Block A corresponds to the LSTM layer, with the internal gating and state-update operations shown in the pink inset.

it=σs(Wixt+Uiht−1+bi),ft=σs(Wfxt+Ufht−1+bf),ot=σs(Woxt+Uoht−1+bo),c~t=σt(Wcxt+Ucht−1+bc),ct=ft⊙ct−1+it⊙c~t,cout,t=σt(ct),ht=ot⊙cout,t,(1)\begin{aligned} i_t &= \sigma_s(W_i x_t + U_i h_{t-1} + b_i),\\ f_t &= \sigma_s(W_f x_t + U_f h_{t-1} + b_f),\\ o_t &= \sigma_s(W_o x_t + U_o h_{t-1} + b_o),\\ \tilde{c}_t &= \sigma_t(W_c x_t + U_c h_{t-1} + b_c),\\ c_t &= f_t \odot c_{t-1} + i_t \odot\tilde{c}_t,\\ c_{\mathrm{out},t} &= \sigma_t(c_t),\\ h_t &= o_t \odot c_{\mathrm{out},t}, \tag*{(1)} \end{aligned}

where xtx_t denotes the eight-dimensional embedding vector corresponding to the input token at timestep tt, ⊙\odot indicates element-wise multiplication, σs(x)=11+e−x\sigma_s(x) = \frac{1}{1+e^{-x}} and σt(x)=tanh⁡(x)\sigma_t(x) = \tanh(x) are the sigmoid and hyperbolic tangent functions, respectively, and {W,U,b}\{W, U, b\} represent the trainable parameters of the LSTM layer. At each timestep tt, the embedded input xtx_t is fed into the LSTM layer together with the previous hidden state ht−1h_{t-1} and cell state ct−1c_{t-1}, which store the memory of the network, resulting in an updated hidden state hth_t and cell state ctc_t. After the entire review is processed by the LSTM layer, the final hidden state h−1h_{-1} is passed to the classifier, which outputs a value in the interval [0,1][0, 1] for binary classification.

We trained multiple LSTM networks sharing the same architecture described above, while varying the dimensionality of the hidden state hth_t across the set {8,16,32,48,64,80,96,112,128}\{8, 16, 32, 48, 64, 80, 96, 112, 128\}. The networks were trained on a dataset of 50, 000 IMDb movie reviews Maas et al. [ [25], labeled as positive opinion (1) or negative opinion (0). To account for the inherent stochasticity in neural network training and to ensure the statistical robustness of the results presented in this work, each configuration was trained using 28 different random seeds; a small number of trained networks were excluded for reasons discussed in Appendix § 11. Across all realizations, most networks saturated before epoch 30 at a test accuracy of approximately 87%, whereas larger networks exhibited delayed optimization for a small subset of random seeds, as illustrated in Figure 2. All observables measured below were computed during inference using the network checkpoints at the specified epoch. For a given hidden-state dimensionality and training epoch, the same quantity was evaluated across all experiments trained with different random seeds, and the results were averaged, with error bars representing the standard deviation unless explicitly stated otherwise. Although reviews in the dataset varied in length, all were truncated or padded to a fixed length of 500 tokens. The dataset was then split into a training set of 35, 000 reviews and a test set of 15, 000 reviews. All networks were trained using an identical set of hyperparameters, summarized in Table 1.

Figure 2. Distribution of optimal epochs (i.e., the epoch with the lowest test loss during training) as a function of hidden-state dimensionality for LSTM networks. Each violin plot summarizes the distribution of optimal epochs across multiple network realizations with different random seeds but identical architectures. Black dots denote individual realizations. The dashed horizontal line marks epoch 30, which corresponds to the epoch at which the majority of networks achieve their best performance across hidden-state dimensions.

Batch sizeEmbedding sizeDropout
12880.1
L2 regularizationLearning rateGradient clipping
0.00050.0055

Table 1. Selected hyperparameters used during LSTM training.

Avalanches

In neuroscience, neuronal avalanches are defined as cascades of spatiotemporal bursts of neural activity separated by intervals of quiescence. Experimentally, such avalanches are typically identified using multielectrode arrays placed onto cortical slices, which record local field potentials (LFPs) from multiple electrodes Beggs and Plenz [ [8], Petermann et al. [ [21], Priesemann et al. [ [26]. Neuronal populations in the vicinity of a given electrode are considered active within a discrete time bin if a negative spike exceeding a predefined threshold is detected in the corresponding LFP signal during that interval. An avalanche is operationally defined as a sequence of consecutively active time bins, where a bin is considered active if it has at least one active electrode, preceded and followed by inactive bins. The size and duration distribution of the avalanches in cortical slices are observed to be scale-free, a hallmark of critical dynamics in biological neural systems.

Motivated by this experimental framework, we modeled each dimension of the hidden states hth_t in LSTM as an artificial neuron and applied an analogous avalanche analysis during network inference, as illustrated in Figure 3(a). Specifically, reviews from the test set of the IMDb dataset were processed using the trained LSTM networks. For each review, a sequence of 500 tokens was sequentially fed into the LSTM layer, yielding hth_t at every timestep. To render the activities of different artificial neurons directly comparable, each dimension of the resulting hth_t was independently standardized across time using zz-score normalization described below.

Extraction of neural activities from an LSTM network

Figure 3. Extraction of neural activities from an LSTM network. (a) Schematic of the analysis pipeline. LSTM processes test reviews to generate hidden states hth_t, which are recorded and normalized for subsequent analysis. (b) Time series of normalized hidden states for selected dimensions ii. A uniform activity threshold (shaded regions) is applied independently to each normalized hidden-state dimension to identify significant activation events. For illustration, the hidden-state dimensionality is set to eight; in the experiments, networks with varying hidden dimensions were analyzed. (c) Binary raster plot of detected neural activities across dimensions after thresholding (top), together with the corresponding XtX_t, defined as the number of active neurons at each timestep tt (bottom). Consecutive timesteps for which Xt≠0X_t \ne0 (red regions) are identified as avalanches.

For a given hidden-state dimension ii, the temporal mean μi\mu_i and standard deviation of the ii-th hidden dimension were computed over the interval t∈[t0,T]t \in[t_0,T], where T=500T = 500. The initial segments t<t0t < t_0 were discarded as a burn-in period, during which the hidden-state dynamics were strongly influenced by the initialization of the hidden states h0h_0, and cell states c0c_0. The normalized neural activities were defined as the absolute deviation from the temporal mean measured in units of the corresponding standard deviation,

μi=1T−t0+1∑t′=t0Tht′,i,∣ht,inorm∣=∣ht,i−μi∣1T−t0+1∑t′=t0T(ht′,i−μi)2(2)\mu_i = \frac{1}{T-t_0+1}\sum_{t'=t_0}^{T} h_{t',i}, \qquad \left|h_{t,i}^{\mathrm{norm}}\right| = \frac{\left|h_{t,i}-\mu_i\right|}{\sqrt{\frac{1}{T-t_0+1}\sum_{t'=t_0}^{T}\left(h_{t',i}-\mu_i\right)^2}} \tag*{(2)}

where ii indexes the hidden-state dimension and T−t0+1T-t_0+1 denotes the length of the time window used for normalization. This normalization step ensures that differences in activity scale across artificial neurons do not bias the subsequent avalanche detection. As shown in Figure 3(b), after normalization, all dimensions of the normalized neural activities ∣htnorm∣\left|h_t^{\mathrm{norm}}\right| were binarized using a uniform threshold θ\theta, which was manually selected and applied consistently across dimensions and time. An artificial neuron was considered active at a given timestep if the amplitude of its normalized hth_t exceeded this threshold, as demonstrated in Figure 3(c). This binarization defines a binary state matrix sts_t with elements

st,i={1,if ∣ht,inorm∣>θ,0otherwise.(3)s_{t,i} = \begin{cases} 1, & \text{if } \left|h_{t,i}^{\mathrm{norm}}\right| > \theta,\\ 0 & \text{otherwise.} \end{cases} \tag*{(3)}

A timestep was classified as active if at least one neuron was active at that moment. An avalanche was then identified as a sequence of consecutive active timesteps bounded by silent periods on both sides. The use of zz-score normalization and threshold-based binarization follows standard practice in studies of neuronal avalanches Plenz et al. [ [19], Yu et al. [ [20], from which our methodology is inspired. This entire pipeline was applied to all 15,000 test reviews, after which we analyzed the avalanche statistics over the entire set.

We analyzed the distribution of avalanche size P(S)P(S), where the size SS is quantified as the total number of active neurons accumulated over the duration of an avalanche, across networks with different sizes. For the network with the smallest hidden-state dimensionality at its optimal epoch, the avalanche size followed a power-law distribution with an exponential cutoff. The presence of this cutoff reflects finite-size effects, which are expected in any system with finite degrees of freedom, including biological neural systems Beggs and Plenz [ [8]. As shown in Figure 4(a)-(Figure 4c), within a reasonable range of threshold values θ\theta, the power-law behavior remained qualitatively stable. However, the estimated power-law exponent varied substantially under different thresholding conditions, indicating limited robustness of the avalanche statistics, similar to what has been reported for neuronal avalanches in biological neural systems. In the early epochs, the smallest network typically exhibited a pronounced exponential decay in the avalanche size distribution, suggesting subcritical dynamics Christensen and Olami [ [22]. In Figure 4(d), for larger networks, the avalanche size generally deviates from the power-law distribution at both early and optimal training epochs. For clarity, only the case with hidden-state dimension 128 is shown in the figure; corresponding results for other large hidden-state dimensions are provided in Appendix § 11. This behavior indicates that large networks remain subcritical during training, with no transition toward criticality observed.

Figure 4. Log–log plots of avalanche size distributions obtained from thresholded hidden-state activities across different operating regimes. Red markers correspond to the optimal training epoch, while blue markers indicate an early training stage. Here, epoch 30 is selected as a representative optimal epoch across realizations, despite variation in the exact optimal epoch for different networks (Figure 2). This definition is used consistently in all subsequent figures unless noted otherwise. Error bars represent statistical variability across networks trained with different random seeds but identical architectures and training epochs. Dashed lines denote power-law guides, fitted to the avalanche size distributions at the optimal epoch, with the corresponding exponent α\alpha shown for reference. Panels (a–c) show results for a network with hidden-state dimensionality 8 at increasing activity threshold ϕ\phi: (a) ϕ=1.0\phi= 1.0, (b) ϕ=1.5\phi= 1.5, and (c) ϕ=2.0\phi= 2.0. Panel (d) shows the avalanche size distribution for a larger network (hidden-state dimension 128) at ϕ=1.5\phi= 1.5. Results for additional large hidden-state dimensions, which exhibit similar behavior, are provided in Appendix § 11. The observed power-law scaling for small networks remains qualitatively consistent across threshold values ϕ\phi, supporting the presence of near-critical dynamics in small LSTM networks.

Branching process

To account for scale-free avalanche statistics, biological neural systems are often modeled as a mean-field sandpile, which can be mathematically described as a branching process governed by an order parameter known as the branching parameter mm Beggs and Plenz [ [8], Harris et al. [ [24]. The dynamics were described as

Xt+1=∑i=1XtYt,i,⟨Yt,i⟩=m,(4)X_{t+1} = \sum_{i=1}^{X_t} Y_{t,i}, \qquad\langle Y_{t,i}\rangle= m, \tag*{(4)}

where XtX_t denotes the number of toppled sites (or active electrodes) at discrete time tt, and Yt,iY_{t,i} represents the number of offspring events generated by the ii-th toppled site at that time. The branching parameter mm quantifies the expected number of descendants per toppled site. As mm approaches one, the sandpile self-organizes into a critical state, where a perturbation can trigger a cascade of events that neither dies out nor explodes. In this critical regime, the avalanche size distribution of this sandpile follows a power law with a critical exponent of τs=32\tau_s = \frac{3}{2}, independent of the threshold value. Notably, the robustness of the exponent τs\tau_s in the mean-field theory is considerably stronger than that observed in cortical slice experiments Beggs and Plenz [ [8], a discrepancy that is commonly attributed to the idealized connectivity structure of the mean-field model. In contrast, when m<1m < 1, cascades eventually dissipate, and the avalanche size distribution will deviate from the power-law behavior. Consequently, measuring the branching parameter mm is crucial for understanding the mechanism underlying the avalanche dynamics in LSTM.

Unlike the mean-field sandpile model or in-vitro brain systems, which are externally driven only after relaxation, artificial neurons in LSTM are continuously driven by external input tokens. To capture the branching dynamics underlying the LSTM avalanche, we hypothesized a driven variation of the branching process given by the following equation:

Xt+1=∑i=1XtYt,i+It,⟨Yt,i⟩=m,(5)X_{t+1} = \sum_{i=1}^{X_t} Y_{t,i} + I_t, \qquad\langle Y_{t,i}\rangle= m, \tag*{(5)}

where Xt=∑ist,iX_t = \sum_i s_{t,i} denotes the number of active artificial neurons at timestep tt, Yt,iY_{t,i} represents the number of descendants generated by the ii-th neuron at that timestep, and ItI_t denotes the external drive at timestep tt, assumed to have finite mean and variance.

We then employed the multi-regressive (MR) estimator Wilting and Priesemann [ [27] to measure the branching parameter mm, as shown in Figure 5(a). Because both the cell state ctc_t and hidden state hth_t were updated at every timestep, the full dynamical system evolved in a state space larger than spanned by hth_t alone. However, since only hth_t was used in our analysis, this procedure effectively constituted spatial subsampling of the underlying dynamics. The MR estimator was explicitly developed to address such subsampling scenarios and is known to yield unbiased estimates of the branching parameter under these conditions. In addition, thresholding in the avalanche detection procedure suppressed low-amplitude activities, which can likewise be interpreted as an additional form of spatial subsampling. For each input review processed by the LSTM, the hidden state hth_t yields an individual activity time series {Xti}\{X_t^i\}, where rr indexes the review. The autocorrelation function (ACF) was computed separately for each activity time series and then averaged across all reviews. The MR estimation was subsequently performed by fitting this review-averaged ACF,

Estimation of the branching parameter from temporal correlations in LSTM activities

Figure 5. Estimation of the branching parameter from temporal correlations in LSTM activities. (a) Schematic of the analysis pipeline. For each trained network, test reviews are processed to obtain the activity signal XtX_t. The ACF C[τ]C[\tau] is computed for each signal and averaged across reviews. The averaged ACF is then fitted with an exponential decay to extract an effective branching parameter. This procedure is repeated across networks with different hidden-state dimensions, across independent training runs governed by different random seeds, and across training epochs. (b) Evolution of the estimated branching parameter mm over training epochs for different hidden-state dimensions, obtained from discrete-count activity signals XtX_t (left) and continuous-valued definitions of XtX_t that eliminate explicit thresholding and binarization of the hidden states hth_t (right). Solid lines indicate the mean across networks sharing the same hidden-state dimensionality but trained under different random seeds, while shaded regions denote variability arising from stochasticity in the training process. The red dashed vertical lines and star markers highlight the epoch interval during which the majority of networks achieve their optimal performance. Larger hidden dimensions systematically yield lower effective branching parameters, while training drives small networks toward a near-critical regime.

C[τ]=1R∑rCov⁡(Xtr,Xt+τr)Var⁡(Xtr),(6)C[\tau] = \frac{1}{R} \sum_r \frac{\operatorname{Cov}(X_t^r, X_{t+\tau}^r)}{\operatorname{Var}(X_t^r)}, \tag*{(6)}

where RR denotes the total number of reviews. Hereafter, the term ACF refers to this review-averaged autocorrelation function. The averaged ACF was fitted with an exponential form bmτbm^\tau over relatively short time lags up to τ=10\tau= 10, owing to the limited sequence length of XtX_t, where bb is a constant. In the left panel of Figure 5(b), after optimization, mm approached values close to 0.9 as the hidden-state dimensionality decreased, indicating increasingly critical dynamics in smaller networks. For small networks, mm increased monotonically over the course of training and attained its maximum in the vicinity of the optimal epoch. Furthermore, note that the MR estimation method does not impose any requirement for XtX_t to be represented as integer counts or real-valued quantities. This flexibility allowed XtX_t to be redefined in terms of the total intensities of the active neurons at each timestep tt. Specifically, we defined

Xt=∑i∣ht,inorm∣,(7)X_t = \sum_i \left|h_{t,i}^{\mathrm{norm}}\right|, \tag*{(7)}

which eliminates the need for an explicit thresholding procedure. Importantly, the conclusions drawn from this alternative continuous-valued definition of XtX_t were consistent with those derived from the integer-valued formulation, as evidenced by the agreement between the left and right panels of Figure 5(b), reinforcing the robustness of our findings.

These observations are qualitatively in agreement with avalanche statistics discussed in the previous section and support the hypothesized driven branching process in Eq. (5) as a plausible mechanism underlying LSTM avalanche dynamics.

Long-range dynamics

While the preceding sections established a framework for avalanche analysis in LSTMs, the connection between the inferred branching mechanism and the observed 1/fβ1/f^\beta noise remains unsolved. This gap is particularly striking in light of the mean-field sandpile theory, where 1/fβ1/f^\beta noise arises from the scale-free duration of avalanches at criticality, while subcritical sandpiles (m<1m < 1) exhibit a finite characteristic timescale that leads to a Lorentzian power spectral density (PSD) Per Bak and Wiesenfeld [ [18], Jensen et al. [ [28], Laurson et al. [ [29]. In contrast to these theoretical expectations, prior work Chong and Feng [ [16] reported robust 1/fβ1/f^\beta noise even in larger LSTM networks despite their relatively low branching parameter mm. This apparent discrepancy suggests that the driven branching model, inferred from the short-range ACF of XtX_t, is insufficient to account for the long-range 1/fβ1/f^\beta noise observed in LSTMs.

It is important to note, however, that the activity signal analyzed in Ref. Chong and Feng [ [16] is not identical to the definition of XtX_t adopted here. Specifically, that study considered the summed hidden-state activity ∑iht,i\sum_i h_{t,i}, whereas in this work XtX_t was defined either as a discrete count of active neurons or as a continuous-valued aggregation of normalized amplitudes. To ensure a consistent and robust characterization of long-range temporal correlations under this generalized definition, we estimated the spectral exponent β\beta of 1/fβ1/f^\beta noise in XtX_t using Detrended Fluctuation Analysis (DFA) Peng et al. [ [30], Hardstone et al. [ [31]. In DFA, for each input review, XtrX_t^r is partitioned into windows of length ss, locally detrended within each window, and the fluctuation function Fr(s)F_r(s) is defined as the root-mean-square (RMS) magnitude of the detrended residuals, averaged over all windows at scale ss. The final fluctuation function F(s)F(s) is then obtained by averaging Fr(s)F_r(s) across all reviews. For scale-invariant processes, F(s)F(s) follows a power-law dependence F(s)∼sαF(s) \sim s^\alpha, from which the spectral exponent is obtained via β=2α−1\beta= 2\alpha- 1 Buldryev et al. [1995]. As shown in Figure 6(b), for LSTMs operating near their optimal epoch, the measured F(s)F(s) exhibits power-law scaling over approximately two decades in window size ss. Correspondingly, as shown in Figure 6(c), the inferred spectral exponent β\beta increases toward values near 0.8 as the hidden-state dimension decreases, in qualitative agreement with Ref. Chong and Feng [ [16]. By contrast, at early training epochs, F(s)F(s) is not well described by a single power law and instead displays a crossover with different effective scaling exponents at small and large window sizes, as illustrated in Figure 6(a).

Training-dependent scaling behavior and temporal correlations in LSTM activities

Figure 6. Training-dependent scaling behavior and temporal correlations in LSTM activities. Panels (a) and (b) show DFA fluctuation functions F(s)F(s) computed from the discrete-count activity signals XtX_t as a function of scale ss for different hidden-state dimensions, evaluated at an early training epoch (a) and at the optimal epoch (b). Panel (c) presents the spectral exponent β\beta of the corresponding 1/fβ1/f^\beta noise at the optimal epoch, obtained by fitting F(s)F(s) in panel (b), shown as a function of the LSTM hidden-state dimension; error bars are defined as in Figure 4. Panels (d) and (e) display the corresponding ACF C[τ]C[\tau] as a function of time lag τ\tau at the early (d) and optimal (e) epochs, respectively; shaded regions are defined as in Figure 5. At the optimal epoch, the activity exhibits well-defined scale-invariant behavior over multiple window sizes together with an extended correlation length. In contrast, at the early training epoch, F(s)F(s) deviates from a single power-law form and the ACF decays rapidly, indicating short-range temporal correlations. Corresponding results for the continuous-valued XtX_t are provided in Appendix § 11.

To elucidate the origin of the observed 1/fβ1/f^\beta noise in XtX_t, it is thus necessary to probe the ACF C[τ]C[\tau] over substantially longer timescales. In Figure 6(e), when evaluated over two decades in time lag τ\tau, C[τ]C[\tau] computed from XtX_t at the optimal epoch of all LSTMs exhibits a heavy-tailed behavior that can be well captured by a power law with an exponential cutoff

C[τ]∼τ−ηe−τ/τc,(8)C[\tau] \sim\tau^{-\eta}e^{-\tau/\tau_c}, \tag*{(8)}

where η\eta is the power-law exponent, and τc\tau_c denotes the characteristic cutoff timescale. To mitigate the statistical fluctuations across different realizations, especially at large τ\tau, we fitted this form to the empirical C[τ]C[\tau], obtained by averaging ACFs across networks trained with different random seeds at the same training epoch and hidden-state dimensionality before fitting the resulting averaged correlation function. This contrasts with the short-range MR analysis, where the branching parameter is fitted separately for each seed and then averaged. The fitted exponent η\eta increases systematically with the hidden-state dimensionality, mirroring our earlier observations regarding the dimensional dependence of β\beta at the optimal epoch. This correspondence is expected on general grounds, since the scaling form in Eq. (8) directly constrains the high-frequency behavior of the PSD. Specifically, a correlation function of the form C[τ]∼τ−ηe−τ/τcC[\tau] \sim\tau^{-\eta}e^{-\tau/\tau_c} yields a PSD that is flat for ω≪ωc\omega\ll\omega_c and scales as S(ω)∼ω−(1−η)S(\omega) \sim\omega^{-(1-\eta)} for ω≫ωc\omega\gg\omega_c, with crossover frequency ωc=1/τc\omega_c = 1/\tau_c (see Appendix § 8 for a derivation). Because τc\tau_c is typically large in our setting, the experimentally accessible spectrum is dominated by the high-frequency regime, implying β=1−η\beta= 1-\eta, in agreement with the empirically observed anti-correlation between β\beta and η\eta. In contrast, C[τ]C[\tau] at the early epochs displays a pure exponential decay, vanishing over very short time lags. Such an exponential ACF implies a Lorentzian power spectrum that is flat at low frequencies and crosses over to 1/f21/f^2 at high frequencies, consistent with the two-regime scaling behavior of F(s)F(s) observed during early epochs.

Motivated by the fact that a single branching process cannot generate a genuinely long-ranged temporal correlation, and by the limitation of our preceding analysis in assuming a homogeneous mechanism across different input reviews, we hypothesized a more general mechanism, termed the mixture branching process (MBP), to account for the observed behavior. In the MBP, each review induces its own branching dynamics so that the activity XtX_t associated with review rr evolves with a review-specific branching parameter mrm_r,

Xt+1r=∑i=1XtrYt,ir+Itr,⟨Yt,ir⟩=mr,(9)X_{t+1}^{r} = \sum_{i=1}^{X_t^{r}} Y_{t,i}^{r} + I_t^{r}, \qquad\langle Y_{t,i}^{r} \rangle= m_r, \tag*{(9)}

where rr indexes the review. The observed dynamics then arise from the superposition of branching processes with heterogeneous mrm_r, which can effectively reproduce the heavy-tailed decay of C[τ]C[\tau]. Within this framework, the review-specific parameters mrm_r are sampled from a distribution W(mr)W(m_r), which can be derived analytically from the scaling form C[τ]∼τ−ηe−τ/τcC[\tau] \sim\tau^{-\eta}e^{-\tau/\tau_c}. The parameters η\eta and τc\tau_c are obtained by the fitting procedure described above. As proven in Appendix B, the analytical form of W(mr)W(m_r) is

W(mr)∝(−ln⁡mr−λc)η−1Γ(η)mr1[mmin⁡,e−λc](mr),(10)W(m_r) \propto\frac{(-\ln m_r-\lambda_c)^{\eta-1}}{\Gamma(\eta)m_r}\mathbf{1}_{[m_{\min},e^{-\lambda_c}]}(m_r), \tag*{(10)}

where Γ(⋅)\Gamma(\cdot) denotes the Gamma function, λc=1/τc\lambda_c = 1/\tau_c, and 1[mmin⁡,e−λc](mr)\mathbf{1}_{[m_{\min},e^{-\lambda_c}]}(m_r) is a boxcar function that equals unity for mr∈[mmin⁡,e−λc]m_r \in[m_{\min},e^{-\lambda_c}] and vanishes otherwise. The introduction of a lower bound mmin⁡m_{\min} is required to prevent an unphysical divergence associated with extremely small mrm_r. Importantly, the specific choice of mmin⁡m_{\min} affects only short-timescale behavior and does not alter the heavy-tailed regime of C[τ]C[\tau], which is the primary focus of our long-range analysis. In the calculations that follow, we therefore fix mmin⁡=0.6m_{\min}=0.6. The upper bound mc=e−λcm_c=e^{-\lambda_c} arises directly from the exponential cutoff in ACF. Motivated by this distribution W(mr)W(m_r), we further defined an ensemble-averaged branching parameter,

⟨mr⟩=∫01mrW(mr) dmr,(11)\langle m_r \rangle= \int_0^1 m_r W(m_r)\,dm_r, \tag*{(11)}

which captures the effective contribution of heterogeneous branching dynamics to the observed long-range correlations. This quantity admits a closed-form expression,

⟨mr⟩=ηmcγ(η,ln⁡mcmmin⁡)(ln⁡mcmmin⁡)η,(12)\langle m_r \rangle= \eta m_c \frac{\gamma\left(\eta,\ln\frac{m_c}{m_{\min}}\right)}{\left(\ln\frac{m_c}{m_{\min}}\right)^\eta}, \tag*{(12)}

where γ(⋅)\gamma(\cdot) denotes the lower incomplete Gamma function. To characterize networks across different training epochs and hidden-state dimensionalities, the fitted parameters η\eta and τc\tau_c were substituted into Eq. (12) to obtain the corresponding value of ⟨mr⟩\langle m_r\rangle. In Figure 7, the resulting ensemble-averaged branching parameter exhibits trends that closely parallel those of the branching parameter mm inferred from the short-range ACFs across networks and training epochs, indicating a consistent characterization of LSTM dynamics across both short and long timescales.

Figure 7. Evolution of the ensemble-averaged branching parameter ⟨mr⟩\langle m_r\rangle as a function of training epoch for different LSTM hidden-state dimensions, inferred from the discrete-count activity signal XtX_t. The red dashed vertical lines and star marker are defined consistently with Figure 5. Across all model sizes, ⟨mr⟩\langle m_r\rangle rapidly increases during early training and subsequently stabilizes. Notably, networks with smaller hidden dimensions converge to larger asymptotic values of ⟨mr⟩\langle m_r\rangle, exhibiting trends consistent with those shown in Figure 5(b).

Within the MBP framework, the avalanche size distribution P(S)P(S) can be derived analytically under certain assumptions. Specifically, assuming that the external drive ItrI_t^r is moderate and its effect can be negligible due to the thresholding procedure, each review-specific branching process with a fixed mr<1m_r < 1 generates avalanches that follow a distribution of the form

P(S,mr)∼S−3/2exp⁡[−SSc(mr)]S→∞,(13)P(S,m_r) \sim S^{-3/2}\exp\left[-\frac{S}{S_c(m_r)}\right] \qquad S \to\infty, \tag*{(13)}

where ScS_c denotes the characteristic scale of the exponential portion of the distribution Christensen and Olami [ [22]. The scale-free behavior of P(S)P(S) reported in § 3 arises only after averaging over the heterogeneous distribution W(mr)W(m_r). As shown in Appendix § 10, the superposition of branching processes whose parameters are distributed according to Eq. (10) yields an avalanche size distribution of the form

P(S)∼S(−32+η)exp⁡[−S(1−mc)2C],(14)P(S) \sim S^{\left(-\frac{3}{2}+\eta\right)}\exp\left[-\frac{S(1-m_c)^2}{C}\right], \tag*{(14)}

where CC is a constant, and the power-law exponent τs=32+η\tau_s = \frac{3}{2}+\eta is directly controlled by the correlation exponent η\eta. During the optimal epochs of the smallest LSTM network, the upper-bound cutoff mcm_c in the weight distribution W(mr)W(m_r) approaches unity, leading to an extended power-law regime, consistent with the avalanche statistics observed empirically near these epochs. This explains why the measured avalanche exponent τs\tau_s is usually found to be larger than 32\frac{3}{2} in small networks. Moreover, this result reinforces the unifying role of the MBP framework: both the long-range temporal correlations and the avalanche size distributions originate from the same underlying heterogeneity in branching dynamics, providing a coherent explanation for critical-like behavior across multiple statistical observables.

Discussion

Our study demonstrates that artificial recurrent neural networks like the LSTM networks exhibit criticality-like dynamics when it is well trained, analogous to neuronal avalanches observed in biological neural systems. Through a combination of avalanche statistics, branching-process analysis, and spectral characterization, we show that multiple hallmarks of critical dynamics consistently emerge in smaller models near their optimal training epochs. The simultaneous appearance of these signatures across independent analytical frameworks provides converging evidence that LSTMs can transiently organize into a regime reminiscent of criticality during learning.

Building on this observation, a central insight of this work is that the observed critical-like behavior reflects a unified dynamical picture spanning both short-range and long-range temporal organization. At short timescales, the dynamics are governed by an effective branching process characterized by a branching parameter close to unity, which controls the propagation and termination of local activity cascades and gives rise to avalanche statistics. At longer timescales, heterogeneity in the effective branching parameters across different input conditions gives rise to a mixture of branching processes, whose superposition naturally generates long-range temporal correlations and 1/fβ1/f^{\beta} spectral behavior. In this sense, short-range avalanche dynamics and long-range memory effects are not independent phenomena but arise from two complementary branching mechanisms operating at different temporal scales within the same system.

Furthermore, our results indicate that such critical behavior is not an intrinsic consequence of the LSTM architecture itself. Instead, it emerges from an interplay between training dynamics and model capacity. Larger models, despite their greater expressive power, tend to operate further from this critical regime, exhibiting lower effective branching parameters and weaker long-range temporal correlations. This suggests that overparameterization may suppress the fine balance between amplification and dissipation required for critical-like dynamics, which could be related to the delayed optimization observed in larger networks (Figure 2), whereas smaller models are naturally driven toward this balance during training. In this sense, criticality appears as an emergent, capacity-dependent property rather than a built-in architectural feature. The training epoch plays a similarly crucial role. Critical signatures are most pronounced near the optimal epoch, where generalization performance is maximized, and diminish at early epochs when dynamics are strongly subcritical and dominated by short timescales. This temporal evolution highlights a close connection between learning, dynamical organization, and criticality. Rather than being static, the dynamical state of an LSTM evolves throughout training, transiently approaching criticality as representations become maximally informative and flexible.

Beyond LSTMs, our findings suggest that branching parameters may provide a useful, architecture-agnostic descriptor of sequential neural network dynamics, such as Transformers Vaswani et al. [2017], MAMBA Gu and Dao [2024] and DeltaNet Schlag et al. [2021]. Because the branching parameter captures how activity propagates over time, it offers a compact and interpretable measure of dynamical regime that is largely independent of specific architectural details. As such, it may serve as a unifying indicator for comparing different recurrent architectures, training protocols, or capacity regimes. More broadly, the emergence of critical-like dynamics in artificial networks reinforces the idea that criticality may be a general organizing principle for efficient information processing, arising naturally in systems that balance stability and adaptability through learning.

Data availability

The data that support the findings of this article are openly available [2026a]. The code used to reproduce the results is available in a GitHub repository [2026b].

Relationship between β\beta and η\eta

In this appendix, we derive the relationship between the spectral exponent β\beta and the correlation exponent η\eta for XtX_t whose ACF exhibits power-law decay with an exponential cutoff

C[τ]∼τ−ηe−τ/τc.(15)C[\tau] \sim\tau^{-\eta}e^{-\tau/\tau_c}. \tag*{(15)}

where 0<η<10 < \eta< 1 characterizes long-range temporal correlations and τc\tau_c denotes a finite correlation timescale. For a stationary process, the PSD is related to the ACF through the Wiener-Khinchin theorem. Assuming that XtX_t is stationary, its PSD is given by

S(ω)=∑τ=−∞∞R[τ]e−jωτ dτ≈∑τ=1∞τ−ηe−τ/τccos⁡(τω)≈τc1−η∫0∞x−ηe−xcos⁡(ωτcx) dx,(16)\begin{aligned} S(\omega) &= \sum_{\tau=-\infty}^{\infty} R[\tau]e^{-j\omega\tau}\,d\tau\\ &\approx\sum_{\tau=1}^{\infty}\tau^{-\eta}e^{-\tau/\tau_c}\cos(\tau\omega) \\ &\approx\tau_c^{1-\eta}\int_{0}^{\infty}x^{-\eta}e^{-x}\cos(\omega\tau_c x)\,dx, \tag*{(16)} \end{aligned}

where R[τ]=E[XtXt+τ]R[\tau] = \mathbb{E}[X_tX_{t+\tau}], which differs from the ACF C[τ]C[\tau] defined earlier, but does not affect the scaling behavior at finite frequencies. In the second line, we exploit the even symmetry of the autocorrelation function, and in the third line we approximate the discrete sum by a continuum integral, which is justified for sufficiently large τc\tau_c. The integral in Eq. (16) can be rewritten in complex form as

τc1−η∫0∞x−ηe−xcos⁡(ωτcx) dx=τc1−ηℜ[∫0∞x−ηe−(1−jωτc)x dx],(17)\tau_c^{1-\eta}\int_0^\infty x^{-\eta}e^{-x}\cos(\omega\tau_c x)\,dx = \tau_c^{1-\eta}\Re\left[\int_0^\infty x^{-\eta}e^{-(1-\mathrm{j}\omega\tau_c)x}\,dx\right], \tag*{(17)}

For 0<η<10<\eta<1, the integral converges and can be evaluated using the identity

∫0∞x−ηe−ax dx=Γ(1−η)aη−1,ℜ(a)>0.(18)\int_0^\infty x^{-\eta}e^{-ax}\,dx=\Gamma(1-\eta)a^{\eta-1},\qquad\Re(a)>0. \tag*{(18)}

Applying this result with a=1−jωτca=1-\mathrm{j}\omega\tau_c yields

S(ω)∼τc1−ηℜ[Γ(1−η)(1−jωτc)1−η].(19)S(\omega)\sim\tau_c^{1-\eta}\Re\left[\frac{\Gamma(1-\eta)}{(1-\mathrm{j}\omega\tau_c)^{1-\eta}}\right]. \tag*{(19)}

For small frequencies where ωτc≪1\omega\tau_c\ll1, the PSD tends to a constant:

S(ω)∼(τc)1−ηΓ(1−η)→constant.(20)S(\omega)\sim(\tau_c)^{1-\eta}\Gamma(1-\eta)\to\text{constant}. \tag*{(20)}

In the large-frequency regime ωτc≫1\omega\tau_c\gg1, we approximate

(1−jωτc)−(1−η)≈(ωτce−jπ2)−(1−η)=(ωτc)−(1−η)ei(1−η)π/2.(21)\begin{aligned} (1-\mathrm{j}\omega\tau_c)^{-(1-\eta)}&\approx(\omega\tau_c e^{-\mathrm{j}\frac{\pi}{2}})^{-(1-\eta)}\\ &=(\omega\tau_c)^{-(1-\eta)}e^{\mathrm{i}(1-\eta)\pi/2}. \tag*{(21)} \end{aligned}

Thus, the PSD becomes

S(ω)∼τc1−η(ωτc)η−1Γ(1−η)sin⁡(πη2)∼Γ(1−η)sin⁡(πη2)ωη−1.(22)\begin{aligned} S(\omega)&\sim\tau_c^{1-\eta}(\omega\tau_c)^{\eta-1}\Gamma(1-\eta)\sin\left(\frac{\pi\eta}{2}\right)\\ &\sim\Gamma(1-\eta)\sin\left(\frac{\pi\eta}{2}\right)\omega^{\eta-1}. \tag*{(22)} \end{aligned}

Defining the spectral exponent β\beta via S(ω)∼ω−βS(\omega)\sim\omega^{-\beta}, we obtain the relation

β+η=1.(23)\beta+\eta=1. \tag*{(23)}

Derivations of long-range dynamics

In this section, we derive the weight distribution W(mr)W(m_r) for the MBP whose ACF follows a power-law decay with an exponential cutoff. We then obtain the analytical expression for the ensemble-averaged branching parameter ⟨mr⟩\langle m_r\rangle from the derived distribution W(mr)W(m_r), and illustrate why it is necessary to introduce a lower-bound truncation mminm_{min} for W(mr)W(m_r). Finally, we quantify the error introduced by this truncation.

Derivation of W(mr)W(m_r)

Since the MBP is mathematically described by Eq. (9), the ACF C[τ]C[\tau], averaged over all XtX_t corresponding to different test reviews, can be written as

C[τ]=∫01W(mr)mrτ dmr.(24)C[\tau]=\int_0^1 W(m_r)m_r^\tau\,dm_r. \tag*{(24)}

We introduce the variable transformation mr=e−λm_r=e^{-\lambda}, which maps the interval mr∈(0,1)m_r\in(0,1) to λ∈(0,∞)\lambda\in(0,\infty). Under this change of variables, Eq. (25) becomes

C[τ]=∫0∞W(e−λ)e−τλe−λ dλ=∫0∞W~(λ)e−τλ dλ,(25)\begin{aligned} C[\tau]&=\int_0^\infty W(e^{-\lambda})e^{-\tau\lambda}e^{-\lambda}\,d\lambda\\ &=\int_0^\infty\widetilde{W}(\lambda)e^{-\tau\lambda}\,d\lambda, \tag*{(25)} \end{aligned}

where we have defined W~(λ)=W(e−λ)e−λ\widetilde{W}(\lambda)=W(e^{-\lambda})e^{-\lambda}. The above expression shows that C[τ]C[\tau] is the Laplace transform of W~(λ)\widetilde{W}(\lambda). A standard Laplace transform pair for ℜ(η)>0\Re(\eta)>0 is given by

∫0∞λη−1Γ(η)e−τλ dλ=τ−η.(26)\int_0^\infty\frac{\lambda^{\eta-1}}{\Gamma(\eta)}e^{-\tau\lambda}\,d\lambda=\tau^{-\eta}. \tag*{(26)}

Furthermore, a shift in λ\lambda–space by λc=1τc\lambda_c=\frac{1}{\tau_c} introduces an exponential factor in the transform,

∫λc∞(λ−λc)η−1Γ(η)e−τλ dλ=e−τλcτ−η.(27)\int_{\lambda_c}^{\infty}\frac{(\lambda-\lambda_c)^{\eta-1}}{\Gamma(\eta)}e^{-\tau\lambda}\,d\lambda=e^{-\tau\lambda_c}\tau^{-\eta}. \tag*{(27)}

Comparing this expression with the assumed form of the ACF C[τ]∼τ−ηe−τλcC[\tau]\sim\tau^{-\eta}e^{-\tau\lambda_c}, we obtain

W~(λ)∝(λ−λc)η−1Γ(η)H(λ−λc),(28)\widetilde{W}(\lambda)\propto\frac{(\lambda-\lambda_c)^{\eta-1}}{\Gamma(\eta)}H(\lambda-\lambda_c), \tag*{(28)}

where H(⋅)H(\cdot) denotes the Heaviside step function. Transforming back to the original variable mrm_r, the weight distribution is given by

W(mr)=1Z(−ln⁡mr−λc)η−1Γ(η)mrH(−ln⁡mr−λc).(29)W(m_r)=\frac{1}{Z}\frac{(-\ln m_r-\lambda_c)^{\eta-1}}{\Gamma(\eta)m_r}H(-\ln m_r-\lambda_c). \tag*{(29)}

where ZZ ensures proper normalization of W(mr)W(m_r) over the domain of mr∈[0,1]m_r\in[0,1]. From the analytical form, the lower bound in mrm_r-space follows directly as

mc=e−1τc.(30)m_c=e^{-\frac{1}{\tau_c}}. \tag*{(30)}

Consequently, W(mr)W(m_r) is non-zero only for 0<mr<mc0<m_r<m_c.

The ensemble-averaged branching parameter

Given the weight distribution W(mr)W(m_r), a natural quantity to consider is an ensemble-averaged branching parameter that captures the heterogeneous branching dynamics across reviews,

⟨mr⟩=∫01mrW(mr) dmr.(31)\langle m_r\rangle=\int_0^1m_rW(m_r)\,dm_r. \tag*{(31)}

However, the normalization integral of ZZ diverges. This divergence follows from the asymptotic behavior of the weight distribution,

W(mr)∼ln⁡(1/mr)η−1Γ(η)mrmr→0+,(32)W(m_r)\sim\frac{\ln(1/m_r)^{\eta-1}}{\Gamma(\eta)m_r}\qquad m_r\to0^+, \tag*{(32)}

which produces an unphysical divergence as mrm_r approaches zero. This pathology reflects the excessive weight assigned to extremely small branching parameters, which do not correspond to meaningful dynamical realizations. Importantly, the ACF C[τ]C[\tau] retains its heavy-tailed form even if a lower-bound cutoff mminm_{min} is introduced and only the remaining portion of W(mr)W(m_r) is retained. Motivated by this observation, we define a truncated weight distribution,

W(mr)=1Z(−ln⁡mr−λc)η−1Γ(η)mr1[mmin,mc](mr).(33)W(m_r) = \frac{1}{Z}\frac{(-\ln m_r-\lambda_c)^{\eta-1}}{\Gamma(\eta)m_r}\mathbf{1}_{[m_{min},m_c]}(m_r). \tag*{(33)}

This truncation removes the spurious divergence at mr→0m_r \to0 while preserving the long-range temporal correlations encoded in the ACF. Under this truncation, the normalization constant ZZ becomes

Z=∫λcλmax(λ−λc)η−1Γ(η) dλ=(λmax−λc)ηηΓ(η)=dηηΓ(η),(34)\begin{aligned} Z &= \int_{\lambda_c}^{\lambda_{max}} \frac{(\lambda-\lambda_c)^{\eta-1}}{\Gamma(\eta)}\,d\lambda\\ &= \frac{(\lambda_{max}-\lambda_c)^\eta}{\eta\Gamma(\eta)} \\ &= \frac{d^\eta}{\eta\Gamma(\eta)}, \tag*{(34)} \end{aligned}

where λmax=−ln⁡mmin\lambda_{max}=-\ln m_{min}, and we have defined d=λmax−λcd=\lambda_{max}-\lambda_c. Substituting the truncated distribution into Eq. (31), the ensemble-averaged branching parameter ⟨mr⟩\langle m_r\rangle is given by

∫01mrW(mr) dmr=1Γ(η)Z∫λcλmax(λ−λc)η−1e−λ dλ=e−λcΓ(η)Z∫0duη−1e−u du=e−λcΓ(η)Zγ(η,d),(35)\begin{aligned} \int_0^1 m_rW(m_r)\,dm_r &= \frac{1}{\Gamma(\eta)Z}\int_{\lambda_c}^{\lambda_{max}}(\lambda-\lambda_c)^{\eta-1}e^{-\lambda}\,d\lambda\\ &= \frac{e^{-\lambda_c}}{\Gamma(\eta)Z}\int_0^d u^{\eta-1}e^{-u}\,du \\ &= \frac{e^{-\lambda_c}}{\Gamma(\eta)Z}\gamma(\eta,d), \tag*{(35)} \end{aligned}

where γ(η,d)\gamma(\eta,d) denotes a lower incomplete Gamma function. After straightforward simplification, the ensemble-averaged branching parameter is obtained as

⟨mr⟩=ηe−λcdηγ(η,d)=ηmcγ(η,ln⁡mcmmin)(ln⁡mcmmin)η.(36)\begin{aligned} \langle m_r\rangle= \frac{\eta e^{-\lambda_c}}{d^\eta}\gamma(\eta,d) \\ &= \eta m_c\frac{\gamma\left(\eta,\ln\frac{m_c}{m_{min}}\right)}{\left(\ln\frac{m_c}{m_{min}}\right)^\eta}. \tag*{(36)} \end{aligned}

A visualization of ⟨mr⟩\langle m_r\rangle as a function of η\eta and mcm_c for mmin=0.6m_{min}=0.6 is shown in Figure A1(a). Within a reasonable range of mminm_{min} the qualitative dependence of ⟨mr⟩\langle m_r\rangle on these parameters remains unchanged.

Figure A1. (a) Ensemble-averaged branching parameter ⟨mr⟩\langle m_r\rangle as a function of the correlation exponent η\eta and the upper bound mcm_c for a fixed lower bound mmin⁡=0.6m_{\min}=0.6. (b) Order of magnitude of the relative truncation error Q(η,τd)Q(\eta,\tau d), shown as a function of η\eta and mmin⁡m_{\min} for mmin⁡=1m_{\min}=1 and τ=5\tau=5. The error is of order 10−k10^{-k} across the explored parameter range, demonstrating that the truncation-induced correction decays rapidly for large τd\tau d and does not affect the long-range temporal behavior of the ACF C[τ]C[\tau].

Error of truncation

The error introduced by the truncation can be quantified by comparing the ACF computed from the untruncated weight distribution W(mr)W(m_r) with that obtained from the truncated distribution. The ACF prior to truncation is given by Eq. (15). The truncated ACF is obtained by substituting Eq. (33) into Eq. (24), yielding

Ctrunc[τ]∝1Γ(η)∫λcλmax(λ−λc)η−1e−τλ dλ=e−τλcτ−ηΓ(η)∫0τduη−1e−u du=e−τλcτ−ηγ(η,τd)Γ(η)=e−τλcτ−ηP(η,τd),(37)\begin{aligned} C_{trunc}[\tau] &\propto\frac{1}{\Gamma(\eta)}\int_{\lambda_c}^{\lambda_{max}}(\lambda-\lambda_c)^{\eta-1}e^{-\tau\lambda}\,d\lambda\\ &= \frac{e^{-\tau\lambda_c}\tau^{-\eta}}{\Gamma(\eta)}\int_0^{\tau d}u^{\eta-1}e^{-u}\,du \\ &= e^{-\tau\lambda_c}\tau^{-\eta}\frac{\gamma(\eta,\tau d)}{\Gamma(\eta)} \\ &= e^{-\tau\lambda_c}\tau^{-\eta}P(\eta,\tau d), \tag*{(37)} \end{aligned}

where γ(η,τd)\gamma(\eta,\tau d) is the lower incomplete Gamma function, and P(η,τd)P(\eta,\tau d) denotes the regularized incomplete Gamma function. The relative error induced by the truncation is therefore given by

err=C[τ]−Ctrunc[τ]C[τ]=Q(η,τd),(38)err=\frac{C[\tau]-C_{trunc}[\tau]}{C[\tau]}=Q(\eta,\tau d), \tag*{(38)}

where Q(η,τd)=1−P(η,τd)Q(\eta,\tau d)=1-P(\eta,\tau d) denotes the regularized upper incomplete Gamma function. This expression shows that the truncation error decays rapidly for τd≫1\tau d\gg1, confirming that the long-range behavior of the ACF is preserved. This behavior is explicitly demonstrated in Figure A1(b), which shows that Q(η,τd)Q(\eta,\tau d) remains small for τ=5\tau=5.

Avalanche size distribution in MBP

In this appendix, we derive the asymptotic form of the avalanche size distribution P(S)P(S) induced by the weight distribution W(mr)W(m_r) in the MBP theory. For each single subcritical branching process with the review-specific branching parameter mrm_r, the avalanche size distribution is given by

P(S,mr)∼S−3/2exp⁡[−SSc(mr)]S→∞,P(S,m_r) \sim S^{-3/2}\exp\left[-\frac{S}{S_c(m_r)}\right]\qquad S\to\infty,
Sc(mr)∼1(1−mr)2mr→1,(39)S_c(m_r) \sim\frac{1}{(1-m_r)^2}\qquad m_r\to1, \tag*{(39)}

Given the weight distribution W(mr)W(m_r), the total avalanche size distribution of the MBP should be

P(S)=∫01W(mr)P(S,mr) dmr,(40)P(S)=\int_0^1 W(m_r)P(S,m_r)\,dm_r, \tag*{(40)}

where we can apply Eq. (33) for W(mr)W(m_r). We introduce the shifted variables

δ=1−mcϵ=mc−m.(41)\delta=1-m_c \qquad\epsilon=m_c-m. \tag*{(41)}

The avalanche-size distribution can be expressed with new variables

P(S)∼S−3/2∫01W(m)exp⁡[−S(1−m)2C] dm=S−3/2∫0ϵmax⁡W(mc−ϵ)exp⁡[−S(δ+ϵ)2C] dϵ,(42)\begin{aligned} P(S)\sim S^{-3/2}\int_0^1 W(m)\exp\left[-\frac{S(1-m)^2}{C}\right]\,dm \\ &= S^{-3/2}\int_0^{\epsilon_{\max}}W(m_c-\epsilon)\exp\left[-\frac{S(\delta+\epsilon)^2}{C}\right]\,d\epsilon, \tag*{(42)} \end{aligned}

where ϵmax⁡=mc−mmin⁡\epsilon_{\max}=m_c-m_{\min} denotes the upper bound of the integration since W(mr)W(m_r) is only valid in the interval m∈[mmin⁡,mc]m\in[m_{\min},m_c], and CC is a constant. Using λc=−ln⁡mc\lambda_c=-\ln m_c,

−ln⁡m−λc=ϵmc+O(ϵ2)ϵ→0+.(43)-\ln m-\lambda_c=\frac{\epsilon}{m_c}+O(\epsilon^2)\qquad\epsilon\to0^+. \tag*{(43)}

Plug the equation into the weight distribution yields

W(mc−ϵ)∼ϵη−1Γ(η)mcηϵ→0+.(44)W(m_c-\epsilon)\sim\frac{\epsilon^{\eta-1}}{\Gamma(\eta)m_c^\eta}\qquad\epsilon\to0^+. \tag*{(44)}

In the large-SS limit, the upper bound of the integral ϵmax⁡\epsilon_{\max} can be extended to infinity, yielding

P(S)∼S−3/2∫0∞ϵη−1exp⁡[−S(δ+ϵ)2C] dϵ.(45)P(S)\sim S^{-3/2}\int_0^\infty\epsilon^{\eta-1}\exp\left[-\frac{S(\delta+\epsilon)^2}{C}\right]\,d\epsilon. \tag*{(45)}

Factoring out the dominant exponential term and neglecting the quadratic term in the exponent gives

P(S)∼S−3/2exp⁡(−Sδ2C)∫0∞ϵη−1e−2Sδϵ/C dϵ=S−3/2exp⁡(−Sδ2C)Γ(η)(2SδC)−η.(46)\begin{aligned} P(S)\sim S^{-3/2}\exp\left(-\frac{S\delta^2}{C}\right)\int_0^\infty\epsilon^{\eta-1}e^{-2S\delta\epsilon/C}\,d\epsilon\\ &= S^{-3/2}\exp\left(-\frac{S\delta^2}{C}\right)\Gamma(\eta)\left(\frac{2S\delta}{C}\right)^{-\eta}. \tag*{(46)} \end{aligned}

Combining all factors, we obtain the asymptotic avalanche-size distribution

P(S)∼S−(32+η)exp⁡[−S(1−mc)2C],(47)P(S)\sim S^{-(\frac{3}{2}+\eta)}\exp\left[-\frac{S(1-m_c)^2}{C}\right], \tag*{(47)}

where mc=e−1/τcm_c=e^{-1/\tau_c}. This result shows that the MBP produces a power-law avalanche-size distribution with exponent 32+η\frac{3}{2}+\eta, followed by an exponential cutoff controlled by the distance 1−mc1-m_c from criticality.

Additional details

Excluded Network Realizations

A small fraction of trained networks was excluded from the avalanche and branching analysis due to pathological behaviors observed during inference. The first exclusion criterion was numerical instability, manifested as NaN values in the hidden states hth_t. The second criterion was the emergence of persistent fast oscillations in the hidden states hth_t, synchronized across all hidden-state dimensions, as illustrated in Figure A2. In this case, the hidden-state vector repeatedly alternates between two distinct points in the state space across successive timesteps, resulting in a pronounced periodic pattern. Such behavior is indicative of a highly ordered dynamical regime and is typically observed at early training epochs. While this oscillatory behavior may be related to bifurcation-like phenomena in recurrent dynamical systems Pascanu et al. [ [38], a detailed investigation of its origin is beyond the scope of the present work. As a consequence, the corresponding autocorrelation function C[τ]C[\tau] displays strong periodic structure rather than the exponential decay assumed in the branching-process framework, rendering these cases unsuitable for our analysis. Both types of pathological behavior occur more frequently in networks with larger hidden-state dimensionality. Nevertheless, for each hidden-state dimension, no more than four out of 28 trained networks were affected, and excluding these cases does not affect the statistical robustness or validity of our conclusions.

Figure A2. Time series of selected normalized hidden-state hth_t from a trained LSTM with hidden-state dimensionality 96, evaluated on a single test review. For clarity, only ten out of the 96 hidden-state dimensions are shown. The displayed components exhibit pronounced, synchronized fast oscillations.

Avalanche statistics in larger LSTM networks

In the main text, the avalanche size distribution P(S)P(S) for a hidden-state dimensionality of 128 is shown as a representative example, as illustrated in Figure 4(d). Here, we present corresponding results for additional larger networks to demonstrate that the observed subcritical behavior is not specific to that particular network size. As shown in Figure A3, LSTM networks with hidden-state dimensionalities ranging from 16 to 96 exhibit avalanche size distributions that deviate from power-law scaling at both early and optimal training epochs, indicating persistent subcritical dynamics. These results confirm that the subcritical avalanche behavior observed in large networks is robust across different hidden-state dimensionalities.

Figure A3. Avalanche size distributions P(S)P(S) for LSTM networks with hidden-state dimensionalities 16–96 at ϕ=1.5\phi= 1.5, shown for early (blue) and optimal (red) training epochs. Error bars indicate variability across random seeds.

Analysis Using Continuous-Valued Activity Signals

To assess the robustness of our results with respect to the definition of the activity signal, we repeated the temporal-correlation and branching analyses using a continuous-valued definition of the activity XtX_t, which removes the need for explicit thresholding and binarization of the hidden states. Figure A4 summarizes the results obtained from this continuous-valued activity signal. The ACFs, spectral exponents β\beta, and ensemble-averaged branching parameters ⟨mr⟩\langle m_r\rangle exhibit trends that closely mirror those observed for the discrete-count activity signals presented in the main text. In particular, small networks operate closer to criticality at the optimal epoch, while larger networks remain subcritical, and training drives the system toward longer temporal correlations. These results demonstrate that our conclusions are not sensitive to the specific choice of activity definition and are therefore robust.

Temporal correlations and branching statistics for continuous-valued activity signals

Figure A4. Temporal correlations and branching statistics for continuous-valued activity signals. (a,b) ACF C[τ]C[\tau] of the continuous-valued activity XtX_t at the early (a) and optimal (b) training epochs for different hidden-state dimensionalities. (c) Spectral exponent β\beta of the corresponding 1/fβ1/f^\beta noise at the optimal epoch, shown as a function of hidden-state dimensionality. (d) Evolution of the ensemble-averaged branching parameter ⟨mr⟩\langle m_r\rangle over training epochs for different hidden-state dimensionalities. Results obtained using discrete-count activity signals are shown in the main text and display qualitatively similar behavior.

References

References

  1. [1]John J Hopfield. Neural networks and physical systems with emergent collective computational abilities. Proceedings of the national academy of sciences, 79(8):2554–2558, 1982.
  2. [2]David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. Learning representations by back-propagating errors. nature, 323(6088):533–536, 1986.
  3. [3]Thierry Mora and William Bialek. Are biological systems poised at criticality? Journal of Statistical Physics, 144(2):268–302, 2011.DOI
  4. [4]Nils Bertschinger, Thomas Natschläger, and Robert Legenstein. At the edge of chaos: Real-time computations and self-organized criticality in recurrent neural networks. Advances in neural information processing systems, 17, 2004.
  5. [5]Joschka Boedecker, Oliver Obst, Joseph T Lizier, N Michael Mayer, and Minoru Asada. Information processing in echo state networks at the edge of chaos. Theory in Biosciences, 131(3):205–213, 2012.DOI
  6. [6]Ling Feng, Lin Zhang, and Choy Heng Lai. Optimal machine intelligence at the edge of chaos. arXiv preprint arXiv:1909.05176, 2019.arxiv.org/abs/1909.05176
  7. [7]Wenbo Wei, Nicholas Chong Jia Le, Choy Heng Lai, and Ling Feng. Multiple descents in deep learning as a sequence of order-chaos transitions. arXiv preprint arXiv:2505.20030, 2025.
  8. [8]John M Beggs and Dietmar Plenz. Neuronal avalanches in neocortical circuits. Journal of neuroscience, 23(35):11167–11177, 2003.pubmed.ncbi.nlm.nih.gov/14657176
  9. [9]Arsham Ghavasieh, Meritxell Vila-Minana, Akanksha Khurd, John Beggs, Gerardo Ortiz, and Santo Fortunato. Toward a physics of deep learning and brains. arXiv preprint arXiv:2509.22649, 2025.arxiv.org/abs/2509.22649
  10. [10]Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In European conference on computer vision, pages 630–645. Springer, 2016.arxiv.org/abs/1603.05027
  11. [11]Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735–1780, 1997.
  12. [12]Klaus Linkenkaer-Hansen, Simo Monto, Heikki Rytsälä, Kirsi Suominen, Erkki Isometsä, and Seppo Kähkönen. Breakdown of long-range temporal correlations in theta oscillations in patients with major depressive disorder. Journal of Neuroscience, 25(44):10131–10137, 2005.
  13. [13]Teresa Montez, Simon-Shlomo Poil, Bethany F Jones, Ilonka Manshanden, Jeroen PA Verbunt, Bob W van Dijk, Arjen B Brussaard, Arjen Van Oo yen, Cornelis J Stam, Philip Scheltens, et al. Altered temporal correlations in parietal alpha and prefrontal theta oscillations in early-stage alzheimer disease. Proceedings of the National Academy of Sciences, 106(5):1614–1619, 2009.DOI
  14. [14]Anupriya Pathania, Melynda Schreiber, Matthew W Miller, Matthew J Euler, and Keith R Lohse. Exploring the reliability and sensitivity of the eeg power spectrum as a biomarker. International Journal of Psychophysiology, 160:18–27, 2021.DOI
  15. [15]Paolo Allegrini, Danilo Menicucci, Remo Bedini, Leone Fronzoni, Angelo Gemignani, Paolo Gr igolini, Bruce J West, and Paolo Paradisi. Spontaneous brain activity as a source of ideal 1/f noise. Physical Review E—Statistical, Nonlinear, and Soft Matter Physics, 80(6):061914, 2009.pubmed.ncbi.nlm.nih.gov/20365197
  16. [16]Nicholas Jia Le Chong and Ling Feng. Self-organization toward 1/f noise in deep neural networks. Chaos: An Interdisciplinary Journal of Nonlinear Science, 34(8), 2024.DOI
  17. [17]Per Bak, Chao Tang, and Kurt Wiesenfeld. Self-organized criticality. Physical review A, 38(1):364, 1988.
  18. [18]TC Per Bak and Kurt Wiesenfeld. Self-organized criticality: and explanation of 1/f noise. Phys. Rev. Let, 59:381–384, 1987.pubmed.ncbi.nlm.nih.gov/10035754
  19. [19]Dietmar Plenz, Tiago L Ribeiro, Stephanie R Miller, Patrick A Kells, Ali Vakili, and Elliott L Capek. Self-organized criticality in the brain. Frontiers in Physics, 9:639389, 2021.arxiv.org/abs/2102.09124
  20. [20]Shan Yu, Andreas Klaus, Hongdian Yang, and Dietmar Plenz. Scale-invariant neuronal avalanche dynamics and the cut-off in size distributions. Plos one, 9(6):e99761, 2014.DOI
  21. [21]Thomas Petermann, Tara C Thiagarajan, Mikhail A Lebedev, Miguel AL Nicolelis, Dante R Chialvo, and Dietmar Plenz. Spontaneous cortical activity in awake monkeys composed of neuronal avalanches. Proceedings of the National Academy of Sciences, 106(37):15921–15926, 2009.DOI
  22. [22]Kim Christensen and Zeev Olami. Sandpile models with and without an underlying spatial structure. Physical Review E, 48(5):3361–3372, 1993.pubmed.ncbi.nlm.nih.gov/9960993
  23. [23]Stefano Zapperi, Kent Bæ kgaard Lauritsen, and H Eugene Stanley. Self-organized branching processes: mean-field theory for avalanches. Physical review letters, 75(22):4071, 1995.pubmed.ncbi.nlm.nih.gov/10059807
  24. [24]Theodore Edward Harris et al. The theory of branching processes, volume 6. Springer Berlin, 1963.
  25. [25]Andrew Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies, pages 142–150, 2011.
  26. [26]Viola Priesemann, Michael Wibral, Mario Valderrama, Robert Pröpper, Michel Le Van Quyen, Theo Geisel, Jochen Triesch, Danko Nikolić, and Matthias HJ Munk. Spike avalanches in vivo suggest a driven, slightly subcritical brain state. Frontiers in systems neuroscience, 8:108, 2014.
  27. [27]Jens Wilting and Viola Priesemann. Inferring collective dynamical states from widely unobserved systems. Nature communications, 9(1):2325, 2018.DOI
  28. [28]Henrik Jeldtoft Jensen, Kim Christensen, and Hans C Fogedby. 1/f noise, distribution of lifetimes, and a pile of sand. Physical Review B, 40(10):7425, 1989.pubmed.ncbi.nlm.nih.gov/9991162
  29. [29]Lasse Laurson, Mikko J Alava, and Stefano Zapperi. Power spectra of self-organized critical sandpiles. Journal of Statistical Mechanics: Theory and Experiment, 2005(11):L11001–L11001, 2005.DOI
  30. [30]C-K Peng, Sergey V Buldyrev, Shlomo Havlin, Michael Simons, H Eugene Stanley, and Ary L Goldberger. Mosaic organization of dna nucleotides. Physical review e, 49(2):1685, 1994.pubmed.ncbi.nlm.nih.gov/9961383
  31. [31]Richard Hardstone, Simon-Shlomo Poil, Giuseppina Schiavone, Rick Jansen, Vadim V Nikulin, Huibert D Mansvelder, and Klaus Linkenkaer-Hansen. Detrended fluctuation analysis: a scale-free view on neuronal oscillations. Frontiers in physiology, 3:23105, 2012.
  32. [38]Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks. In International conference on machine learning, pages 1310–1318. Pmlr, 2013.arxiv.org/abs/1211.5063

Paper details

vlnc:W_u6vgj4sb
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Physics and Astronomy

Not recorded for this paper

View statement
Looking for related papers…
Contents