WorksPhysics and Astronomy
Towards Critical Branching Mechanism in Recurrent Neural Networks
Abstract
Criticality has been proposed as a key organizing principle in biological neural systems, yet its origin and relevance in artificial neural networks remain unclear. We analyze hidden-state dynamics in trained long short-term memory (LSTM) networks and show that small networks near their optimal training epochs (steps) exhibit scale-free avalanche statistics and branching parameters close to unity, indicative of near-critical dynamics, while larger models remain subcritical. To explain the coexistence of subcritical branching with robust noise, we introduce a mixture branching process framework that links heterogeneous branching dynamics to long-range temporal correlations. These results identify critical-like behavior in LSTMs as an emergent, capacity-dependent dynamical regime.
Introduction
Scientists have long drawn inspiration from biological neural systems in the design of Artificial Neural Networks (ANNs), with an emphasis on mimicking their structural and functional characteristics. By investigating the dynamical behavior of biological brains, researchers seek to identify principles that can be transferred to ANN design Hopfield [ [1], Rumelhart et al. [ [2]. One key finding in this area is that both biological brains and ANNs tend to operate near the edge of chaos, a critical state that maximizes information-processing capacity Mora and Bialek [ [3], Bertschinger et al. [ [4], Boedecker et al. [ [5], Feng et al. [ [6], Wei et al. [ [7]. Motivated by the discovery of neuronal avalanches in biological systems Beggs and Plenz [ [8], scale-free avalanche-like activity propagating across layers has also been observed in deep feedforward neural networks such as ResNet Ghavasieh et al. [ [9], He et al. [ [10]. Furthermore, accumulating evidence suggests that both biological neural systems and Long Short-Term Memory (LSTM) networks Hochreiter and Schmidhuber [ [11] display noise in their activity, a statistical hallmark often associated with self-organized criticality (SOC) Linkenkaer-Hansen et al. [ [12], Montez et al. [ [13], Pathania et al. [ [14], Allegrini et al. [ [15], Chong and Feng [ [16], Bak et al. [ [17], Per Bak and Wiesenfeld [ [18].
In the context of neuroscience, the SOC hypothesis has been extensively investigated, particularly in in-vitro cortical networks, where neural activity organizes into scale-free cascades known as neuronal avalanches Beggs and Plenz [ [8], Plenz et al. [ [19], Yu et al. [ [20], Petermann et al. [ [21]. These avalanches are characterized by power-law distributions of event sizes and durations, indicating the absence of characteristic scales. From a theoretical perspective, such dynamics can be modeled using a mean-field sandpile model, which is mathematically equivalent to a critical branching process Christensen and Olami [ [22], Zapperi et al. [ [23]. Within this framework, the branching parameter serves as a control variable that quantifies the balance between activity propagation and decay, with criticality emerging when the system is poised between quiescence and runaway excitation Harris et al. [ [24].
While the SOC paradigm and avalanche dynamics in biological neural systems are well established, the relevance to ANNs remains far less understood. In particular, although recent studies have reported the presence of noise and long-range temporal correlations in LSTM networks Chong and Feng [ [16], a unifying theoretical framework explaining the emergence of such scale-free behavior in trained networks is still lacking. Unlike biological systems, ANNs are deterministic, parameterized models trained via gradient-based optimization, raising fundamental questions about how critical-like dynamics arise in the absence of explicit self-organizing mechanisms.
Here, we propose a framework to extend the concept of neuronal avalanches from neuroscience to the realm of ANNs, with a particular focus on LSTM networks. By defining avalanche-like events in LSTM hidden-state dynamics and analyzing their statistical properties, we establish a direct connection between LSTM avalanches and the branching process. Furthermore, we demonstrate that the emergence of noise in LSTMs is closely linked to the underlying branching structure of network dynamics, providing a possible mechanistic interpretation of scale-free temporal correlations in LSTMs.
The remainder of this paper is organized as follows. In § 2, we introduce the LSTM networks, dataset, and training protocols in our analysis. In § 3, we define avalanche events in LSTM dynamics and demonstrate the emergence of the scale-free avalanche size distribution at specific stages of training and for particular network configurations. In § 4, we adopt a branching-process framework to account for the observed avalanche behaviors and apply an established method to estimate the branching parameter, which serves as a quantitative indicator of the dynamical state of the network. However, the branching-process analysis in § 4 primarily captures short-range temporal correlations and does not account for the long-range noise observed in LSTM activity. To address this limitation, in § 5 we investigate the long-range temporal correlations in LSTMs and introduce an extended branching mechanism capable of reproducing the observed behavior. This framework yields an ensemble-averaged branching parameter whose evolution mirrors that of the conventional branching parameter across training epochs and network configurations. Throughout the paper, an epoch denotes a discrete step in the training process corresponding to one full pass over the dataset.
LSTM networks
The LSTM architecture employed in this work consists of three components: an embedding layer, a single LSTM layer, and a linear classifier implemented as a dense layer followed by a sigmoid function, as shown in Figure 1. The embedding layer maps each discrete input token onto a continuous representation in an eight-dimensional (vector) latent space. The single-layer LSTM can be mathematically described as Hochreiter and Schmidhuber [ [11]:
Figure 1. Schematic of the LSTM architecture used for binary sentiment classification. Block A corresponds to the LSTM layer, with the internal gating and state-update operations shown in the pink inset.
where denotes the eight-dimensional embedding vector corresponding to the input token at timestep , indicates element-wise multiplication, and are the sigmoid and hyperbolic tangent functions, respectively, and represent the trainable parameters of the LSTM layer. At each timestep , the embedded input is fed into the LSTM layer together with the previous hidden state and cell state , which store the memory of the network, resulting in an updated hidden state and cell state . After the entire review is processed by the LSTM layer, the final hidden state is passed to the classifier, which outputs a value in the interval for binary classification.
We trained multiple LSTM networks sharing the same architecture described above, while varying the dimensionality of the hidden state across the set . The networks were trained on a dataset of 50, 000 IMDb movie reviews Maas et al. [ [25], labeled as positive opinion (1) or negative opinion (0). To account for the inherent stochasticity in neural network training and to ensure the statistical robustness of the results presented in this work, each configuration was trained using 28 different random seeds; a small number of trained networks were excluded for reasons discussed in Appendix § 11. Across all realizations, most networks saturated before epoch 30 at a test accuracy of approximately 87%, whereas larger networks exhibited delayed optimization for a small subset of random seeds, as illustrated in Figure 2. All observables measured below were computed during inference using the network checkpoints at the specified epoch. For a given hidden-state dimensionality and training epoch, the same quantity was evaluated across all experiments trained with different random seeds, and the results were averaged, with error bars representing the standard deviation unless explicitly stated otherwise. Although reviews in the dataset varied in length, all were truncated or padded to a fixed length of 500 tokens. The dataset was then split into a training set of 35, 000 reviews and a test set of 15, 000 reviews. All networks were trained using an identical set of hyperparameters, summarized in Table 1.
Figure 2. Distribution of optimal epochs (i.e., the epoch with the lowest test loss during training) as a function of hidden-state dimensionality for LSTM networks. Each violin plot summarizes the distribution of optimal epochs across multiple network realizations with different random seeds but identical architectures. Black dots denote individual realizations. The dashed horizontal line marks epoch 30, which corresponds to the epoch at which the majority of networks achieve their best performance across hidden-state dimensions.
| Batch size | Embedding size | Dropout |
| 128 | 8 | 0.1 |
| L2 regularization | Learning rate | Gradient clipping |
| 0.0005 | 0.005 | 5 |
Table 1. Selected hyperparameters used during LSTM training.
Avalanches
In neuroscience, neuronal avalanches are defined as cascades of spatiotemporal bursts of neural activity separated by intervals of quiescence. Experimentally, such avalanches are typically identified using multielectrode arrays placed onto cortical slices, which record local field potentials (LFPs) from multiple electrodes Beggs and Plenz [ [8], Petermann et al. [ [21], Priesemann et al. [ [26]. Neuronal populations in the vicinity of a given electrode are considered active within a discrete time bin if a negative spike exceeding a predefined threshold is detected in the corresponding LFP signal during that interval. An avalanche is operationally defined as a sequence of consecutively active time bins, where a bin is considered active if it has at least one active electrode, preceded and followed by inactive bins. The size and duration distribution of the avalanches in cortical slices are observed to be scale-free, a hallmark of critical dynamics in biological neural systems.
Motivated by this experimental framework, we modeled each dimension of the hidden states in LSTM as an artificial neuron and applied an analogous avalanche analysis during network inference, as illustrated in Figure 3(a). Specifically, reviews from the test set of the IMDb dataset were processed using the trained LSTM networks. For each review, a sequence of 500 tokens was sequentially fed into the LSTM layer, yielding at every timestep. To render the activities of different artificial neurons directly comparable, each dimension of the resulting was independently standardized across time using -score normalization described below.

Figure 3. Extraction of neural activities from an LSTM network. (a) Schematic of the analysis pipeline. LSTM processes test reviews to generate hidden states , which are recorded and normalized for subsequent analysis. (b) Time series of normalized hidden states for selected dimensions . A uniform activity threshold (shaded regions) is applied independently to each normalized hidden-state dimension to identify significant activation events. For illustration, the hidden-state dimensionality is set to eight; in the experiments, networks with varying hidden dimensions were analyzed. (c) Binary raster plot of detected neural activities across dimensions after thresholding (top), together with the corresponding , defined as the number of active neurons at each timestep (bottom). Consecutive timesteps for which (red regions) are identified as avalanches.
For a given hidden-state dimension , the temporal mean and standard deviation of the -th hidden dimension were computed over the interval , where . The initial segments were discarded as a burn-in period, during which the hidden-state dynamics were strongly influenced by the initialization of the hidden states , and cell states . The normalized neural activities were defined as the absolute deviation from the temporal mean measured in units of the corresponding standard deviation,
where indexes the hidden-state dimension and denotes the length of the time window used for normalization. This normalization step ensures that differences in activity scale across artificial neurons do not bias the subsequent avalanche detection. As shown in Figure 3(b), after normalization, all dimensions of the normalized neural activities were binarized using a uniform threshold , which was manually selected and applied consistently across dimensions and time. An artificial neuron was considered active at a given timestep if the amplitude of its normalized exceeded this threshold, as demonstrated in Figure 3(c). This binarization defines a binary state matrix with elements
A timestep was classified as active if at least one neuron was active at that moment. An avalanche was then identified as a sequence of consecutive active timesteps bounded by silent periods on both sides. The use of -score normalization and threshold-based binarization follows standard practice in studies of neuronal avalanches Plenz et al. [ [19], Yu et al. [ [20], from which our methodology is inspired. This entire pipeline was applied to all 15,000 test reviews, after which we analyzed the avalanche statistics over the entire set.
We analyzed the distribution of avalanche size , where the size is quantified as the total number of active neurons accumulated over the duration of an avalanche, across networks with different sizes. For the network with the smallest hidden-state dimensionality at its optimal epoch, the avalanche size followed a power-law distribution with an exponential cutoff. The presence of this cutoff reflects finite-size effects, which are expected in any system with finite degrees of freedom, including biological neural systems Beggs and Plenz [ [8]. As shown in Figure 4(a)-(Figure 4c), within a reasonable range of threshold values , the power-law behavior remained qualitatively stable. However, the estimated power-law exponent varied substantially under different thresholding conditions, indicating limited robustness of the avalanche statistics, similar to what has been reported for neuronal avalanches in biological neural systems. In the early epochs, the smallest network typically exhibited a pronounced exponential decay in the avalanche size distribution, suggesting subcritical dynamics Christensen and Olami [ [22]. In Figure 4(d), for larger networks, the avalanche size generally deviates from the power-law distribution at both early and optimal training epochs. For clarity, only the case with hidden-state dimension 128 is shown in the figure; corresponding results for other large hidden-state dimensions are provided in Appendix § 11. This behavior indicates that large networks remain subcritical during training, with no transition toward criticality observed.
Figure 4. Log–log plots of avalanche size distributions obtained from thresholded hidden-state activities across different operating regimes. Red markers correspond to the optimal training epoch, while blue markers indicate an early training stage. Here, epoch 30 is selected as a representative optimal epoch across realizations, despite variation in the exact optimal epoch for different networks (Figure 2). This definition is used consistently in all subsequent figures unless noted otherwise. Error bars represent statistical variability across networks trained with different random seeds but identical architectures and training epochs. Dashed lines denote power-law guides, fitted to the avalanche size distributions at the optimal epoch, with the corresponding exponent shown for reference. Panels (a–c) show results for a network with hidden-state dimensionality 8 at increasing activity threshold : (a) , (b) , and (c) . Panel (d) shows the avalanche size distribution for a larger network (hidden-state dimension 128) at . Results for additional large hidden-state dimensions, which exhibit similar behavior, are provided in Appendix § 11. The observed power-law scaling for small networks remains qualitatively consistent across threshold values , supporting the presence of near-critical dynamics in small LSTM networks.
Branching process
To account for scale-free avalanche statistics, biological neural systems are often modeled as a mean-field sandpile, which can be mathematically described as a branching process governed by an order parameter known as the branching parameter Beggs and Plenz [ [8], Harris et al. [ [24]. The dynamics were described as
where denotes the number of toppled sites (or active electrodes) at discrete time , and represents the number of offspring events generated by the -th toppled site at that time. The branching parameter quantifies the expected number of descendants per toppled site. As approaches one, the sandpile self-organizes into a critical state, where a perturbation can trigger a cascade of events that neither dies out nor explodes. In this critical regime, the avalanche size distribution of this sandpile follows a power law with a critical exponent of , independent of the threshold value. Notably, the robustness of the exponent in the mean-field theory is considerably stronger than that observed in cortical slice experiments Beggs and Plenz [ [8], a discrepancy that is commonly attributed to the idealized connectivity structure of the mean-field model. In contrast, when , cascades eventually dissipate, and the avalanche size distribution will deviate from the power-law behavior. Consequently, measuring the branching parameter is crucial for understanding the mechanism underlying the avalanche dynamics in LSTM.
Unlike the mean-field sandpile model or in-vitro brain systems, which are externally driven only after relaxation, artificial neurons in LSTM are continuously driven by external input tokens. To capture the branching dynamics underlying the LSTM avalanche, we hypothesized a driven variation of the branching process given by the following equation:
where denotes the number of active artificial neurons at timestep , represents the number of descendants generated by the -th neuron at that timestep, and denotes the external drive at timestep , assumed to have finite mean and variance.
We then employed the multi-regressive (MR) estimator Wilting and Priesemann [ [27] to measure the branching parameter , as shown in Figure 5(a). Because both the cell state and hidden state were updated at every timestep, the full dynamical system evolved in a state space larger than spanned by alone. However, since only was used in our analysis, this procedure effectively constituted spatial subsampling of the underlying dynamics. The MR estimator was explicitly developed to address such subsampling scenarios and is known to yield unbiased estimates of the branching parameter under these conditions. In addition, thresholding in the avalanche detection procedure suppressed low-amplitude activities, which can likewise be interpreted as an additional form of spatial subsampling. For each input review processed by the LSTM, the hidden state yields an individual activity time series , where indexes the review. The autocorrelation function (ACF) was computed separately for each activity time series and then averaged across all reviews. The MR estimation was subsequently performed by fitting this review-averaged ACF,

Figure 5. Estimation of the branching parameter from temporal correlations in LSTM activities. (a) Schematic of the analysis pipeline. For each trained network, test reviews are processed to obtain the activity signal . The ACF is computed for each signal and averaged across reviews. The averaged ACF is then fitted with an exponential decay to extract an effective branching parameter. This procedure is repeated across networks with different hidden-state dimensions, across independent training runs governed by different random seeds, and across training epochs. (b) Evolution of the estimated branching parameter over training epochs for different hidden-state dimensions, obtained from discrete-count activity signals (left) and continuous-valued definitions of that eliminate explicit thresholding and binarization of the hidden states (right). Solid lines indicate the mean across networks sharing the same hidden-state dimensionality but trained under different random seeds, while shaded regions denote variability arising from stochasticity in the training process. The red dashed vertical lines and star markers highlight the epoch interval during which the majority of networks achieve their optimal performance. Larger hidden dimensions systematically yield lower effective branching parameters, while training drives small networks toward a near-critical regime.
where denotes the total number of reviews. Hereafter, the term ACF refers to this review-averaged autocorrelation function. The averaged ACF was fitted with an exponential form over relatively short time lags up to , owing to the limited sequence length of , where is a constant. In the left panel of Figure 5(b), after optimization, approached values close to 0.9 as the hidden-state dimensionality decreased, indicating increasingly critical dynamics in smaller networks. For small networks, increased monotonically over the course of training and attained its maximum in the vicinity of the optimal epoch. Furthermore, note that the MR estimation method does not impose any requirement for to be represented as integer counts or real-valued quantities. This flexibility allowed to be redefined in terms of the total intensities of the active neurons at each timestep . Specifically, we defined
which eliminates the need for an explicit thresholding procedure. Importantly, the conclusions drawn from this alternative continuous-valued definition of were consistent with those derived from the integer-valued formulation, as evidenced by the agreement between the left and right panels of Figure 5(b), reinforcing the robustness of our findings.
These observations are qualitatively in agreement with avalanche statistics discussed in the previous section and support the hypothesized driven branching process in Eq. (5) as a plausible mechanism underlying LSTM avalanche dynamics.
Long-range dynamics
While the preceding sections established a framework for avalanche analysis in LSTMs, the connection between the inferred branching mechanism and the observed noise remains unsolved. This gap is particularly striking in light of the mean-field sandpile theory, where noise arises from the scale-free duration of avalanches at criticality, while subcritical sandpiles () exhibit a finite characteristic timescale that leads to a Lorentzian power spectral density (PSD) Per Bak and Wiesenfeld [ [18], Jensen et al. [ [28], Laurson et al. [ [29]. In contrast to these theoretical expectations, prior work Chong and Feng [ [16] reported robust noise even in larger LSTM networks despite their relatively low branching parameter . This apparent discrepancy suggests that the driven branching model, inferred from the short-range ACF of , is insufficient to account for the long-range noise observed in LSTMs.
It is important to note, however, that the activity signal analyzed in Ref. Chong and Feng [ [16] is not identical to the definition of adopted here. Specifically, that study considered the summed hidden-state activity , whereas in this work was defined either as a discrete count of active neurons or as a continuous-valued aggregation of normalized amplitudes. To ensure a consistent and robust characterization of long-range temporal correlations under this generalized definition, we estimated the spectral exponent of noise in using Detrended Fluctuation Analysis (DFA) Peng et al. [ [30], Hardstone et al. [ [31]. In DFA, for each input review, is partitioned into windows of length , locally detrended within each window, and the fluctuation function is defined as the root-mean-square (RMS) magnitude of the detrended residuals, averaged over all windows at scale . The final fluctuation function is then obtained by averaging across all reviews. For scale-invariant processes, follows a power-law dependence , from which the spectral exponent is obtained via Buldryev et al. [1995]. As shown in Figure 6(b), for LSTMs operating near their optimal epoch, the measured exhibits power-law scaling over approximately two decades in window size . Correspondingly, as shown in Figure 6(c), the inferred spectral exponent increases toward values near 0.8 as the hidden-state dimension decreases, in qualitative agreement with Ref. Chong and Feng [ [16]. By contrast, at early training epochs, is not well described by a single power law and instead displays a crossover with different effective scaling exponents at small and large window sizes, as illustrated in Figure 6(a).

Figure 6. Training-dependent scaling behavior and temporal correlations in LSTM activities. Panels (a) and (b) show DFA fluctuation functions computed from the discrete-count activity signals as a function of scale for different hidden-state dimensions, evaluated at an early training epoch (a) and at the optimal epoch (b). Panel (c) presents the spectral exponent of the corresponding noise at the optimal epoch, obtained by fitting in panel (b), shown as a function of the LSTM hidden-state dimension; error bars are defined as in Figure 4. Panels (d) and (e) display the corresponding ACF as a function of time lag at the early (d) and optimal (e) epochs, respectively; shaded regions are defined as in Figure 5. At the optimal epoch, the activity exhibits well-defined scale-invariant behavior over multiple window sizes together with an extended correlation length. In contrast, at the early training epoch, deviates from a single power-law form and the ACF decays rapidly, indicating short-range temporal correlations. Corresponding results for the continuous-valued are provided in Appendix § 11.
To elucidate the origin of the observed noise in , it is thus necessary to probe the ACF over substantially longer timescales. In Figure 6(e), when evaluated over two decades in time lag , computed from at the optimal epoch of all LSTMs exhibits a heavy-tailed behavior that can be well captured by a power law with an exponential cutoff
where is the power-law exponent, and denotes the characteristic cutoff timescale. To mitigate the statistical fluctuations across different realizations, especially at large , we fitted this form to the empirical , obtained by averaging ACFs across networks trained with different random seeds at the same training epoch and hidden-state dimensionality before fitting the resulting averaged correlation function. This contrasts with the short-range MR analysis, where the branching parameter is fitted separately for each seed and then averaged. The fitted exponent increases systematically with the hidden-state dimensionality, mirroring our earlier observations regarding the dimensional dependence of at the optimal epoch. This correspondence is expected on general grounds, since the scaling form in Eq. (8) directly constrains the high-frequency behavior of the PSD. Specifically, a correlation function of the form yields a PSD that is flat for and scales as for , with crossover frequency (see Appendix § 8 for a derivation). Because is typically large in our setting, the experimentally accessible spectrum is dominated by the high-frequency regime, implying , in agreement with the empirically observed anti-correlation between and . In contrast, at the early epochs displays a pure exponential decay, vanishing over very short time lags. Such an exponential ACF implies a Lorentzian power spectrum that is flat at low frequencies and crosses over to at high frequencies, consistent with the two-regime scaling behavior of observed during early epochs.
Motivated by the fact that a single branching process cannot generate a genuinely long-ranged temporal correlation, and by the limitation of our preceding analysis in assuming a homogeneous mechanism across different input reviews, we hypothesized a more general mechanism, termed the mixture branching process (MBP), to account for the observed behavior. In the MBP, each review induces its own branching dynamics so that the activity associated with review evolves with a review-specific branching parameter ,
where indexes the review. The observed dynamics then arise from the superposition of branching processes with heterogeneous , which can effectively reproduce the heavy-tailed decay of . Within this framework, the review-specific parameters are sampled from a distribution , which can be derived analytically from the scaling form . The parameters and are obtained by the fitting procedure described above. As proven in Appendix B, the analytical form of is
where denotes the Gamma function, , and is a boxcar function that equals unity for and vanishes otherwise. The introduction of a lower bound is required to prevent an unphysical divergence associated with extremely small . Importantly, the specific choice of affects only short-timescale behavior and does not alter the heavy-tailed regime of , which is the primary focus of our long-range analysis. In the calculations that follow, we therefore fix . The upper bound arises directly from the exponential cutoff in ACF. Motivated by this distribution , we further defined an ensemble-averaged branching parameter,
which captures the effective contribution of heterogeneous branching dynamics to the observed long-range correlations. This quantity admits a closed-form expression,
where denotes the lower incomplete Gamma function. To characterize networks across different training epochs and hidden-state dimensionalities, the fitted parameters and were substituted into Eq. (12) to obtain the corresponding value of . In Figure 7, the resulting ensemble-averaged branching parameter exhibits trends that closely parallel those of the branching parameter inferred from the short-range ACFs across networks and training epochs, indicating a consistent characterization of LSTM dynamics across both short and long timescales.
Figure 7. Evolution of the ensemble-averaged branching parameter as a function of training epoch for different LSTM hidden-state dimensions, inferred from the discrete-count activity signal . The red dashed vertical lines and star marker are defined consistently with Figure 5. Across all model sizes, rapidly increases during early training and subsequently stabilizes. Notably, networks with smaller hidden dimensions converge to larger asymptotic values of , exhibiting trends consistent with those shown in Figure 5(b).
Within the MBP framework, the avalanche size distribution can be derived analytically under certain assumptions. Specifically, assuming that the external drive is moderate and its effect can be negligible due to the thresholding procedure, each review-specific branching process with a fixed generates avalanches that follow a distribution of the form
where denotes the characteristic scale of the exponential portion of the distribution Christensen and Olami [ [22]. The scale-free behavior of reported in § 3 arises only after averaging over the heterogeneous distribution . As shown in Appendix § 10, the superposition of branching processes whose parameters are distributed according to Eq. (10) yields an avalanche size distribution of the form
where is a constant, and the power-law exponent is directly controlled by the correlation exponent . During the optimal epochs of the smallest LSTM network, the upper-bound cutoff in the weight distribution approaches unity, leading to an extended power-law regime, consistent with the avalanche statistics observed empirically near these epochs. This explains why the measured avalanche exponent is usually found to be larger than in small networks. Moreover, this result reinforces the unifying role of the MBP framework: both the long-range temporal correlations and the avalanche size distributions originate from the same underlying heterogeneity in branching dynamics, providing a coherent explanation for critical-like behavior across multiple statistical observables.
Discussion
Our study demonstrates that artificial recurrent neural networks like the LSTM networks exhibit criticality-like dynamics when it is well trained, analogous to neuronal avalanches observed in biological neural systems. Through a combination of avalanche statistics, branching-process analysis, and spectral characterization, we show that multiple hallmarks of critical dynamics consistently emerge in smaller models near their optimal training epochs. The simultaneous appearance of these signatures across independent analytical frameworks provides converging evidence that LSTMs can transiently organize into a regime reminiscent of criticality during learning.
Building on this observation, a central insight of this work is that the observed critical-like behavior reflects a unified dynamical picture spanning both short-range and long-range temporal organization. At short timescales, the dynamics are governed by an effective branching process characterized by a branching parameter close to unity, which controls the propagation and termination of local activity cascades and gives rise to avalanche statistics. At longer timescales, heterogeneity in the effective branching parameters across different input conditions gives rise to a mixture of branching processes, whose superposition naturally generates long-range temporal correlations and spectral behavior. In this sense, short-range avalanche dynamics and long-range memory effects are not independent phenomena but arise from two complementary branching mechanisms operating at different temporal scales within the same system.
Furthermore, our results indicate that such critical behavior is not an intrinsic consequence of the LSTM architecture itself. Instead, it emerges from an interplay between training dynamics and model capacity. Larger models, despite their greater expressive power, tend to operate further from this critical regime, exhibiting lower effective branching parameters and weaker long-range temporal correlations. This suggests that overparameterization may suppress the fine balance between amplification and dissipation required for critical-like dynamics, which could be related to the delayed optimization observed in larger networks (Figure 2), whereas smaller models are naturally driven toward this balance during training. In this sense, criticality appears as an emergent, capacity-dependent property rather than a built-in architectural feature. The training epoch plays a similarly crucial role. Critical signatures are most pronounced near the optimal epoch, where generalization performance is maximized, and diminish at early epochs when dynamics are strongly subcritical and dominated by short timescales. This temporal evolution highlights a close connection between learning, dynamical organization, and criticality. Rather than being static, the dynamical state of an LSTM evolves throughout training, transiently approaching criticality as representations become maximally informative and flexible.
Beyond LSTMs, our findings suggest that branching parameters may provide a useful, architecture-agnostic descriptor of sequential neural network dynamics, such as Transformers Vaswani et al. [2017], MAMBA Gu and Dao [2024] and DeltaNet Schlag et al. [2021]. Because the branching parameter captures how activity propagates over time, it offers a compact and interpretable measure of dynamical regime that is largely independent of specific architectural details. As such, it may serve as a unifying indicator for comparing different recurrent architectures, training protocols, or capacity regimes. More broadly, the emergence of critical-like dynamics in artificial networks reinforces the idea that criticality may be a general organizing principle for efficient information processing, arising naturally in systems that balance stability and adaptability through learning.
Data availability
The data that support the findings of this article are openly available [2026a]. The code used to reproduce the results is available in a GitHub repository [2026b].
Relationship between and
In this appendix, we derive the relationship between the spectral exponent and the correlation exponent for whose ACF exhibits power-law decay with an exponential cutoff
where characterizes long-range temporal correlations and denotes a finite correlation timescale. For a stationary process, the PSD is related to the ACF through the Wiener-Khinchin theorem. Assuming that is stationary, its PSD is given by
where , which differs from the ACF defined earlier, but does not affect the scaling behavior at finite frequencies. In the second line, we exploit the even symmetry of the autocorrelation function, and in the third line we approximate the discrete sum by a continuum integral, which is justified for sufficiently large . The integral in Eq. (16) can be rewritten in complex form as
For , the integral converges and can be evaluated using the identity
Applying this result with yields
For small frequencies where , the PSD tends to a constant:
In the large-frequency regime , we approximate
Thus, the PSD becomes
Defining the spectral exponent via , we obtain the relation
Derivations of long-range dynamics
In this section, we derive the weight distribution for the MBP whose ACF follows a power-law decay with an exponential cutoff. We then obtain the analytical expression for the ensemble-averaged branching parameter from the derived distribution , and illustrate why it is necessary to introduce a lower-bound truncation for . Finally, we quantify the error introduced by this truncation.
Derivation of
Since the MBP is mathematically described by Eq. (9), the ACF , averaged over all corresponding to different test reviews, can be written as
We introduce the variable transformation , which maps the interval to . Under this change of variables, Eq. (25) becomes
where we have defined . The above expression shows that is the Laplace transform of . A standard Laplace transform pair for is given by
Furthermore, a shift in –space by introduces an exponential factor in the transform,
Comparing this expression with the assumed form of the ACF , we obtain
where denotes the Heaviside step function. Transforming back to the original variable , the weight distribution is given by
where ensures proper normalization of over the domain of . From the analytical form, the lower bound in -space follows directly as
Consequently, is non-zero only for .
The ensemble-averaged branching parameter
Given the weight distribution , a natural quantity to consider is an ensemble-averaged branching parameter that captures the heterogeneous branching dynamics across reviews,
However, the normalization integral of diverges. This divergence follows from the asymptotic behavior of the weight distribution,
which produces an unphysical divergence as approaches zero. This pathology reflects the excessive weight assigned to extremely small branching parameters, which do not correspond to meaningful dynamical realizations. Importantly, the ACF retains its heavy-tailed form even if a lower-bound cutoff is introduced and only the remaining portion of is retained. Motivated by this observation, we define a truncated weight distribution,
This truncation removes the spurious divergence at while preserving the long-range temporal correlations encoded in the ACF. Under this truncation, the normalization constant becomes
where , and we have defined . Substituting the truncated distribution into Eq. (31), the ensemble-averaged branching parameter is given by
where denotes a lower incomplete Gamma function. After straightforward simplification, the ensemble-averaged branching parameter is obtained as
A visualization of as a function of and for is shown in Figure A1(a). Within a reasonable range of the qualitative dependence of on these parameters remains unchanged.
Figure A1. (a) Ensemble-averaged branching parameter as a function of the correlation exponent and the upper bound for a fixed lower bound . (b) Order of magnitude of the relative truncation error , shown as a function of and for and . The error is of order across the explored parameter range, demonstrating that the truncation-induced correction decays rapidly for large and does not affect the long-range temporal behavior of the ACF .
Error of truncation
The error introduced by the truncation can be quantified by comparing the ACF computed from the untruncated weight distribution with that obtained from the truncated distribution. The ACF prior to truncation is given by Eq. (15). The truncated ACF is obtained by substituting Eq. (33) into Eq. (24), yielding
where is the lower incomplete Gamma function, and denotes the regularized incomplete Gamma function. The relative error induced by the truncation is therefore given by
where denotes the regularized upper incomplete Gamma function. This expression shows that the truncation error decays rapidly for , confirming that the long-range behavior of the ACF is preserved. This behavior is explicitly demonstrated in Figure A1(b), which shows that remains small for .
Avalanche size distribution in MBP
In this appendix, we derive the asymptotic form of the avalanche size distribution induced by the weight distribution in the MBP theory. For each single subcritical branching process with the review-specific branching parameter , the avalanche size distribution is given by
Given the weight distribution , the total avalanche size distribution of the MBP should be
where we can apply Eq. (33) for . We introduce the shifted variables
The avalanche-size distribution can be expressed with new variables
where denotes the upper bound of the integration since is only valid in the interval , and is a constant. Using ,
Plug the equation into the weight distribution yields
In the large- limit, the upper bound of the integral can be extended to infinity, yielding
Factoring out the dominant exponential term and neglecting the quadratic term in the exponent gives
Combining all factors, we obtain the asymptotic avalanche-size distribution
where . This result shows that the MBP produces a power-law avalanche-size distribution with exponent , followed by an exponential cutoff controlled by the distance from criticality.
Additional details
Excluded Network Realizations
A small fraction of trained networks was excluded from the avalanche and branching analysis due to pathological behaviors observed during inference. The first exclusion criterion was numerical instability, manifested as NaN values in the hidden states . The second criterion was the emergence of persistent fast oscillations in the hidden states , synchronized across all hidden-state dimensions, as illustrated in Figure A2. In this case, the hidden-state vector repeatedly alternates between two distinct points in the state space across successive timesteps, resulting in a pronounced periodic pattern. Such behavior is indicative of a highly ordered dynamical regime and is typically observed at early training epochs. While this oscillatory behavior may be related to bifurcation-like phenomena in recurrent dynamical systems Pascanu et al. [ [38], a detailed investigation of its origin is beyond the scope of the present work. As a consequence, the corresponding autocorrelation function displays strong periodic structure rather than the exponential decay assumed in the branching-process framework, rendering these cases unsuitable for our analysis. Both types of pathological behavior occur more frequently in networks with larger hidden-state dimensionality. Nevertheless, for each hidden-state dimension, no more than four out of 28 trained networks were affected, and excluding these cases does not affect the statistical robustness or validity of our conclusions.
Figure A2. Time series of selected normalized hidden-state from a trained LSTM with hidden-state dimensionality 96, evaluated on a single test review. For clarity, only ten out of the 96 hidden-state dimensions are shown. The displayed components exhibit pronounced, synchronized fast oscillations.
Avalanche statistics in larger LSTM networks
In the main text, the avalanche size distribution for a hidden-state dimensionality of 128 is shown as a representative example, as illustrated in Figure 4(d). Here, we present corresponding results for additional larger networks to demonstrate that the observed subcritical behavior is not specific to that particular network size. As shown in Figure A3, LSTM networks with hidden-state dimensionalities ranging from 16 to 96 exhibit avalanche size distributions that deviate from power-law scaling at both early and optimal training epochs, indicating persistent subcritical dynamics. These results confirm that the subcritical avalanche behavior observed in large networks is robust across different hidden-state dimensionalities.
Figure A3. Avalanche size distributions for LSTM networks with hidden-state dimensionalities 16–96 at , shown for early (blue) and optimal (red) training epochs. Error bars indicate variability across random seeds.
Analysis Using Continuous-Valued Activity Signals
To assess the robustness of our results with respect to the definition of the activity signal, we repeated the temporal-correlation and branching analyses using a continuous-valued definition of the activity , which removes the need for explicit thresholding and binarization of the hidden states. Figure A4 summarizes the results obtained from this continuous-valued activity signal. The ACFs, spectral exponents , and ensemble-averaged branching parameters exhibit trends that closely mirror those observed for the discrete-count activity signals presented in the main text. In particular, small networks operate closer to criticality at the optimal epoch, while larger networks remain subcritical, and training drives the system toward longer temporal correlations. These results demonstrate that our conclusions are not sensitive to the specific choice of activity definition and are therefore robust.

Figure A4. Temporal correlations and branching statistics for continuous-valued activity signals. (a,b) ACF of the continuous-valued activity at the early (a) and optimal (b) training epochs for different hidden-state dimensionalities. (c) Spectral exponent of the corresponding noise at the optimal epoch, shown as a function of hidden-state dimensionality. (d) Evolution of the ensemble-averaged branching parameter over training epochs for different hidden-state dimensionalities. Results obtained using discrete-count activity signals are shown in the main text and display qualitatively similar behavior.
References
References
- [1]John J Hopfield. Neural networks and physical systems with emergent collective computational abilities. Proceedings of the national academy of sciences, 79(8):2554–2558, 1982.
- [2]David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. Learning representations by back-propagating errors. nature, 323(6088):533–536, 1986.
- [3]Thierry Mora and William Bialek. Are biological systems poised at criticality? Journal of Statistical Physics, 144(2):268–302, 2011.DOI
- [4]Nils Bertschinger, Thomas Natschläger, and Robert Legenstein. At the edge of chaos: Real-time computations and self-organized criticality in recurrent neural networks. Advances in neural information processing systems, 17, 2004.
- [5]Joschka Boedecker, Oliver Obst, Joseph T Lizier, N Michael Mayer, and Minoru Asada. Information processing in echo state networks at the edge of chaos. Theory in Biosciences, 131(3):205–213, 2012.DOI
- [6]Ling Feng, Lin Zhang, and Choy Heng Lai. Optimal machine intelligence at the edge of chaos. arXiv preprint arXiv:1909.05176, 2019.arxiv.org/abs/1909.05176
- [7]Wenbo Wei, Nicholas Chong Jia Le, Choy Heng Lai, and Ling Feng. Multiple descents in deep learning as a sequence of order-chaos transitions. arXiv preprint arXiv:2505.20030, 2025.
- [8]John M Beggs and Dietmar Plenz. Neuronal avalanches in neocortical circuits. Journal of neuroscience, 23(35):11167–11177, 2003.pubmed.ncbi.nlm.nih.gov/14657176
- [9]Arsham Ghavasieh, Meritxell Vila-Minana, Akanksha Khurd, John Beggs, Gerardo Ortiz, and Santo Fortunato. Toward a physics of deep learning and brains. arXiv preprint arXiv:2509.22649, 2025.arxiv.org/abs/2509.22649
- [10]Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In European conference on computer vision, pages 630–645. Springer, 2016.arxiv.org/abs/1603.05027
- [11]Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735–1780, 1997.
- [12]Klaus Linkenkaer-Hansen, Simo Monto, Heikki Rytsälä, Kirsi Suominen, Erkki Isometsä, and Seppo Kähkönen. Breakdown of long-range temporal correlations in theta oscillations in patients with major depressive disorder. Journal of Neuroscience, 25(44):10131–10137, 2005.
- [13]Teresa Montez, Simon-Shlomo Poil, Bethany F Jones, Ilonka Manshanden, Jeroen PA Verbunt, Bob W van Dijk, Arjen B Brussaard, Arjen Van Oo yen, Cornelis J Stam, Philip Scheltens, et al. Altered temporal correlations in parietal alpha and prefrontal theta oscillations in early-stage alzheimer disease. Proceedings of the National Academy of Sciences, 106(5):1614–1619, 2009.DOI
- [14]Anupriya Pathania, Melynda Schreiber, Matthew W Miller, Matthew J Euler, and Keith R Lohse. Exploring the reliability and sensitivity of the eeg power spectrum as a biomarker. International Journal of Psychophysiology, 160:18–27, 2021.DOI
- [15]Paolo Allegrini, Danilo Menicucci, Remo Bedini, Leone Fronzoni, Angelo Gemignani, Paolo Gr igolini, Bruce J West, and Paolo Paradisi. Spontaneous brain activity as a source of ideal 1/f noise. Physical Review E—Statistical, Nonlinear, and Soft Matter Physics, 80(6):061914, 2009.pubmed.ncbi.nlm.nih.gov/20365197
- [16]Nicholas Jia Le Chong and Ling Feng. Self-organization toward 1/f noise in deep neural networks. Chaos: An Interdisciplinary Journal of Nonlinear Science, 34(8), 2024.DOI
- [17]Per Bak, Chao Tang, and Kurt Wiesenfeld. Self-organized criticality. Physical review A, 38(1):364, 1988.
- [18]TC Per Bak and Kurt Wiesenfeld. Self-organized criticality: and explanation of 1/f noise. Phys. Rev. Let, 59:381–384, 1987.pubmed.ncbi.nlm.nih.gov/10035754
- [19]Dietmar Plenz, Tiago L Ribeiro, Stephanie R Miller, Patrick A Kells, Ali Vakili, and Elliott L Capek. Self-organized criticality in the brain. Frontiers in Physics, 9:639389, 2021.arxiv.org/abs/2102.09124
- [20]Shan Yu, Andreas Klaus, Hongdian Yang, and Dietmar Plenz. Scale-invariant neuronal avalanche dynamics and the cut-off in size distributions. Plos one, 9(6):e99761, 2014.DOI
- [21]Thomas Petermann, Tara C Thiagarajan, Mikhail A Lebedev, Miguel AL Nicolelis, Dante R Chialvo, and Dietmar Plenz. Spontaneous cortical activity in awake monkeys composed of neuronal avalanches. Proceedings of the National Academy of Sciences, 106(37):15921–15926, 2009.DOI
- [22]Kim Christensen and Zeev Olami. Sandpile models with and without an underlying spatial structure. Physical Review E, 48(5):3361–3372, 1993.pubmed.ncbi.nlm.nih.gov/9960993
- [23]Stefano Zapperi, Kent Bæ kgaard Lauritsen, and H Eugene Stanley. Self-organized branching processes: mean-field theory for avalanches. Physical review letters, 75(22):4071, 1995.pubmed.ncbi.nlm.nih.gov/10059807
- [24]Theodore Edward Harris et al. The theory of branching processes, volume 6. Springer Berlin, 1963.
- [25]Andrew Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies, pages 142–150, 2011.
- [26]Viola Priesemann, Michael Wibral, Mario Valderrama, Robert Pröpper, Michel Le Van Quyen, Theo Geisel, Jochen Triesch, Danko Nikolić, and Matthias HJ Munk. Spike avalanches in vivo suggest a driven, slightly subcritical brain state. Frontiers in systems neuroscience, 8:108, 2014.
- [27]Jens Wilting and Viola Priesemann. Inferring collective dynamical states from widely unobserved systems. Nature communications, 9(1):2325, 2018.DOI
- [28]Henrik Jeldtoft Jensen, Kim Christensen, and Hans C Fogedby. 1/f noise, distribution of lifetimes, and a pile of sand. Physical Review B, 40(10):7425, 1989.pubmed.ncbi.nlm.nih.gov/9991162
- [29]Lasse Laurson, Mikko J Alava, and Stefano Zapperi. Power spectra of self-organized critical sandpiles. Journal of Statistical Mechanics: Theory and Experiment, 2005(11):L11001–L11001, 2005.DOI
- [30]C-K Peng, Sergey V Buldyrev, Shlomo Havlin, Michael Simons, H Eugene Stanley, and Ary L Goldberger. Mosaic organization of dna nucleotides. Physical review e, 49(2):1685, 1994.pubmed.ncbi.nlm.nih.gov/9961383
- [31]Richard Hardstone, Simon-Shlomo Poil, Giuseppina Schiavone, Rick Jansen, Vadim V Nikulin, Huibert D Mansvelder, and Klaus Linkenkaer-Hansen. Detrended fluctuation analysis: a scale-free view on neuronal oscillations. Frontiers in physiology, 3:23105, 2012.
- [38]Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks. In International conference on machine learning, pages 1310–1318. Pmlr, 2013.arxiv.org/abs/1211.5063






