WorksComputer Science
Towards Understanding the Impact of Plasticity Loss on Reinforcement Learning in Stochastic Environments
Abstract
Deep reinforcement learning (RL) is a powerful framework for sequential decision-making. Despite many high-profile successes in recent years, deep RL is known to be brittle. An emerging line of research attributes this brittleness to a loss of plasticity in neural networks. However, plasticity loss has so far been understudied in stochastic environments. We study this interplay systematically, training SAC agents on three DeepMind Control tasks augmented with controlled levels of observation and action noise. We find that plasticity-preserving methods increase learning robustness on the two harder tasks, with their advantage growing as observation noise increases, while combining resets with layer normalization underperforms resets alone at every observation noise level tested. Further, stochasticity can accelerate the loss of plasticity in harder environments, whereas in easier settings noise can act as a regularizer that sustains plasticity. Lastly, plasticity loss is most pronounced in the early stages of training. In the most challenging stochastic settings, TD instability rather than gradual plasticity loss becomes the dominant failure mode.
Introduction
Over the last decade, deep Reinforcement Learning (RL) has achieved impressive results from game playing [51] over energy markets [40, 20] to control of complex real world systems [23] [6, 10, 31]. Predominantly however, RL research still is focused on simulated environments with deterministic transition dynamics [53, 8, 52]. However, stochasticity is very much inherent in real-world applications, where sensors are noisy, actuators imprecise, and dynamics uncertain. A natural question is therefore how well modern deep RL algorithms learn under such conditions, and what the root cause is for the observed brittleness of RL [24] in the presence of noise.
Multiple lines of research have emerged to understand and mitigate the brittleness of RL. On the side of mitigation, many works address this by automating design choices present in RL pipelines [45]. Some work in this direction is more concerned with augmenting environments, e.g., via curriculum learning [43, 32] to expose agents to more diverse experiences, while others are trying to address the brittleness via hyperparameter tuning [11]. While such works are able to produce more stable and performative RL agents, they provide little insight into the ultimate causes.
On the side of understanding the causes, attention has shifted towards the loss of plasticity of neural networks in deep RL pipelines: agents are known to overfit to transitions collected early in training, when the policy is still far from optimal [42, 28], gradually losing the ability to update their representations in response to new experience [34]. Interventions that counteract this, such as periodic resets and normalization, measurably improve learning on standard benchmarks [42, 41]. Closely related work has attributed the primacy bias to temporal difference (TD) instability, i.e. the early divergence of -values [27].
The interplay between plasticity loss and environment stochasticity remains largely unexamined, although two considerations suggest it warrants study. The first is observational: in a benchmarking study of robust RL algorithms, Glossop et al. [ [19] find that the presence of noise during training is considerably more detrimental than the same noise at deployment. The second is mechanistic: RL algorithms optimize an expected return, and the samples used to estimate it are collected throughout training. Zero-mean noise should therefore average out, provided realizations are weighted equally regardless of when they were collected. Plasticity loss might break this, inducing a bias towards early samples. Accordingly, our object of study is learning robustness, the ability of an algorithm to learn a task at all from noisy samples, rather than robustness in the more common sense of a deployed controller that remains stable or safe under disturbances [9].
This motivates our central question: can preserving plasticity improve learning robustness? We investigate this with a systematic study of Soft Actor-Critic (SAC) [21] on three proprioceptive control tasks from the DeepMind Control suite (DMC suite) [52], augmented with controlled levels of observation and action noise. We do not verify the mechanism directly, but our setting yields a testable prediction: if plasticity loss is more damaging under noise, the advantage of plasticity-preserving methods should grow with the noise level. We probe plasticity throughout training to verify that the tested methods indeed preserve it and to establish plasticity as a potential factor in learning robustness under noise. We additionally track maximum -values to separate the effects of plasticity loss and TD instability [27].
Our key findings are:
Plasticity-preserving methods — in particular periodic resets [42] — improve learning robustness under observation noise on the two harder tasks, with their advantage growing at higher noise levels.
The relationship between noise and plasticity loss is bidirectional as noise accelerates plasticity loss, and low plasticity in turn compounds learning failure. On easier tasks, however, noise can act as a regularizer that actually sustains plasticity.
Plasticity loss is most pronounced in the early stages of training and is exacerbated by noise, but in the most challenging settings the dominant failure mode is TD instability rather than gradual plasticity loss and stochastic environments make this catastrophic failure mode more likely to occur.
Related Work
Controlled noise in simulated learning tasks. Previous work has augmented simulators with observation and action noise to reflect stochastic dynamics in real world applications, with the objective of obtaining policies that bridge the sim-to-real gap [46] or to study the robustness of learned policies under disturbances [55]. Farebrother et al. [2024] compare categorical and MSE regression on the Arcade-Learning-Environment [37] with (default) and without action-noise. They find that categorical regressions’ improved performance over MSE is contingent upon the presence of action noise and attribute it to being less prone to overfitting on noisy targets. Glossop et al. [ [19] study how different noise types (dynamics, observation, action) affect the score achievable by PPO [49] and SAC [21] on the cart-pole environment and find noise on dynamics and observations to be much more detrimental to performance. Interestingly, they find that the presence of observation noise during training is already detrimental to performance under any noise level at test time and lower than performance if trained without noise. Our work starts exactly here, aiming to study how noise affects the learning dynamics.
RL that matters. To mitigate the known and well documented brittleness of RL [24, 12, 2], various research directions are actively being explored. Under the umbrella-term AutoRL, Parker-Holder et al. [ [45] identify multiple dimensions that are (automatically) adapted to improve robustness of learning pipelines. For example, various methods and techniques are proposed to automatically adapt the hyperparameters of RL pipelines, to increase the final performance [see, e.g., Jaderberg et al., 2017, Franke et al., 2021, Parker-Holder et al., 2020], as well as improve robustness [13, 11]. Further, environment design approaches propose to augment different elements of the environment, including reward function augmentation [39, 16] (Hu et al. 2020), action spaces [38, 14, 5] as well as observation space augmentation [47, 54]. The latter is of particular interest to our study as such work considers automatic selection of transformations (such as crop, rotate and flip) to image observations, such that policies are invariant to these transformations while maximizing rewards. In contrast to these approaches, which primarily prioritize maximizing final rewards, our study shifts the focus toward deepening the understanding of reinforcement learning in noisy environments and exploring how plasticity-preserving methods can enhance robustness under such conditions. Thus, we focus on settings in which we draw random samples, rather than randomly perturbing samples.
Background
MDPs, RL and the Effect of Different Sources of Noise
The goal of RL is to find the optimal policy in a Markov Decision Process (MDP). An MDP is defined by a tuple where is the state space, is the action space, defines the transition probability distribution, is the reward function and is the discount factor. An optimal policy then needs to maximize the expected discounted return .
Perturbing the observations of the agent at time as breaks the Markov property and gives rise to a Partially Observable Markov Decision Process (POMDP) where is the observation space. Hence, solving the POMDP generally involves a state estimation problem to maintain a belief over the true state [30].
In contrast, perturbations on the actions as yield a transformed MDP with a transition probability function which “absorbs” the action noise:
Quantifying Plasticity Loss
The loss of plasticity of a neural network describes its decreasing ability to fit a changing data distribution or to “respond to different possible learning signals” [34]. Lyle et al. [ [34] propose to quantify it as the ability of a (trained) neural network parameterized by to fit a high frequency transformation of the output of a randomly initialized model from the same class: , where and is a bias to compensate for the offset of the mean prediction of . This definition focuses on the ability to reduce the training loss, whereas other work has identified similar phenomena in terms of generalization capabilities of neural networks [3, 28].
A broad range of work explains plasticity loss by a reduced representational and signal propagation capacity when compared to the time of initialization. Prominent measures to detect such capacity loss are dormant neurons [50], a low effective rank of feature representations [33] or linearization of activation functions [35]. Further, optimization pathologies such as loss landscape sharpness and a degenerate neural-tangent kernel [35] have been connected to the cause of plasticity loss.
Previous work suggests that preserving the plasticity of the critic is more important in SAC [41] (Ma et al. 2024). This likely is due to the fact that the phenomena treated under the term plasticity loss have also been studied more specifically in the context of TD learning. Kumar et al. [ [33] studied and explained the decrease of the effective rank in deep RL as a direct result of TD learning. And Hussing et al. [ [27] attributed the primacy-bias not to overfitting on early data but to early value divergence due to limited action-space coverage of samples at the beginning of training.
We will refer to -value divergence as TD instability as one specific mechanism that may cause severe plasticity loss, in accordance with the findings by Lyle et al. [ [35] that regression on high offset targets results in particularly severe loss of plasticity.
Maintaining Comparable Plasticity Probing Throughout Training
To assess the ability of a neural network implementing a parametric function to learn from non-stationary data which was gathered in a stochastic environment, we aim to directly quantify the plasticity of its parameters . To this end we adopt the definition of plasticity as proposed by Lyle et al. [ [34]:
Definition 1 (Optimization procedure). Given dataset and a loss function , we define an optimization (or update) procedure as a function that returns updated parameters , by performing steps minimizing starting from .
Together with a neural network architecture, implements the function . The optimization procedure may have further updatable parameters, such as dynamic learning rates or gradient moment statistics.2 For the sake of a clear presentation, we subsume these parameters into and note that they have no immediate effect on the computation of . Likewise the optimization procedure is implicitly specified by the choice of the loss function and the dataset and we omit the explicit dependence for the sake of brevity.
Definition 2 (Neural network plasticity). Given an optimization procedure as in definition 1, a neural network parameterized by , a probing dataset and a number of probing steps we define plasticity as:
That is, plasticity is the fraction of the initial loss that can be removed by the update procedure on the task within probing steps.
Following Lyle et al. [ [34], we define the probing task as an MSE-regression task, however with an important distinction: We generate the probing task by adding randomly-sampled and sine-transformed offsets to the current network’s predictions. I.e., given a sample we set the target as , where is sampled from a uniform distribution and has the same dimensionality as the output of .
The central difference to Lyle et al. [ [34] is that we offset of the probing samples specific to the respective input instead of using a global offset , which they set to the mean prediction of over . By considering the “shape” of we keep the initial probing loss approximately constant since all probing samples lie within an -ball around . We illustrate the difference between the original and our approach in Figure 1. We can think of this as probing the ability of the model to fit perturbations in arbitrary directions.

Figure 1. Illustration of the probing task. As implemented in this study (left); as originally proposed by Lyle et al. [2023] (right).
Experiments
Setup and Evaluation Protocol
Stochastic environments. We study additive observation and action noise for two reasons: (1) both can be implemented by wrapping existing DMC suite environments (humanoid-walk, quadruped-run and walker-run) without modifying the physics simulation, making the approach straightforward to extend to other benchmarks [48]; (2) they affect learning in fundamentally different ways: observation noise breaks the Markov property and gives rise to a POMDP, whereas action noise induces a transformed MDP that adds stochasticity to otherwise deterministic dynamics (§ 3.1). Following Rajan et al. [ [48], the wrapper transforms the agent’s action before passing it to the environment as , and the next-state observation to , before returning it to the agent.3
Learning algorithm. We train SAC agents [21, 22] for one million environment steps with default hyperparameters (see Appendix § 9). We vary the replay ratio (gradient steps per environment step) over to investigate whether higher update frequencies exacerbate overfitting under noise [42, 28]. When adding layer normalization (LN) or resets we follow the implementation details of the respective papers [42, 34, 35]: LN is inserted before each ReLU activation and resets are applied to all training networks (actor, critics) and Adam optimizer statistics every steps.
Evaluation protocol. We report performance as the mean over 5 evaluation episodes in a separately seeded evaluation environment. Increasing noise levels reduce the performance attainable by any method (Figure 2), compressing the range of achievable scores toward the random-agent baseline. As a result, we deem an equal absolute performance gap between two methods to carry more weight under high noise than in a noise-free setting. To account for this, we normalize scores per task and noise level using the Normalized Score of Agarwal et al. [ [1]:

Figure 2. Final performance of SAC under increasing observation noise (left) and action noise (right) after training steps.
In absence of a clearly defined reference performance, such as a game solved in ATARI, we set the target score as the mean over the best configuration per task and training regime, and the minimal score is given by the performance of a random agent in the environment. This normalization ensures that method comparisons are expressed relative to the difficulty of each noise condition, making differences across noise levels and tasks commensurable. For reporting per method we select the best replay ratio based on mean training performance over the last 5000 environment steps, decoupling hyperparameter selection from evaluation. We report the raw learning curves in Appendix § 10.
The Effect of Noise During Training on SAC
Glossop et al. [ [19] show that even low levels of observation noise during training are far more detrimental than the same noise applied only at evaluation time, suggesting that noise actively disrupts the learning process rather than merely making the task harder to execute. We confirm this finding on proprioceptive control tasks from DMC suite. Figure 2 shows that SAC is already strongly impaired by mild observation noise on the challenging tasks quadruped-run and humanoid-walk, whereas action noise causes a noticeable drop only at higher magnitudes.
Note that noise magnitudes across types are not directly comparable. The action noise is first absorbed by the environment’s dynamics before affecting the next state, so its effective impact on the state distribution can differ substantially from the nominal (see Appendix § 11). This makes it difficult to attribute the stronger impact of observation noise unambiguously to the Markov property violation versus the perturbation of observation dimensions critical for action selection.
Can Plasticity-Preserving Methods Improve Training Robustness Under Noise?
Following the motivation laid out in § 1, we ask whether methods proposed to preserve plasticity and counteract overfitting to early transitions [42, 28] also improve robustness when learning under noise.
Two interventions stand out as particularly effective on the DMC suite [41], each targeting a distinct mechanism. LN [Ba et al., 2016] has been motivated from two complementary angles. On the one hand, Lyle et al. [ [35] show that distribution shifts can cause large pre-activation changes that push neurons into saturation and thereby reduce plasticity, and that LN mitigates this. On the other hand, constraining feature activation norms stabilizes TD learning [7, 18, 27]. Periodic parameter resets [42] directly counteract overfitting to early samples by periodically reinitializing network weights, erasing the temporal ordering bias accumulated in the replay buffer [28].
Adding LN, periodic resets, or both improves final performance on the two harder tasks but not on walker-run (Figure 3), with the largest gains on quadruped-run where baseline SAC degrades rapidly under observation noise. On humanoid-walk, however, all four configurations converge at . Aggregated normalized scores (Figure 4, left) confirm that the relative advantage of periodic resets grows with increasing observation noise. Resetting alone outperforms resets combined with LN at all three observation noise levels, while baseline SAC and LN alone fare progressively worse. Under action noise (Figure 4, right), reset-based methods retain their advantage over baseline SAC, but it no longer grows with the noise level and the ranking between the two reset variants flips.

Figure 3. Episode returns from final evaluation after steps. Each point aggregates to mean and std over 5 seeds per method and its best replay ratio.

Figure 4. Aggregated normalized scores (stratified bootstrapping, inter-quartile mean (IQM) with 95% CI [1]) across environments. Per method, environment and noise level the results for the best replay ratio were selected for aggregation. Left (observation noise): the relative effectiveness of periodic resets grows with noise magnitude; resets alone consistently outperform resets combined with LN, while baseline SAC and LN alone deteriorate at higher noise levels. Right (action noise): no consistent ranking emerges and the relative performance of SAC becomes increasingly uncertain.
Can Plasticity Preservation Explain Robustness Under Noise?
The results in § 5.3 show that plasticity-preserving methods improve robustness under noise, but cannot fully prevent performance degradation. This is unsurprising in part as observation noise turns the MDP into a POMDP and thus some performance loss is inevitable regardless of plasticity. Beyond this, we use our results to better understand how noise and plasticity loss interact during learning. Specifically, we ask whether this relationship is unidirectional, with noise impairing plasticity which then harms performance, or whether learning failure and plasticity loss mutually reinforce each other.
To this end, Figure 5 plots mean critic plasticity across training against final normalized score, for each combination of method, environment, and observation noise level. We use mean plasticity as a proxy for the agent’s sustained ability to adapt its representations in response to the learning signal.4

Figure 5. Mean critic plasticity over training vs. final normalized score, per environment and observation noise level (). Per plot, an individual point is a combination of (method, seed, replay ratio). An -regression line is fit across all runs per plot.
Hard tasks: noise and plasticity loss reinforce each other. On quadruped-run, most runs that fail to improve over the random baseline show a catastrophic loss of plasticity, whereas runs that retain higher levels of plasticity tend to reach a better final return (Figure 5). Crucially, higher noise also shifts the distribution of learners toward low-plasticity outcomes, indicating the relationship is not unidirectional. Noise does not merely make the task harder for an otherwise intact agent; it also accelerates plasticity loss, which then further undermines learning. On humanoid-walk the same trend appears but is less pronounced. Agents with only mild plasticity loss frequently fail under high noise, and some failure is already observed in the noise-free setting, which shows that plasticity loss is only one of several factors making this particular task difficult.
Easy tasks: noise can prevent overfitting. Compared to the previous observations, walker-run shows the opposite pattern. Adding observation noise leads to more plastic learners on average, and final performance is weakly negatively correlated with plasticity. We hypothesize that noise acts as a mild regularizer in this setting. By making the task slightly harder, it prevents early convergence to simple behaviors that exhibit reduced plasticity.
Action noise. We observe qualitatively similar trends under the highest investigated level of action noise (), but not at lower magnitudes, consistent with the smaller absolute performance drops seen at those levels (Figure 3; see also Appendix § 12). These findings indicate that an environment’s stochasticity itself is a factor reducing learning robustness, and that this cannot be traced back to partial observability alone.
When Does Plasticity Loss Occur and Are Early Dynamics Critical?
In § 5.3 we motivated periodic resets as a remedy for overfitting to the noise-corrupted early transitions that dominate the replay buffer. This raises a more specific question: “During which training phase does plasticity loss actually occur, and is the early phase particularly critical?” If so, this would motivate more targeted interventions that apply resets aggressively early but less frequently later, trading plasticity preservation for learning speed.
To investigate, we focus on the most challenging settings in our study, quadruped-run and humanoid-walk under high observation noise (), and plot mean critic plasticity throughout training (Figure 6). We use a replay ratio of 4 to amplify any effects on plasticity and TD stability.

Figure 6. Evolution of critic plasticity throughout training on the quadruped-run and humanoid-walk tasks. Each curve shows the mean over 5 seeds. Plasticity values have been clipped to [0, 1] before averaging.
Consistent with prior work [27], plasticity loss is most pronounced in the very early stages of training and is further exacerbated by observation noise (Figure 6). For methods with periodic resets, the plasticity drop following each subsequent reset becomes progressively less severe. Notably, combining resets with LN does not fully eliminate this pattern: a persistent plasticity drop follows each reset, whereas resets alone eventually yield a regime with negligible plasticity loss between cycles.
To further distinguish plasticity loss from TD instability, which Hussing et al. [ [27] identified as the primary failure mode, we track the maximum -value within training batches (Figure 7). For baseline SAC on quadruped-run, -values diverge dramatically under high observation noise, indicating that TD updates become unstable. On humanoid-walk no comparable overestimation occurs, consistent with the smaller performance differences between methods seen on that task. These results suggest that in stochastic environments, TD instability is a particularly important failure mode to prevent. While TD instability is catastrophic in any setting, noise makes it more likely to occur whereas plasticity loss and overfitting to early samples alone tend to cause only gradual drops in performance.

Figure 7. Evolution of the maximum -value within training batches throughout training on the quadruped-run and humanoid-walk tasks. Each curve shows the mean over 5 seeds.
Limitations
Due to the computational overhead of evaluation over different noise types and levels we had to limit our experiments to a subset of benchmarking environments. Nevertheless, in terms of the evaluation setting we see a bigger limitation in the characteristics of the studied noise types, in particular applying the same level of noise indiscriminately of the observation dimension. We also believe that studying noisy transition dynamics is of great interest to better understand the interplay between plasticity loss and potential overfitting to early, noisy samples without directly transforming the setting from an MDP into a POMDP. We expect its implementation to be more difficult in relative terms as it will be necessary to implement this on the simulation level and can not be achieved by a simple wrapper. In line with prior works, we opted to apply unrealistically high levels of action noise. However, Glossop et al. [2022] suggest that true transition noise would result in a noticeable impact earlier. Lastly, while plasticity loss and TD stability have been established as important factors for the successful application of deep RL, they are only two of many and studying their interplay with other aspects, such as exploration, is required to gain a holistic understanding of the effectiveness of the evaluated methods.
Conclusion
Aggregated across the three DMC suite tasks studied, we find that plasticity-preserving methods improve learning robustness in stochastic environments: reset-based agents achieve a higher aggregate normalized score than baseline SAC at every noise type and level tested. Under observation noise this advantage grows with the noise level, meeting the criterion set out in § 1, whereas under action noise it is persistent but independent of the noise level. Moreover, we observe an increased loss of plasticity in these settings, and in the most drastic case (baseline SAC on quadruped-run under ) observation noise led to severe value overestimation from which training did not recover. Since stochasticity makes environments inherently more difficult, it is unsurprising that the methods investigated can mitigate but not fully prevent the decrease in performance. Overall, our findings suggest that for SAC in stochastic environments, maintaining TD stability is more critical than preventing overfitting to perturbed, early samples. We found LN to be primarily effective in preventing value divergence while permitting some loss of plasticity. Under observation noise, resetting was the most effective of the four configurations tested, providing both long-term TD stability and plasticity preservation. Somewhat surprisingly, combining resetting with LN underperformed resetting alone at every observation noise level tested and led to an ongoing, albeit slow, loss of plasticity throughout training.
We should expect data collected in the real world to be stochastic by nature and hence we identify a need for benchmark environments that allow the assessment of robust learning under such conditions for the holistic assessment of an algorithm’s capabilities. While we resorted to a simple extension of existing benchmark environments we envision benchmarks with more realistic non-deterministic characteristics that we should expect in the real world such as stochastic environment dynamics, random observation delays or observation noise that is well-calibrated with regard to the overall signal magnitude on the respective channels. Once access to such simulators exists, we should aim to further understand the interplay between environment stochasticity, plasticity loss and further central components of RL algorithms. Lastly, evaluation in such environments should inform the further development and refinement of algorithms in general and plasticity preserving methods in particular.
Acknowledgments
The authors acknowledge support by the state of Baden - Württemberg through bwHPC and the German Research Foundation (DFG) through grant INST 35/1597-1 FUGG. The authors further acknowledge funding from The Carl Zeiss Foundation through the research network “Responsive and Scalable Learning for Robots Assisting Humans” (ReScaLe) of the University of Freiburg. This research was funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under grant number 539134284, through EFRE (FEIH_2698644) and the state of Baden-Württemberg.
Hyperparameter Settings Soft Actor Critic
For our study we use the SAC implementation of CleanRL [26]. We keep the default hyperparameters and only vary the replay ratio. We report the hyperparameters in Table 1. Further details about hyperparameters can be seen in our code repository.
| optimizer | Adam |
| batch size | 256 |
| (Adam) | 0.9 |
| (Adam) | 0.999 |
| learning rate | |
| buffer size | |
| (discount factor) | 0.99 |
| (target smoothing coefficient) | 0.005 |
| (entropy coefficient, fixed) | 0.2 |
| train frequency | 1 |
| target network frequency | 1 |
| replay ratio | [1, 4, 8] |
| learning starts | 1000 |
Table 1. Hyperparameters for SAC.
Learning Curves and Normalized Scores per Environment
We first provide the reward curves per considered replay ratio, environment and observation noise (Figure 8) as well as action noise (Figure 9). We further provide the plasticity (Figure 10 and Figure 11) and evolution (Figure 12 and Figure 13) plots.

Figure 8. Learning curves for the evaluation episode return across replay ratios and observation noise levels.

Figure 9. Learning curves for the evaluation episode return across replay ratios and action noise levels.

Figure 10. Learning curves for the plasticity evolution across replay ratios and observation noise levels.

Figure 11. Learning curves for the plasticity evolution across replay ratios and action noise levels.

Figure 12. Learning curves for the across replay ratios and observation noise levels.

Figure 13. Learning curves for the across replay ratios and action noise levels.
Reward curves. Figure 8 shows the reward curves across observation noise levels.
Figure 9 shows the reward curves across action noise levels.
Plasticity evolution. Figure 10 shows the evolution of critic plasticity across observation noise levels.
Figure 11 shows the evolution of critic plasticity across action noise levels.
Maximum -value. Figure 12 shows the evolution of across observation noise levels.
Figure 13 shows the evolution of across action noise levels.
Normalized scores. Here we report the final episode returns for the best replay ratio across all prior combinations.

Figure 14. Normalized final episode return of the best replay ratio per method, environment and noise level.
Noise Effect on Transition Dynamics

Figure 15. Distribution of per-dimension next-state observation sensitivity to action noise for three dm_control environments. For each step along a trajectory, independent action–noise pairs , with , are evaluated from the same physics state. The per-step standard deviation of the resulting observation errors is computed for each observation dimension and averaged over steps (1000 steps 3 seeds). Bars show the proportion of observation dimensions whose mean std falls within each magnitude bin.
Plasticity vs. Learning Robustness Under Action Noise

Figure 16. Mean critic plasticity over training vs. final normalized score, per environment and action noise level (). Within one plot each point is a combination of (method, seed, replay ratio). A -regression line is fit across all runs per plot.
Replay-Ratio Sensitivity to Environment Noise

Figure 17. Episode returns from final evaluation after steps per noise level and across replay ratios. The goal is to detect whether high replay ratios are more affected by high noise levels due to overfitting (which is generally not the case). Each point aggregates to mean and std over 5 seeds.
References
- [1]R. Agarwal, M. Schwarzer, P. S. Castro, A. C. Courville, and M. G. Bellemare. Deep reinforcement learning at the edge of the statistical precipice. In M. Ranzato, A. Beygelzimer, K. Nguyen, P. Liang, J. Vaughan, and Y. Dauphin, editors, Proceedings of the 34th International Conference on Advances in Neural Information Processing Systems (NeurIPS’21), pages 29304–29320. Curran Associates, 2021.arxiv.org/abs/2108.13264
- [2]M. Andrychowicz, A. Raichuk, P. Stańczyk, M. Orsini, S. Girgin, R. Marinier, L. Hussenot, M. Geist, O. Pietquin, M. Michalski, S. Gelly, and O. Bachem. What matters for on-policy deep actor-critic methods? a large-scale study. In The Ninth International Conference on Learning Representations (ICLR’21). ICLR, 2021.
- [3]J. Ash and R. P. Adams. On warm-starting neural network training. In H. Larochelle, M. Ranzato, R. Hadsell, M.-F. Balcan, and H. Lin, editors, Proceedings of the 33rd International Conference on Advances in Neural Information Processing Systems (NeurIPS’20), pages 3884–3894. Curran Associates, 2020.arxiv.org/abs/1910.08475
- [5]A. Bagaria, J. K. Senthil, M. Slivinski, and G. Konidaris. Robustly learning composable options in deep reinforcement learning. In Z. Zhou, editor, Proceedings of the 30th International Joint Conference on Artificial Intelligence (IJCAI’21), pages 2161–2169, 2021.
- [7]A. Bhatt, D. Palenicek, B. Belousov, M. Argus, A. Amiranashvili, T. Brox, and J. Peters. Crossq: Batch normalization in deep reinforcement learning for greater sample efficiency and simplicity. In The Twelfth International Conference on Learning Representations (ICLR’24). ICLR, 2024.
- [8]G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba. OpenAI gym. arXiv:1606.01540, 2016.arxiv.org/abs/1606.01540
- [9]L. Brunke, M. Greiff, A. W. Hall, Z. Yuan, S. Zhou, J. Panerati, and A. P. Schoellig. Safe learning in robotics: From learning-based control to safe reinforcement learning. Annual Review of Control, Robotics, and Autonomous Systems, 5:411–444, 2022.arxiv.org/abs/2108.06266
- [10]J. Degrave, F. Felici, J. Buchli, M. Neunert, B. Tracey, F. Carpanese, T. Ewalds, R. Hafner, A. Abdolmaleki, D. D. L. Casas, C. Donner, L. Fritz, C. Galperti, A. Huber, J. Keeling, M. Tsimpou kelli, J. Kay, A. Merle, J.-M. Moret, S. Noury, F. Pesamosca, D. Pfau, O. Sauter, C. Sommariva, S. Coda, B. Duval, A. Fasoli, P. Kohli, K. Kavukcuoglu, D. Hassabis, and M. R. Riedmiller. Magnetic control of tokamak plasmas through deep reinforcement learning. Nature, 602(7897):414–419, 2022.DOI
- [11]T. Eimer, M. Lindauer, and R. Raileanu. Hyperparameters in reinforcement learning and how to tune them. In A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning (ICML’23), volume 202 of Proceedings of Machine Learning Research. PMLR, 2023.arxiv.org/abs/2306.01324
- [12]L. Engstrom, A. Ilyas, S. Santurkar, D. Tsipras, F. Janoos, L. Rudolph, and A. Madry. Implementation matters in deep RL: A case study on PPO and TRPO. In The Eighth International Conference on Learning Representations (ICLR’20). ICLR, 2020.
- [13]S. Falkner, A. Klein, and F. Hutter. BOHB: Robust and efficient Hyperparameter Optimization at scale. In J. Dy and A. Krause, editors, Proceedings of the 35th International Conference on Machine Learning (ICML’18), volume 80, pages 1437–1446. Proceedings of Machine Learning Research, 2018.arxiv.org/abs/1807.01774
- [14]M. D. Farahani and N. Mozayani. Automatic construction and evaluation of macro-actions in reinforcement learning. Applied Soft Computing, 82, 2019.DOI
- [16]A. Faust, A. Francis, and D. Mehta. Evolving rewards to automate reinforcement learning. In K. Eggensperger, M. Feurer, F. Hutter, and J. Vanschoren, editors, ICML workshop on Automated Machine Learning (AutoML workshop 2019), 2019.arxiv.org/abs/1905.07628
- [18]M. Gallici, M. Fellows, B. Ellis, B. Pou, I. Masmitja, J. Foerster, and M. Martin. Simplifying deep temporal difference learning. In The Thirteenth International Conference on Learning Representations (ICLR’25), pages 78148–78190. ICLR, 2025.arxiv.org/abs/2407.04811
- [19]C. Glossop, J. Panerati, A. Krishnan, Z. Yuan, and A. P. Schoellig. Characterising the robustness of reinforcement learning for continuous control using disturbance injection. In NeurIPS 2022 Workshop on Distribution Shifts: Connecting Methods and Applications, 2022.arxiv.org/abs/2210.15199
- [20]B. Grooten, P. MacAlpine, K. Subramanian, P. R. Wurman, and P. Stone. Out-of-distribution generalization with a sparc: Racing 100 unseen vehicles with a single policy. In S. Koenig, C. Jenkins, and M. E. Taylor, editors, Proceedings of the Fourteenth AAAI Conference on Artificial Intelligence (AAAI’26). AAAI Press, 2026.
- [21]T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In J. Dy and A. Krause, editors, Proceedings of the 35th International Conference on Machine Learning (ICML’18), volume 80, pages 1861–1870. Proceedings of Machine Learning Research, 2018.arxiv.org/abs/1801.01290
- [22]T. Haarnoja, A. Zhou, K. Hartikainen, G. Tucker, S. Ha, J. Tan, V. Kumar, H. Zhu, A. Gupta, P. Abbeel, and S. Levine. Soft actor-critic algorithms and applications. arXiv:1812.05905 [cs.LG], 2019.arxiv.org/abs/1812.05905
- [23]N. Harder, R. Qussous, and A. Weidlich. Fit for purpose: Modeling wholesale electricity markets realistically with multi-agent deep reinforcement learning. Energy and AI, 14, 2023.DOI
- [24]P. Henderson, R. Islam, P. Bachman, J. Pineau, D. Precup, and D. Meger. Deep reinforcement learning that matters. In S. McIlraith and K. Weinberger, editors, Proceedings of the Thirty-Second Conference on Artificial Intelligence (AAAI’18). AAAI Press, 2018.arxiv.org/abs/1709.06560
- [26]S. Huang, R. F. J. Dossa, C. Ye, J. Braga, D. Chakraborty, K. Mehta, and J. G. Araújo. Cleanrl: High-quality single-file implementations of deep reinforcement learning algorithms. Journal of Machine Learning Research, 23(274):1–18, 2022.arxiv.org/abs/2111.08819
- [27]M. Hussing, C. A. Voelcker, I. Gilitschenski, A. Farahmand, and E. Eaton. Dissecting deep RL with high update ratios: Combatting value divergence. Reinforcement Learning Journal, 2:995–1018, 2024.arxiv.org/abs/2403.05996
- [28]M. Igl, G. Farquhar, J. Luketina, W. Boehmer, and S. Whiteson. Transient non-stationarity and generalisation in deep reinforcement learning. In The Ninth International Conference on Learning Representations (ICLR’21). ICLR, 2021.
- [30]L. P. Kaelbling, M. L. Littman, and A. R. Cassandra. Planning and acting in partially observable stochastic domains. Artificial Intelligence, 101(1):99–134, 1998.
- [31]E. Kaufmann, L. Bauersfeld, A. Loquercio, M. Müller, V. Koltun, and D. Scaramuzza. Champion-level drone racing using deep reinforcement learning. Nature, 620(7976):982–987, 2023.DOI
- [32]P. Klink, C. D’Eramo, J. Peters, and J. Pajarinən. Self-paced deep reinforcement learning. In H. Larochelle, M. Ranzato, R. Hadsell, M.-F. Balcan, and H. Lin, editors, Proceedings of the 33rd International Conference on Advances in Neural Information Processing Systems (NeurIPS’20), pages 9216–9227. Curran Associates, 2020.arxiv.org/abs/2004.11812
- [33]A. Kumar, R. Agarwal, D. Ghosh, and S. Levine. Implicit under-parameterization inhibits data-efficient deep reinforcement learning. In The Ninth International Conference on Learning Representations (ICLR’21). ICLR, 2021.arxiv.org/abs/2010.14498
- [34]C. Lyle, Z. Zheng, E. Nikishin, B. A. Pires, R. Pascanu, and W. Dabney. Understanding plasticity in neural networks. In A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning (ICML’23), volume 202 of Proceedings of Machine Learning Research, pages 23190–23211. PMLR, 2023.arxiv.org/abs/2303.01486
- [35]C. Lyle, Z. Zheng, K. Khetarpal, H. V. Hasselt, R. Pascanu, J. Martens, and W. Dabney. Disentangling the causes of plasticity loss in neural networks. In V. Lomonaco, S. Melacci, T. Tuytelaars, S. Chandar, and R. Pascanu, editors, Proceedings of The 3rd Conference on Lifelong Learning Agents, volume 274, pages 750–783. Proceedings of Machine Learning Research, 2024.arxiv.org/abs/2402.18762
- [37]M. C. Machado, M. G. Bellemare, E. Talvitie, J. Veness, M. J. Hausknecht, and M. Bowling. Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents. Journal of Artificial Intelligence Research, 61:523–562, 2018.arxiv.org/abs/1709.06009
- [38]D. J. Mankowitz, T. A. Mann, P. Bacon, D. Precup, and S. Mannor. Learning robust options. In S. McIlraith and K. Weinberger, editors, Proceedings of the Thirty-Second Conference on Artificial Intelligence (AAAI’18), pages 6409–6416. AAAI Press, 2018.arxiv.org/abs/1802.03236
- [39]B. Marthi. Automatic shaping and decomposition of reward functions. In Z. Ghahramani, editor, Proceedings of the 24th International Conference on Machine Learning (ICML’07), pages 601–608. Omnipress, 2007.
- [40]V. Mnih, K. Kavukcuoglu, D. Silver, A. Rusu, J. Veness, M. Bellemare, A. Graves, M. Riedmiller, A. Fidjeland, G. Ostrovski, S. Petersen, C. Beattie, A. Sadik, I. Antonoglou, H. King, D. Kumaran, D. Wierstra, S. Legg, and D. Hassabis. Human-level control through deep reinforcement learning. Nature, 518(7540):529–533, 2015.DOI
- [41]M. Nauman, M. Bortkiewicz, P. Miłoś, T. Trzcinski, M. Ostaszewski, and M. Cygan. Overestimation, overfitting, and plasticity in actor-critic: the bitter lesson of reinforcement learning. In R. Salakhutdinov, Z. Kolter, K. Heller, A. Weller, N. Oliver, J. Scarlett, and F. Berkenkamp, editors, Proceedings of the 41st International Conference on Machine Learning (ICML’24), volume 251 of Proceedings of Machine Learning Research, pages 37342–37364. PMLR, 2024.arxiv.org/abs/2403.00514
- [42]E. Nikishin, M. Schwarzer, P. D’Orо, P. Bacon, and A. C. Courville. The primacy bias in deep reinforcement learning. In K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvári, G. Niu, and S. Sabato, editors, Proceedings of the 39th International Conference on Machine Learning (ICML’22), volume 162 of Proceedings of Machine Learning Research, pages 16828–16847. PMLR, 2022.arxiv.org/abs/2205.07802
- [43]OpenAI, I. Akkaya, M. Andrychowicz, M. Chociej, M. Litwin, B. McGrew, A. Petron, A. Paino, M. Plappert, G. Powell, R. Ribas, J. Schneider, N. Tezak, J. Tworek, P. Welinder, L. Weng, Q. Yuan, W. Zaremba, and L. Zhang. Solving rubik’s cube with a robot hand. arXiv:1910.07113, 2019.arxiv.org/abs/1910.07113
- [45]J. Parker-Holder, R. Rajan, X. Song, A. Biedenkapp, Y. Miao, T. Eimer, B. Zhang, V. Nguyen, R. Calandra, A. Faust, F. Hutter, and M. Lindauer. Automated reinforcement learning (AutoRL): A survey and open problems. Journal of Artificial Intelligence Research, 74:517–568, 2022.
- [46]X. B. Peng, M. Andrychowicz, W. Zaremba, and P. Abbeel. Sim-to-real transfer of robotic control with dynamics randomization. In Proceedings of IEEE International Conference on Robotics and Automation (ICRA’18), pages 1–8, 2018.
- [47]R. Raileanu, M. Goldstein, D. Yarats, I. Kostrikov, and R. Fergus. Automatic data augmentation for generalization in reinforcement learning. In M. Ranzato, A. Beygelzimer, K. Nguyen, P. Liang, J. Vaughan, and Y. Dauphin, editors, Proceedings of the 34th International Conference on Advances in Neural Information Processing Systems (NeurIPS’21), pages 5402–5415. Curran Associates, 2021.
- [48]R. Rajan, J. L. B. Diaz, S. Guttikonda, F. Ferreira, A. Biedenkapp, J. O. V. Hartz, and F. Hutter. MDP playground: An analysis and debug testbed for reinforcement learning. Journal of Machine Learning Research, 77:821–890, 2023.DOI
- [49]J. Schulman, F. Wolski, P. Dhar iwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv:1707.06347 [cs.LG], 2017.arxiv.org/abs/1707.06347
- [50]G. Sokar, R. Agarwal, P. S. Castro, and U. Evci. The dormant neuron phenomenon in deep reinforcement learning. In A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning (ICML’23), volume 202 of Proceedings of Machine Learning Research. PMLR, 2023.arxiv.org/abs/2302.12902
- [51]R. S. Sutton and A. G. Barto. Reinforcement learning - an introduction, 2nd Edition. MIT Press, 2018.
- [52]Y. Tassa, Y. Doron, A. Muldal, T. Erez, Y. Li, D. D. L. Casas, D. Budden, A. Abdolmaleki, J. Merel, A. Lefrancq, T. P. Lillicrap, and M. A. Riedmiller. Deepmind control suite. arXiv:1801.00690, 2018.arxiv.org/abs/1801.00690
- [53]E. Todorov, T. Erez, and Y. Tassa. MuJoCo: A physics engine for model-based control. In International Conference on Intelligent Robots and Systems (IROS’12), pages 5026–5033. ieeeсis, IEEE, 2012.DOI
- [54]D. Yarats, I. Kostrikov, and R. Fergus. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. In The Ninth International Conference on Learning Representations (ICLR’21). ICLR, 2021.
- [55]Z. Yuan, A. W. Hall, S. Zhou, L. Brunke, M. Greeff, J. Panerati, and A. P. Schoellig. Safe-control-gym: A unified benchmark suite for safe learning-based control and reinforcement learning in robotics. IEEE Robotics and Automation Letters, 7(4):11142–11149, 2022.arxiv.org/abs/2109.06325