betaFeedback

WorksComputer Science

The Stability Gap in Continual Reinforcement Learning

Abstract

Continual reinforcement learning (CRL) aims to enable agents to adapt to changing tasks while preserving previously acquired capabilities. Evaluation often relies on final aggregate metrics that can hide temporary performance degradation during adaptation. This work introduces a simple CRL benchmark that varies observation representations while preserving environment dynamics and rewards, supporting proof-of-concept demonstrations, inexpensive experimentation, and easily reproducible sanity checks. Using this benchmark, we demonstrate the stability gap in CRL: a substantial drop in prior-task performance after the introduction of new tasks, followed by partial or complete recovery. This pattern occurs despite the agent having complete access to past tasks.

Introduction

Sequential training can severely degrade neural network performance on previously learned tasks, a phenomenon known as catastrophic forgetting [1]. Continual learning (CL) aims to adapt to changing tasks while retaining acquired knowledge. Evaluation typically relies on task-boundary measures such as final performance, forgetting, and backward transfer, which can overlook temporary failures during adaptation. In supervised CL, De Lange et al. [2] identified the stability gap: a sharp drop in previous-task performance after a task transition, followed by partial or complete recovery. This gap can occur even when all past-task data remain available [3], a setting referred to as incremental joint-training.

Whether a stability gap also occurs in continual reinforcement learning (CRL) remains unexplored. Existing CRL evaluations emphasize final performance, or they look at transfer and forgetting measured only at task boundaries or after convergence [4]. Temporary performance drops may therefore have remained hidden. To isolate these dynamics, we introduce a lightweight transformed-state benchmark inspired by Permuted-MNIST [5]. Tasks share environment dynamics, rewards, and action spaces but apply different orthogonal transformations to the state representation. We compare value-based and actor-critic algorithms across discrete and continuous action spaces, using frequent evaluation of past tasks to test whether a stability gap arises in CRL.

Benchmark

We construct a controlled CRL benchmark in a way that can be applied to any environment with vector observations.1 Each task shares the same dynamics, rewards, and action space but uses a different observation representation. For each experimental seed, we independently sample NN orthogonal matrices Ti∈Rd×dT_i \in\mathbb{R}^{d \times d} and transform the state s∈Rd\mathbf{s} \in\mathbb{R}^{d} as

xi=[Tisei]\mathbf{x}_i = \begin{bmatrix} T_i\mathbf{s} \\ \mathbf{e}_i \end{bmatrix}

where ei\mathbf{e}_i is a one-hot task identifier. We obtain TiT_i by QR decomposition of Gaussian matrices, correcting column signs using the diagonal of RR. Orthogonality preserves angles, distances, and information, retaining the geometry needed learn the underlying dynamics. The task identifier can be omitted for a harder variant of the benchmark.

Figure 1 presents the general benchmark construction (a) and its instantiation on CartPole (b), which we use alongside InvertedPendulum for discrete and continuous control, respectively. Both environments have four-dimensional states (d=4d = 4), and we construct three tasks (N=3N = 3) for each. We verified that with our experimental settings, all algorithms we consider can learn all three tasks under multitask training, yet exhibit catastrophic forgetting under sequential fine-tuning. Panel (c) illustrates this for SAC-D.

Figure 1. Benchmark construction and CartPole instantiation. (a) Task-specific agent inputs are constructed by applying an orthogonal transformation TiT_i to a base state s\mathbf{s} and concatenating the result with a one-hot task identity. (b) An instantiation on three CartPole tasks: each four-dimensional state is transformed by a task-specific matrix and paired with a three-dimensional task identifier, yielding a seven-dimensional input. (c) All three tasks can be learned by multitask training (left), whereas sequential fine-tuning (right) shows catastrophic forgetting. Illustrated here for SAC-D on CartPole.

Experimental setup

Algorithms We evaluate DQN [6] and discrete SAC (SAC-D) [7] on CartPole, and DDPG [8] and SAC [9] on InvertedPendulum. All algorithms are trained and evaluated under an incremental joint-training paradigm: tasks are introduced sequentially, and the agent continues to train on all tasks encountered thus far. Unlike with methods that rely on limited replay or other approximations of prior-task experience, the agent thus retains full access to all past tasks, making any stability gap especially notable. For all algorithms, we maintain a separate replay buffer for each task and sample equal proportions of transitions from these buffers for each update. At each task transition, we reset the replay buffers and optimizer states while retaining the learned network parameters. Then, we collect 1,000 transitions from each encountered task before resuming network updates. We evaluate performance on all tasks every 15 environment steps during the first 30,000 steps of each task and every 250 steps thereafter. Performance on a task at a given timestep is measured as the mean return over 15 episodes.

For all experiments, we use the Adam optimizer, networks with two fully-connected hidden layers of 256 units each, a batch size of 128, and a discount factor γ=0.99\gamma= 0.99. We tune the learning rate separately for each algorithm, selecting the configuration with the highest mean return across all tasks at the end of training, averaged over 10 seeds. Tuning consists of an initial logarithmic sweep followed by up to three additional values near the best-performing region. The selected learning rates are 10−410^{-4} for DQN, 3×10−43 \times10^{-4} for DDPG and SAC, and 10−310^{-3} for SAC-D. We then run the experiments with the selected learning rates on 50 new seeds. Regarding the exploration hyperparameters, for DQN we follow [6] and employ an ϵ\epsilon-greedy policy with an annealing schedule. In our experiments, ϵ\epsilon decreases from 1.0 to 0.05 over the first 10% of timesteps, after which it remains constant. When it comes to SAC and SAC-D, we use automatic temperature tuning as proposed in [10, 7], with target entropies of −da-d_a and 0.98log⁡∣A∣0.98 \log|\mathcal{A}|, respectively. Here, dad_a denotes the dimensionality of the continuous action space, and ∣A∣|\mathcal{A}| denotes the number of discrete actions.

Linear Interpolation For each of the 50 experimental seeds, we evaluate a 200×200200 \times200 parameter grid built from the policies θ1,θ2,θ3\theta_1, \theta_2, \theta_3 learned after Tasks 1–3 in that run. The extrapolated fourth corner is θ4=θ1+θ3−θ2\theta_4 = \theta_1 + \theta_3 - \theta_2, and each grid policy is formed by bilinear interpolation:

θ(u,v)=v[(1−u)θ1+uθ2]+(1−v)[(1−u)θ4+uθ3],u,v∈[−0.5,1.5].\theta(u,v) = v\left[(1-u)\theta_1 + u\theta_2\right] + (1-v)\left[(1-u)\theta_4 + u\theta_3\right], \qquad u,v \in[-0.5,1.5].

At each grid point, we evaluate the performance on Task 1 by averaging the return over 15 episodes per seed.

Results

All four algorithms exhibit a sharp, substantial drop in previous-task performance after at least one task transition, followed by partial or complete recovery (Figure 2). Control experiments confirm that this pattern persists when optimizer states and replay buffers are retained across task transitions, when exploration parameters are reset at task transitions, and when returns are aggregated across seeds using the interquartile rather than the arithmetic mean.

Figure 2. A stability gap consistently shows up in CRL. Performance on Task 1 and 2 during incremental joint-training on three tasks for (a) DQN, (b) SAC-D, (c) DDPG, and (d) SAC. Dashed vertical lines mark task transitions. Curves show mean normalized returns across seeds, with shaded 95% bootstrapped confidence intervals, revealing transient drops in past-task performance in all settings.

Figure 3 compares Task 1 return along linear interpolations between successive checkpoints with Task 1 return during training. For SAC-D and SAC, Task 1 returns along the interpolated paths are, on average across seeds, close to those at the checkpoints, even though training shows pronounced transient drops. Thus, stable high-return linear paths between checkpoints can coexist with a stability gap along the training trajectory. For all algorithms except DQN, the dips in mean return during training are larger than the corresponding dips along the interpolated paths.

Figure 3. Training trajectories tend to boost larger performance drops than corresponding linear paths. Task 1 return along a three-checkpoint parameter-space path for DQN, SAC-D, DDPG, and SAC. Top: heatmaps show normalized mean Task 1 return; the red dashed path connects θ1\theta_1, θ2\theta_2, and θ3\theta_3. Bottom: Task 1 return during training on Task 2 and 3 (solid orange) and along linear interpolations between successive checkpoints (dashed red), with shaded 95% bootstrapped confidence intervals. Interpolation coefficients are mapped linearly onto the corresponding training intervals for visualization. Interpolation produces pronounced return drops for DQN and DDPG, while SAC-D and SAC remain near their checkpoint returns.

Conclusion

We demonstrate the existence of the stability gap in CRL: previous-task performance can drop sharply following a task transition, even under incremental joint-training. Consistent with prior work on unstable optimization trajectories in supervised CL [11, 3], our finding shows that even an objective that jointly incorporates all tasks encountered thus far is not, by itself, sufficient to maintain stable performance throughout training. The optimization trajectory also matters. This motivates future work on CRL methods that change the way optimization is done, particularly during the updates following task transitions, when the stability gap may emerge.

Additionally, we propose a simple benchmark for CRL in which tasks vary only in their observation representations, while rewards and environment dynamics are stable. ics remain fixed. Although this design does not capture the full range of task changes encountered in CRL, its simplicity serves three purposes: (i) it supports proof-of-principle demonstrations, as in this study; (ii) it provides a standardized, inexpensive playground for initial iterations on new CRL methods; and (iii) it offers an easily reproducible sanity check for new algorithms.

References

  1. [1]Michael McCloskey and Neal J. Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Gordon H. Bower, editor, Psychology of Learning and Motivation, volume 24, pages 109–165. Academic Press, 1989.DOI
  2. [2]Matthias De Lange, Gido M. van de Ven, and Tinne Tuytelaars. Continual evaluation for lifelong learning: Identifying the stability gap. In International Conference on Learning Representations, 2023.arxiv.org/abs/2205.13452
  3. [3]Timm Hess, Tinne Tuytelaars, and Gido M. van de Ven. Two complementary perspectives to continual learning: Ask not only what to optimize, but also how. In Proceedings of the 1st ContinualAI Unconference, 2023, volume 249 of Proceedings of Machine Learning Research, pages 37–61. PMLR, 2024.
  4. [4]Maciej Wolczyk, Michał Zając, Razvan Pascanu, Łukasz Kuciński, and Piotr Miłos. Continual world: A robotic benchmark for continual reinforcement learning. In Advances in Neural Information Processing Systems, volume 34, pages 28496–28510, 2021.
  5. [5]Ian J Goodfellow, Mehdi Mirza, Da Xiao, Aaron Courville, and Yoshua Bengio. An empirical investigation of catastrophic forgetting in gradient-based neural networks. arXiv preprint arXiv:1312.6211, 2013.arxiv.org/abs/1312.6211
  6. [6]Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing Atari with deep reinforcement learning. NeurIPS Deep Learning Workshop, 2013.arxiv.org/abs/1312.5602
  7. [7]Petros Christodoulou. Soft actor-critic for discrete action settings. arXiv preprint arXiv:1910.07207, 2019.arxiv.org/abs/1910.07207
  8. [8]Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In International Conference on Learning Representations, 2016.arxiv.org/abs/1509.02971
  9. [9]Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 1861–1870. PMLR, 2018.arxiv.org/abs/1801.01290
  10. [10]Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine. Soft actor-critic algorithms and applications. arXiv preprint arXiv:1812.05905, 2018.arxiv.org/abs/1812.05905
  11. [11]Jason Yoo, Yunpeng Liu, Frank Wood, and Geoff Pleiss. Layerwise proximal replay: A proximal point method for online continual learning. In Proceedings of the 41st International Conference on Machine Learning, pages 57199–57216, 2024.arxiv.org/abs/2402.09542

Paper details

vlnc:W_y4nvf4ah
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents