betaFeedback

WorksComputer Science

rl_rob: A transparent, from-scratch Python library for teaching reinforcement learning from tabular methods to deep policy gradients

Abstract

rl_rob is an open-source Python library for teaching reinforcement learning. It implements the arc of a modern course, from tabular temporal-difference learning through linear function approximation to deep value-based and policy-gradient methods, with every algorithm written from first principles so that its update rule is visible in a few lines of code. All agents share one interface, so learners can compare methods directly across stages and trace a deep reinforcement learning update from the temporal-difference error to the weight change. The library supports teaching and assessment in which students implement, modify and explain update rules, rather than only produce working code.

Code metadata

Nr.Code metadata descriptionMetadata
C1Current code versionv1.0.0
C2Permanent link to code/repository used for this code versionhttps://github.com/AltahhanAi/rl_rob
C3Permanent link to Reproducible CapsuleNot applicable
C4Legal Code LicenseGNU Affero General Public License v3.0 (AGPL-3.0)
C5Code versioning system usedgit
C6Software code languages, tools, and services usedPython; NumPy, Matplotlib, PyTorch, Gymnasium, Jupyter; GitHub Actions; Google Colab
C7Compilation requirements, operating environments and dependenciesPython 3.9 or later on Linux, macOS or Windows, or Google Colab; numpy, matplotlib, ipython, tqdm, gymnasium, minigrid, torch, opencv-python-headless, scikit-learn; optional ROS 2 for the Gazebo robot environment
C8If available, link to developer documentation/manualhttps://github.com/AltahhanAi/rl_rob#readme
C9Support email for questions[email protected]

Table 1.

Motivation and significance

Reinforcement learning (RL) is now a core part of postgraduate curricula in artificial intelligence, yet it remains one of the hardest subjects to teach well. Its central ideas, bootstrapping, credit assignment over time and the interaction between exploration and learning, are easy to state and hard to see. Students meet them first as pseudocode in a textbook [1] and then, if they continue, as calls to production libraries such as Stable-Baselines3 [2] or CleanRL [3]. Between the two there is a gap. Production libraries are excellent for applying algorithms, but they hide the update rules behind optimisers, automatic differentiation and engineering layers that are not the subject of an introductory course. Educational resources such as Spinning Up [4] concentrate on deep RL and assume that the tabular and linear foundations are already in place.

In practice an instructor faces an uncomfortable choice. Students can be asked to implement every algorithm and its environment from scratch, which consumes much of a course in plumbing, or they can be given a framework in which the learning itself is invisible. Neither serves the aim of a course, which is for students to understand why an algorithm behaves as it does and when one method should be preferred to another.

rl_rob was written to close this gap. It covers the tabular, linear and neural stages of the standard progression [1] within one consistent interface, so that moving from Q-learning to a deep Q-network [5] changes one class name rather than the whole program. In the neural stage, PyTorch [6] supplies only the network layers and their gradients; the temporal-difference target, the replay buffer, eligibility traces and the policy-gradient updates are written out explicitly. A student can therefore trace a deep RL update from the temporal-difference error to the change in the weights, using the same vocabulary and code structure they met in the tabular stage.

The design also responds to a newer pressure on teaching. Where generative AI tools can produce working code for a standard algorithm on request, assessment that rewards working code alone no longer evidences understanding. Because rl_rob supplies the environments, experiment loop and visualisation, instructors can set tasks in which students implement or modify the core update of an algorithm, predict and explain its behaviour, and compare it with alternatives on the same problem. These tasks remain meaningful when code generation is freely available, because the object of assessment is the student's reasoning about the update rule.

Software description

Software architecture

rl_rob is a pure Python package of about seventy classes, organised in two layers: algorithms (package rl) and environments (package env). Its dependencies are standard scientific Python libraries; the tabular and linear stages use only NumPy [7] and Matplotlib [8].

The algorithm layer is built on a small hierarchy of base classes in rl.base. A prediction base (MRP) holds what every agent shares: hyperparameters, seeding, the episode loop, logging, and the plotting and animation harness. A control base (MDP) adds action selection and exploration, and a policy-gradient base (PG) adds parameterised policies. Control and policy-gradient behaviour are attached to each stage through class factories, so that the tabular, linear and neural stages reuse the same control logic and differ only in how values or policies are represented and updated. Each concrete algorithm is then a short class whose main content is its update rule. This is the property the library is built around: the code a student reads for Sarsa, for linear Sarsa(λ\lambda) or for a neural actor-critic is the update itself, and the surrounding machinery is identical across them.

The modules follow the stages of a course: rl.tabular and rl.dp for tabular and dynamic-programming methods; rl.linear for linear function approximation; rl.neural and rl.nn for neural function approximation, where rl.nn builds configurable convolutional trunks and fully connected heads; and rl.select, which manages repeated runs and comparisons between algorithms.

The environment layer provides gridworlds, mazes, random walks, windy and cliff-walking grids with tabular, linear and neural state encodings (env.grid); wrappers for Gymnasium control tasks [9] with additive discretisation and hashed multi-tiling tile coding (env.gym); and an optional ROS 2 interface for robot navigation in the Gazebo simulator (env.gazebo), which lets the same agents control a simulated mobile robot.

Software functionalities

The library implements the following methods, each reachable through the same constructor-and-interact pattern.

  • Tabular methods: Monte Carlo and TD(0) prediction, n-step methods, Sarsa, Q-learning, expected Sarsa, double Q-learning, Dyna-Q and prioritised sweeping, together with dynamic programming.

  • Linear function approximation: prediction and control with features, n-step Sarsa, eligibility traces including true online TD(λ\lambda) [10], REINFORCE and actor-critic with softmax and Gaussian policies, applied to Mountain Car, Acrobot, Pendulum and CartPole through discretisation and tile coding.

  • Neural function approximation: neural Sarsa and Q-learning, DQN, double and duelling DQN with experience replay and target networks, neural TD(λ\lambda) and Sarsa(λ\lambda), REINFORCE, n-step actor-critic and proximal policy optimisation (PPO) [11].

Every agent is trained by a single call to .interact(), which runs the experiment and returns learning curves, value maps and trajectory views. A configuration dictionary switches on a visual mode that renders the agent moving through its environment inside a Jupyter notebook as it learns, alongside steps-per-episode and return curves, so that students can watch a policy improve rather than read it only from a curve. The rl.select module repeats runs over seeds and compares algorithms or hyperparameters, which supports the classic experiments of the textbook, such as the error of TD(λ\lambda) as a function of step size and λ\lambda. The repository includes runnable scripts, notebooks that open directly in Google Colab, and automated tests that train a short run of each stage on every change through continuous integration.

Illustrative examples

Every algorithm is used in the same way: it is constructed with an environment and hyperparameters, and trained with .interact(). Tabular Sarsa on a gridworld is one line:

from rl.tabular import *
from env.grid.tabular import *

sarsa = Sarsa(env=grid(), α=.8, episodes=50, seed=10).interact()

The same pattern trains a deep Q-network with experience replay and a target network on a sparse-reward maze; replacing DQN with DDQN, nnSarsa or nnQlearn changes the algorithm and nothing else:

from rl.neural import *

dqn = DQN(env=imaze(reward='sparse'), α=1e-4, γ=.98, episodes=30,
          trunk=[(8, 4, 2), (4, 4, 4)], nbuffer=1000, nbatch=64).interact()

Linear control on a Gymnasium task uses tile coding, and the visual mode animates the car as it learns to climb the hill:

demo = {'plotT': True, 'plotR': True, 'visual': True, 'animate': True}
vsarsa = vSarsa(env=vGymDiscretes(**MountainCar), α=.1, ε=0,
                episodes=500, seed=1, **demo).interact()
Eligibility traces RMS error plots

Figure 2. Eligibility traces: TD(λ\lambda) RMS error against step size α\alpha for a range of λ\lambda (left) and true online TD(λ\lambda) (right), reproduced with the comparison utilities in rl.select.

Deep value-based control on a sparse-reward maze

Figure 3. Deep value-based control: DQN on a sparse-reward maze, steps and return per episode, trained through the same interface as the tabular agent in Figure 1.

Tabular Sarsa on a gridworld and steps per episode

Figure 1. Tabular Sarsa on a gridworld, produced directly by .interact(): the learned trajectory (left) and steps per episode (right).

Because the interface is shared, a single notebook can place tabular and neural Sarsa side by side on the same maze. Students see directly what function approximation changes, in speed of learning, stability and generalisation, without rewriting the experiment.

Impact

rl_rob took its present form in the Reinforcement Learning module of the online MSc in Artificial Intelligence at the University of Leeds, a module designed and taught by the author since 2021/22. Its students are mostly part-time working professionals studying online, for whom installation problems and opaque frameworks are a serious barrier. Over five years the library has been used by several hundred postgraduate students.

The module is built around a companion set of Jupyter worksheets that take students from one-step temporal-difference learning to PPO, each with explicit learning outcomes. In each worksheet, students implement or adapt the update rule of an algorithm and then study its behaviour with the library's experiment and plotting tools. The summative assessment mirrors the arc of the module: students take an agent from tabular to linear to non-linear function approximation, measure what each stage gains, meet the slow convergence and divergence they will face in practice, and compare algorithms to choose one that suits their environment. To remove installation barriers, the library also runs in a cloud laboratory and in Google Colab.

Two lessons from this use shaped the design. First, students engage more deeply with an algorithm when the code they write is the update rule itself and the rest of the experiment is handled for them. Second, a single interface across the three stages makes the effect of function approximation concrete, because methods can be compared on the same problem with the same tools. The approach to teaching RL embodied in the library was also the basis of invited tutorials on experience replay and deep RL at the International Joint Conference on Neural Networks, and the library has served as the starting point for doctoral research on RL.

Beyond its home module, rl_rob is released openly so that instructors and self-learners elsewhere can use it alongside the standard textbook [1]. It is suited to postgraduate and advanced undergraduate courses, to self-study, and to instructors who want one code base across a whole module rather than a different framework at each stage.

Conclusions

rl_rob provides a single, transparent code base for teaching reinforcement learning across its full arc, from tabular methods to deep policy gradients. By writing every update rule explicitly while supplying environments, experiments and visualisation, it lets students study the part of an algorithm that matters for understanding, and lets instructors set tasks that remain meaningful when code generation is freely available. Planned work includes fuller API documentation, further environments, and a structured evaluation of the library's effect on student learning.

Declaration of competing interest

The author declares that he has no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Declaration of generative AI and AI-assisted technologies in the writing process

During the preparation of this work the author used Claude (Anthropic) to assist with drafting and editing the manuscript text. The author reviewed and edited the output as needed and takes full responsibility for the content of the published article.

Acknowledgements

The author thanks the students of the Reinforcement Learning module of the online MSc in Artificial Intelligence at the University of Leeds, whose questions and feedback shaped the library.

References

  1. [1]R.S. Sutton, A.G. Barto, Reinforcement Learning: An Introduction, second ed., MIT Press, Cambridge, MA, 2018.
  2. [2]A. Raffin, A. Hill, A. Gleave, A. Kanervisto, M. Ernestus, N. Dormann, Stable-Baselines3: Reliable reinforcement learning implementations, J. Mach. Learn. Res. 22 (268) (2021) 1–8.
  3. [3]S. Huang, R.F.J. Dossa, C. Ye, J. Braga, D. Chakraborty, K. Mehta, J.G.M. Araújo, CleanRL: High-quality single-file implementations of deep reinforcement learning algorithms, J. Mach. Learn. Res. 23 (274) (2022) 1–18.arxiv.org/abs/2111.08819
  4. [4]J. Achiam, Spinning Up in Deep Reinforcement Learning, OpenAI, 2018. https://spinningup.openai.com
  5. [5]V. Mnih, K. Kavukcuoglu, D. Silver, et al., Human-level control through deep reinforcement learning, Nature 518 (7540) (2015) 529–533.DOI
  6. [6]A. Paszke, S. Gross, F. Massa, et al., PyTorch: An imperative style, high-performance deep learning library, in: Advances in Neural Information Processing Systems 32, 2019, pp. 8024–8035.arxiv.org/abs/1912.01703
  7. [7]C.R. Harris, K.J. Millman, S.J. van der Walt, et al., Array programming with NumPy, Nature 585 (7825) (2020) 357–362.
  8. [8]J.D. Hunter, Matplotlib: A 2D graphics environment, Comput. Sci. Eng. 9 (3) (2007) 90–95.DOI
  9. [9]M. Towers, K. Wiatkowski, J. Terry, et al., Gymnasium: A standard interface for reinforcement learning environments, arXiv preprint arXiv:2407.17032, 2024.arxiv.org/abs/2407.17032
  10. [10]H. van Seijen, R.S. Sutton, True online TD(λ), in: Proceedings of the 31st International Conference on Machine Learning, PMLR 32, 2014, pp. 692–700.
  11. [11]J. Schulman, F. Wolski, P. Dhariwal, A. Radford, O. Klimov, Proximal policy optimization algorithms, arXiv preprint arXiv:1707.06347, 2017.arxiv.org/abs/1707.06347

Paper details

vlnc:W_n462dyhs
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 6

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Claude (Anthropic)

View statement
Looking for related papers…
Contents