betaFeedback

WorksComputer Science

Delayed Heuristic-Behavior-Target Coupling for Off-Policy Reinforcement Learning in Inventory Control

Abstract

Many reinforcement learning methods tie data collection to the task policy, while coordinating independently optimized behavior and target policies remains challenging. Policies learned by these methods can perform unreliably in long-horizon tasks under uncertainty, such as inventory control, where failures can incur extreme costs. We introduce Heuristic Behavior Policy Optimization (HBPO), an off-policy framework that routes heuristic guidance through persistent behavior learning, coupled with target-policy, updating separately with residual-based advantage estimation and emphatic weighting. Under matched environment interaction budgets on DSIS and its risk-augmented variant DSIS-RB, HBPO recovers many high-cost initial policies and improves most non-collapse comparison groups. Cold starts achieve competitive costs without final cost collapse across all tested seeds. We identify a transmission chain in the delayed heuristic-behavior-target coupling as a key driver of recovery and improvement. Generation-level diagnostics trace heuristic uptake and subsequent selective target responses, while component ablations support the contribution of this transmission mechanism to policy recovery and improvement.

Introduction

Learning a policy for long-horizon planning requires choosing how to collect the experience used to improve it [41]. In inventory control, an order changes the pipeline of future deliveries, and its consequences depend on subsequent demand [4]. Decisions therefore shape not only future costs but also the states encountered during learning. This makes the behavior policy that collects data an important intervention point: if the behavior policy is itself learned, what should it be optimized to do for the learning process as a whole? Concretely, can heuristic guidance applied to it and behavior policy persistently influence the target policy, which is the policy finally want to optimize, to achieve a better performance?

Existing designs give reason to doubt that this mechanism happens in most RL frameworks. Importantly, failures are observed in long-horizon planning tasks. In a studied multi-echelon inventory network, PPO converged consistently only after inventory transitions were constrained during training [10], and our comparisons show high-cost runs for PPO, Discrete SAC, and adapted GPS (§ 5). These methods share a default: experience is collected by the task policy or a recent snapshot of it [34, 15]. Off-policy learning permits a distinct behavior policy [7], but leaves open how that policy should improve and how its deviations should inform target updates. Behavior–target mismatch can impair approximate off-policy updates, and fitting a policy to a teacher’s actions need not preserve the teacher’s long-horizon performance. Our focus is therefore to construct explicit coupling updating rules for the transmission path itself: which behavioral deviations should a separately updated target reinforce or counteract, and when should planning-derived guidance enter its updates?

We introduce Heuristic Behavior Policy Optimization (HBPO), an off-policy coupled framework that routes heuristic guidance through a persistent behavior policy to a separately and selectively optimized target policy. HBPO forms an explicit delayed heuristic–behavior–target transmission: heuristic guidance, constructed from the current generation target policy, first shapes the successor behavior policy then reaches two-next target updates through four combined optimizations. This transmission is based on data collected by behavior, residual-based advantage estimation supporting both policies and emphatic weighting carrying heuristic confidence forward. Under same interaction budgets on DSIS and DSIS-RB, HBPO recovers many high-cost initial policies, lowers mean costs across most initialization sources, and improves most non-collapse comparison groups. Cold starts achieve competitive costs without cost collapse across all tested seeds. We identify the transmission of delayed heuristic–behavior–target coupling as a key driver of recovery and improvement. An anonymized implementation is included in the supplementary material and will be made publicly available upon acceptance.

Related Work

Behavior learning and selective transfer. Off-policy learning permits the collector to differ from the target [7]. On-policy methods remove the difference by collecting only from the current policy (policy-gradient, natural-gradient, trust-region, and asynchronous families) [42, 37, 33, 20, 32, 34, 27], so the data source cannot outgrow the target. Common algorithms derive their evolving behavior distributions from learned value functions or actors and exploration rules [26, 39, 31, 24, 9, 15]. Correction schemes persistently track the behavior (truncated importance sampling, Retrace, V-trace) reconcile it with target updates without shaping it [40, 29, 8, 14], so correction grows with the gap. Heuristic shaping (optimism, count-based bonuses, curiosity, learned novelty policies, variance-optimized behavior) can improve data collection, but these gains need not translate into consistent target-policy improvement [2, 3, 30, 5, 41, 43, 25, 12]. HBPO couples three in one loop: a temporary heuristic controller guides a persistent behavior learner, and behavior data plus an advantage-filtered transfer term weighting signed target-advantage scores by absolute behavior–target disagreement feed target updates. ACER uses a one-sided residual to correct importance-weight truncation bias [40]; HBPO uses absolute behavior–target disagreement to weight an additional advantage-based transfer objective. Our experiments support the contribution of this heuristic–behavior–target coupling (Section 5).

Coupled updates and transmission. With the heuristic, behavior, and target roles fixed, the key question is how improvement travels through their coupled updates. GPS fits local controllers directly into the global policy [22]; conservative and regularized policy iteration fit an improved action distribution into the actor [19, 28, 1]; TRPO, PPO, and GRPO make each improved policy the immediate data collector [32, 34, 35]; and TPO fits the policy to a softmax-improved target distribution [18]. HBPO instead claims a delayed coupling: information and policy impacts enter the next generation separately, rather than the current one, making the transmission πg→qH,g→μg+1→πg+2\pi_g \to q_{H,g} \to\mu_{g+1} \to\pi_{g+2} explicit; § 5 examines how this transmission operates during recovery and improvement.

Emphatic state weighting. Emphatic temporal-difference and actor–critic methods relate behavior sampling to the state weighting of an off-policy learning objective [38, 16, 13, 44], and expected emphatic traces and adaptive interest extend this framework to learned state emphasis [17, 21]. HBPO adapts this perspective through separate learned masses for target-return learning and behavior transfer, with heuristic-improvement confidence supplying interest for the transfer mass.

Inventory control as a testbed. Deep RL has proven effective across lost-sales, dual-sourcing, and multi-echelon inventory problems [11], though training conditions materially affect learning [10]. Böttcher et al. [4] optimize dual-sourcing policies by embedding inventory dynamics in recurrent neural networks. We adapt their problem formulation for our testbed. We choose inventory control as a testbed because poor decisions carry severe downside risk, and the systems admit a compact Markov formulation whose known or learnable dynamics support planning-derived guidance. § 3 defines our inventory control problem.

Problem Settings

We adapt the single-item dual-sourcing model of Böttcher et al. [4] into DSIS, with regular deliveries after four periods and costlier, immediate expedited deliveries. DSIS has IID demand and uninterrupted supply; our extension, DSIS-RB, adds persistent demand and supply disruptions.

We formulate both benchmarks as underlying Markov decision processes (X,A,P,r,γ)(\mathcal{X},\mathcal{A},\mathcal{P},r,\gamma). At period tt, DSIS observations st=(It,pt,Bt)s_t = (I_t,p_t,B_t) contain net inventory, a four-order pipeline, and current supply availability. DSIS-RB adds previous demand dt−1d_{t-1}, increasing the dimension from six to seven. Actions at=(qtr,qte)∈{0,…,8}2a_t = (q_t^r,q_t^e) \in\{0,\ldots,8\}^2 specify regular and expedited orders. DSIS-RB training pools trajectories across demand disruption-risk levels ρ\rho, fixed within each trajectory and withheld from all learners and simulation workers. All learned networks in HBPO use only the current observation sts_t, with fixed preprocessing. The full environment state is xt=(st,ρ)x_t = (s_t,\rho) for DSIS-RB and xt=stx_t = s_t for DSIS. Risk variables and their averaging are omitted for DSIS throughout. The environment kernel P\mathcal{P} governs order acceptance, arrivals, demand, supply availability, and inventory carryover through xt+1∼P(⋅∣xt,at)x_{t+1} \sim\mathcal{P}(\cdot\mid x_t,a_t) (Appendix C). Period cost ctc_t comprises expediting, holding, and backlogging. We maximize expected discounted return using rt=−ct/495r_t = -c_t/495 and γ=0.995\gamma= 0.995, where 495 is the per-unit, per-period backlog penalty. All shared information access and real-interaction budgets. DSIS dynamics are public; HBPO and GPS construct DSIS-RB models from shared public simulation specification P^g\hat{\mathcal{P}}_g and their observed data, without access to true stochastic parameters (Appendix C). Delayed deliveries, accumulated backlogs, and severe shortage penalties make these tasks a testbed for long-horizon decision-making under uncertainty, where failures can incur extreme costs.

Heuristic Behavior Policy Optimization Methods

Figure 1. HBPO Learning Framework. Red highlights the delayed heuristic transmission.

HBPO maintains two separately parameterized, persistent policies, μg=μgθμ\mu_g = \mu_g^{\theta_\mu} and πg=πgθπ\pi_g = \pi_g^{\theta_\pi}. The behavior policy μg\mu_g collects interaction data Dg\mathcal{D}_g, while the target policy πg\pi_g is evaluated for task performance and defines the continuation policy for model evaluation. Under the simulation specification P^g\hat{\mathcal{P}}_g, frozen within generation gg, long-horizon comparisons of alternative first actions followed by πg\pi_g construct a temporary heuristic policy qH,gq_{H,g}. A shared advantage face scores policy changes and behavior–target discrepancies using discounted target-policy advantages, defining the A,J,PA,J,P objectives. Together with heuristic guidance HH, these yield four gradients: behavior combines target-advantage learning gAg_A with heuristic guidance gHg_H, while target combines its own advantage learning gJg_J with selective transfer from behavior gPg_P.

The frozen reference vg(s)=15∑j=15Vg,j(s)v_g(s) = \frac{1}{5}\sum_{j=1}^{5}V_{g,j}(s), averaging five separately parameterized scalar critics, supports heuristic evaluation and residual-completed estimation of the discounted model action value Qgπg(s,a)Q_g^{\pi_g}(s,a); successor critics are fitted by minimizing squared TD error. Positive masses MgχM_g^\chi provide mean-one emphatic weights wgχ(s)w_g^\chi(s), χ∈{J,P}\chi\in\{J,P\}, for the state-weighted parts of learning; their construction and the critic updates are specified in Section 4.3.

Heuristic Objectives Guidance

To construct a temporary heuristic policy from πg\pi_g, anchor states are sampled from Dg\mathcal{D}_g, with each anchor state ss defining local time k=0k=0. We compare the long-horizon consequences of candidate first actions aa over T=1000T=1000 periods, with all subsequent actions drawn from the frozen parent target πg\pi_g. Let SkS_k, AkA_k, and RkR_k denote the observation, action, and normalized reward at local decision stage kk. Writing EH\mathbb{E}_{\mathrm{H}} for expectation under P^g\widehat{P}_g, including the synthetic-risk draw for each DSIS-RB tape, we define the expected return GT,g(s,a)G_{T,g}(s,a) and the centered estimated utility ug(s,a)u_g(s,a) as

GT,g(s,a)=EH[∑k=0T−1Rk∣S0=s, A0=aAk∼πg(⋅∣Sk) (1≤k<T)],ug(s,a)=G^T,g(s,a)−∑a′∈Aπg(a′∣s)G^T,g(s,a′).G_{T,g}(s,a)=\mathbb{E}_{\mathrm{H}}\left[\left.\sum_{k=0}^{T-1}R_k\right|_{\substack{S_0=s,\ A_0=a\\ A_k\sim\pi_g(\cdot\mid S_k)\ (1\leq k<T)}}\right],\qquad u_g(s,a)=\widehat{G}_{T,g}(s,a)-\sum_{a'\in\mathcal{A}}\pi_g(a'\mid s)\widehat{G}_{T,g}(s,a').

Here, G^T,g\widehat{G}_{T,g} is a Monte Carlo estimate of GT,gG_{T,g}, constructed using the frozen critic vgv_g as a control variate and 32 simulation tapes shared across the candidate first-action branches at each distinct query observation. All queries are performed within the model, requiring no additional environment interactions. Appendices § 11.1.3 and § 11.§ 4.3 detail the simulation specification and the utility estimator.

For a candidate action distribution qq at state ss, ⟨q,ug(s,⋅)⟩=∑a(qa−πg(a∣s))G^T,g(s,a)\langle q,u_g(s,\cdot)\rangle=\sum_a(q_a-\pi_g(a\mid s))\widehat{G}_{T,g}(s,a) measures the estimated return gain from replacing the πg(⋅∣s)\pi_g(\cdot\mid s) with qq and following πg\pi_g thereafter. Maximizing this gain alone would concentrate probability on utility-maximizing actions. We therefore construct a temporary stochastic policy for heuristic guidance through Hellinger regularization

qH,g(⋅∣s)=arg⁡max⁡q∈Δ(A){⟨q,ug(s,⋅)⟩−2τ∑a(qa−πg(a∣s))2}.(1)q_{H,g}(\cdot\mid s)=\underset{q\in\Delta(\mathcal{A})}{\arg\max}\left\{\langle q,u_g(s,\cdot)\rangle-2\tau\sum_a\left(\sqrt{q_a}-\sqrt{\pi_g(a\mid s)}\right)^2\right\}. \tag*{(1)}

The utility term favors higher estimated returns, while the symmetric, bounded squared Hellinger penalty moderates departures from the target prior to help stabilize heuristic guidance. The coefficient τ>0\tau>0 controls this trade-off and remains fixed throughout each run after calibration under the shared hyperparameter-selection protocol HPO. With a strictly positive target prior, scalar root finding yields the temporary guidance policy at each queried state. The resulting policy has no trainable actor parameters. Appendix § 11.4.7 details the calibration, controller solution, and derivation; Appendix § 12 reports the hyperparameter-selection protocol.

Denote νgJ\nu_g^J as the empirical distribution of behavior states weighted by wgJw_g^J, introduced in § 4.3. Starting from anchors drawn from νgJ\nu_g^J, short model rollouts under qH,gq_{H,g} generate the empirical training distribution ξg\xi_g by retaining states at stages 0,…,80,\ldots,8. Each retained state receives a controller query with a full T=1000T=1000 horizon, so the short rollouts select training locations without shortening the utility horizon. Using qH,g(⋅∣s)q_{H,g}(\cdot\mid s) as the fitting targets at each retained state, define the behavior-guidance gradient by

References

  1. [1]Abbas Abdolmaleki, Jost Tobias Springenberg, Yuval Tassa, Remi Munos, Nicolas Heess, and Martin Riedmiller. Maximum a posteriori policy optimization. In International Conference on Learning Representations, 2018. URL https://openreview.net/forum?id=S1ANxQW0b.arxiv.org/abs/1806.06920
  2. [2]Peter Auer, Nicolò Cesa-Bianchi, and Paul Fischer. Finite-time analysis of the multiarmed bandit problem. Machine Learning, 47(2–3):235–256, 2002. doi: 10.1023/A:1013689704352. URL https://doi.org/10.1023/A:1013689704352.
  3. [3]Marc G. Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. In Advances in Neural Information Processing Systems, volume 29, 2016. URL https://arxiv.org/abs/1606.01868
  4. [4]Lucas Böttcher, Thomas Asikis, and Ioannis Fragkos. Control of dual-sourcing inventory systems using recurrent neural networks. INFORMS Journal on Computing, 35(6):1308–1328, 2023. doi: 10.1287/ijoc.2022.0136. URL https://pubsonline.informs.org/doi/10.1287/ijoc.2022.0136.DOI
  5. [5]Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. In International Conference on Learning Representations, 2019. URL https://arxiv.org/abs/1810.12894.
  6. [6]Petros Christodoulou. Soft actor-critic for discrete action settings. arXiv preprint arXiv:1910.07207, 2019. URL https://arxiv.org/abs/1910.07207.
  7. [7]Thomas Degris, Martha White, and Richard S. Sutton. Off-policy actor-critic. In Proceedings of the 29th International Conference on Machine Learning, 2012. URL https://arxiv.org/abs/1205.4839.
  8. [8]Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Volodymyr Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, Iain Dunning, Shane Legg, and Koray Kavukcuoglu. IMPALA: Scalable distributed deep-RL with importance weighted actor-learner architectures. In Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 1407–1416. PMLR, 2018. URL https://proceedings. mlr.press/v80/espeholt18a.html.
  9. [9]Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 1587–1596. PMLR, 2018. URL https://proceedings.mlr.press/v80/fujimoto18a.html.
  10. [10]Kevin Geevers, Lotte van Hezewijk, and Martijn R. K. Mes. Multi-echelon inventory optimization using deep reinforcement learning. Central European Journal of Operations Research, 32:653–683, 2024. doi: 10.1007/s10100-023-00872-2. URL https://link.springer.com/article/10.1007/s10100-023-00872-2.DOI
  11. [11]Joren Gijsbrechts, Robert N. Boute, Jan A. Van Mieghem, and Dennis J. Zhang. Can deep reinforcement learning improve inventory management? Performance on lost sales, dual-sourcing, and multi-echelon problems. Manufacturing & Service Operations Management, 24(3):1349–1368, 2022. doi: 10.1287/msom.2021.1064. URL https://pubsonline.informs.org/doi/10.1287/msom.2021.1064.DOI
  12. [12]Alex W. Goodall, Edwin Hamel-De Le Court, and Francesco Belardinelli. Behaviour policy optimization: Provably lower variance return estimates for off-policy reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, pp. 21324–21332, 2026. doi: 10.1609/aaai.v40i26.39278. URL https://ojs.aaai.org/index.php/AAAI/article/view/39278.DOI
  13. [13]Eric Graves, Ehsan Imani, Raksha Kumaraswamy, and Martha White. Off-policy actor-critic with emphatic weightings. Journal of Machine Learning Research, 24(146):1–63, 2023. URL https://jmlr.org/papers/v24/21-1350.html.
  14. [14]Audrunas Gruslys, Will Dabney, Mohammad Gheshlaghi Azar, Bilal Piot, Marc Bellemare, and Remi Munos. The reactor: A fast and sample-efficient actor-critic agent for reinforcement learning. In International Conference on Learning Representations, 2018. URL https://arxiv.org/abs/1704.04651.
  15. [15]Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 1861–1870. PMLR, 2018. URL https://proceedings.mlr.press/v80/haarnoja18b.html.arxiv.org/abs/1801.01290
  16. [16]Ehsan Imani, Eric Graves, and Martha White. An off-policy policy gradient theorem using emphatic weightings. In Advances in Neural Information Processing Systems, volume 31, 2018. URL https://papers.neurips.cc/paper_files/paper/2018/hash/3ef815416f775098fe977004015c6193-Abstract.html.arxiv.org/abs/1811.09013
  17. [17]Ray Jiang, Shangtong Zhang, Veronica Chelu, Adam White, and Hado van Hasselt. Learning expected emphatic traces for deep RL. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 7015–7023, 2022. doi: 10.1609/aaai.v36i6.20660. URL https://ojs.aaai.org/index.php/AAAI/article/view/20660.DOI
  18. [18]Jean Kaddour. Target policy optimization. arXiv preprint arXiv:2604.06159, 2026. URL https://arxiv.org/abs/2604.06159v1.
  19. [19]Sham Kakade and John Langford. Approximately optimal approximate reinforcement learning. In Proceedings of the 19th International Conference on Machine Learning, pp. 267–274, 2002. URL https://people.eecs.berkeley.edu/~pabbeel/cs287-fa09/readings/KakadeLangford-icml2002.pdf.
  20. [20]Sham M. Kakade. A natural policy gradient. In Advances in Neural Information Processing Systems, volume 14, 2001. URL https://proceedings.neurips.cc/paper/2001/hash/4b86abe48d358ecf194c56c69108433e-Abstract.html.
  21. [21]Martin Klissarov, Rasool Fakoor, Jonas W Mueller, Kavosh Asadi, Taesup Kim, and Alexander Smola. Adaptive interest for empathic reinforcement learning. In Advances in Neural Information Processing Systems, volume 35, pp. 95–108. Curran Associates, Inc., 2022. doi: 10.52202/068431-0008. URL https://proceedings.neurips.cc/paper_files/paper/2022/file/008079ec00eec9760ee93af5434e932-Paper_Conference.pdf.DOI
  22. [22]Sergey Levine and Vladlen Koltun. Guided policy search. In Proceedings of the 30th International Conference on Machine Learning, volume 28 of Proceedings of Machine Learning Research, pp. 1–9. PMLR, 2013. URL https://proceedings.mlr.press/v28/levine13.html.
  23. [23]Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuomotor policies. Journal of Machine Learning Research, 17(39):1–40, 2016. URL https://jmlr.org/papers/v17/15-522.html.arxiv.org/abs/1504.00702
  24. [24]Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In International Conference on Learning Representations, 2016. URL https://arxiv.org/abs/1509.02971.
  25. [25]Anji Liu, Yitao Liang, and Guy Van den Broeck. Off-policy deep reinforcement learning with analogous disentangled exploration. In Proceedings of the 19th International Conference on Autonomous Agents and Multi-Agent Systems, 2020. URL https://starai.cs.ucla.edu/papers/LiuAAMAS20.pdf.arxiv.org/abs/2002.10738
  26. [26]Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrej A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 518(7540):529–533, 2015. doi: 10.1038/nature14236. URL https://www.nature.com/articles/nature14236.DOI
  27. [27]Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In Proceedings of the 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pp. 1928–1937. PMLR, 2016. URL https://proceedings.mlr.press/v48/mniha16.html.arxiv.org/abs/1602.01783
  28. [28]William H. Montgomery and Sergey Levine. Guided policy search via approximate mirror descent. In Advances in Neural Information Processing Systems, volume 29, 2016. URL https://papers.neurips.cc/paper_files/paper/2016/hash/a00e5eb0973d24649a4a920fc53d9564-Abstract.html.
  29. [29]Rémi Munos, Tom Stepleton, Anna Harutyunyan, and Marc G. Bellemare. Safe and efficient off-policy reinforcement learning. In Advances in Neural Information Processing Systems, volume 29, 2016. URL https://proceedings.neurips.cc/paper/2016/file/c392e9a68c5ae12bd18488bc579b30d-Paper.pdf.arxiv.org/abs/1606.02647
  30. [30]Deepak Pathak, Pulkit Agrawal, Alexei A. Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 2778–2787. PMLR, 2017. URL https://proceedings.mlr.press/v70/pathak17a.html.arxiv.org/abs/1705.05363
  31. [31]Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay. In International Conference on Learning Representations, 2016. URL https://arxiv.org/abs/1511.05952.
  32. [32]John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pp. 1889–1897. PMLR, 2015. URL https://proceedings.mlr.press/v37/schulman15.html.arxiv.org/abs/1502.05477
  33. [33]John Schulman, Philipp Moritz, Sergey Levine, Michael I. Jordan, and Pieter Abbeel. High-dimensional continuous control using generalized advantage estimation. In International Conference on Learning Representations, 2016. URL https://arxiv.org/abs/1506.02438.
  34. [34]John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. URL https://arxiv.org/abs/1707.06347.
  35. [35]Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. DeepSeekMath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024. URL https://arxiv.org/abs/2402.03300.
  36. [36]Richard S. Sutton. Dyna, an integrated architecture for learning, planning, and reacting. ACM SIGART Bulletin, 2(4):160–163, 1991. doi: 10.1145/122344.122377. URL https://doi.org/10.1145/122344.122377.
  37. [37]Richard S. Sutton, David A. McAllester, Satinder P. Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. In Advances in Neural Information Processing Systems, volume 12, 1999. URL https://proceedings.neurips.cc/paper/1999/hash/464d828b85b0bed98e80ade0a5c43b0f-Abstract.html.
  38. [38]Richard S. Sutton, A. Rupam Mahmood, and Martha White. An emphatic approach to the problem of off-policy temporal-difference learning. Journal of Machine Learning Research, 17(73):1–29, 2016. URL https://jmlr.org/papers/v17/14-488.html.arxiv.org/abs/1503.04269
  39. [39]Hado van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q-learning. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, volume 30, pp. 2094–2100, 2016. doi: 10.1609/aaai.v30i1.10295. URL https://ojs.aaai.org/index.php/AAAI/article/view/10295.DOI
  40. [40]Ziyu Wang, Victor Bapst, Nicolas Heess, Volodymyr Mnih, Remi Munos, Koray Kavukcuoglu, and Nando de Freitas. Sample efficient actor-critic with experience replay. In International Conference on Learning Representations, 2017. URL https://openreview.net/forum?id=HyM25Mqel.arxiv.org/abs/1611.01224
  41. [41]William F. Whitney, Michael Bloesch, Jost Tobias Springenberg, Abbas Abdolmaleki, Kyunghyun Cho, and Martin Riedmiller. Decoupled exploration and exploitation policies for sample-efficient reinforcement learning. arXiv preprint arXiv:2101.09458, 2021. URL https://arxiv.org/abs/2101.09458.
  42. [42]Ronald J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning, 8(3–4):229–256, 1992. doi: 10.1007/BF00992696. URL https://doi.org/10.1007/BF00992696.
  43. [43]Tianbing Xu, Qiang Liu, Liang Zhao, and Jian Peng. Learning to explore via meta-policy gradient. In Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 5463–5472. PMLR, 2018. URL https://proceedings.mlr.press/v80/xu18d.html.
  44. [44]Shangtong Zhang, Wendelin Boehmer, and Shimon Whiteson. Generalized off-policy actor-critic. In Advances in Neural Information Processing Systems, volume 32, 2019. URL https://proceedings.neurips.cc/paper/2019/hash/0e095e054ee94774d6a496099eb1cf6a-Abstract.html.arxiv.org/abs/1903.11329

Paper details

vlnc:W_hbhsmcv8
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents