betaFeedback

WorksComputer Science

Heteroscedastic Gaussian Process Q-learning for Sample Efficient Experimentation

Abstract

Self-driving laboratories must learn to control a process from tens to hundreds of expensive experiments, too few for deep reinforcement learning (RL). We learn the Q-function directly with a Gaussian process (GP), which avoids learning the dynamics, and show that the GP posterior supplies the learning rate of Q-learning. If each Bellman target is treated as a noisy measurement of the Q-function, the learning rate that minimises the mean squared error is the Kalman gain, and applying it to every observed transition is equivalent to GP regression. Heteroscedastic Gaussian Process Q-learning (HGPQ) sets the noise variance of each target with a heteroscedastic GP, which learns how unreliable the targets are across the states and inputs. Targets from noisy operating conditions, or from regions where the value function has large gradients, then move the estimate less than targets from low-noise regions, so HGPQ learns quickly where the data are reliable without being misled where they are not. For kernel value iteration with per-target variances that are assigned when a target is collected, we prove a high-probability bound on the error of the value estimate, uniform over the explored region. Pooled over five simulated benchmarks, which include a chemical reactor and a bioprocess, HGPQ approaches the best return found by any method in fewer episodes than GP, kernel and deep RL baselines.

Introduction

Self-driving laboratories run experimental campaigns in chemistry and biology with automated equipment and a learning algorithm that decides what to do next [2]. Each episode is a real experiment, in which the controller measures the state of a reactor or culture, sets inputs such as a feed rate, and is often rewarded only once the batch is complete [39, 10]. Each campaign is therefore a finite-horizon Markov decision process (MDP) with continuous states and inputs and unknown dynamics. The controller must learn from tens to hundreds of episodes, too few to also tune the hyperparameters of the learner.

Reinforcement learning (RL) with Gaussian processes (GPs) usually learns a model of the dynamics and plans through it [16, 10, 6], but model errors compound over the horizon [30, 44], especially when the reward arrives only at the end. We instead place a GP prior [46] on the QQ-function [55], which the controller maximises directly to choose its inputs. Similarly, Bayesian optimisation models the objective of an experiment rather than the process behind it [48].

Q-learning moves its estimate towards each new Bellman target by a learning rate, which must be tuned for each problem [23]. The noise of a laboratory process is rarely known in advance and changes between instruments, batches of material and operating conditions [4, 43, 25]. Kalman temporal difference (TD) methods set the rate to a Kalman gain by treating each target as a noisy measurement of the Q-function, but tune the noise variance of the targets in place of the rate [23, 22, 52], and existing GP Q-learning methods tune the rate itself [7, 13]. Variance-aware value iteration instead weights each target by the inverse of its variance, which in theory needs fewer samples [40] (Yin et al. 2022). Heteroscedastic Gaussian Process Q-learning (HGPQ) makes this practical. It estimates the variance of each target with a heteroscedastic GP and uses it in the Kalman gain, computed as GP regression. Our contributions are:

  • Method. HGPQ, a GP Q-learning method whose learning rate is the Kalman gain and whose target variances are estimated by a heteroscedastic GP, so that no learning rate is tuned.

  • Theory. A high-probability bound on the error of the posterior mean, uniform over the explored region, for per-target variances that are assigned when a target is collected.

  • Results. Pooled over five benchmarks, HGPQ approaches the best return found by any method in fewer episodes than GP, kernel and deep RL baselines with equal or larger budgets.

Related work. The bias-adjusted Kalman filter (BAKF) and the optimal stepsize algorithm for approximate value iteration (OSAVI) [23, 47], GP temporal differences (GPTD) and GP SARSA (GPSARSA) [18, 19] and Kalman temporal differences (KTD) for Q-learning [22] set the TD learning rate from a single noise variance. Kalman filter Q-learning [52], assumed density filtering Q-learning [31] and deep ensembles [36] give each target its own noise or weight, but from the epistemic uncertainty of the successor’s value. HGPQ instead follows variance-aware value iteration [40] (Yin et al. 2022). This weights each target by the inverse of an estimated variance of the next value plus a floor, and it has sharper regret and sample complexity than unweighted regression [28, 3] (Zhou et al. 2021). HGPQ estimates these variances nonparametrically with a heteroscedastic GP [24, 34]. Like optimistic kernel value iteration [32, 14] (Yang et al. 2020), whose regret is order-optimal for Matérn kernels [53], HGPQ fits a regularised estimate per time step [33]. Like fitted Q-iteration [20], it regresses Bellman targets on stored transitions, and unlike GP dynamic programming [15], which models both the value function and the dynamics with GPs, it needs no model of the dynamics. We bound its error uniformly over the explored region of the state-input space.

Problem setting and Gaussian processes

Finite-horizon MDPs. At time steps t=0,…,T−1t = 0, \ldots, T - 1 the controller observes a state xt\mathbf{x}_t in a compact set X⊂Rdx\mathcal{X} \subset\mathbb{R}^{d_x} and applies an input ut\mathbf{u}_t from a box U⊂Rdu\mathcal{U} \subset\mathbb{R}^{d_u}. We write z=(x,u)\mathbf{z} = (\mathbf{x}, \mathbf{u}), Z:=X×U\mathcal{Z} := \mathcal{X} \times\mathcal{U} and d:=dx+dud := d_x + d_u. The pair zt\mathbf{z}_t produces a random stage reward rt∈[rmin⁡,rmax⁡]r_t \in[r_{\min}, r_{\max}] with mean rˉt(z):=E[rt∣zt=z]\bar{r}_t(\mathbf{z}) := \mathbb{E}[r_t \mid\mathbf{z}_t = \mathbf{z}] and a random next state xt+1∼Pt(⋅∣zt)\mathbf{x}_{t+1} \sim P_t(\cdot\mid\mathbf{z}_t), with PtP_t the transition kernel, and the initial state is drawn from a density p0p_0. The final state xT\mathbf{x}_T produces a random terminal reward rT∈[rmin⁡T,rmax⁡T]r_T \in[r_{\min}^T, r_{\max}^T] with mean rˉT(x):=E[rT∣xT=x]\bar{r}_T(\mathbf{x}) := \mathbb{E}[r_T \mid\mathbf{x}_T = \mathbf{x}], such as the yield of a batch. The goal is a policy π=(π0,…,πT−1)\pi= (\pi_0, \ldots, \pi_{T-1}) that maximises the expected return Eπ[∑t=0T−1rt+rT]\mathbb{E}_{\pi}\left[\sum_{t=0}^{T-1} r_t + r_T\right]. The optimal Q-function Qt∗(z)Q_t^*(\mathbf{z}), the expected return of applying u\mathbf{u} in state x\mathbf{x} at time step tt and acting optimally afterwards, satisfies the Bellman equation, solved backwards from the terminal value VT∗:=rˉTV_T^* := \bar{r}_T,

Qt∗(z)=rˉt(z)+Ex′∼Pt(⋅∣z)[Vt+1∗(x′)],Vt∗(x):=max⁡u∈UQt∗(x,u),t=T−1,…,0(1)Q_t^*(\mathbf{z}) = \bar{r}_t(\mathbf{z}) + \mathbb{E}_{\mathbf{x}' \sim P_t(\cdot\mid\mathbf{z})}\left[V_{t+1}^*(\mathbf{x}')\right], \qquad V_t^*(\mathbf{x}) := \max_{\mathbf{u} \in\mathcal{U}} Q_t^*(\mathbf{x}, \mathbf{u}), \qquad t = T-1, \ldots, 0 \tag*{(1)}

with rˉT≡0\bar{r}_T \equiv0 without a terminal reward. The greedy policy πt∗(x)∈arg⁡max⁡uQt∗(x,u)\pi_t^*(\mathbf{x}) \in\arg\max_{\mathbf{u}} Q_t^*(\mathbf{x}, \mathbf{u}) is optimal, so HGPQ learns Q0∗,…,QT−1∗Q_0^*, \ldots, Q_{T-1}^* from data and needs no model of the dynamics.

Gaussian process regression. Under a GP prior f∼GP(0,κ)f \sim\mathcal{GP}(0, \kappa) on f:Z→Rf : \mathcal{Z} \to\mathbb{R} (Rasmussen and Williams, [46]), given observations oi=f(zi)+ηio_i = f(\mathbf{z}_i) + \eta_i, i=1,…,ni = 1, \ldots, n, with independent zero-mean noise of variance λi\lambda_i, write o:=(o1,…,on)T\mathbf{o} := (o_1, \ldots, o_n)^{\mathsf{T}}, Λ:=diag⁡(λ1,…,λn)\Lambda:= \operatorname{diag}(\lambda_1, \ldots, \lambda_n), K:=[κ(zi,zj)]i,j≤nK := [\kappa(\mathbf{z}_i,\mathbf{z}_j)]_{i,j \le n} and κn(z):=(κ(z1,z),…,κ(zn,z))T\kappa_n(\mathbf{z}) := (\kappa(\mathbf{z}_1,\mathbf{z}), \ldots, \kappa(\mathbf{z}_n,\mathbf{z}))^{\mathsf{T}}. Conditioning on o\mathbf{o} gives the posterior GP(μ,Σ)\mathcal{GP}(\mu,\Sigma) with

μ(z)=κn(z)T(K+Λ)−1o,Σ(z,z′)=κ(z,z′)−κn(z)T(K+Λ)−1κn(z′)(2)\mu(\mathbf{z}) = \kappa_n(\mathbf{z})^{\mathsf{T}}(K+\Lambda)^{-1}\mathbf{o}, \qquad\Sigma(\mathbf{z},\mathbf{z}') = \kappa(\mathbf{z},\mathbf{z}') - \kappa_n(\mathbf{z})^{\mathsf{T}}(K+\Lambda)^{-1}\kappa_n(\mathbf{z}') \tag*{(2)}

and we write σ2(z):=Σ(z,z)\sigma^2(\mathbf{z}) := \Sigma(\mathbf{z},\mathbf{z}). The noise is heteroscedastic when the λi\lambda_i differ. We use the Matérn-5/25/2 kernel κ(z,z′)=σf2k5/2(∥diag⁡(ℓ)−1(z−z′)∥)\kappa(\mathbf{z},\mathbf{z}') = \sigma_f^2 k_{5/2}(\lVert\operatorname{diag}(\boldsymbol{\ell})^{-1}(\mathbf{z}-\mathbf{z}')\rVert) with signal variance σf2\sigma_f^2 and one lengthscale ℓj\ell_j per dimension of z\mathbf{z}, except on the LMDP, where the kernel is linear in fixed features (Appendix § 8).

Heteroscedastic Gaussian Process Q-learning

Bellman targets

Like least-squares value iteration (Jin et al., [32]) and kernel optimistic least-squares value iteration (KOVI, Yang et al., 2020), HGPQ keeps one GP per time step, Qt∼GP(μt,Σt)Q_t \sim\mathcal{GP}(\mu_t,\Sigma_t) for t=0,…,T−1t = 0, \ldots, T-1, refitted backwards after each episode, and a dataset Dt\mathcal{D}_t of the transitions observed at time step tt. The GPs have independent priors, because the Q-function of a finite-horizon problem differs between time steps, which are coupled only through the targets. Each stored transition gives the Bellman target, the Q-learning target with the GP mean in place of a table of values,

yt,i=ri+Vt+1(xi′),Vt+1(x):=max⁡u∈Uμt+1(x,u)(3)y_{t,i} = r_i + V_{t+1}(\mathbf{x}_i'), \qquad V_{t+1}(\mathbf{x}) := \max_{\mathbf{u} \in\mathcal{U}} \mu_{t+1}(\mathbf{x}, \mathbf{u}) \tag*{(3)}

where VTV_T is the terminal value. Our benchmarks give the reward of the final state on the last transition, so VT≡0V_T \equiv0, and only the synthetic MDPs of § 5.2 have a known terminal reward, VT=rˉTV_T = \bar{r}_T. Appendix § 8 covers other terminal rewards and the clipping of Vt+1V_{t+1}.

Learning rate from the GP posterior

Tabular Q-learning (Watkins and Dayan, [55]) moves a stored value Q^t(zi)\hat{Q}_t(\mathbf{z}_i) towards the target of a transition (zi,ri,xi′)(\mathbf{z}_i,r_i,\mathbf{x}_i') observed at time step tt by α(yt,i−Q^t(zi))\alpha\left(y_{t,i}-\hat{Q}_t(\mathbf{z}_i)\right), with a learning rate α∈(0,1]\alpha\in(0,1] that must often be tuned. We instead hold a GP belief Qt∼GP(μt,Σt)Q_t \sim\mathcal{GP}(\mu_t,\Sigma_t) about the Q-function at time step tt and treat each target as a noisy measurement of it,

yt,i=ri+max⁡u∈Uμt+1(xi′,u)=Qt(zi)+ηt,i,E[ηt,i]=0,Var⁡(ηt,i)=λt,i(4)y_{t,i} = r_i + \max_{\mathbf{u} \in\mathcal{U}} \mu_{t+1}(\mathbf{x}_i',\mathbf{u}) = Q_t(\mathbf{z}_i) + \eta_{t,i}, \qquad\mathbb{E}[\eta_{t,i}] = 0, \qquad\operatorname{Var}(\eta_{t,i}) = \lambda_{t,i} \tag*{(4)}

References

  1. [1]Yasin Abbasi-Yadkori, Dávid Pál, and Csaba Szepesvári. Improved algorithms for linear stochastic bandits. In Advances in Neural Information Processing Systems, volume 24, 2011.
  2. [2]Milad Abolhasani and Eugenia Kumacheva. The rise of self-driving labs in chemical and materials sciences. Nature Synthesis, 2(6):483–492, 2023.DOI
  3. [3]Alekh Agarwal, Yujia Jin, and Tong Zhang. VOQL: Towards optimal regret in model-free RL with nonlinear function approximation. In Conference on Learning Theory, pages 987–1063, 2023.arxiv.org/abs/2212.06069
  4. [4]Matteo Aldeghi, Florian Häse, Riley J Hickman, Isaac Tamblyn, and Alán Aspuru-Guzik. Golem: An algorithm for robust experiment and process optimization. Chemical Science, 12(44):14792–14807, 2021.DOI
  5. [5]François Bachoc. Cross validation and maximum likelihood estimations of hyper-parameters of Gaussian processes with model misspecification. Computational Statistics & Data Analysis, 66:55–69, 2013.arxiv.org/abs/1301.4320
  6. [6]Jasmine Bayrooti, Sattar Vakili, Amanda Prorok, and Carl Henrik Ek. No-regret Thompson sampling for finite-horizon Markov decision processes with Gaussian processes. In Advances in Neural Information Processing Systems, volume 38, 2025.arxiv.org/abs/2510.20725
  7. [7]Maximilian Bloor, Tom Savage, Calvin Tsay, Antonio del Rio Chanona, and Max Mowbray. Gaussian process Q-learning for finite-horizon Markov decision processes. Reinforcement Learning Journal, 6:1916–1930, 2025a.
  8. [8]Maximilian Bloor, José Torraca, Ilya Orson Sandoval, Akhil Ahmed, Martha White, Mehmet Mercangöz, Calvin Tsay, Antonio del Rio Chanona, and Max Mowbray. PC-gym: Benchmark environments for process control problems. Computers & Chemical Engineering, page 109363, 2025b.arxiv.org/abs/2410.22093
  9. [9]Eric Bradford and Lars Imsland. Economic stochastic model predictive control using the unscented Kalman filter. IFAC-PapersOnLine, 51(18):417–422, 2018.
  10. [10]Eric Bradford, Lars Imsland, Dongda Zhang, and Antonio del Rio Chanona. Stochastic data-driven model predictive control using Gaussian processes. Computers & Chemical Engineering, 139:106844, 2020.DOI
  11. [11]David R Burt, Carl Edward Rasmussen, and Mark van der Wilk. Convergence of sparse variational inference in Gaussian processes regression. Journal of Machine Learning Research, 21(131):1–63, 2020.arxiv.org/abs/2008.00323
  12. [12]Richard H Byrd, Peihuang Lu, Jorge Nocedal, and Ciyou Zhu. A limited memory algorithm for bound constrained optimization. SIAM Journal on Scientific Computing, 16(5):1190–1208, 1995.
  13. [13]Girish Chowdhary, Miao Liu, Robert Grande, Thomas Walsh, Jonathan How, and Lawrence Carin. Off-policy reinforcement learning with Gaussian processes. IEEE/CAA Journal of Automatica Sinica, 1(3):227–238, 2014.DOI
  14. [14]Sayak Ray Chowdhury and Rafael Oliveira. Value function approximations via kernel embeddings for no-regret reinforcement learning. In Asian Conference on Machine Learning, pages 249–264, 2023.
  15. [15]Marc Peter Deisenroth, Carl Edward Rasmussen, and Jan Peters. Gaussian process dynamic programming. Neurocomputing, 72(7–9):1508–1524, 2009.DOI
  16. [16]Marc Peter Deisenroth, Dieter Fox, and Carl Edward Rasmussen. Gaussian processes for data-efficient learning in robotics and control. IEEE Transactions on Pattern Analysis and Machine Intelligence, 37(2):408–423, 2015.DOI
  17. [17]David E. Edmunds and Hans Triebel. Function Spaces, Entropy Numbers, Differential Operators. Cambridge University Press, 1996.DOI
  18. [18]Yaakov Engel, Shie Mannor, and Ron Meir. Bayes meets Bellman: The Gaussian process approach to temporal difference learning. In International Conference on Machine Learning, pages 154–161, 2003.
  19. [19]Yaakov Engel, Shie Mannor, and Ron Meir. Reinforcement learning with Gaussian processes. In International Conference on Machine Learning, pages 201–208, 2005.
  20. [20]Damien Ernst, Pierre Geurts, and Louis Wehenkel. Tree-based batch mode reinforcement learning. Journal of Machine Learning Research, 6:503–556, 2005.DOI
  21. [21]Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International Conference on Machine Learning, pages 1587–1596, 2018.
  22. [22]Matthieu Geist and Olivier Pietquin. Kalman temporal differences. Journal of Artificial Intelligence Research, 39:483–532, 2010.
  23. [23]Abraham P. George and Warren B. Powell. Adaptive stepsizes for recursive estimation with applications in approximate dynamic programming. Machine Learning, 65(1):167–198, 2006.DOI
  24. [24]Paul W. Goldberg, Christopher K. I. Williams, and Christopher M. Bishop. Regression with input-dependent noise: A Gaussian process treatment. In Advances in Neural Information Processing Systems, volume 10, pages 493–499, 1998.
  25. [25]Ryan-Rhys Griffiths, Alexander A Aldrick, Miguel Garcia-Ortegon, Vidhi Lalchand, and Alpha A Lee. Achieving robustness to aleatoric uncertainty with heteroscedastic Bayesian optimisation. Machine Learning: Science and Technology, 3(1):015004, 2022.DOI
  26. [26]Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International Conference on Machine Learning, pages 1861–1870, 2018.
  27. [27]Nikolaus Hansen, Anne Auger, Raymond Ros, Olaf Mersmann, Tea Tušar, and Dimo Brockhoff. COCO: A platform for comparing continuous optimizers in a black-box setting. Optimization Methods and Software, 36(1):114–144, 2021.DOI
  28. [28]Jiafan He, Heyang Zhao, Dongruo Zhou, and Quanquan Gu. Nearly minimax optimal reinforcement learning for linear Markov decision processes. In International Conference on Machine Learning, pages 12790–12822, 2023.arxiv.org/abs/2212.06132
  29. [29]Wassily Hoeffding. Probability inequalities for sums of bounded random variables. Journal of the American Statistical Association, 58(301):13–30, 1963.
  30. [30]Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model-based policy optimization. In Advances in Neural Information Processing Systems, volume 32, 2019.arxiv.org/abs/1906.08253
  31. [31]Heejin Jeong, Clark Zhang, George J. Pappas, and Daniel D. Lee. Assumed density filtering Q-learning. In International Joint Conference on Artificial Intelligence, 2019.DOI
  32. [32]Chi Jin, Zhuoran Yang, Zhaoran Wang, and Michael I. Jordan. Probably efficient reinforcement learning with linear function approximation. In Conference on Learning Theory, pages 2137–2143, 2020.arxiv.org/abs/1907.05388
  33. [33]Motonobu Kanagawa, Philipp Hennig, Dino Sejdinovic, and Bharath K Sriperumbudur. Gaussian processes and kernel methods: A review on connections and equivalences. arXiv preprint arXiv:1807.02582, 2018.arxiv.org/abs/1807.02582
  34. [34]Kristian Kersting, Christian Plagemann, Patrick Pfaff, and Wolfram Burgard. Most likely het-eroscedastic Gaussian process regression. In International Conference on Machine Learning, pages 393–400, 2007.
  35. [35]Armin Lederer, Jonas Umlauft, and Sandra Hirche. Uniform error and posterior variance bounds for Gaussian process regression with application to safe control. arXiv preprint arXiv:2101.05328, 2021.arxiv.org/abs/2101.05328
  36. [36]Kimin Lee, Michael Laskin, Aravind Srinivas, and Pieter Abbeel. SUNRISE: A simple unified framework for ensemble learning in deep reinforcement learning. In International Conference on Machine Learning, volume 139, pages 6131–6141, 2021.arxiv.org/abs/2007.04938
  37. [37]Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In International Conference on Learning Representations, 2016.arxiv.org/abs/1509.02971
  38. [38]Raman K Mehra. On the identification of variances and adaptive Kalman filtering. IEEE Transactions on Automatic Control, 15(2):175–184, 1970.
  39. [39]Ali Mesbah. Stochastic model predictive control: An overview and perspectives for future research. IEEE Control Systems, 36(6):30–44, 2016.DOI
  40. [40]Yifei Min, Tianhao Wang, Dongruo Zhou, and Quanquan Gu. Variance-aware off-policy evaluation with linear function approximation. In Advances in Neural Information Processing Systems, volume 34, 2021.arxiv.org/abs/2106.11960
  41. [41]Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529–533, 2015.DOI
  42. [42]Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped DQN. In Advances in Neural Information Processing Systems, volume 29, 2016.arxiv.org/abs/1602.04621
  43. [43]Panagiotis Petsagkourakis, Ilya Orson Sandoval, Eric Bradford, Dongda Zhang, and Antonio del Rio Chanona. Reinforcement learning for batch bioprocess optimization. Computers & Chemical Engineering, 133:106649, 2020.arxiv.org/abs/1904.07292
  44. [44]Kyriakos Polymenakos, Luca Laurenti, Andrea Patane, Jan-Peter Calliess, Luca Cardelli, Marta Kwiatkowska, Alessandro Abate, and Stephen Roberts. Safety guarantees for iterative predictions with Gaussian processes. In IEEE Conference on Decision and Control, pages 3187–3193, 2020.
  45. [45]Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. In Advances in Neural Information Processing Systems, volume 20, 2007.
  46. [46]Carl Edward Rasmussen and Christopher K I Williams. Gaussian Processes for Machine Learning. MIT Press, 2006.DOI
  47. [47]Ilya O. Ryzhow, Peter I. Frazier, and Warren B. Powell. A new optimal stepsize for approximate dynamic programming. IEEE Transactions on Automatic Control, 60(3):743–758, 2015.
  48. [48]Bobak Shahriari, Kevin Swersky, Ziyu Wang, Ryan P. Adams, and Nando de Freitas. Taking the human out of the loop: A review of Bayesian optimization. Proceedings of the IEEE, 104(1): 148–175, 2016.DOI
  49. [50]William R. Thompson. On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika, 25(3/4):285–294, 1933.DOI
  50. [51]Michalis Titsias. Variational learning of inducing variables in sparse Gaussian processes. In International Conference on Artificial Intelligence and Statistics, pages 567–574, 2009.
  51. [52]Charles Tripp and Ross D Shachter. Approximate Kalman filter Q-learning for continuous state-space MDPs. In Conference on Uncertainty in Artificial Intelligence, 2013.arxiv.org/abs/1309.6868
  52. [53]Sattar Vakili and Julia Olkhovskaya. Kernelized reinforcement learning with order optimal regret bounds. In Advances in Neural Information Processing Systems, volume 36, 2023.arxiv.org/abs/2306.07745
  53. [54]Sattar Vakili, Kia Khezeli, and Victor Picheny. On information gain and regret bounds in Gaussian process bandits. In International Conference on Artificial Intelligence and Statistics, pages 82–90, 2021.arxiv.org/abs/2009.06966
  54. [55]Christopher J. C. H. Watkins and Peter Dayan. Q-learning. Machine Learning, 8(3):279–292, 1992.DOI
  55. [56]Gellért Weisz, Philip Amortila, and Csaba Szepesvári. Exponential lower bounds for planning in MDPs with linearly-realizable optimal action-value functions. In Algorithmic Learning Theory, pages 1237–1264, 2021.arxiv.org/abs/2010.01374

Paper details

vlnc:W_5quyerh3
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents