betaFeedback

WorksComputer Science

Inferring Transferable Rewards via Active Inverse Reinforcement Learning

Abstract

Inverse Reinforcement Learning (IRL) aims to recover expert rewards that transfer across environments with differing dynamics. Prior work has shown that such transfer is possible when gathering expert demonstrations across sufficiently different dynamics. In practice, however, expert demonstrations are costly, making it critical to select the right dynamics to reduce the quantity of expert data required for learning transferable rewards. We propose a two-phase approach: we first cycle through the available dynamics to obtain a reliable reward estimate, then actively select dynamics to reduce our uncertainty about the expert reward. Crucially, this yields novel sample complexity guarantees on transfer that hold for any dynamics, with a rate independent of the number of available dynamics. Empirically, our method significantly reduces the number of expert demonstrations needed to learn a transferable reward.

Introduction

Learning a transferable representation of expert behavior makes it possible to predict how the expert would act in previously unseen circumstances. Inverse Reinforcement Learning (IRL) [1] formalizes this by inferring a reward function from expert demonstrations. Unlike imitation learning methods such as Behavior Cloning (BC) [2], which directly replicate observed behavior, IRL aims to recover the underlying objective the expert is optimizing. In contrast to a cloned policy tied to its training dynamics, a reward can, in principle, transfer to unseen dynamics.

Despite its appeal, IRL is fundamentally ill-posed [1]. Under entropy regularization [3], the standard setting in identifiability works [4, 5], demonstrations from a fixed Markov decision process (MDP) identify the reward only up to subspace specific to that MDP. Consequently, such rewards transfer only to dynamics whose identifiable subspace sufficiently overlaps with that of the original MDP [6]. Recent works show that observing expert demonstrations in as few as two sufficiently different dynamics resolves this ambiguity [4, 5, 6]. However, these works leave open how to actively select such dynamics to reduce the ambiguity with as few expert demonstrations as possible.

In practice, expert demonstrations are costly, and only a fixed set of dynamics is available. We must therefore choose carefully which dynamics to query the expert in. This raises the following central question:

How should we select dynamics to learn a transferable reward from a minimal number of expert demonstrations?

To address this question, we propose an active IRL setting, in which the learner chooses the dynamics under which the expert demonstrates the task. This setting is natural whenever reconfiguring the environment is cheap compared to collecting expert demonstrations. For example, a robotics lab can cheaply vary the initial placement, mass, or friction of manipulated objects [7], or a driving simulator can vary road conditions [8]; collecting demonstrations, by contrast, requires a skilled human operator for each one [9].

Closest to our work, Buening et al. [10] adaptively design dynamics to identify the expert’s reward. They empirically show that, compared to randomly chosen environments, this yields more transferable rewards from fewer demonstrations, but provide no theoretical guarantees. To the best of our knowledge, our work provides the first active IRL algorithms with sample complexity guarantees for learning transferable rewards through dynamics selection.

Contributions. Our main contributions are:

  1. We formalize active IRL as a sequential problem: over nn rounds, the learner chooses dynamics from a finite set on which to request an expert demonstration, aiming to recover a reward that transfers as much as possible across the set (Equation (1)).

  1. We derive novel confidence sets for the expert reward (Lemma 3.1) and show that transfer to each dynamics is controlled by how informative they are relative to the information already accumulated (Lemma 3.2).

  1. This yields a two-phase algorithm: cycle through the set of dynamics until the reward estimate is reliable (Lemma 3.3), then query the expert where information gain is greatest. After nn rounds of dynamics selection and expert demonstrations, this results in a transfer error of O~(1/n)\widetilde{O}(1/n) across the set of available dynamics, independent of its cardinality (Theorem 3.6). The same rate extends to any dynamics, up to an irreducible bias (Theorem 3.7).

  1. We validate our approach empirically against imitation learning and IRL baselines, as well as dynamics design for IRL [10]. Our active approach learns an equally transferable reward using up to five times fewer demonstrations than the baselines, and its learned reward generalizes best to unseen dynamics (§ 4).

Problem formulation

Notations. Let ∥⋅∥2\lVert\cdot\rVert_2 denote the Euclidean norm. For a symmetric positive semidefinite matrix X∈Rd×dX \in\mathbb{R}^{d \times d}, ∥X∥2\lVert X\rVert_2 denotes its spectral norm, X†X^\dagger its Moore–Penrose inverse, and ∥v∥X:=⟨v,Xv⟩\lVert v\rVert_X := \sqrt{\langle v, Xv\rangle} the associated Mahalanobis norm of a vector v∈Rdv \in\mathbb{R}^d. For a finite set Z\mathcal{Z}, ΔZ\Delta^{\mathcal{Z}} denotes the probability simplex over Z\mathcal{Z}.

Regularized MDPs. A regularized MDP is a tuple M=(S,A,T,ν1,r,H,η)\mathcal{M} = (\mathcal{S}, \mathcal{A}, T, \nu_1, r, H, \eta). Here, S\mathcal{S} and A\mathcal{A} are finite state and action spaces, T:S×A→ΔST : \mathcal{S} \times\mathcal{A} \to\Delta^{\mathcal{S}} is the transition function, ν1∈ΔS\nu_1 \in\Delta^{\mathcal{S}} denotes the initial state distribution, r:S×A→Rr : \mathcal{S} \times\mathcal{A} \to\mathbb{R} denotes the reward, and H∈N+H \in\mathbb{N}^{+} is the horizon. We consider entropy-regularized MDPs, with η>0\eta> 0 denoting the inverse temperature that controls the strength of the regularization. Starting from state s1∼ν1s_1 \sim\nu_1, an agent interacting with the MDP takes at each timestep hh an action ah∈Aa_h \in\mathcal{A}, receives the reward r(sh,ah)r(s_h, a_h) and the environment transitions to a new state sh+1∼T(⋅∣sh,ah)s_{h+1} \sim T(\cdot\mid s_h, a_h).

Denoting the dynamics by P=(ν1,T)P = (\nu_1, T), the optimal solution of M\mathcal{M} is a policy π=(πh)h=1H\pi= (\pi_h)_{h=1}^{H}, with πh:S→ΔA\pi_h : \mathcal{S} \to\Delta^{\mathcal{A}}, maximizing the regularized value VPπ(r):=EPT[∑h=1Hr(sh,ah)−η−1log⁡(πh(ah∣sh))]V^\pi_P(r) := \mathbb{E}^T_P\left[\sum_{h=1}^{H} r(s_h,a_h) - \eta^{-1}\log\bigl(\pi_h(a_h\mid s_h)\bigr)\right]. A state ss is unreachable at timestep hh if no policy visits it with positive probability at hh. As a convention, all policies follow an arbitrarily fixed action distribution (e.g., uniform) on unreachable states, which leaves trajectory laws and values unchanged. The MDP M\mathcal{M} then admits a unique optimal policy [3]. For a reward rr and dynamics PP, we write VP∗(r)V^*_P(r) and πP(r)\pi_P(r) for the corresponding optimal value and policy.

Setting. Given an unknown reward rEr^{\mathrm{E}}, an expert follows the optimal policy πPE:=πP(rE)\pi_P^{\mathrm{E}} := \pi_P(r^{\mathrm{E}}) under any chosen dynamics PP. We consider a fixed, finite family of MDPs {Mi}i\{\mathcal{M}_i\}_i that share rEr^{\mathrm{E}} but differ in their dynamics P:={Pi}i\mathcal{P} := \{P_i\}_i. A demonstration in Mi\mathcal{M}_i is a trajectory τ=((sh,ah))h=1H\tau= ((s_h,a_h))_{h=1}^{H} generated by πPiE\pi_{P_i}^{\mathrm{E}}.

We consider an active setting in which a learner selects dynamics sequentially, and an expert is queried after each selection. Over a budget of n∈N+n \in\mathbb{N}^{+} rounds, the learner selects at each round kk dynamics Pk∈PP_k \in\mathcal{P}, and observes a batch of mm expert trajectories generated under PkP_k, for a total of nmnm trajectories, gathering them in a dataset DE:=⋃k=1n{τk,j}j=1m\mathcal{D}^{\mathrm{E}} := \bigcup_{k=1}^{n}\{\tau_{k,j}\}_{j=1}^{m}.

Objective. Given this budget, our goal is to select dynamics (Pk)k=1n(P_k)_{k=1}^{n} from P\mathcal{P} and estimate a reward rr from DE\mathcal{D}^{\mathrm{E}} that is transferable across P\mathcal{P}. We quantify transferability by the worst-case suboptimality,

max⁡P∈PSubOpt⁡(r,P)=max⁡P∈PVP∗(rE)−VPπP(r)(rE),(1)\max_{P\in\mathcal{P}} \operatorname{SubOpt}(r,P) = \max_{P\in\mathcal{P}} V^*_P(r^{\mathrm{E}}) - V^{\pi_P(r)}_P(r^{\mathrm{E}}), \tag*{(1)}

and we seek a reward rr minimizing (1). This objective captures the ground-truth performance of the recovered optimal policy πP(r)\pi_P(r) across P\mathcal{P} and coincides with the notion of transferability studied by Schlaginhaufen and Kamgarpour [6] for fixed dynamics. Furthermore, it can be viewed as a regularized analogue of the imitation gap studied in imitation learning [11].

Structural assumption. Before presenting the algorithm for selecting the dynamics PkP_k and estimating the reward, we introduce our main structural assumption, which enables efficient estimation of the expert reward rEr^{\mathrm{E}} from expert demonstrations DE\mathcal{D}^{\mathrm{E}}.

Assumption 2.1 (Expert bounded linear rewards). Let ϕ:S×A→Rd\phi: \mathcal{S} \times\mathcal{A} \to\mathbb{R}^{d} be a known feature map with ∥ϕ(s,a)∥2≤Bϕ\lVert\phi(s,a)\rVert_2 \le B_\phi for all (s,a)∈S×A(s,a) \in\mathcal{S}\times\mathcal{A}, and let Θ:={θ∈Rd:∥θ∥2≤Bθ}\Theta:= \{\theta\in\mathbb{R}^{d} : \lVert\theta\rVert_2 \le B_\theta\}, where Bϕ,Bθ>0B_\phi,B_\theta> 0 are known constants. We assume the expert reward is linear in unknown parameters θE∈Θ\theta^{\mathrm{E}} \in\Theta, i.e. rE(s,a)=⟨θE,ϕ(s,a)⟩r^{\mathrm{E}}(s,a) = \langle\theta^{\mathrm{E}},\phi(s,a)\rangle.

Linear rewards are standard in RL [12] and inverse RL [1]. We index rewards rθ=⟨θ,ϕ⟩r_\theta= \langle\theta,\phi\rangle by their parame-

Algorithm 1 Active IRL
1: input: set of dynamics $\mathcal{P}$, warm-up length $n_0$
2: // Warm-up phase
3: for round $k = 1,\ldots,n_0$ do
4:   Select $P_k$ round-robin over $\mathcal{P}$ (6)
5:   Observe $m$ trajectories in $P_k$
6:   Estimate $\theta_{n_0}$ by minimizing (2) (Section 3.1)
7: // Active phase
8: for round $k = n_0+1,\ldots,n$ do
9:   Actively select $P_k$ via (7) (Section 3.2)
10:  Observe $m$ trajectories in $P_k$
11:  Estimate $\theta_k$ by minimizing (2) (Section 3.1)

ters θ\theta, and write ϕ(τ)=∑(s,a)∈τϕ(s,a)\phi(\tau)=\sum_{(s,a)\in\tau}\phi(s,a) for trajectory features.

Finally, we assume that the expert is optimal for θE\theta^{\mathrm{E}}, with a known temperature. Appendix G extends our results to misspecified experts, e.g., when the assumed temperature is incorrect, in which case all our transferability results hold with an additional error proportional to the expert’s suboptimality.

Actively learning transferable rewards

We now present our active method for learning transferable rewards minimizing (1), summarized in Algorithm 1. Its key idea is to query the expert under the most informative dynamics, i.e., those revealing parts of the reward that matter for transfer across P\mathcal{P} but remain poorly determined by previous demonstrations.

The method proceeds in two phases. In a warm-up phase, the learner has too little data to tell which dynamics are informative, so it queries the expert under each available dynamics in turn. It then switches to an active phase, in which it uses its reward estimate to select the most informative dynamics.

To make this strategy precise, we first quantify the accuracy of reward estimates from adaptively collected demonstrations through ellipsoidal confidence bounds (§ 3.1). These bounds determine when the estimate is reliable enough to guide dynamics selection. In § 3.2, we use them to construct the two-phase selection algorithm, before deriving transfer guarantees for the learned reward in § 3.3.

Reward estimation

To select informative dynamics, we need to quantify how much information each dynamics provides about θE\theta^{\mathrm{E}}, and how much has already been accumulated. We first introduce the loss and use its curvature to characterize the information provided under each dynamics.

We then show how the accumulated information yields an ellipsoidal confidence bound for θE\theta^{\mathrm{E}} (Lemma 3.1). Proofs are deferred to Appendix B.

IRL loss. After kk rounds, we estimate θE\theta^{\mathrm{E}} by minimizing the empirical loss over Θ\Theta,

Lk(θ):=∑i=1kli(θ),li(θ):=VPi∗(ΠPiθ)−⟨θ,ΠPiϕ^i⟩,(2)L_k(\theta) := \sum_{i=1}^{k} l_i(\theta), \qquad l_i(\theta) := V_{P_i}^{*}(\Pi_{P_i}\theta)-\langle\theta,\Pi_{P_i}\hat{\phi}_i\rangle, \tag*{(2)}

and write θk∈arg⁡min⁡θ∈ΘLk(θ)\theta_k\in\arg\min_{\theta\in\Theta}L_k(\theta). Here PiP_i are the dynamics selected at round ii, ϕ^i=1m∑j=1mϕ(τi,j)\hat{\phi}_i=\frac{1}{m}\sum_{j=1}^{m}\phi(\tau_{i,j}) is the empirical mean of the expert’s trajectory features, and ΠPi\Pi_{P_i} is the orthogonal projector onto the identifiable subspace UPiU_{P_i} defined below. Without the projections, each lil_i is an entropy-regularized version of the min-max IRL loss [13] under PiP_i.

Curvature and identifiability. The optimal policy induced by a reward depends on the underlying dynamics. Hence, by selecting the dynamics under which the expert demonstrates, the learner chooses which parts of the reward can be estimated, and how efficiently. We characterize this through the Hessian of the optimal value function,

HP(θ):=∇θ2VP∗(θ),\mathcal{H}_{P}(\theta):=\nabla_{\theta}^{2}V_{P}^{*}(\theta),

References

  1. [1]Andrew Y Ng, Stuart Russell, et al. Algorithms for inverse reinforcement learning. In Icml, volume 1, page 2, 2000.
  2. [2]Dean A. Pomerleau. Alvinn: An autonomous land vehicle in a neural network. In D. Touretzky, editor, Advances in Neural Information Processing Systems, volume 1. Morgan-Kaufmann, 1988. URL https://proceedings.neurips.cc/paper_files/paper/1988/file/812b4ba287f5ee0bc9d43bbf5bbe87fb-Paper.pdf
  3. [3]Matthieu Geist, Bruno Scherrer, and Olivier Pietquin. A theory of regularized markov decision processes. In International conference on machine learning, pages 2160–2169. PMLR, 2019.arxiv.org/abs/1901.11275
  4. [4]Haoyang Cao, Samuel Cohen, and Lukasz Szpruch. Identifiability in inverse reinforcement learning. Advances in Neural Information Processing Systems, 34:12362–12373, 2021.arxiv.org/abs/2106.03498
  5. [5]Paul Rolland, Luca Viano, Norman Schürhoff, Boris Nikolov, and Volkan Cevher. Identifiability and generalizability from multiple experts in inverse reinforcement learning. Advances in Neural Information Processing Systems, 35:550–564, 2022.
  6. [6]Andreas Schlaginhaufen and Maryam Kamgarpour. Towards the transferability of rewards recovered via regularized inverse reinforcement learning. Advances in Neural Information Processing Systems, 37:21461–21501, 2024.arxiv.org/abs/2406.01793
  7. [7]Xue Bin Peng, Marcin Andrychowicz, Wojciech Zaremba, and Pieter Abbeel. Sim-to-real transfer of robotic control with dynamics randomization. In 2018 IEEE international conference on robotics and automation (ICRA), pages 3803–3810. IEEE, 2018.
  8. [8]Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. Carla: An open urban driving simulator. In Conference on robot learning, pages 1–16. PMLR, 2017.arxiv.org/abs/1711.03938
  9. [9]Ajay Mandlekar, Yuke Zhu, Animesh Garg, Jonathan Bohger, Max Spero, Albert Tung, Julian Gao, John Emmons, Anchit Gupta, Emre Orbay, et al. Roboturk: A crowdsourcing platform for robotic skill learning through imitation. In Conference on Robot Learning, pages 879–893. PMLR, 2018.arxiv.org/abs/1811.02790
  10. [10]Thomas Kleine Buening, Victor Villin, and Christos Dimatrakakis. Environment design for inverse reinforcement learning. In Proceedings of the 41st International Conference on Machine Learning, pages 24808–24828, 2024.
  11. [11]Gokul Swamy, Sanjiban Choudhury, J Andrew Bagnell, and Steven Wu. Of moments and matching: A game-theoretic framework for closing the imitation gap. In International Conference on Machine Learning, pages 10022–10032. PMLR, 2021.arxiv.org/abs/2103.03236
  12. [12]Chi Jin, Zhuoran Yang, Zhaoran Wang, and Michael I Jordan. Probably efficient reinforcement learning with linear function approximation. In Jacob Abernethy and Shivani Agarwal, editors, Proceedings of Thirty Third Conference on Learning Theory, volume 125 of Proceedings of Machine Learning Research, pages 2137–2143. PMLR, 09–12 Jul 2020. URL https://proceedings.mlr.press/v125/jin20a.html.arxiv.org/abs/1907.05388
  13. [13]Umar Syed and Robert E Schapire. A game-theoretic approach to apprenticeship learning. Advances in neural information processing systems, 20, 2007.
  14. [14]Andrew Y Ng, Daishi Harada, and Stuart Russell. Policy invariance under reward transformations: Theory and application to reward shaping. In Icml, volume 99, pages 278–287. Citeseer, 1999.
  15. [15]Francis Bach. Self-concordant analysis for logistic regression. Electronic Journal of Statistics, 4:384–414, 2010.arxiv.org/abs/0910.4627
  16. [16]Andreas Schlaginhaufen and Maryam Kamgarpour. Fast rates for inverse reinforcement learning, 2026.arxiv.org/abs/2605.14599
  17. [17]Kamalika Chaudhuri, Sham M Kakade, Praneeth Netrapalli, and Sujay Sanghavi. Convergence rates of active learning for maximum likelihood estimation. Advances in Neural Information Processing Systems, 28, 2015.arxiv.org/abs/1506.02348
  18. [18]Joongkyu Lee and Min-hwan Oh. Improved online confidence bounds for multinomial logistic bandits. arXiv preprint arXiv:2502.10020, 2025.arxiv.org/abs/2502.10020
  19. [19]Jack Kiefer and Jacob Wolfowitz. Optimum designs in regression problems. The annals of mathematical statistics, 30(2):271–294, 1959.
  20. [20]Jack Kiefer. General equivalence theory for optimum designs (approximate theory). The annals of Statistics, pages 849–879, 1974.
  21. [21]Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. Advances in neural information processing systems, 29, 2016.arxiv.org/abs/1606.03476
  22. [22]Jongseong Chae, Seungyul Han, Whiyoung Jung, Myungsik Cho, Sungho Choi, and Youngchul Sung. Robust imitation learning against variations in environment dynamics. In International Conference on Machine Learning, pages 2828–2852. PMLR, 2022.arxiv.org/abs/2206.09314
  23. [23]Aurél Galántai. Finite Projection Methods for Linear Systems, pages 83–115. Springer US, Boston, MA, 2004. doi: 10.1007/978-1-4419-9180-5_3. URL https://doi.org/10.1007/978-1-4419-9180-5_3.
  24. [24]M. Lopes, Francisco S. Melo, and L. Montesano. Active learning for reward estimation in inverse reinforcement learning. In ECML/PKDD, 2009. URL https://doi.org/10.1007/978-3-642-04174-7_3.
  25. [25]Yuchen Cui and Scott Niekum. Active reward learning from critiques. 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 6907–6914, 2018. URL https://api.semanticscholar.org/CorpusID:4490670.DOI
  26. [26]Ondrej Bajgar, Dewi SW Gould, Jonathon Liu, Alessandro Abate, Konstantinos Gatsis, and Michael A Osborne. Pac apprenticeship learning with bayesian active inverse reinforcement learning. arXiv preprint arXiv:2508.03693, 2025.arxiv.org/abs/2508.03693
  27. [27]David Lindner, Andreas Krause, and Giorgia Ramponi. Active exploration for inverse reinforcement learning, 2023. URL https://arxiv.org/abs/2207.08645.
  28. [28]Guang-Yuan Hao, Lars van der Laan, Aurélien Bibaut, and Nathan Kallus. Reward transfer from inverse reinforcement learning: A coupled minimax approach. arXiv preprint arXiv:2605.27834, 2026.arxiv.org/abs/2605.27834
  29. [29]Filippo Lazzati and Alberto Maria Metelli. Robustness in the face of partial identifiability in reward learning. In International Conference on Learning Representations, volume 2026, pages 107353–107390, 2026.arxiv.org/abs/2501.06376
  30. [30]Filippo Lazzati and Alberto Maria Metelli. Generalizing behavior via inverse reinforcement learning with closed-form reward centroids. arXiv preprint arXiv:2509.12010, 2025.arxiv.org/abs/2509.12010
  31. [31]Kareem Amin, Nan Jiang, and Satinder Singh. Repeated inverse reinforcement learning. Advances in neural information processing systems, 30, 2017.arxiv.org/abs/1705.05427
  32. [32]Paul Nitschke, Lars Lien Ankile, Eura Nofshin, Siddharth Swaroop, Finale Doshi-Velez, and Weiwei Pan. Amber: An entropy maximizing environment design algorithm for inverse reinforcement learning. In ICML 2024 Workshop on Models of Human Feedback for AI Alignment, 2024.
  33. [33]Ondrej Bajgar, Sid William Gould, R. Mitta, Jonathon Liu, Oliver Newcombe, and Jackson Golden. Toward information theoretic active inverse reinforcement learning. ArXiv, abs/2501.00381, 2024. URL https://api.semanticscholar.org/CorpusId:275212510.arxiv.org/abs/2501.00381
  34. [34]Jordan Ash, Surbhi Goel, Akshay Krishnamurthy, and Sham Kakade. Gone fishing: Neural active learning with fisher embeddings. Advances in Neural Information Processing Systems, 34:8927–8939, 2021.arxiv.org/abs/2106.09675
  35. [35]Titouan Renard, Andreas Schlaginhaaufen, Tingting Ni, and Maryam Kamgarpour. Convergence of a model-free entropy-regularized inverse reinforcement learning algorithm. arXiv preprint arXiv:2403.16829, 2024.arxiv.org/abs/2403.16829

Paper details

vlnc:W_6dag2wfr
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 10

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents