betaFeedback

WorksComputer Science

Learning to Undo: Transfer Reinforcement Learning under State Space Transformations

Abstract

Transfer learning in reinforcement learning (RL) has shown strong empirical success. In this work, we take a more principled perspective by studying when and how transferring knowledge between MDPs (a source and a target) can be provably beneficial. Specifically, we consider the case where there exists an undo map such that applying this map to the target's state space recovers the source exactly. We propose an algorithm that learns this map by matching state feature statistics gathered from both MDPs, and then uses it to transfer the source policy to the target. We theoretically justify the algorithm by analyzing the setting when the undo map is linear and the source is linearly-Q⋆Q^\star realizable, where our approach has strictly better sample complexity than tabula rasa RL in the target MDP. Empirically, we demonstrate that these benefits extend beyond this regime: on challenging continuous control tasks and Atari games, our method achieves significantly better sample efficiency. Overall, our results highlight how shared structure between tasks can be leveraged for efficient transfer of policies across environments.

Introduction

Reinforcement learning (RL) has seen rapid progress in recent years and has achieved strong performance in complex tasks, such as video games, locomotion, manipulation, and navigation [16, 14, 30]. Despite this, learning high-quality policies from scratch typically requires millions of environment interactions, which limits the applicability of RL in real-world domains where data collection is costly or time-consuming [7].

Transfer learning aims to overcome this challenge by leveraging knowledge learned from source tasks to accelerate learning on related target tasks [22]. It has the potential to drastically improve sample efficiency by effectively reusing prior experience. While transfer learning has seen strong empirical success in RL [31], our goal is not to design a state-of-the-art transfer method that works across arbitrary variations in tasks. Instead, we study the problem from a more principled lens, where we explicitly model the similarity between tasks and exploit it to obtain an algorithm that not only has provable sample complexity improvements over learning from scratch, but also works empirically.

An interesting and practically relevant setting with such structure is that of state space transformations: the source and target tasks differ due to transformations in the state space. For instance, real-world transformations such as color transformation from grayscale to RGB, change in frame of reference, and sensor fusion – all are state space transformations (see Figure 2). Such transformations provide a rich and practical setting to design principled transfer learning methods that explicitly model them. One might assume that such transformations are trivial to handle since the underlying task remains the same. However, this intuition is misleading: as shown in Figure 1, even a 10×10 \times increase in observation size in the sensor fusion setting, where readings must be aggregated meaningfully, is enough to render tabula rasa RL severely suboptimal, which highlights that principled transfer is necessary.

Average return comparison between ANT and FusionANT under a 10× observation size increase

Figure 1. State space transformations can significantly decrease sample efficiency (e.g., increasing observation size by 10×10 \times).

Figure 2. State space transformations are grounded in real-world applications.

State-space and observation transformations naturally occur in many practical RL scenarios where the underlying task dynamics remain unchanged, but the agent’s interface to the environment changes. For instance, in robotics, policies are often trained in simulation and then deployed on real systems, since real-world training is significantly more costly and difficult to parallelize [29]. In the transition from simulation to the real system, differences between simulated and physical sensors cause discrepancies in sensing and perception, and thus, introduce shifts in the agent’s observation space [3]. In such settings, relearning policies from scratch can be very sample-inefficient even though the underlying control problem essentially remains the same.

Furthermore, low-dimensional privileged state features are often available during development, but deployment must operate under high-dimensional inputs such as camera observations. Even when both types of observations are available, learning with compact representations is often substantially easier [11], which highlights the utility of learning under state-space transformations.

In this work, we address this setting by learning an undo map that transforms the state space in the target back to the source. Instead of learning a policy from scratch, we recover a function that undoes the transformation applied to the target states, which allows reusing source policies in the target task.

To this end, we introduce a distributional notion of undo discrepancy that measures the quality of a candidate undo map. Minimizing this discrepancy is equivalent to matching the state feature distributions in the source MDP and the corresponding undone target feature distributions at each timestep under open-loop action sequences. We then propose a practical transfer learning algorithm that optimizes this objective using samples from the target. Theoretically, we show that when the undo map is linear and the source is linearly-Q∗Q^* realizable, our approach has strictly better sample complexity than learning from scratch.

To empirically evaluate our method, we use challenging continuous control tasks and Atari games, and construct target tasks by applying transformations such as changes in frame of reference and coordinate system, sensor fusion-inspired transformations, and color shifts. We also consider perception gaps such as moving from sensor observations in the source to pixel observations in the target, and moving from semantic bitmaps in the source to pixel observations in the target. Our results show that learning the undo map consistently outperforms learning the target policy from scratch in terms of sample efficiency, where it often achieves near-optimal performance with an order of magnitude fewer samples. These results highlight the practical utility of our approach for transfer RL.

To summarize, our work makes the following contributions:

  1. We propose a novel transfer learning method, LUMOS (Learning to Undo by Matching Observation Statistics), that performs regression on state feature statistics to undo state-space transformations in the target MDP (§ 3.1 and § 3.2).

  2. Theoretically, we prove that our method achieves strictly better sample complexity than learning from scratch when the undo map is linear and the source is linearly-Q∗Q^* realizable (§ 3.3).

  3. We empirically demonstrate the practicality of our method via experiments on challenging continuous control tasks and Atari games involving state-space transformations (§ 4).

Problem Setup

Tasks. We model each task as a finite-horizon episodic Markov Decision Process (MDP), defined by the 7-tuple (S,A,P,r,H,μ,ϕ)(\mathcal{S}, \mathcal{A}, \mathbb{P}, r, H, \mu, \phi), where S\mathcal{S} is the state space, A\mathcal{A} is the action space, P:S×A→P(S)\mathbb{P}:\mathcal{S}\times\mathcal{A}\to\mathcal{P}(\mathcal{S}) defines the transition dynamics, r:S×A→Rr:\mathcal{S}\times\mathcal{A}\to\mathbb{R} is the reward function, HH is the fixed horizon length, μ∈P(S)\mu\in\mathcal{P}(\mathcal{S}) is the initial state distribution, and ϕ:S→Rd\phi:\mathcal{S}\to\mathbb{R}^{d} is a fixed feature map. W.l.o.g., we assume that d≤Hd\le H. The agent interacts with the task using a non-stationary policy πh:Rd→P(A)\pi_h:\mathbb{R}^{d}\to\mathcal{P}(\mathcal{A}), and for 0≤h<H0\le h<H, chooses an action ah∼πh(⋅∣ϕ(sh))a_h\sim\pi_h(\cdot\mid\phi(s_h)). The value of the policy π\pi is defined as the expected return Vπ≜Es0∼μ,π,P[∑h=0H−1r(sh,ah)]V^\pi\triangleq\mathbb{E}_{s_0\sim\mu,\pi,\mathbb{P}}\left[\sum_{h=0}^{H-1}r(s_h,a_h)\right], and the agent’s goal is to learn a policy π⋆\pi^\star that maximizes this quantity, i.e., V⋆=max⁡πVπV^\star=\max_\pi V^\pi. The state-action value function Qπ(s,a)Q^\pi(s,a) denotes the expected return when taking action aa in state ss, and following policy π\pi thereafter.

Transfer Learning. The transfer learning setting follows the protocol in which the agent interacts with a source task MS=(SS,AS,PS,rS,H,μS,ϕS)\mathcal{M}_S=(\mathcal{S}^S,\mathcal{A}^S,\mathbb{P}^S,r^S,H,\mu^S,\phi_S) to learn a policy πS\pi_S, and then leverages the learned policy to speed up learning in the target task MT=(ST,AT,PT,rT,H,μT,ϕT)\mathcal{M}_T=(\mathcal{S}^T,\mathcal{A}^T,\mathbb{P}^T,r^T,H,\mu^T,\phi_T). We assume that the agent has complete access to the source MDP MS\mathcal{M}_S, either by knowing its dynamics and reward function exactly or by being allowed to interact with it arbitrarily without incurring any sample complexity cost. In contrast, interactions with the target MDP MT\mathcal{M}_T are costly. Further, the target MDP does not provide reward feedback during adaptation.

Shared Structure and Undo Map. We assume that both tasks are exactly the same except that the agent observes states through task-specific feature maps: ϕS:S→RdS\phi_S:\mathcal{S}\to\mathbb{R}^{d_S} in the source and ϕT:S→RdT\phi_T:\mathcal{S}\to\mathbb{R}^{d_T} in the target. These feature maps are related by an unknown transformation (the undo map) U⋆:RdT→RdSU^\star:\mathbb{R}^{d_T}\to\mathbb{R}^{d_S}, such that for all s∈Ss\in\mathcal{S}, ϕS(s)=U⋆(ϕT(s))\phi_S(s)=U^\star(\phi_T(s)). This relationship induces a correspondence between policies: for any policy πS:RdS→P(A)\pi_S:\mathbb{R}^{d_S}\to\mathcal{P}(\mathcal{A}) defined in the source feature space, we define the corresponding policy in the target task as πT(ϕT(s))=πS(U⋆(ϕT(s)))\pi_T(\phi_T(s))=\pi_S(U_\star(\phi_T(s))). These policies are equivalent in terms of their value functions: VSπS(s)=VTπT(s)V_S^{\pi_S}(s)=V_T^{\pi_T}(s) for all s∈Ss\in\mathcal{S}. This allows source policies to transfer to the target despite differing state representations.

Objective. Given complete access to the source MDP MS\mathcal{M}_S, our objective is to learn a near-optimal policy for the target MDP MT\mathcal{M}_T using as few samples from the target as possible.

Lumos: Learning to Undo by Matching Observation Statistics

In this section, we show how the undo map can be learned by leveraging the invariance of the action space across the source and target MDPs. First, we formally define a distributional measure to evaluate the quality of a candidate undo map § 3.1. We then propose a practical learning algorithm, LUMOS § 3.2, that matches observation statistics between the source MDP and the target MDP viewed through an undo map. This algorithm, illustrated in Figure 3, enables transfer of the source policy by composing it with the learned undo map. Theoretically, under certain assumptions, we show that LUMOS enjoys strictly better sample complexity than learning from scratch § 3.3.

Figure 3. Overview of LUMOS. We match state feature statistics between source and undone target trajectories.

Distributional Measure of Undo Map Quality

To estimate the undo map, we match state feature distributions induced by an open-loop sequence of actions in the source MDP MS\mathcal{M}_S with those induced by the undone trajectories in the target MDP MT\mathcal{M}_T. This leverages the fact that while the observations differ, the underlying dynamics remain consistent.

For an arbitrary undo map UU, we now introduce the notion of source and undone target state features.

Definition 3.1 (Source and Undone Target Features). Given an open-loop action sequence a0:h−1a_{0:h-1}, define the source feature random variable at timestep hh as

ψhS(a0:h−1)≜ϕS(sh),(1)\psi_h^S(a_{0:h-1}) \triangleq\phi_S(s_h), \tag*{(1)}

and, for an arbitrary undo map UU, define the undone target feature random variable as

ψh,UT(a0:h−1)≜U(ϕT(sh)),(2)\psi_{h,U}^T(a_{0:h-1}) \triangleq U(\phi_T(s_h)), \tag*{(2)}

where (s0,…,sh)(s_0,\ldots,s_h) is the state sequence obtained by executing a0,…,ah−1a_0,\ldots,a_{h-1} in the respective MDP.

Algorithm 1 Learning to Undo with LUMOS

  1. Input: source MS\mathcal{M}_{S}, target MT\mathcal{M}_{T}, trajectories nn, iterations NN, moments MM, truncated horizon H†H^{\dagger}

  2. Initialize undo map parameters ϕ\phi.

  3. for i∈[N]i \in[N] do

  4. Sample an action sequence a0:H†−1a_{0:H^{\dagger}-1} uniformly at random.

    ▷ Exactly compute source feature mean and marginal central moments.

  5. Compute mean μS(h)←E[ψhS(a0:h−1)]\mu_{S}(h) \leftarrow\mathbb{E}[\psi^{S}_{h}(a_{0:h-1})], ∀h∈[H†]\forall h \in[H^{\dagger}].

  6. Compute moments ck,S(h)←E[(ψhS(a0:h−1)−μS(h))⊙k]c_{k,S}(h) \leftarrow\mathbb{E}\left[(\psi^{S}_{h}(a_{0:h-1})-\mu_{S}(h))^{\odot k}\right], ∀k∈[M]\forall k \in[M], ∀h∈[H†]\forall h \in[H^{\dagger}].

    ▷ Estimate undone target feature mean and marginal central moments.

  7. Collect nn trajectories in MT\mathcal{M}_{T} using action sequence a0:H†−1a_{0:H^{\dagger}-1}.

  8. for h∈[H†]h \in[H^{\dagger}] do

  9. For each trajectory jj, compute undone target features ψh,UϕT,(j)(a0:h−1)\psi^{T,(j)}_{h,U_{\phi}}(a_{0:h-1}).

  10. Compute mean μ^T(h)←1n∑j=1nψh,UϕT,(j)(a0:h−1)\hat{\mu}_{T}(h) \leftarrow\frac{1}{n}\sum_{j=1}^{n}\psi^{T,(j)}_{h,U_{\phi}}(a_{0:h-1}).

  11. Compute moments c^k,T(h)←1n∑j=1n(ψh,UϕT,(j)(a0:h−1)−μ^T(h))⊙k\hat{c}_{k,T}(h) \leftarrow\frac{1}{n}\sum_{j=1}^{n}\left(\psi^{T,(j)}_{h,U_{\phi}}(a_{0:h-1})-\hat{\mu}_{T}(h)\right)^{\odot k}, ∀k∈[M]\forall k \in[M].

    ▷ Minimize empirical undo discrepancy.

  12. Update ϕ\phi to minimize ∑h=0H†−1[∥μS(h)−μ^T(h)∥2+∑k=2M∥ck,S(h)−c^k,T(h)∥2]\sum_{h=0}^{H^{\dagger}-1}\left[\lVert\mu_{S}(h)-\hat{\mu}_{T}(h)\rVert_{2}+\sum_{k=2}^{M}\lVert c_{k,S}(h)-\hat{c}_{k,T}(h)\rVert_{2}\right].

  13. Output: undo map parameters ϕ\phi

By the definition of the true undo map U⋆U_{\star}, we have for all h∈[H]h \in[H],

ψhS(a0:h−1)=dψh,U⋆T(a0:h−1),(3)\psi^{S}_{h}(a_{0:h-1}) \mathrel{\overset{d}{=}} \psi^{T}_{h,U_{\star}}(a_{0:h-1}), \tag*{(3)}

where =d\overset{d}{=} denotes equality in distribution, which means that applying U⋆U_{\star} to the target observations recovers exactly the source distribution, conditioned on any open-loop action sequence a0:h−1a_{0:h-1}.

Motivated by Eq. (3), our goal is to evaluate a candidate undo map UU by measuring how closely the undone target features match the source features in distribution, under open-loop rollouts. To quantify distributional mismatch, we use the Central Moment Discrepancy (CMD) metric [28], which compares distributions via their means and central moments.

Definition 3.2 (Central Moment Discrepancy (CMD) [28]). Let pp and qq be probability distributions supported on a compact domain [a,b]N[a,b]^{N}, and let X∼pX \sim p and Y∼qY \sim q be bounded random vectors. The Central Moment Discrepancy (CMD) between pp and qq is defined as

CMD⁡(p,q)≜1∣b−a∣∥E[X]−E[Y]∥2+∑k=2∞1∣b−a∣k∥ck(X)−ck(Y)∥2,(4)\operatorname{CMD}(p,q) \triangleq\frac{1}{\lvert b-a\rvert}\lVert\mathbb{E}[X]-\mathbb{E}[Y]\rVert_{2}+\sum_{k=2}^{\infty}\frac{1}{\lvert b-a\rvert^{k}}\lVert c_{k}(X)-c_{k}(Y)\rVert_{2}, \tag*{(4)}

where ck(⋅)c_{k}(\cdot) denotes the kk-th order central moment vector.

CMD⁡(p,q)=0\operatorname{CMD}(p,q)=0 iff p=qp=q. Moreover, if CMD⁡(pn,p)→0\operatorname{CMD}(p_{n},p)\to0, then pn⟶dpp_{n}\overset{d}{\longrightarrow}p, where ⟶d\overset{d}{\longrightarrow} denotes convergence in distribution [28].

For notational convenience, given random variables XX and YY, we write dCM(X,Y)d_{\mathrm{CM}}(X,Y) to denote the CMD between the distributions induced by XX and YY. We now use this to define a measure for the quality of an arbitrary undo map.

Figure 4. We evaluate LUMOS under structured transformations ranging from geometric and sensor-level changes to gaps in the level of abstraction (e.g., semantic maps in the source and pixels in the target), where the source task is often easier to learn than the target.

Definition 3.3 (Undo Discrepancy). For an undo map UU, we define its discrepancy as

D(U)≜Ea0:H−1[∑h=0H−1dCM(ψhS(a0:h−1),ψh,UT(a0:h−1))].(5)\mathcal{D}(U) \triangleq\mathbb{E}_{a_{0:H-1}}\left[\sum_{h=0}^{H-1}d_{\mathrm{CM}}\left(\psi^S_h(a_{0:h-1}),\psi^T_{h,U}(a_{0:h-1})\right)\right]. \tag*{(5)}

where ψhS(a0:h−1)\psi^S_h(a_{0:h-1}) and ψh,UT(a0:h−1)\psi^T_{h,U}(a_{0:h-1}) denote the source and undone target feature random variables at timestep hh, and the expectation is over uniformly sampled action sequences.

Thus, minimizing D(U)\mathcal{D}(U) is equivalent to matching the undone feature distributions in the target to the corresponding feature distributions in the source at each timestep for open-loop sequences of actions.

Learning the Undo Map

With a distributional measure of undo map quality defined, we now describe how to learn the map.

Empirical Undo Discrepancy. To empirically estimate D(U)\mathcal{D}(U) from samples, we approximate CMD using only the first MM moments and compute empirical marginal central moments across state feature coordinates. Concretely, given samples {xi}i=1n∼p\{x_i\}_{i=1}^n \sim p, the kk-th order empirical marginal central moment vector, c^k(p)\hat{c}_k(p), is given by 1n∑i=1n(xi−μ^p)⊙k\frac{1}{n}\sum_{i=1}^n(x_i-\hat{\mu}_p)^{\odot k}, where (⋅)⊙k(\cdot)^{\odot k} denotes element-wise exponentiation and μ^p=1n∑i=1nxi\hat{\mu}_p=\frac{1}{n}\sum_{i=1}^n x_i is the empirical mean.

Learning Algorithm. We parameterize the undo map as UϕU_\phi and optimize its parameters ϕ\phi to match state feature distributions in the source and the undone target. More concretely, we minimize the empirical undo discrepancy D^(Uϕ)\widehat{\mathcal{D}}(U_\phi), defined via a truncated empirical CMD estimator using the first MM marginal central moments. Our algorithm samples multiple action sequences and optimizes these moments over the resulting rollouts. As samples from the source MDP MS\mathcal{M}_S are free, we can compute these moments in the source exactly, while they are empirically estimated using samples from the target MDP MT\mathcal{M}_T. Moreover, we use a truncated horizon H†<HH^\dagger< H to allow more action sequences to be sampled for training under a given budget of samples; we justify this choice in § 4.

Algorithm 1 gives the pseudocode for our algorithm, LUMOS. In summary, we iteratively sample an action sequence, compute marginal central moments of source state features, estimate marginal central moments of the corresponding undone target state features via nn trajectories, and update ϕ\phi to minimize the empirical discrepancy. In practice, the source expectations are also estimated from samples. We then transfer the source policy to the target by composing it with the learned undo map.

Theoretical Analysis

In this section, we show that LUMOS is more sample efficient than learning from scratch. For this analysis, we specialize our formalism to state-action features used to represent the optimal QQ-function. We assume that the source and target state-action features are related by an undo map. Recovering it allows the source optimal QQ-function (and therefore its greedy policy) to be transferred to the target.

We first list the assumptions used in our analysis. The source and target MDPs have feature maps ϕS:SS×A→RdS\phi_S : \mathcal{S}^S \times\mathcal{A} \to\mathbb{R}^{d_S} and ϕT:ST×A→RdT\phi_T : \mathcal{S}^T \times\mathcal{A} \to\mathbb{R}^{d_T} that represent state–action pairs as dSd_S and dTd_T-dimensional vectors. Let ΦS∈R∣SS×A∣×dS\Phi_S \in\mathbb{R}^{|\mathcal{S}^S \times\mathcal{A}| \times d_S} and ΦT∈R∣ST×A∣×dT\Phi_T \in\mathbb{R}^{|\mathcal{S}^T \times\mathcal{A}| \times d_T} denote the corresponding feature matrices, with rows ϕS(s,a)\phi_S(s,a) and ϕT(s,a)\phi_T(s,a). We assume that the feature vectors in both MDPs satisfy ∥ϕ(s,a)∥2≤1\|\phi(s,a)\|_2 \leq1, and that rewards are bounded within the range [−1,1][-1,1]. We further assume that the source is linearly-Q⋆Q^\star realizable.

Assumption 3.4 (Linear-Q⋆Q^\star realizability in MS\mathcal{M}_S [[25]). There exists a vector θS⋆∈RdS\theta^\star_S \in\mathbb{R}^{d_S} such that for every state–action pair (s,a)∈SS×A(s,a) \in\mathcal{S}^S \times\mathcal{A} in the source MDP MS\mathcal{M}_S,

QS⋆(s,a)=⟨ϕS(s,a),θS⋆⟩.Q^\star_S(s,a) = \langle\phi_S(s,a), \theta^\star_S \rangle.

In this setup, given an action sequence a0:h−1a_{0:h-1}, we define ψh(a0:h−1)\psi_h(a_{0:h-1}) to be the state-action feature random variable at timestep h−1h-1:

ψh(a0:h−1)≜ϕ(sh−1,ah−1).(6)\psi_h(a_{0:h-1}) \triangleq\phi(s_{h-1},a_{h-1}). \tag*{(6)}

The source and the target are exactly the same except for the state-action features, which are related via a linear undo map U⋆U_\star: ΦS=ΦTU⋆T\Phi_S = \Phi_T U_\star^T. We assume that the ℓ2\ell_2 norm of each row of U⋆U_\star is upper bounded by a constant. In the source, the learner has access to the parameter vector θS⋆\theta^\star_S for the state-action values of the optimal policy, and an oracle O\mathcal{O} for the expected state-action features.

Assumption 3.5 (Oracle for expected state-action features in MS\mathcal{M}_S). We denote by O\mathcal{O} an oracle that when queried with a0:h−1a_{0:h-1} and h∈[H]h \in[H], outputs ψˉhS(a0:h−1)≜E[ψhS(a0:h−1)]\bar{\psi}^S_h(a_{0:h-1}) \triangleq\mathbb{E}[\psi^S_h(a_{0:h-1})].

We would like to note that this assumption is only for convenience; the same analysis holds when the source expectations are estimated empirically via random-design linear regression arguments.

Since the source and target MDPs differ only through a linear transformation of the state-action features, the expected state-action features in the source and the target (denoted by ψˉhS(a0:h−1)\bar{\psi}^S_h(a_{0:h-1}) and ψˉhT(a0:h−1)\bar{\psi}^T_h(a_{0:h-1}), respectively) are related via the undo map U⋆U_\star as follows:

ψˉhS(a0:h−1)=U⋆ψˉhT(a0:h−1), ∀h∈[H].(7)\bar{\psi}^S_h(a_{0:h-1}) = U_\star\bar{\psi}^T_h(a_{0:h-1}), \ \forall h \in[H]. \tag*{(7)}

Remark. When the undo map is linear, higher-order central moments of the transformed features are completely determined by the first-order statistics through the same linear operator. Therefore, matching higher-order moments provides no additional constraints beyond mean matching.

Learning Algorithm. Our learning algorithm first queries O\mathcal{O} to obtain ψˉhS(a0:h−1)\bar{\psi}^S_h(a_{0:h-1}) in MS\mathcal{M}_S, for all h∈[H]h \in[H]. Next, it rolls out nn trajectories in the target using the open-loop action sequence a0:H−1a_{0:H-1}. It then views the state-action features ψhT,(i)(a0:h−1)\psi^{T,(i)}_h(a_{0:h-1}) (from the ii-th sample trajectory) as noisy features for ψˉhS(a0:h−1)\bar{\psi}^S_h(a_{0:h-1}). Subsequently, it estimates U⋆U_\star with U^\widehat{U} via least squares.

Policy Transfer. From Eq. (7), the relation QT⋆=ΦTU⋆TθS⋆Q^\star_T = \Phi_T U_\star^T \theta^\star_S holds. Thus, a natural way to evaluate the output U^\widehat{U} of LUMOS is to bound the performance difference with respect to the optimal policy in the target, if we act greedily with respect to ΦTU^TθS⋆\Phi_T \widehat{U}^T \theta^\star_S. Let π^\hat{\pi} denote this policy.

We show that our algorithm has the following guarantee; the proof is provided in Appendix § 7.

Theorem 3.6 (Performance Difference Bound for LUMOS). Define the covariance matrix

Σ≜1H∑h=1Hψˉh(a0:h−1)ψˉh(a0:h−1)T,\Sigma\triangleq\frac{1}{H}\sum_{h=1}^{H}\bar{\psi}_{h}(a_{0:h-1})\bar{\psi}_{h}(a_{0:h-1})^{\mathsf{T}},

and let λmin⁡\lambda_{\min} denote λmin⁡(Σ)\lambda_{\min}(\Sigma).

For all n≥6Hλmin⁡(log⁡3dTdSδ)n \ge\frac{6}{H\lambda_{\min}}\left(\log\frac{3d_{\mathsf{T}}d_{\mathsf{S}}}{\delta}\right), with probability at least 1−δ1-\delta, the performance difference satisfies

∥VT⋆−V~T∥∞≤O~(Hnλmin⁡dT+2log⁡(3dSδ)).\left\|V^{\star}_{\mathsf{T}}-\widetilde{V}_{\mathsf{T}}\right\|_{\infty}\le\widetilde{O}\left(\sqrt{\frac{H}{n\lambda_{\min}}}\sqrt{d_{\mathsf{T}}+2\log\left(\frac{3d_{\mathsf{S}}}{\delta}\right)}\right).

The theorem shows that if the open-loop action sequence a0:H−1a_{0:H-1} is good, i.e., λmin⁡(Σ)\lambda_{\min}(\Sigma) is sufficiently large (on the order of 1/dT1/d_{\mathsf{T}}), then LUMOS achieves a sample complexity upper bound that scales as O~(dTH)\widetilde{O}(d_{\mathsf{T}}\sqrt{H}). This improves upon the lower bound for linearly-Q⋆Q^{\star} realizable MDPs, which scales exponentially with the feature dimension dTd_{\mathsf{T}} or the horizon HH [25].

Remark on Linear MDPs. A special case of MDPs with linear-Q⋆Q^{\star} realizability is linear MDPs [13]. In this setting, the minimax lower bound has a dTH3/2d_{\mathsf{T}}H^{3/2} dependence on dTd_{\mathsf{T}} and HH [9], whereas LUMOS enjoys a dTHd_{\mathsf{T}}\sqrt{H} dependence, up to logarithmic factors.

Experimental Evaluation

In this section, we empirically evaluate our method to address the following research questions: (i) Does transfer via LUMOS outperform learning from scratch? Concretely, does learning the undo map U⋆U_{\star} yield better target policies than single-task training? (ii) How does LUMOS compare with transfer learning baselines that assume reward feedback in the target, which is an orthogonal setting to ours? (iii) Does matching higher-order moments contribute to performance gains? (iv) How should the training budget be allocated between the number of iterations NN and the truncated horizon H†H^{\dagger}?

We begin by describing the transfer settings. Next, we provide an overview of the training process, followed by empirical results. We additionally analyze the robustness of LUMOS in Appendix § 10.

Environments

We evaluate our method on a suite of continuous control tasks and Atari games that are challenging to learn from scratch. As source tasks, we consider standard MuJoCo environments: Swimmer, HalfCheetah, Walker2d, Ant, and CartPole, as well as Atari games: Breakout and SpaceInvaders. We construct corresponding target tasks by applying structured transformations to the state space. Detailed descriptions are provided in Appendix § 8.

Training Procedure

We use the rl-baselines3-zoo framework [18] to train both the source policies and the target policies used in the learning-from-scratch baseline, with PPO [20] as the learning algorithm. For each environment, we adopt the tuned hyperparameters provided in the framework. Our Atari experiments are based on the MinAtar benchmark [27].

To estimate the state feature means and marginal central moments for undo discrepancy, we use a large number of samples from the source MDP to obtain accurate statistics, while using fewer samples from the target. We employ multiple short action sequences for good coverage of the features.

The undo map is parameterized by a neural network and trained via LUMOS. Additional implementation details are provided in Appendix § 9.

Results

In this section, we first compare our method, LUMOS, against learning the target policy from scratch (listed as SCRATCH). Next, we evaluate the contribution of higher-order moments to performance gains. Finally, we examine the effect of using a truncated horizon H†H^\dagger.

Results with LUMOS. We evaluate LUMOS in transfer settings that go beyond the regime of our theoretical analysis. Figure 5 and Figure 6 report the average return (with standard deviation over five random seeds) as a function of the target sampling budget. We stop LUMOS early upon reaching near-source performance, after which the dashed line extends the final performance to larger budgets.

Figure 5. Comparison of avg. return (standard deviation over five random seeds) for varying budgets of target samples. Transfer learning via LUMOS consistently achieves significantly better sample efficiency than learning from scratch.

Figure 6. Comparison of avg. return (standard deviation over five random seeds) for varying budgets of target samples. Transfer learning via LUMOS consistently achieves significantly better sample efficiency than learning from scratch.

Across all tasks, LUMOS achieves consistent and substantial improvements over single-task training (SCRATCH). The FUSION and PIXEL targets are especially challenging for learning from scratch, due to the increased observation dimensionality and the fact that useful information must be recovered through structured aggregation (across sensors in FUSION, or from high-dimensional visual inputs in PIXEL). In contrast, LUMOS explicitly exploits this shared structure between source and target to achieve near-optimal performance with few target samples.

Overall, these results provide empirical evidence that undo-map-based transfer leads to sample-efficiency gains in RL under structured state-space transformations.

Comparison with transfer RL baselines. While our setup assumes no reward feedback in the target, for completeness we compare LUMOS against transfer RL baselines that operate with reward feedback in the target. Figure 7 includes a standard transfer baseline (FINETUNE) that initializes the target policy from the source policy (except for the input layer when dimensions differ). We also evaluate MBReg [21], a baseline that learns a reward and dynamics model over latent states in the source and uses it for model regularization in the target. Despite the favorable conditions afforded to these baselines (namely, access to reward feedback in the target), LUMOS consistently outperforms them across a variety of environments.

Figure 7. Comparison of avg. return (standard deviation over five random seeds). LUMOS consistently achieves better sample efficiency than transfer baselines that require rewards in the target.

Ablation w.r.t. the number of moments. We use PIXELCARTPOLE to study the effect of matching higher-order central moments when learning the undo map. Specifically, we run LUMOS with M=1M = 1 (mean matching only) and M=2M = 2 (including second-order moments). Figure 8(a) shows that using higher-order moments yields clear performance improvements.

Figure 8. Ablation results. (a) Effect of the number of matched moments MM. (b) Effect of allocating a fixed sampling budget between the number of action sequences NN and their truncated horizon H†H^{\dagger}.

Ablation w.r.t. the truncated horizon. We use PIXELCARTPOLE to justify our algorithmic choice of truncating trajectories at horizon H†H^\dagger. Keeping the overall sampling budget fixed, we vary the allocation between the number of sampled action sequences NN and their length H†H^\dagger. Concretely, we consider the configurations (N,H†)∈{(5,500),(50,50),(500,5)}(N, H^\dagger) \in\{(5, 500), (50, 50), (500, 5)\}, with the product NH†NH^\dagger kept constant. Figure 8(b) indicates that moderately short sequences perform best: using shorter sequences improves performance relative to very long rollouts (e.g., (50, 50) vs. (5, 500)), while overly short horizons can hurt due to potentially insufficient state-space coverage (e.g., (500, 5)).

Related Work

Transfer Learning. Empirical methods for transfer learning in RL often lack provable guarantees or rely on application-specific assumptions that do not generalize. For instance, Sun et al. [2022] assume shared latent dynamics across observation spaces and that target observations can be transformed into the source domain; they do not learn this mapping explicitly and offer no theoretical guarantees on transfer performance. Similarly, Watahiki et al. [2024] assume a shared latent MDP structure between tasks but their method is not theoretically grounded. Other works, such as Agarwal et al. [2022], study transfer through reusing prior policies across design iterations, which is orthogonal to our setting. Methods like Chen et al. [2024], Yi et al. [2023] rely on very specific visual or object-level assumptions, which limits their general applicability. The notion of an undo map to reverse state space transformations in reinforcement learning was first introduced in Gupta et al. [2022], where the problem is approached through a distributional lens by aligning trajectories across the source and target tasks. However, the method does not scale beyond toy tabular environments. Orthogonal to our setup, works such as Qu et al. [2025], Wagenmaker et al. [2024] study transfer under dynamics shift.

Representational Transfer in Low-Rank MDPs. Several works have studied representational transfer under low-rank or linear MDP assumptions. For instance, Agarwal et al. [2023], Cheng et al. [2022] use reward-free exploration in the source task to learn a good representation for the target. Sam et al. [2024] extend this to a more general low-rank setting. Lu et al. [2021] learn a linear representation using least squares for multitask linear MDPs, and Ishfaq et al. [2024] consider the offline multitask case. Chai et al. [2025] provide guarantees for transfer Q-learning in the composite MDP setting, but their method does not scale empirically and only works for finite state-action spaces.

In contrast, our work approaches transfer RL through a more principled lens by making concrete assumptions about the structural relationship between the source and target tasks. We propose an algorithm that explicitly learns a mapping between their observation spaces by framing the problem as supervised learning, and therefore eliminates the need for RL in the target task. Under certain assumptions, this leads to provable improvements in sample complexity over learning from scratch. Moreover, our method is practical and demonstrates strong empirical performance. Essentially, our work takes a step toward more principled and broadly applicable approaches to transfer in RL.

Our goal is not to develop a state-of-the-art transfer RL method competing across benchmarks, but rather to design a theoretically grounded approach that is provably better than learning from scratch when suitable shared structure exists, and is also empirically better than single-task RL.

Concluding Discussion

In this paper, we introduced a principled approach to transfer learning in reinforcement learning, where the source and target tasks are related by structured transformations of the state space. By explicitly learning an undo map, our method achieves significant gains in sample efficiency compared to learning target policies from scratch, as demonstrated across a range of challenging continuous control tasks and Atari games. Theoretically, we showed that when the undo map is linear and the source is linearly-Q∗Q^{*} realizable, our approach achieves strictly better sample complexity than learning from scratch. A promising direction for future work is to study more general notions of task equivalence, such as MDP homomorphisms where the action space may also be transformed. Extending our guarantees beyond the linear realizability assumption is another interesting direction.

Broader impact. Our work is algorithmic, and we do not see any direct societal implications.

Funding. No external funding was received.

Competing Interests. The authors declare no competing interests.

In this section, we provide an outline of the contents provided in the paper’s appendices.

  • Appendix B contains the proof of Theorem 3.6.

  • Appendix C describes the environments used in the experimental evaluation.

  • Appendix D provides additional implementation details for the experimental evaluation.

  • Appendix E provides additional results examining the robustness of LUMOS.

Proofs

In this section, we provide the proof of Theorem 3.6.

Proof Structure. The overall structure of our proof is:

  1. View the estimation of each row of U^\widehat{U} as a random-design linear regression problem and bound its covariance-weighted estimation error.

  2. Use λmin⁡(Σ)\lambda_{\min}(\Sigma) to convert this covariance-weighted error into a max-norm bound on U^−U⋆\widehat{U}-U_\star.

  3. Use the relation QT⋆=ΦTU⋆⊤θS⋆Q_T^\star=\Phi_T U_\star^\top\theta_S^\star to translate undo-map estimation error into error in the approximate target QQ-function Q^T=ΦTU^⊤θS⋆\widehat{Q}_T=\Phi_T\widehat{U}^\top\theta_S^\star.

  4. Apply a standard performance bound for the policy greedy with respect to Q^T\widehat{Q}_T to obtain the final value-difference guarantee.

We first relate the performance difference between the optimal policy in the target MT\mathcal{M}_T and the policy that acts greedily with respect to ΦTU^⊤θS⋆\Phi_T\widehat{U}^\top\theta_S^\star to the estimation error in U^\widehat{U}, where U^\widehat{U} estimates U⋆U_\star.

Lemma B.1 (Performance Difference). ∥VT⋆−VTπ^∥∞≤2H∥U^−U⋆∥max⁡\lVert V_T^\star-V_T^{\widehat{\pi}}\rVert_\infty\le2H\lVert\widehat{U}-U_\star\rVert_{\max}.

Here, ∥⋅∥max⁡\lVert\cdot\rVert_{\max} denotes the max norm. To bound the estimation error ∥U^−U⋆∥max⁡\lVert\widehat{U}-U_\star\rVert_{\max}, we use the following per-row guarantee:

Lemma B.2 (Estimation Error Bound). Let M[i]M[i] denote the ii-th row of a matrix MM. Fix i∈[dS]i\in[d_S]. Then, for all n≥6Hλmin⁡(log⁡3dTδ)n\ge\frac{6}{H\lambda_{\min}}\left(\log\frac{3d_T}{\delta}\right), with probability at least 1−δ1-\delta, the output U^\widehat{U} of LUMOS satisfies:

∥U^[i]−U⋆[i]∥Σ≤1nHdT+2log⁡(3δ)+2dTlog⁡(3δ)+o(1nH).\lVert\widehat{U}[i]-U_\star[i]\rVert_\Sigma\le\sqrt{\frac{1}{nH}}\sqrt{d_T+2\log\left(\frac{3}{\delta}\right)}+2\sqrt{d_T\log\left(\frac{3}{\delta}\right)}+o\left(\sqrt{\frac{1}{nH}}\right).

Since Σ\Sigma is positive semidefinite (PSD), we can relate the Σ\Sigma-norm to the ℓ∞\ell_\infty norm via:

∥x∥∞≤∥x∥Σλmin⁡.(8)\lVert x\rVert_\infty\le\frac{\lVert x\rVert_\Sigma}{\sqrt{\lambda_{\min}}}. \tag*{(8)}

Applying a union bound over all i∈[dS]i\in[d_S], with per-row failure probability set to δ/dS\delta/d_S, gives a high-probability bound on ∥U^−U⋆∥max⁡\lVert\widehat{U}-U_\star\rVert_{\max}. Substituting this into Lemma B.1 concludes the proof.

Proof of Lemma B.1. Let ff denote the approximate target QQ-function induced by the estimated undo map: f(s,a)≜⟨U^ϕT(s,a),θS⋆⟩f(s,a)\triangleq\left\langle\widehat{U}\phi_T(s,a),\theta_S^\star\right\rangle. For any s∈STs\in S^T,

VT⋆(s)−VTπ^(s)=QT⋆(s,π⋆(s))−QT⋆(s,π^(s))+QT⋆(s,π^(s))−QTπ^(s,π^(s))≤QT⋆(s,π⋆(s))−f(s,π⋆(s))+f(s,π^(s))−QT⋆(s,π^(s))+Es′∼PT(s,π^(s))[VT⋆(s′)−VTπ^(s′)]≤2∥f−QT⋆∥∞+Es′∼PT(s,π^(s))[VT⋆(s′)−VTπ^(s′)].\begin{aligned} V_T^\star(s)-V_T^{\widehat{\pi}}(s) &=Q_T^\star(s,\pi^\star(s))-Q_T^\star(s,\widehat{\pi}(s))+Q_T^\star(s,\widehat{\pi}(s))-Q_T^{\widehat{\pi}}(s,\widehat{\pi}(s))\\ &\le Q_T^\star(s,\pi^\star(s))-f(s,\pi^\star(s))+f(s,\widehat{\pi}(s))-Q_T^\star(s,\widehat{\pi}(s))\\ &\qquad+\mathbb{E}_{s'\sim P_T(s,\widehat{\pi}(s))}\left[V_T^\star(s')-V_T^{\widehat{\pi}}(s')\right]\\ &\le2\lVert f-Q_T^\star\rVert_\infty+\mathbb{E}_{s'\sim P_T(s,\widehat{\pi}(s))}\left[V_T^\star(s')-V_T^{\widehat{\pi}}(s')\right]. \end{aligned}

Unrolling this recursion for HH steps,

∥VT⋆−VTπ^∥∞≤2H∥f−QT⋆∥∞.\left\|V_T^\star- V_T^{\hat{\pi}}\right\|_\infty\le2H\left\|f-Q_T^\star\right\|_\infty.

Substituting f=ΦTU^TθS⋆f=\Phi_T\hat{U}^{\mathsf{T}}\theta_S^\star and QT⋆=ΦTU⋆θS⋆Q_T^\star=\Phi_TU^\star\theta_S^\star completes the proof.

Proof of Lemma B.2. Fix a source-feature coordinate i∈[dS]i\in[d_S]. Estimating U⋆[i]U_\star[i] can be viewed as a random-design linear regression problem in which the target state-action features (of dimensionality dTd_T) are the covariates and the corresponding expected source-feature coordinate is the response. The regression parameter is U⋆[i]U_\star[i], and the covariance matrix is Σ\Sigma. Hsu et al. [2014] bound the covariance-weighted estimation error of least squares under three conditions.

In our setting:

  1. The regression noise is bounded in [−1,1][-1,1] and is therefore 1-subgaussian, so we set σ=1\sigma=1.

  2. The linear model is correctly specified through ψhS=U⋆ψhT\psi_h^S=U_\star\psi_h^T. Therefore, there is no approximation bias, and we set b0=0b_0=0.

  3. Since the ℓ2\ell_2-norm of ϕT(s,a)\phi_T(s,a) is at most 1 and the minimum eigenvalue of Σ\Sigma is λmin⁡\lambda_{\min}, the statistical leverage parameter ρ0\rho_0 satisfies ρ0≤1λmin⁡dT\rho_0\le\frac{1}{\sqrt{\lambda_{\min}d_T}} by Remark 2 of [10].

Substituting these quantities into Theorem 1 and Remark 9 of [10] completes the proof.

Environments

Types of Transformations

Changed frame of reference. Here, we rotate the global coordinate frame in the source tasks, which results in rotated variants of the MuJoCo tasks (e.g., ROTATEDANT). This models geometric discrepancies that arise, for example, from changes in coordinate conventions across simulators.

Sensor fusion. In this setting, multiple sensor modalities are available, which results in sensor-fused variants (e.g., FUSIONANT). Instead of observing a dSd_S-dimensional vector, the agent receives a 10dS10d_S-dimensional observation (10×10\times sensors), which can be combined to recover the original values. This captures settings where sensors differ at deployment.

Changed coordinate system. We construct polar-coordinate variants POLARHALFCHEETAH and POLARANT, where sensor readings are expressed in polar rather than Cartesian coordinates. This captures settings with differing state parameterizations.

Sensor-to-pixel observations. We consider CARTPOLE with sensor observations as the source task and pixel-based observations as the target, which results in PIXELCARTPOLE. This introduces an abstraction gap where the source provides structured low-dimensional inputs, while the target requires learning directly from raw visual observations.

Color transformation. For Atari games, we treat grayscale observations as the source and RGB observations as the target, which results in the variants COLORBREAKOUT and COLORSPACEINVADERS. This setting models rendering shifts at deployment time.

Semantic bitmaps to pixels. Finally, we consider a setting where the source provides object-centric semantic bitmaps, but the target provides raw pixel observations (PIXELBREAKOUT). This represents an abstraction gap where learning from scratch is significantly harder.

Environment-Specific Details

ROTATEDSWIMMER. This task requires the agent to swim forward through a viscous fluid. We apply a 45∘45^\circ rotation to the (x,y)(x,y) plane to modify the frame in which the agent’s motion is observed. This affects both the heading of the swimmer and its linear velocity vector. The observation is updated accordingly by adjusting the front tip’s angle and rotating the (x,y)(x,y) velocity components.

ROTATEDHALFCHEETAH. This task requires the agent to run forward in a planar environment. The original environment operates in a 2D plane. We perform a 45∘45^\circ rotation on the (x,z)(x,z) components of the agent’s torso position and velocity. The joint states and angular velocities remain unchanged.

RotatedWalker2D. This task requires the agent to walk forward while maintaining balance. Similar to HalfCheetah, we rotate the observation of the agent’s torso height and linear velocity in the (x,z)(x,z) plane by 45∘45^\circ. This alters how progress and vertical motion are perceived, while the actuator and sensor states are kept intact.

RotatedAnt. In this task, the agent is a quadruped and must navigate a 2D surface using four legs. We rotate the global (x,y)(x,y) velocity of the torso by 45∘45^\circ. When present, external contact forces are also rotated in the same plane. The result is a consistent shift in perceived motion direction across the ant’s high-dimensional observation space.

Sensor Fusion. In the fusion environments, instead of observing a dSd_S-dimensional observation vector, the agent receives a 10×dS10 \times d_S-dimensional observation. More concretely, for each source observation element ii, the target task provides 10 measurements of the form: xi(j)=wj⋅xi+cjx_i^{(j)} = w_j \cdot x_i + c_j, for j∈[10]j \in[10], where the weights wjw_j sum to 1 and the offsets cjc_j sum to 0. This simulates a sensor fusion scenario in which the undo map U⋆U_\star sums the 10 measurements corresponding to each original dimension to recover the source observation.

PolarHalfCheetah. Here, we change the coordinate system of selected observation dimensions from Cartesian to polar coordinates. In particular, pairs of state variables such as the angular velocities of the back thigh and shin are transformed so that their joint (x,y)(x,y) representation becomes (r,θ)(r,\theta), where r=x2+y2r = \sqrt{x^2+y^2} and θ=arctan⁡2(y,x)\theta= \arctan2(y,x). This alters how the agent perceives limb velocities.

PolarAnt. In this task, the angular velocity components of the torso along the xx and yy axes are represented in polar form. This affects how rotational motion is encoded.

PixelCartPole. In this task, the agent receives pixel observations instead of low-dimensional sensor states. We extract a cropped 40×6040 \times60 image centered around the cart, and the observation is the difference between two consecutive frames to capture dynamics. Since the cart’s absolute position and velocity cannot be inferred from pixels alone, we mask those variables from the source task to ensure comparability. This setting, inspired by Sun et al. [ [21], evaluates whether our method can transfer across a modality shift from sensor states to pixels.

ColorBreakout and ColorSpaceInvaders. In these tasks, the source provides grayscale observations while the target provides RGB.

PixelBreakout. In this task, the source provides semantic bitmaps corresponding to the ball, bricks, paddle, and trail, whereas the target provides grayscale pixel observations.

Implementation Details

In our implementation, if an episode terminates early, we reset the environment but continue the action sequence indexing. This ensures that we can always collect HtH^t steps from an action sequence. The action sequences used to compute the state feature statistics are either randomly sampled or derived from the source policy πS\pi_S. Concretely, we perform a rollout of πS\pi_S in the source MDP and record the action sequence taken by the policy. We then use this recorded sequence. One can potentially include both kinds of action sequences, which makes this procedure task-independent and effectively combines the benefits of both sampling strategies.

Rotation and Fusion Tasks. The undo map is a single linear layer, and we use LUMOS with M=1M=1 (learning rate 0.001, weight decay 0.0001, 10,000 epochs). For Swimmer and Walker2D, we use randomly sampled action sequences of lengths 8 and 17, respectively, with 200 and 100 source trajectories and 3 target trajectories per sequence. For HalfCheetah and Ant, the action sequences are obtained from the source policy (with lengths 17 and 27, respectively, and 100 and 60 source trajectories and 3 target trajectories per sequence). We use up to 600 action sequences.

Polar Tasks. The undo map is an MLP with two hidden layers (64 units per layer), and we use LUMOS with M=1M=1 (learning rate 0.001, weight decay 0.0001, 10,000 epochs). We obtain 100 action sequences of length 100 from the source policy. Target statistics are computed from up to 10 (PolarHalfCheetah) and 20 (PolarAnt) trajectories per sequence.

Sensor-to-Pixel Task. The undo map is an MLP with one hidden layer (16 units), and we use LUMOS with M=2M=2 (learning rate 0.001, weight decay 0.0001, 10,000 epochs). We use 50 randomly sampled action sequences of length 50. Target statistics are estimated with up to 20 trajectories per sequence.

Color Tasks. The undo map is a single linear layer, and we use LUMOS with M=1M=1 (learning rate 0.01, ℓ1\ell_1 penalty of 0.001, 3,000 epochs). We use 10 sequences of length 300 (COLORBREAKOUT) or 100 (COLORSPACEINVADERS), obtained from the source policy. Target statistics are estimated with up to 50 (COLORBREAKOUT) and 100 (COLORSPACEINVADERS) trajectories per sequence.

Bitmaps-to-Pixel Task. The undo map is an MLP with one hidden layer (2048 units), and we use LUMOS with M=2M=2 (learning rate 0.001, weight decay 0.0001, 2,000 epochs). Here, we use 20 source-policy action sequences of length 200, with up to 50 target trajectories per sequence.

More details are provided in the codebase. For the transfer baselines, we use the default hyperparameters provided in the original codebases.

Compute resources. All experiments were conducted on nodes of a shared cluster with Intel Xeon Gold 6426Y CPUs (up to 4.1 GHz, 250 GB RAM per node) and NVIDIA L40S GPUs (46 GB VRAM). Each experiment took approximately 5 to 15 minutes per seed.

Robustness Analysis

We conduct additional robustness experiments on PIXELCARTPOLE using five random seeds. PIXELCARTPOLE is a nontrivial transfer setting because the source policy operates on low-dimensional sensor observations, whereas the target provides only high-dimensional rendered pixel observations.

We consider two stochastic observation corruptions for which an exact deterministic (memoryless) undo map generally does not exist.

  1. Random Pixel Masking: Each target pixel is independently set to zero with probability pmp_m.

  2. Sticky Frames: The previous observation is emitted instead of the current with probability psp_s.

Table 1 reports the results at multiple target-sample budgets for the individual corruptions, as well as for a combined stochastic corruption consisting of pixel masking [pm=0.05p_m = 0.05] and sticky frames [ps=0.01p_s = 0.01]. All results are averaged over five random seeds.

Setting12.5k25k37.5k50k
Exact transformation86.12 ± 18.20114.20 ± 6.31115.08 ± 5.85118.96 ± 3.82
Pixel masking [pm=0.05p_m = 0.05]79.08 ± 11.5199.96 ± 13.0899.52 ± 7.00107.28 ± 13.02
Pixel masking [pm=0.10p_m = 0.10]70.00 ± 20.5576.64 ± 9.7687.56 ± 8.7891.36 ± 13.63
Sticky frames [ps=0.01p_s = 0.01]90.56 ± 9.6795.76 ± 16.62104.12 ± 9.05111.76 ± 5.75
Sticky frames [ps=0.10p_s = 0.10]48.04 ± 13.2850.80 ± 14.7649.40 ± 9.0557.68 ± 10.52
Combined [pm=0.05p_m = 0.05, ps=0.01p_s = 0.01]74.04 ± 17.1091.92 ± 27.1198.88 ± 13.2298.80 ± 19.18

Table 1. Average return (standard deviation over five random seeds) on PIXELCARTPOLE under stochastic observation corruptions at different target-sample budgets.

Under 5% pixel masking and 1% sticky frames, LUMOS remains close to its performance under the exact transformation. Performance decreases as the corruption becomes more severe with 10% pixel masking or sticky frames. Sticky frames lead to a larger degradation because the entire current observation is occasionally replaced by stale information.

The results show that performance generally improves with additional target samples. Combining the two (mild) stochastic corruptions reduces performance relative to individual corruptions. However, it does not result in a drastic degradation of performance; LUMOS reaches an average return of 98.80 ± 19.18 after 50k samples from the target.

Overall, these results show that LUMOS remains effective under moderate stochastic observation corruption and degrades as the exact undo-map assumption is increasingly violated.

References

  1. [3]Steven Bohez, Tim Verbel en, Elias De Coninck, Bert Vankersbilck, Pieter Simoens, and Bart Dhoedt. Sensor Fusion for Robot Control through Deep Reinforcement Learning. IROS, 2017.arxiv.org/abs/1703.04550
  2. [7]Gabriel Dulac-Arnold, Daniel J. Mankowitz, and Todd Hester. Challenges of Real-World Reinforcement Learning. CoRR, abs/1904.12901, 2019.
  3. [9]Jiafan He, Heyang Zhao, Dongruo Zhou, and Quanquan Gu. Nearly Minimax Optimal Reinforcement Learning for Linear Markov Decision Processes. ICML, 2023.arxiv.org/abs/2212.06132
  4. [10]Daniel J. Hsu, Sham M. Kakade, and Tong Zhang. Random Design Analysis of Ridge Regression. Found. Comput. Math., 14(3):569–600, 2014.
  5. [11]Edward S. Hu, James Springer, Oleh Rybkin, and Dinesh Jayaraman. Privileged Sensing Scaffolds Reinforcement Learning. ICLR, 2024.arxiv.org/abs/2405.14853
  6. [13]Chi Jin, Zhuoran Yang, Zhaoran Wang, and Michael I Jordan. Provably Efficient Reinforcement Learning with Linear Function Approximation. COLT, 2020.arxiv.org/abs/1907.05388
  7. [14]Joonho Lee, Alexey Dosovitskiy, Dario Bellicoso, Vassilios Tsounis, Vladlen Koltun, and Marco Hutter. Learning agile and dynamic motor skills for legged robots. Sci. Robotics, 4(26), 2019.
  8. [16]Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Belle-mare, Alex Graves, Martin A. Riedmiller, Andreas Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nat., 518(7540):529–533, 2015.DOI
  9. [18]Antonin Raffin. RL Baselines3 Zoo. https://github.com/DLR-RM/rl-baselines3-zoo, 2020.
  10. [20]John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal Policy Optimization Algorithms. CoRR, abs/1707.06347, 2017.arxiv.org/abs/1707.06347
  11. [21]Yanchao Sun, Ruijie Zheng, Xia­o Wang, Andrew Cohen, and Furong Huang. Transfer RL across Observation Feature Spaces via Model-Based Regularization. ICLR, 2022.arxiv.org/abs/2201.00248
  12. [22]Matthew E. Taylor and Peter Stone. Transfer Learning for Reinforcement Learning Domains: A Survey. JMLR, 2009.DOI
  13. [25]Gellért Weisz, Philip Amortila, and Csaba Szepesvári. Exponential Lower Bounds for Planning in MDPs With Linearly-Realizable Optimal Action-Value Functions. ALT, 2021.arxiv.org/abs/2010.01374
  14. [27]Kenny Young and Tian Tian. MinAtar: An Atari-Inspired Testbed for Thorough and Reproducible Reinforcement Learning Experiments. CoRR, abs/1903.03176, 2019.arxiv.org/abs/1903.03176
  15. [28]Werner Zellinger, Thomas Grubinger, Edwin David Lughofer, Thomas Natschläger, and Susanne Saminger-Platz. Central Moment Discrepancy (CMD) for Domain-Invariant Representation Learning. ICLR, 2017.arxiv.org/abs/1702.08811
  16. [29]Wenshuai Zhao, Jorge Peña Queralta, and Tomi Westerlund. Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Survey. IEEE SSCI, 2020.
  17. [30]Henry Zhu, Abhishek Gupta, Aravind Rajeswaran, Sergey Levine, and Vikash Kumar. Dexterous Manipulation with Deep Reinforcement Learning: Efficient, General, and Low-Cost. ICRA, 2019.arxiv.org/abs/1810.06045
  18. [31]Zhuangdi Zhu, Kaixiang Lin, Anil K. Jain, and Jiayu Zhou. Transfer Learning in Deep Reinforcement Learning: A Survey. IEEE TPAMI, 2020.arxiv.org/abs/2009.07888

Paper details

vlnc:W_pqv8sa6x
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents