betaFeedback

WorksComputer Science

RIFT: Reward Improvement Field Transfer for Cross-Domain Offline RL

Abstract

Learning from limited target data is a central challenge in cross-domain offline reinforcement learning. Abundant source data can help, but dynamics differences can make direct transition reuse unreliable. We propose Reward Improvement Field Transfer (RIFT), which instead learns how low- and high-reward source transitions differ. RIFT encodes source and target transitions in a shared latent space, pairs low- and high-reward source transitions using optimal transport, and fits an affine field to their displacements. It applies this field to low-reward target anchors and decodes the results to augment the target dataset. The approach builds on the hypothesis that related domains can share useful low-to-high reward structure despite different dynamics. We bound synthesis error under shared class geometry, identify conditions favoring target anchoring over source-high transplant, and derive a conditional connection to downstream return. Experiments on dense-reward locomotion tasks under gravity, morphology, and friction shifts show improved aggregate performance over existing baselines.

Introduction

Learning a good policy often requires more target-domain experience than is practical to collect. A robot may have abundant simulation data but little real-world data, or its dynamics may change with friction, payload, or body structure [40, 2]. Cross-domain offline reinforcement learning (RL) studies how to use abundant source data and scarce target data without further interaction [22, 26].

These datasets provide different kinds of information. Source data offer extensive experience, but dynamics mismatch makes direct reuse difficult: the same state and action can produce different next states, even under a shared task and reward-scoring rule [33]. Target data reflect the deployment dynamics, but may provide too little coverage to learn a good policy alone. The challenge is therefore to extract useful source information without assuming that source transitions accurately describe the target environment.

One possible source of information is how low- and high-reward transitions differ. This relationship may be easier to estimate from abundant source data than from scarce target data. For example, a locomotion dataset may contain transitions with similar forward progress but different control costs, suggesting ways to achieve comparable progress with less effort. A change in gravity or body structure may alter the exact motion needed, while some of this reward-related structure may remain useful. This motivates our central question: instead of transferring source transitions directly, can we transfer how behavior improves?

We propose Reward Improvement Field Transfer (RIFT). RIFT maps source and target transitions into a shared latent space, matches low- and high-reward source transitions, and learns a simple transformation from their differences. It then applies this transformation to low-reward target transitions to generate additional target-domain data (Figure 1). Here, reward improvement refers to a transformation between low- and high-reward transition distributions. In this way, RIFT uses immediate reward to guide augmentation while estimating a long-horizon value function through the downstream IQL learner.

Motivation behind RIFT: source and target datasets with existing methods compared to RIFT

Figure 1. Motivation behind RIFT. A scarce dataset makes it difficult to learn in the target domain. How can we instead use existing structures from the source dataset to transfer what we learn?

Our contributions are: (1) we introduce RIFT, which transfers the change from low- to high-reward behavior across domains and uses it to generate additional target-domain data; (2) we bound target-anchored synthesis error, give conditions for greater accuracy than source-high transplant, and connect synthesis error to a conditional downstream return guarantee; and (3) we evaluate RIFT on cross-domain locomotion tasks with changes in gravity, morphology, and friction, using approximately one million source transitions and only 5,000 target transitions, and show improved aggregate performance across several source- and target-data settings.

Related Work

Cross-Domain reinforcement learning. Reinforcement-learning policies often need to adapt across changes in embodiment, observations, action spaces, or dynamics. Prior work studies these shifts through domain randomization, system identification, meta-RL, imitation learning, and domain adaptation [9, 13, 5, 7], as well as learned representations, data selection, and reward modification [30, 47, 11]. In fully offline settings, DARA, BOSA, and SRPO modify rewards or constrain learning to supported regions [22, 23, 48]; IGDF, OTDF, and ROOT select or reweight source transitions [45, 26, 3]; and STC, DmC, and MOBODY correct or generate transitions [49, 42, 12]. DROCO addresses training- and test-time dynamics robustness [35], while REAG and V2A align return or value information across domains [44] (Qiao et al. 2026a). RIFT builds on this work on cross-domain transfer and data augmentation, but transfers low-to-high reward displacements learned in the source domain and applies them to target anchors to generate additional transitions.

Representations under domain shift. Domain adaptation seeks representations that preserve useful structure across domains [52]. Conditional-invariance methods align distributions within labels or outcomes [38] (Li et al. 2018), including regression targets in COD [50]. This provides a way to align task-relevant structure rather than only the overall feature distribution. Further, latent-space regularization and structured priors can also help organize learned representations [14, 37]. Our method builds on this literature by using reward classes to encourage shared low- and high-reward structure across domains, supporting transfer of the improvement field.

Reward-structured learning and data augmentation. Reward and preference information can guide both policy learning and data generation. RLHF uses preferences to learn reward models or directly optimize policies [32, 10, 36], while LENS [39] augments limited preference data in a learned VAE latent space. In offline RL, MetaDiffuser, GTA, and GODA use return information to guide generation [29, 18, 15], while DiffStitch connects low- and high-reward trajectory segments [19]. These methods use reward-related structure to construct additional data for policy learning. RIFT uses this structure for cross-domain augmentation, learning low-to-high reward transformations from source data and applying them to scarce target data.

Preliminaries

We model each environment as a Markov decision process (MDP) M=(S,A,P,r,γ,ρ)\mathcal{M} = (\mathcal{S}, \mathcal{A}, P, r, \gamma, \rho), with state space S\mathcal{S}, action space A\mathcal{A}, transition kernel PP, expected one-step reward rr, discount factor γ∈[0,1)\gamma\in[0, 1), and initial-state distribution ρ\rho. A policy π\pi maximizes the expected discounted return

JM(π)=Eρ,P,π[∑t=0T−1γtRt]J_{\mathcal{M}}(\pi) = \mathbb{E}_{\rho,P,\pi}\left[\sum_{t=0}^{T-1}\gamma^t R_t\right]

where TT is the episode length, including the final transition, or ∞\infty for continuing tasks.

We consider source and target contexts AA and BB that share S\mathcal{S}, A\mathcal{A}, γ\gamma, ρ\rho, and the task objective, with known termination functions. Their dynamics PAP_A, PBP_B and expected rewards rAr_A, rBr_B may differ: even a shared rule for scoring transitions can yield different rewards at the same (s,a)(s, a) when dynamics change. Given fixed offline datasets DA\mathcal{D}_A and DB\mathcal{D}_B, with ∣DB∣≪∣DA∣|\mathcal{D}_B| \ll|\mathcal{D}_A|, our goal is to learn a target policy using both datasets without further environment interaction.

Methodology

Reward Improvement Field Transfer (RIFT) learns a low-to-high reward transformation from abundant source data in context AA and applies it to scarce target data in context BB to generate additional transitions Figure 2. This design addresses target-data scarcity: DB\mathcal{D}_B may be too small to fit a reliable dynamics model or a flexible transition generator, motivating a simple source-learned transformation applied to target anchors.

Overview of RIFT

Figure 2. Overview of RIFT. The method encodes source and target data, learns a reward-improvement field from source transitions, transfers it to target anchors, and augments downstream IQL training.

Algorithm

First, we learn a shared latent representation from Dmix=DA∪DB\mathcal{D}_{\mathrm{mix}} = \mathcal{D}_A \cup\mathcal{D}_B. Within each context, we label transitions in the bottom and top reward quantiles:

y(x)={0,rc≤Qc(q),1,rc≥Qc(1−q),c∈{A,B}.y(x) = \begin{cases} 0, & r_c \le Q_c(q),\\ 1, & r_c \ge Q_c(1-q), \end{cases} \qquad c \in\{A,B\}.

where QcQ_c is the reward quantile function and q∈(0,1/2)q \in(0,1/2) is fixed. Smaller qq increases or preserves the reward separation between classes but leaves fewer samples for field fitting and target anchoring.

We then train a shared variational autoencoder (VAE) (Kingma and Welling, [16]) on transitions x=(st,at,st+1)x = (s_t, a_t, s_{t+1}). Alongside the standard normal prior N(0,I)\mathcal{N}(0,I), we add class-conditioned regularization on one latent coordinate, z0z_0, to encourage a shared low-to-high reward direction:

p(z0∣y=0)=N(−cq,sq2),p(z0∣y=1)=N(cq,sq2).p(z_0 \mid y=0) = \mathcal{N}(-c_q,s_q^2), \qquad p(z_0 \mid y=1) = \mathcal{N}(c_q,s_q^2).

These means and variance match those of the lower and upper qq-tails of a standard normal and are therefore determined by qq. After training, the encoder posterior mean z=eϑ(x)z = e_{\vartheta}(x) represents each transition, and the decoder gϑg_{\vartheta} maps latent points back to transitions.

Next, we learn a low-to-high reward transformation from source context AA. We append the observed reward to each latent point, giving u=(z,r)u = (z,r), and normalize source points to target coordinates:

u~=μB+σB⋅u−μAσA.\tilde{u} = \mu_B + \sigma_B \cdot\frac{u-\mu_A}{\sigma_A}.

where μc\mu_c and σc\sigma_c are the coordinate-wise means and standard deviations over all samples in context cc. Mini-batch optimal transport (Villani, [43]) then pairs these normalized low- and high-reward source points using squared Euclidean distance between their latent components. For each matched low/high pair (u~i,v~i)(\tilde{u}_i,\tilde{v}_i), we compute the joint latent–reward displacement δi=v~i−u~i\delta_i = \tilde{v}_i - \tilde{u}_i. We fit an affine field δ^(u)=W^u+b^\hat{\delta}(u) = \widehat{W}u + \widehat{b} to these displacements by ordinary least squares. For each low-reward target anchor uBu^B, we generate

u~B=uB+δ^(uB).\tilde{u}^B = u^B + \hat{\delta}(u^B).

We decode its latent component into a transition and use its reward component as the synthetic reward. The known target termination function assigns the terminal flag from the decoded next state, and samples with terminal initial states are discarded.

Finally, we train the downstream policy with implicit Q-learning (IQL) (Kostrikov et al., [17]) and evaluate it in context BB. While immediate reward provides the local signal for augmentation, downstream IQL learns the long-horizon values through Bellman backups. We standardize observations using means and standard deviations. Further, each minibatch draws a fraction 1−λ1-\lambda of its transitions uniformly from DAD_A and the remaining fraction λ\lambda uniformly from DB∪DB′D_B \cup D'_B, where DB′D'_B contains RIFT’s synthetic target transitions. The weight λ\lambda balances source coverage against source reward and dynamics mismatch. The realized synthetic fraction within the target pool is

θ=∣DB′∣∣DB∣+∣DB′∣.\theta= \frac{\lvert D'_B\rvert}{\lvert D_B\rvert+\lvert D'_B\rvert}.

Theoretical Analysis

We formalize when a source improvement field can transfer across domains, bound synthesis error, and compare target anchoring with directly decoding high-reward source points. Proofs and the conditional downstream guarantee appear in Appendix § 7. All appendices are provided in the supplementary material.

Setting. Let ηcy\eta_c^y denote the transition–reward distribution in context cc and immediate-reward class yy. Define

E(x,r)=(eϑ(x),r),D(z,r)=(gϑ(z),r),μcy=E#ηcy.E(x,r)=(e_\vartheta(x),r), \qquad D(z,r)=(g_\vartheta(z),r), \qquad\mu_c^y=E_{\#}\eta_c^y.

where #\# denotes the pushforward of a distribution. Write μˉAy=S#μAy\bar{\mu}_A^y=S_{\#}\mu_A^y for source points normalized to target coordinates as in § 4.1. We fix the representation, normalization, and fitted field. All norms and Wasserstein distances use Euclidean coordinates, and all distributions have finite second moments.

Let πA∈Π(μˉA0,μˉA1)\pi_A\in\Pi(\bar{\mu}_A^0,\bar{\mu}_A^1) be a population coupling that minimizes squared distance between the latent components. Its population affine fit is

δaff(u)=Wu+b∈arg⁡min⁡δ affine E(X,Y)∼πA∥Y−X−δ(X)∥2.\delta_{\mathrm{aff}}(u)=Wu+b\in\underset{\delta\ \mathrm{affine}}{\arg\min}\ \mathbb{E}_{(X,Y)\sim\pi_A}\lVert Y-X-\delta(X)\rVert^2.

Choose one minimizer if necessary. The fitted generator and its pre-filtering synthetic distribution are

F=id⁡+δ^,ν=(D∘F)#μB0.F=\operatorname{id}+\hat{\delta}, \qquad\nu=(D\circ F)_{\#}\mu_B^0.

Assumption 1 (Shared class geometry). There is a common invertible affine map h(u)=Hu+th(u)=Hu+t such that

W2(μcy,h#μˉAy)≤ρy,y=0,1.W_2(\mu_c^y,h_{\#}\bar{\mu}_A^y)\leq\rho_y,\qquad y=0,1.

The map hh describes the remaining cross-context geometry after normalization. The reward-conditioned VAE regularization is designed to encourage this correspondence.

Synthesis error. Define the field mismatch Γh(u)=δaff(u)−Hδaff(h−1(u))\Gamma_h(u)=\delta_{\mathrm{aff}}(u)-H\delta_{\mathrm{aff}}(h^{-1}(u)) and the errors

εaff=(EπA∥Y−X−δaff(X)∥2)1/2,εest=∥δ^−δaff∥L2(μB0),\varepsilon_{\mathrm{aff}}=\left(\mathbb{E}_{\pi_A}\lVert Y-X-\delta_{\mathrm{aff}}(X)\rVert^2\right)^{1/2}, \qquad\varepsilon_{\mathrm{est}}=\lVert\hat{\delta}-\delta_{\mathrm{aff}}\rVert_{L^2(\mu_B^0)},
εconj=∥Γh∥L2(μB0),εdec=(E(x,r)∼ηB1∥gϑ(eϑ(x))−x∥2)1/2.\varepsilon_{\mathrm{conj}}=\lVert\Gamma_h\rVert_{L^2(\mu_B^0)}, \qquad\varepsilon_{\mathrm{dec}}=\left(\mathbb{E}_{(x,r)\sim\eta_B^1}\lVert g_\vartheta(e_\vartheta(x))-x\rVert^2\right)^{1/2}.

These measure source affine approximation, field estimation on target anchors, cross-context field mismatch, and target-high reconstruction, respectively. Let Lh=∥H(I+W)H−1∥L_h=\lVert H(I+W)H^{-1}\rVert and LD=max⁡(1,Lg)L_D=\max(1,L_g), where the decoder is LgL_g-Lipschitz. The population transfer-error budget is then

εtr=εconj+∥H∥εaff+Lhρ0+ρ1.\varepsilon_{\mathrm{tr}}=\varepsilon_{\mathrm{conj}}+\lVert H\rVert\varepsilon_{\mathrm{aff}}+L_h\rho_0+\rho_1.

Theorem 1 (Accuracy of transferred synthesis). Under Assumption 1 and the setting above,

W2(ν,ηB1)≤εsyn:=LD(εest+εtr)+εdec.(1)W_2(\nu,\eta_B^1) \le\varepsilon_{\mathrm{syn}} := L_D(\varepsilon_{\mathrm{est}}+\varepsilon_{\mathrm{tr}})+\varepsilon_{\mathrm{dec}}. \tag*{(1)}

Thus, generated and genuine target-high points admit a coupling with root-mean-square distance at most εsyn\varepsilon_{\mathrm{syn}}.

For a residual translation h(u)=u+th(u)=u+t, εconj=∥Wt∥\varepsilon_{\mathrm{conj}}=\lVert Wt\rVert. A large common shift can therefore produce little field mismatch when the improvement field is nearly constant.

Proposition 1 (Comparison with source-high transplant). Suppose the encoder is LeL_e-Lipschitz, put LE=max⁡(1,Le)L_E=\max(1,L_e), and let εcyc=(E(z,r)∼μˉA1∥eϑ(gϑ(z))−z∥2)1/2\varepsilon_{\mathrm{cyc}}=\left(\mathbb{E}_{(z,r)\sim\bar{\mu}_A^1}\lVert e_\vartheta(g_\vartheta(z))-z\rVert^2\right)^{1/2}. With finite second moments,

W2(D#μˉA1,ηB1)≥W2(μˉA1,μB1)−εcycLE.(2)W_2(D_{\#}\bar{\mu}_A^1,\eta_B^1)\ge\frac{W_2(\bar{\mu}_A^1,\mu_B^1)-\varepsilon_{\mathrm{cyc}}}{L_E}. \tag*{(2)}

If this lower bound exceeds εsyn\varepsilon_{\mathrm{syn}} under Theorem 1’s assumptions, RIFT is strictly closer to ηB1\eta_B^1 than transplant is. Under Assumption 1 with h(u)=u+th(u)=u+t, transplant’s lower bound is at least (∥t∥−ρ1−εcyc)/LE(\lVert t\rVert-\rho_1-\varepsilon_{\mathrm{cyc}})/L_E, whereas RIFT’s field-incompatibility contribution is LD∥Wt∥L_D\lVert Wt\rVert.

From source fitting to synthesis accuracy. Under the regularity, population-coupling, and sample-size conditions in Appendix § 7.§ 3, the fitted field satisfies

εest≤ε‾est(J):=κM[Bk+C2log⁡(2/η)J]\varepsilon_{\mathrm{est}}\le\overline{\varepsilon}_{\mathrm{est}}(J):=\kappa_M\left[B_k+C_2\sqrt{\frac{\log(2/\eta)}{J}}\right]

with probability at least 1−η1-\eta, where JJ is the number of independent source batches, BkB_k bounds matching bias for batches of kk points per class, and κM\kappa_M converts source prediction error to error on target anchors. The constant C2C_2 depends on source boundedness and moment conditioning. Substituting into Theorem 1 gives

W2(ν,ηB1)≤LD[ε‾est(J)+εtr]+εdec(3)W_2(\nu,\eta_B^1)\le L_D\left[\overline{\varepsilon}_{\mathrm{est}}(J)+\varepsilon_{\mathrm{tr}}\right]+\varepsilon_{\mathrm{dec}} \tag*{(3)}

with the same probability. If this upper bound is below the transplant lower bound in equation (2), the strict comparison therefore holds for the estimated field.

Increasing JJ reduces sampling error, but not matching bias, transfer error, or decoding error. If LD(κMBk+εtr)+εdecL_D(\kappa_M B_k+\varepsilon_{\mathrm{tr}})+\varepsilon_{\mathrm{dec}} is below the transplant lower bound, sufficiently many independent source batches establish the comparison for fixed kk and η\eta. This motivates learning an improvement rule from abundant source data, while recognizing that more source data alone cannot resolve field mismatch.

Sampling interpretation. Theorem 1 concerns the pre-filtering distribution ν\nu, not a finite generated dataset. When εsyn=0\varepsilon_{\mathrm{syn}}=0, independent target-low anchors passed through the fixed generator produce independent target-high samples. Finite datasets still have sampling error, which we analyze in Appendix § 7.§ 2.

Conditional downstream consequence. Under target and surrogate regularity, occupancy domination, and a surrogate learning-error allowance εlearn\varepsilon_{\mathrm{learn}}, Theorem A.1 in Appendix § 7.§ 4 gives

JB(π^)≥JB(πc)−εlearn−Rλ,θ(π^)−Rλ,θ(πc).J_B(\hat{\pi}) \ge J_B(\pi_c) - \varepsilon_{\mathrm{learn}} - \mathcal{R}_{\lambda,\theta}(\hat{\pi}) - \mathcal{R}_{\lambda,\theta}(\pi_c).

The budget R\mathcal{R} includes synthesis error, source reward and dynamics mismatch, filtering, and surrogate fitting, with terminal-flag errors included for episodic tasks. This connects synthesis accuracy to return, motivating our overall approach.

Experiments

We evaluate RIFT on the ODRL cross-domain offline RL benchmark ([24]) and use ablations to examine its components and hyperparameters. Hyperparameters and additional results appear in Appendices C and D.

Setup

Tasks and evaluation protocol. We evaluate HalfCheetah, Hopper, Walker2d, and Ant under ODRL’s gravity, morphology, and friction shifts, following the benchmark settings used by OTDF ([26]) and STC ([49]). Target gravity and friction coefficients are halved in their respective settings; the medium morphology shift modifies one body component per task. We use the full source dataset DAD_A, containing approximately one million transitions, and 5,000 target transitions in DBD_B, held fixed across policy seeds. Dataset qualities follow D4RL ([8]): medium (med), medium-replay (med-r), medium-expert (med-e), and expert. Further dataset details appear in Appendix § 9.

Baselines. We compare against pooled source–target IQL ([17]) and six cross-domain methods. DARA adjusts source rewards using domain classifiers ([22]), BOSA constrains policy and value learning to supported regions ([23]), and SRPO shapes rewards using stationary-state information ([48]). IGDF uses contrastive scores to select source transitions ([45]), while OTDF scores and filters them using optimal transport ([26]). STC corrects source transitions toward the target domain and filters unreliable corrections ([49]). Baseline results in the source- and target-quality comparisons are taken from STC under the same ODRL protocol, using one million gradient steps and five seeds. Implementation details appear in Appendix C.

Main Results

Performance across source qualities. Table 1 varies source-data quality while fixing the target dataset to 5,000 medium-quality transitions. RIFT achieves the highest mean in 16 of 24 gravity and morphology settings and the highest aggregate normalized score of 1318.7, a 26.2% gain over STC (1045.3), the strongest baseline in aggregate. Gains are largest on HalfCheetah and Hopper, particularly under morphology shifts, and less consistent on Walker2d and Ant. OTDF and STC generally outperform earlier adaptation methods, highlighting the value of addressing transition mismatch. RIFT further improves aggregate performance by learning low-to-high reward transformations from source data.

TypeSourceTargetIQLDARABOSASRPOIGDFOTDFSTCRIFT (ours)
Gravityhalf-medmed39.6±3.339.6 \pm3.341.2±3.941.2 \pm3.938.9±4.038.9 \pm4.036.9±4.536.9 \pm4.536.6±5.536.6 \pm5.540.7±7.740.7 \pm7.742.4±5.342.4 \pm5.347.3±6.647.3 \pm6.6
half-med-rmed20.1±5.020.1 \pm5.017.6±6.217.6 \pm6.220.0±4.920.0 \pm4.917.5±5.217.5 \pm5.214.4±2.214.4 \pm2.221.5±6.521.5 \pm6.526.7±2.226.7 \pm2.231.4±5.331.4 \pm5.3
half-med-emed38.6±6.038.6 \pm6.037.8±3.337.8 \pm3.341.8±5.141.8 \pm5.142.5±2.342.5 \pm2.337.7±7.337.7 \pm7.339.5±3.539.5 \pm3.539.2±4.239.2 \pm4.249.6±5.349.6 \pm5.3
hopp-medmed11.2±1.111.2 \pm1.117.3±3.817.3 \pm3.815.2±3.315.2 \pm3.312.4±1.012.4 \pm1.015.3±3.515.3 \pm3.532.4±8.032.4 \pm8.043.4±6.143.4 \pm6.160.4±7.360.4 \pm7.3
hopp-med-rmed13.9±2.913.9 \pm2.910.7±4.310.7 \pm4.33.3±1.93.3 \pm1.914.0±2.614.0 \pm2.615.3±4.415.3 \pm4.431.1±13.431.1 \pm13.436.8±17.836.8 \pm17.861.3±4.361.3 \pm4.3
hopp-med-emed19.1±6.619.1 \pm6.618.5±12.318.5 \pm12.315.9±5.915.9 \pm5.919.7±8.519.7 \pm8.522.3±5.422.3 \pm5.426.4±10.126.4 \pm10.145.3±7.545.3 \pm7.565.2±6.565.2 \pm6.5
walk-medmed28.1±12.928.1 \pm12.928.4±13.728.4 \pm13.738.0±11.238.0 \pm11.221.4±7.021.4 \pm7.022.1±8.422.1 \pm8.436.6±2.336.6 \pm2.341.6±4.041.6 \pm4.046.5±5.546.5 \pm5.5
walk-med-rmed14.6±2.514.6 \pm2.514.1±6.114.1 \pm6.17.6±5.87.6 \pm5.817.9±3.817.9 \pm3.811.6±4.611.6 \pm4.632.7±7.032.7 \pm7.029.0±1.929.0 \pm1.933.0±8.833.0 \pm8.8
walk-med-emed39.9±13.139.9 \pm13.141.6±13.041.6 \pm13.032.3±7.232.3 \pm7.246.4±3.546.4 \pm3.533.8±3.133.8 \pm3.130.2±9.830.2 \pm9.834.9±9.134.9 \pm9.140.3±7.640.3 \pm7.6
ant-medmed10.2±1.810.2 \pm1.89.4±0.99.4 \pm0.912.4±2.012.4 \pm2.011.7±1.011.7 \pm1.011.3±1.311.3 \pm1.345.1±12.445.1 \pm12.442.6±8.442.6 \pm8.440.8±10.340.8 \pm10.3
ant-med-rmed18.9±2.618.9 \pm2.621.7±2.121.7 \pm2.113.9±1.513.9 \pm1.518.7±1.718.7 \pm1.719.6±1.019.6 \pm1.029.6±10.729.6 \pm10.740.9±5.640.9 \pm5.646.7±3.546.7 \pm3.5
ant-med-emed9.8±2.49.8 \pm2.48.1±1.88.1 \pm1.88.1±3.08.1 \pm3.08.4±2.18.4 \pm2.18.9±1.58.9 \pm1.518.6±11.918.6 \pm11.939.2±9.239.2 \pm9.238.1±7.038.1 \pm7.0
Morphhalf-medmed24.5±2.424.5 \pm2.421.0±3.921.0 \pm3.924.2±5.624.2 \pm5.618.1±1.818.1 \pm1.823.7±3.423.7 \pm3.421.1±7.621.1 \pm7.619.5±2.219.5 \pm2.257.9±4.457.9 \pm4.4
half-med-rmed11.0±1.211.0 \pm1.29.5±2.39.5 \pm2.34.7±2.94.7 \pm2.98.9±1.28.9 \pm1.29.2±0.69.2 \pm0.66.5±1.46.5 \pm1.413.0±5.313.0 \pm5.349.2±4.649.2 \pm4.6
half-med-emed21.1±2.821.1 \pm2.819.2±2.219.2 \pm2.223.2±3.923.2 \pm3.921.1±1.921.1 \pm1.918.6±1.318.6 \pm1.320.8±2.520.8 \pm2.516.8±8.816.8 \pm8.860.5±3.160.5 \pm3.1
hopp-medmed15.9±6.815.9 \pm6.817.8±10.117.8 \pm10.112.8±0.112.8 \pm0.121.7±7.721.7 \pm7.725.3±9.725.3 \pm9.716.4±7.116.4 \pm7.143.1±23.943.1 \pm23.969.4±13.269.4 \pm13.2
hopp-med-rmed12.9±0.312.9 \pm0.312.8±0.112.8 \pm0.12.0±1.22.0 \pm1.212.4±0.712.4 \pm0.712.5±1.712.5 \pm1.713.3±0.113.3 \pm0.122.9±6.122.9 \pm6.161.8±10.661.8 \pm10.6
hopp-med-emed14.9±3.114.9 \pm3.111.1±5.611.1 \pm5.614.4±1.814.4 \pm1.816.6±1.916.6 \pm1.918.3±7.518.3 \pm7.525.4±9.425.4 \pm9.453.4±20.853.4 \pm20.865.7±10.465.7 \pm10.4
walk-medmed31.5±8.631.5 \pm8.635.0±10.835.0 \pm10.826.7±6.626.7 \pm6.638.6±5.138.6 \pm5.138.5±8.438.5 \pm8.442.5±3.142.5 \pm3.156.7±8.156.7 \pm8.152.8±8.252.8 \pm8.2
walk-med-rmed41.5±3.041.5 \pm3.038.5±7.938.5 \pm7.915.3±7.615.3 \pm7.636.0±4.436.0 \pm4.424.2±8.624.2 \pm8.617.9±13.417.9 \pm13.463.1±8.063.1 \pm8.046.9±6.746.9 \pm6.7
walk-med-emed32.8±4.332.8 \pm4.341.4±10.941.4 \pm10.945.1±13.745.1 \pm13.739.8±14.339.8 \pm14.337.9±4.237.9 \pm4.255.3±2.255.3 \pm2.262.1±8.162.1 \pm8.161.3±7.061.3 \pm7.0
ant-medmed71.4±2.471.4 \pm2.471.5±6.871.5 \pm6.854.8±13.254.8 \pm13.272.8±2.272.8 \pm2.271.8±2.771.8 \pm2.775.1±2.375.1 \pm2.377.2±2.877.2 \pm2.876.2±2.476.2 \pm2.4
ant-med-rmed65.9±5.565.9 \pm5.562.3±8.262.3 \pm8.215.2±2.315.2 \pm2.359.3±4.059.3 \pm4.065.0±5.365.0 \pm5.363.1±5.963.1 \pm5.976.2±2.676.2 \pm2.678.1±0.478.1 \pm0.4
ant-med-emed70.2±7.370.2 \pm7.364.3±5.864.3 \pm5.864.0±6.064.0 \pm6.068.5±4.468.5 \pm4.466.8±11.066.8 \pm11.076.4±1.976.4 \pm1.979.3±0.279.3 \pm0.278.3±0.378.3 \pm0.3
Total (displayed means)677.6670.8549.8681.3660.7818.21045.31318.7

Table 1. Performance under gravity and morphology shifts when the target dataset contains medium-quality transitions and source-data quality varies. Values report the mean ±\pm standard deviation over five seeds, with the highest mean bolded.

Performance across target qualities. We also fix source quality to medium and varies target quality (Appendix D.1). RIFT achieves the highest mean in 13 of 20 settings and the highest aggregate score of 940.9, a 14.6% improvement over STC (820.9). RIFT’s advantage is generally larger with medium-quality target data and decreases as target quality improves. This is consistent with its motivation: augmentation offers more benefit when useful target behavior is not already well represented in the dataset.

Ablations

We examine augmentation, representation design, and sensitivity to reward conditioning and target-data size across eight medium-to-medium gravity and morphology settings. All experiments follow the main protocol except for the component varied. Our reproduced baseline, IQL*, uses the same selection protocol as RIFT: for each task, we report the choice of λ∈{0.5,1}\lambda\in\{0.5, 1\} with the higher mean target return. This distinguishes it from the published IQL results used in our main comparisons.

Augmentation and source-field controls. Table 2 compares RIFT with Mixup (Zhang et al., [51]) and SMOTE (Chawla et al., [1]), using either target data alone or pooled source and target data. Each uses a 1,000-transition budget. RIFT achieves the highest aggregate score of 451.3, compared with 394.5 for the strongest interpolation baseline, although results vary by task. We also compare against a target-only field using the same VAE and anchors, and resampling real high-reward target transitions. These score 423.8 and 384.5, respectively, supporting the benefit of learning the field from source data (Appendix D.2.3).

Table 2: Comparison of augmentation strategies across eight medium-to-medium gravity and morphology settings. Mixup and SMOTE generate 1,000 synthetic transitions using either pooled source and target data (A∪BA \cup B) or target data alone (BB). RIFT generates up to 1,000 synthetic transitions. Values report the mean ±\pm standard deviation over five seeds, with the highest mean bolded.

ShiftTaskIQL*Mixup A∪BA \cup BMixup BBSMOTE A∪BA \cup BSMOTE BBRIFT
GravityAnt29.8 ±\pm 4.929.5 ±\pm 1.926.1 ±\pm 6.928.2 ±\pm 7.534.4 ±\pm 7.240.8 ±\pm 10.3
HalfCheetah42.0 ±\pm 2.342.9 ±\pm 2.143.3 ±\pm 2.045.2 ±\pm 4.745.4 ±\pm 4.947.3 ±\pm 6.6
Hopper52.7 ±\pm 12.549.1 ±\pm 12.144.1 ±\pm 12.861.0 ±\pm 6.660.8 ±\pm 8.660.4 ±\pm 7.3
Walker2d30.6 ±\pm 4.343.9 ±\pm 6.136.5 ±\pm 13.645.1 ±\pm 6.640.5 ±\pm 5.546.5 ±\pm 5.5
MorphologyAnt76.9 ±\pm 0.374.1 ±\pm 2.876.2 ±\pm 0.777.3 ±\pm 0.577.3 ±\pm 0.376.2 ±\pm 2.4
HalfCheetah22.8 ±\pm 2.418.8 ±\pm 1.517.3 ±\pm 1.818.8 ±\pm 2.520.2 ±\pm 4.857.9 ±\pm 4.4
Hopper55.5 ±\pm 6.459.9 ±\pm 13.946.6 ±\pm 10.354.3 ±\pm 12.160.2 ±\pm 11.269.4 ±\pm 13.2
Walker2d49.2 ±\pm 5.154.1 ±\pm 8.659.0 ±\pm 9.454.3 ±\pm 7.855.7 ±\pm 10.952.8 ±\pm 8.2
Total (displayed means)359.5372.3349.1384.2394.5451.3

Table 2.

Representation design. Balanced source–target VAE sampling improves the aggregate score from 394.6 to 432.7, and adding reward regularization raises it to 451.3 (Appendix D.2.2). For the quantile, we chose q=0.2q = 0.2 as the default before running this ablation. It also achieves the highest aggregate score among the tested quantiles, although the preferred value varies by task (Appendix D.2.1).

Conditioning horizon and target-data size. Using 5-, 10-, and 20-step-return classes gives broadly similar aggregate performance, with no consistent gain as the horizon increases (Appendix D.2.4). This supports immediate reward in these dense-reward tasks while showing that RIFT accommodates longer-horizon signals, which may be useful for sparse or delayed rewards. Further, RIFT improves aggregate performance over IQL* with 1,000 and 2,500 target transitions, but underperforms at 500, possibly because limited target coverage weakens representation learning and target anchoring (Appendix D.2.5).

Latent Space Representation

Figure 3 shows shared VAE representations for HalfCheetah and Walker2d under gravity shifts. Balanced sampling and reward regularization produce a similar low-to-high reward direction across domains, consistent with the shared-geometry motivation. Reward regularization also preserves reconstruction quality, explaining over 98% of the variance in all settings (Appendix D.§ 3).

PCA projections of shared VAE representations under gravity shifts

Figure 3. PCA projections of shared VAE representations under gravity shifts. Colors distinguish domains and reward classes. The regularized representation shows a similar low-to-high reward direction across domains.

We also evaluate held-out reconstruction and replay generated transitions in the target simulator. Per-dimension reconstruction MSE ranges from 0.001–0.07 times the corresponding variance, while reward errors range from 0.05–0.19 target reward-class gaps. These results support reconstruction quality and the accuracy of generated reward labels, although next-state accuracy varies by task and does not consistently track policy performance (Appendix D.§ 3).

Conclusions

We introduced RIFT, which learns low-to-high reward transformations from abundant source data and applies them to target anchors to generate additional training transitions. Our analysis bounds synthesis error, identifies conditions favoring target anchoring over source-high transplant, and gives a conditional connection to target return. Experiments under gravity, morphology, and friction shifts show improved aggregate performance across source- and target-data qualities. Controlled comparisons further support the contribution of the source-derived field beyond high-reward target resampling or learning the field from target data alone. These findings suggest that source data can provide useful relationships between transitions even when the transitions themselves do not match target dynamics. Transferring these relationships, specifically for reward improvement, offers a way to use existing experience when collecting additional target data is difficult or expensive.

RIFT also has limitations. Reward regularization encourages the source field to remain useful in the target domain but does not guarantee compatibility or detect failure under larger dynamics shifts. Our evaluation focuses on dense-reward locomotion; sparse or delayed rewards may require other conditioning signals. Longer-horizon labels offer one option, but evaluation on these tasks and manipulation environments remains future work. Finally, the theory fixes the representation and population coupling under additional regularity and sampling assumptions; it does not prove that end-to-end training satisfies them or that accurate synthesis alone guarantees improved return.

Generative AI Use

We used generative AI tools for writing and LaTeX editing, code development, related-work search, and assistance with mathematical formalization, proof writing, and consistency checks. The core research ideas and mathematical reasoning were developed by the authors. We reviewed the AI-assisted text, code, proofs, and references and take responsibility for the final content.

Theoretical Results and Proofs

Formal setting and conventions

A labeled transition is ξ=(x,r)∈Ξ:=Rdx×R\xi=(x,r)\in\Xi:=\mathbb{R}^{d_x}\times\mathbb{R} with x=(s,a,s′)x=(s,a,s'), and ηcy\eta_c^y denotes its population law in reward class y∈{0,1}y\in\{0,1\} of context c∈{A,B}c\in\{A,B\}. For q∈(0,1/2)q\in(0,1/2), classes are the bottom and top qq fractions of rewards. At atoms on a threshold, independent uniform ranks break ties so that each class has probability qq.

Write U=Rdz×R\mathcal{U}=\mathbb{R}^{d_z}\times\mathbb{R} and du=dz+1d_u=d_z+1. The lift E(x,r)=(eϑ(x),r)E(x,r)=(e_\vartheta(x),r) and decoder D(z,r)=(gϑ(z),r)D(z,r)=(g_\vartheta(z),r) give μcy=E#ηcy\mu_c^y=E_\#\eta_c^y. The fixed normalization S=SB−1∘SAS=S_B^{-1}\circ S_A uses the coordinatewise means and positive standard deviations of each full context distribution, including all reward classes; empirically, these statistics use all samples from that context. Thus μˉAy=S#μAy\bar{\mu}_A^y=S_\#\mu_A^y. Squared latent matching is computed in these target coordinates, not before an additional anisotropic rescaling.

The Euclidean norm is used on both joint latent–reward and decoded transition–reward coordinates; the same symbol on matrices means the operator norm, with ∥⋅∥F\|\cdot\|_F the Frobenius norm. Write ∥f∥L2(μ)=(∫∥f∥2dμ)1/2\|f\|_{L^2(\mu)}=\left(\int\|f\|^2d\mu\right)^{1/2}, and let W2W_2 denote quadratic Wasserstein distance. All laws entering these distances have finite second moments. Standard coupling and pushforward properties are used below [43].

The source coupling in § 4.2 can be any fixed minimizer of

inf⁡π∈Π(μˉA0,μˉA1)E(X,Y)∼π∥ZY−ZX∥2.\inf_{\pi\in\Pi(\bar{\mu}_A^0,\bar{\mu}_A^1)}\mathbb{E}_{(X,Y)\sim\pi}\|Z_Y-Z_X\|^2.

where ZX,ZYZ_X,Z_Y are the latent blocks. Its endpoints remain joint latent–reward points. The affine fit and residual therefore include the reward displacement. If the affine minimizer is not unique, all subsequent quantities refer to one fixed representative. Synthesis uses only the coupling’s marginals, not its optimality, determinism, or an absolutely continuous class law.

The theorem, Theorem 1, conditions on the learned representation, normalization, coupling, and field. The sampling and estimation results below use separate conditional-i.i.d. assumptions; they do not provide a generalization guarantee for jointly learned quantities.

Synthesis, transplant, and sampling

Proof [Proof of Theorem 1] Define the affine comparison map

Gh=h∘(id⁡+δaff)∘h−1,Lip⁡(Gh)=∥H(I+W)H−1∥=Lh.G_h=h\circ(\operatorname{id}+\delta_{\mathrm{aff}})\circ h^{-1},\qquad\operatorname{Lip}(G_h)=\|H(I+W)H^{-1}\|=L_h.

Then

F−Gh=(δ^−δaff)+Γh,∥F−Gh∥L2(μB0)≤εest+εconj.F-G_h=(\hat{\delta}-\delta_{\mathrm{aff}})+\Gamma_h,\qquad\|F-G_h\|_{L^2(\mu_B^0)}\leq\varepsilon_{\mathrm{est}}+\varepsilon_{\mathrm{conj}}.

For (X,Y)∼πA(X,Y)\sim\pi_A, the pair (Gh(h(X)),h(Y))(G_h(h(X)),h(Y)) couples (Gh)#h#μˉA0(G_h)_\#h_\#\bar{\mu}_A^0 and h#μˉA1h_\#\bar{\mu}_A^1, with difference H(X+δaff(X)−Y)H(X+\delta_{\mathrm{aff}}(X)-Y). Its root-mean-square cost is at most ∥H∥εaff\|H\|\varepsilon_{\mathrm{aff}}. The triangle inequality and Lipschitz pushforward bound therefore give

W2(F#μB0,μB1)≤∥F−Gh∥L2(μB0)+LhW2(μB0,h#μˉA0)+∥H∥εaff+ρ1≤εest+εconj+∥H∥εaff+Lhρ0+ρ1.(4)\begin{aligned} W_2(F_\#\mu_B^0,\mu_B^1) &\leq\|F-G_h\|_{L^2(\mu_B^0)}+L_hW_2(\mu_B^0,h_\#\bar{\mu}_A^0)+\|H\|\varepsilon_{\mathrm{aff}}+\rho_1\\ &\leq\varepsilon_{\mathrm{est}}+\varepsilon_{\mathrm{conj}}+\|H\|\varepsilon_{\mathrm{aff}}+L_h\rho_0+\rho_1. \tag*{(4)} \end{aligned}

The nonlinear or nondeterministic affine-approximation error is paid only under the source coupling. The lifted decoder is LD=max⁡(1,Lg)L_D=\max(1,L_g)-Lipschitz. Coupling D(E(ξ))D(E(\xi)) to ξ∼ηB0\xi\sim\eta_B^0 gives W2(D#μB1,ηB1)≤εdecW_2(D_{\#}\mu_B^1,\eta_B^1) \le\varepsilon_{\mathrm{dec}}. Hence

W2(ν,ηB1)≤LDW2(F#μB0,μB1)+εdec≤εsyn.W_2(\nu,\eta_B^1) \le L_D W_2(F_{\#}\mu_B^0,\mu_B^1) + \varepsilon_{\mathrm{dec}} \le\varepsilon_{\mathrm{syn}}.

An optimal coupling of the decoded laws gives the claimed root-mean-square interpretation.

Context interpretation. For h(u)=Hu+th(u)=Hu+t,

Γh(u)=(W−HWH−1)u+HWH−1t+(I−H)b.\Gamma_h(u)=(W-HWH^{-1})u+HWH^{-1}t+(I-H)b.

For a shift this equals WtWt. For a rotation H=OH=O, t=0t=0, and a constant field W=0W=0, it equals (I−O)b(I-O)b. Thus a constant improvement is invariant to a common translation but does not need to be invariant to a rotation. The result does not assume that normalization removes every residual class-relative or cross-coordinate change.

Proof [Proof of Proposition 1] The encoder lift is LE=max⁡(1,Le)L_E=\max(1,L_e)-Lipschitz, so

W2(D#μˉA1,ηB1)≥LE−1W2((E∘D)#μˉA1,μB1).W_2(D_{\#}\bar{\mu}_A^1,\eta_B^1)\ge L_E^{-1}W_2((E\circ D)_{\#}\bar{\mu}_A^1,\mu_B^1).

Since (E∘D)(z,r)=(eϑ(gϑ(z)),r)(E\circ D)(z,r)=(e_\vartheta(g_\vartheta(z)),r), coupling by the same source-high point gives W2((E∘D)#μˉA1,μˉA1)≤εcycW_2((E\circ D)_{\#}\bar{\mu}_A^1,\bar{\mu}_A^1)\le\varepsilon_{\mathrm{cyc}}. The reverse triangle inequality proves equation (2). Under Assumption 1, a second reverse triangle inequality yields

W2(μˉA1,μB1)≥W2(μˉA1,h#μˉA1)−ρ1.W_2(\bar{\mu}_A^1,\mu_B^1)\ge W_2(\bar{\mu}_A^1,h_{\#}\bar{\mu}_A^1)-\rho_1.

For a shift, (X,X+t)(X,X+t) is a coupling with cost ∥t∥\lVert t\rVert, while every coupling satisfies E∥X−Y∥2≥∥E(X−Y)∥2=∥t∥2\mathbb{E}\lVert X-Y\rVert^2\ge\lVert\mathbb{E}(X-Y)\rVert^2=\lVert t\rVert^2. Thus W2(μˉA1,h#μˉA1)=∥t∥W_2(\bar{\mu}_A^1,h_{\#}\bar{\mu}_A^1)=\lVert t\rVert. Combining the lower bound with Theorem 1 gives the strict comparison.

Finite synthetic samples

Under the conditional-i.i.d. anchor assumption of Appendix § 7.§ 1, let ξ^i=D(F(Ui))\hat{\xi}_i=D(F(U_i)), i=1,…,mi=1,\ldots,m, be the actual generated points and ν^m\hat{\nu}_m their empirical law. On an enlarged probability space they admit partners ξi⋆∼ηB1\xi_i^\star\sim\eta_B^1 such that the pairs (ξ^i,ξi⋆)(\hat{\xi}_i,\xi_i^\star) are i.i.d. and E∥ξ^i−ξi⋆∥2≤εsyn2\mathbb{E}\lVert\hat{\xi}_i-\xi_i^\star\rVert^2\le\varepsilon_{\mathrm{syn}}^2. With ν^m⋆\hat{\nu}_m^\star their partners’ empirical law,

EW22(ν^m,ν^m⋆)≤εsyn2,EW2(ν^m,ηB1)≤εsyn+EW2(ν^m⋆,ηB1).(5)\mathbb{E}W_2^2(\hat{\nu}_m,\hat{\nu}_m^\star)\le\varepsilon_{\mathrm{syn}}^2,\qquad \mathbb{E}W_2(\hat{\nu}_m,\eta_B^1)\le\varepsilon_{\mathrm{syn}}+\mathbb{E}W_2(\hat{\nu}_m^\star,\eta_B^1). \tag*{(5)}

Proof Disintegrate an optimal coupling of (ν,ηB1)(\nu,\eta_B^1) at each actual ξ^i\hat{\xi}_i, using independent auxiliary draws across indices to obtain ξi⋆\xi_i^\star. Euclidean spaces admit these conditional laws. The pairs have the claimed laws and squared cost. Matching pairs proves the first inequality; the triangle inequality and Jensen prove the second.

If εsyn=0\varepsilon_{\mathrm{syn}}=0, then ν=ηB1\nu=\eta_B^1. For a measurable region OO fixed under the conditioning, at least one of the mm independent generated points hits OO with probability 1−(1−ηB1(O))m1-(1-\eta_B^1(O))^m. Thus synthesis can fill empirical coverage gaps without expanding the support of the target conditional law. These statements concern candidates before rejection and condition on the retained anchor sample.

Estimating the field with latent-only minibatch matching

This section analyzes the matching cost used by RIFT under an independent-batch sampling model. The additional regularity and sampling conditions below are needed only for this estimation result, not for Theorem 1.

Population conditions. Suppose the joint source classes are supported in {∥u∥≤Ru}\{\lVert u\rVert\le R_u\}, where Ru>0R_u > 0. Put U~=(U,1)\widetilde{U}=(U,1) and define

MA=EμˉA0[U~U~T]≻0,MB=EμB0[U~U~T],M_A=\mathbb{E}_{\bar{\mu}_A^0}[\widetilde{U}\widetilde{U}^{\mathsf{T}}]\succ0,\qquad M_B=\mathbb{E}_{\mu_B^0}[\widetilde{U}\widetilde{U}^{\mathsf{T}}],
a∗=λmin⁡(MA),κM=∥MB1/2MA−1/2∥.a_*=\lambda_{\min}(M_A),\qquad\kappa_M=\lVert M_B^{1/2}M_A^{-1/2}\rVert.

Let Py=μˉA,zyP_y=\bar{\mu}_{A,z}^y denote the source latent marginal for class yy. Suppose that Tz=∇φzT_z=\nabla\varphi_z pushes P0P_0 to P1P_1, where φz\varphi_z is globally convex and LTL_T-smooth, with LT>0L_T>0. On the normalized source high class, assume

R=ψA(Z)R=\psi_A(Z)

almost surely, where ψA\psi_A is LψL_\psi-Lipschitz on its support.

Reference coupling. For this estimation analysis, specialize the population source coupling used in Theorem 1 to

TA(z,r)=(Tz(z),ψA(Tz(z))),πA=(id⁡,TA)#μˉA0.T_A(z,r)=\bigl(T_z(z),\psi_A(T_z(z))\bigr),\qquad\pi_A=(\operatorname{id},T_A)_\#\bar{\mu}_A^0.

The lift TAT_A is needed only μˉA0\bar{\mu}_A^0-almost surely.

Estimation experiment. Conditionally on the fixed representation and normalization, draw NAN_A independent points from each source class, with the two samples independent. Divide each sample into

J=NAkJ=\frac{N_A}{k}

disjoint batches of size kk, where k≥1k\ge1 divides NAN_A. Within each pair of batches, solve the latent assignment problem exactly, using a fixed measurable tie rule and independent batchwise randomness when the matching procedure is randomized. Fit δ^\widehat{\delta} by ordinary least squares to all resulting joint latent–reward displacements.

Minibatch-bias quantity. Let P^ky\widehat{P}_k^y denote the empirical latent law of kk independent draws from PyP_y. Define the expected excess matching cost and its associated bias certificate by

excess⁡k:=EW22(P^k0,P^k1)−W22(P0,P1)≥0,Bk:=(1+Lψ2)LTexcess⁡k.(6)\operatorname{excess}_k:=\mathbb{E}W_2^2(\widehat{P}_k^0,\widehat{P}_k^1)-W_2^2(P_0,P_1)\ge0,\qquad B_k:=\sqrt{(1+L_\psi^2)L_T\operatorname{excess}_k}. \tag*{(6)}

Proposition A.1 (Finite-sample field estimation). Under the setup above, there exist constants C1,C2C_1,C_2, depending only on RuR_u and a∗a_*, such that for every η∈(0,1)\eta\in(0,1) and

J≥C1log⁡(2/η),J\ge C_1\log(2/\eta),

with probability at least 1−η,1-\eta,

εest≤κM[Bk+C2log⁡(2/η)J].(7)\varepsilon_{\mathrm{est}}\le\kappa_M\left[B_k+C_2\sqrt{\frac{\log(2/\eta)}{J}}\right]. \tag*{(7)}

Scope. The smooth latent transport, source-high reward graph, positive-design condition, and independent-batch experiment are sufficient conditions for Proposition A.1; they are not required by Theorem 1. The smooth transport is assumed and does not need to exist for arbitrary singular class laws. The condition MA≻0M_A \succ0 can also fail, for example when the source low-class reward is an affine function of its latent coordinates. In such cases, Theorem 1 still applies to a fixed affine field and its actual estimation error, but Proposition A.1 does not provide a certificate for that error.

Lemma A.1 (Same-marginal excess-cost control) Let P,QP, Q have finite second moments. Suppose T=∇φT = \nabla\varphi pushes PP to QQ, where φ\varphi is convex and globally LL-smooth, L>0L > 0. For every π∈Π(P,Q)\pi\in\Pi(P, Q),

Eπ∥Y−T(X)∥2≤L[Eπ∥X−Y∥2−W22(P,Q)].(8)\mathbb{E}_{\pi}\lVert Y-T(X)\rVert^2 \le L\left[\mathbb{E}_{\pi}\lVert X-Y\rVert^2-W_2^2(P,Q)\right]. \tag*{(8)}

Proof. This is the transport-plan estimate of Li and Nochetto [20]; the following gap argument gives the stated finite-moment version. For the ordinary conjugate, evaluating at w=x+(y−T(x))/Lw=x+(y-T(x))/L and using smoothness gives

φ(x)+φ∗(y)−⟨x,y⟩≥∥y−T(x)∥22L.\varphi(x)+\varphi^*(y)-\langle x,y\rangle\ge\frac{\lVert y-T(x)\rVert^2}{2L}.

Global smoothness bounds ∣φ(x)∣\lvert\varphi(x)\rvert quadratically and ∥T(x)∥\lVert T(x)\rVert linearly. Fenchel equality at T(X)T(X) therefore makes φ∗\varphi^* integrable under Q=T#PQ=T_{\#}P. The gap is nonnegative and zero on (id⁡,T)#P(\operatorname{id},T)_{\#}P, so this coupling minimizes quadratic cost. With fixed marginals, the integrated gap is half the excess cost in equation A.5. Integration proves the claim.

Proof [Proof of Proposition A.1] Let Πk\Pi_k be the law of a uniformly selected matched joint pair from one batch. A permutation uses every point once, so Πk∈Π(μˉA0,μˉA1)\Pi_k \in\Pi(\bar{\mu}_A^0,\bar{\mu}_A^1); its latent cost is EW22(P^k0,P^k1)\mathbb{E}W_2^2(\hat{P}_k^0,\hat{P}_k^1). This proves nonnegativity in equation A.3. Let δk\delta_k be the population affine fit under Πk\Pi_k. Since MB⪯κM2MAM_B \preceq\kappa_M^2M_A, every affine ff satisfies ∥f∥L2(μˉB0)≤κM∥f∥L2(μˉA0)\lVert f\rVert_{L^2(\bar{\mu}_B^0)}\le\kappa_M\lVert f\rVert_{L^2(\bar{\mu}_A^0)}.

Bias. Put a=δk−δaffa=\delta_k-\delta_{\mathrm{aff}}. The normal equations and common input marginal give

∥a∥L2(μˉA0)2=EΠk⟨Y−TA(X),a(X)⟩≤(EΠk∥Y−TA(X)∥2)1/2∥a∥L2(μˉA0).\lVert a\rVert_{L^2(\bar{\mu}_A^0)}^2=\mathbb{E}_{\Pi_k}\langle Y-T_A(X),a(X)\rangle\le\left(\mathbb{E}_{\Pi_k}\lVert Y-T_A(X)\rVert^2\right)^{1/2}\lVert a\rVert_{L^2(\bar{\mu}_A^0)}.

Both high endpoints lie on the source reward graph, so

∥Y−TA(X)∥2≤(1+Lψ2)∥ZY−Tz(ZX)∥2.\lVert Y-T_A(X)\rVert^2\le(1+L_\psi^2)\lVert Z_Y-T_z(Z_X)\rVert^2.

Apply Lemma A.1 to the latent marginal of the mean population coupling Πk\Pi_k, not to an individual empirical coupling with different marginals. It follows that ∥δk−δaff∥L2(μˉA0)≤Bk\lVert\delta_k-\delta_{\mathrm{aff}}\rVert_{L^2(\bar{\mu}_A^0)}\le B_k.

Sampling. Write δk(u)=Θku~\delta_k(u)=\Theta_k\tilde{u}, δ~(u)=Θ~u~\tilde{\delta}(u)=\tilde{\Theta}\tilde{u}, and R1=Ru2+1R_1=\sqrt{R_u^2+1}. The normal equations give

Θk=EΠk[(Y−X)X~T]MA−1,\Theta_k=\mathbb{E}_{\Pi_k}[(Y-X)\tilde{X}^{\mathsf{T}}]M_A^{-1},

so ∥Θk∥≤2RuR1/a∗\lVert\Theta_k\rVert\le2R_uR_1/a_*. For batch jj set

Mj=1k∑i∈jX~iX~iT,Zj=1k∑i∈j(Yi−Xi−ΘkX~i)X~iT.M_j=\frac{1}{k}\sum_{i\in j}\tilde{X}_i\tilde{X}_i^{\mathsf{T}},\qquad Z_j=\frac{1}{k}\sum_{i\in j}(Y_i-X_i-\Theta_k\tilde{X}_i)\tilde{X}_i^{\mathsf{T}}.

These are independent across batches, with EMj=MA\mathbb{E}M_j=M_A, EZj=0\mathbb{E}Z_j=0, ∥Mj∥F≤R12\lVert M_j\rVert_F\le R_1^2, and ∥Zj∥F≤B:=2RuR1(1+R12/a∗)\lVert Z_j\rVert_F\le B:=2R_uR_1(1+R_1^2/a_*). For independent matrices AjA_j with ∥Aj∥F≤B0\lVert A_j\rVert_F\le B_0, independence and Jensen bound the expected centered-average Frobenius norm by B0/JB_0/\sqrt{J}. McDiarmid’s inequality [28], with per-batch sensitivity 2B0/J2B_0/J, yields

∥1J∑j(Aj−EAj)∥F≤B0J(1+2log⁡(1/ζ))\left\|\frac{1}{J}\sum_j(A_j-\mathbb{E}A_j)\right\|_F \le\frac{B_0}{\sqrt{J}}\left(1+\sqrt{2\log(1/\zeta)}\right)

with probability at least 1−ζ1-\zeta. Apply this to MjM_j and ZjZ_j with ζ=η/2\zeta=\eta/2 each. Put M^A=J−1∑jMj\widehat{M}_A=J^{-1}\sum_j M_j and Z^=J−1∑jZj\widehat{Z}=J^{-1}\sum_j Z_j. For J≥C1log⁡(2/η)J\ge C_1\log(2/\eta) the event gives ∥M^A−MA∥≤a∗/2\|\widehat{M}_A-M_A\|\le a_*/2, hence M^A⪰MA/2\widehat{M}_A\succeq M_A/2 and ∥M^A−1∥≤2/a∗\|\widehat{M}_A^{-1}\|\le2/a_*. In particular,

∥M^A−1MA1/2∥2=∥M^A−1MAM^A−1∥≤2∥M^A−1∥≤4/a∗.\left\|\widehat{M}_A^{-1}M_A^{1/2}\right\|^2=\left\|\widehat{M}_A^{-1}M_A\widehat{M}_A^{-1}\right\|\le2\|\widehat{M}_A^{-1}\|\le4/a_*.

The empirical normal equations now give

Θ^−Θk=Z^M^A−1,∥δ^−δk∥L2(μˉ 0)=∥Z^M^A−1MA1/2∥F≤2a∗∥Z^∥F≤C2log⁡(2/η)J.\widehat{\Theta}-\Theta_k=\widehat{Z}\widehat{M}_A^{-1},\qquad\|\widehat{\delta}-\delta_k\|_{L^2(\bar{\mu}^{\,0})}=\|\widehat{Z}\widehat{M}_A^{-1}M_A^{1/2}\|_F\le\frac{2}{\sqrt{a_*}}\|\widehat{Z}\|_F\le C_2\sqrt{\frac{\log(2/\eta)}{J}}.

For c∗=1/log⁡2+2c_* = 1/\sqrt{\log2}+\sqrt{2}, valid choices are C1=4c∗2R14/a∗2C_1=4c_*^2R_1^4/a_*^2 and C2=2c∗B/a∗C_2=2c_*B/\sqrt{a_*}. Combining bias, sampling, and the affine source-to-target norm comparison proves equation A.4. ■

Rate and scope. The expected absolute quadratic-cost error of Chizat et al. [4], rescaled from diameter one to a radius-RuR_u ball, implies

excess⁡k≤CdzRu2rdz(k),rd(k)={k−1/2,d<4,k−1/2log⁡(1+k),d=4,k−2/d,d>4.(9)\operatorname{excess}_k\le C_{d_z}R_u^2r_{d_z}(k),\qquad r_d(k)= \begin{cases} k^{-1/2}, & d<4,\\ k^{-1/2}\log(1+k), & d=4,\\ k^{-2/d}, & d>4. \end{cases} \tag*{(9)}

The expected excess is nonnegative and bounded by that absolute error. See Manole and Niles-Weed [27] for the broader smooth-cost analysis. Thus the high-dimensional bias certificate is of order k−1/dzk^{-1/d_z}, while the sampling term uses J=NA/kJ=N_A/k independent batches. Bias does not need to vanish as NAN_A grows at fixed kk [6]. Radius and moment conditioning also carry dimension dependence; the bound is not dimension-free.

Note that proposition A.1 is an idealized independent-batch result. Its assumptions—including the reward graph, fixed representation, and positive design condition—are sufficient conditions for the bound rather than empirical diagnostics of the reported experiments.

Conditional target-return guarantee

The following result connects synthesis accuracy to a specified fitted or analytical surrogate.

(R1) Target and data laws. The target has discount γ∈[0,1)\gamma\in[0,1), initial law ρ\rho, bounded reward rB(s,a)r_B(s,a), and deterministic dynamics S′=fB(S,A)S'=f_B(S,A). The reward and dynamics are respectively LrL_r- and LfL_f-Lipschitz on a domain containing the real, candidate synthetic, and comparison coordinates. Let ηA,ηB\eta_A,\eta_B be the full-context transition–reward laws underlying ηcy\eta_c^y. Genuine target tuples obey R=rB(S,A)R=r_B(S,A) and S′=fB(S,A)S'=f_B(S,A). Source rewards are bounded but does not need to equal rB(S,A)r_B(S,A), and source dynamics may differ.

(R2) Filtering and joint mixture. Fix a measurable acceptance event KK with pkeep=ν(K)>0p_{\mathrm{keep}}=\nu(K)>0, and let νK=ν(⋅∣K)\nu_K=\nu(\cdot\mid K). Retained actions are feasible at their states. Without filtering take K=ΞK=\Xi and pkeep=1p_{\mathrm{keep}}=1. For fixed mixture weights λ,θ∈[0,1]\lambda,\theta\in[0,1], let

ηλ,θ=(1−λ)ηA+λ(1−θ)ηB+λθνK,dλ,θ=its state–action marginal.(10)\eta_{\lambda,\theta}=(1-\lambda)\eta_A+\lambda(1-\theta)\eta_B+\lambda\theta\nu_K,\qquad d_{\lambda,\theta}=\text{its state--action marginal}. \tag*{(10)}

For uniform sampling within the empirical target pool, the corresponding synthetic fraction is θ=mkeep/(nB+mkeep)\theta=m_{\mathrm{keep}}/(n_B+m_{\mathrm{keep}}). The population training backup is

(Bλ,θV)(s,a)=Eηλ,θ[R+γV(S′)∣S=s,A=a].(11)(\mathcal{B}_{\lambda,\theta}V)(s,a)=\mathbb{E}_{\eta_{\lambda,\theta}}[R+\gamma V(S')\mid S=s,A=a]. \tag*{(11)}

It conditions the joint mixture: component weights does not need to remain constant after conditioning on (s,a)(s,a).

(R3) Surrogate regularity and fitting. Let M^\widehat{\mathcal{M}} be a fitted or analytical surrogate with the same spaces, discount, and initial law, bounded rewards, and state--action backup B^\widehat{\mathcal{B}}. Conditional laws, backups, policies, and values have fixed measurable definitions at all coordinates used below; extensions beyond observed support are part of the chosen model. For π∈{π^,πc}\pi\in\{\widehat{\pi},\pi_c\}, its actual policy value V^π=VM^π\widehat{V}^{\pi}=V^{\pi}_{\widehat{\mathcal{M}}} satisfies

∥V^π∥∞≤Vmax⁡,Lip⁡(V^π)≤LV.\lVert\widehat{V}^{\pi}\rVert_{\infty}\le V_{\max},\qquad\operatorname{Lip}(\widehat{V}^{\pi})\le L_V.

The bounds also apply on the extensions used in the backups. Define

εfit(π)=Edλ,θ[∣(B^−Bλ,θ)V^π(S,A)∣].(12)\varepsilon_{\mathrm{fit}}(\pi)=\mathbb{E}_{d_{\lambda,\theta}}\left[\left|(\widehat{\mathcal{B}}-\mathcal{B}_{\lambda,\theta})\widehat{V}^{\pi}(S,A)\right|\right]. \tag*{(12)}

(R4) Occupancy domination. For both policies, let

dBπ=(1−γ)∑t=0∞γtPr⁡Bπ((St,At)∈⋅),dBπ≤Cπdλ,θ(13)d_B^{\pi}=(1-\gamma)\sum_{t=0}^{\infty}\gamma^t\Pr_B^{\pi}((S_t,A_t)\in\cdot),\qquad d_B^{\pi}\le C_{\pi}d_{\lambda,\theta} \tag*{(13)}

for a finite CπC_{\pi}. All displayed error expectations are finite. This is a population coverage condition; in continuous spaces a finite atomic empirical law cannot generally replace dλ,θd_{\lambda,\theta}.

Continuing error budget. Define source discrepancies in the decoded coordinates and reward units used by the policy learner:

εA,r=EηA∣R−rB(S,A)∣,εA,f=EηA∥S′−fB(S,A)∥.\varepsilon_{A,r}=\mathbb{E}_{\eta_A}|R-r_B(S,A)|,\qquad\varepsilon_{A,f}=\mathbb{E}_{\eta_A}\lVert S'-f_B(S,A)\rVert.

Write

Ksyn=1+(γLV)2+(Lr+γLVLf)2,(14)K_{\mathrm{syn}}=\sqrt{1+(\gamma L_V)^2+(L_r+\gamma L_VL_f)^2}, \tag*{(14)}

from the reward, next-state, and state--action blocks, and set

εdata=(1−λ)(εA,r+γLVεA,f)+λθKsynεsynpkeep,Rλ,θ(π)=Cπ1−γ[εfit(π)+εdata].(15)\varepsilon_{\mathrm{data}}=(1-\lambda)(\varepsilon_{A,r}+\gamma L_V\varepsilon_{A,f})+\lambda\theta K_{\mathrm{syn}}\frac{\varepsilon_{\mathrm{syn}}}{\sqrt{p_{\mathrm{keep}}}},\qquad\mathcal{R}_{\lambda,\theta}(\pi)=\frac{C_{\pi}}{1-\gamma}[\varepsilon_{\mathrm{fit}}(\pi)+\varepsilon_{\mathrm{data}}]. \tag*{(15)}

Here εsyn\varepsilon_{\mathrm{syn}} is exactly equation 1. Source immediate-reward error has no factor γ\gamma; it vanishes under shared state--action rewards.

Episodic definitions

For terminating tasks keep (R1)–(R4), with the following replacements. Let τB(s′)∈{0,1}\tau_B(s') \in\{0,1\} be the measurable target termination function, term⁡B(s,a)=τB(fB(s,a))\operatorname{term}_B(s,a)=\tau_B(f_B(s,a)), and TB={s′:τB(s′)=1}\mathcal{T}_B=\{s':\tau_B(s')=1\}. Genuine transitions have their own context’s correct flag dd; synthetic transitions use d=τB(s′)d=\tau_B(s'). The initial law is supported on target nonterminal states and the final transition’s reward is included. The boundary of TB\mathcal{T}_B is in the ambient Euclidean state space.

Attach the flags to the same mixture laws in equation A.7. Replace the training backup by

(Bλ,θV)(s,a)=Eηλ,θ[R+γ(1−d)V(S′)∣S=s,A=a].(\mathcal{B}_{\lambda,\theta}V)(s,a)=\mathbb{E}_{\eta_{\lambda,\theta}}\left[R+\gamma(1-d)V(S')\mid S=s,A=a\right].

The surrogate includes its own continuation flag, with (R3) still required. Hard termination does not need to preserve value regularity; Appendix § 7.§ 5 supplies a concrete sufficient surrogate class. If TT is the target termination time, use the survival occupancy

dBπ=(1−γ)∑t=0∞γtPBπ((St,At)∈⋅,t<T)d_B^\pi=(1-\gamma)\sum_{t=0}^{\infty}\gamma^t\mathbb{P}_B^\pi((S_t,A_t)\in\mathord{\cdot},t<T)

in (R4), and use flagged backups in εfit\varepsilon_{\mathrm{fit}}.

Put εA,d=EηA∣d−term⁡B(S,A)∣\varepsilon_{A,d}=\mathbb{E}_{\eta_A}|d-\operatorname{term}_B(S,A)| and bB(t)=ηB1(dist⁡(S′,∂TB)≤t)b_B(t)=\eta_B^1(\operatorname{dist}(S',\partial\mathcal{T}_B)\le t). For ε>0\varepsilon>0 and p>0p>0, define

ΩB(ε,p)=min⁡{1,1pinf⁡t>0[bB(t)+max⁡{1,Lf2}ε2t2]}.(16)\Omega_B(\varepsilon,p)=\min\left\{1,\frac{1}{p}\inf_{t>0}\left[b_B(t)+\frac{\max\{1,L_f^2\}\varepsilon^2}{t^2}\right]\right\}. \tag*{(16)}

Set ΩB(0,p)=0\Omega_B(0,p)=0; distance to an empty boundary is +∞+\infty. For fixed pp, the positive-error limit is min⁡{1,ηB1(S′∈∂TB)/p}\min\{1,\eta_B^1(S'\in\partial\mathcal{T}_B)/p\}, so it tends to zero when the boundary has zero target-high mass. Exact partners still have identical flags at zero error, including on the boundary.

In the episodic version, add

Eterm=γVmax⁡[(1−λ)εA,d+λθΩB(εsyn,pkeep)](17)\mathcal{E}_{\mathrm{term}}=\gamma V_{\max}\left[(1-\lambda)\varepsilon_{A,d}+\lambda\theta\Omega_B(\varepsilon_{\mathrm{syn}},p_{\mathrm{keep}})\right] \tag*{(17)}

to Edata\mathcal{E}_{\mathrm{data}} in equation A.12, and use the resulting Rλ,θ\mathcal{R}_{\lambda,\theta}. For continuing tasks all flags and Eterm\mathcal{E}_{\mathrm{term}} are zero and T=∞T=\infty.

Theorem A.1 (Synthesis-to-return comparison). Assume (R1)–(R4), using the continuing or episodic definitions, and the decoded coupling bound in Theorem 1. Then for π∈{π^,πc}\pi\in\{\hat{\pi},\pi_c\},

∣JB(π)−JM^(π)∣≤Rλ,θ(π).(18)|J_B(\pi)-J_{\widehat{M}}(\pi)|\le\mathcal{R}_{\lambda,\theta}(\pi). \tag*{(18)}

If JM^(π^)≥JM^(πc)−εlearnJ_{\widehat{M}}(\hat{\pi})\ge J_{\widehat{M}}(\pi_c)-\varepsilon_{\mathrm{learn}} for εlearn≥0\varepsilon_{\mathrm{learn}}\ge0, then

JB(π^)≥JB(πc)−εlearn−Rλ,θ(π^)−Rλ,θ(πc).(19)J_B(\hat{\pi})\ge J_B(\pi_c)-\varepsilon_{\mathrm{learn}}-\mathcal{R}_{\lambda,\theta}(\hat{\pi})-\mathcal{R}_{\lambda,\theta}(\pi_c). \tag*{(19)}

The result holds for each realized surrogate and policies satisfying these conditions; it requires neither comparator optimality nor nested policy classes.

Subtracting any baseline return JB(π0)J_B(\pi_0) gives the corresponding improvement comparison without a coverage assumption on π0\pi_0. Strict gain follows only if the comparator’s advantage over that baseline exceeds the error budget. No positive comparator advantage, IQL-specific rate, or data-driven certificate of these population quantities is supplied by the theorem.

Lemma A.2 (Uniform training-law consistency). Under (R1)–(R2) and the decoded coupling bound, every VV with ∥V∥∞≤Vmax⁡\lVert V\rVert_\infty\le V_{\max} and Lip⁡(V)≤LV\operatorname{Lip}(V) \le L_V satisfies

Eηλ,θ∣bV∣≤Edata,bV(ξ,d)=r+γ(1−d)V(s′)−(BBV)(s,a),\mathbb{E}_{\eta_\lambda,\theta}|b_V| \le\mathcal{E}_{\mathrm{data}},\qquad b_V(\xi,d)=r+\gamma(1-d)V(s')-(\mathcal{B}_B V)(s,a),

where (BBV)(s,a)=rB(s,a)+γ(1−term⁡B(s,a))V(fB(s,a))(\mathcal{B}_B V)(s,a)=r_B(s,a)+\gamma(1-\operatorname{term}_B(s,a))V(f_B(s,a)). Use zero flags and the continuing budget in continuing tasks.

Proof. For any flagged transition, adding and subtracting (1−d)V(fB(s,a))(1-d)V(f_B(s,a)) gives

∣bV(ξ,d)∣≤∣r−rB(s,a)∣+γLV∥s′−fB(s,a)∥+γVmax⁡∣d−term⁡B(s,a)∣.(20)|b_V(\xi,d)| \le|r-r_B(s,a)|+\gamma L_V\lVert s'-f_B(s,a)\rVert+\gamma V_{\max}|d-\operatorname{term}_B(s,a)|. \tag*{(20)}

Couple ξ^∼ν\hat{\xi}\sim\nu to ξ⋆∼ηB1\xi^\star\sim\eta_B^1 as in Theorem 1, and put Z=∥ξ^−ξ⋆∥Z=\lVert\hat{\xi}-\xi^\star\rVert, so EZ2≤εsyn2\mathbb{E}Z^2\le\varepsilon_{\mathrm{syn}}^2. The phantom is target-consistent. Lipschitz continuity and Cauchy–Schwarz on the reward, next-state, and state–action blocks bound the first two terms by KsynZK_{\mathrm{syn}}Z. Conditioning on acceptance gives mean at most

KsynE[Z1K]pkeep≤Ksynεsynpkeep.K_{\mathrm{syn}}\frac{\mathbb{E}[Z1_K]}{p_{\mathrm{keep}}}\le K_{\mathrm{syn}}\frac{\varepsilon_{\mathrm{syn}}}{\sqrt{p_{\mathrm{keep}}}}.

For termination put M⋆=dist⁡(s⋆′,∂TB)M_\star=\operatorname{dist}(s^{\star'},\partial\mathcal{T}_B) and a=max⁡(1,Lf)a=\max(1,L_f). Both s^′\hat{s}' and fB(s^,a^)f_B(\hat{s},\hat{a}) lie within aZaZ of s⋆′s^{\star'}. If M⋆>aZM_\star>aZ, that ball avoids the boundary, so their flags agree. Splitting on Z>t/aZ>t/a and using Markov’s inequality gives, for each t>0t>0,

pkeepEνK∣d−term⁡B(S,A)∣≤bB(t)+a2εsyn2/t2.p_{\mathrm{keep}}\mathbb{E}_{\nu_K}|d-\operatorname{term}_B(S,A)|\le b_B(t)+a^2\varepsilon_{\mathrm{syn}}^2/t^2.

Taking the infimum and capping the conditional probability at one yields equation A.13 for positive error. For zero error the partners coincide almost surely, so flags agree even on the boundary. Thus

EνK∣bV∣≤Ksynεsynpkeep+γVmax⁡ΩB(εsyn,pkeep).\mathbb{E}_{\nu_K}|b_V|\le K_{\mathrm{syn}}\frac{\varepsilon_{\mathrm{syn}}}{\sqrt{p_{\mathrm{keep}}}}+\gamma V_{\max}\Omega_B(\varepsilon_{\mathrm{syn}},p_{\mathrm{keep}}).

On target data bV=0b_V=0; on source data equation A.17 gives EηA∣bV∣≤εA,r+γLVεA,f+γVmax⁡εA,d\mathbb{E}_{\eta_A}|b_V|\le\varepsilon_{A,r}+\gamma L_V\varepsilon_{A,f}+\gamma V_{\max}\varepsilon_{A,d}. Expanding the mixture proves the claim. All flag terms vanish in the continuing case. This bounds retained-sample consistency, not the filtered law’s Wasserstein distance. ▪

Proof [Proof of Theorem A.1] Covered Bellman error. For either policy write eπ=(B^−BB)V^πe_\pi=(\widehat{\mathcal{B}}-\mathcal{B}_B)\widehat{V}^{\pi}. Conditional Jensen, Lemma A.2, the fitting definition, and occupancy domination give

EdBπ∣eπ∣≤CπEdλ,θ∣eπ∣≤Cπ[εfit(π)+Edata].(21)\mathbb{E}_{d_B^\pi}|e_\pi|\le C_\pi\mathbb{E}_{d_{\lambda,\theta}}|e_\pi|\le C_\pi[\varepsilon_{\mathrm{fit}}(\pi)+\mathcal{E}_{\mathrm{data}}]. \tag*{(21)}

The consistency estimate is uniform over bounded Lipschitz VV, so no independence of the learned surrogate is needed.

Trajectory telescoping. We use the same-policy Bellman-residual identity (e.g., [e.g. 46 Lemma 1]), retaining survival indicators for episodic tasks. Let It=1{t<T}I_t=1_{\{t<T\}} along a target trajectory, extended arbitrarily after termination, and let rt=rB(St,At)r_t=r_B(S_t,A_t). Then It(1−term⁡B(St,At))=It+1I_t(1-\operatorname{term}_B(S_t,A_t))=I_{t+1}. The surrogate policy Bellman equation, averaged over the action at each live state, yields

∑t=0n−1γtEBπ[Iteπ(St,At)]=JM^(π)−EBπ∑t=0n−1γtItrt−γnEBπ[InV^π(Sn)].\sum_{t=0}^{n-1}\gamma^t\mathbb{E}_{B}^{\pi}[I_t e_\pi(S_t,A_t)] =J_{\widehat{\mathcal{M}}}(\pi)-\mathbb{E}_{B}^{\pi}\sum_{t=0}^{n-1}\gamma^t I_t r_t-\gamma^n\mathbb{E}_{B}^{\pi}[I_n\widehat{V}^{\pi}(S_n)].

The final term is bounded by γnVmax⁡\gamma^n V_{\max}. Bounded rewards and values allow n→∞n \to\infty, giving

JM^(π)−JB(π)=11−γEπDBϵπ.J_{\widehat{\mathcal{M}}}(\pi)-J_B(\pi)=\frac{1}{1-\gamma}\mathbb{E}_{\pi_D^B}\epsilon_\pi.

Together with equation A.18, this proves equation A.15. The learned-policy condition then gives

JB(π^)≥JM^(π^)−Rλ,θ(π^)≥JB(πc)−ϵlearn−Rλ,θ(π^)−Rλ,θ(πc),J_B(\hat{\pi}) \ge J_{\widehat{\mathcal{M}}}(\hat{\pi})-\mathcal{R}_{\lambda,\theta}(\hat{\pi}) \ge J_B(\pi_c)-\epsilon_{\mathrm{learn}}-\mathcal{R}_{\lambda,\theta}(\hat{\pi})-\mathcal{R}_{\lambda,\theta}(\pi_c),

which is equation A.16. Setting T=∞T=\infty and all flags to zero proves the continuing case with the same argument.

Interpretation. Filtering and termination enter through the displayed consistency penalties, while source reward and dynamics mismatch can remain even with perfect synthesis. The theorem leaves the source-data effects on coverage, fitting, and optimization explicit, so it does not select an optimal λ\lambda. The quantities ϵfit\epsilon_{\mathrm{fit}} and ϵlearn\epsilon_{\mathrm{learn}} are theoretical error terms rather than observed training losses.

A regular analytical surrogate

Hard continuation flags can make exact surrogate values discontinuous even at nonterminal states. The correction ΩB\Omega_B controls flag disagreement, not those value jumps. The following construction verifies (R3) for a common analytical reference. It is a finite-outcome model of the kernel-interpolation type ([31]).

Fix outcomes (yi,ri,di)(y_i,r_i,d_i), i=1,…,n∗i=1,\ldots,n_*, with ∣ri∣≤R∗|r_i|\le R_*, R∗>0R_*>0, di∈{0,1}d_i\in\{0,1\}, and yiy_i in the state domain, including required extensions. Choose nonnegative smooth weights with

∑iwi(u)=1,∑i∣wi(u)−wi(v)∣≤Lw∥u−v∥,Lw<∞.\sum_i w_i(u)=1,\qquad\sum_i |w_i(u)-w_i(v)|\le L_w\lVert u-v\rVert,\qquad L_w<\infty.

At (s,a)(s,a), select ii with probability wi(s,a)w_i(s,a), receive rir_i, and stop if di=1d_i=1; otherwise continue at yiy_i. The resulting stochastic surrogate has the same discount and initial law, and

(B^V)(s,a)=∑iwi(s,a)[ri+γ(1−di)V(yi)].(\widehat{B}V)(s,a)=\sum_i w_i(s,a)\left[r_i+\gamma(1-d_i)V(y_i)\right].

Stopping uses the sampled flag, not an additional hard stopping rule imposed by the target terminal set. Both continuation probability and the weighted continuation outcomes vary smoothly.

Lemma A.3 (Value regularity of the interpolated surrogate) For any deterministic policy with Lip⁡(π)≤Lπ\operatorname{Lip}(\pi)\le L_\pi on this domain, the exact surrogate value satisfies

∥V^π∥∞≤V∗:=R∗1−γ,Lip⁡(V^π)≤V∗Lw1+Lπ2.\lVert\widehat{V}^{\pi}\rVert_\infty\le V_*:=\frac{R_*}{1-\gamma},\qquad\operatorname{Lip}(\widehat{V}^{\pi})\le V_*L_w\sqrt{1+L_\pi^2}.

No contraction of physical dynamics in the Euclidean state metric is assumed.

Proof. The policy Bellman operator is a sup-norm contraction with modulus at most γ\gamma, and bounded rewards give V∗V_*. At its fixed point, V^π(s)=∑iwi(s,π(s))biπ\widehat{V}^{\pi}(s)=\sum_i w_i(s,\pi(s))b_i^\pi, where biπ=ri+γ(1−di)V^π(yi)b_i^\pi=r_i+\gamma(1-d_i)\widehat{V}^{\pi}(y_i) is independent of the current state and ∣biπ∣≤R∗+γV∗=V∗|b_i^\pi|\le R_*+\gamma V_*=V_*. Therefore

∣V^π(s)−V^π(t)∣≤V∗∑i∣wi(s,π(s))−wi(t,π(t))∣≤V∗Lw1+Lπ2∥s−t∥.(22)|\widehat{V}^{\pi}(s)-\widehat{V}^{\pi}(t)|\le V_*\sum_i|w_i(s,\pi(s))-w_i(t,\pi(t))|\le V_*L_w\sqrt{1+L_\pi^2}\lVert s-t\rVert. \tag*{(22)}

For deterministic Lipschitz π^,πc\hat{\pi}, \pi_c, one common surrogate satisfies (R3) with Vmax⁡=V∗V_{\max}=V_* and LV=V∗Lwmax⁡π∈{π^,πc}1+Lπ2L_V=V_*L_w\max_{\pi\in\{\hat{\pi},\pi_c\}}\sqrt{1+L_\pi^2}. The result holds for every realized outcome set and weight function satisfying these conditions, including fitted choices.

Fitting cost and scope. For any measurable conditional coupling Γ\Gamma of training outcomes (R,S′,d)(R,S',d) and surrogate outcomes (R~,S~′,d~)(\widetilde{R},\widetilde{S}',\widetilde{d}) at the same (S,A)(S,A), with expectation also over dλ,θd_{\lambda,\theta}, conditional Jensen and the add-and-subtract argument in equation A.17 give

εfit(π)≤EΓ[∣R−R~∣+γLV∥S′−S~′∥+γVmax⁡∣d−d~∣].\varepsilon_{\mathrm{fit}}(\pi)\leq\mathbb{E}_{\Gamma}\left[|R-\widetilde{R}|+\gamma L_V\|S'-\widetilde{S}'\|+\gamma V_{\max}|d-\widetilde{d}|\right].

Smoothing discrepancy enters εfit\varepsilon_{\mathrm{fit}}. The target-flag and filtering penalties, including ΩB\Omega_B, still apply. Steeper weights can enlarge LVL_V, while broader averaging can enlarge fitting error; neither is asserted small. Coverage and the learning comparison must hold for this same surrogate. This construction gives a sufficient condition for value regularity for the surrogate class considered here.

Environment Setting

We construct the same environment settings as the ODRL benchmark ([24]) and use the ODRL offline datasets directly. This matches the benchmark protocol used by recent cross-domain offline RL methods, including OTDF, STC, and DmC ([26, 49, 42]), allowing for comparison under the same settings.

Source and Target Datasets

We consider four MuJoCo locomotion tasks: Ant, HalfCheetah, Hopper, and Walker2d. The source datasets are the corresponding D4RL datasets distributed with ODRL. We consider medium (med), medium-replay (med-r), and medium-expert (med-e) source datasets. The full source dataset is used in each experiment and contains approximately one million transitions, depending on the D4RL dataset ([8]).

For the target domain, we use the limited offline datasets provided by ODRL. ODRL provides medium and expert target datasets containing 5,000 transitions. To construct the medium-expert dataset, we follow the OTDF and STC protocol of combining two whole trajectories from the medium dataset with three from the expert dataset, to get a total of 5,000 transitions. We use the provided target datasets directly rather than recollecting trajectories. For each experimental setting, the same target dataset is held fixed across all five policy training seeds.

Target Dynamics Shifts

The source environment uses the default MuJoCo dynamics. The target environment modifies these dynamics according to the ODRL benchmark. The descriptions below follow its released XML assets ([25]). We study gravity, friction, and morphology shifts.

Gravity shift. For the gravity shift, the magnitude of the gravitational acceleration is reduced by a factor of 0.5 while preserving its direction. In the MuJoCo XML, this corresponds to changing

<option gravity="0 0 -9.81" ... /> to

<option gravity="0 0 -4.905" ... />

for the target environment.

Friction shift. ODRL halves explicitly specified geometry friction values. In Hopper, the torso, thigh, and leg entries change from friction="0.9" to friction="0.45", and the foot entry changes from 2.0 to 1.0. The other tasks halve their explicit tuples and geometry-specific overrides; omitted coefficients retain inherited defaults.

Morphology shift. For the morphology shift, we use ODRL’s medium-level change to one body part of each robot. Unlike the gravity and friction shifts, this alters model geometry rather than gravity or friction coefficients:

  • HalfCheetah (thigh): the back and front thigh capsules are shortened from approximate half-lengths 0.145 and 0.133 to 0.057 and 0.053, with the shin and foot bodies repositioned accordingly.

  • Hopper (foot): the foot capsule shrinks from radius 0.06 and half-length 0.195 to 0.036 and 0.117, and the foot body moves downward by 0.10.

  • Walker2d (leg): on both legs, the thigh capsule lengthens from half-length 0.225 to 0.35 and the shin shortens from 0.25 to 0.125, with the foot bodies moved downward by 0.10.

  • Ant (legs): the four ankle capsules are shortened from approximate half-length 0.283 to 0.141 by halving their endpoint offsets; the ankle-body origins are unchanged.

For example, Hopper’s global-coordinate XML changes the foot capsule from

<geom name="foot_geom" size="0.06"
      fromto="-0.13 0 0.1 0.26 0 0.1" ... />

to

<geom name="foot_geom" size="0.036"
      fromto="-0.078 0 0.1 0.156 0 0.1" ... />

in the target environment.

Evaluation

Policies are evaluated only in the corresponding target environment. Following ODRL [24 b], the evaluation statistic GG is the mean undiscounted episodic return, distinct from the discounted JMJ_{\mathrm{M}} used in the theory. The D4RL-style normalized score is

NS=100⋅G−GrandomGexpert−Grandom\mathrm{NS} = 100 \cdot\frac{G - G_{\mathrm{random}}}{G_{\mathrm{expert}} - G_{\mathrm{random}}}

where GrandomG_{\mathrm{random}} and GexpertG_{\mathrm{expert}} are the benchmark reference scores for the corresponding task and shift. Unless otherwise stated, reported results use the final policy checkpoint and average over five policy seeds. The target dataset is fixed across those seeds. For RIFT, each seed refits the VAE, recomputes the OT couplings and field, and retrains the policy, so variation covers full-pipeline training rather than independently sampled target datasets. Appendix D.2.6 separately studies variance from differences in target-data.

Algorithmic Implementation

We follow the implementation settings used in the ODRL benchmark along with their corresponding baseline configurations. We first verify our IQL implementation against the reference results reported by STC, and then list the hyperparameters used for all methods.

IQL Replication

Table 3 compares our IQL implementation with the IQL results reported by STC. We reproduce both the pooled source–target setting and the target-only setting on the eight medium-to-medium gravity and morphology tasks. The pooled aggregate is close to STC (234.0 vs. 232.4), while the target-only aggregate is higher (312.4 vs. 282.0); individual task scores differ.

ShiftEnvironmentIQL (reported by STC)IQL (ours)RIFT
DA∪DB\mathcal{D}_A \cup\mathcal{D}_BDB\mathcal{D}_BDA∪DB\mathcal{D}_A \cup\mathcal{D}_BDB\mathcal{D}_B
GravityAnt10.2 ± 1.832.0 ± 7.610.1 ± 1.429.8 ± 4.940.8 ± 10.3
GravityHalfCheetah39.6 ± 3.30.6 ± 0.542.0 ± 2.32.6 ± 0.647.3 ± 6.6
GravityHopper11.2 ± 1.146.2 ± 3.714.1 ± 4.852.7 ± 12.560.4 ± 7.3
GravityWalker2d28.1 ± 12.928.5 ± 7.430.6 ± 4.326.7 ± 4.546.5 ± 5.5
MorphologyAnt71.4 ± 2.476.8 ± 2.865.5 ± 2.076.9 ± 0.376.2 ± 2.4
MorphologyHalfCheetah24.5 ± 2.412.3 ± 3.722.8 ± 2.419.0 ± 6.657.9 ± 4.4
MorphologyHopper15.9 ± 6.842.5 ± 17.318.1 ± 7.155.5 ± 6.469.4 ± 13.2
MorphologyWalker2d31.5 ± 8.643.1 ± 11.530.8 ± 9.249.2 ± 5.152.8 ± 8.2
Total (displayed means)232.4282.0234.0312.4451.3

Table 3. IQL reference results across eight medium-to-medium gravity and morphology settings. Values report the mean ± standard deviation over five seeds, with the highest mean bolded.

Hyperparameter Setup

We include the baseline algorithms and hyperparameter settings as reported in STC [49] and DmC [42].

IQL. IQL [17] learns a value function using expectile regression,

LV=E(s,a)∼D[L2τ(Q(s,a)−V(s))],L_V = \mathbb{E}_{(s,a)\sim\mathcal{D}}\left[L_2^\tau\left(Q(s,a)-V(s)\right)\right],

where L2τ(u)=∣τ−1{u<0}∣u2L_2^\tau(u)=\left|\tau-1_{\{u<0\}}\right|u^2. Its Q-function uses temporal-difference regression,

LQ=E(s,a,r,s′,d)∼D[(r+γ(1−d)V(s′)−Q(s,a))2].L_Q = \mathbb{E}_{(s,a,r,s',d)\sim\mathcal{D}}\left[\left(r+\gamma(1-d)V(s')-Q(s,a)\right)^2\right].

Here dd denotes environmental termination in the displayed discounted backup, not time-limit truncation by itself. The policy is trained by advantage-weighted behavior cloning,

Lπ=−E(s,a)∼D[exp⁡(βIQL(Q(s,a)−V(s)))log⁡π(a∣s)].\mathcal{L}_{\pi}=-\mathbb{E}_{(s,a)\sim\mathcal{D}}\left[\exp\left(\beta_{\mathrm{IQL}}(Q(s,a)-V(s))\right)\log\pi(a\mid s)\right].

DARA. DARA ([22]) trains domain classifiers to estimate the dynamics discrepancy between source and target transitions. The resulting discrepancy Δr\Delta r is used to modify the source reward as

r^=r−αΔr,\hat{r}=r-\alpha\Delta r,

with the reward penalty clipped to [−10,10][-10,10]. The corrected source transitions are then used for downstream IQL training.

BOSA. BOSA ([23]) models target-domain support using a CVAE and restricts value learning to transitions supported by the estimated target dynamics. Its critic objective takes the form

LBOSA=EDmix[δ(Q)1(log⁡P^tar(s′∣s,a)>ϵth′)]+E(s,a)∼DA[Q(s,a)],\mathcal{L}_{\mathrm{BOSA}}=\mathbb{E}_{\mathcal{D}_{\mathrm{mix}}}\left[\delta(Q)\mathbb{1}\left(\log\hat{P}_{\mathrm{tar}}(s'\mid s,a)>\epsilon'_{\mathrm{th}}\right)\right]+\mathbb{E}_{(s,a)\sim\mathcal{D}_{A}}[Q(s,a)],

where δ(Q)\delta(Q) denotes squared TD error, and ϵth′\epsilon'_{\mathrm{th}} is a log-likelihood threshold. The dynamics ensemble uses its minimum log-likelihood for this test. BOSA additionally constrains policy updates to behavior support in the pooled data ([25]).

IGDF. IGDF ([45]) learns a contrastive representation that assigns each transition a score h(s,a,s′)h(s,a,s') reflecting source–target dynamics mismatch. Target samples are always used for value learning, while source samples are selected according to this score:

LQ=12EDB[(Q−TQ)2]+α2EDA[1(h>hξ%)(Q−TQ)2].\mathcal{L}_{Q}=\frac{1}{2}\mathbb{E}_{\mathcal{D}_{B}}\left[(Q-\mathcal{T}Q)^2\right]+\frac{\alpha}{2}\mathbb{E}_{\mathcal{D}_{A}}\left[\mathbb{1}(h>h_{\xi\%})(Q-\mathcal{T}Q)^2\right].

The threshold hξ%h_{\xi\%} determines the retained fraction of source transitions.

SRPO. SRPO ([48]) estimates stationary-state information using a discriminator trained from high- and low-value transitions. The discriminator is then used to modify the reward,

r^SRPO=r+λDδ(s)1−Dδ(s),\hat{r}_{\mathrm{SRPO}}=r+\lambda\frac{D_{\delta}(s)}{1-D_{\delta}(s)},

and policy learning proceeds using the modified rewards.

OTDF. OTDF ([26]) uses optimal transport to measure the dynamics discrepancy between source and target transitions. Given the optimal coupling μ∗\mu^{*}, it assigns each source transition utu_t the deviation score

d(ut)=−∑t′C(ut,ut′)μt,t′∗.d(u_t)=-\sum_{t'}C(u_t,u_{t'})\mu^{*}_{t,t'}.

In its practical implementation, OTDF computes the coupling using entropy-regularized optimal transport with the Sinkhorn algorithm and normalizes the resulting deviation scores. Source transitions are then selected according to the top-ξ\xi scores and weighted in the TD objective according to their target-domain compatibility. OTDF additionally trains a CVAE on the target dataset and uses it to regularize the learned policy toward the support of the target behavior policy.

STC. STC [49] mitigates the dynamics gap by correcting source transitions toward the target domain. It first pretrains an inverse policy model, forward dynamics model, and reward model on the target dataset for 50,000 steps. For each source transition, STC corrects the action using the inverse policy model and estimates the corrected reward using a first-order Taylor approximation. The corrected transition is retained only when its estimated target-domain dynamics discrepancy satisfies the correction criterion controlled by threshold λ\lambda. The value function is then trained with TD regression, while the policy uses a Q-weighted behavior-cloning objective with coefficient β\beta.

DmC. DmC [42] estimates source–target dynamics proximity using a kk-nearest-neighbor score over transition tuples (s,a,s′)(s, a, s^{\prime}). The resulting score is used both to select and weight source transitions during value learning and to condition a diffusion model that generates additional source samples closer to the target domain. DmC further applies a target-policy regularization term through a conditional VAE to reduce bias toward the source dataset. We include the DmC comparison in Appendix D.§ 5.

IQL.* IQL* uses the same IQL implementation and hyperparameters as above, but follows the same data-mixing protocol used by RIFT, modifying the batch target-to-source ratio.

RIFT. We train a shared VAE using balanced source–target minibatches and add the additional reward-conditioned regularization from § 4.1 to the loss:

Lclass=ηEc∈{A,B}Ey∈{0,1}Ex∼Dyc[KL⁡(qϑ(z0∣x)∥N((2y−1)cq,sq2))].\mathcal{L}_{\mathrm{class}} = \eta\mathbb{E}_{c \in\{A,B\}}\mathbb{E}_{y \in\{0,1\}}\mathbb{E}_{x \sim\mathcal{D}_{y}^{c}} \left[\operatorname{KL}\left(q_{\vartheta}(z_0 \mid x) \mathbin{\|} \mathcal{N}\left((2y-1)c_q,s_q^2\right)\right)\right].

Here qϑq_{\vartheta} is the encoder posterior and η\eta weights the class penalty. We use mini-batch optimal transport with squared Euclidean cost to pair low- and high-reward source samples. Their joint latent–reward displacements are fit with an affine model using ordinary least squares; matching itself uses only squared latent distance. The learned field is then applied to low-reward target anchors to generate synthetic transitions. We similarly modify the batch target-to-source ratio.

Specifications. We summarize the hyperparameter settings for all methods in Table 4. For IQL, DARA, and BOSA, we use the fixed configurations reported by STC [49]. For SRPO, the reward coefficient is selected from {0.1,0.3}\{0.1, 0.3\}. For IGDF, the data selection ratio is fixed to 75%, while the representation dimension is selected from {16,64}\{16, 64\}. For OTDF, we follow the reported implementation settings, using a fixed data selection ratio of 80% and a task-dependent policy coefficient in {0.1,0.5}\{0.1, 0.5\}. For STC, the reward gradient coefficient is fixed to 0.5, while the correction threshold is selected from {1.0,5.0}\{1.0, 5.0\} and the Q-weighted loss coefficient from {0.5,5.0}\{0.5, 5.0\}, with the best configuration reported for each environment. For RIFT, we fix the representation and field-fitting hyperparameters across tasks and evaluate two downstream training configurations, λ∈{0.5,1}\lambda\in\{0.5, 1\}.

MethodHyperparameterValue
Shared IQL backbone
Actor network(256, 256)
Critic network(256, 256)
Learning rate3×10−43 \times10^{-4}
OptimizerAdam
Discount factor0.99
NonlinearityReLU
Target update rate5×10−35 \times10^{-3}
Source-domain batch size128
Target-domain batch size128
Temperature coefficient0.2
Maximum log std2
Minimum log std−20-20
Inverse temperature βIQL\beta_{\mathrm{IQL}}3.0
Expectile τ\tau0.7
DARAClassifier network(256, 256)
Reward penalty coefficient0.1
BOSAPolicy regularization coefficient0.1
Transition coefficient0.1
Threshold parameters ϵ,ϵ′\epsilon, \epsilon^{\prime}log⁡(0.01)\log(0.01)
Value weight0.1
CVAE ensemble size5
SRPODiscriminator network(256, 256)
Data selection ratio0.5
Reward coefficient{0.1,0.3}\{0.1, 0.3\}
IGDFRepresentation dimension{16,64}\{16, 64\}
Contrastive encoder network(256, 256)
Encoder pretraining steps7,000
Importance coefficient1.0
Data selection ratio75%
OTDFCVAE training steps10,000
CVAE learning rate10−310^{-3}
Number of sampled latent variables10
Gaussian standard deviation0.1\sqrt{0.1}
Cost functioncosine
Data selection ratio80%
Policy coefficient{0.1,0.5}\{0.1, 0.5\}
STCCorrection threshold λ\lambda{1.0,5.0}\{1.0, 5.0\}
Reward gradient coefficient α\alpha0.5
Q-weighted loss coefficient β\beta{0.5,5.0}\{0.5, 5.0\}
DmCkk-nearest neighbor5
Data selection ratio50%
Policy regularization coefficient0.1

Table 4. Reported hyperparameter settings: shared IQL settings and method-specific configurations.

Guided-sampling score threshold90%
IQL*Data-mixing coefficient λ\lambda{0.5,1}\{0.5, 1\}
RIFTReward quantile qq0.2
VAE latent dimension16
VAE hidden layers(256, 256)
VAE learning rate / batch / steps10−310^{-3} / 256 / 100,000
OT mini-batch size2,000 per side, 8 problems
Data-mixing coefficient λ\lambda{0.5,1}\{0.5, 1\}

Additional Results

We provide the detailed tables summarized in the main text and additional results.

Target-Quality Results

Table 5 gives the complete target-quality comparison summarized in § 5.2.

TypeSourceTargetIQLDARABOSASRPOIGDFOTDFSTCRIFT (ours)
Gravityhalf-medmed39.6 ±\pm 3.341.2 ±\pm 3.938.9 ±\pm 4.036.9 ±\pm 4.536.6 ±\pm 5.540.7 ±\pm 7.742.4 ±\pm 5.347.3 ±\pm 6.6
half-medmed-e39.6 ±\pm 3.740.7 ±\pm 2.840.4 ±\pm 3.040.7 ±\pm 2.338.7 ±\pm 6.228.6 ±\pm 3.237.3 ±\pm 1.546.2 ±\pm 3.0
half-medexpert42.4 ±\pm 3.839.8 ±\pm 4.440.5 ±\pm 3.939.4 ±\pm 1.639.6 ±\pm 4.636.1 ±\pm 5.343.1 ±\pm 5.745.5 ±\pm 2.6
hopp-medmed11.2 ±\pm 1.117.3 ±\pm 3.815.2 ±\pm 3.312.4 ±\pm 1.015.3 ±\pm 3.532.4 ±\pm 8.043.4 ±\pm 6.160.4 ±\pm 7.3
hopp-medmed-e14.7 ±\pm 3.615.4 ±\pm 2.521.1 ±\pm 9.314.2 ±\pm 1.815.1 ±\pm 3.624.2 ±\pm 3.623.4 ±\pm 6.430.5 ±\pm 17.2
hopp-medexpert12.5 ±\pm 1.619.3 ±\pm 10.512.7 ±\pm 1.711.8 ±\pm 0.914.8 ±\pm 4.033.7 ±\pm 7.837.1 ±\pm 14.148.2 ±\pm 20.8
walk-medmed28.1 ±\pm 12.928.4 ±\pm 13.738.0 ±\pm 11.221.4 ±\pm 7.022.1 ±\pm 8.436.6 ±\pm 2.341.6 ±\pm 4.046.5 ±\pm 5.5
walk-medmed-e35.7 ±\pm 4.730.7 ±\pm 9.740.9 ±\pm 7.234.0 ±\pm 9.935.4 ±\pm 9.144.8 ±\pm 7.536.9 ±\pm 6.548.8 ±\pm 8.5
walk-medexpert37.3 ±\pm 8.036.0 ±\pm 7.041.3 ±\pm 8.639.5 ±\pm 3.836.2 ±\pm 13.644.0 ±\pm 4.049.9 ±\pm 10.142.6 ±\pm 7.6
ant-medmed10.2 ±\pm 1.89.4 ±\pm 0.912.4 ±\pm 2.011.7 ±\pm 1.011.3 ±\pm 1.345.1 ±\pm 12.442.6 ±\pm 8.440.8 ±\pm 10.3
ant-medmed-e9.4 ±\pm 1.210.0 ±\pm 0.911.6 ±\pm 1.310.2 ±\pm 1.29.4 ±\pm 1.433.9 ±\pm 5.423.2 ±\pm 6.125.0 ±\pm 11.7
ant-medexpert10.2 ±\pm 0.39.8 ±\pm 0.611.8 ±\pm 0.49.5 ±\pm 0.69.7 ±\pm 1.633.2 ±\pm 9.035.6 ±\pm 5.615.9 ±\pm 4.7
Morphhalf-medmed24.5 ±\pm 2.421.0 ±\pm 3.924.2 ±\pm 5.618.1 ±\pm 1.823.7 ±\pm 3.421.1 ±\pm 7.619.5 ±\pm 2.257.9 ±\pm 4.4
half-medexpert11.1 ±\pm 1.511.3 ±\pm 0.69.1 ±\pm 1.710.1 ±\pm 1.610.1 ±\pm 0.87.4 ±\pm 2.37.7 ±\pm 1.212.3 ±\pm 1.6
hopp-medmed15.9 ±\pm 6.817.8 ±\pm 10.112.8 ±\pm 0.121.7 ±\pm 7.725.3 ±\pm 9.716.4 ±\pm 7.143.1 ±\pm 23.969.4 ±\pm 13.2
hopp-medexpert15.5 ±\pm 4.113.0 ±\pm 1.44.5 ±\pm 4.010.4 ±\pm 3.113.4 ±\pm 2.014.0 ±\pm 4.053.2 ±\pm 50.362.4 ±\pm 40.9
walk-medmed31.5 ±\pm 8.635.0 ±\pm 10.826.7 ±\pm 6.638.6 ±\pm 5.138.5 ±\pm 8.442.5 ±\pm 3.156.7 ±\pm 8.152.8 ±\pm 8.2
walk-medexpert37.0 ±\pm 6.043.7 ±\pm 5.531.3 ±\pm 8.643.8 ±\pm 8.438.5 ±\pm 4.349.9 ±\pm 5.432.9 ±\pm 7.333.2 ±\pm 18.4
ant-medmed71.4 ±\pm 2.471.5 ±\pm 6.854.8 ±\pm 13.272.8 ±\pm 2.271.8 ±\pm 2.775.1 ±\pm 2.377.2 ±\pm 2.876.2 ±\pm 2.4
ant-medexpert63.9 ±\pm 8.168.9 ±\pm 7.647.5 ±\pm 11.959.0 ±\pm 4.766.1 ±\pm 4.063.8 ±\pm 6.474.1 ±\pm 21.279.0 ±\pm 12.9
Total (displayed means)561.7580.2535.7556.2571.6723.5820.9940.9

Table 5. Performance under gravity and morphology shifts when the source dataset is fixed to medium quality and target-data quality varies. Values report the mean ±\pm standard deviation over five seeds, with the highest mean bolded.

Detailed Ablations

We provide detailed results for reward quantiles, VAE design, source-field controls, conditioning horizons, target-data size and sampling, and fixed mixing weights. Unless otherwise stated, the experiments cover eight medium-to-medium gravity and morphology settings and follow the main evaluation protocol.

Reward Quantile

Table 6 compares reward quantiles. Our default q=0.2q = 0.2 achieves the highest aggregate score, although the best value varies by task.

ShiftTask0 (no augmentation)0.10.2 (ours)0.30.40.5
GravityAnt29.8 ±\pm 4.934.5 ±\pm 11.940.8 ±\pm 10.336.8 ±\pm 14.137.8 ±\pm 4.431.5 ±\pm 6.9
HalfCheetah42.0 ±\pm 2.349.4 ±\pm 2.747.3 ±\pm 6.649.5 ±\pm 3.648.3 ±\pm 3.951.4 ±\pm 2.6
Hopper52.7 ±\pm 12.562.0 ±\pm 14.060.4 ±\pm 7.360.2 ±\pm 8.658.7 ±\pm 6.541.9 ±\pm 12.9
Walker2d30.6 ±\pm 4.334.7 ±\pm 3.246.5 ±\pm 5.542.4 ±\pm 7.941.4 ±\pm 10.040.8 ±\pm 10.9
MorphologyAnt76.9 ±\pm 0.377.1 ±\pm 0.276.2 ±\pm 2.478.0 ±\pm 0.378.0 ±\pm 0.176.8 ±\pm 1.3
HalfCheetah22.8 ±\pm 2.454.5 ±\pm 5.057.9 ±\pm 4.458.0 ±\pm 3.851.9 ±\pm 2.350.9 ±\pm 6.1
Hopper55.5 ±\pm 6.467.7 ±\pm 8.269.4 ±\pm 13.253.3 ±\pm 16.465.7 ±\pm 14.666.3 ±\pm 10.9
Walker2d49.2 ±\pm 5.142.9 ±\pm 15.752.8 ±\pm 8.247.3 ±\pm 18.651.3 ±\pm 8.647.1 ±\pm 4.6
Total (displayed means)359.5422.8451.3425.5433.1406.7

Table 6. Comparison of reward quantiles across eight medium-to-medium gravity and morphology settings. Values report mean ±\pm standard deviation over five seeds, with the highest mean bolded.

VAE Design

Table 7 compares uniformly pooled VAE training (base), balanced source–target sampling (bal), and balanced sampling with reward regularization (bal + reg). Balanced sampling improves the aggregate score from 394.6 to 432.7, and reward regularization further increases it to 451.3. The effects vary by task.

ShiftTaskbasebalbal + reg (ours)
GravityAnt35.8 ±\pm 6.231.5 ±\pm 13.940.8 ±\pm 10.3
HalfCheetah48.3 ±\pm 5.047.6 ±\pm 4.547.3 ±\pm 6.6
Hopper39.2 ±\pm 19.660.4 ±\pm 7.860.4 ±\pm 7.3
Walker2d37.3 ±\pm 6.542.1 ±\pm 8.246.5 ±\pm 5.5
MorphologyAnt78.1 ±\pm 0.277.9 ±\pm 0.276.2 ±\pm 2.4
HalfCheetah58.3 ±\pm 2.857.3 ±\pm 5.157.9 ±\pm 4.4
Hopper64.0 ±\pm 10.768.2 ±\pm 12.069.4 ±\pm 13.2
Walker2d33.6 ±\pm 12.347.7 ±\pm 9.552.8 ±\pm 8.2
Total (displayed means)394.6432.7451.3

Table 7. Comparison of VAE design across eight medium-to-medium gravity and morphology settings. Values report mean ±\pm standard deviation over five seeds, with the highest mean bolded.

Source-Field and High-Reward Resampling Controls

We compare two controls while keeping the VAE and normalization fixed. The target-only field learns the low-to-high field from target transitions and applies it to the same target anchors. Highreward resampling repeats real high-reward target transitions with replacement. Each uses a 1,000-transition augmentation budget; RIFT retains up to 1,000 synthetic transitions after filtering.

ShiftTaskIQL*High-reward resamplingTarget-only fieldRIFT
GravityAnt29.8 ±\pm 4.936.1 ±\pm 6.727.3 ±\pm 9.740.8 ±\pm 10.3
HalfCheetah42.0 ±\pm 2.347.3 ±\pm 7.148.4 ±\pm 4.447.3 ±\pm 6.6
Hopper52.7 ±\pm 12.552.2 ±\pm 8.561.1 ±\pm 10.260.4 ±\pm 7.3
Walker2d30.6 ±\pm 4.342.5 ±\pm 6.738.3 ±\pm 7.946.5 ±\pm 5.5
MorphologyAnt76.9 ±\pm 0.377.0 ±\pm 0.477.6 ±\pm 0.576.2 ±\pm 2.4
HalfCheetah22.8 ±\pm 2.419.2 ±\pm 2.456.7 ±\pm 5.157.9 ±\pm 4.4
Hopper55.5 ±\pm 6.453.2 ±\pm 7.163.5 ±\pm 6.769.4 ±\pm 13.2
Walker2d49.2 ±\pm 5.157.0 ±\pm 7.050.9 ±\pm 8.952.8 ±\pm 8.2
Total (displayed means)359.5384.5423.8451.3

Table 8. Source-field and resampling controls. Values report mean ±\pm standard deviation. IQL* and RIFT use the original paper reference results.

RIFT exceeds the target-only field by 27.5 aggregate points and high-reward resampling by 66.8 points, although the advantage varies by task. The target-only field retains source information in the shared VAE and tests where the field is learned, rather than providing a fully source-free control. These comparisons do not directly measure field invariance or isolate OT from alternative source couplings.

Horizon of the Reward-Class Signal

Immediate reward describes the transition being augmented, while longer-horizon returns also depend on subsequent actions and future dynamics. We compare immediate-reward classes with classes defined by the summed reward over the next nn steps within each transition’s episode, for n∈{5,10,20}n \in\{5, 10, 20\}.

The aggregate scores change by +0.2%, −2.9%, and −0.1% relative to immediate-reward conditioning. We hypothesize that dense rewards already provide useful local task information, limiting the benefit of longer-horizon labels. These results support immediate reward as a simple, effective choice in the evaluated tasks.

Sensitivity to Target-Data Size

We evaluate 500, 1,000, 2,500, and 5,000 target transitions. The 5,000-transition setting uses the original paper results.

RIFT improves aggregate performance at three of the four sizes, but underperforms at 500 transitions. Source data therefore do not remove the need for sufficient target information. Limited coverage or representation quality may contribute to this result, but the experiment does not isolate the cause.

Sensitivity to Target-Data Sampling

Training seeds on a fixed dataset measure training variability, but not sensitivity to which target transitions are available. From each 5,000-transition ODRL target dataset, we draw four random

ShiftTask1 step5 steps10 steps20 steps
GravityAnt40.8±10.340.8 \pm10.335.9±2.235.9 \pm2.244.9±8.944.9 \pm8.940.3±9.440.3 \pm9.4
HalfCheetah47.3±6.647.3 \pm6.648.3±2.948.3 \pm2.951.3±2.151.3 \pm2.150.7±2.150.7 \pm2.1
Hopper60.4±7.360.4 \pm7.354.4±6.854.4 \pm6.853.2±12.553.2 \pm12.560.5±6.660.5 \pm6.6
Walker2d46.5±5.546.5 \pm5.546.1±3.746.1 \pm3.736.5±4.636.5 \pm4.639.9±11.339.9 \pm11.3
MorphologyAnt76.2±2.476.2 \pm2.476.6±2.876.6 \pm2.876.3±1.876.3 \pm1.874.2±1.174.2 \pm1.1
HalfCheetah57.9±4.457.9 \pm4.459.0±2.159.0 \pm2.157.2±2.857.2 \pm2.857.2±4.457.2 \pm4.4
Hopper69.4±13.269.4 \pm13.277.7±5.377.7 \pm5.362.6±7.562.6 \pm7.579.4±4.179.4 \pm4.1
Walker2d52.8±8.252.8 \pm8.254.0±9.554.0 \pm9.556.1±10.656.1 \pm10.648.8±5.848.8 \pm5.8
Total (displayed means)451.3452.0438.1451.0

Table 9. Conditioning-horizon comparison across eight medium-to-medium gravity and morphology settings. Values report mean ±\pm standard deviation. The one-step column uses the original paper results.

Target transitionsIQL∗^{*}RIFTRelative difference
500257.6245.6−4.7%-4.7\%
1,000288.9293.6+1.6%+1.6\%
2,500336.8398.6+18.3%+18.3\%
5,000359.5451.3+25.5%+25.5\%

Table 10. Aggregate performance at different target-data sizes. Relative differences use IQL∗^{*} as the denominator.

subsets of 4,000 transitions. For each subset, we refit the VAE, recompute the OT couplings and field, and retrain the policy.

ShiftTaskIQL*RIFT
GravityAnt22.8 ± 5.543.5 ± 5.6
HalfCheetah41.2 ± 2.648.3 ± 2.1
Hopper49.2 ± 10.856.4 ± 9.2
Walker2d47.8 ± 4.437.4 ± 12.6
MorphologyAnt76.6 ± 0.477.3 ± 0.5
HalfCheetah17.2 ± 2.556.2 ± 6.0
Hopper49.8 ± 19.968.8 ± 6.1
Walker2d48.9 ± 9.241.1 ± 11.1
Total (displayed means)353.5429.0

Table 11. Sensitivity to target-data removal. The full pipeline is rebuilt for each of four 4,000-transition subsets of the original 5,000-transition target dataset.

RIFT achieves higher mean scores in six of eight settings, with aggregate scores of 429.0 versus 353.5. Its advantage therefore persists after removing some target transitions. The subsets overlap and come from the same dataset, so this does not establish robustness across independently collected target datasets.

Fixed Mixing Weights

For RIFT and the augmentation comparisons, we select the better mixing arm for each task using mean evaluation return across seeds. Table 12 fixes λ\lambda across tasks.

Methodλ=0.5\lambda= 0.5λ=1\lambda= 1
IQL234.0312.4
SMOTE262.1337.2
RIFT280.5392.3

Table 12. Aggregate performance with a fixed mixing weight. At λ=0.5\lambda= 0.5, half the policy minibatch uses raw source data; at λ=1\lambda= 1, none does.

RIFT retains the highest aggregate score among these methods under either mixing choice.

HalfCheetah Contribution

Excluding Morph-HalfCheetah from the selected-arm comparison leaves aggregate scores of 393.4 for RIFT and 336.7 for IQL∗\mathrm{IQL}^{*}, a gain of 56.7 points. Excluding both HalfCheetah settings leaves 346.1 versus 294.7, a gain of 51.4 points. The aggregate advantage is therefore not confined to HalfCheetah.

Learning Curves

Figure 4 shows learning curves for augmentation, reward-quantile, and VAE-design ablations under morphology shifts. Differences are larger on Hopper than on Ant.

Learning curves for augmentation, reward-quantile, and VAE-design ablations under morphology shifts

Figure 4. Learning curves for augmentation, reward-quantile, and VAE-design ablations under morphology shifts.

Held-Out Reconstruction and Synthetic-Transition Validity

We filter terminal initial states, and also evaluate both held-out reconstruction and target-simulator replay on synthetic transitions. For these diagnostics, we exclude 500 target transitions (10%) and 5,000 source transitions from VAE fitting.

On held-out target transitions, per-dimension reconstruction MSE ranges from 0.001 to 0.07 times the corresponding dimension’s variance across the eight settings. We also encode and decode each held-out transition, restore the decoded state, apply the decoded action, and compare the simulator’s successor and reward with the generated values.

Generated-transition replay. We apply the same replay diagnostic to 1,000 RIFT proposals per setting and to Mixup and SMOTE proposals built on the same anchors. Successor error is the norm of the difference between the generated and replayed successor, divided by the median norm of a real one-step state change in the corresponding target dataset. An error of 1 therefore represents a typical real-step displacement. Reward MAE compares the reward stored in the generated row with the simulator reward and is divided by the target high-minus-low class mean reward. Realized reward is (rsim−rˉB,low)/(rˉB,high−rˉB,low)(r_{\mathrm{sim}}-\bar{r}_{B,\mathrm{low}})/(\bar{r}_{B,\mathrm{high}}-\bar{r}_{B,\mathrm{low}}); 0 and 1 denote the low- and high-class means. Values can exceed 1. Table 13 reports RIFT’s results.

ShiftTaskHeld-out decoded successor errorGenerated successor errorGenerated reward MAE (class gaps)Generated realized reward
GravityAnt0.250.570.180.87
HalfCheetah0.050.110.120.99
Hopper0.331.460.051.06
Walker2d0.657.070.080.80
MorphologyAnt0.160.290.140.91
HalfCheetah0.050.080.051.08
Hopper0.190.600.190.87
Walker2d0.592.200.050.88

Table 13. Target-simulator replay diagnostics.

Replaying real target transitions yields next-step error of at most 0.05 real steps and reward error below 0.003 class gaps. Averaged equally across the eight settings, RIFT proposals achieve realized reward of 0.93, compared with 0.57 for Mixup and 0.55 for SMOTE. However, we note that next-step accuracy does not predict downstream gains and that these measurements do not guarantee exactly accurate transitions.

Reconstruction across VAE variants. Table 14 reports reconstruction variance explained for HalfCheetah and Walker2d under gravity shifts. This diagnostic is separate from the held-out per-dimension reconstruction errors above.

TaskVAESourceTarget
HalfCheetahbase99.17±0.0399.17 \pm0.0396.41±0.2396.41 \pm0.23
bal98.77±0.0498.77 \pm0.0499.38±0.0699.38 \pm0.06
bal + reg98.52±0.0398.52 \pm0.0399.30±0.0699.30 \pm0.06
Walker2dbase99.36±0.0399.36 \pm0.0398.32±0.0998.32 \pm0.09
bal98.95±0.0298.95 \pm0.0299.65±0.0399.65 \pm0.03
bal + reg98.80±0.0298.80 \pm0.0299.62±0.0399.62 \pm0.03

Table 14. Reconstruction variance explained (%) by the three shared-VAE variants under gravity shifts. Results are mean ± standard deviation.

Friction Results

Table 15 reports results under friction shifts. RIFT obtains the highest aggregate score across the twelve settings, although STC or other baselines perform better on several individual tasks.

SourceTargetIQLDARABOSASRPOIGDFOTDFSTCRIFT (ours)
halfcheetah-medmed69.7 ± 1.366.2 ± 4.168.5 ± 1.266.7 ± 2.166.1 ± 1.966.8 ± 0.667.4 ± 1.072.4 ± 2.6
halfcheetah-medmed-e66.8 ± 0.660.4 ± 11.969.2 ± 0.666.8 ± 3.968.6 ± 0.661.3 ± 5.469.4 ± 1.071.4 ± 1.4
halfcheetah-medexpert68.2 ± 0.467.9 ± 0.569.7 ± 1.367.1 ± 2.368.2 ± 2.568.3 ± 0.667.3 ± 0.670.5 ± 3.2
hopper-medmed24.5 ± 1.724.2 ± 2.225.4 ± 1.926.5 ± 2.027.2 ± 4.329.3 ± 4.627.4 ± 2.038.8 ± 2.9
hopper-medmed-e26.4 ± 2.521.7 ± 5.623.0 ± 1.724.3 ± 3.725.7 ± 2.326.4 ± 2.423.4 ± 5.123.8 ± 11.0
hopper-medexpert21.5 ± 1.124.6 ± 2.525.7 ± 0.821.1 ± 2.723.6 ± 3.825.9 ± 1.141.6 ± 26.137.8 ± 15.2
walker2d-medmed72.6 ± 6.273.9 ± 11.772.1 ± 5.267.5 ± 6.070.3 ± 3.670.4 ± 6.673.6 ± 8.674.1 ± 6.2
walker2d-medmed-e59.0 ± 1.361.5 ± 20.242.8 ± 17.557.4 ± 7.959.1 ± 5.459.9 ± 10.271.2 ± 5.869.7 ± 10.0
walker2d-medexpert52.5 ± 3.660.0 ± 11.751.5 ± 12.952.3 ± 12.754.9 ± 6.149.6 ± 14.667.9 ± 6.664.1 ± 11.9
ant-medmed58.2 ± 2.358.7 ± 2.057.6 ± 4.056.9 ± 2.555.2 ± 3.258.3 ± 0.260.2 ± 3.158.5 ± 0.6
ant-medmed-e59.3 ± 2.558.3 ± 1.060.5 ± 0.358.4 ± 0.457.8 ± 0.558.1 ± 0.445.7 ± 10.558.6 ± 0.4
ant-medexpert58.2 ± 0.358.3 ± 0.559.3 ± 1.857.2 ± 2.058.3 ± 0.256.5 ± 2.360.6 ± 1.866.0 ± 16.7
Total (displayed means)636.9635.7625.3622.2635.0630.8675.7705.7

Table 15. Performance under friction shifts when the source dataset is fixed to medium quality and target-data quality varies. Values report the mean ± standard deviation over five seeds, with the highest mean bolded.

Comparison with DmC

We additionally compare against DmC [42], whose official implementation is available in [41], on the gravity-shift settings reported in its paper. Table 16 reports best-over-training scores: the maximum normalized score over evaluation checkpoints for each seed, followed by the mean and population standard deviation across seeds. This statistic selects checkpoints using target returns, unlike the final-checkpoint statistic in the main experiments, so we present this comparison separately.

SourceTargetDmCRIFT (ours)
half-medmed48.0 ± 0.655.0 ± 1.0
half-med-rmed32.2 ± 0.843.6 ± 1.9
half-med-emed49.9 ± 1.456.6 ± 0.6
hopp-medmed30.6 ± 5.574.4 ± 5.2
hopp-med-rmed35.4 ± 0.771.9 ± 4.3
hopp-med-emed52.2 ± 6.675.0 ± 2.1
walk-medmed52.5 ± 2.056.2 ± 3.0
walk-med-rmed42.2 ± 5.953.9 ± 4.3
walk-med-emed57.5 ± 3.368.0 ± 5.9
ant-medmed56.9 ± 2.264.0 ± 5.6
ant-med-rmed41.8 ± 4.767.8 ± 4.3
ant-med-emed55.7 ± 8.159.9 ± 4.0
Total (displayed means)554.9746.3

Table 16. Best-over-training performance under gravity shifts when the target dataset is fixed to medium quality and source-data quality varies. Values report the mean ± population standard deviation over five seeds, with the highest mean bolded.

Computing Infrastructure

We list the computing infrastructure used to run all experiments in this paper in Table 17. All experiments are CPU-only. Each IQL training run took around 2 hours on a training core, the main bottleneck for compute.

ComponentSpecification
CPUAMD EPYC 9474F / 9654
Coresup to 96 in parallel
Memory4 GB per run

Table 17. Computing infrastructure used to run all experiments in this paper.

References

  1. [1]Nitesh V. Chawla, Kevin W. Bowyer, Lawrence O. Hall, and W. Philip Kegelmeyer. SMOTE: Synthetic minority over-sampling technique. Journal of Artificial Intelligence Research, 16:321–357, 2002. doi: 10.1613/jair.953.DOI
  2. [2]Yevgen Chebotar, Ankur Handa, Viktor Makoviychuk, Miles Macklin, Jan Issac, Nathan Ratliff, and Dieter Fox. Closing the sim-to-real loop: Adapting simulation randomization with real world experience. In 2019 International Conference on Robotics and Automation (ICRA), pages 8973–8979. IEEE, 2019.arxiv.org/abs/1810.05687
  3. [3]Wenyu Chen, Yujia Zhang, Wei Guo, Linli Ma, Yanbo Wang, Pinle Qin, and Jianchao Zeng. Transport or discard: Robust unbalanced optimal transport for cross-domain policy adaptation. In Proceedings of the 43rd International Conference on Machine Learning, volume 306 of Proceedings of Machine Learning Research, pages 18099–18123, 2026. URL https://proceedings.mlr.press/v306/chen26gd.html.
  4. [4]Lénaïc Chizat, Pierre Roussillon, Flavien Léger, François-Xavier Vialard, and Gabriel Peyré. Faster wasserstein distance estimation with the sinkhorn divergence. In Advances in Neural Information Processing Systems, volume 33, pages 2257–2269, 2020.arxiv.org/abs/2006.08172
  5. [5]Benjamin Eysenbach, Swapnil Asawa, Shreyas Chaudhari, Sergey Levine, and Ruslan Salakhutdinov. Off-dynamics reinforcement learning: Training for transfer with domain classifiers. In International Conference on Learning Representations, 2021.arxiv.org/abs/2006.13916
  6. [6]Kilian Fatras, Younes Zine, Rémi Flamary, Rémi Gribonval, and Nicolas Courty. Learning with minibatch wasserstein: Asymptotic and gradient properties. In Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, volume 108 of Proceedings of Machine Learning Research, pages 2131–2141. PMLR, 2020. URL https://proceedings.mlr.press/v108/fatras20a.html.
  7. [7]Arnaud Fickinger, Samuel Cohen, Stuart Russell, and Brandon Amos. Cross-domain imitation learning via optimal transport. In International Conference on Learning Representations, 2022.arxiv.org/abs/2110.03684
  8. [8]Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4RL: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020.arxiv.org/abs/2004.07219
  9. [9]Shani Gamrian and Yoav Goldberg. Transfer learning for related reinforcement learning tasks via image-to-image translation. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 2063–2072. PMLR, 2019.arxiv.org/abs/1806.07377
  10. [10]Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 10835–10866. PMLR, 2023.arxiv.org/abs/2210.10760
  11. [11]Yihong Guo, Yixuan Wang, Yuanyuan Shi, Pan Xu, and Anqi Liu. Off-dynamics reinforcement learning via domain adaptation and reward augmented imitation. In Advances in Neural Information Processing Systems, volume 37, 2024. doi: 10.52202/079017-4331.DOI
  12. [12]Yihong Guo, Yu Yang, Pan Xu, and Anqi Liu. MOBODY: Model-based off-dynamics offline reinforcement learning. In International Conference on Learning Representations, 2026. URL https://openreview.net/forum?id=5r9gWZvtJg.arxiv.org/abs/2506.08460
  13. [13]Donald Hejna, Lerrel Pinto, and Pieter Abbeel. Hierarchically decoupled imitation for morphological transfer. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 4159–4171. PMLR, 2020.arxiv.org/abs/2003.01709
  14. [14]Irina Higgins, Loïc Matthey, Arka Pal, Christopher P. Burgess, Xavier Glorot, Matthew M. Botvinick, Shakir Mohamed, and Alexander Lerchner. β-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework. In International Conference on Learning Representations, 2017.
  15. [15]Xingshuai Huang, Di Wu, and Benoit Boulet. Goal-conditioned data augmentation for offline reinforcement learning. Transactions on Machine Learning Research, 2025.arxiv.org/abs/2412.20519
  16. [16]Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In International Conference on Learning Representations, 2014.arxiv.org/abs/1312.6114
  17. [17]Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=68n2s9ZJWf8.arxiv.org/abs/2110.06169
  18. [18]Jaewoo Lee, Sujin Yun, Taeyoung Yun, and Jinkyoo Park. GTA: Generative trajectory augmentation with guidance for offline reinforcement learning. In Advances in Neural Information Processing Systems, volume 37, 2024. doi: 10.52202/079017-1808.
  19. [19]Guanghe Li, Yixiang Shan, Zhengbang Zhu, Ting Long, and Weinan Zhang. DiffStitch: Boosting offline reinforcement learning with diffusion-based trajectory stitching. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 28597–28609. PMLR, 2024.arxiv.org/abs/2402.02439
  20. [20]Wenbo Li and Ricardo H. Nochetto. Quantitative stability and error estimates for optimal transport plans. IMA Journal of Numerical Analysis, 41(3):1941–1965, 2021. doi: 10.1093/imanum/draa045.DOI
  21. [22]Jinxin Liu, Hongyin Zhang, and Donglin Wang. DARA: Dynamics-aware reward augmentation in offline reinforcement learning. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=9SDQB3b68K.arxiv.org/abs/2203.06662
  22. [23]Jinxin Liu, Ziqi Zhang, Zhenyu Wei, Zifeng Zhuang, Yachen Kang, Sibo Gai, and Donglin Wang. Beyond OOD state actions: Supported cross-domain offline reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 13945–13953, 2024. doi: 10.1609/aaai.v38i12.29302.DOI
  23. [24]Jiafei Lyu, Kang Xu, Jiacheng Xu, Mengbei Yan, Jingwen Yang, Zongzhang Zhang, Chenjia Bai, Zongqing Lu, and Xiu Li. ODRL: A benchmark for off-dynamics reinforcement learning. In Advances in Neural Information Processing Systems, volume 37, 2024a. doi: 10.52202/079017-1912.DOI
  24. [25]Jiafei Lyu, Kang Xu, Jiacheng Xu, Mengbei Yan, Jingwen Yang, Zongzhang Zhang, Chenjia Bai, Zongqing Lu, and Xiu Li. ODRL: Off-dynamics reinforcement learning benchmark code, 2024b. URL https://github.com/OffDynamicsRL/off-dynamics-rl. Official implementation of the ODRL benchmark.DOI
  25. [26]Jiafei Lyu, Mengbei Yan, Zhongjian Qiao, Runze Liu, Xiaoteng Ma, Deheng Ye, Jing-Wen Yang, Zongqing Lu, and Xiu Li. Cross-domain offline policy adaptation with optimal transport and dataset constraint. In International Conference on Learning Representations, 2025.
  26. [27]Tudor Manole and Jonathan Niles-Weed. Sharp convergence rates for empirical optimal transport with smooth costs. The Annals of Applied Probability, 34(1B):1108–1135, 2024. doi: 10.1214/23-AAP1986.DOI
  27. [28]Colin McDiarmid. On the method of bounded differences. In Surveys in Combinatorics, 1989, volume 141 of London Mathematical Society Lecture Note Series, pages 148–188. Cambridge University Press, 1989. doi: 10.1017/CBO9781107359949.008.DOI
  28. [29]Fei Ni, Jianye Hao, Yao Mu, Yifu Yuan, Yan Zheng, Bin Wang, and Zhixuan Liang. MetaDiffuser: Diffusion model as conditional planner for offline meta-RL. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 26087–26105. PMLR, 2023.arxiv.org/abs/2305.19923
  29. [30]Haoyi Niu, Shubham Sharma, Yiwen Qiu, Ming Li, Guyue Zhou, Jianming Hu, and Xianyuan Zhan. When to trust your simulator: Dynamics-aware hybrid offline-and-online reinforcement learning. arXiv preprint arXiv:2206.13464, 2022.arxiv.org/abs/2206.13464
  30. [31]Dirk Ormoneit and Šaunak Sen. Kernel-based reinforcement learning. Machine Learning, 49(2–3):161–178, 2002.DOI
  31. [32]Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems, volume 35, 2022. doi: 10.52202/068431-2011.DOI
  32. [33]Xue Bin Peng, Marcin Andrychowicz, Wojciech Zaremba, and Pieter Abbeel. Sim-to-real transfer of robotic control with dynamics randomization. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 3803–3810. IEEE, 2018. doi: 10.1109/ICRA.2018.8460528.
  33. [35]Zhongjian Qiao, Rui Yang, Jiafei Lyu, Xiu Li, Zhongxiang Dai, Zhuoran Yang, Siyang Gao, and Shuang Qiu. Dual-robust cross-domain offline reinforcement learning against dynamics shifts. In International Conference on Learning Representations, 2026b. URL https://openreview.net/forum?id=oKGe5wcMbt.arxiv.org/abs/2512.02486
  34. [36]Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. In Advances in Neural Information Processing Systems, volume 36, 2023.arxiv.org/abs/2305.18290
  35. [37]Xinxing Shi, Xiaoyu Jiang, and Mauricio A. Álvarez. Neighbour-driven gaussian process variational autoencoders for scalable structured latent modelling. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pages 54934–54959. PMLR, 2025.arxiv.org/abs/2505.16481
  36. [38]Remi Tachet des Combes, Han Zhao, Yu-Xiang Wang, and Geoffrey J. Gordon. Domain adaptation with conditional distribution matching and generalized label shift. In Advances in Neural Information Processing Systems, volume 33, 2020.arxiv.org/abs/2003.04475
  37. [39]Leitian Tao, Xuefeng Du, and Sharon Li. Limited preference data? learning better reward model with latent space synthesis. In Advances in Neural Information Processing Systems, volume 38, 2025.arxiv.org/abs/2509.26074
  38. [40]Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, and Pieter Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. In 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 23–30. IEEE, 2017.arxiv.org/abs/1703.06907
  39. [41]Linh Le Pham Van, Minh Hoang Nguyen, Duc Kieu, Hung Le, Hung The Tran, and Sunil Gupta. DmC: Official code and experimental logs, 2025a. URL https://github.com/linhlpv/DmC. Experimental logs available under offdynamics/logs.
  40. [42]Linh Le Pham Van, Minh Hoang Nguyen, Duc Kieu, Hung Le, Hung The Tran, and Sunil Gupta. DmC: Nearest neighbor guidance diffusion model for offline cross-domain reinforcement learning. In ECAI 2025, 2025b. doi: 10.3233/FAIA251077.
  41. [43]Cédric Villani. Optimal Transport: Old and New, volume 338 of Grundlehren der mathematischen Wissenschaften. Springer Berlin, Heidelberg, 2009. doi: 10.1007/978-3-540-71050-9.
  42. [44]Ruhan Wang, Yu Yang, Zhishuai Liu, Dongruo Zhou, and Pan Xu. Return augmented decision transformer for off-dynamics reinforcement learning. Transactions on Machine Learning Research, 2026.
  43. [45]Xiaoyu Wen, Chenjia Bai, Kang Xu, Xudong Yu, Yang Zhang, Xuelong Li, and Zhen Wang. Contrastive representation for data filtering in cross-domain offline reinforcement learning. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 52720–52743. PMLR, 2024. URL https://proceedings.mlr.press/v235/wen24a.html.arxiv.org/abs/2405.06192
  44. [46]Tengyang Xie and Nan Jiang. Q* approximation schemes for batch reinforcement learning: A theoretical comparison. In Proceedings of the 36th Conference on Uncertainty in Artificial Intelligence, volume 124 of Proceedings of Machine Learning Research, pages 550–559. PMLR, 2020. URL https://proceedings.mlr.press/v124/xie20a.html.arxiv.org/abs/2003.03924
  45. [47]Kang Xu, Chenjia Bai, Xiaoteng Ma, Dong Wang, Bin Zhao, Zhen Wang, Xuelong Li, and Wei Li. Cross-domain policy adaptation via value-guided data filtering. In Advances in Neural Information Processing Systems, volume 36, 2023. doi: 10.52202/075280-3210.DOI
  46. [48]Zhenghai Xue, Qingpeng Cai, Shuchang Liu, Dong Zheng, Peng Jiang, Kun Gai, and Bo An. State regularized policy optimization on data with dynamics shift. In Advances in Neural Information Processing Systems, volume 36, 2023. doi: 10.52202/075280-1427.DOI
  47. [49]Mengbei Yan, Jiafei Lyu, Shengjie Sun, Zhongjian Qiao, Jingwen Yang, Zichuan Lin, Deheng Ye, and Xiu Li. Cross-domain offline policy adaptation via selective transition correction. Transactions on Machine Learning Research, 2026.arxiv.org/abs/2602.05776
  48. [50]Hao-Ran Yang, Chuan-Xian Ren, and You-Wei Luo. COD: Learning conditional invariant representation for domain adaptation regression. In European Conference on Computer Vision, pages 108–125, 2024.arxiv.org/abs/2408.06638
  49. [51]Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In International Conference on Learning Representations, 2018.
  50. [52]Han Zhao, Remi Tachet des Combes, Kun Zhang, and Geoffrey Gordon. On learning invariant representations for domain adaptation. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 7523–7532. PMLR, 2019.

Paper details

vlnc:W_curuj7vq
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

generative AI tools

View statement
Looking for related papers…
Contents