betaFeedback

WorksComputer Science

Meta-Reinforcement Learning with Zero-Shot Reinforcement Learning

Abstract

Meta-reinforcement learning (meta-RL) trains agents to adapt to novel tasks from test-time experience, but requires diverse training environments and reward functions. While environments are becoming cheaper to generate, it remains difficult to anticipate every objective an agent may be asked to pursue at test time. Behavior Foundation Models (BFMs) learn policies from reward-free data and support zero-shot RL (ZSRL): adaptation to new objectives at test time using a large reward-labeled dataset. Recent work has shown that BFM task inference can instead be performed online without prior task-specific data. We note that this capability positions BFMs and meta-RL as alternative approaches for online adaptation to new reward functions in a fixed environment. We then ask whether we can extend BFMs to compete with meta-RL more broadly, including in novel environments, and identify four challenges: environment identification, online data collection, test-time exploration, and learning from mixed reward supervision. We study each individually, then propose MetaBFM, a hybrid method that addresses all four. During training, it combines supervised reward-following with unsupervised learning; at test time, it can interpolate between exploration, exploitation, and ZSRL-style reward inference. We demonstrate that MetaBFM can recover the in-distribution performance of supervised meta-RL while generalizing to reward functions outside the training set. Our work suggests that BFMs and meta-RL are complementary research directions whose combined strengths allow adaptive agents to learn more diverse behaviors from a fixed set of supervised objectives.

Introduction

Meta-reinforcement learning agents improve on novel tasks by adapting their decision-making from test-time experience [1]. This capability relies on diverse training sets of environments and reward functions, and benefits from online data collection over many trials. Despite much algorithmic progress, meta-RL remains limited by the cost of generating these training sets. Most work focuses on toy problems where similar environments and/or reward functions are easily sampled; large-scale applications like AdA [2], where a team of developers repurposed a commercial game engine to generate complex tasks and curricula, are rare exceptions. Generating a diverse set of learnable reward functions that covers the desired range of behavior may be inherently difficult. Reward functions are challenging to design at scale [3, 4], and imperfect proxies can break down under optimization pressure [5]. However, with hardware-accelerated simulators [6, 7], LLM-assisted coding [8], and promptable world models [9, 10], the cost of generating environments is rapidly declining.

Behavior Foundation Models (BFMs) are an emerging class of unsupervised RL methods [11] that train in a reward-free environment by learning to maximize many possible objectives [12]. At test time, a target objective is specified by model inference on a large dataset of trajectories labeled with a particular reward function. BFMs’ unsupervised training and supervised testing datasets are loosely analogous to the pretraining and finetuning paradigm of foundation models in other areas [13]. This adaptation process is often called zero-shot RL (ZSRL) [14] and has shown promising results in domains like humanoid locomotion [15, 16]. Meta-RL and BFMs both enable generalization without additional training, but ZSRL requires a large supervised adaptation dataset in advance, while meta-RL assumes the policy actively collects its own (far smaller) dataset. This distinction makes the two approaches applicable to different types of adaptation problems, and their test-time performance is difficult to compare directly. However, recent work by Rupf et al. [17] demonstrated that ZSRL task inference can be done online, without a preexisting dataset, under the same conditions as meta-RL. As a result, meta-RL and BFMs can now be considered direct competitors on problems that require adaptation to novel reward functions in a fixed environment.

If BFMs can match a broader range of meta-RL’s capabilities, they could help extend generalist RL to domains where train-time reward supervision covers only a narrow range of behaviors. We argue that this requires BFMs to (1) identify and generalize to novel environments, (2) learn effectively without a pre-collected training dataset, (3) make exploration-exploitation tradeoffs at test time, and (4) benefit from available reward supervision. We study each of these subproblems in turn with detailed experiments on a diagnostic toy problem and simulated humanoid locomotion before arriving at a unified framework that addresses all four. The resulting method, MetaBFM, trains a policy that spans meta-RL, ZSRL, and intrinsic exploration. During training, MetaBFM combines supervised reward-following with unsupervised learning while collecting training data online. At test time, we are free to interpolate between exploration, exploitation, and ZSRL-style reward inference. This hybrid method draws on the strengths of meta-RL and ZSRL: it retains meta-RL’s ability to adapt within the training distribution while opening the possibility of generalizing to out-of-distribution reward functions. Our work suggests that BFMs and meta-RL are complementary research directions and that, together, they can shift the emphasis from specifying additional training objectives to designing diverse environments.

Background

Meta-Reinforcement Learning

A meta-RL policy π\pi can be evaluated by its cumulative return over the first kk attempts, or episodes, in a new task. This objective creates an exploration-exploitation tradeoff at test time: the policy may sacrifice performance in early attempts to learn more about the task if exploiting that knowledge later leads to higher cumulative return [18]. Methods differ in whether this tradeoff is managed by the designer as a hyperparameter [19] or learned by the policy itself [20]. Policies may also be given kk free episodes to explore without consequence before being evaluated on attempt k+1k+1; this k-shot setting creates explicit exploration and exploitation phases [21, 22].

Meta-RL’s test-time adaptation is enabled by training on a diverse set of tasks. In this paper, a “task” refers specifically to a combination of an environment and an objective—or, equivalently, to a pair of parameters (cd,cr)(c_d, c_r) that alter the transition dynamics and reward function, respectively. The basic goal of a context-based meta-learner is to use the multi-episodic trajectory1 of interaction with the current task so far, τ:t={s0,a0,r1,s1,a1,…,st}\tau_{:t} = \{s_0, a_0, r_1, s_1, a_1, \ldots, s_t\}, to infer the unknown (cd,cr)(c_d, c_r), exploit the unique characteristics of the task, and maximize returns. Methods broadly differ in whether the parameters (cd,cr)(c_d, c_r) are estimated by supervised regression to ground-truth values [23], as the byproduct of a self-supervised objective such as dynamics modeling [24], or implicitly through the objective of maximizing returns [25]. In each case, the idea is to establish a training objective that is minimized by correctly distinguishing (cd,cr)(c_d, c_r) and then conditioning π\pi on the resulting representation.

Meta-RL is at its best when we can afford sample-inefficient online data collection [2, 26, 27], which may be acceptable when we are willing to trade train-time compute for efficient test-time adaptation. Meta-learning is harder when restricted to a fixed dataset [28, 29, 30, 31], as it suffers from an extreme form of the distribution shift problem that defines offline RL more broadly [32]. Methods also differ in whether task rewards are available during meta-training. Standard meta-RL is supervised by a distribution of reward-labeled tasks, whereas unsupervised meta-RL learns from reward-free interaction and only encounters rewards at test time [1]. We focus on online meta-RL that spans both regimes, meta-training on a set of reward functions while simultaneously learning a reward-free BFM that supports ZSRL-style task inference at test time. This hybrid framing leads to a method distinct from prior unsupervised meta-RL [33, 34, 35], which omits reward supervision entirely, and from prior ZSRL/BFM work, which does not meta-train across environments.

Behavior Foundation Models

BFMs are unsupervised RL agents based on successor features [12, 36]. The idea is to learn a latent-conditioned policy π(a∣s,z)\pi(a \mid s, z) from a reward-free dataset and infer the zz that maximizes a particular task of interest from a reward-labeled dataset, without further training [37]. In general, we let ϕ:S→Rd\phi: \mathcal{S} \to\mathbb{R}^{d} define latent rewards rz(s)=ϕ(s)⊤zr_z(s) = \phi(s)^{\top}z and learn successor features ψ(s,a,z)≈Eπz[∑t≥0γtϕ(st+1)∣s0=s,a0=a]\psi(s, a, z) \approx\mathbb{E}_{\pi_z}\left[\sum_{t\geq0} \gamma^t\phi(s_{t+1}) \mid s_0 = s, a_0 = a\right], so that QSF(s,a,z)=ψ(s,a,z)⊤zQ_{\mathrm{SF}}(s, a, z) = \psi(s, a, z)^{\top}z can serve as the critic in a standard off-policy actor-critic update [38] over some distribution of zz, e.g., Lactor=−Es∼D,z∼p(z),a∼π(⋅∣s,z)[QSF(s,a,z)]\mathcal{L}_{\mathrm{actor}} = -\mathbb{E}_{s\sim\mathcal{D},z\sim p(z),a\sim\pi(\cdot\mid s,z)}[Q_{\mathrm{SF}}(s, a, z)]. In this work, the details of training ψ\psi and ϕ\phi will be interchangeable and are not necessary to follow the main text, so we defer a more complete introduction to Appendix § 7.§ 2. Most recent work is based on the FB algorithm [14, 39], but there are many alternatives and extensions [40, 41, 42, 43, 44, 45]. Our experiments primarily use TD-JEPA [46] because it shares similarities with off-policy supervised RL and lets us transfer implementation details from an optimized meta-RL baseline when appropriate.

BFMs benefit from diverse training datasets with broad state-action coverage, and are most commonly applied to offline datasets originally collected by a policy that optimized for exploration [11, 47]. They adapt to a test-time task by fitting z∗=arg⁡min⁡z∑Dr(ri−ϕ(si)⊤z)2z^{*} = \arg\min_{z}\sum_{\mathcal{D}_r}(r_i - \phi(s_i)^{\top}z)^2 on a reward-labeled dataset to deploy π(⋅∣s,z∗)\pi(\cdot\mid s, z^{*}), and we will refer to this inference process as zero-shot RL (ZSRL). Meta-RL’s inference of crc_r plays a similar role as ZSRL’s inference of z∗z^{*}: both parameters estimate the active reward function and therefore enable generalization to novel objectives. ZSRL relies on an existing offline dataset Dr\mathcal{D}_r to fit z∗z^{*}, after which this parameter is fixed. In contrast, meta-RL typically infers crc_r from τ:t\tau_{:t} (an online Dr\mathcal{D}_r it has collected itself) by updating a belief between timesteps or episodes [48]. This discrepancy makes ZSRL and meta-RL difficult to compare directly. In fact, “zero-shot RL” is, by design, unsuited to zero-shot generalization as typically defined (§ 2.1), where we expect a policy to perform well in a novel task on the first attempt with no prior data. OptiBFM [17] helps to close this gap by demonstrating that, as in meta-RL, we can maintain a belief over z∗z^{*} that is updated between timesteps or episodes.

Generalization of BFMs and Supervised Meta-RL

With OptiBFM’s online task inference enabled, BFMs and meta-RL can be directly compared. We will use a didactic ColorGrid problem as a running example through § 3. ColorGrid creates an n×nn \times n continuous 2D gridworld where the agent receives a reward based on the unit square cell of its current location, and the objective is to navigate to the cell with the highest reward and remain there. The underlying cell layout can be randomly permuted such that any two cells may be adjacent. In short, we can generate (n2)!(n^2)! distinct environments where nearly every transition is unpredictable without a reasonable estimate of the active grid permutation, cdc_d. Randomly sampling a reward for each cell generates infinitely many crc_r values. Additional environment details are in Appendix § 8.

As a first experiment, we split cdc_d and crc_r into train/test sets and generate tasks from their cross product, measuring environment and reward generalization separately. To replicate BFMs’ high-coverage offline setting, we collect large random-policy datasets. Our BFM baseline is the better of FB and TD-JEPA, while our meta-RL baseline is an off-policy variant of RL2 [20] that makes offline training reasonable, if still slightly unfair—we return to this later. Figure 1 reports results for n=9n = 9. Meta-RL follows a predictable trend: adding environments or reward functions improves generalization, but closing the train/test gap requires a diverse mixture of both. In this sense, supervised meta-RL is designed for the bottom-right corner of Figure 1. OptiBFM allows BFMs to match the standard z∗z^* inference procedure, though adaptation barely saturates within the k=6k = 6 test episodes. In contrast, meta-RL has trained on reward functions and is incentivized to learn a narrow prior over crc_r. For now, we are not concerned with the performance gap between BFMs and meta-RL where BFMs show meaningful adaptation. Instead, we focus on BFMs’ relative performance: the train curves in the left column of Figure 1 represent the established BFM setting, and performance holds as evaluated reward functions grow in a fixed environment, even though the reward labels go unused and therefore act as a test set themselves. However, BFMs immediately collapse with multiple training environments, a problem also identified by Bobrin et al. [49].

Figure 1. BFM and Meta-RL Generalization. In offline ColorGrid (n=9n = 9), we vary training environments (columns) and rewards (rows), then evaluate k=6k = 6 episode adaptation on train and held-out tasks. Meta-RL improves with training set diversity but needs many reward functions to generalize; BFMs adapt in the fixed-environment setting (left column), but fail across environments.

BFMs as Meta-RL Agents

BFMs can adapt to unseen reward functions over kk attempts without train-time rewards. However, to be useful in meta-RL, they must also generalize across environments. This is only one of several gaps: meta-RL benefits from online training, while BFMs default to offline datasets; meta-RL manages exploration and exploitation over kk test episodes, while BFMs do not; and complete reward-free learning is often unnecessary in simulated online RL domains. We can usually define some reward-labeled tasks, even if we cannot cover every behavior needed at deployment. We investigate these gaps before proposing a solution that enables a competitive hybrid between meta-RL and BFMs.

Adapting to Novel Dynamics

An RL policy that cannot condition π\pi on estimates of the current dynamics, cdc_d, and reward function, crc_r, learns an average policy under p(cd,cr)p(c_d,c_r) that is generally not optimal for any specific task [50]. BFMs share this problem, but z∗z^* already plays the role of crc_r, leaving us to infer cdc_d. Following context-based meta-RL [1], we augment each trajectory τ\tau with a learned estimate c~d\tilde{c}_d and train π(a∣s,c~d)\pi(a \mid s,\tilde{c}_d). Because context labels are assumed to be unavailable, we use a self-supervised dynamics objective. Unlike BeliefFB [49], we target timestep-level online task inference with latent dynamics representations that avoid method-specific changes to the BFM update.

Figure 2 shows our Latent Context Learning (LCL) architecture, which produces c~d\tilde{c}_d at every timestep during rollouts. We embed each timestep of the trajectory τ:t\tau_{:t} to create the input sequence {h0,…,ht}\{h_0,\ldots,h_t\}, then pass it through a causal sequence model. Let Sθ(τ:t)∈RdS_\theta(\tau_{:t}) \in\mathbb{R}^d be the output representation at timestep tt. This output is trained to identify the active dynamics through self-supervised inverse dynamics modeling (IDM) and latent dynamics modeling (LDM). Because accurate estimates of cdc_d improve these objectives, and because SθS_\theta is the only representation with access to the necessary history, we treat c~t=Sθ(τ:t)\tilde{c}_t = S_\theta(\tau_{:t}) and can train a standard BFM on augmented states sˉt=(st,c~t)\bar{s}_t = (s_t,\tilde{c}_t). SθS_\theta observes action choices to infer dynamics, but we avoid leaking the inverse dynamics model label by staggering the action input by two timesteps. We prefer latent dynamics modeling because forward dynamics modeling (FDM) scales with the size of the state space. Following latent world modeling and self-supervised sequence models [51, 52, 53], we use an EMA target network for ht+1h_{t+1} labels [54], and can optionally encourage c~t\tilde{c}_t to be dissimilar across sequences in each training batch [55]. Appendix § 9.§ 1 and Algorithm 1 provide additional details. By default, we train IDM and LDM jointly; detached FDM prediction error is used only as a diagnostic metric.

Figure 2. Latent Context Learning. (Left) We learn a timestep embedding hth_t and sequence model Sθ(τt)S_{\theta}(\tau_t) that generates an online estimate of the environment parameters, c~t\tilde{c}_t. The representation is trained with self-supervised inverse and latent dynamics objectives; Algorithm 1 gives the full procedure. Action inputs ata_t are necessary context to infer dynamics, but are time-shifted to avoid leaking the IDM label. (Right) Context features learned by a feedforward (FF) or Transformer (Seq.) SθS_{\theta} are presented to either a feedforward or sequence-based supervised RL policy π\pi in ColorGrid.

Context estimates should improve supervised RL by reducing or eliminating the need for long-term memory policies, and Figure 2 (Right) verifies this effect in ColorGrid. By augmenting states with context estimates, we can also make BFM networks context-aware without the instability of sequence models. Figure 3 shows that, when the context representation Sθ(τ)∈RT×dS_\theta(\tau) \in\mathbb{R}^{T \times d} has enough capacity to represent cdc_d, joint IDM and LDM learns accurate dynamics estimates that improve TD-JEPA performance. In Figure 4, we define train/test splits of (cd,cr)(c_d,c_r) task pairs with separate offline datasets, and evaluate generalization as the number of training tasks grows. Since LCL and the downstream BFM agent are independent, they can be trained separately. Pretraining and freezing LCL on the largest task set improves BFM generalization at small training set sizes, but gives no clear advantage over training both systems from scratch on the same offline dataset. This is encouraging because joint training is the only realistic option when the dataset is collected online.

LCL and Generalization

Figure 3. LCL and Generalization. TD-JEPA in ColorGrid n=5n = 5 with various context embedding sizes and training objectives. Performance correlates with lower FDM and IDM error. LDM representations recover similar dynamics prediction as FDM without training on this objective.

Latent context learning improves BFM generalization

Figure 4. Latent context learning improves BFM generalization. We increase the number of training (cd,cr)(c_d,c_r) task pairs with separate offline datasets. Standard TD-JEPA collapses across environments, while LCL-augmented TD-JEPA recovers adaptation and improves test performance as the training set grows. Pretraining LCL is most helpful at small training set sizes, but joint training approaches the same performance at scale.

To further illustrate BFMs’ failure across environments and test whether context learning addresses it, we turn to motion tracking for the G1 humanoid, building on BFM-Zero [16]. We randomize the dynamics through wind and gravity perturbations of varying strength, and evaluate tracking performance on the LAFAN dataset [56], which contains 4.6 hours of motion sequences. We compare a BFM trained without these perturbations (BFM-Single) to one trained across the full range of dynamics (BFM-Multi), with and without the FDM and LDM context-learning objectives. Consistent with our analysis above, a BFM trained across multiple dynamics averages its actions over them, which can degrade performance: in the in-distribution setting, BFM-Multi performs even worse than BFM-Single, suggesting that BFMs suffer substantially when they cannot infer the environment’s dynamics. Context learning recovers much of this gap (Table 1), suggesting its promise for training across domains. Surprisingly, it also improves performance modestly in the out-of-distribution setting. Appendix § 8.§ 5 illustrates these BFM failures and provides additional details on the task and environment.

MethodIn-DistributionOut-of-Distribution
mpjpe-l(↓\downarrow)vel. dist.(↓\downarrow)accel. dist.(↓\downarrow)emd(↓\downarrow)mpjpe-l(↓\downarrow)vel. dist.(↓\downarrow)accel. dist.(↓\downarrow)emd(↓\downarrow)
BFM-Single117.21.642.751.01194.72.814.831.80
BFM-Multi132.71.772.961.13151.62.133.631.39
FDM+BFM-Multi125.31.732.901.09149.32.093.541.37
LDM+BFM-Multi121.51.682.851.05145.92.063.521.36

Table 1. BFMs with latent context learning improve humanoid motion tracking. We compare motion-tracking metrics for humanoid locomotion under in-distribution and out-of-distribution dynamics. BFM-Single trains without wind/gravity randomization and can be viewed as an in-distribution oracle for BFM-Multi, or as a worst-case baseline under OOD dynamics.

Exploring, Exploiting, and Learning Online

BFM algorithms are primarily developed on offline RL benchmarks [11] with behavior datasets generated by an online policy optimizing an intrinsic reward [47, 57, 58]. BFMs rely on high state-action coverage and struggle to learn online because the actor objective Lactor\mathcal{L}_{\mathrm{actor}} collapses onto narrow behavior. Appendix § 11.§ 1 explores the difficulty of online learning and task inference in a standard locomotion benchmark. DVFB [59] addresses dataset diversity by maximizing a combination of QSFQ_{\mathrm{SF}} and a supervised critic trained on intrinsic curiosity rewards, which we will call QRQ_R (Equation (1)). Recent applications of BFMs to humanoid robotics [15, 16] use a similar idea, but derive QRQ_R rewards from behavior in an expert motion-capture dataset.

LDVFB=−Es∼D, z∼p(z), a∼π(⋅∣s,z)[QSF(s,a,z)+αQR(s,a,z)].(1)\mathcal{L}_{\mathrm{DVFB}}=-\mathbb{E}_{s\sim\mathcal{D},\,z\sim p(z),\,a\sim\pi(\cdot\mid s,z)}\left[Q_{\mathrm{SF}}(s,a,z)+\alpha Q_R(s,a,z)\right]. \tag*{(1)}

In meta-RL problems, we can take this idea further. First, we replace the global intrinsic reward—one that decreases as a function of total training steps, such as RND [47]—with an episodic reward that decays within a kk-episode meta-trial [60]. An agent maximizing this reward is incentivized to explore new states across each episode at test time. We use an intrinsic reward that combines an off-the-shelf episodic bonus (NGU [61] or E3B [62]) on LCL timestep features hth_t (Figure 2) with the LCL prediction error from the previous timestep. The latter is a core meta-RL idea [63, 64]: it incentivizes exploration that improves the c~d\tilde{c}_d estimate, which can also improve in-context. Second, because episodic rewards are non-Markov, QRQ_R and π\pi should have memory and take LCL-augmented trajectories τˉ\bar{\tau} as input. Finally, we sample α\alpha during training and condition QRQ_R on the active value:

Lα-BFM=−Eτ∼D,z∼p(z),α∼p(α),a∼π(⋅∣τˉ,z,α)[(1−α)QSF(sˉ,a,z)+αQR(τˉ,a,z,α)].(2)\mathcal{L}_{\alpha\text{-BFM}}=-\mathbb{E}_{\substack{\tau\sim\mathcal{D},\,z\sim p(z),\\ \alpha\sim p(\alpha),\,a\sim\pi(\cdot\mid\bar{\tau},z,\alpha)}}\left[(1-\alpha)Q_{\mathrm{SF}}(\bar{s},a,z)+\alpha Q_R(\bar{\tau},a,z,\alpha)\right]. \tag*{(2)}

The result is a policy that generates more diverse data during training while allowing α\alpha to vary at test time. Due to our choice of intrinsic reward, α=1\alpha= 1 drives the policy to explore new behavior each episode in hopes of (1) finding rewards that improve OptiBFM’s online task inference of z∗z^* and (2) improving the context estimate c~d\tilde{c}_d. In contrast, α=0\alpha= 0 exploits that knowledge to maximize returns under z∗z^*. We can schedule α\alpha to recover meta-RL’s kk-shot exploration-exploitation tradeoff at test time, as demonstrated in Figure 5.

Figure 5. Exploring at Test Time. ColorGrid (n=5n = 5) policies trained by Eq. (2) can vary α\alpha over k=6k = 6 test episodes. High α\alpha explores more of the state space, improving online task estimates, which can then be exploited to outperform the standard BFM policy (α=0\alpha= 0, blue) when rewards are sparse.

Learning from Available Rewards

BFMs assume access to zero reward labels during training. While this may be necessary when learning offline, such as from a web dataset that was not intended for RL, the constraint is unnecessary when learning online in simulated environments. When we create meta-RL or multi-task RL training domains, we can usually create some reward-labeled tasks; the challenge is creating enough to cover all behaviors we may want during deployment. Adding training tasks should improve generalization with diminishing returns, creating a clear path to improving the agent by improving its dataset. Ideally, we would bring this relationship to BFMs. Each new train-time objective can help span the space of possible behaviors and supervise learning progress, while unsupervised learning fills in gaps and finds solutions to objectives we had not considered.

We can approach this problem from two directions. The first is to let reward labels improve the BFM update itself. For example, BFMs learn features ϕ(s)\phi(s) and assume ϕ(s)⊤z=r\phi(s)^\top z = r; when reward labels are available during training, we can enforce this relationship directly with an auxiliary loss. It has also become standard to sample training z∼p(z)z \sim p(z) from recent values of ϕ(s)\phi(s), implicitly assuming evaluation on sparse state-based goal-reaching [14, 39]. With reward labels, we can extend this idea to arbitrary reward functions by using online task inference along the labeled trajectory to estimate z∗z^*, as if the same states and rewards appeared at test time. Appendix § 11.§ 2 evaluates both ideas in ColorGrid and DMC ExoRL, finding that they make ϕ\phi more predictive of rewards but have mixed effects on policy performance. A second approach is to restore supervised meta-RL where rewards are available and create a true hybrid method. We modify the “intrinsic” critic QRQ_R from § 3.2 to learn from a weighted combination of intrinsic rewards, rir_i, and extrinsic (supervised) rewards rer_e:

QR(τˉ,a,z,α,β)≈(1−β)ri+βre+γQR(τˉ,a′,z,α,β)(3)Q_R(\bar{\tau}, a, z, \alpha, \beta) \approx(1-\beta)r_i+\beta r_e+\gamma Q_R(\bar{\tau}, a', z, \alpha, \beta) \tag*{(3)}

where τ′\tau' and a′a' are the next timestep in the standard TD backup. We are then free to repeat the idea from Equation (2) of sampling the weight β\beta during training and conditioning the actor and critic on the current value so that it may also be adjusted at deployment:

Lαβ-BFM=−Eτ∼D,z∼p(z),α∼p(α),β∼p(β),a∼π(⋅∣τˉ,z,α,β)[(1−α)QSF(sˉ,a,z)+αQR(τˉ,a,z,α,β)](4)\mathcal{L}_{\alpha\beta\text{-BFM}} = -\mathbb{E}_{\substack{\tau\sim\mathcal{D},\,z\sim p(z),\\ \alpha\sim p(\alpha),\,\beta\sim p(\beta),\\ a\sim\pi(\cdot\mid\bar{\tau},z,\alpha,\beta)}}\left[(1-\alpha)Q_{\mathrm{SF}}(\bar{s},a,z)+\alpha Q_R(\bar{\tau},a,z,\alpha,\beta)\right] \tag*{(4)}

When β=0\beta= 0, we recover the exploratory ZSRL objective of Eq. (2), and when α=β=0\alpha= \beta= 0, we recover standard ZSRL. Most importantly, when α=β=1\alpha= \beta= 1, we recover supervised meta-RL. In practice, this makes it possible to compete with a standard supervised meta-RL baseline on in-distribution tasks, while opening the possibility of generalizing via zz to novel tasks when meta-RL fails. Figure 6 visualizes this space of possible test-time settings.

Figure 6. A Unified Objective. The Lαβ-BFM\mathcal{L}_{\alpha\beta\text{-BFM}} loss (Eq. (4)) interpolates between zero-shot RL, intrinsic curiosity, and supervised meta-RL. α\alpha and β\beta can be configured to adjust generalization behavior.

MetaBFM

We can now combine the pieces from § 3 into a unified update, MetaBFM, that captures the in-distribution performance of supervised meta-RL and the out-of-distribution generalization of unsupervised BFMs. At each training step, we first update the Latent Context Learning module with its self-contained objective (Algorithm 1). LCL’s primary role is to augment each state with a context estimate c~d\tilde{c}_d, allowing the BFM networks to learn from data generated by multiple environment dynamics (§ 3.1). LCL also produces per-timestep representations that can be passed to off-the-shelf episodic curiosity methods to compute intrinsic rewards rir_i for the trajectory. We then update the BFM networks, excluding the actor π\pi, over a distribution p(z)p(z). Any actions from π\pi required for the BFM update are sampled from π(⋅∣τˉ,z,α=0,β=0)\pi(\cdot\mid\bar{\tau}, z, \alpha= 0, \beta= 0), which recovers the original BFM policy; the remaining details are unchanged, and many ZSRL algorithms are interchangeable here, though we use TD-JEPA. Next, we update the critic head QRQ_R with a standard off-policy backup on weighted intrinsic and extrinsic rewards (Equation (3)) and compute the actor loss Lαβ-BFM\mathcal{L}_{\alpha\beta\text{-BFM}} (Equation (4)), both over samples from p(α,β)p(\alpha, \beta). We collect trajectories from the online environment by sampling from the same distribution of zz, α\alpha, and β\beta.

While this update is more expensive than training the BFM component in isolation, in practice we share network backbones for efficiency. For example, π\pi and QRQ_R share a Transformer backbone to enable long-term memory. Many such training details are copied directly from a strong off-policy meta-RL implementation [65], and Appendix § 9 provides additional discussion. To make extrinsic rewards optional, we force β=0\beta= 0 when collecting or training on reward-free trajectories. Early layers that take extrinsic reward inputs to QRQ_R and π\pi have their outputs multiplied by β\beta, allowing us to insert and automatically ignore an arbitrary placeholder reward.

Figure 1 showed that BFMs can learn adaptive behavior from limited training domains where supervised meta-RL overfits, but also that meta-RL improves with diverse datasets while BFMs do not. MetaBFM’s main goal is to allow for test-time configuration to take advantage of both of these methods’ strengths while being able to avoid their weaknesses. Figure 7 Left returns to this experiment and shows that MetaBFM captures both modes: a single policy can be deployed in “ZSRL mode” (α=0,β=0\alpha= 0, \beta= 0) when meta-RL fails to generalize, or in “reward-following mode” (α=1,β=1\alpha= 1, \beta= 1) when the dataset is diverse enough to support it.

Figure 7. Configurable MetaBFM Inference. (Left) When learning from limited training objectives, MetaBFM in ZSRL mode (β=0\beta= 0) avoids supervised meta-RL’s overfitting. (Center Left) With more objectives, supervised mode recovers the meta-RL baseline. (Center Right) On held-out in-distribution Wind tasks, MetaBFM remains competitive with supervised meta-RL. (Right) On out-of-distribution Wind tasks, ZSRL-style inference adapts over long adaptation horizons.

An extension of learning more general behavior from fewer train-time objectives is generalizing to test-time objectives that are clearly out of distribution from the supervised signals seen during training. We study this with a variant of the Wind environment [29, 66], a common benchmark for OOD generalization. Agents navigate from the origin toward a randomly placed goal, crc_r, while adjusting for wind in a random direction, cdc_d. The training set contains all wind directions but excludes goals in the upper-left quadrant. At test time, we therefore evaluate both held-out in-distribution tasks (any wind direction, goals in the three training quadrants) and out-of-distribution tasks (any wind direction, goals in the excluded quadrant). Figure 7 Right shows the adaptation curves for each set. MetaBFM significantly outperforms the supervised baseline on the OOD test set when configured to maximize successor-feature values, while nearly matching its performance on in-distribution tasks where train-time rewards are useful. Perhaps the most interesting OOD result is the gap between the train-time trajectory length of k=3k = 3 episodes and the length at which online task inference saturates (k>20k > 20). This kind of length extrapolation is rare in black-box meta-RL and may be enabled by offloading the usual context estimation of crc_r outside of the policy weights and into ZSRL’s z∗z^*.

We repeat this experiment on the “walker” locomotion task in the DeepMind Control Suite [67], using a variant of the meta-RL setup from Rothfuss et al. [68]. We generate cdc_d by randomizing body geometry, joint ranges, motor damping, and ground friction. Agents train over k=2k = 2 episodes in 10k random walker variations using only the standard walk reward, and are evaluated on held-out dynamics (Appendix § 8.§ 3). At test time, we evaluate on walk, two closely related objectives (stand and run), and three objectives requiring behaviors not encouraged during supervised training (walk backward, crouch, and crouch walk). Figure 8 shows the same pattern as Wind: on walk, MetaBFM in reward-following mode performs similarly to supervised meta-RL, while ZSRL mode is noticeably weaker. On the three novel objectives, supervised meta-RL fails to follow the test-time reward signal, while performance with ZSRL-style inference continues to improve well past the training horizon.

Figure 8. DMC Walker. Adaptation in held-out environments after training on 10k environments with only the walk reward (left). MetaBFM recovers supervised meta-RL on the train reward and its close variants, and generalizes to novel rewards in ZSRL mode. ZSRL evaluations run for 30 episodes; x-axes are truncated after performance plateaus.

Finally, we evaluate MetaBFM on the more complex MetaWorld continuous-control benchmark [69], using the hardware-accelerated variant from MTBench [70]. Despite learning to maximize a range of actor objectives at both training and test time, MetaBFM converges to near-perfect performance on six ML1 tasks, each testing generalization across hidden-parameter variations of a single tabletop manipulation skill (Appendix Figure 18). Figure 9 reports results for the full multi-task ML45 setting. To reduce computational cost, we initialize off-policy learning with a replay buffer of demonstrations collected from checkpoints of a multi-day meta-RL training run, with exploration noise enabled. We compare baselines trained offline on this dataset with meta-RL and MetaBFM agents that continue collecting online experience. On the training tasks, MetaBFM again performs comparably to supervised meta-RL (Figure 9, left). In exploration mode, MetaBFM continues to visit new regions of the workspace in each episode, while reward-following mode largely repeats its first attempt (Figure 9, center). However, MetaBFM fails to generalize to MetaWorld’s held-out manipulation tasks in either reward-following or ZSRL inference mode (Appendix Figure 19). To test adaptation to novel objectives that require in-distribution behaviors, we add a goal-reaching reward to all 50 training and held-out environments, with hidden targets drawn from states visited during exploration rollouts (Appendix § 8.§ 4). Performance with ZSRL-style inference continues to improve well beyond the k=2k = 2 training horizon, while success rates in reward-following and exploration modes remain near chance (Figure 9, right).

Figure 9. MetaWorld ML45. (Left) In reward-following mode, MetaBFM recovers supervised meta-RL performance on the 45 training tasks. Boxes show interquartile ranges; diamonds show mean success rates. Behavior Data and Best Policy report success rates for the seed dataset and its best contributing policy, respectively. (Center) In exploration mode, MetaBFM continues to expand the gripper’s coverage of the tabletop workspace over a trial. (Right) In ZSRL mode, MetaBFM adapts to a simple unseen goal-reaching reward across all 50 environments.

Conclusion

Meta-RL requires diverse training reward functions, but anticipating every test-time objective remains difficult even as environments become cheaper to generate. Building on OptiBFM’s online task inference, we extend BFMs to match a broader range of meta-RL’s capabilities, including adaptation to novel environments. MetaBFM combines a self-predictive latent context for environment identification, intrinsic curiosity for online data collection and test-time exploration, and supervised value learning to take advantage of mixed reward supervision. The resulting policy recovers the in-distribution performance of supervised meta-RL while generalizing to reward functions outside the training set. Continued progress in meta-RL and BFMs could be mutually reinforcing, advancing generalist RL by shifting the emphasis from specifying additional training objectives to generating more diverse environments.

Acknowledgments and Disclosure of Funding

This work was supported by the National Science Foundation (CAREER Award FRR-2145283 and EFRI-2318065), the Office of Naval Research (N00014-24-1-2550), the Army Research Lab (W911NF-25-1-0065), the KIST-UT collaboration (UTAUS-FA00004578), the JP Morgan Chase Faculty Fellowship in AI, and the Sony Research Awards.

Additional Background

Reinforcement Learning

A Markov Decision Process (MDP) [71] is defined as a tuple (S,A,μ,P,r)(\mathcal{S}, \mathcal{A}, \mu, P, r) where S\mathcal{S} is called the state space, A\mathcal{A} is the action space, μ\mu is the initial state distribution, P:S×A→Δ(S)P : \mathcal{S} \times\mathcal{A} \to\Delta(\mathcal{S}) is the transition function that maps a given state and action to a distribution of next states and r:S→Rr : \mathcal{S} \to\mathbb{R} is the scalar reward function. The agent executes a policy, π:S→Δ(A)\pi: \mathcal{S} \to\Delta(\mathcal{A}), to interact with the environment. The Reinforcement Learning (RL) objective is to find the optimal policy, π∗\pi^{*}, defined as,

π∗=argmax⁡πEπ[∑tγtr(st)](5)\pi^{*} = \underset{\pi}{\operatorname{argmax}} \mathbb{E}_{\pi}\left[\sum_{t}\gamma^{t}r(s_{t})\right] \tag*{(5)}

Behavior Foundation Models

A Behavior Foundation Model (BFM), for a given dynamical system, allows an agent to produce near-optimal policies for any given reward function using little or no additional computation. As a result, this paradigm has often been referred to as Zero-Shot Reinforcement Learning. Specifically, in this work, we consider successor-measure-based BFMs that learn to represent successor measures for a wide range of policies. Mathematically, the successor measure for a policy is defined as,

Mπ(s,a,X)=Eπ[∑t=0∞γtIπ(st∈X∣s0=s,a0=a)],∀X⊂S.(6)M^{\pi}(s, a, X) = \mathbb{E}_{\pi}\left[\sum_{t=0}^{\infty}\gamma^{t}\mathbb{I}^{\pi}(s_{t} \in X \mid s_{0}=s, a_{0}=a)\right], \forall X \subset\mathcal{S}. \tag*{(6)}

BFMs represent MπzM^{\pi_{z}} (successor measure for policy parameterized using zz) as Mπz(s,a,s+)=ψ(s,a,z)Tϕ(s+)M^{\pi_{z}}(s, a, s^{+}) = \psi(s, a, z)^{T}\phi(s^{+}). Connections have been drawn [12, 39, 44] between this representation and successor features with ψ\psi being the successor feature for the state feature Eρ[ϕϕT]ϕ(s)\mathbb{E}_{\rho}[\phi\phi^{T}]\phi(s). Methods like [39, 44, 55] use the following objective to learn these representations,

L(ϕ,ψ)=−Es,a,s′∼ρ[ψ(s,a,z)Tϕ(s′)]+12Es,a,s′∼ρ,s+∼ρ[(ψ(s,a,z)Tϕ(s+)−γψˉ(s′,πz(s′),z)Tϕˉ(s+))2](7)\mathcal{L}(\phi,\psi) = -\mathbb{E}_{s,a,s'\sim\rho}[\psi(s,a,z)^{T}\phi(s')] + \frac{1}{2}\mathbb{E}_{s,a,s'\sim\rho,s^{+}\sim\rho}\left[(\psi(s,a,z)^{T}\phi(s^{+})-\gamma\bar{\psi}(s',\pi_{z}(s'),z)^{T}\bar{\phi}(s^{+}))^{2}\right] \tag*{(7)}

where ρ\rho is the offline dataset. These BFMs consider tasks defined by rewards, rz(s)=ϕ(s)Tzr_{z}(s)=\phi(s)^{T}z for which the corresponding optimal Q function can be written as, Qz∗(s,a)=ψ(s,a,z)TzQ^{*}_{z}(s,a)=\psi(s,a,z)^{T}z with the optimal policy πz=argmax⁡aQz∗(s,a)=argmax⁡aψ(s,a,z)Tz\pi_{z}=\operatorname{argmax}_{a}Q^{*}_{z}(s,a)=\operatorname{argmax}_{a}\psi(s,a,z)^{T}z. Overall, BFMs can be represented using (ϕ,ψ,πz)(\phi,\psi,\pi_{z}). All three networks ϕ\phi, ψ\psi and πz\pi_{z} are learned using reward-free interactions with ϕ\phi and ψ\psi being trained using (7) and πz\pi_{z} as the optimal policy for the reward rz(s)=ϕ(s)Tzr_{z}(s)=\phi(s)^{T}z.

At inference, when a reward function is provided, the BFMs simply need to find the zz for the given reward function as the corresponding policy πz\pi_{z} is already trained to be optimal for rzr_{z}. This zz can be inferred simply by linear regression on reward samples, i.e. z∗=argmin⁡zEρ[(r(s)−ϕ(s)Tz)2]z^{*}=\operatorname{argmin}_{z}\mathbb{E}_{\rho}[(r(s)-\phi(s)^{T}z)^{2}]. Several methods [39, 44, 45, 46, 55] have been proposed with a similar construction of BFMs.

Environment Details

ColorGrid

ColorGrid is an n×nn \times n continuous 2D navigation task in which a hidden reward grid crc_{r} assigns a scalar payoff to each unit cell. The agent does not observe its true position; instead, it sees the cell label scrambled by a hidden permutation cdc_{d}. A task is the pair (cd,cr)(c_{d}, c_{r}), so the agent must resolve two independent unknowns within a trial: how its movements map to observations, and which cell pays the most.

Dynamics and reward. The true state is a continuous position st∈[0,n]2s_t \in[0,n]^2. A 2D velocity action at∈[−1,1]2a_t \in[-1,1]^2 updates it as st+1=clip⁡(st+at,0,n)s_{t+1} = \operatorname{clip}(s_t + a_t,0,n), and the reward is the value of the current true cell, rt=cr[⌊st(y)⌋,⌊st(x)⌋]r_t = c_r[\lfloor s_t^{(y)} \rfloor,\lfloor s_t^{(x)} \rfloor]. Movement and reward follow the true grid; cdc_d only changes what the agent observes. The observation relabels the integer part of the position by cdc_d and keeps the fractional part, so motion within a cell looks smooth, but crossing a cell boundary makes the observed position jump to an unrelated cell (Figure 10). To navigate, the agent must implicitly invert the permutation. Inner episodes last 60 steps for n=5n = 5 and 80 steps for n=9n = 9. We report oracle-normalized return: per-episode return divided by that of an oracle that teleports from the random start cell to arg⁡max⁡cr\arg\max c_r and stays there, so a score of 1.0 matches the oracle.

An example ColorGrid transition for $n = 5$

Figure 10. An example ColorGrid transition for n=5n = 5.

Observations. The observation is three-dimensional: the scrambled 2D position described above, and a flag that is set for one step at each boundary between inner episodes of a trial. Neither cdc_d nor crc_r is observed.

Task distribution. Permutations and reward grids come from two fixed, deduplicated pools, with reward entries sampled i.i.d. from N(0,3)\mathcal{N}(0,3) and clipped to [−10,10][-10,10]. Each pool is split into 10,000 training entries and 10,000 disjoint held-out entries. By default, the ii-th task pairs the ii-th entry of each pool, (cdi,cri)(c_d^i,c_r^i). In Figure 1, tasks instead come from the cross product of the two pools, so a policy can train on many reward functions in a single permutation, or vice versa.

Wind

Wind is a 2D point meta-RL navigation task introduced by Dorfman et al. [29] and adapted by Ni et al. [66] as part of their POMDP suite. The agent commands position deltas on the plane, and the world adds a hidden, task-specific wind vector to its motion at every step. A task is an unobserved (w,g)(w,g) pair in which ww perturbs the dynamics while gg defines the reward landscape. The two hidden variables stress meta-RL along complementary axes. The wind is identifiable only from the difference between commanded actions and observed motion, and the goal only from reward, so a competent policy must jointly infer the dynamics and search for an unmarked target.

Dynamics and reward. The agent’s state is a 2D position st∈R2s_t \in\mathbb{R}^2, advanced by an action at∈[−0.1,0.1]2a_t \in[-0.1,0.1]^2 and a per-task wind w∈[−0.08,0.08]2w \in[-0.08,0.08]^2 via st+1=st+at+ws_{t+1} = s_t + a_t + w. Inner episodes last 75 steps and terminate early on entry into a small disk around the goal gg. The reward is rt=e−∥st−g∥+1[st∈goal disk]r_t = e^{-\lVert s_t-g\rVert} + \mathbf{1}[s_t \in\text{goal disk}], bounded everywhere on the plane with a sharp +1+1 bonus inside the disk.

Observations. The agent receives its own 2D position together with an episode-boundary flag that fires for one step at the boundary between inner episodes within a trial. Both ww and gg are hidden.

Task distribution. During training, we sample 80 winds uniformly from [−0.08,0.08]2[-0.08,0.08]^2 and 80 goal locations on the unit circle, with the circle partitioned into the four standard quadrants. One quadrant (Q2Q_2 in our experiments) is held out from the training goal set, so the agent is never rewarded for entering the upper-left region of the plane during training. The two test environments measure distinct generalization gaps that we report separately.

DMC Walker

We use the dynamics-generalization walker from DMC Remastered [72]: the DeepMind Control walker [67] with its embodiment resampled from a dynamics seed. Observations are state-based, so visual randomization is disabled.

Dynamics and task distribution. Each dynamics seed defines a body by resampling torso and foot geometry, knee and ankle joint ranges, motor damping and armature, and ground friction. We draw 11,000 distinct seeds once. The first nn form the training pool, so smaller pools nest inside larger ones, and the final 1,000 form a disjoint held-out pool shared across all nn. § 4 trains on n=10,000n = 10{,}000 environments.

Observations and actions. The agent receives the 24-D proprioceptive walker state together with an episode-boundary flag that fires for one step at the boundary between inner episodes. The action is a 6-D vector of joint torques in [−1,1]6[-1,1]^6, repeated for two control steps. Inner episodes last 250 agent steps (500 control steps). The body is hidden.

Rewards. Every reward lies in [0,1][0,1] per control step, and we report episode returns rescaled to the standard 0–1000 DMC score. Training uses only the native walk reward (torso height above 1.2 m, upright, forward speed of at least 1 m/s). The native stand and run rewards drop the speed term or raise the target to 8 m/s. Crouch and crouch walk replace the standing height condition of stand and walk with a two-sided band of [0.6,0.9][0.6,0.9] m and a margin of 0.2 m. Walk backward multiplies the standing height term by a velocity ramp that pays 0 when walking forward at 1 m/s, 0.5 at rest, and 1 when walking backward at 1 m/s.

MetaWorld

We base our experiments on the MetaWorld manipulation suite [69], with the reward and goal-masking corrections of McLean et al. [73], run through the GPU-accelerated MTBench reimplementation [70] for batched simulation. Tasks are simulated in NVIDIA IsaacGym [6] with a Franka Panda arm, and episodes are 250 steps.

Tasks and goals. MetaWorld consists of 50 single-arm tabletop manipulation task families (sliding, picking, button-pressing, door opening, etc.), each parameterized by a goal vector specifying a target site or object position drawn from a task-specific range. Each training task family exposes 100 sampled goal variations, giving ML45 a total of 4,500 train configurations across 45 families and 5 held-out test families. The goal vector is masked to zero in the agent’s observation at every timestep, so the agent must infer both the task family and the specific goal from interaction alone.

Action and observation spaces. The action is a 4-D continuous vector a∈[−1,1]4a \in[-1,1]^4. Three components specify a Cartesian end-effector delta and the fourth controls the parallel gripper. The observation is a 39-D continuous vector containing the end-effector position and gripper, the position and quaternion of up to two task-relevant objects (zero-padded for tasks with fewer), the previous-timestep object pose, and the goal slot (masked as described above).

Reward. At each timestep, the agent receives the sum of a dense shaping term and a sparse success bonus. MetaWorld reward functions have changed several times: MetaWorld v2 placed rewards on a more consistent scale across tasks to avoid indirectly leaking task identity, while MTBench scaled the dense reward by 0.01 to reduce returns and improve multi-task RL. Our experiments avoid this issue with categorical value function networks, so we restore the original dense reward scale, as dense rewards are expected to improve OptiB𝖥M online task inference.

Goal-reaching reward. Because ZSRL inference fails on the five held-out ML45 tasks (Figure 19), we instead evaluate a novel behavior in all 50 train and held-out environments (Figure 9 Right) by porting the objective of MetaWorld’s reach task. Objects and dynamics are unchanged, but the task reward is replaced by a function of the distance dd between the gripper and a hidden target: each step yields 10max⁡(0,1−d/1.2)10\max(0,1-d/1.2), or a success bonus of 20 when d<5d < 5 cm. Unlike MetaWorld’s sparse shaping, which is scaled by the initial distance to the goal, this linear ramp gives a constant gradient across the workspace and keeps success a clear step above the dense reward. The target is resampled at the start of each trial and fixed across its episodes. We only sample targets near gripper positions visited by MetaBFM’s exploration-mode rollouts (α=1\alpha= 1, β=0\beta= 0) in the same environment. This lets us verify that every target behavior is in-distribution for the exploration data the BFM is trained on, so it is a behavior ZSRL inference could learn to recover.

Figure 19. ZSRL Inference Noise in ML45. Success of pure ZSRL inference (α=0\alpha= 0, β=0\beta= 0) for different Thompson-sampling noise levels σ\sigma; dashed lines fix a single random zz for the whole trial. Larger σ\sigma trades early success for steady improvement on the train tasks, while the held-out tasks remain unsolved.

Humanoid Locomotion

Following Li et al. [16], we create environments in IsaacLab [7] for humanoids with locomotion tasks. The humanoids are evaluated on motion tracking with LAFAN [56], a diverse and challenging motion dataset.

Although RL methods on robotics usually include domain randomization to achieve robust sim-to-real transfer, we intentionally try to create more dynamic and random environments, for example, different wind and gravity. So apart from the domain randomization on mass, friction and so on, we randomly sample the scale and direction of wind, and gravity scale as shown in Figure 11. Dynamics setting details are shown in Figure 12.

Wind/gravity illustration

Figure 11. Wind/Gravity illustration. During online data collection, the wind and gravity strength in each environment are different – the horizontal arrows indicate the direction and strength of the wind (red as the strongest, white as the weakest), while the length of vertical arrows indicate the strength of gravity.

Domain RandomizationAdditive Observation Noise
ParameterRangeObservationRange
COM Offset [m]U([−0.02,0.02])\mathcal{U}([-0.02, 0.02])qt−qˉq_t-\bar{q}U([−0.01,0.01])\mathcal{U}([-0.01, 0.01])
Link MassU([0.95,1.05])\mathcal{U}([0.95, 1.05])q˙t\dot{q}_tU([−0.5,0.5])\mathcal{U}([-0.5, 0.5])
FrictionU([−0.5,1.25])\mathcal{U}([-0.5, 1.25])gravt\mathrm{grav}_tU([−0.05,0.05])\mathcal{U}([-0.05, 0.05])
Default Joint Pos [m]U([−0.02,0.02])\mathcal{U}([-0.02, 0.02])ω˙troot/4\dot{\omega}^{\mathrm{root}}_t/4U([−0.05,0.05])\mathcal{U}([-0.05, 0.05])
Push Robots [m/s]U([0,0.5])\mathcal{U}([0, 0.5])
Wind Direction [Rad]U([−π,π])\mathcal{U}([-\pi, \pi])
Wind MagnitudeU([0.0,30.0])\mathcal{U}([0.0, 30.0])
Gravity Magnitude [g]U([0.8,1.2])\mathcal{U}([0.8, 1.2])

Table 12. Details in training environment.

State and Observation. The privileged state st∈R463s_t \in\mathbb{R}^{463} includes root height, full body pose, body orientation, as well as linear and angular velocities. The observable state is defined as ot≜(qt−qˉ,q˙t,14ωtroot,gt)∈R64o_t \triangleq(q_t-\bar{q}, \dot{q}_t, \frac{1}{4}\omega_t^{\mathrm{root}}, g_t) \in\mathbb{R}^{64}, where qt∈R29q_t \in\mathbb{R}^{29} is the joint position normalized by the nominal pose qˉ\bar{q}, q˙t∈R29\dot{q}_t \in\mathbb{R}^{29} is the joint velocity, ωtroot∈R3\omega_t^{\mathrm{root}} \in\mathbb{R}^{3} is the root angular velocity, and gt∈R3g_t \in\mathbb{R}^{3} is the projected gravity. To mitigate partial observability, we use an observable history ot,H≜(ot−H,at−H,…,ot)∈R93H+64o_{t,H} \triangleq(o_{t-H}, a_{t-H}, \ldots, o_t) \in\mathbb{R}^{93H+64}, which concatenates past observations and actions over a horizon of length HH. All observation components (except root height in the privileged state) are normalized with respect to the humanoid’s current facing direction and root position. During BFM training, we assume access to a dataset of unlabeled motion trajectories D={τ}\mathcal{D} = \{\tau\}, where each trajectory τ=(o1,s1,…,oT,sT)\tau= (o_1, s_1, \ldots, o_T, s_T) contains both observable and privileged states.

Action. Following PHC [74], we use a proportional-derivative (PD) controller at each DoF of the humanoid, with the action ata_t specifying the PD target. With the target joint set as qtd=atq_t^d = a_t, the torque applied at each joint is τi=kp∘(at−qt)−kd∘q˙t\tau^i = k^p \circ(a_t-q_t) - k^d \circ\dot{q}_t. This is different from the residual action representation used in prior motion imitation methods, where the action is added to the reference pose: qtd=q^t+atq_t^d = \hat{q}_t + a_t to speed up training. We do not use any external forces or meta-PD control. For the 29-degree-of-freedom (DoF) humanoid we use – Unitree G1, the action a∈R29a \in\mathbb{R}^{29} contains the PD controller targets for all DoFs.

BFM failures under multiple dynamics. BFMs trained with multi-dynamics data might suffer from taking actions from the wrong environment dynamics. This can be shown directly via the visualizations in Figure 13. For BFM-Multi, in the stage of trying to get up from the ground, the robot acts as if there were wind from certain directions, loses balance and finally tracks badly. As a comparison, BFMs with context learning can identify the dynamics, so the states are close to the BFM-Single, which is trained solely on one type of dynamics.

Qualitative comparison of humanoid motion tracking showing Reference Motions, BFM-Single, BFM-Multi, and Ours

Figure 13. Qualitative Comparison on Humanoid Motion Tracking.

Implementation Details

Latent Context Learning

Algorithm 1 Latent Context Learning

Input: trajectory batch τ={τ(i)}i=1B\tau= \{\tau^{(i)}\}_{i=1}^{B} with τ(i)={(s0,a0,…,sT)}\tau^{(i)} = \{(s_0, a_0, \ldots, s_T)\}; EMA rate ρ\rho; networks EθE_\theta, SθS_\theta, LDMθ\mathrm{LDM}_\theta, IDMθ\mathrm{IDM}_\theta, FDMθ\mathrm{FDM}_\theta; EMA target encoder E−E^{-}.

Output: augmented trajectory τˉ={(sˉt,at,sˉt+1)}t\bar{\tau} = \{(\bar{s}_t, a_t, \bar{s}_{t+1})\}_t with sˉt=(st,ht,c~t)\bar{s}_t = (s_t, h_t, \tilde{c}_t).

// Augment each τ\tau with dynamics features

ht=Eθ(st,at−2),ht−=E−(st,at−2),c~t=Sθ(τ:t),sˉt=(st,ht,c~t).h_t = E_\theta(s_t, a_{t-2}), \quad h_t^{-} = E^{-}(s_t, a_{t-2}), \quad\tilde{c}_t = S_\theta(\tau_{:t}), \quad\bar{s}_t = (s_t, h_t, \tilde{c}_t).

// Self-supervised dynamics losses

L^LDM=−1T∑i,tcos⁡(LDMθ(ht,at,c~t),ht+1−)\hat{\mathcal{L}}_{\mathrm{LDM}} = -\frac{1}{T}\sum_{i,t}\cos(\mathrm{LDM}_\theta(h_t, a_t, \tilde{c}_t), h_{t+1}^{-})

L^IDM=12T∑i,t∥IDMθ(ht,ht+1,c~t,c~t+1)−at∥2\hat{\mathcal{L}}_{\mathrm{IDM}} = \frac{1}{2T}\sum_{i,t}\lVert\mathrm{IDM}_\theta(h_t, h_{t+1}, \tilde{c}_t, \tilde{c}_{t+1}) - a_t\rVert^2

L^FDM=12T∑i,t∥FDMθ(st,at,sg⁡(c~t))−st+1∥2//c~t detached by default (diagnostic only)\hat{\mathcal{L}}_{\mathrm{FDM}} = \frac{1}{2T}\sum_{i,t}\lVert\mathrm{FDM}_\theta(s_t, a_t, \operatorname{sg}(\tilde{c}_t)) - s_{t+1}\rVert^2 \quad// \tilde{c}_t\ \text{detached by default (diagnostic only)}

L^REG=−1B(B−1)T∑i≠j,t[1−cos⁡(ct(i),ct(j))]\hat{\mathcal{L}}_{\mathrm{REG}} = -\frac{1}{B(B-1)T}\sum_{i\ne j,t}\left[1-\cos(c_t^{(i)}, c_t^{(j)})\right]

Take a gradient step on θ\theta to minimize a weighted sum of (normalized) L^LDM,L^IDM,L^FDM,L^REG\hat{\mathcal{L}}_{\mathrm{LDM}}, \hat{\mathcal{L}}_{\mathrm{IDM}}, \hat{\mathcal{L}}_{\mathrm{FDM}}, \hat{\mathcal{L}}_{\mathrm{REG}}.

EMA target update: E−←(1−ρ)E−+ρEθE^{-} \leftarrow(1-\rho)E^{-}+\rho E_\theta.

return augmented trajectory τˉ\bar{\tau}.

HyperparameterColorGridWindMetaWorldDMC Walker
ϕ\phi-encoder output dim (dim ϕ\phi)200160220256
ψ\psi-encoder output dim (dim ψ\psi)50408050
ϕ\phi-predictor hidden width102480010241024
ψ\psi-predictor hidden width102480010241024
Successor-feature discount (γsf\gamma_{\mathrm{sf}})0.970.970.980.98
Reward multiplier (SF target / BLR)101011
ϕ\phi–ψ\psi orthogonality weight0.100.250.100.10

Table 3. BFM (TD-JEPA) hyperparameters. In reference to notation in Bagatella et al. [46]. Shared across domains: two-tower predictor with three layers (one shared trunk plus two per-head tower layers).

Algorithm 1 summarizes the Latent Context Learning (LCL) procedure, which augments states with features used for BFM training and intrinsic rewards. Per-domain choices are summarized in Table 2. All four domains use small Transformer dynamics encoders. The optional cross-trajectory regularizer is disabled in § 4, though preliminary experiments in Figure 4 enabled it.

HyperparameterColorGridWindMetaWorldDMC Walker
Latent feature dim (dim hh)324168
Inverse-dynamics feature dim3241612
Inverse-dynamics MLP hidden width400128300256
Latent-dynamics predictor hidden width512128300256
Diagnostic probe hidden width400128256256
SθS_\theta (dmodeld_{\mathrm{model}}, layers, heads)Tformer (512, 6, 8)Tformer (128, 2, 4)Tformer (128, 2, 4)Tformer (256, 3, 4)

Table 2. Latent Context Learning hyperparameters. Shared across domains: latent-dynamics and inverse-dynamics loss weights both 1; EMA target rate ρ=5×10−3\rho=5\times10^{-3}; sphere-normalized features; cross-trajectory regularizer weight 0; probe gradients detached.

BFM

While many BFM updates could in principle be used interchangeably, our results use TD-JEPA [46], primarily because it shares the most surface area with off-policy actor–critic RL. The two heads ϕ\phi (state representation) and ψ\psi (task representation) each use a two-tower predictor, sharing a small input trunk before splitting into per-head towers. We additionally apply a cross-trajectory orthogonality regularizer on the predicted features. A scalar reward multiplier (ten for ColorGrid and Wind, one for MetaWorld and DMC Walker) is applied to the per-step reward inside both the SF prediction target and the BLR likelihood at deployment, so it sets the natural magnitude of the regression target r≈⟨ψ(s′),z⟩r \approx\langle\psi(s'), z\rangle and therefore couples directly to the Thompson-sampling noise level σ\sigma discussed in Appendix § 9.§ 3.

Online Task Inference

Following OptiBFM [17], we estimate the latent task vector z∗z^* online with Bayesian linear regression on the forward features ϕ\phi, resampling at every timestep. Our experiments use Thompson sampling rather than the optimistic alternative because it scales to the long meta-rollouts in our evaluations, and preliminary experiments found little performance gap. The posterior is governed by an observation-noise scale σ\sigma and prior precision λ\lambda; we found σ\sigma to be the key hyperparameter, scaling roughly linearly with the typical magnitude of the scaled per-step reward.

HyperparameterColorGridWindMetaWorldDMC Walker
σ\sigma101550.03

Table 4. Online task inference (Thompson Sampling) hyperparameters. Shared across domains: Bayesian linear regression with prior precision λ=1\lambda=1, posterior decay 1, exploration bias κ=0\kappa=0, posterior resampled every environment step.

HyperparameterColorGridWindMetaWorldDMC Walker
Episodic novelty bonus weight1.01.01.01.0
Intrinsic critic reward scale1.01.01.01.0
Episodic novelty: nearest neighbors kk10101010
Episodic novelty: cluster distance threshold8×10−38 \times10^{-3}1×10−31 \times10^{-3}3×10−23 \times10^{-2}8×10−38 \times10^{-3}
Episodic novelty: maximum kernel similarity8.08.08.08.0
Episodic novelty: kernel ϵ\epsilon1×10−41 \times10^{-4}1×10−51 \times10^{-5}3×10−43 \times10^{-4}1×10−41 \times10^{-4}

Table 5. Intrinsic reward hyperparameters.

Intrinsic Rewards

The intrinsic reward at each timestep is a sum of two terms: an episodic novelty bonus computed in the style of NGU [61] on the latent context features, and a term proportional to the previous-timestep latent-dynamics prediction error (returned by Alg. 1). The combined scalar is normalized and exposed to the policy as an additional observation, so the actor can directly condition on the current intrinsic signal in addition to driving the β\beta-blended critic.

Actor–Critic Update

Sequence backbone. The actor and the intrinsic–extrinsic critic share a transformer trajectory encoder with rotary position embeddings [75] and sliding-window FlashAttention [76].

Actor and Critic. The critic, QRQ_R, is a small ensemble of two-hot regression heads over a fixed return range [77], which helps absorb the large variation in TD-target scale induced by randomizing α\alpha and β\beta. Both the actor and critic heads consume a Fourier encoding of the (α,β)(\alpha,\beta) scalars in Equation (4) as additional input features. The actor is an MLP with a standard Tanh-Gaussian output.

Computational Resources and Statistical Significance

All models are trained on an NVIDIA RTX A5000 GPU, with training runs for the MetaBFM (§ 4) experiments lasting 12–36 hours depending on the domain. Shaded error bars in results figures indicate the mean and standard deviation of at least three trials, except for MetaWorld experiments, which use a single training run due to computational constraints. Results indicating test-time adaptation curves (performance vs. kk) average these results over the last 3 evaluation cycles during training, then over seeds.

Limitations

BFM and ZSRL algorithms are an active area of research, and reported gains often depend on hyperparameters, implementation details, and benchmark choice. Our experiments use TD-JEPA as a representative state-of-the-art BFM update, but broader comparisons across BFM objectives and online task inference methods remain important future work. MetaBFM also introduces test-time choices through α\alpha and β\beta. These parameters are useful because they let a single policy interpolate between exploration, exploitation, and ZSRL-style reward inference, but they also create a model-selection problem. In our experiments, we evaluate a few interpretable settings; in deployment, one may need validation tasks or an online rule for choosing α\alpha and β\beta.

HyperparameterColorGridWindMetaWorldDMC Walker
Training sequence length360226501128
Actor / critic context length96128128128
Policy attention window64 timestepsfull attention64 timestepsfull attention
Actor MLP hidden width512300400400
Critic MLP hidden width300300400300
Critic output bins (two-hot)646496256
Critic return range [rmin⁡,rmax⁡][r_{\min}, r_{\max}][−400,600][-400, 600][−50,400][-50, 400][−300,4000][-300, 4000][−100,500][-100, 500]
Target network rate (τ\tau)5×10−35 \times10^{-3}3×10−33 \times10^{-3}5×10−35 \times10^{-3}5×10−35 \times10^{-3}
Number of Fourier frequencies for (α,β)(\alpha, \beta) encoding16889

Table 6. Actor–critic hyperparameters.

Additional Results

Online ZSRL

BFMs are inherently proposed as offline algorithms [14, 44], trained on offline exploratory datasets, which can then be used to directly infer policies for diverse reward functions. Inference solves a linear regression problem (Appendix § 7.§ 2) using samples drawn from the same offline datasets. Prior work [59] has shown that these BFMs are not effective for online training. Methods such as [15, 16] have used expert datasets to guide the online training of these models.

We compare the performance of BFMs when trained offline and online on standard DM Control tasks [67]. We compare (Figure 14) the offline-trained BFM (on ExoRL datasets [78]) using offline inference (labeled offline) with the same model using online inference (labeled offline-online) as proposed by [17] and with a fully online BFM that does not assume access to any external dataset during training or inference.

Offline vs Online BFMs on Walker, Cheetah, and Quadruped

Figure 14. Offline vs Online BFMs: Comparison of fully offline, offline training and online inference and fully online BFMs on DM Control tasks.

It is clear that BFMs are optimally suited to fully offline settings. While using online inference degrades the performance slightly, the performance of online-trained BFM is significantly worse. BFMs require high coverage, which is hard to obtain during online training, leading to poor performance. On the other hand, meta-RL often improves when trained online, as observed from Figure 15.

Figure 15. Online vs. Offline Meta-RL. In ColorGrid (n=5n = 5), offline meta-RL from a static dataset generated by a uniform behavior policy leads to significant distribution shift in long-context representations; online learning substantially improves performance.

Utilizing Available Rewards

BFMs are designed to be trained using reward-free interactions. We design a training procedure by providing occasional reward information to the BFMs to see whether they can make use of the available rewards. On the trajectories where rewards are available, the BFMs exploit the reward function to interact with the environment using an optimal policy (optimal for the partially trained BFM). It adds regularization to the state features to ensure they remain sufficiently predictive of the available rewards, as discussed in § 3.3.

We compare the performance of online-trained BFMs with and without the use of reward functions in Figure 16. The improvement in performance when using available rewards is not consistent. While an improvement is observed in the walker, similar improvements are not observed in the other two environments. On ColorGrid, the BFMs show improvement in performance when leveraging the available rewards, as shown in Figure 17.

Effect of using rewards during training of BFMs

Figure 16. Effect of using rewards during training of BFMs: Comparing online BFMs with and without using available rewards during training.

Figure 17. Reward Regression and zz-sampling. Sampling reward-estimated zz values and fitting known reward labels (using an auxiliary loss with weight λ\lambda) fits better linear reward estimates (center), focuses learning on more similar latent tasks (right), and marginally improves early-episode adaptation in the ColorGrid domain (left).

MetaWorld

Figure 18 evaluates MetaBFM on six single-task ML1 problems, where train and test tasks share one manipulation skill and differ only in hidden goal positions.

Figure 18. Online Learning in MetaWorld ML1. MetaBFM evaluated in the three main inference configurations on six MetaWorld ML1 problems.

Figure 19 studies pure ZSRL inference (α=0\alpha= 0, β=0\beta= 0) on ML45 as a function of the Thompson-sampling noise σ\sigma (Appendix § 9.§ 3). On the 45 train tasks, small σ\sigma commits to a task estimate early and plateaus, while σ∈[5,10]\sigma\in[5, 10] trades a worse second episode for steady improvement through 25 episodes, well beyond the k=2k = 2 training horizon. On the five held-out tasks, success stays near zero at every σ\sigma.

References

  1. [1]Jacob Beck, Risto Vuorio, Evan Zheran Liu, Zheng Xiong, Luisa Zintgraf, Chelsea Finn, and Shimon Whiteson. A survey of meta-reinforcement learning. arXiv preprint arXiv:2301.08028, 2023.
  2. [2]Adaptive Agent Team, Jakob Bauer, Kate Baumli, Satinder Baveja, Feryal Behbahani, Avishkar Bhoopchand, Nathalie Bradley-Schmieg, Michael Chang, Natalie Clay, Adrian Collister, et al. Human-timescale adaptation in an open-ended task space. arXiv preprint arXiv:2301.07608, 2023.arxiv.org/abs/2301.07608
  3. [3]Serena Booth, W Bradley Knox, Julie Shah, Scott Niekum, Peter Stone, and Alessandro Allievi. The perils of trial-and-error reward design: misdesign through overfitting and invalid task specifications. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 5920–5929, 2023.
  4. [4]W Bradley Knox, Alessandro Allievi, Holger Banzhaf, Felix Schmitt, and Peter Stone. Reward (mis) design for autonomous driving. Artificial Intelligence, 316:103829, 2023.arxiv.org/abs/2104.13906
  5. [5]Jacek Karwowski, Oliver Hayman, Xingjian Bai, Klaus Kiendlhofer, Charlie Griffin, and Joar Max Viktor Skalse. Goodhart’s law in reinforcement learning. In The Twelfth International Conference on Learning Representations, 2023.arxiv.org/abs/2310.09144
  6. [6]Viktor Makoviychuk, Lukasz Wawrzyniak, Yunlong Guo, Michelle Lu, Kier Storey, Miles Macklin, David Hoeller, Nikita Rudin, Arthur Allshire, Ankur Handa, et al. Isaac gym: High performance gpu-based physics simulation for robot learning. arXiv preprint arXiv:2108.10470, 2021.arxiv.org/abs/2108.10470
  7. [7]Mayank Mittal, Pascal Roth, James Tigue, Antoine Richard, Octi Zhang, Peter Du, Antonio Serrano-Munoz, Xinjie Yao, René Zurbrügg, Nikita Rudin, et al. Isaac lab: A gpu-accelerated simulation framework for multi-modal robot learning. arXiv preprint arXiv:2511.04831, 2025.arxiv.org/abs/2511.04831
  8. [8]Seth Karten, Rahul Dev Appapogu, and Chi Jin. Automatic generation of high-performance rl environments. arXiv preprint arXiv:2603.12145, 2026.arxiv.org/abs/2603.12145
  9. [9]Jake Bruce, Michael D Dennis, Ashley Edwards, Jack Parker-Holder, Yuge Shi, Edward Hughes, Matthew Lai, Aditi Mavlankar, Richie Steigerwald, Chris Apps, et al. Genie: Generative interactive environments. In Forty-first International Conference on Machine Learning, 2024.arxiv.org/abs/2402.15391
  10. [10]Haoxuan Che, Xuanhua He, Quande Liu, Cheng Jin, and Hao Chen. Gamengen-x: Interactive open-world game video generation. arXiv preprint arXiv:2411.00769, 2024.arxiv.org/abs/2411.00769
  11. [11]Michael Laskin, Denis Yarats, Hao Liu, Kimin Lee, Albert Zhan, Kevin Lu, Catherine Cang, Lerrel Pinto, and Pieter Abbeel. URLB: Unsupervised reinforcement learning benchmark. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021. URL https://openreview.net/forum?id=lwrPkQP_is.arxiv.org/abs/2110.15191
  12. [12]Siddhant Agarwal, Caleb Chuck, Harshit Sikchi, Jia heng Hu, Max Rudolph, Scott Niekum, Peter Stone, and Amy Zhang. A unified framework for unsupervised reinforcement learning algorithms. In Workshop on Reinforcement Learning Beyond Rewards @ Reinforcement Learning Conference 2025, 2025. URL https://openreview.net/forum?id=tQ0t75p5HA.
  13. [13]Ce Zhou, Qian Li, Chen Li, Jun Yu, Yixin Liu, Guangjing Wang, Kai Zhang, Cheng Ji, Qiben Yan, Lifang He, et al. A comprehensive survey on pretrained foundation models: A history from bert to chatgpt. International Journal of Machine Learning and Cybernetics, 16(12):9851–9915, 2025.arxiv.org/abs/2302.09419
  14. [14]Ahmed Touati, Jérémy Rapin, and Yann Ollivier. Does zero-shot reinforcement learning exist? arXiv preprint arXiv:2209.14935, 2022.arxiv.org/abs/2209.14935
  15. [15]Andrea Tirinzoni, Ahmed Touati, Jesse Farebrother, Mateusz Guzek, Anssi Kanervisto, Yingchen Xu, Alessandro Lazaric, and Matteo Pirotta. Zero-shot whole-body humanoid control via behavioral foundation models. In The Thirteenth International Conference on Learning Representations, 2025.arxiv.org/abs/2504.11054
  16. [16]Yitang Li, Zhengyi Luo, Tonghe Zhang, Cunxi Dai, Anssi Kanervisto, Andrea Tirinzoni, Haoyang Weng, Kris Kitani, Mateusz Guzek, Ahmed Touati, et al. Bfm-zero: A promptable behavioral foundation model for humanoid control using unsupervised reinforcement learning. arXiv preprint arXiv:2511.04131, 2025.arxiv.org/abs/2511.04131
  17. [17]Thomas Rupf, Marco Bagatella, Marin Vlastelica, and Andreas Krause. Optimistic task inference for behavior foundation models. arXiv preprint arXiv:2510.20264, 2025.arxiv.org/abs/2510.20264
  18. [18]Luisa Zintgraf. Fast adaptation via meta reinforcement learning. PhD thesis, University of Oxford, 2022.DOI
  19. [19]Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International conference on machine learning, pages 1126–1135. PMLR, 2017.arxiv.org/abs/1703.03400
  20. [20]Yan Duan, John Schulman, Xi Chen, Peter L Bartlett, Ilya Sutskever, and Pieter Abbeel. R1²: Fast reinforcement learning via slow reinforcement learning. arXiv preprint arXiv:1611.02779, 2016.
  21. [21]Evan Z Liu, Aditi Raghunathan, Percy Liang, and Chelsea Finn. Decoupling exploration and exploitation for meta-reinforcement learning without sacrifices. In *International conference on machine learning*, pages 6925–6935. PMLR, 2021.arxiv.org/abs/2008.02790
  22. [22]Bradly C Stadie, Ge Yang, Rein Houthooft, Xi Chen, Yan Duan, Yuhuai Wu, Pieter Abbeel, and Ilya Sutskever. Some considerations on learning to explore via meta-reinforcement learning. *arXiv preprint arXiv:1803.01118*, 2018.arxiv.org/abs/1803.01118
  23. [23]Jan Humplik, Alexandre Galashov, Leonard Hasenclever, Pedro A Ortega, Yee Whye Teh, and Nicolas Heess. Meta reinforcement learning as task inference. *arXiv preprint arXiv:1905.06424*, 2019.arxiv.org/abs/1905.06424
  24. [24]Luisa Zintgraf, Sebastian Schulze, Cong Lu, Leo Feng, Maximilian Igl, Kyriacos Shiarlis, Yarin Gal, Katja Hofmann, and Shimon Whiteson. Varibad: Variational bayes-adaptive deep rl via meta-learning. *Journal of Machine Learning Research*, 22(289):1–39, 2021.
  25. [25]Jane X Wang, Zeb Kurth-Nelson, Dhruva Tirumala, Hubert Soyer, Joel Z Leibo, Remi Munos, Charles Blundell, Dharshan Kumaran, and Matt Botvinick. Learning to reinforcement learn. *arXiv preprint arXiv:1611.05763*, 2016.arxiv.org/abs/1611.05763
  26. [26]Alexander Nikulin, Vladislav Kurenkov, Ilya Zisman, Artem Agarkov, Viacheslav Sinii, and Sergey Kolesnikov. Xland-minigrid: Scalable meta-reinforcement learning environments in jax. *Advances in Neural Information Processing Systems*, 37:43809–43835, 2024.arxiv.org/abs/2312.12044
  27. [27]Chris Lu, Yannick Schroecker, Albert Gu, Emilio Parisotto, Jakob Foerster, Satinder Singh, and Feryal Behbahani. Structured state space models for in-context reinforcement learning. *arXiv preprint arXiv:2303.03982*, 2023.arxiv.org/abs/2303.03982
  28. [28]Vitchyr H Pong, Ashvin V Nair, Laura M Smith, Catherine Huang, and Sergey Levine. Offline meta-reinforcement learning with online self-supervision. In *International Conference on Machine Learning*, pages 17811–17829. Pmlr, 2022.arxiv.org/abs/2107.03974
  29. [29]Ron Dorfman, Idan Shenfeld, and Aviv Tamar. Offline meta learning of exploration. *arXiv preprint arXiv:2008.02598*, 2020.arxiv.org/abs/2008.02598
  30. [30]Yunkai Gao, Rui Zhang, Jiaming Guo, Fan Wu, Qi Yi, Shaohui Peng, Siming Lan, Ruizhi Chen, Zidong Du, Xing Hu, et al. Context shift reduction for offline meta-reinforcement learning. *Advances in Neural Information Processing Systems*, 36:80024–80043, 2023.arxiv.org/abs/2311.03695
  31. [31]Lanqing Li, Hai Zhang, Xinyu Zhang, Shatong Zhu, Yang Yu, Junqiao Zhao, and Pheng-Ann Heng. Towards an information theoretic framework of context-based offline meta-reinforcement learning. *Advances in Neural Information Processing Systems*, 37:75642–75667, 2024.arxiv.org/abs/2402.02429
  32. [32]Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. *arXiv preprint arXiv:2005.01643*, 2020.arxiv.org/abs/2005.01643
  33. [33]Abhishek Gupta, Benjamin Eysenbach, Chelsea Finn, and Sergey Levine. Unsupervised meta-learning for reinforcement learning. *arXiv preprint arXiv:1806.04640*, 2018.arxiv.org/abs/1806.04640
  34. [34]Allan Jabri, Kyle Hsu, Abhishek Gupta, Ben Eysenbach, Sergey Levine, and Chelsea Finn. Unsupervised curricula for visual meta-reinforcement learning. *Advances in Neural Information Processing Systems*, 32, 2019.arxiv.org/abs/1912.04226
  35. [35]Octavio Papalardo. Unsupervised learning of efficient exploration: Pre-training adaptive policies via self-imposed goals. *arXiv preprint arXiv:2601.19810*, 2026.
  36. [36]Jacopo Di Ventura, Jan Felix Kleuker, Aske Plaat, and Thomas Moerland. A unified framework for zero-shot reinforcement learning. *arXiv preprint arXiv:2510.20542*, 2025.arxiv.org/abs/2510.20542
  37. [37]Scott Jeen, Tom Bewley, and Jonathan M Cullen. Zero-shot reinforcement learning under partial observability. *arXiv preprint arXiv:2506.15446*, 2025.arxiv.org/abs/2506.15446
  38. [38]Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. *arXiv preprint arXiv:1509.02971*, 2015.arxiv.org/abs/1509.02971
  39. [39]Ahmed Touati and Yann Ollivier. Learning one representation to optimize all rewards. Advances in Neural Information Processing Systems, 34:13–23, 2021.arxiv.org/abs/2103.07945
  40. [40]Raymond Chua, Arna Ghosh, Christos Kaplanis, Blake A Richards, and Doina Precup. Learning successor features the simple way. Advances in Neural Information Processing Systems, 37: 49957–50030, 2024.arxiv.org/abs/2410.22133
  41. [41]Marco Bagatella, Thomas Rupf, Georg Martius, and Andreas Krause. Soft forward-backward representations for zero-shot reinforcement learning with general utilities. arXiv preprint arXiv:2602.06769, 2026.
  42. [42]Chongyi Zheng, Royina Karegoudra Jayanth, and Benjamin Eysenbach. Can we really learn one representation to optimize all rewards? arXiv preprint arXiv:2602.11399, 2026.
  43. [43]Scott Jeen, Tom Bewley, and Jonathan M Cullen. Zero-shot reinforcement learning from low quality data. Advances in Neural Information Processing Systems, 37:16894–16942, 2024.
  44. [44]Siddhant Agarwal, Harshit Sikchi, Peter Stone, and Amy Zhang. Proto successor measure: Representing the behavior space of an rl agent. In International Conference on Machine Learning, pages 566–586. PMLR, 2025.arxiv.org/abs/2411.19418
  45. [45]Seohong Park, Tobias Kreiman, and Sergey Levine. Foundation policies with hilbert representations. arXiv preprint arXiv:2402.15567, 2024.arxiv.org/abs/2402.15567
  46. [46]Marco Bagatella, Matteo Pirotta, Ahmed Touati, Alessandro Lazaric, and Andrea Tirinzoni. Td-jepa: Latent-predictive representations for zero-shot reinforcement learning. arXiv preprint arXiv:2510.00739, 2025.
  47. [47]Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. arXiv preprint arXiv:1810.12894, 2018.arxiv.org/abs/1810.12894
  48. [48]Kate Rakelly, Aurick Zhou, Chelsea Finn, Sergey Levine, and Deirdre Quillen. Efficient off-policy meta-reinforcement learning via probabilistic context variables. In International conference on machine learning, pages 5331–5340. PMLR, 2019.
  49. [49]Maksim Bobrin, Ilya Zisman, Alexander Nikulin, Vladislav Kurenkov, and Dmitry Dylov. Zero-shot adaptation of behavioral foundation models to unseen dynamics. arXiv preprint arXiv:2505.13150, 2025.
  50. [50]Dibya Ghosh, Abhishek Gupta, Ashwin Reddy, Justin Fu, Coline Devin, Benjamin Eysenbach, and Sergey Levine. Learning to reach goals via iterated supervised learning. arXiv preprint arXiv:1912.06088, 2019.
  51. [51]Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748, 2018.arxiv.org/abs/1807.03748
  52. [52]Max Schwarzer, Ankesh Anand, Rishab Goel, R Devon Hjelm, Aaron Courville, and Philip Bachman. Data-efficient reinforcement learning with self-predictive representations. arXiv preprint arXiv:2007.05929, 2020.
  53. [53]Max Schwarzer, Johan Samir Obando Ceron, Aaron Courville, Marc G Bellemare, Rishabh Agarwal, and Pablo Samuel Castro. Bigger, better, faster: Human-level atari with human-level efficiency. In International Conference on Machine Learning, pages 30365–30380. PMLR, 2023.arxiv.org/abs/2305.19452
  54. [54]Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Ghesh laghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems, 33:21271–21284, 2020.
  55. [55]Pranaya Jajoo, Harshit Sikchi, Siddhant Agarwal, Amy Zhang, Scott Niekum, and Martha White. Regularized latent dynamics prediction is a strong baseline for behavioral foundation models. In The Fourteenth International Conference on Learning Representations, 2026. URL https://openreview.net/forum?id=jdL6WB5JHZ.
  56. [56]Félix G. Harvey, Mike Yurick, Derek Nowrouzezahrai, and Christopher Pal. Robust motion in-betweening. ACM Transactions on Graphics (Proceedings of ACM SIGGRAPH), 39(4), 2020.DOI
  57. [57]Younggyo Seo, Li Li Chen, Jinwoo Shin, Honglak Lee, Pieter Abbeel, and Kimin Lee. State en- tropy maximization with random encoders for efficient exploration. In International conference on machine learning, pages 9443–9454. PMLR, 2021.arxiv.org/abs/2102.09430
  58. [58]Roberta Raileanu and Tim Rocktäschel. Ride: Rewarding impact-driven exploration for procedurally-generated environments. In International Conference on Learning Representations, 2020. URL https://openreview.net/forum?id=rkg-TJBFPB.arxiv.org/abs/2002.12292
  59. [59]Jingbo Sun, Songjun Tu, qichao Zhang, Haoran Li, Xin Liu, Yaran Chen, Ke Chen, and Dongbin Zhao. Unsupervised zero-shot reinforcement learning via dual-value forward-backward representation. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=OQnKnt4110.
  60. [60]Mikael Henaff, Minqi Jiang, and Roberta Raileanu. A study of global and episodic bonuses for exploration in contextual mdps. In International Conference on Machine Learning, pages 12972–12999. PMLR, 2023.arxiv.org/abs/2306.03236
  61. [61]Adrià Puigdomènech Badia, Pablo Sprechmann, Alex V. Vitsvitskyi, Daniel Guo, Bilal Piot, Steven Kapturowski, Olivier Tieleman, Martín Arjovsky, Alexander Pritzel, Andrew Bolt, et al. Never give up: Learning directed exploration strategies. arXiv preprint arXiv:2002.06038, 2020.arxiv.org/abs/2002.06038
  62. [62]Mikael Henaff, Roberta Raileanu, Minqi Jiang, and Tim Rocktäschel. Exploration via elliptical episodic bonuses. Advances in Neural Information Processing Systems, 35:37631–37646, 2022.arxiv.org/abs/2210.05805
  63. [63]Luisa M Zintgraf, Leo Feng, Cong Lu, Maximilian Igl, Kristian Hartikainen, Katja Hofmann, and Shimon Whiteson. Exploration in approximate hyper-state space for meta reinforcement learning. In International Conference on Machine Learning, pages 12991–13001. PMLR, 2021.
  64. [64]Jin Zhang, Jianhao Wang, Hao Hu, Tong Chen, Yingfeng Chen, Changjie Fan, and Chongjie Zhang. Metacure: Meta reinforcement learning with empowerment-driven exploration. In International Conference on Machine Learning, pages 12600–12610. PMLR, 2021.arxiv.org/abs/2006.08170
  65. [65]Jake Grigsby, Justin Sasek, Samyak Parajuli, Daniel Adebi, Amy Zhang, and Yuke Zhu. Amago- 2: Breaking the multi-task barrier in meta-reinforcement learning with transformers. Advances in Neural Information Processing Systems, 37:87473–87508, 2024.arxiv.org/abs/2411.11188
  66. [66]Tianwei Ni, Benjamin Eysenbach, and Ruslan Salakhutdinov. Recurrent model-free rl can be a strong baseline for many pomdps, 2022.arxiv.org/abs/2110.05038
  67. [67]Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint arXiv:1801.00690, 2018.arxiv.org/abs/1801.00690
  68. [68]Jonas Rothfuss, Dennis Lee, Ignasi Clavera, Tamim Asfour, and Pieter Abbeel. Promp: Proximal meta-policy search. arXiv preprint arXiv:1810.06784, 2018.arxiv.org/abs/1810.06784
  69. [69]Tianhe Yu, Deirdre Quillen, Zhangpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on robot learning, pages 1094–1100. PMLR, 2020.arxiv.org/abs/1910.10897
  70. [70]Viraj Joshi, Zifan Xu, Bo Liu, Peter Stone, and Amy Zhang. Benchmarking massively paral- lelized multi-task reinforcement learning for robotics tasks. arXiv preprint arXiv:2507.23172, 2025.arxiv.org/abs/2507.23172
  71. [71]Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014.
  72. [72]Jake Grigsby and Yanjun Qi. Measuring visual generalization in continuous control from pixels. arXiv preprint arXiv:2010.06740, 2020.arxiv.org/abs/2010.06740
  73. [73]Reginald McLean, Evangelos Chatzaroulas, Luc McCutcheon, Frank Röder, Tianhe Yu, Zhanpeng He, KR Zentner, Ryan Julian, Jordan K Terry, Isaac Woungang, et al. Meta-world+: An improved, standardized, rl benchmark. arXiv preprint arXiv:2505.11289, 2025.arxiv.org/abs/2505.11289
  74. [74]Zhengyi Luo, Jinkun Cao, Kris Kitani, Weipeng Xu, et al. Perpetual humanoid control for real-time simulated avatars. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10895–10904, 2023.arxiv.org/abs/2305.06456
  75. [75]Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063, 2024.arxiv.org/abs/2104.09864
  76. [76]Tri Dao, Daniel Y Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness. arXiv preprint arXiv:2205.14135, 2022.arxiv.org/abs/2205.14135
  77. [77]Danijar Hafner, Jur̄gis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models. arXiv preprint arXiv:2301.04104, 2023.arxiv.org/abs/2301.04104
  78. [78]Denis Yarats, David Brandfonbrener, Hao Liu, Michael Laskin, Pieter Abbeel, Alessandro Lazaric, and Lerrel Pinto. Don’t change the algorithm, change the data: Exploratory data for offline reinforcement learning. arXiv preprint arXiv:2201.13425, 2022.arxiv.org/abs/2201.13425

Paper details

vlnc:W_jecxzgvy
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents