In this paper
AbstractIntroductionBackgroundMeta-Reinforcement LearningBehavior Foundation ModelsGeneralization of BFMs and Supervised Meta-RLBFMs as Meta-RL AgentsAdapting to Novel DynamicsExploring, Exploiting, and Learning OnlineLearning from Available RewardsMetaBFMConclusionAcknowledgments and Disclosure of FundingAdditional BackgroundReinforcement LearningBehavior Foundation ModelsEnvironment DetailsColorGridWindDMC WalkerMetaWorldHumanoid LocomotionImplementation DetailsLatent Context LearningBFMOnline Task InferenceIntrinsic RewardsActor–Critic UpdateComputational Resources and Statistical SignificanceLimitationsAdditional ResultsOnline ZSRLUtilizing Available RewardsMetaWorldReferencesWorksComputer Science
Meta-Reinforcement Learning with Zero-Shot Reinforcement Learning
Abstract
Meta-reinforcement learning (meta-RL) trains agents to adapt to novel tasks from test-time experience, but requires diverse training environments and reward functions. While environments are becoming cheaper to generate, it remains difficult to anticipate every objective an agent may be asked to pursue at test time. Behavior Foundation Models (BFMs) learn policies from reward-free data and support zero-shot RL (ZSRL): adaptation to new objectives at test time using a large reward-labeled dataset. Recent work has shown that BFM task inference can instead be performed online without prior task-specific data. We note that this capability positions BFMs and meta-RL as alternative approaches for online adaptation to new reward functions in a fixed environment. We then ask whether we can extend BFMs to compete with meta-RL more broadly, including in novel environments, and identify four challenges: environment identification, online data collection, test-time exploration, and learning from mixed reward supervision. We study each individually, then propose MetaBFM, a hybrid method that addresses all four. During training, it combines supervised reward-following with unsupervised learning; at test time, it can interpolate between exploration, exploitation, and ZSRL-style reward inference. We demonstrate that MetaBFM can recover the in-distribution performance of supervised meta-RL while generalizing to reward functions outside the training set. Our work suggests that BFMs and meta-RL are complementary research directions whose combined strengths allow adaptive agents to learn more diverse behaviors from a fixed set of supervised objectives.
Introduction
Meta-reinforcement learning agents improve on novel tasks by adapting their decision-making from test-time experience [1]. This capability relies on diverse training sets of environments and reward functions, and benefits from online data collection over many trials. Despite much algorithmic progress, meta-RL remains limited by the cost of generating these training sets. Most work focuses on toy problems where similar environments and/or reward functions are easily sampled; large-scale applications like AdA [2], where a team of developers repurposed a commercial game engine to generate complex tasks and curricula, are rare exceptions. Generating a diverse set of learnable reward functions that covers the desired range of behavior may be inherently difficult. Reward functions are challenging to design at scale [3, 4], and imperfect proxies can break down under optimization pressure [5]. However, with hardware-accelerated simulators [6, 7], LLM-assisted coding [8], and promptable world models [9, 10], the cost of generating environments is rapidly declining.
Behavior Foundation Models (BFMs) are an emerging class of unsupervised RL methods [11] that train in a reward-free environment by learning to maximize many possible objectives [12]. At test time, a target objective is specified by model inference on a large dataset of trajectories labeled with a particular reward function. BFMs’ unsupervised training and supervised testing datasets are loosely analogous to the pretraining and finetuning paradigm of foundation models in other areas [13]. This adaptation process is often called zero-shot RL (ZSRL) [14] and has shown promising results in domains like humanoid locomotion [15, 16]. Meta-RL and BFMs both enable generalization without additional training, but ZSRL requires a large supervised adaptation dataset in advance, while meta-RL assumes the policy actively collects its own (far smaller) dataset. This distinction makes the two approaches applicable to different types of adaptation problems, and their test-time performance is difficult to compare directly. However, recent work by Rupf et al. [17] demonstrated that ZSRL task inference can be done online, without a preexisting dataset, under the same conditions as meta-RL. As a result, meta-RL and BFMs can now be considered direct competitors on problems that require adaptation to novel reward functions in a fixed environment.
If BFMs can match a broader range of meta-RL’s capabilities, they could help extend generalist RL to domains where train-time reward supervision covers only a narrow range of behaviors. We argue that this requires BFMs to (1) identify and generalize to novel environments, (2) learn effectively without a pre-collected training dataset, (3) make exploration-exploitation tradeoffs at test time, and (4) benefit from available reward supervision. We study each of these subproblems in turn with detailed experiments on a diagnostic toy problem and simulated humanoid locomotion before arriving at a unified framework that addresses all four. The resulting method, MetaBFM, trains a policy that spans meta-RL, ZSRL, and intrinsic exploration. During training, MetaBFM combines supervised reward-following with unsupervised learning while collecting training data online. At test time, we are free to interpolate between exploration, exploitation, and ZSRL-style reward inference. This hybrid method draws on the strengths of meta-RL and ZSRL: it retains meta-RL’s ability to adapt within the training distribution while opening the possibility of generalizing to out-of-distribution reward functions. Our work suggests that BFMs and meta-RL are complementary research directions and that, together, they can shift the emphasis from specifying additional training objectives to designing diverse environments.
Background
Meta-Reinforcement Learning
A meta-RL policy can be evaluated by its cumulative return over the first attempts, or episodes, in a new task. This objective creates an exploration-exploitation tradeoff at test time: the policy may sacrifice performance in early attempts to learn more about the task if exploiting that knowledge later leads to higher cumulative return [18]. Methods differ in whether this tradeoff is managed by the designer as a hyperparameter [19] or learned by the policy itself [20]. Policies may also be given free episodes to explore without consequence before being evaluated on attempt ; this k-shot setting creates explicit exploration and exploitation phases [21, 22].
Meta-RL’s test-time adaptation is enabled by training on a diverse set of tasks. In this paper, a “task” refers specifically to a combination of an environment and an objective—or, equivalently, to a pair of parameters that alter the transition dynamics and reward function, respectively. The basic goal of a context-based meta-learner is to use the multi-episodic trajectory1 of interaction with the current task so far, , to infer the unknown , exploit the unique characteristics of the task, and maximize returns. Methods broadly differ in whether the parameters are estimated by supervised regression to ground-truth values [23], as the byproduct of a self-supervised objective such as dynamics modeling [24], or implicitly through the objective of maximizing returns [25]. In each case, the idea is to establish a training objective that is minimized by correctly distinguishing and then conditioning on the resulting representation.
Meta-RL is at its best when we can afford sample-inefficient online data collection [2, 26, 27], which may be acceptable when we are willing to trade train-time compute for efficient test-time adaptation. Meta-learning is harder when restricted to a fixed dataset [28, 29, 30, 31], as it suffers from an extreme form of the distribution shift problem that defines offline RL more broadly [32]. Methods also differ in whether task rewards are available during meta-training. Standard meta-RL is supervised by a distribution of reward-labeled tasks, whereas unsupervised meta-RL learns from reward-free interaction and only encounters rewards at test time [1]. We focus on online meta-RL that spans both regimes, meta-training on a set of reward functions while simultaneously learning a reward-free BFM that supports ZSRL-style task inference at test time. This hybrid framing leads to a method distinct from prior unsupervised meta-RL [33, 34, 35], which omits reward supervision entirely, and from prior ZSRL/BFM work, which does not meta-train across environments.
Behavior Foundation Models
BFMs are unsupervised RL agents based on successor features [12, 36]. The idea is to learn a latent-conditioned policy from a reward-free dataset and infer the that maximizes a particular task of interest from a reward-labeled dataset, without further training [37]. In general, we let define latent rewards and learn successor features , so that can serve as the critic in a standard off-policy actor-critic update [38] over some distribution of , e.g., . In this work, the details of training and will be interchangeable and are not necessary to follow the main text, so we defer a more complete introduction to Appendix § 7.§ 2. Most recent work is based on the FB algorithm [14, 39], but there are many alternatives and extensions [40, 41, 42, 43, 44, 45]. Our experiments primarily use TD-JEPA [46] because it shares similarities with off-policy supervised RL and lets us transfer implementation details from an optimized meta-RL baseline when appropriate.
BFMs benefit from diverse training datasets with broad state-action coverage, and are most commonly applied to offline datasets originally collected by a policy that optimized for exploration [11, 47]. They adapt to a test-time task by fitting on a reward-labeled dataset to deploy , and we will refer to this inference process as zero-shot RL (ZSRL). Meta-RL’s inference of plays a similar role as ZSRL’s inference of : both parameters estimate the active reward function and therefore enable generalization to novel objectives. ZSRL relies on an existing offline dataset to fit , after which this parameter is fixed. In contrast, meta-RL typically infers from (an online it has collected itself) by updating a belief between timesteps or episodes [48]. This discrepancy makes ZSRL and meta-RL difficult to compare directly. In fact, “zero-shot RL” is, by design, unsuited to zero-shot generalization as typically defined (§ 2.1), where we expect a policy to perform well in a novel task on the first attempt with no prior data. OptiBFM [17] helps to close this gap by demonstrating that, as in meta-RL, we can maintain a belief over that is updated between timesteps or episodes.
Generalization of BFMs and Supervised Meta-RL
With OptiBFM’s online task inference enabled, BFMs and meta-RL can be directly compared. We will use a didactic ColorGrid problem as a running example through § 3. ColorGrid creates an continuous 2D gridworld where the agent receives a reward based on the unit square cell of its current location, and the objective is to navigate to the cell with the highest reward and remain there. The underlying cell layout can be randomly permuted such that any two cells may be adjacent. In short, we can generate distinct environments where nearly every transition is unpredictable without a reasonable estimate of the active grid permutation, . Randomly sampling a reward for each cell generates infinitely many values. Additional environment details are in Appendix § 8.
As a first experiment, we split and into train/test sets and generate tasks from their cross product, measuring environment and reward generalization separately. To replicate BFMs’ high-coverage offline setting, we collect large random-policy datasets. Our BFM baseline is the better of FB and TD-JEPA, while our meta-RL baseline is an off-policy variant of RL2 [20] that makes offline training reasonable, if still slightly unfair—we return to this later. Figure 1 reports results for . Meta-RL follows a predictable trend: adding environments or reward functions improves generalization, but closing the train/test gap requires a diverse mixture of both. In this sense, supervised meta-RL is designed for the bottom-right corner of Figure 1. OptiBFM allows BFMs to match the standard inference procedure, though adaptation barely saturates within the test episodes. In contrast, meta-RL has trained on reward functions and is incentivized to learn a narrow prior over . For now, we are not concerned with the performance gap between BFMs and meta-RL where BFMs show meaningful adaptation. Instead, we focus on BFMs’ relative performance: the train curves in the left column of Figure 1 represent the established BFM setting, and performance holds as evaluated reward functions grow in a fixed environment, even though the reward labels go unused and therefore act as a test set themselves. However, BFMs immediately collapse with multiple training environments, a problem also identified by Bobrin et al. [49].
Figure 1. BFM and Meta-RL Generalization. In offline ColorGrid (), we vary training environments (columns) and rewards (rows), then evaluate episode adaptation on train and held-out tasks. Meta-RL improves with training set diversity but needs many reward functions to generalize; BFMs adapt in the fixed-environment setting (left column), but fail across environments.
BFMs as Meta-RL Agents
BFMs can adapt to unseen reward functions over attempts without train-time rewards. However, to be useful in meta-RL, they must also generalize across environments. This is only one of several gaps: meta-RL benefits from online training, while BFMs default to offline datasets; meta-RL manages exploration and exploitation over test episodes, while BFMs do not; and complete reward-free learning is often unnecessary in simulated online RL domains. We can usually define some reward-labeled tasks, even if we cannot cover every behavior needed at deployment. We investigate these gaps before proposing a solution that enables a competitive hybrid between meta-RL and BFMs.
Adapting to Novel Dynamics
An RL policy that cannot condition on estimates of the current dynamics, , and reward function, , learns an average policy under that is generally not optimal for any specific task [50]. BFMs share this problem, but already plays the role of , leaving us to infer . Following context-based meta-RL [1], we augment each trajectory with a learned estimate and train . Because context labels are assumed to be unavailable, we use a self-supervised dynamics objective. Unlike BeliefFB [49], we target timestep-level online task inference with latent dynamics representations that avoid method-specific changes to the BFM update.
Figure 2 shows our Latent Context Learning (LCL) architecture, which produces at every timestep during rollouts. We embed each timestep of the trajectory to create the input sequence , then pass it through a causal sequence model. Let be the output representation at timestep . This output is trained to identify the active dynamics through self-supervised inverse dynamics modeling (IDM) and latent dynamics modeling (LDM). Because accurate estimates of improve these objectives, and because is the only representation with access to the necessary history, we treat and can train a standard BFM on augmented states . observes action choices to infer dynamics, but we avoid leaking the inverse dynamics model label by staggering the action input by two timesteps. We prefer latent dynamics modeling because forward dynamics modeling (FDM) scales with the size of the state space. Following latent world modeling and self-supervised sequence models [51, 52, 53], we use an EMA target network for labels [54], and can optionally encourage to be dissimilar across sequences in each training batch [55]. Appendix § 9.§ 1 and Algorithm 1 provide additional details. By default, we train IDM and LDM jointly; detached FDM prediction error is used only as a diagnostic metric.
Figure 2. Latent Context Learning. (Left) We learn a timestep embedding and sequence model that generates an online estimate of the environment parameters, . The representation is trained with self-supervised inverse and latent dynamics objectives; Algorithm 1 gives the full procedure. Action inputs are necessary context to infer dynamics, but are time-shifted to avoid leaking the IDM label. (Right) Context features learned by a feedforward (FF) or Transformer (Seq.) are presented to either a feedforward or sequence-based supervised RL policy in ColorGrid.
Context estimates should improve supervised RL by reducing or eliminating the need for long-term memory policies, and Figure 2 (Right) verifies this effect in ColorGrid. By augmenting states with context estimates, we can also make BFM networks context-aware without the instability of sequence models. Figure 3 shows that, when the context representation has enough capacity to represent , joint IDM and LDM learns accurate dynamics estimates that improve TD-JEPA performance. In Figure 4, we define train/test splits of task pairs with separate offline datasets, and evaluate generalization as the number of training tasks grows. Since LCL and the downstream BFM agent are independent, they can be trained separately. Pretraining and freezing LCL on the largest task set improves BFM generalization at small training set sizes, but gives no clear advantage over training both systems from scratch on the same offline dataset. This is encouraging because joint training is the only realistic option when the dataset is collected online.

Figure 3. LCL and Generalization. TD-JEPA in ColorGrid with various context embedding sizes and training objectives. Performance correlates with lower FDM and IDM error. LDM representations recover similar dynamics prediction as FDM without training on this objective.

Figure 4. Latent context learning improves BFM generalization. We increase the number of training task pairs with separate offline datasets. Standard TD-JEPA collapses across environments, while LCL-augmented TD-JEPA recovers adaptation and improves test performance as the training set grows. Pretraining LCL is most helpful at small training set sizes, but joint training approaches the same performance at scale.
To further illustrate BFMs’ failure across environments and test whether context learning addresses it, we turn to motion tracking for the G1 humanoid, building on BFM-Zero [16]. We randomize the dynamics through wind and gravity perturbations of varying strength, and evaluate tracking performance on the LAFAN dataset [56], which contains 4.6 hours of motion sequences. We compare a BFM trained without these perturbations (BFM-Single) to one trained across the full range of dynamics (BFM-Multi), with and without the FDM and LDM context-learning objectives. Consistent with our analysis above, a BFM trained across multiple dynamics averages its actions over them, which can degrade performance: in the in-distribution setting, BFM-Multi performs even worse than BFM-Single, suggesting that BFMs suffer substantially when they cannot infer the environment’s dynamics. Context learning recovers much of this gap (Table 1), suggesting its promise for training across domains. Surprisingly, it also improves performance modestly in the out-of-distribution setting. Appendix § 8.§ 5 illustrates these BFM failures and provides additional details on the task and environment.
| Method | In-Distribution | Out-of-Distribution | ||||||
| mpjpe-l() | vel. dist.() | accel. dist.() | emd() | mpjpe-l() | vel. dist.() | accel. dist.() | emd() | |
| BFM-Single | 117.2 | 1.64 | 2.75 | 1.01 | 194.7 | 2.81 | 4.83 | 1.80 |
| BFM-Multi | 132.7 | 1.77 | 2.96 | 1.13 | 151.6 | 2.13 | 3.63 | 1.39 |
| FDM+BFM-Multi | 125.3 | 1.73 | 2.90 | 1.09 | 149.3 | 2.09 | 3.54 | 1.37 |
| LDM+BFM-Multi | 121.5 | 1.68 | 2.85 | 1.05 | 145.9 | 2.06 | 3.52 | 1.36 |
Table 1. BFMs with latent context learning improve humanoid motion tracking. We compare motion-tracking metrics for humanoid locomotion under in-distribution and out-of-distribution dynamics. BFM-Single trains without wind/gravity randomization and can be viewed as an in-distribution oracle for BFM-Multi, or as a worst-case baseline under OOD dynamics.
Exploring, Exploiting, and Learning Online
BFM algorithms are primarily developed on offline RL benchmarks [11] with behavior datasets generated by an online policy optimizing an intrinsic reward [47, 57, 58]. BFMs rely on high state-action coverage and struggle to learn online because the actor objective collapses onto narrow behavior. Appendix § 11.§ 1 explores the difficulty of online learning and task inference in a standard locomotion benchmark. DVFB [59] addresses dataset diversity by maximizing a combination of and a supervised critic trained on intrinsic curiosity rewards, which we will call (Equation (1)). Recent applications of BFMs to humanoid robotics [15, 16] use a similar idea, but derive rewards from behavior in an expert motion-capture dataset.
In meta-RL problems, we can take this idea further. First, we replace the global intrinsic reward—one that decreases as a function of total training steps, such as RND [47]—with an episodic reward that decays within a -episode meta-trial [60]. An agent maximizing this reward is incentivized to explore new states across each episode at test time. We use an intrinsic reward that combines an off-the-shelf episodic bonus (NGU [61] or E3B [62]) on LCL timestep features (Figure 2) with the LCL prediction error from the previous timestep. The latter is a core meta-RL idea [63, 64]: it incentivizes exploration that improves the estimate, which can also improve in-context. Second, because episodic rewards are non-Markov, and should have memory and take LCL-augmented trajectories as input. Finally, we sample during training and condition on the active value:
The result is a policy that generates more diverse data during training while allowing to vary at test time. Due to our choice of intrinsic reward, drives the policy to explore new behavior each episode in hopes of (1) finding rewards that improve OptiBFM’s online task inference of and (2) improving the context estimate . In contrast, exploits that knowledge to maximize returns under . We can schedule to recover meta-RL’s -shot exploration-exploitation tradeoff at test time, as demonstrated in Figure 5.
Figure 5. Exploring at Test Time. ColorGrid () policies trained by Eq. (2) can vary over test episodes. High explores more of the state space, improving online task estimates, which can then be exploited to outperform the standard BFM policy (, blue) when rewards are sparse.
Learning from Available Rewards
BFMs assume access to zero reward labels during training. While this may be necessary when learning offline, such as from a web dataset that was not intended for RL, the constraint is unnecessary when learning online in simulated environments. When we create meta-RL or multi-task RL training domains, we can usually create some reward-labeled tasks; the challenge is creating enough to cover all behaviors we may want during deployment. Adding training tasks should improve generalization with diminishing returns, creating a clear path to improving the agent by improving its dataset. Ideally, we would bring this relationship to BFMs. Each new train-time objective can help span the space of possible behaviors and supervise learning progress, while unsupervised learning fills in gaps and finds solutions to objectives we had not considered.
We can approach this problem from two directions. The first is to let reward labels improve the BFM update itself. For example, BFMs learn features and assume ; when reward labels are available during training, we can enforce this relationship directly with an auxiliary loss. It has also become standard to sample training from recent values of , implicitly assuming evaluation on sparse state-based goal-reaching [14, 39]. With reward labels, we can extend this idea to arbitrary reward functions by using online task inference along the labeled trajectory to estimate , as if the same states and rewards appeared at test time. Appendix § 11.§ 2 evaluates both ideas in ColorGrid and DMC ExoRL, finding that they make more predictive of rewards but have mixed effects on policy performance. A second approach is to restore supervised meta-RL where rewards are available and create a true hybrid method. We modify the “intrinsic” critic from § 3.2 to learn from a weighted combination of intrinsic rewards, , and extrinsic (supervised) rewards :
where and are the next timestep in the standard TD backup. We are then free to repeat the idea from Equation (2) of sampling the weight during training and conditioning the actor and critic on the current value so that it may also be adjusted at deployment:
When , we recover the exploratory ZSRL objective of Eq. (2), and when , we recover standard ZSRL. Most importantly, when , we recover supervised meta-RL. In practice, this makes it possible to compete with a standard supervised meta-RL baseline on in-distribution tasks, while opening the possibility of generalizing via to novel tasks when meta-RL fails. Figure 6 visualizes this space of possible test-time settings.
Figure 6. A Unified Objective. The loss (Eq. (4)) interpolates between zero-shot RL, intrinsic curiosity, and supervised meta-RL. and can be configured to adjust generalization behavior.
MetaBFM
We can now combine the pieces from § 3 into a unified update, MetaBFM, that captures the in-distribution performance of supervised meta-RL and the out-of-distribution generalization of unsupervised BFMs. At each training step, we first update the Latent Context Learning module with its self-contained objective (Algorithm 1). LCL’s primary role is to augment each state with a context estimate , allowing the BFM networks to learn from data generated by multiple environment dynamics (§ 3.1). LCL also produces per-timestep representations that can be passed to off-the-shelf episodic curiosity methods to compute intrinsic rewards for the trajectory. We then update the BFM networks, excluding the actor , over a distribution . Any actions from required for the BFM update are sampled from , which recovers the original BFM policy; the remaining details are unchanged, and many ZSRL algorithms are interchangeable here, though we use TD-JEPA. Next, we update the critic head with a standard off-policy backup on weighted intrinsic and extrinsic rewards (Equation (3)) and compute the actor loss (Equation (4)), both over samples from . We collect trajectories from the online environment by sampling from the same distribution of , , and .
While this update is more expensive than training the BFM component in isolation, in practice we share network backbones for efficiency. For example, and share a Transformer backbone to enable long-term memory. Many such training details are copied directly from a strong off-policy meta-RL implementation [65], and Appendix § 9 provides additional discussion. To make extrinsic rewards optional, we force when collecting or training on reward-free trajectories. Early layers that take extrinsic reward inputs to and have their outputs multiplied by , allowing us to insert and automatically ignore an arbitrary placeholder reward.
Figure 1 showed that BFMs can learn adaptive behavior from limited training domains where supervised meta-RL overfits, but also that meta-RL improves with diverse datasets while BFMs do not. MetaBFM’s main goal is to allow for test-time configuration to take advantage of both of these methods’ strengths while being able to avoid their weaknesses. Figure 7 Left returns to this experiment and shows that MetaBFM captures both modes: a single policy can be deployed in “ZSRL mode” () when meta-RL fails to generalize, or in “reward-following mode” () when the dataset is diverse enough to support it.
Figure 7. Configurable MetaBFM Inference. (Left) When learning from limited training objectives, MetaBFM in ZSRL mode () avoids supervised meta-RL’s overfitting. (Center Left) With more objectives, supervised mode recovers the meta-RL baseline. (Center Right) On held-out in-distribution Wind tasks, MetaBFM remains competitive with supervised meta-RL. (Right) On out-of-distribution Wind tasks, ZSRL-style inference adapts over long adaptation horizons.
An extension of learning more general behavior from fewer train-time objectives is generalizing to test-time objectives that are clearly out of distribution from the supervised signals seen during training. We study this with a variant of the Wind environment [29, 66], a common benchmark for OOD generalization. Agents navigate from the origin toward a randomly placed goal, , while adjusting for wind in a random direction, . The training set contains all wind directions but excludes goals in the upper-left quadrant. At test time, we therefore evaluate both held-out in-distribution tasks (any wind direction, goals in the three training quadrants) and out-of-distribution tasks (any wind direction, goals in the excluded quadrant). Figure 7 Right shows the adaptation curves for each set. MetaBFM significantly outperforms the supervised baseline on the OOD test set when configured to maximize successor-feature values, while nearly matching its performance on in-distribution tasks where train-time rewards are useful. Perhaps the most interesting OOD result is the gap between the train-time trajectory length of episodes and the length at which online task inference saturates (). This kind of length extrapolation is rare in black-box meta-RL and may be enabled by offloading the usual context estimation of outside of the policy weights and into ZSRL’s .
We repeat this experiment on the “walker” locomotion task in the DeepMind Control Suite [67], using a variant of the meta-RL setup from Rothfuss et al. [68]. We generate by randomizing body geometry, joint ranges, motor damping, and ground friction. Agents train over episodes in 10k random walker variations using only the standard walk reward, and are evaluated on held-out dynamics (Appendix § 8.§ 3). At test time, we evaluate on walk, two closely related objectives (stand and run), and three objectives requiring behaviors not encouraged during supervised training (walk backward, crouch, and crouch walk). Figure 8 shows the same pattern as Wind: on walk, MetaBFM in reward-following mode performs similarly to supervised meta-RL, while ZSRL mode is noticeably weaker. On the three novel objectives, supervised meta-RL fails to follow the test-time reward signal, while performance with ZSRL-style inference continues to improve well past the training horizon.
Figure 8. DMC Walker. Adaptation in held-out environments after training on 10k environments with only the walk reward (left). MetaBFM recovers supervised meta-RL on the train reward and its close variants, and generalizes to novel rewards in ZSRL mode. ZSRL evaluations run for 30 episodes; x-axes are truncated after performance plateaus.
Finally, we evaluate MetaBFM on the more complex MetaWorld continuous-control benchmark [69], using the hardware-accelerated variant from MTBench [70]. Despite learning to maximize a range of actor objectives at both training and test time, MetaBFM converges to near-perfect performance on six ML1 tasks, each testing generalization across hidden-parameter variations of a single tabletop manipulation skill (Appendix Figure 18). Figure 9 reports results for the full multi-task ML45 setting. To reduce computational cost, we initialize off-policy learning with a replay buffer of demonstrations collected from checkpoints of a multi-day meta-RL training run, with exploration noise enabled. We compare baselines trained offline on this dataset with meta-RL and MetaBFM agents that continue collecting online experience. On the training tasks, MetaBFM again performs comparably to supervised meta-RL (Figure 9, left). In exploration mode, MetaBFM continues to visit new regions of the workspace in each episode, while reward-following mode largely repeats its first attempt (Figure 9, center). However, MetaBFM fails to generalize to MetaWorld’s held-out manipulation tasks in either reward-following or ZSRL inference mode (Appendix Figure 19). To test adaptation to novel objectives that require in-distribution behaviors, we add a goal-reaching reward to all 50 training and held-out environments, with hidden targets drawn from states visited during exploration rollouts (Appendix § 8.§ 4). Performance with ZSRL-style inference continues to improve well beyond the training horizon, while success rates in reward-following and exploration modes remain near chance (Figure 9, right).
Figure 9. MetaWorld ML45. (Left) In reward-following mode, MetaBFM recovers supervised meta-RL performance on the 45 training tasks. Boxes show interquartile ranges; diamonds show mean success rates. Behavior Data and Best Policy report success rates for the seed dataset and its best contributing policy, respectively. (Center) In exploration mode, MetaBFM continues to expand the gripper’s coverage of the tabletop workspace over a trial. (Right) In ZSRL mode, MetaBFM adapts to a simple unseen goal-reaching reward across all 50 environments.
Conclusion
Meta-RL requires diverse training reward functions, but anticipating every test-time objective remains difficult even as environments become cheaper to generate. Building on OptiBFM’s online task inference, we extend BFMs to match a broader range of meta-RL’s capabilities, including adaptation to novel environments. MetaBFM combines a self-predictive latent context for environment identification, intrinsic curiosity for online data collection and test-time exploration, and supervised value learning to take advantage of mixed reward supervision. The resulting policy recovers the in-distribution performance of supervised meta-RL while generalizing to reward functions outside the training set. Continued progress in meta-RL and BFMs could be mutually reinforcing, advancing generalist RL by shifting the emphasis from specifying additional training objectives to generating more diverse environments.
Acknowledgments and Disclosure of Funding
This work was supported by the National Science Foundation (CAREER Award FRR-2145283 and EFRI-2318065), the Office of Naval Research (N00014-24-1-2550), the Army Research Lab (W911NF-25-1-0065), the KIST-UT collaboration (UTAUS-FA00004578), the JP Morgan Chase Faculty Fellowship in AI, and the Sony Research Awards.
Additional Background
Reinforcement Learning
A Markov Decision Process (MDP) [71] is defined as a tuple where is called the state space, is the action space, is the initial state distribution, is the transition function that maps a given state and action to a distribution of next states and is the scalar reward function. The agent executes a policy, , to interact with the environment. The Reinforcement Learning (RL) objective is to find the optimal policy, , defined as,
Behavior Foundation Models
A Behavior Foundation Model (BFM), for a given dynamical system, allows an agent to produce near-optimal policies for any given reward function using little or no additional computation. As a result, this paradigm has often been referred to as Zero-Shot Reinforcement Learning. Specifically, in this work, we consider successor-measure-based BFMs that learn to represent successor measures for a wide range of policies. Mathematically, the successor measure for a policy is defined as,
BFMs represent (successor measure for policy parameterized using ) as . Connections have been drawn [12, 39, 44] between this representation and successor features with being the successor feature for the state feature . Methods like [39, 44, 55] use the following objective to learn these representations,
where is the offline dataset. These BFMs consider tasks defined by rewards, for which the corresponding optimal Q function can be written as, with the optimal policy . Overall, BFMs can be represented using . All three networks , and are learned using reward-free interactions with and being trained using (7) and as the optimal policy for the reward .
At inference, when a reward function is provided, the BFMs simply need to find the for the given reward function as the corresponding policy is already trained to be optimal for . This can be inferred simply by linear regression on reward samples, i.e. . Several methods [39, 44, 45, 46, 55] have been proposed with a similar construction of BFMs.
Environment Details
ColorGrid
ColorGrid is an continuous 2D navigation task in which a hidden reward grid assigns a scalar payoff to each unit cell. The agent does not observe its true position; instead, it sees the cell label scrambled by a hidden permutation . A task is the pair , so the agent must resolve two independent unknowns within a trial: how its movements map to observations, and which cell pays the most.
Dynamics and reward. The true state is a continuous position . A 2D velocity action updates it as , and the reward is the value of the current true cell, . Movement and reward follow the true grid; only changes what the agent observes. The observation relabels the integer part of the position by and keeps the fractional part, so motion within a cell looks smooth, but crossing a cell boundary makes the observed position jump to an unrelated cell (Figure 10). To navigate, the agent must implicitly invert the permutation. Inner episodes last 60 steps for and 80 steps for . We report oracle-normalized return: per-episode return divided by that of an oracle that teleports from the random start cell to and stays there, so a score of 1.0 matches the oracle.

Figure 10. An example ColorGrid transition for .
Observations. The observation is three-dimensional: the scrambled 2D position described above, and a flag that is set for one step at each boundary between inner episodes of a trial. Neither nor is observed.
Task distribution. Permutations and reward grids come from two fixed, deduplicated pools, with reward entries sampled i.i.d. from and clipped to . Each pool is split into 10,000 training entries and 10,000 disjoint held-out entries. By default, the -th task pairs the -th entry of each pool, . In Figure 1, tasks instead come from the cross product of the two pools, so a policy can train on many reward functions in a single permutation, or vice versa.
Wind
Wind is a 2D point meta-RL navigation task introduced by Dorfman et al. [29] and adapted by Ni et al. [66] as part of their POMDP suite. The agent commands position deltas on the plane, and the world adds a hidden, task-specific wind vector to its motion at every step. A task is an unobserved pair in which perturbs the dynamics while defines the reward landscape. The two hidden variables stress meta-RL along complementary axes. The wind is identifiable only from the difference between commanded actions and observed motion, and the goal only from reward, so a competent policy must jointly infer the dynamics and search for an unmarked target.
Dynamics and reward. The agent’s state is a 2D position , advanced by an action and a per-task wind via . Inner episodes last 75 steps and terminate early on entry into a small disk around the goal . The reward is , bounded everywhere on the plane with a sharp bonus inside the disk.
Observations. The agent receives its own 2D position together with an episode-boundary flag that fires for one step at the boundary between inner episodes within a trial. Both and are hidden.
Task distribution. During training, we sample 80 winds uniformly from and 80 goal locations on the unit circle, with the circle partitioned into the four standard quadrants. One quadrant ( in our experiments) is held out from the training goal set, so the agent is never rewarded for entering the upper-left region of the plane during training. The two test environments measure distinct generalization gaps that we report separately.
DMC Walker
We use the dynamics-generalization walker from DMC Remastered [72]: the DeepMind Control walker [67] with its embodiment resampled from a dynamics seed. Observations are state-based, so visual randomization is disabled.
Dynamics and task distribution. Each dynamics seed defines a body by resampling torso and foot geometry, knee and ankle joint ranges, motor damping and armature, and ground friction. We draw 11,000 distinct seeds once. The first form the training pool, so smaller pools nest inside larger ones, and the final 1,000 form a disjoint held-out pool shared across all . § 4 trains on environments.
Observations and actions. The agent receives the 24-D proprioceptive walker state together with an episode-boundary flag that fires for one step at the boundary between inner episodes. The action is a 6-D vector of joint torques in , repeated for two control steps. Inner episodes last 250 agent steps (500 control steps). The body is hidden.
Rewards. Every reward lies in per control step, and we report episode returns rescaled to the standard 0–1000 DMC score. Training uses only the native walk reward (torso height above 1.2 m, upright, forward speed of at least 1 m/s). The native stand and run rewards drop the speed term or raise the target to 8 m/s. Crouch and crouch walk replace the standing height condition of stand and walk with a two-sided band of m and a margin of 0.2 m. Walk backward multiplies the standing height term by a velocity ramp that pays 0 when walking forward at 1 m/s, 0.5 at rest, and 1 when walking backward at 1 m/s.
MetaWorld
We base our experiments on the MetaWorld manipulation suite [69], with the reward and goal-masking corrections of McLean et al. [73], run through the GPU-accelerated MTBench reimplementation [70] for batched simulation. Tasks are simulated in NVIDIA IsaacGym [6] with a Franka Panda arm, and episodes are 250 steps.
Tasks and goals. MetaWorld consists of 50 single-arm tabletop manipulation task families (sliding, picking, button-pressing, door opening, etc.), each parameterized by a goal vector specifying a target site or object position drawn from a task-specific range. Each training task family exposes 100 sampled goal variations, giving ML45 a total of 4,500 train configurations across 45 families and 5 held-out test families. The goal vector is masked to zero in the agent’s observation at every timestep, so the agent must infer both the task family and the specific goal from interaction alone.
Action and observation spaces. The action is a 4-D continuous vector . Three components specify a Cartesian end-effector delta and the fourth controls the parallel gripper. The observation is a 39-D continuous vector containing the end-effector position and gripper, the position and quaternion of up to two task-relevant objects (zero-padded for tasks with fewer), the previous-timestep object pose, and the goal slot (masked as described above).
Reward. At each timestep, the agent receives the sum of a dense shaping term and a sparse success bonus. MetaWorld reward functions have changed several times: MetaWorld v2 placed rewards on a more consistent scale across tasks to avoid indirectly leaking task identity, while MTBench scaled the dense reward by 0.01 to reduce returns and improve multi-task RL. Our experiments avoid this issue with categorical value function networks, so we restore the original dense reward scale, as dense rewards are expected to improve OptiB𝖥M online task inference.
Goal-reaching reward. Because ZSRL inference fails on the five held-out ML45 tasks (Figure 19), we instead evaluate a novel behavior in all 50 train and held-out environments (Figure 9 Right) by porting the objective of MetaWorld’s reach task. Objects and dynamics are unchanged, but the task reward is replaced by a function of the distance between the gripper and a hidden target: each step yields , or a success bonus of 20 when cm. Unlike MetaWorld’s sparse shaping, which is scaled by the initial distance to the goal, this linear ramp gives a constant gradient across the workspace and keeps success a clear step above the dense reward. The target is resampled at the start of each trial and fixed across its episodes. We only sample targets near gripper positions visited by MetaBFM’s exploration-mode rollouts (, ) in the same environment. This lets us verify that every target behavior is in-distribution for the exploration data the BFM is trained on, so it is a behavior ZSRL inference could learn to recover.
Figure 19. ZSRL Inference Noise in ML45. Success of pure ZSRL inference (, ) for different Thompson-sampling noise levels ; dashed lines fix a single random for the whole trial. Larger trades early success for steady improvement on the train tasks, while the held-out tasks remain unsolved.
Humanoid Locomotion
Following Li et al. [16], we create environments in IsaacLab [7] for humanoids with locomotion tasks. The humanoids are evaluated on motion tracking with LAFAN [56], a diverse and challenging motion dataset.
Although RL methods on robotics usually include domain randomization to achieve robust sim-to-real transfer, we intentionally try to create more dynamic and random environments, for example, different wind and gravity. So apart from the domain randomization on mass, friction and so on, we randomly sample the scale and direction of wind, and gravity scale as shown in Figure 11. Dynamics setting details are shown in Figure 12.

Figure 11. Wind/Gravity illustration. During online data collection, the wind and gravity strength in each environment are different – the horizontal arrows indicate the direction and strength of the wind (red as the strongest, white as the weakest), while the length of vertical arrows indicate the strength of gravity.
| Domain Randomization | Additive Observation Noise | ||
| Parameter | Range | Observation | Range |
| COM Offset [m] | |||
| Link Mass | |||
| Friction | |||
| Default Joint Pos [m] | |||
| Push Robots [m/s] | |||
| Wind Direction [Rad] | |||
| Wind Magnitude | |||
| Gravity Magnitude [g] |
Table 12. Details in training environment.
State and Observation. The privileged state includes root height, full body pose, body orientation, as well as linear and angular velocities. The observable state is defined as , where is the joint position normalized by the nominal pose , is the joint velocity, is the root angular velocity, and is the projected gravity. To mitigate partial observability, we use an observable history , which concatenates past observations and actions over a horizon of length . All observation components (except root height in the privileged state) are normalized with respect to the humanoid’s current facing direction and root position. During BFM training, we assume access to a dataset of unlabeled motion trajectories , where each trajectory contains both observable and privileged states.
Action. Following PHC [74], we use a proportional-derivative (PD) controller at each DoF of the humanoid, with the action specifying the PD target. With the target joint set as , the torque applied at each joint is . This is different from the residual action representation used in prior motion imitation methods, where the action is added to the reference pose: to speed up training. We do not use any external forces or meta-PD control. For the 29-degree-of-freedom (DoF) humanoid we use – Unitree G1, the action contains the PD controller targets for all DoFs.
BFM failures under multiple dynamics. BFMs trained with multi-dynamics data might suffer from taking actions from the wrong environment dynamics. This can be shown directly via the visualizations in Figure 13. For BFM-Multi, in the stage of trying to get up from the ground, the robot acts as if there were wind from certain directions, loses balance and finally tracks badly. As a comparison, BFMs with context learning can identify the dynamics, so the states are close to the BFM-Single, which is trained solely on one type of dynamics.

Figure 13. Qualitative Comparison on Humanoid Motion Tracking.
Implementation Details
Latent Context Learning
Algorithm 1 Latent Context Learning
Input: trajectory batch with ; EMA rate ; networks , , , , ; EMA target encoder .
Output: augmented trajectory with .
// Augment each with dynamics features
// Self-supervised dynamics losses
Take a gradient step on to minimize a weighted sum of (normalized) .
EMA target update: .
return augmented trajectory .
| Hyperparameter | ColorGrid | Wind | MetaWorld | DMC Walker |
| -encoder output dim (dim ) | 200 | 160 | 220 | 256 |
| -encoder output dim (dim ) | 50 | 40 | 80 | 50 |
| -predictor hidden width | 1024 | 800 | 1024 | 1024 |
| -predictor hidden width | 1024 | 800 | 1024 | 1024 |
| Successor-feature discount () | 0.97 | 0.97 | 0.98 | 0.98 |
| Reward multiplier (SF target / BLR) | 10 | 10 | 1 | 1 |
| – orthogonality weight | 0.10 | 0.25 | 0.10 | 0.10 |
Table 3. BFM (TD-JEPA) hyperparameters. In reference to notation in Bagatella et al. [46]. Shared across domains: two-tower predictor with three layers (one shared trunk plus two per-head tower layers).
Algorithm 1 summarizes the Latent Context Learning (LCL) procedure, which augments states with features used for BFM training and intrinsic rewards. Per-domain choices are summarized in Table 2. All four domains use small Transformer dynamics encoders. The optional cross-trajectory regularizer is disabled in § 4, though preliminary experiments in Figure 4 enabled it.
| Hyperparameter | ColorGrid | Wind | MetaWorld | DMC Walker |
| Latent feature dim (dim ) | 32 | 4 | 16 | 8 |
| Inverse-dynamics feature dim | 32 | 4 | 16 | 12 |
| Inverse-dynamics MLP hidden width | 400 | 128 | 300 | 256 |
| Latent-dynamics predictor hidden width | 512 | 128 | 300 | 256 |
| Diagnostic probe hidden width | 400 | 128 | 256 | 256 |
| (, layers, heads) | Tformer (512, 6, 8) | Tformer (128, 2, 4) | Tformer (128, 2, 4) | Tformer (256, 3, 4) |
Table 2. Latent Context Learning hyperparameters. Shared across domains: latent-dynamics and inverse-dynamics loss weights both 1; EMA target rate ; sphere-normalized features; cross-trajectory regularizer weight 0; probe gradients detached.
BFM
While many BFM updates could in principle be used interchangeably, our results use TD-JEPA [46], primarily because it shares the most surface area with off-policy actor–critic RL. The two heads (state representation) and (task representation) each use a two-tower predictor, sharing a small input trunk before splitting into per-head towers. We additionally apply a cross-trajectory orthogonality regularizer on the predicted features. A scalar reward multiplier (ten for ColorGrid and Wind, one for MetaWorld and DMC Walker) is applied to the per-step reward inside both the SF prediction target and the BLR likelihood at deployment, so it sets the natural magnitude of the regression target and therefore couples directly to the Thompson-sampling noise level discussed in Appendix § 9.§ 3.
Online Task Inference
Following OptiBFM [17], we estimate the latent task vector online with Bayesian linear regression on the forward features , resampling at every timestep. Our experiments use Thompson sampling rather than the optimistic alternative because it scales to the long meta-rollouts in our evaluations, and preliminary experiments found little performance gap. The posterior is governed by an observation-noise scale and prior precision ; we found to be the key hyperparameter, scaling roughly linearly with the typical magnitude of the scaled per-step reward.
| Hyperparameter | ColorGrid | Wind | MetaWorld | DMC Walker |
| 10 | 15 | 5 | 0.03 |
Table 4. Online task inference (Thompson Sampling) hyperparameters. Shared across domains: Bayesian linear regression with prior precision , posterior decay 1, exploration bias , posterior resampled every environment step.
| Hyperparameter | ColorGrid | Wind | MetaWorld | DMC Walker |
| Episodic novelty bonus weight | 1.0 | 1.0 | 1.0 | 1.0 |
| Intrinsic critic reward scale | 1.0 | 1.0 | 1.0 | 1.0 |
| Episodic novelty: nearest neighbors | 10 | 10 | 10 | 10 |
| Episodic novelty: cluster distance threshold | ||||
| Episodic novelty: maximum kernel similarity | 8.0 | 8.0 | 8.0 | 8.0 |
| Episodic novelty: kernel |
Table 5. Intrinsic reward hyperparameters.
Intrinsic Rewards
The intrinsic reward at each timestep is a sum of two terms: an episodic novelty bonus computed in the style of NGU [61] on the latent context features, and a term proportional to the previous-timestep latent-dynamics prediction error (returned by Alg. 1). The combined scalar is normalized and exposed to the policy as an additional observation, so the actor can directly condition on the current intrinsic signal in addition to driving the -blended critic.
Actor–Critic Update
Sequence backbone. The actor and the intrinsic–extrinsic critic share a transformer trajectory encoder with rotary position embeddings [75] and sliding-window FlashAttention [76].
Actor and Critic. The critic, , is a small ensemble of two-hot regression heads over a fixed return range [77], which helps absorb the large variation in TD-target scale induced by randomizing and . Both the actor and critic heads consume a Fourier encoding of the scalars in Equation (4) as additional input features. The actor is an MLP with a standard Tanh-Gaussian output.
Computational Resources and Statistical Significance
All models are trained on an NVIDIA RTX A5000 GPU, with training runs for the MetaBFM (§ 4) experiments lasting 12–36 hours depending on the domain. Shaded error bars in results figures indicate the mean and standard deviation of at least three trials, except for MetaWorld experiments, which use a single training run due to computational constraints. Results indicating test-time adaptation curves (performance vs. ) average these results over the last 3 evaluation cycles during training, then over seeds.
Limitations
BFM and ZSRL algorithms are an active area of research, and reported gains often depend on hyperparameters, implementation details, and benchmark choice. Our experiments use TD-JEPA as a representative state-of-the-art BFM update, but broader comparisons across BFM objectives and online task inference methods remain important future work. MetaBFM also introduces test-time choices through and . These parameters are useful because they let a single policy interpolate between exploration, exploitation, and ZSRL-style reward inference, but they also create a model-selection problem. In our experiments, we evaluate a few interpretable settings; in deployment, one may need validation tasks or an online rule for choosing and .
| Hyperparameter | ColorGrid | Wind | MetaWorld | DMC Walker |
| Training sequence length | 360 | 226 | 501 | 128 |
| Actor / critic context length | 96 | 128 | 128 | 128 |
| Policy attention window | 64 timesteps | full attention | 64 timesteps | full attention |
| Actor MLP hidden width | 512 | 300 | 400 | 400 |
| Critic MLP hidden width | 300 | 300 | 400 | 300 |
| Critic output bins (two-hot) | 64 | 64 | 96 | 256 |
| Critic return range | ||||
| Target network rate () | ||||
| Number of Fourier frequencies for encoding | 16 | 8 | 8 | 9 |
Table 6. Actor–critic hyperparameters.
Additional Results
Online ZSRL
BFMs are inherently proposed as offline algorithms [14, 44], trained on offline exploratory datasets, which can then be used to directly infer policies for diverse reward functions. Inference solves a linear regression problem (Appendix § 7.§ 2) using samples drawn from the same offline datasets. Prior work [59] has shown that these BFMs are not effective for online training. Methods such as [15, 16] have used expert datasets to guide the online training of these models.
We compare the performance of BFMs when trained offline and online on standard DM Control tasks [67]. We compare (Figure 14) the offline-trained BFM (on ExoRL datasets [78]) using offline inference (labeled offline) with the same model using online inference (labeled offline-online) as proposed by [17] and with a fully online BFM that does not assume access to any external dataset during training or inference.

Figure 14. Offline vs Online BFMs: Comparison of fully offline, offline training and online inference and fully online BFMs on DM Control tasks.
It is clear that BFMs are optimally suited to fully offline settings. While using online inference degrades the performance slightly, the performance of online-trained BFM is significantly worse. BFMs require high coverage, which is hard to obtain during online training, leading to poor performance. On the other hand, meta-RL often improves when trained online, as observed from Figure 15.
Figure 15. Online vs. Offline Meta-RL. In ColorGrid (), offline meta-RL from a static dataset generated by a uniform behavior policy leads to significant distribution shift in long-context representations; online learning substantially improves performance.
Utilizing Available Rewards
BFMs are designed to be trained using reward-free interactions. We design a training procedure by providing occasional reward information to the BFMs to see whether they can make use of the available rewards. On the trajectories where rewards are available, the BFMs exploit the reward function to interact with the environment using an optimal policy (optimal for the partially trained BFM). It adds regularization to the state features to ensure they remain sufficiently predictive of the available rewards, as discussed in § 3.3.
We compare the performance of online-trained BFMs with and without the use of reward functions in Figure 16. The improvement in performance when using available rewards is not consistent. While an improvement is observed in the walker, similar improvements are not observed in the other two environments. On ColorGrid, the BFMs show improvement in performance when leveraging the available rewards, as shown in Figure 17.

Figure 16. Effect of using rewards during training of BFMs: Comparing online BFMs with and without using available rewards during training.
Figure 17. Reward Regression and -sampling. Sampling reward-estimated values and fitting known reward labels (using an auxiliary loss with weight ) fits better linear reward estimates (center), focuses learning on more similar latent tasks (right), and marginally improves early-episode adaptation in the ColorGrid domain (left).
MetaWorld
Figure 18 evaluates MetaBFM on six single-task ML1 problems, where train and test tasks share one manipulation skill and differ only in hidden goal positions.
Figure 18. Online Learning in MetaWorld ML1. MetaBFM evaluated in the three main inference configurations on six MetaWorld ML1 problems.
Figure 19 studies pure ZSRL inference (, ) on ML45 as a function of the Thompson-sampling noise (Appendix § 9.§ 3). On the 45 train tasks, small commits to a task estimate early and plateaus, while trades a worse second episode for steady improvement through 25 episodes, well beyond the training horizon. On the five held-out tasks, success stays near zero at every .
References
- [1]Jacob Beck, Risto Vuorio, Evan Zheran Liu, Zheng Xiong, Luisa Zintgraf, Chelsea Finn, and Shimon Whiteson. A survey of meta-reinforcement learning. arXiv preprint arXiv:2301.08028, 2023.
- [2]Adaptive Agent Team, Jakob Bauer, Kate Baumli, Satinder Baveja, Feryal Behbahani, Avishkar Bhoopchand, Nathalie Bradley-Schmieg, Michael Chang, Natalie Clay, Adrian Collister, et al. Human-timescale adaptation in an open-ended task space. arXiv preprint arXiv:2301.07608, 2023.arxiv.org/abs/2301.07608
- [3]Serena Booth, W Bradley Knox, Julie Shah, Scott Niekum, Peter Stone, and Alessandro Allievi. The perils of trial-and-error reward design: misdesign through overfitting and invalid task specifications. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 5920–5929, 2023.
- [4]W Bradley Knox, Alessandro Allievi, Holger Banzhaf, Felix Schmitt, and Peter Stone. Reward (mis) design for autonomous driving. Artificial Intelligence, 316:103829, 2023.arxiv.org/abs/2104.13906
- [5]Jacek Karwowski, Oliver Hayman, Xingjian Bai, Klaus Kiendlhofer, Charlie Griffin, and Joar Max Viktor Skalse. Goodhart’s law in reinforcement learning. In The Twelfth International Conference on Learning Representations, 2023.arxiv.org/abs/2310.09144
- [6]Viktor Makoviychuk, Lukasz Wawrzyniak, Yunlong Guo, Michelle Lu, Kier Storey, Miles Macklin, David Hoeller, Nikita Rudin, Arthur Allshire, Ankur Handa, et al. Isaac gym: High performance gpu-based physics simulation for robot learning. arXiv preprint arXiv:2108.10470, 2021.arxiv.org/abs/2108.10470
- [7]Mayank Mittal, Pascal Roth, James Tigue, Antoine Richard, Octi Zhang, Peter Du, Antonio Serrano-Munoz, Xinjie Yao, René Zurbrügg, Nikita Rudin, et al. Isaac lab: A gpu-accelerated simulation framework for multi-modal robot learning. arXiv preprint arXiv:2511.04831, 2025.arxiv.org/abs/2511.04831
- [8]Seth Karten, Rahul Dev Appapogu, and Chi Jin. Automatic generation of high-performance rl environments. arXiv preprint arXiv:2603.12145, 2026.arxiv.org/abs/2603.12145
- [9]Jake Bruce, Michael D Dennis, Ashley Edwards, Jack Parker-Holder, Yuge Shi, Edward Hughes, Matthew Lai, Aditi Mavlankar, Richie Steigerwald, Chris Apps, et al. Genie: Generative interactive environments. In Forty-first International Conference on Machine Learning, 2024.arxiv.org/abs/2402.15391
- [10]Haoxuan Che, Xuanhua He, Quande Liu, Cheng Jin, and Hao Chen. Gamengen-x: Interactive open-world game video generation. arXiv preprint arXiv:2411.00769, 2024.arxiv.org/abs/2411.00769
- [11]Michael Laskin, Denis Yarats, Hao Liu, Kimin Lee, Albert Zhan, Kevin Lu, Catherine Cang, Lerrel Pinto, and Pieter Abbeel. URLB: Unsupervised reinforcement learning benchmark. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021. URL https://openreview.net/forum?id=lwrPkQP_is.arxiv.org/abs/2110.15191
- [12]Siddhant Agarwal, Caleb Chuck, Harshit Sikchi, Jia heng Hu, Max Rudolph, Scott Niekum, Peter Stone, and Amy Zhang. A unified framework for unsupervised reinforcement learning algorithms. In Workshop on Reinforcement Learning Beyond Rewards @ Reinforcement Learning Conference 2025, 2025. URL https://openreview.net/forum?id=tQ0t75p5HA.
- [13]Ce Zhou, Qian Li, Chen Li, Jun Yu, Yixin Liu, Guangjing Wang, Kai Zhang, Cheng Ji, Qiben Yan, Lifang He, et al. A comprehensive survey on pretrained foundation models: A history from bert to chatgpt. International Journal of Machine Learning and Cybernetics, 16(12):9851–9915, 2025.arxiv.org/abs/2302.09419
- [14]Ahmed Touati, Jérémy Rapin, and Yann Ollivier. Does zero-shot reinforcement learning exist? arXiv preprint arXiv:2209.14935, 2022.arxiv.org/abs/2209.14935
- [15]Andrea Tirinzoni, Ahmed Touati, Jesse Farebrother, Mateusz Guzek, Anssi Kanervisto, Yingchen Xu, Alessandro Lazaric, and Matteo Pirotta. Zero-shot whole-body humanoid control via behavioral foundation models. In The Thirteenth International Conference on Learning Representations, 2025.arxiv.org/abs/2504.11054
- [16]Yitang Li, Zhengyi Luo, Tonghe Zhang, Cunxi Dai, Anssi Kanervisto, Andrea Tirinzoni, Haoyang Weng, Kris Kitani, Mateusz Guzek, Ahmed Touati, et al. Bfm-zero: A promptable behavioral foundation model for humanoid control using unsupervised reinforcement learning. arXiv preprint arXiv:2511.04131, 2025.arxiv.org/abs/2511.04131
- [17]Thomas Rupf, Marco Bagatella, Marin Vlastelica, and Andreas Krause. Optimistic task inference for behavior foundation models. arXiv preprint arXiv:2510.20264, 2025.arxiv.org/abs/2510.20264
- [18]Luisa Zintgraf. Fast adaptation via meta reinforcement learning. PhD thesis, University of Oxford, 2022.DOI
- [19]Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International conference on machine learning, pages 1126–1135. PMLR, 2017.arxiv.org/abs/1703.03400
- [20]Yan Duan, John Schulman, Xi Chen, Peter L Bartlett, Ilya Sutskever, and Pieter Abbeel. R1²: Fast reinforcement learning via slow reinforcement learning. arXiv preprint arXiv:1611.02779, 2016.
- [21]Evan Z Liu, Aditi Raghunathan, Percy Liang, and Chelsea Finn. Decoupling exploration and exploitation for meta-reinforcement learning without sacrifices. In *International conference on machine learning*, pages 6925–6935. PMLR, 2021.arxiv.org/abs/2008.02790
- [22]Bradly C Stadie, Ge Yang, Rein Houthooft, Xi Chen, Yan Duan, Yuhuai Wu, Pieter Abbeel, and Ilya Sutskever. Some considerations on learning to explore via meta-reinforcement learning. *arXiv preprint arXiv:1803.01118*, 2018.arxiv.org/abs/1803.01118
- [23]Jan Humplik, Alexandre Galashov, Leonard Hasenclever, Pedro A Ortega, Yee Whye Teh, and Nicolas Heess. Meta reinforcement learning as task inference. *arXiv preprint arXiv:1905.06424*, 2019.arxiv.org/abs/1905.06424
- [24]Luisa Zintgraf, Sebastian Schulze, Cong Lu, Leo Feng, Maximilian Igl, Kyriacos Shiarlis, Yarin Gal, Katja Hofmann, and Shimon Whiteson. Varibad: Variational bayes-adaptive deep rl via meta-learning. *Journal of Machine Learning Research*, 22(289):1–39, 2021.
- [25]Jane X Wang, Zeb Kurth-Nelson, Dhruva Tirumala, Hubert Soyer, Joel Z Leibo, Remi Munos, Charles Blundell, Dharshan Kumaran, and Matt Botvinick. Learning to reinforcement learn. *arXiv preprint arXiv:1611.05763*, 2016.arxiv.org/abs/1611.05763
- [26]Alexander Nikulin, Vladislav Kurenkov, Ilya Zisman, Artem Agarkov, Viacheslav Sinii, and Sergey Kolesnikov. Xland-minigrid: Scalable meta-reinforcement learning environments in jax. *Advances in Neural Information Processing Systems*, 37:43809–43835, 2024.arxiv.org/abs/2312.12044
- [27]Chris Lu, Yannick Schroecker, Albert Gu, Emilio Parisotto, Jakob Foerster, Satinder Singh, and Feryal Behbahani. Structured state space models for in-context reinforcement learning. *arXiv preprint arXiv:2303.03982*, 2023.arxiv.org/abs/2303.03982
- [28]Vitchyr H Pong, Ashvin V Nair, Laura M Smith, Catherine Huang, and Sergey Levine. Offline meta-reinforcement learning with online self-supervision. In *International Conference on Machine Learning*, pages 17811–17829. Pmlr, 2022.arxiv.org/abs/2107.03974
- [29]Ron Dorfman, Idan Shenfeld, and Aviv Tamar. Offline meta learning of exploration. *arXiv preprint arXiv:2008.02598*, 2020.arxiv.org/abs/2008.02598
- [30]Yunkai Gao, Rui Zhang, Jiaming Guo, Fan Wu, Qi Yi, Shaohui Peng, Siming Lan, Ruizhi Chen, Zidong Du, Xing Hu, et al. Context shift reduction for offline meta-reinforcement learning. *Advances in Neural Information Processing Systems*, 36:80024–80043, 2023.arxiv.org/abs/2311.03695
- [31]Lanqing Li, Hai Zhang, Xinyu Zhang, Shatong Zhu, Yang Yu, Junqiao Zhao, and Pheng-Ann Heng. Towards an information theoretic framework of context-based offline meta-reinforcement learning. *Advances in Neural Information Processing Systems*, 37:75642–75667, 2024.arxiv.org/abs/2402.02429
- [32]Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. *arXiv preprint arXiv:2005.01643*, 2020.arxiv.org/abs/2005.01643
- [33]Abhishek Gupta, Benjamin Eysenbach, Chelsea Finn, and Sergey Levine. Unsupervised meta-learning for reinforcement learning. *arXiv preprint arXiv:1806.04640*, 2018.arxiv.org/abs/1806.04640
- [34]Allan Jabri, Kyle Hsu, Abhishek Gupta, Ben Eysenbach, Sergey Levine, and Chelsea Finn. Unsupervised curricula for visual meta-reinforcement learning. *Advances in Neural Information Processing Systems*, 32, 2019.arxiv.org/abs/1912.04226
- [35]Octavio Papalardo. Unsupervised learning of efficient exploration: Pre-training adaptive policies via self-imposed goals. *arXiv preprint arXiv:2601.19810*, 2026.
- [36]Jacopo Di Ventura, Jan Felix Kleuker, Aske Plaat, and Thomas Moerland. A unified framework for zero-shot reinforcement learning. *arXiv preprint arXiv:2510.20542*, 2025.arxiv.org/abs/2510.20542
- [37]Scott Jeen, Tom Bewley, and Jonathan M Cullen. Zero-shot reinforcement learning under partial observability. *arXiv preprint arXiv:2506.15446*, 2025.arxiv.org/abs/2506.15446
- [38]Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. *arXiv preprint arXiv:1509.02971*, 2015.arxiv.org/abs/1509.02971
- [39]Ahmed Touati and Yann Ollivier. Learning one representation to optimize all rewards. Advances in Neural Information Processing Systems, 34:13–23, 2021.arxiv.org/abs/2103.07945
- [40]Raymond Chua, Arna Ghosh, Christos Kaplanis, Blake A Richards, and Doina Precup. Learning successor features the simple way. Advances in Neural Information Processing Systems, 37: 49957–50030, 2024.arxiv.org/abs/2410.22133
- [41]Marco Bagatella, Thomas Rupf, Georg Martius, and Andreas Krause. Soft forward-backward representations for zero-shot reinforcement learning with general utilities. arXiv preprint arXiv:2602.06769, 2026.
- [42]Chongyi Zheng, Royina Karegoudra Jayanth, and Benjamin Eysenbach. Can we really learn one representation to optimize all rewards? arXiv preprint arXiv:2602.11399, 2026.
- [43]Scott Jeen, Tom Bewley, and Jonathan M Cullen. Zero-shot reinforcement learning from low quality data. Advances in Neural Information Processing Systems, 37:16894–16942, 2024.
- [44]Siddhant Agarwal, Harshit Sikchi, Peter Stone, and Amy Zhang. Proto successor measure: Representing the behavior space of an rl agent. In International Conference on Machine Learning, pages 566–586. PMLR, 2025.arxiv.org/abs/2411.19418
- [45]Seohong Park, Tobias Kreiman, and Sergey Levine. Foundation policies with hilbert representations. arXiv preprint arXiv:2402.15567, 2024.arxiv.org/abs/2402.15567
- [46]Marco Bagatella, Matteo Pirotta, Ahmed Touati, Alessandro Lazaric, and Andrea Tirinzoni. Td-jepa: Latent-predictive representations for zero-shot reinforcement learning. arXiv preprint arXiv:2510.00739, 2025.
- [47]Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. arXiv preprint arXiv:1810.12894, 2018.arxiv.org/abs/1810.12894
- [48]Kate Rakelly, Aurick Zhou, Chelsea Finn, Sergey Levine, and Deirdre Quillen. Efficient off-policy meta-reinforcement learning via probabilistic context variables. In International conference on machine learning, pages 5331–5340. PMLR, 2019.
- [49]Maksim Bobrin, Ilya Zisman, Alexander Nikulin, Vladislav Kurenkov, and Dmitry Dylov. Zero-shot adaptation of behavioral foundation models to unseen dynamics. arXiv preprint arXiv:2505.13150, 2025.
- [50]Dibya Ghosh, Abhishek Gupta, Ashwin Reddy, Justin Fu, Coline Devin, Benjamin Eysenbach, and Sergey Levine. Learning to reach goals via iterated supervised learning. arXiv preprint arXiv:1912.06088, 2019.
- [51]Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748, 2018.arxiv.org/abs/1807.03748
- [52]Max Schwarzer, Ankesh Anand, Rishab Goel, R Devon Hjelm, Aaron Courville, and Philip Bachman. Data-efficient reinforcement learning with self-predictive representations. arXiv preprint arXiv:2007.05929, 2020.
- [53]Max Schwarzer, Johan Samir Obando Ceron, Aaron Courville, Marc G Bellemare, Rishabh Agarwal, and Pablo Samuel Castro. Bigger, better, faster: Human-level atari with human-level efficiency. In International Conference on Machine Learning, pages 30365–30380. PMLR, 2023.arxiv.org/abs/2305.19452
- [54]Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Ghesh laghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems, 33:21271–21284, 2020.
- [55]Pranaya Jajoo, Harshit Sikchi, Siddhant Agarwal, Amy Zhang, Scott Niekum, and Martha White. Regularized latent dynamics prediction is a strong baseline for behavioral foundation models. In The Fourteenth International Conference on Learning Representations, 2026. URL https://openreview.net/forum?id=jdL6WB5JHZ.
- [56]Félix G. Harvey, Mike Yurick, Derek Nowrouzezahrai, and Christopher Pal. Robust motion in-betweening. ACM Transactions on Graphics (Proceedings of ACM SIGGRAPH), 39(4), 2020.DOI
- [57]Younggyo Seo, Li Li Chen, Jinwoo Shin, Honglak Lee, Pieter Abbeel, and Kimin Lee. State en- tropy maximization with random encoders for efficient exploration. In International conference on machine learning, pages 9443–9454. PMLR, 2021.arxiv.org/abs/2102.09430
- [58]Roberta Raileanu and Tim Rocktäschel. Ride: Rewarding impact-driven exploration for procedurally-generated environments. In International Conference on Learning Representations, 2020. URL https://openreview.net/forum?id=rkg-TJBFPB.arxiv.org/abs/2002.12292
- [59]Jingbo Sun, Songjun Tu, qichao Zhang, Haoran Li, Xin Liu, Yaran Chen, Ke Chen, and Dongbin Zhao. Unsupervised zero-shot reinforcement learning via dual-value forward-backward representation. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=OQnKnt4110.
- [60]Mikael Henaff, Minqi Jiang, and Roberta Raileanu. A study of global and episodic bonuses for exploration in contextual mdps. In International Conference on Machine Learning, pages 12972–12999. PMLR, 2023.arxiv.org/abs/2306.03236
- [61]Adrià Puigdomènech Badia, Pablo Sprechmann, Alex V. Vitsvitskyi, Daniel Guo, Bilal Piot, Steven Kapturowski, Olivier Tieleman, Martín Arjovsky, Alexander Pritzel, Andrew Bolt, et al. Never give up: Learning directed exploration strategies. arXiv preprint arXiv:2002.06038, 2020.arxiv.org/abs/2002.06038
- [62]Mikael Henaff, Roberta Raileanu, Minqi Jiang, and Tim Rocktäschel. Exploration via elliptical episodic bonuses. Advances in Neural Information Processing Systems, 35:37631–37646, 2022.arxiv.org/abs/2210.05805
- [63]Luisa M Zintgraf, Leo Feng, Cong Lu, Maximilian Igl, Kristian Hartikainen, Katja Hofmann, and Shimon Whiteson. Exploration in approximate hyper-state space for meta reinforcement learning. In International Conference on Machine Learning, pages 12991–13001. PMLR, 2021.
- [64]Jin Zhang, Jianhao Wang, Hao Hu, Tong Chen, Yingfeng Chen, Changjie Fan, and Chongjie Zhang. Metacure: Meta reinforcement learning with empowerment-driven exploration. In International Conference on Machine Learning, pages 12600–12610. PMLR, 2021.arxiv.org/abs/2006.08170
- [65]Jake Grigsby, Justin Sasek, Samyak Parajuli, Daniel Adebi, Amy Zhang, and Yuke Zhu. Amago- 2: Breaking the multi-task barrier in meta-reinforcement learning with transformers. Advances in Neural Information Processing Systems, 37:87473–87508, 2024.arxiv.org/abs/2411.11188
- [66]Tianwei Ni, Benjamin Eysenbach, and Ruslan Salakhutdinov. Recurrent model-free rl can be a strong baseline for many pomdps, 2022.arxiv.org/abs/2110.05038
- [67]Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint arXiv:1801.00690, 2018.arxiv.org/abs/1801.00690
- [68]Jonas Rothfuss, Dennis Lee, Ignasi Clavera, Tamim Asfour, and Pieter Abbeel. Promp: Proximal meta-policy search. arXiv preprint arXiv:1810.06784, 2018.arxiv.org/abs/1810.06784
- [69]Tianhe Yu, Deirdre Quillen, Zhangpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on robot learning, pages 1094–1100. PMLR, 2020.arxiv.org/abs/1910.10897
- [70]Viraj Joshi, Zifan Xu, Bo Liu, Peter Stone, and Amy Zhang. Benchmarking massively paral- lelized multi-task reinforcement learning for robotics tasks. arXiv preprint arXiv:2507.23172, 2025.arxiv.org/abs/2507.23172
- [71]Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014.
- [72]Jake Grigsby and Yanjun Qi. Measuring visual generalization in continuous control from pixels. arXiv preprint arXiv:2010.06740, 2020.arxiv.org/abs/2010.06740
- [73]Reginald McLean, Evangelos Chatzaroulas, Luc McCutcheon, Frank Röder, Tianhe Yu, Zhanpeng He, KR Zentner, Ryan Julian, Jordan K Terry, Isaac Woungang, et al. Meta-world+: An improved, standardized, rl benchmark. arXiv preprint arXiv:2505.11289, 2025.arxiv.org/abs/2505.11289
- [74]Zhengyi Luo, Jinkun Cao, Kris Kitani, Weipeng Xu, et al. Perpetual humanoid control for real-time simulated avatars. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10895–10904, 2023.arxiv.org/abs/2305.06456
- [75]Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063, 2024.arxiv.org/abs/2104.09864
- [76]Tri Dao, Daniel Y Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness. arXiv preprint arXiv:2205.14135, 2022.arxiv.org/abs/2205.14135
- [77]Danijar Hafner, Jur̄gis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models. arXiv preprint arXiv:2301.04104, 2023.arxiv.org/abs/2301.04104
- [78]Denis Yarats, David Brandfonbrener, Hao Liu, Michael Laskin, Pieter Abbeel, Alessandro Lazaric, and Lerrel Pinto. Don’t change the algorithm, change the data: Exploratory data for offline reinforcement learning. arXiv preprint arXiv:2201.13425, 2022.arxiv.org/abs/2201.13425










