betaFeedback

WorksComputer Science

Action-Aligned Latent Planning for Versatile Humanoid Loco-Manipulation

Abstract

Learning a single perceptive policy for diverse humanoid loco-manipulation requires unifying high-level task decisions without losing the state-dependent physical responses that make each interaction succeed. This makes the representation between high-level planning and whole-body control critical: kinematic or contact targets provide compact abstractions but do not directly encode interaction-specific control responses, while directly predicting low-level actions requires the planner to model high-frequency closed-loop feedback. We propose FIBER, a framework that uses the executable latent space of a frozen behavioral foundation model (BFM) as a shared planning representation for heterogeneous interaction skills. Our latent policy projection (LPP) maps independently acquired task-specific policies into this representation by identifying latent commands whose induced actions match their local control responses, enabling lower-rate planning while retaining the BFM's high-frequency state feedback. After projection, we aggregate closed-loop latent rollouts from all skills and train a single perceptive flow planner entirely offline. We further introduce AH-BFM, an adaptive-horizon BFM trained with multi-gamma forward-backward (FB) learning to support both precise control and temporally extended whole-body behavior. We evaluate FIBER on box pickup, door opening, and object grasping in simulation and on a real Unitree G1 humanoid. Across 128 diverse variations per task in simulation, FIBER achieves 42.0% mean success, more than tripling projected latent replay (11.6%) and substantially outperforming kinematic planning (3.5%). Real-world experiments further demonstrate sim-to-real transfer, skill chaining, and conditional steering through the same unified planner.

One Autonomous Vision-Conditioned Policy, Diverse Real-World Interactions

Figure 1. A single autonomous perceptive latent planner enables diverse real-world interactions, including object grasping in (a,b), door opening in (c), whole-body box pickup in (d), waypoint-conditioned steering in (e), and crouched locomotion while carrying in (f).

Project website: https://fiberaa.github.io/.

Introduction

Humanoid loco-manipulation requires both deciding what interaction to perform from visual and semantic context and determining how to realize it through balance, contact, and coordinated whole-body control. Recent work has achieved strong task-specific interaction skills, including object manipulation, articulated-object interaction, and object transport [1, 2], often using task-specific objectives, observations, and training procedures. Scaling these successes to a versatile policy requires unifying diverse task-level decisions without losing the physical responses that make each skill succeed.

A common strategy separates task-level decision making from whole-body control. A high-level policy predicts structured targets such as body motion, end-effector poses, or contacts, while a low-level controller realizes them on the robot. However, the same target may require different actions under different contacts, object states, or disturbances, so such representations do not directly capture the state-dependent control responses required for interaction. Predicting low-level actions avoids this abstraction, but offline learning then requires broad coverage of closed-loop corrective responses, which is difficult for contact-rich interaction. This exposes a central challenge for humanoid planning: how can we abstract low-level control complexity without abstracting away the physical responses that make interaction succeed?

A learned behavioral representation offers a way to resolve this tension. We build on a Behavior Foundation Model (BFM) that exposes an executable behavioral latent space. Given the current robot state and a latent command, BFM produces low-level actions at high frequency. A latent therefore indexes a state-dependent control policy rather than merely a desired motion, allowing high-level planning to operate over compact behavioral commands while the BFM retains high-frequency closed-loop physical feedback.

Using such an interface for versatile loco-manipulation still requires acquiring the interaction capabilities that the planner should invoke. One approach is to collect demonstrations for the desired interactions through a common data-acquisition pipeline [3, 4], but obtaining broad coverage of contact-rich and corrective behavior is expensive. Another approach is to first train task-specific interaction policies and distill them into a generalist. On-policy multi-expert distillation can improve closed-loop coverage by querying these policies at states visited by the generalist, but requires all policies and their environments to remain available during integration. We instead consider a decoupled setting: each interaction policy can be acquired independently, and only afterward incorporated into the shared behavioral interface.

Under this decoupled setting, different interaction policies need not share their observations, objectives, or task representations. They nevertheless share one quantity that directly describes their physical realization: the actions applied to the robot. A latent command from the frozen BFM is grounded in the same quantity, because it also produces robot actions conditioned on the current robot state. This common action space provides a direct way to relate an independently acquired interaction policy to the BFM latent representation.

Based on this insight, we propose FIBER, which represents independently acquired interaction policies, or experts, in the executable latent space of a frozen BFM. At the core of FIBER is Latent Policy Projection (LPP), which identifies latent commands by matching the actions induced by the BFM to an expert’s local state-dependent control responses. The resulting projected policy executes through the frozen BFM, preserving high-frequency state feedback while expressing heterogeneous interaction skills through a common lower-rate representation.

Because each expert is projected independently, the experts need not share an acquisition procedure or remain jointly available during subsequent training. After projection, we roll out the projected policies and collect their latent commands together with the perceptive and proprioceptive observations available to the final policy. These closed-loop latent rollouts are aggregated across skills to train a single perceptive planner entirely offline. At deployment, the planner predicts future latent commands from onboard observations and a high-level steering condition, while the frozen BFM provides the state-dependent whole-body responses required to execute them.

Finally, projection and planning impose different temporal requirements on the latent representation. Matching an expert’s control response favors precise short-horizon behavior, whereas lower-rate planning benefits from stable commands that preserve behavior over longer horizons. We therefore introduce AH-BFM, an adaptive-horizon BFM trained with multi-γ\gamma Forward-Backward (FB) learning to support both control fidelity and temporally extended behavioral structure.

We evaluate FIBER on three heterogeneous interaction tasks—box pickup, door opening, and object grasping-in simulation and on a real Unitree G1. In simulation, LPP largely preserves expert performance on box pickup and door opening near the interaction regime. The unified perceptive planner achieves 42.0% mean success, compared with 11.6% for fixed projected latent replay and 3.5% for kinematic planning. Real-world experiments further demonstrate direct sim-to-real transfer, interaction chaining, and waypoint-conditioned steering through the same planner.

Our contributions are threefold:

  • A shared executable representation for humanoid interaction planning. We use the latent space of a frozen BFM as a behavioral interface that supports lower-rate high-level planning while retaining state-dependent interaction realization responses.

  • Action-aligned latent policy projection. We introduce LPP, which maps independently acquired interaction experts into the shared BFM representation by aligning their local control responses in robot action space, decoupling skill acquisition from subsequent integration.

  • Adaptive-horizon BFM and SF-guided behavior reuse. We introduce multi-γ\gamma FB learning and AH-BFM that balance precise control with temporally extended behavioral structure, together with Successor-Feature (SF)-guided interaction-preserving behaviors reuse.

Related Work

Humanoid Controller and Behavioral Foundation Models

Humanoid control has increasingly moved from skill-specific policies toward shared controllers that support broad motor repertoires through a common conditioning interface. Motion-conditioned approaches unify locomotion and whole-body behaviors through dense or sparse motion specifications [5, 6], while recent systems extend unified control toward language-conditioned behaviors and humanoid loco-manipulation [7, 8]. These works demonstrate that diverse behaviors can be consolidated within a single controller, but the supported behaviors are typically acquired through the controller’s predefined training and conditioning interface. Our work instead considers a complementary setting: integrating interaction behaviors acquired independently of this interface into an already-trained controller.

BFMs expose a learned behavioral latent space as such a shared control interface. BFM-Zero [9] uses FB to learn a promptable humanoid behavioral space, while subsequent behavioral inference methods show that expert behaviors can be recovered by optimizing within a frozen FB latent space [10]. These results establish latent inference as an effective mechanism for behavior reproduction. We build on this capability but study its use as an interface for interaction representation, where the latent sequence must capture temporally varying local control responses rather than only recover a stationary behavior or desired state trajectory. This setting further exposes a trade-off in conventional FB learning: the successor horizon favorable for precise local tracking differs from that required to preserve long-term behavior. Our adaptive-horizon BFM is designed to retain both properties within the same behavioral space.

Humanoid Loco-Manipulation and Representation

Humanoid loco-manipulation requires specifying not only the desired motion of the robot, but also how it should respond to physical interaction with objects and the environment. A broad family of methods makes interaction explicit through structured task representations, including contact sequences, end-effector constraints, task-motion objectives, distance fields, and interaction-preserving references [11, 12, 13, 14, 15, 16, 17]. Such representations provide effective and often interpretable interfaces for learning diverse skills. However, they specify interaction through variables chosen in advance; the resulting task description does not directly encode the state-dependent control response induced by contact and environment dynamics.

Alternatively, interaction responses can be learned directly through task-specific reinforcement learning (RL) or demonstrations [18, 19, 20, 21, 22, 23, 24, 25]. Teleoperation and human-video-based pipelines further broaden the sources from which such behaviors can be acquired [26, 27, 28, 29, 30, 31, 32]. These approaches can capture interaction-specific control responses through RL, but the acquired behavior generally remains tied to the specialist policy or downstream training process. In contrast, we decouple interaction acquisition from execution: demonstrated state-action responses are projected into the native latent interface of a frozen behavioral foundation controller, allowing independently acquired interactions to share a common executable substrate.

Methodology

Overview

Our goal is to express independently acquired interaction skills in a shared behavioral interface that preserves their state-dependent physical realization and supports unified perceptive planning. FIBER uses the latent space of a frozen BFM as this interface and maps each interaction expert into it by aligning the expert’s local control responses with the actions induced by BFM latent commands.

As illustrated in Figure 2, FIBER proceeds in three stages. First, interaction skills are acquired independently with task-appropriate supervision. Second, LPP maps each expert into the frozen BFM latent space by matching the actions induced by latent commands to the expert’s local control responses. The resulting projected policies are rolled out directly through the shared behavioral interface, after which the original experts are no longer required. Finally, their closed-loop latent rollouts are aggregated to train a single perceptive flow planner entirely offline, conditioned on onboard observations and high-level steering commands.

Figure 2. Overview of FIBER. (a) Heterogeneous interaction experts are acquired through motion imitation, reward shaping, or successor-feature-guided behavior reuse. (b) Action-aligned latent policy projection maps their short-horizon control responses into a frozen adaptive-horizon BFM; closed-loop rollouts produce executable latent trajectories. (c) These trajectories train a unified perceptive planner entirely offline. Language commands or waypoint conditions are encoded as steering tokens, and a DiT planner predicts future latent trajectories. During deployment, the planner replans from onboard observations while the frozen BFM provides high-frequency feedback.

This latent space must support both precise local interaction and temporally extended whole-body behavior. We therefore introduce AH-BFM, an adaptive-horizon BFM trained with multi-γ\gamma FB learning. We consider

Γ={γs,γl},γs<γl,(1)\Gamma= \{\gamma_s,\gamma_l\}, \qquad\gamma_s < \gamma_l, \tag*{(1)}

where γs\gamma_s emphasizes local transition fidelity and γl\gamma_l preserves longer-horizon behavioral structure. We condition the forward representation FF on γ\gamma while sharing the backward representation BB, and normalize the corresponding values as

Q~γFB=(1−γ)QγFB.(2)\widetilde{Q}^{\mathrm{FB}}_{\gamma} = (1-\gamma)Q^{\mathrm{FB}}_{\gamma}. \tag*{(2)}

The shared actor adaptively aggregates the horizon-specific objectives,

Jπ=E[τlog⁡∑γ∈Γp(γ)exp⁡(Q~γFB(s,πBFM(s,z),z)τ)],(3)J_{\pi} = \mathbb{E}\left[\tau\log\sum_{\gamma\in\Gamma} p(\gamma)\exp\left(\frac{\widetilde{Q}^{\mathrm{FB}}_{\gamma}(s,\pi_{\mathrm{BFM}}(s,z),z)}{\tau}\right)\right], \tag*{(3)}

with gradient weights

wγ∝p(γ)exp⁡(Q~γFB(s,πBFM(s,z),z)τ).(4)w_{\gamma} \propto p(\gamma)\exp\left(\frac{\widetilde{Q}^{\mathrm{FB}}_{\gamma}(s,\pi_{\mathrm{BFM}}(s,z),z)}{\tau}\right). \tag*{(4)}

Here, p(γ)p(\gamma) defines a prior over horizons and τ\tau controls the aggregation sharpness. Because the weights depend on the current state and latent command, the shared actor can combine short- and long-horizon supervision within the same behavioral space.

Heterogeneous Skill Acquisition

FIBER allows interaction skills to be acquired with task-appropriate supervision, observations, and training objectives. We consider three complementary acquisition routes.

a) Motion-based acquisition: For interactions with suitable humanoid-object motion references, such as grasping [33] and object pickup [29], we train task-specific experts by combining motion imitation with interaction objectives.

b) Reward-shaping-based acquisition: Some interactions admit a natural decomposition into successive stages and can therefore be acquired directly through staged reward shaping. For example, opening an articulated door [2] can be trained by progressively rewarding approach, contact establishment, handle manipulation, and door opening. This provides a direct route to task-specific experts without requiring motion references, while allowing each stage to use task-appropriate observations and rewards.

c) SF-guided behavior reuse: The pretrained BFM provides an additional route for extending acquired interactions with whole-body capabilities already represented in its latent space. For an interaction II, such as carrying box, we learn a latent adapter AψIA_{\psi_I} while keeping the BFM frozen:

z^t=AψI(st,zN),at=πBFM(st,z^t),(5)\hat{z}_t = A_{\psi_I}(s_t,z^N), \qquad a_t = \pi_{\mathrm{BFM}}(s_t,\hat{z}_t), \tag*{(5)}

where zNz^N specifies the native behavior to be incorporated.

We optimize the adapter using the interaction reward together with a successor-feature-guided preservation term,

rt=rtI+λSFB(st)⊤zN.(6)r_t = r_t^I + \lambda_{\mathrm{SF}} B(s_t)^\top z^N. \tag*{(6)}

For a fixed zNz^N,

E[∑t=0∞γtB(st)⊤zN]=E[∑t=0∞γtB(st)]⊤zN,(7)\mathbb{E}\left[\sum_{t=0}^{\infty}\gamma^t B(s_t)^\top z^N\right] = \mathbb{E}\left[\sum_{t=0}^{\infty}\gamma^t B(s_t)\right]^\top z^N, \tag*{(7)}

so the second term encourages the adapted rollout to preserve the behavior specified by zNz^N while rtIr_t^I maintains the interaction. This enables acquired interactions to be extended with native skills, such as running or kneeling while carrying.

Motion imitation, task-specific reward learning, and successor-feature-guided behavior reuse each produce task-specific experts, all of which are subsequently projected into the shared BFM latent space through LPP.

Action-Aligned Latent Policy Projection

Externally acquired experts must be converted into a representation suitable for offline planner training. Kinematic trajectories provide compact behavioral targets, but do not uniquely determine the control responses required to realize an interaction. Expert actions provide this missing physical information, yet directly modeling high-frequency actions requires covering a broad state-dependent control distribution. We instead use the frozen BFM to map expert behavior into a lower-rate latent representation that preserves both its executable response and native behavioral semantics.

For expert πiE\pi_i^E, we learn a local projector PψiP_{\psi_i} that maps the expert observation at update time tkt_k to a latent,

zk=Pψi(otk).(8)z_k = P_{\psi_i}(o_{t_k}). \tag*{(8)}

Between consecutive updates, we interpolate from the previous latent zk−1z_{k-1} to zkz_k using SLERP to obtain a latent command at each control step,

ztk+h=SLERP⁡(zk−1,zk,αh),h=0,…,HP−1.(9)z_{t_k+h} = \operatorname{SLERP}(z_{k-1},z_k,\alpha_h), \qquad h=0,\ldots,H_P-1. \tag*{(9)}

We identify the latent command primarily through its induced actions. Given expert actions

atEi=πiE(oi,tE),(10)a_{t}^{E_i} = \pi_i^E(o_{i,t}^E), \tag*{(10)}

we define the short-horizon action-alignment objective

Lact=EPψi[1HP∑h=0HP−1∥πBFM(stk+h,ztk+h)−atk+hEi∥22].(11)\mathcal{L}_{\mathrm{act}} = \mathbb{E}_{P_{\psi_i}}\left[\frac{1}{H_P}\sum_{h=0}^{H_P-1}\left\|\pi_{\mathrm{BFM}}(s_{t_k+h},z_{t_k+h})-a_{t_k+h}^{E_i}\right\|_2^2\right]. \tag*{(11)}

Here, PψiP_{\psi_i} denotes the on-policy state distribution induced by the current projector executed through the frozen BFM. This directly constrains the projected latent by matching the expert’s responses, rather than by a predefined kinematic or contact representation target.

MethodPickup BoxOpen DoorGrasp ObjectMean
w/ appr.w/o appr.w/ appr.w/o appr.
Task-specific policies
Expert Policy†100.0±0.090.8±0.589.3±0.297.9±0.897.0±0.895.4
LPP w/ SONIC [34]†0.0±0.00.0±0.083.1±0.788.1±0.65.1±0.729.4
LPP w/ Fixed-γ\gamma BFM†75.4±0.280.2±1.361.8±1.972.4±0.769.3±1.268.8
LPP w/ AH-BFM†81.9±0.987.4±2.483.5±1.897.3±0.452.9±2.172.8
Reference-prompted execution
SONIC [34]0.0±0.00.0±0.00.4±0.03.1±0.012.1±0.04.2
FALCON [21]0.0±0.00.0±0.00.4±0.02.7±0.011.9±0.34.1
B(s)B(s) Tracking0.0±0.00.0±0.00.4±0.05.6±0.513.9±0.54.8
LPP Replay11.2±0.817.7±1.212.2±0.949.5±0.611.3±1.111.6
Perceptive planning
OmniContact∗ [17]29.9±0.249.6±0.0––––
LessMimic [14]38.3±0.028.5±0.0––––
Action Planner0.0±0.00.0±0.00.0±0.00.0±0.00.0±0.00.0
Kinematic Planner0.0±0.00.0±0.00.0±0.00.0±0.010.6±0.53.5
Ours – FIBER42.1±0.644.1±1.053.5±4.851.2±1.030.3±2.342.0

Table I. Simulation success rates (%) across heterogeneous loco-manipulation tasks. “w/ appr.” includes the approaching phase, whereas “w/o appr.” initializes the humanoid near the interaction state. † denotes a task-specific policy, and ∗ denotes access to privileged object states instead of egocentric observations. Mean is computed using the “w/ appr.” results.

Action alignment alone, however, can leave the latent underdetermined: different commands may induce similar local actions while corresponding to different behavioral directions in the BFM space. Recall that the BFM latent zz acts as a policy index through the forward–backward objective F(s,a)⊤zF(s,a)^{\top}z, while B(s)B(s) provides the corresponding stateside behavioral representation. We therefore anchor each projected command to the native semantic representation of its local state,

Lsem=E[∥zk−B(stk)∥22].(12)\mathcal{L}_{\mathrm{sem}} = \mathbb{E}\left[\left\|z_k - B(s_{t_k})\right\|_2^2\right]. \tag*{(12)}

The complete projection objective is

Lproj=Lact+λsemLsem.(13)\mathcal{L}_{\mathrm{proj}} = \mathcal{L}_{\mathrm{act}} + \lambda_{\mathrm{sem}}\mathcal{L}_{\mathrm{sem}}. \tag*{(13)}

The action term determines the latent according to the expert’s required control response, while the semantic term selects solutions consistent with the native behavioral geometry of the BFM. This reduces action-equivalent latent ambiguity and yields more coherent latent trajectories for downstream planning. To obtain a robust projection beyond nominal expert rollouts, we train each PψiP_{\psi_i} independently using single-skill on-policy data aggregation [35].

After projection, the original expert is no longer required. We roll out PψiP_{\psi_i} under randomized initial states and environment configurations and collect

DiZ={(dt,pt,ut,zt)},(14)\mathcal{D}_i^Z = \left\{(d_t,p_t,u_t,z_t)\right\}, \tag*{(14)}

where dtd_t and ptp_t denote the perceptive and proprioceptive observations available to the downstream planner, and utu_t denotes the corresponding high-level task condition. These latent rollouts provide executable and semantically regularized targets that can be combined across independently acquired skills for offline latent planning.

Unified Latent Flow Planning

Given the collected latent rollouts, we train a single perceptive planner to generate future behavioral commands entirely offline. At each planning step, the policy is conditioned on onboard depth dtd_t, proprioception ptp_t, and a compact steering token ctc_t:

ct=Esteer(ut).(15)c_t = E_{\mathrm{steer}}(u_t). \tag*{(15)}

The steering input utu_t specifies the desired high-level behavior and can be instantiated by different conditioning modalities, such as a language command or a navigation waypoint. This separates the representation of task intent from the perceptive latent planner itself.

The planner predicts a horizon of future BFM commands

x1=vec⁡(zt,zt+1,…,zt+H−1)(16)x_1 = \operatorname{vec}(z_t,z_{t+1},\ldots,z_{t+H-1}) \tag*{(16)}

using conditional flow matching. Given Gaussian noise x0∼N(0,I)x_0 \sim\mathcal{N}(0,I) and flow time τ∼U(0,1)\tau\sim\mathcal{U}(0,1), we construct the linear interpolation xτ=(1−τ)x0+τx1x_\tau= (1-\tau)x_0+\tau x_1. Then a conditional network predicts the clean latent, and is trained with

Lflow=E[∥x^1−x1∥22].(17)\mathcal{L}_{\mathrm{flow}} = \mathbb{E}\left[\left\|\hat{x}_1-x_1\right\|_2^2\right]. \tag*{(17)}

At deployment, the planner repeatedly observes the current depth and proprioceptive state, constructs the steering token from the desired high-level condition, and generates a future latent trajectory. The leading latent commands are executed through the frozen BFM, after which the planner replans from the newly observed state. This receding-horizon execution allows the same policy to continuously adapt its whole-body behavior to the perceived scene while following different steering conditions.

Through this pipeline, skills originating from heterogeneous acquisition procedures are converted into a common executable latent representation and subsequently learned by a single offline perceptive planner.

Experiments

We evaluate FIBER through simulation and real-world experiments to answer four questions:

Q1: How faithfully can heterogeneous interaction experts be projected into the shared latent space of a frozen BFM?

Q2: Can these projected skills be jointly learned by a single planner for versatile humanoid loco-manipulation?

Q3: Can the unified planner chain projected skills and flexibly steer their execution through various conditions?

Q4: Which components of FIBER are critical for reliable latent projection and downstream planning?

For each projected skill, we collect 20K episodes, and merge them across tasks for FIBER training. Language conditions are assigned according to the underlying task, while waypoint conditions are labeled from the robot trajectories in the collected rollouts.

Simulation Evaluation

We evaluate in IsaacSim, with 128 variations per task: box size and initial placement for pickup, door geometry for opening, and object identity for grasping. We do not report on-policy multi-expert distillation because the experts use different environments, observation spaces, and codebases, and therefore do not share the online rollout interface required for joint distillation.

a) Action-aligned latent projection: To answer Q1, we compare each task-specific expert with its LPP projector in Tab. I. LPP largely preserves expert performance on box pickup and door opening, while fine-grained grasping is more challenging, dropping from 97.0% to 52.9%.

We further compare representations using targets derived from the same successful expert trajectories. Reference-tracking baselines, including the BFM’s own backward representation B(s)B(s), perform poorly, whereas action-aligned projected replay achieves substantially higher success on box pickup and door opening. Notably, FALCON [21], despite learning force-adaptive end-effector tracking under randomized external forces, achieves only 0.4%/2.7% on door opening. These results show that kinematic, behavioral, or generic force-adaptive tracking alone does not recover the task-specific control responses required for interaction, motivating action-aligned latent projection.

b) Unified perceptive planning: To answer Q2, we compare replaying LPP-produced latent trajectories with the unified perceptive planner. FIBER improves mean success from 11.6% to 42.0%, showing that high-frequency BFM feedback alone is insufficient and that task-level latent replanning remains important. Under the same perceptive planning setup, the Action Planner fails on all tasks, while the Kinematic Planner reaches only 3.5% mean success. Directly planning low-level actions removes high-frequency state feedback between planner updates, whereas predicting kinematic targets does not specify the interaction-specific control response, which leads to failure on contact-rich tasks; FIBER retains both through latent planning over BFM.

c) Skill chaining and conditional steering: To answer Q3, we further test whether the unified planner can compose projected skills under changing high-level commands. As shown in Figure 3, the same policy sequentially opens the door, picks up the box, and carries it back through the doorway within a single rollout, demonstrating closed-loop chaining of heterogeneous interaction skills.

Skill chaining with FIBER

Figure 3. Skill chaining with FIBER. In a single rollout, the policy sequentially opens the door in (i), picks up the box in (ii), and carries it back through the doorway in (iii), demonstrating closed-loop chaining of multiple interaction skills under changing high-level commands.

Real-World Evaluation

We further deploy FIBER on a Unitree G1 equipped with an onboard RealSense depth camera to evaluate real-world transfer of the unified planner (Q2) and interaction chaining and steering (Q3). Both the planner and AH-BFM run on a single NVIDIA RTX PRO 5000 Blackwell laptop. With TensorRT, the planner runs at 5 Hz using 10 Heun steps and Real Time Chunking (RTC) (~20 ms per inference), while AH-BFM runs at 50 Hz (∼\sim1 ms per forward pass).

Tab. II reports success rates for box pickup, door opening, and object grasping. All experts, latent projectors, and the unified planner are trained entirely in simulation. FIBER succeeds in 6/10 box-pickup trials and 4/10 door-opening trials, demonstrating direct sim-to-real transfer of the latent planning interface. Object grasping is more challenging (2/10), where inaccurate depth estimates for the small object compound the fine-grained control difficulty. Beyond task success, Figure 4 highlights the key advantage of our representation: planned latents remain grounded in closed-loop physical responses. When the box becomes unstable, the robot repositions its hands underneath for support in (a); when opening a heavy door, it leans its whole body into the interaction to generate sufficient force in (c).

Qualitative real-world behaviors of FIBER

Figure 4. Qualitative real-world behaviors of FIBER. (a) During box carrying, the robot reacts to an unstable grasp by briefly lifting the box and repositioning its hands underneath for more secure support, illustrating high-frequency closed-loop adjustment. (b) After picking up the box, the same policy follows waypoint commands while carrying it; after stumbling, the robot stabilizes on its knees and recovers before continuing, demonstrating interaction-preserving skill chaining, steering, and recovery. (c) For a heavy door, the robot leans its whole body into the interaction to generate sufficient pushing force, showing adaptive whole-body realization of contact-rich tasks.

To answer Q3, we further evaluate post-pickup behavior chaining and waypoint steering. After picking up a box, the same planner follows root-waypoint commands while maintaining the grasp; when disturbed during carrying, it can stabilize on its knees, recover, and continue execution, as shown in Figure 4 (b). These results show that the unified planner can chain interaction and locomotion behaviors while remaining steerable through high-level waypoint commands.

Ablation and Analysis

To answer Q4, we analyze the two components underlying our latent planning interface: the adaptive-horizon BFM and the action-aligned latent representation.

MethodPickup BoxOpen DoorGrasp ObjectMean
FIBER6/104/102/1040%

Table 2. Real-world success rates of the unified FIBER planner on the Unitree G1.

MethodJoint-MAE ↓\downarrowBE-EMD ↓\downarrow
SONIC [34]0.1315––––
BFM γ0.8\gamma_{0.8}0.11070.0380.2000.4950.792
BFM γ0.98\gamma_{0.98}0.13270.0480.2130.4510.601
AH-BFM0.10200.0490.1950.4220.590

Table 3. Ablation of multi-horizon BFM tracking and behavior embedding performance on 100STYLE [36].

a) Effect of the adaptive-horizon BFM: We first evaluate how the BFM temporal horizon affects both behavior representation and downstream projection.

For behavior representation, we evaluate AH-BFM using two complementary metrics. Joint-MAE measures instantaneous motion-tracking accuracy, while BE-EMD measures how well a fixed latent command preserves a reference behavior over increasing embedding temporal horizons: for each reference clip, we infer a single latent through LPP, roll out the BFM under this fixed command, and measure its discrepancy from the reference.

As shown in Tab. III, the short-horizon model γ=0.8\gamma= 0.8 achieves the lowest BE-EMD at 0.2 s, whereas γ=0.98\gamma= 0.98 better preserves behavior over longer horizons. AH-BFM achieves the lowest Joint-MAE and the lowest BE-EMD at 0.5, 1, and 2 s. Adaptive multi-γ\gamma learning therefore provides a better trade-off between precise local control and temporally extended behavioral structure than either fixed horizon alone.

This advantage also transfers to interaction projection: in Tab. I, replacing the fixed-γ\gamma BFM (γ=0.98\gamma= 0.98) with AH-BFM increases mean LPP success from 68.8% to 72.8%, with substantial gains on box pickup and door opening.

**b) Effect of action-aligned latent representation: ** The reference-prompted results in Tab. I isolate the importance of action alignment. B(s)B(s) tracking and projected replay are derived from the same successful expert trajectories, but the former identifies behavior from the expert state trajectory whereas LPP identifies latent commands from the expert’s action responses. Their large performance difference on pickup box and door opening shows that behavioral-state matching alone does not retain interaction-critical feedback. At the other extreme, the Action Planner fails in all tasks, showing that low-rate action planning cannot replace the high-frequency state feedback provided by the frozen BFM.

The gap between the feedback-conditioned LPP projector and fixed projected replay further shows that projected behaviors cannot generally be reduced to open-loop latent trajectories. Conversely, the improvement from projected replay to FIBER shows that a unified perceptive planner can recover task-level latent feedback from onboard observations. Together, these comparisons separate the roles of action-aligned representation, latent feedback, and perceptive replanning in FIBER.

Conclusions and Limitations

We presented FIBER, an action-aligned latent planning framework for versatile humanoid loco-manipulation. LPP projects heterogeneous interaction experts into a frozen BFM by matching their short-horizon control responses, producing executable latent trajectories for unified offline perceptive planning. Adaptive multi-γ\gamma FB learning supports both precise control and temporally extended behavior, while successor-feature-guided learning enables interaction-preserving behavior reuse. Our simulation and real-world experiments demonstrate effective expert projection and perception-conditioned execution across diverse interaction tasks.

While our approach enables versatile humanoid loco-manipulation through action-aligned latent planning, extending the underlying BFM toward more dynamic whole-body behaviors, such as stair traversal, remains worth exploring. Moreover, incorporating richer visual and tactile feedback together with online latent adaptation could support interaction under more diverse contact conditions and facilitate the continual integration of new skills. We hope our work provides insights into scalable humanoid skill acquisition and opens new directions for unified whole-body planning.

Appendix I: Adaptive-Horizon Behavioral Foundation Model

This appendix follows the order of the methodology in the main paper. We first detail the adaptive-horizon BFM that defines the shared executable space, then the three routes for skill acquisition, action-aligned latent policy projection, and finally unified offline latent planning.

The same BFM latent space serves two roles in FIBER: it is the executable interface used to project interaction experts and the prediction space of the unified planner. Projection favors locally precise responses, while latent planning benefits from commands with longer temporal coherence. The adaptive-horizon BFM is designed to support both requirements within one shared behavioral space.

Executable Behavioral Interface and BFM Observations

The body controller commands the 29 actuated joints of the Unitree G1 at 50 Hz, with control interval Δt=0.02\Delta t = 0.02 s. Tasks using Dex3 hands additionally predict 14 hand-joint targets, seven per hand. These hand targets are produced by a separate branch and are not part of the BFM body action.

Let dzd_z denote the BFM latent dimension. Executable body commands lie on

Z={z∈Rdz:∥z∥2=dz},ΠZ(v)=dzv∥v∥2,(18)\mathcal{Z} = \{z \in\mathbb{R}^{d_z} : \lVert z\rVert_2 = \sqrt{d_z}\}, \qquad\Pi_{\mathcal{Z}}(v) = \sqrt{d_z}\frac{v}{\lVert v\rVert_2}, \tag*{(18)}

with a fixed nonzero reference vector used only for the degenerate zero-norm case. We use dz=256d_z = 256. The fixed-radius parameterization keeps the average squared coordinate magnitude independent of the latent dimension.

The BFM actor consumes the current robot state together with causal history. Its current proprioceptive state is

stprop=[qt−qdefault;q˙t;gtbody;0.25ωtbody]∈R64,(19)s_t^{\mathrm{prop}} = \left[q_t - q^{\mathrm{default}}; \dot{q}_t; g_t^{\mathrm{body}}; 0.25\omega_t^{\mathrm{body}}\right] \in\mathbb{R}^{64}, \tag*{(19)}

where gtbodyg_t^{\mathrm{body}} is the gravity direction and ωtbody\omega_t^{\mathrm{body}} is the body angular velocity. The actor also receives the previous scaled body action and nine causal proprioceptive/action history frames. Each history frame contains 93 entries: 29 action entries, three angular velocities, 29 joint positions, 29 joint velocities, and three gravity entries. The actor therefore receives 64+29+9×93=93064 + 29 + 9 \times93 = 930 proprioceptive/action coordinates before the latent command is appended.

The backward representation additionally receives 463 dimensions of heading-frame whole-body kinematics during BFM training and simulation-based projection. These privileged quantities are not consumed by the deployed BFM actor. Domain-randomization variables and external forces may likewise enter training-time representations and critics without entering the deployed policy.

Multi-Horizon Forward–Backward Representation

For a latent-conditioned policy, the forward representation admits the successor-feature interpretation

Fγ(st,at,z)≈E[∑h=0∞γhB(st+h+1)∣st,at,πBFM(⋅,z)].(20)F_\gamma(s_t, a_t, z) \approx\mathbb{E}\left[\left.\sum_{h=0}^{\infty}\gamma^h B(s_{t+h+1})\right|s_t, a_t, \pi_{\mathrm{BFM}}(\cdot,z)\right]. \tag*{(20)}

Multi-Horizon Forward–Backward Representation

The corresponding FB value is QγFB(s,a,z)=Fγ(s,a,z)⊤zQ_\gamma^{\mathrm{FB}}(s,a,z) = F_\gamma(s,a,z)^\top z. We condition the forward representation on γ\gamma while sharing both the backward map BB and the executable actor across horizons. For every replay transition, a horizon is sampled from the configured set Γ\Gamma, and the same γ\gamma is used in the online prediction and bootstrapped target. The output of BB is projected to Z\mathcal{Z}.

For a minibatch of nn transitions (si,ai,si+,di)(s_i,a_i,s_i^+,d_i), with termination indicator did_i, define

Mij=Fγi(si,ai,zi)⊤B(sj+),(21)M_{ij} = F_{\gamma_i}(s_i,a_i,z_i)^\top B(s_j^+), \tag*{(21)}
Rij=Mij−γi(1−di)Mˉij,(22)R_{ij} = M_{ij} - \gamma_i(1-d_i)\bar{M}_{ij}, \tag*{(22)}

where Mˉij\bar{M}_{ij} is a stopped-gradient target constructed from target representations and a next action sampled from the latent-conditioned actor. Suppressing ensemble notation, the FB minibatch objective is

LFB=∑i≠jRij22n(n−1)−1n∑iRii.(23)\mathcal{L}_{\mathrm{FB}} = \frac{\sum_{i\ne j} R_{ij}^{2}}{2n(n-1)} - \frac{1}{n}\sum_i R_{ii}. \tag*{(23)}

The shared backward map is regularized through

Lorth=∑i≠j(bi⊤bj)22n(n−1)−1n∑i∥bi∥22,bi=B(si+),(24)\mathcal{L}_{\mathrm{orth}} = \frac{\sum_{i\ne j}(b_i^\top b_j)^2}{2n(n-1)} - \frac{1}{n}\sum_i \lVert b_i\rVert_2^2,\qquad b_i = B(s_i^+), \tag*{(24)}

and the representation update minimizes LFB+λorthLorth\mathcal{L}_{\mathrm{FB}} + \lambda_{\mathrm{orth}}\mathcal{L}_{\mathrm{orth}}. Motion-discriminator and auxiliary-critic objectives follow the underlying BFM training procedure; the multi-horizon modification applies to the FB representation and the shared actor objective below.

Adaptive Actor Objective

Values at different discounts have different natural scales. We therefore use the normalized horizon-specific objective

Vγ(s,z)=(1−γ)QγFB(s,πBFM(s,z),z).(25)V_\gamma(s,z) = (1-\gamma)Q_\gamma^{\mathrm{FB}}\left(s,\pi_{\mathrm{BFM}}(s,z),z\right). \tag*{(25)}

The factor (1−γ)(1-\gamma) removes the leading discounted-sum scale, making the short- and long-horizon terms directly comparable. The shared actor maximizes

JFB(s,z)=τlog⁡∑γ∈Γp(γ)exp⁡(Vγ(s,z)τ),(26)J_{\mathrm{FB}}(s,z) = \tau\log\sum_{\gamma\in\Gamma} p(\gamma)\exp\left(\frac{V_\gamma(s,z)}{\tau}\right), \tag*{(26)}

where p(γ)≥0p(\gamma) \ge0, ∑γp(γ)=1\sum_\gamma p(\gamma) = 1, and τ>0\tau> 0. Actor gradient is

∇θJFB=∑γ∈Γwγ(s,z)∇θVγ(s,z),(27)\nabla_\theta J_{\mathrm{FB}} = \sum_{\gamma\in\Gamma} w_\gamma(s,z)\nabla_\theta V_\gamma(s,z), \tag*{(27)}
wγ(s,z)=p(γ)exp⁡(Vγ(s,z)/τ)∑γ′∈Γp(γ′)exp⁡(Vγ′(s,z)/τ).(28)w_\gamma(s,z) = \frac{p(\gamma)\exp(V_\gamma(s,z)/\tau)}{\sum_{\gamma'\in\Gamma}p(\gamma')\exp(V_{\gamma'}(s,z)/\tau)}. \tag*{(28)}

Thus the update remains a convex combination of the horizon-specific policy gradients, but the combination is selected from the current state and latent command rather than fixed globally.

a) Stability of the adaptive FB-to-actor objective: The relevant stability property is not only that the horizon aggregator is non-expansive, but that it does not amplify perturbations introduced through the FB backup. Let

Q~γ=(1−γ)QγFB,(29)\widetilde{Q}_{\gamma}=(1-\gamma)Q^{\mathrm{FB}}_{\gamma}, \tag*{(29)}

and collect the normalized action values as Q~=[Q~γ]γ∈Γ\widetilde{Q}=[\widetilde{Q}_{\gamma}]_{\gamma\in\Gamma}. Holding the current backward map BB fixed, let rzBr^{B}_{z} denote the corresponding latent-conditioned reward. For a fixed latent policy π\pi, the normalized FB Bellman operator is

(BπQ~)γ(s,a,z)=E[(1−γ)rzB(s′)+γ(1−d)Q~γ(s′,π(s′,z),z)∣s,a].(30)(\mathcal{B}^{\pi}\widetilde{Q})_{\gamma}(s,a,z) =\mathbb{E}\left[(1-\gamma)r^{B}_{z}(s')+\gamma(1-d)\widetilde{Q}_{\gamma}(s',\pi(s',z),z)\mid s,a\right]. \tag*{(30)}

Equip the product space with the norm

∥Q~∥Γ,∞=max⁡γ∈Γ∥Q~γ∥∞.(31)\|\widetilde{Q}\|_{\Gamma,\infty}=\max_{\gamma\in\Gamma}\|\widetilde{Q}_{\gamma}\|_{\infty}. \tag*{(31)}

For any two value collections Q~\widetilde{Q} and Q~′\widetilde{Q}', the reward terms cancel, yielding

∥BπQ~−BπQ~′∥Γ,∞≤γmax⁡∥Q~−Q~′∥Γ,∞,(32)\|\mathcal{B}^{\pi}\widetilde{Q}-\mathcal{B}^{\pi}\widetilde{Q}'\|_{\Gamma,\infty}\leq\gamma_{\max}\|\widetilde{Q}-\widetilde{Q}'\|_{\Gamma,\infty}, \tag*{(32)}

where γmax⁡=max⁡Γ<1\gamma_{\max}=\max\Gamma<1. Thus, for fixed BB and π\pi, the multi-horizon FB Bellman backup is a γmax⁡\gamma_{\max}-contraction.

We next define the adaptive actor-value functional

(AτπQ~)(s,z)=τlog⁡∑γ∈Γp(γ)exp⁡(Q~γ(s,π(s,z),z)τ).(33)(\mathcal{A}^{\pi}_{\tau}\widetilde{Q})(s,z)=\tau\log\sum_{\gamma\in\Gamma}p(\gamma)\exp\left(\frac{\widetilde{Q}_{\gamma}(s,\pi(s,z),z)}{\tau}\right). \tag*{(33)}

The log-sum-exp map is monotone and translation equivariant. Let

δ=∥Q~−Q~′∥Γ,∞.(34)\delta=\|\widetilde{Q}-\widetilde{Q}'\|_{\Gamma,\infty}. \tag*{(34)}

Then, for every (s,a,z)(s,a,z) and γ∈Γ\gamma\in\Gamma,

Q~γ′(s,a,z)−δ≤Q~γ(s,a,z)≤Q~γ′(s,a,z)+δ.(35)\widetilde{Q}'_{\gamma}(s,a,z)-\delta\leq\widetilde{Q}_{\gamma}(s,a,z)\leq\widetilde{Q}'_{\gamma}(s,a,z)+\delta. \tag*{(35)}

Monotonicity and translation equivariance therefore imply

∥AτπQ~−AτπQ~′∥∞≤∥Q~−Q~′∥Γ,∞.(36)\|\mathcal{A}^{\pi}_{\tau}\widetilde{Q}-\mathcal{A}^{\pi}_{\tau}\widetilde{Q}'\|_{\infty}\leq\|\widetilde{Q}-\widetilde{Q}'\|_{\Gamma,\infty}. \tag*{(36)}

Hence the adaptive actor-value functional is non-expansive with respect to perturbations of the horizon-specific FB values.

The same bound holds for the exact improved-value functional

(IτQ~)(s,z)=max⁡aτlog⁡∑γ∈Γp(γ)exp⁡(Q~γ(s,a,z)τ),(37)(\mathcal{I}_{\tau}\widetilde{Q})(s,z)=\max_{a}\tau\log\sum_{\gamma\in\Gamma}p(\gamma)\exp\left(\frac{\widetilde{Q}_{\gamma}(s,a,z)}{\tau}\right), \tag*{(37)}

since

∣max⁡af(a)−max⁡ag(a)∣≤max⁡a∣f(a)−g(a)∣.(38)\left|\max_{a}f(a)-\max_{a}g(a)\right|\leq\max_{a}|f(a)-g(a)|. \tag*{(38)}

This statement concerns the improved value functional rather than the arg max policy map itself.

Combining (32) and (36) gives

∥AτπBπQ~−AτπBπQ~′∥∞≤γmax⁡∥Q~−Q~′∥Γ,∞.(39)\|\mathcal{A}^{\pi}_{\tau}\mathcal{B}^{\pi}\widetilde{Q}-\mathcal{A}^{\pi}_{\tau}\mathcal{B}^{\pi}\widetilde{Q}'\|_{\infty}\leq\gamma_{\max}\|\widetilde{Q}-\widetilde{Q}'\|_{\Gamma,\infty}. \tag*{(39)}

Therefore, adaptive horizon aggregation preserves the contractive sensitivity of the fixed-policy FB backup: perturbations in the horizon-specific FB values are not amplified by the joint actor objective.

ComponentHidden widthHidden layers
Body actor20486
Forward representation20486
Critics20486
Backward encoder10241
Motion discriminator10243
Executable latent dimension256
Body action dimension29
Forward/critic ensemble size2

Table IV. BFM architecture used in our implementation.

Consequently, relative to a single-horizon objective with γ=γmax⁡\gamma=\gamma_{\max}, adaptive aggregation introduces no worse worst-case sensitivity to critic perturbations while allowing the shared policy to exploit value estimates at multiple temporal horizons.

b) Adaptive aggregation versus a fixed weighted sum: A natural multi-γ\gamma alternative is the fixed convex average

JWS(s,z)=∑γ∈Γp(γ)Vγ(s,z).(40)J_{\mathrm{WS}}(s,z)=\sum_{\gamma\in\Gamma}p(\gamma)V_{\gamma}(s,z). \tag*{(40)}

Both JWSJ_{\mathrm{WS}} and (26) are scale controlled after the (1−γ)(1-\gamma) normalization; non-expansiveness alone therefore does not distinguish the two. The key difference is that the weighted-sum baseline uses the same horizon weights for every state and behavior, whereas (26) adapts the effective weights to the current horizon-specific values. As τ→0\tau\to0, the update concentrates on the largest normalized value; for large τ\tau, its gradient approaches the prior-weighted average. To isolate this effect from simply exposing the actor to multiple discounts, we additionally compare against (40) using the same multi-horizon representation. The adaptive objective empirically outperforms this fixed weighted-sum baseline, supporting state- and behavior-dependent horizon weighting rather than a fixed global mixture.

For intuition, a discount γ\gamma at control interval Δt\Delta t has effective duration approximately Δt/(1−γ)\Delta t/(1-\gamma). At 50 Hz, γ=0.8\gamma=0.8 corresponds to approximately 0.10.1 s and γ=0.98\gamma=0.98 to approximately 11 s. This is an interpretation of the temporal weighting, not a hard execution horizon for the latent command.

Architecture and Optimization

The actor, forward representations, and critics use residual MLP backbones. The forward representations and critics use two ensemble members; the actor is a single shared network conditioned on the current latent and outputs 29 body actions. The backward encoder maps proprioceptive and privileged kinematic features to the normalized behavioral space. The scalar discount is embedded by a small MLP before entering the forward representation and is not an input to the actor. Details are shown in Tab. IV.

The BFM is frozen during expert projection, projected-data collection, and offline planner training. These stages therefore use the same latent geometry, state convention, and body-action decoder.

ParameterValue
Forward-map learning rate3×10−43 \times10^{-4}
Backward-map learning rate10−510^{-5}
Actor / critic learning rate3×10−43 \times10^{-4}
Discriminator learning rate10−510^{-5}
Minibatch size1024
Training updates100,000
Horizon set Γ\Gamma{0.8,0.98}\{0.8, 0.98\}
Horizon prior p(0.8)/p(0.98)p(0.8)/p(0.98)0.625/0.375
Aggregation temperature τ\tau1
Replay capacity5.12×1065.12 \times10^{6}
FB target update rate ρ\rho0.01
Critic target update rate ρ\rho0.005
Backward orthogonality coefficient100

Table V. BFM optimization parameters. Target networks use Polyak updates θˉ←(1−ρ)θˉ+ρθ\bar{\theta} \leftarrow(1-\rho)\bar{\theta}+\rho\theta.

Appendix II: Heterogeneous Skill Acquisition

As in the main paper, skill acquisition is deliberately decoupled from the shared representation. Experts may use different supervision, observations, and training procedures; projection only requires access to the resulting robot actions. We use motion-based and reward-shaped experts for the base interactions, and SF-guided behavior reuse when we specifically want to create diverse compositions between an interaction and native BFM behaviors.

Task-Specific Interaction Experts

Motion-based experts combine humanoid motion tracking with object and contact objectives. For box pickup and object grasping, the tracked quantities include both body motion and manipulated-object pose, while contact rewards encourage the demonstrated interaction to be realized physically. The door expert is trained with stage-specific objectives [2] for approach, contact establishment, handle manipulation, and opening. These experts may use different privileged observations and rewards; none of those task-specific inputs are required by the unified planner after projection.

For reproducibility, the HDMI-style motion expert uses a privileged observation encoder with a 256-dimensional embedding and a reference-action residual actor. Its actor hidden widths are [512,256,256][512, 256, 256], critic hidden widths are [512,256,128][512, 256, 128], and the activation is Mish. The PPO settings for this route are summarized in Table VI.

ParameterMotion expertSF adapter
Rollout steps / environment3232
Learning rate3×10−43 \times10^{-4}10−510^{-5}
Optimization epochs35
Minibatches / rollout84
PPO clipping parameter0.20.2
Discount0.990.99
GAE parameter0.950.95
Entropy coefficient0.0010
Gradient-norm limit11

Table VI. PPO parameters for the motion expert and the SF-guided composition adapter.

SF-Guided Behavior Reuse for Skill Composition

The third acquisition route targets interaction–behavior composition. Given an acquired interaction, we use behaviors already represented by the BFM to create additional physically valid variants: the interaction reward preserves the task-specific physical requirements, while a native BFM latent specifies the whole-body behavior to be expressed simultaneously. This expands one interaction into a family of compositions without requiring a separate motion reference or task-specific expert for every combination.

In our experiments, we instantiate this idea with box carrying. By varying the requested native latent, the same carrying interaction can be composed with crouching, kneeling, walking, or running. For example, carrying while crouching requires the controller to maintain support of the box while realizing a substantially different whole-body configuration. The resulting rollouts add diverse compositional behavior to the downstream planner data.

For an interaction II, let zN∈Zz^{N} \in\mathcal{Z} denote the native BFM behavior to be composed with it. We freeze the BFM and learn an adapter AψIA_{\psi_I} that outputs an executable latent,

z^t=ΠZ(zN+AψIres(ot,zN)),at=πBFM(st,z^t).(41)\hat{z}_t = \Pi_{\mathcal{Z}}\left(z^N + A_{\psi_I}^{\mathrm{res}}(o_t,z^N)\right), \qquad a_t = \pi_{\mathrm{BFM}}(s_t,\hat{z}_t). \tag*{(41)}

The residual parameterization lets the adapter modify the native command only as needed to satisfy the interaction. The deployable adapter uses causal robot inputs and the requested native latent; privileged object state and interaction errors may be used by the critic and reward computation during simulation training.

We optimize the adapter with

rt=rtI+λSFB(st)⊤zN,(42)r_t = r_t^I + \lambda_{\mathrm{SF}} B(s_t)^\top z^N, \tag*{(42)}

where rtIr_t^I preserves the interaction and the SF term encourages the rollout to remain aligned with the requested native behavior. For fixed zNz^N,

E[∑t=0∞γtB(st)⊤zN]=E[∑t=0∞γtB(st)]⊤zN,(43)\mathbb{E}\left[\sum_{t=0}^{\infty}\gamma^t B(s_t)^\top z^N\right] = \mathbb{E}\left[\sum_{t=0}^{\infty}\gamma^t B(s_t)\right]^\top z^N, \tag*{(43)}

so the preservation term evaluates the discounted state occupancy along the behavioral direction selected by zNz^N. The two terms therefore play distinct roles: rtIr_t^I keeps the interaction valid, while the SF term determines which native behavior is composed with that interaction.

For the carrying compositions used in our experiments, rtIr_t^I contains upper-body pose, object-position, object-orientation, and contact terms. The three tracking terms are

rpose=exp⁡(−eq/0.252),(44)r_{\mathrm{pose}} = \exp\left(-e_q/0.25^2\right), \tag*{(44)}
rpos=exp⁡(−∥ep∥22/0.12),(45)r_{\mathrm{pos}} = \exp\left(-\left\lVert e_p\right\rVert_2^2/0.1^2\right), \tag*{(45)}
rori=exp⁡(−θobj2/0.352),(46)r_{\mathrm{ori}} = \exp\left(-\theta_{\mathrm{obj}}^2/0.35^2\right), \tag*{(46)}

with weights 1, 2, and 1, respectively; the contact term has weight 1. Here eqe_q is the mean squared upper-body jointposition error, epe_p is the object-position error in the torso frame, and θobj\theta_{\mathrm{obj}} is the object-orientation discrepancy in that frame. The SF term has weight 0.1, the latent-smoothness penalty has weight −0.5-0.5, and object-drop termination contributes −100-100. These interaction terms are shared across the composed variants; behavioral diversity is introduced by changing zNz^N.

Because both B(st)B(s_t) and zNz^N lie in the radius dz\sqrt{d_z} sphere,

B(st)⊤zNdz=cos⁡∠(B(st),zN).(47)\frac{B(s_t)^\top z^N}{d_z}=\cos\angle\left(B(s_t),z^N\right). \tag*{(47)}

The normalized and unnormalized forms therefore differ only by a constant scale absorbed into λSF\lambda_{\mathrm{SF}}. Importantly, the preservation target is the requested native behavior zNz^N, not the adapted latent z^t\hat{z}_t.

The adapter actor and critic use two residual feed-forward blocks of width 512 with feed-forward width 2048, GELU activations, and observation normalization. We additionally regularize rapid latent changes with the normalized cost

csmooth,t=∥z^t−z^t−1∥22dz,(48)c_{\mathrm{smooth},t}=\frac{\left\lVert\hat{z}_t-\hat{z}_{t-1}\right\rVert^2}{2d_z}, \tag*{(48)}

which equals one minus cosine similarity on Z\mathcal{Z}.

Appendix III: Action-Aligned Latent Policy Projection

After each expert is acquired, LPP maps its local closed-loop responses into the executable BFM space. This section follows the same order as the main method: first establish the shared action convention, then define the projector and short-horizon latent execution, the action/semantic objectives, and finally the on-policy aggregation used to obtain robust projected rollouts.

Common Action Space

Experts may use different normalized action conventions, so action alignment is performed only after both expert and BFM outputs are converted to the same physical body joint-target space. The frozen BFM actor produces at=πBFM(st,zt)∈R29a_t=\pi_{\mathrm{BFM}}(s_t,z_t)\in\mathbb{R}^{29}. Under the BFM control profile,

qdes=qdefault+0.25rq⊙clip⁡(6at,−6,6),(49)q^{\mathrm{des}}=q^{\mathrm{default}}+0.25r_q\odot\operatorname{clip}(6a_t,-6,6), \tag*{(49)}

where rqr_q contains the per-joint action scales. Joint order, default pose, and scale vectors are kept consistent across expert conversion, projection, projected-rollout collection, and execution. The resulting targets are realized by the robot PD controller with the gains and actuator limits of the selected control profile.

Projector and Latent Execution

Each expert is projected independently. At update time tkt_k, a projector PψiP_{\psi_i} maps the expert observation to BFM latent,

zk=Pψi(otkEi),zk∈Z.(50)z_k=P_{\psi_i}\left(o_{t_k}^{E_i}\right),\qquad z_k\in\mathcal{Z}. \tag*{(50)}

The projector is a pre-normalized residual MLP consisting of an input LayerNorm and linear projection, residual feed-forward blocks, an output LayerNorm, and a latent head followed by ΠZ\Pi_{\mathcal{Z}}. A residual block is

h~ℓ=GELU⁡(W1ℓLN⁡(hℓ)+b1ℓ),(51)\tilde{h}^{\ell}=\operatorname{GELU}\left(W_1^{\ell}\operatorname{LN}(h^{\ell})+b_1^{\ell}\right), \tag*{(51)}
hℓ+1=hℓ+W2ℓh~ℓ+b2ℓ.(52)h^{\ell+1}=h^{\ell}+W_2^{\ell}\tilde{h}^{\ell}+b_2^{\ell}. \tag*{(52)}

Projector and Latent Execution

The hidden width is 1024 with feed-forward expansion ratio four. The motion-based projector uses four residual blocks and the door projector uses two.

Projection updates occur every five body-control steps, corresponding to a 0.1 s interval and a 10 Hz projector. Between two projector updates, the latent command is interpolated on the sphere,

ztk+h=SLERP⁡(zk−1,zk,αh),h=0,…,HP−1,(53)z_{t_k+h}=\operatorname{SLERP}(z_{k-1},z_k,\alpha_h),\qquad h=0,\ldots,H_P-1, \tag*{(53)}

where HP=5H_P=5. For endpoints a,b∈Za,b\in\mathcal{Z} with angular separation ϑ\vartheta,

SLERP⁡(a,b,α)=sin⁡((1−α)ϑ)sin⁡ϑa+sin⁡(αϑ)sin⁡ϑb.(54)\operatorname{SLERP}(a,b,\alpha)=\frac{\sin((1-\alpha)\vartheta)}{\sin\vartheta}a+\frac{\sin(\alpha\vartheta)}{\sin\vartheta}b. \tag*{(54)}

The cosine used to compute ϑ\vartheta is clamped for numerical stability.

Action Alignment and Semantic Regularization

Let TBT_B and TiT_i convert the BFM and expert outputs to the same physical body joint-target space. Over the short execution interval beginning at tkt_k, we use

eha=TB(πBFM(stk+h,ztk+h))−Ti(πiE(otk+hEi)),(55)e_h^a=T_B\left(\pi_{\mathrm{BFM}}(s_{t_k+h},z_{t_k+h})\right)-T_i\left(\pi_i^{E}\left(o_{t_k+h}^{E_i}\right)\right), \tag*{(55)}
Lact=EPψi[∑h=0HP−1mh∥eha∥2229∑h=0HP−1mh],(56)\mathcal{L}_{\mathrm{act}}=\mathbb{E}_{P_{\psi_i}}\left[\frac{\sum_{h=0}^{H_P-1}m_h\left\lVert e_h^a\right\rVert_2^2}{29\sum_{h=0}^{H_P-1}m_h}\right], \tag*{(56)}

where mhm_h masks samples beyond an episode boundary. Comparing physical joint targets prevents normalized-action conventions of different experts from changing the meaning of the alignment loss.

Action matching alone can leave the latent underdetermined. We therefore anchor the predicted latent to the native BFM representation of the same local state,

Lsem=E[∥z−B(st)∥22].(57)\mathcal{L}_{\mathrm{sem}}=\mathbb{E}\left[\left\lVert z-B(s_t)\right\rVert_2^2\right]. \tag*{(57)}

For two vectors on Z\mathcal{Z},

∥z−B(s)∥22=2dz(1−z⊤B(s)dz),(58)\left\lVert z-B(s)\right\rVert_2^2=2d_z\left(1-\frac{z^\top B(s)}{d_z}\right), \tag*{(58)}

so squared distance and cosine gap are equivalent up to a constant factor. The full objective is

Lproj=Lact+λsemLsem.(59)\mathcal{L}_{\mathrm{proj}}=\mathcal{L}_{\mathrm{act}}+\lambda_{\mathrm{sem}}\mathcal{L}_{\mathrm{sem}}. \tag*{(59)}

The action term identifies commands by the expert’s required local control response, whereas the semantic term resolves action-equivalent solutions in a way that remains coherent with the native BFM geometry. Training details in Tab. VII.

For skills with dexterous hands, a separate projector head regresses the expert’s physical hand-joint targets. Its loss is evaluated only when valid hand labels are available. The sphere constraint applies only to the body latent; hand outputs retain their physical joint-target parameterization.

ParameterValue
OptimizerAdamW
Learning rate3×10−43 \times10^{-4}
Weight decay10−410^{-4}
Gradient-norm limit1
Minibatches / rollout4
Optimization passes / rollout5
Semantic-loss weight λsem\lambda_{\mathrm{sem}}0.1
Body-control steps / latent interval HPH_P5
Executed latent noise std.0.1

Table VII. Latent-projector optimization parameters.

On-Policy Projection and Data Aggregation

Projection is trained with single-skill on-policy data aggregation. The current projector is executed through the frozen BFM under randomized task initializations. At the states it visits, we query the corresponding expert for body actions and record the BFM inputs, expert observations, and reset masks. Short execution windows are then replayed through the frozen BFM actor to evaluate (56) and (57). The projector is updated and new rollouts are collected with the updated policy. Gradients pass through the frozen BFM actor to the predicted latent, but no BFM parameter is changed and no simulator gradient is required. This DAgger-style procedure [35] exposes the projector to corrective states that are absent from nominal expert trajectories.

Exploration noise is applied to the executed latent before projection back to Z\mathcal{Z} and is never added to the expert action label. Once projection is complete, the original expert is no longer required for projected execution or planner-data collection.

The projected latent should be interpreted locally: it is identified by the control response induced by the frozen BFM on states visited during projection, rather than by assuming that the expert and BFM policies are globally identical. This is why projected closed-loop rollouts, rather than nominal expert trajectories alone, are used as the supervision for the unified planner.

Appendix IV: Unified Latent Flow Planning

Once each expert has been projected, its closed-loop latent rollouts can be combined because they now share the same executable representation. We train the unified planner entirely offline on these projected trajectories, matching the data construction, conditioning variables, and flow objective. Details of the planner network are shown in Table VIII.

Component or parameterConfigurationComponent or parameterConfiguration
Executable body latentdz=256d_z = 256Transformer width / latent blocks1024 / 16
Prediction horizon1 sAttention heads / per-head width8 / 128
Body / optional hand state58 / 14Feed-forward expansion / activation4 / GELU
Optional hand target14 per latent stepAttention / feed-forward dropout0 / 0.2
Depth image / patch shape144 × 256 / 12 × 16Conditioning registers8
Depth range / spatial tokens0.12–3.0 m / 192Flow-time embedding scale1000
Depth-encoder blocks / registers2 / 8Pooled language embedding768 (frozen T5-base)
Waypoint input10, at 0.5 and 1.0 sSteering blocks / output tokens2 / 8

Table VIII. Architecture of the unified latent flow planner.

Projected Rollout Dataset and Prediction Target

For each projected skill, we collect 20K closed-loop episodes and aggregate them for offline planner training. Each record contains onboard depth, proprioception, the high-level condition, and the action-aligned latent actually executed by the projected policy. Replacing these targets with B(s)B(s) would change the supervision from projected control responses to behavioral-state encoding and is therefore not equivalent to our data construction.

The planner predicts a one-second sequence of future BFM commands. We use one latent target every 0.1 s, corresponding to five 50 Hz BFM control steps, and predict H=10H = 10 latent commands per sequence. Matching the trajectory notation used in the main paper,

Zt=[zt,zt+1,…,zt+H−1]∈RH×dz,H=10,(60)Z_t = [z_t, z_{t+1}, \ldots, z_{t+H-1}] \in\mathbb{R}^{H \times d_z}, \qquad H = 10, \tag*{(60)}

Each future latent is represented by one temporal token in the transformer. For hand-bearing data, a separate target Yt∈RH×14Y_t \in\mathbb{R}^{H \times14} stores physical hand-joint targets in a fixed hand-major ordering. Hand targets are normalized with their own statistics and are not projected onto Z\mathcal{Z}.

Observation and Steering Conditioning

Unlike the task-specific experts, the unified planner uses only onboard observations. Its proprioceptive input is pt=[qt;q˙t]∈R58p_t = [q_t; \dot{q}_t] \in\mathbb{R}^{58}; for hand-bearing samples, the measured 14 hand-joint positions are included through a separate projection. Continuous features are standardized coordinate-wise using fixed training-set statistics, and the normalization statistics and joint ordering are stored with the model.

a) Depth: The planner processes a metric egocentric depth image of size 144×256144 \times256, clipped to the valid range 0.120.12–3.03.0 m. A strided convolution with 12×1612 \times16 kernel and stride produces a 12×1612 \times16 grid of 192 spatial tokens. Learned spatial embeddings are added before two self-attention blocks. Eight learned register tokens participate in the depth encoder and are discarded afterward, leaving the 192 spatial tokens as conditioning memory.

b) Proprioception: The 58-dimensional joint state is standardized coordinate-wise and linearly embedded to the model width. For coordinate jj, pˉj=(pj−μj)/max⁡(σj,ϵ)\bar{p}_j = (p_j - \mu_j)/\max(\sigma_j,\epsilon). When hands are enabled, their measured joint positions are standardized separately and added through an additional projection.

c) Language: Task instructions are encoded by a frozen T5-base text encoder. Masked pooling produces a 768-dimensional sentence embedding, followed by LayerNorm and a two-layer SiLU MLP that maps it to the planner width. The text encoder is not updated by the planner objective.

d) Waypoint: Waypoint supervision is derived from future root poses. At horizons δ∈{0.5,1.0}\delta\in\{0.5, 1.0\} s, we use

utδ=[Δxtδ,Δytδ,sin⁡Δψtδ,cos⁡Δψtδ,ht+δ],(61)u_t^\delta= [\Delta x_t^\delta, \Delta y_t^\delta, \sin\Delta\psi_t^\delta, \cos\Delta\psi_t^\delta, h_{t+\delta}], \tag*{(61)}

where displacement is expressed in the current heading frame, Δψ\Delta\psi is relative yaw, and hh is future pelvis height. Concatenating the two horizons gives a ten-dimensional waypoint condition. Missing waypoint conditions are masked rather than represented by a zero-valued physical command.

e) Steering representation: Language and waypoint embeddings are converted to a small fixed set of steering tokens by a self-attention encoder. The latent transformer cross-attends to these steering tokens, the 192 spatial depth tokens, one proprioceptive state token, and eight learned conditioning-register tokens. Unavailable language or waypoint modalities are masked explicitly.

Conditional Latent Transformer

Each noisy future latent is linearly embedded and combined with a learned temporal-position embedding. If hand targets are present, a separate projection of the noisy hand target is added to the corresponding temporal token. Flow time is sinusoidally embedded and passed through an MLP. Each transformer block contains temporal self-attention, cross-attention to the observation/steering memory, and a feed-forward MLP. Adaptive LayerNorm modulation supplies shift, scale, and residual-gate vectors from the flow-time condition. For example,

h′=h+gτ⊙Attn⁡((1+sτ)⊙LN⁡(h)+bτ).(62)h^\prime= h + g_\tau\odot\operatorname{Attn}\left((1+s_\tau)\odot\operatorname{LN}(h)+b_\tau\right). \tag*{(62)}

Separate linear heads decode the clean body latents and optional hand targets.

ParameterValue
OptimizerAdamW
Learning rate2×10−42 \times10^{-4}
Weight decay10−210^{-2}
Warm-up updates100
Post-warm-up scheduleConstant
Batch size256
Training epochs500
Gradient-norm limit1
Body endpoint-loss weight1
Hand endpoint-loss weight1
Decoded body-action loss weight0
Default windows / worker16
Validation fraction0.2
Flow-time distributionU(0,1)\mathcal{U}(0,1)
Noise distributionN(0,I)\mathcal{N}(0,I)

Table IX. Offline planner optimization parameters.

Offline Flow-Matching Objective

Let ξt\xi_t denote the encoded onboard observations and available steering conditions. We sample x0∼N(0,I)x_0 \sim\mathcal{N}(0,I) and τ∼U(0,1)\tau\sim\mathcal{U}(0,1), and construct

xτ=(1−τ)x0+τx1.(63)x_\tau= (1-\tau)x_0+\tau x_1. \tag*{(63)}

The network predicts the clean endpoint fη(xτ,τ,ξt)f_\eta(x_\tau,\tau,\xi_t) and is trained with

Lbody=E[1Hdz∥fη(xτ,τ,ξt)−x1∥22].(64)\mathcal{L}_{\mathrm{body}}=\mathbb{E}\left[\frac{1}{Hd_z}\left\|f_\eta(x_\tau,\tau,\xi_t)-x_1\right\|_2^2\right]. \tag*{(64)}

The clean latent targets lie on Z\mathcal{Z}, but the Gaussian interpolation and raw endpoint prediction are carried out in Euclidean coordinates. Sphere projection is applied only when sampled outputs are converted to executable BFM commands.

An analogous Gaussian interpolation is applied to the hand targets when available. The complete loss is

Lplanner=Lbody+λhandLhand,(65)\mathcal{L}_{\mathrm{planner}}=\mathcal{L}_{\mathrm{body}}+\lambda_{\mathrm{hand}}\mathcal{L}_{\mathrm{hand}}, \tag*{(65)}

where Lhand\mathcal{L}_{\mathrm{hand}} is evaluated only on valid hand samples. No decoded body-action auxiliary loss is used in the reported planner training; action alignment is already incorporated when constructing the projected latent supervision.

An endpoint predictor induces the velocity field

vη(xτ,τ,ξt)=fη(xτ,τ,ξt)−xτ1−τ,τ<1.(66)v_\eta(x_\tau,\tau,\xi_t)=\frac{f_\eta(x_\tau,\tau,\xi_t)-x_\tau}{1-\tau},\qquad\tau<1. \tag*{(66)}

For a numerical grid 0=τ0<⋯<τN=10=\tau_0<\cdots<\tau_N=1, Euler integration uses

xj+1=xj+(τj+1−τj)vη(xj,τj,ξt).(67)x_{j+1}=x_j+(\tau_{j+1}-\tau_j)v_\eta(x_j,\tau_j,\xi_t). \tag*{(67)}

Heun integration uses the corresponding predictor-corrector update. After sampling, every body latent is independently projected by ΠZ\Pi_{\mathcal{Z}}, while hand outputs are denormalized using their own statistics.

Planner Optimization

Shown in Tab. IX, training windows are sampled within episode boundaries, and train/validation partitions are defined at the episode level. The model checkpoint stores the architecture, optimizer state, proprioceptive normalization, and hand-target normalization. In the exported configuration, state-condition dropout is 0.2 and waypoint-condition dropout is 0.1; waypoint tokens are omitted for episodes without waypoint supervision.

Receding-Horizon Execution

At deployment, the planner executes the leading segment of the sampled latent sequence and then replans from the new onboard observation. Consecutive latent updates are connected by SLERP to supply the BFM with a command at each 50 Hz body-control step. The BFM actor therefore retains high-frequency state-dependent feedback while the planner operates at a lower rate. Hand joint targets follow the same receding-horizon schedule as the corresponding body-latent sequence.

Appendix V: Evaluation and Real-World Deployment

Simulation Evaluation Protocol

We evaluate each task over diverse object or environment variations in IsaacSim. For pickup and door opening, the w/ appr. setting initializes the humanoid approximately 0.5–1.5 m from the interaction target and requires it to approach before execution, whereas w/o appr. initializes the humanoid directly in front of the target. Pickup succeeds if the box is lifted by at least 0.3 m and maintained for at least 1 s; grasping succeeds if the object is lifted by at least 0.1 m and maintained for at least 1 s. Door opening succeeds when the door is opened and the humanoid passes through the doorway. Reported ±\pm values are computed from repeated evaluations with different random seeds to account for simulation stochasticity. Detailed object variation configurations used in evaluation are shown in Table X.

TaskVariationEvaluation range
Box pickupBox geometry, placement, and massBase size 20×30×4020 \times30 \times40 cm, independently scaled by sx,sy∈[0.5,1.5]s_x, s_y \in[0.5, 1.5] and sz∈[0.5,1.0]s_z \in[0.5, 1.0], yielding sizes from 10×15×2010 \times15 \times20 cm to 30×45×4030 \times45 \times40 cm. Boxes are placed either on the ground or on supports 0.25–0.50 m high, with mass sampled from 0.5–1.5 kg.
Door openingDoor geometry and handle placementDoor width 0.8–1.1 m and height 1.9–2.2 m. Handle height is 0.85–0.95 m above the ground, with lateral inset 0.08–0.15 m.
Object graspingObject identity and shape128 distinct object instances are evaluated with four takes per object (512 episodes total), selected through teacher-feasibility screening from 685 object instances and 2,991 usable demonstrations.

Table X. Simulation evaluation variations.

Real-World Execution

The real-world system uses onboard RealSense depth and proprioception. Both the unified planner and the frozen BFM run on a single NVIDIA RTX PRO 5000 Blackwell laptop. The TensorRT planner replans at 5 Hz using 10 Heun integration steps, with approximately 20 ms inference time per planner call, while the BFM runs at 50 Hz with approximately 1 ms per forward pass.

We use RTC during deployment to avoid discontinuities between consecutively sampled latent trajectories. At each replanning step, the flow planner predicts a chunk of future latent commands while the robot continues executing the current chunk. The newly generated chunk overlaps temporally with the remaining portion of the previous prediction, and RTC conditions the overlapping prefix of the new sample toward the previously committed trajectory during flow integration. This preserves continuity across replanning boundaries while still allowing later latent commands in the chunk to adapt to the latest onboard observation. In this way, RTC decouples the planner inference latency from continuous robot execution and avoids abrupt latent changes caused by independently sampling each plan.

References

  1. [1]T. He, Z. Wang, H. Xue, Q. Ben, Z. Luo, W. Xiao, Y. Yuan, X. Da, F. Castañeda, S. Sastry, C. Liu, G. Shi, L. Fan, and Y. Zhu, “Viral: Visual sim-to-real at scale for humanoid loco-manipulation,” in Conference on Computer Vision and Pattern Recognition (CVPR), 2026, pp. 13 430–13 441.arxiv.org/abs/2511.15200
  2. [2]H. Xue, T. He, Z. Wang, Q. Ben, W. Xiao, Z. Luo, X. Da, F. Castañeda, G. Shi, S. Sastry, L. J. Fan, and Y. Zhu, “Opening the sim-to-real door for humanoid pixel-to-action policy transfer,” in Conference on Computer Vision and Pattern Recognition (CVPR), 2026.
  3. [3]Z. Li, Z. Zhang, Y. Wei, W. Zhang, X. Yuan, P. Zhi, G. Li, X. Guo, F. Gao, J. Yang, and S. Zhang, “ω-0: A latent predictive world action model for concurrent humanoid loco-manipulation,” 2026.arxiv.org/abs/2608.06375
  4. [4]J. Zheng, T. Ma, Y. Fan, Z. Wang, S. Yang, and J. Liang, “Mo-tionWAM: Towards foundation world action models for real-time humanoid loco-manipulation,” 2026.
  5. [5]C. Tessler, Y. Guo, O. Nabati, G. Gchechik, and X. Peng, “Masked-mimic: Unified physics-based character control through masked motion inpainting,” ACM Transactions on Graphics (TOG), vol. 43, no. 6, pp. 1–21, 2024.
  6. [6]Q. Liao, T. E. Truong, X. Huang, Y. Gao, G. Tevet, K. Sreenath, and C. K. Liu, “Beyondmimic: From motion tracking to versatile humanoid control via guided diffusion,” Science Robotics, vol. 11, no. 117, p. eadx8924, 2026.DOI
  7. [7]H. Jiang, J. Chen, Q. Bu, L. Chen, M. Shi, Y. Zhang, D. Li, C. Suo, C. Wang, Z. Peng, and H. Li, “Wholebodyvla: Towards unified latent via for whole-body loco-manipulation control,” in International Conference on Learning Representations (ICLR), 2026.
  8. [8]S. Wei, H. Jing, B. Li, Z. Zhao, J. Mao, Z. Ni, S. He, S. Zang, X. Liu, K. Kang, J. Liu, W. Yuan, M. Pavone, D. Huang, and Y. Wang, “ψ₀: An open foundation model towards universal humanoid loco-manipulation,” in Robotics: Science and Systems (RSS), 2026.
  9. [9]Y. Li, Z. Luo, T. Zhang, C. Dai, A. Kanervisto, A. Tirinzoni, H. Weng, K. Kitani, M. Guzek, A. Touati, A. Lazaric, M. Pirotta, and G. Shi, “BFM-Zero: A promptable behavioral foundation model for humanoid control using unsupervised reinforcement learning,” in International Conference on Learning Representations (ICLR), 2026.
  10. [10]M. Pirotta, A. Tirinzoni, A. Touati, A. Lazaric, and Y. Ollivier, “Fast imitation via behavior foundation models,” in International Conference on Learning Representations (ICLR), 2024.
  11. [11]C. Zhang, W. Xiao, T. He, and G. Shi, “Wococo: Learning whole-body humanoid control with sequential contacts,” in Conference on Robot Learning (CoRL), PMLR, 2024, pp. 455–472.arxiv.org/abs/2406.06005
  12. [12]Y. Li, Y. Zhang, W. Xiao, C. Pan, H. Weng, G. He, T. He, and G. Shi, “Hold my beer: Learning gentle humanoid locomotion and end-effector stabilization control,” in Conference on Robot Learning (CoRL), ser. Proceedings of Machine Learning Research, vol. 305, 2025, pp. 4506–4523.arxiv.org/abs/2505.24198
  13. [13]J. Ren, J. Long, T. Huang, H. Wang, Z. Wang, F. Jia, W. Zhang, J. Wang, P. Luo, and J. Pang, “Humanoid goalkeeper: Learning from position conditioned task-motion constraints,” 2025.arxiv.org/abs/2510.18002
  14. [14]Y. Lin, J. Cui, Y. Li, W. Liang, B. Jia, Y. Zhu, and S. Huang, “Less-Mimic: Versatile humanoid-object interaction with unified distance field representations,” in Advances in Neural Information Processing Systems (NeurIPS), 2026.
  15. [15]L. Yang, X. Huang, Z. Wu, A. Kanazawa, P. Abbeel, C. Sferrazza, C. K. Liu, R. Duan, and G. Shi, “Omnimanipret: Interaction-preserving data generation for humanoid whole-body loco-manipulation and scene interaction,” in International Conference on Robotics and Automation (ICRA), 2026.
  16. [16]S. Chen, S. Zhao, Z. Wu, J. Li, G. Shi, and C. K. Liu, “Scenebot: Contact-prompted general humanoid whole body tracking with scene-interaction,” 2026.arxiv.org/abs/2606.27581
  17. [17]R. Yu, X. Lin, J. Ma, Y. Wang, K. Luo, J. Ji, H. Wang, W. Wang, R. Zhang, P. Tan, T. Wu, R. Dai, Q. Chen, and L. Han, “Omnicontact: Chaining meta-skills via contact flow for generalizable humanoid loco-manipulation,” 2026.
  18. [18]Y. Wang, J. Lin, A. Zeng, Z. Luo, J. Zhang, and L. Zhang, “Physhoi: Physics-based imitation of dynamic human-object interaction,” 2023.arxiv.org/abs/2312.04393
  19. [19]Y. Wang, Q. Zhao, R. Yu, H. W. Tsui, A. Zeng, J. Lin, Z. Luo, J. Yu, X. Li, Q. Chen, J. Zhang, L. Zhang, and P. Tan, “Skillmimic: Learning basketball interaction skills from demonstrations,” in Conference on Computer Vision and Pattern Recognition (CVPR), 2025, pp. 17 540–17 549.arxiv.org/abs/2408.15270
  20. [20]R. Yu, Y. Wang, Q. Zhao, H. W. Tsui, J. Wang, P. Tan, and Q. Chen, “Skillmimic-v2: Learning robust and generalizable interaction skills from sparse and noisy demonstrations,” in ACM SIGGRAPH Conference Papers, 2025, pp. 1–11.arxiv.org/abs/2505.02094
  21. [21]Y. Zhang, Y. Yuan, P. Gurunath, I. Gupta, S. Omidshafiei, A.-a. Aghamohammadi, M. Vazquez-Chanlatte, L. Pedersen, T. He, and G. Shi, “Falcon: Learning force-adaptive humanoid loco-manipulation,” in Learning for Dynamics and Control (L4DC), 2026, pp. 265–281.arxiv.org/abs/2505.06776
  22. [22]Y. Du, Y. Li, B. Jia, Y. Lin, P. Zhou, W. Liang, Y. Yang, and S. Huang, “Learning human-humanoid coordination for collaborative object carrying,” in International Conference on Robotics and Automation (ICRA), 2026.arxiv.org/abs/2510.14293
  23. [23]Y. Wang, Q. Zhao, Y. F. Lau, R. Yu, H. W. Tsui, Q. Chen, J. Wang, J. Pang, and P. Tan, “Humamx: Toward agile and generalizable humanoid interaction skills from human videos,” 2026.arxiv.org/abs/2602.02473
  24. [24]D. Li, X. Chen, Q. Wu, B. Chen, S. Wu, H. Wu, G. Zhang, L. Li, M. Zhou, D. Xiang, J. Ma, Q. Zhang, and R. Xu, “Haic: Humanoid agile object interaction control via dynamics-aware world model,” in Robotics: Science and Systems (RSS), 2026.arxiv.org/abs/2602.11758
  25. [25]Z. Wang, Z. Wu, P. Abbeel, R. Duan, J. Malik, C. Sferrazza, C. K. Liu, G. Shi, and A. Kanazawa, “Counterfactual video generation enables scalable humanoid loco-manipulation,” 2026.arxiv.org/abs/2609.38172
  26. [26]T. He, Z. Luo, X. He, W. Xiao, C. Zhang, W. Zhang, K. Kitani, C. Liu, and G. Shi, “Omnih2o: Universal and dexterous human-to-humanoid whole-body teleoperation and learning,” in Conference on Robot Learning (CoRL), 2024, pp. 1516–1540.arxiv.org/abs/2406.08858
  27. [27]Y. Ze, Z. Chen, J. P. Araújo, Z. Cao, X. B. Peng, J. Wu, and C. K. Liu, “Twist: Teleoperated whole-body imitation system,” in Conference on Robot Learning (CoRL), 2025, pp. 2143–2154.arxiv.org/abs/2505.02833
  28. [28]Y. Li, Y. Lin, J. Cui, T. Liu, W. Liang, Y. Zhu, and S. Huang, “Clone: Closed-loop whole-body humanoid teleoperation for long-horizon tasks,” in Conference on Robot Learning (CoRL), ser. Proceedings of Machine Learning Research, vol. 305, 2025, pp. 4493–4505.arxiv.org/abs/2506.08931
  29. [29]H. Weng, Y. Li, N. Sobanbabu, Z. Wang, Z. Luo, T. He, D. Ramanan, and G. Shi, “Hdmi: Learning interactive humanoid whole-body control from human videos,” 2025.arxiv.org/abs/2509.16757
  30. [30]R. Nai, B. Zheng, J. Zhao, H. Zhu, S. Dai, Z. Chen, Y. Hu, Y. Hu, T. Zhang, C. Wen, and Y. Gao, “Humanoid manipulation interface: Humanoid whole-body manipulation from robot-free demonstrations,” 2026.
  31. [31]M. Shi, S. Peng, J. Chen, H. Jiang, T. Li, P. Luo, D. Huang, H. Li, and L. Chen, “Unlocking in-the-wild loco-manipulation with robot-free egocentric demonstration,” in Robotics: Science and Systems (RSS), 2026.
  32. [32]T. Wu, X. Kong, Y. Chen, Q. Yu, H. Ye, J. Li, Y. Wang, and H. Dong, “Sugar: A scalable human-video-driven generalizable humanoid loco-manipulation learning framework,” 2026.arxiv.org/abs/2605.20373
  33. [33]T. Xie, H. Zhang, J. Park, Z. Wang, B. Wen, J. Li, X. Li, Q. Ben, H. Weng, Y. Ye, D. Minor, T. Wang, C. Jiang, S. Fidler, J. Kautz, L. Fan, Y. Zhu, Z. Luo, U. Iqbal, and Y. Yuan, “GRAIL: Generating humanoid loco-manipulation from 3D assets and video priors,” 2026.
  34. [34]Z. Luo, Y. Yuan, T. Wang, C. Li, F. Castañeda, S. Chen, Z.-A. Cao, J. Li, D. Minor, Q. Ben, J. Park, D. Sami, Z. Wang, X. Da, R. Ding, C. Hogg, L. Song, E. Lim, E. Jeong, T. He, H. Xue, W. Xiao, S. Yuen, J. Kautz, Y. Chang, U. Iqbal, L. J. Fan, and Y. Zhu, “SONIC: Supersizing motion tracking for natural humanoid whole-body control,” Science Robotics, vol. 11, no. 117, p. ead4592, 2026.DOI
  35. [35]S. Ross, G. Gordon, and D. Bagnell, “A reduction of imitation learning and structured prediction to no-regret online learning,” in International Conference on Artificial Intelligence and Statistics (AISTATS), ser. Proceedings of Machine Learning Research, vol. 15, 2011, pp. 627–635.arxiv.org/abs/1011.0686
  36. [36]I. Mason, S. Starke, and T. Komura, “Real-time style modelling of human locomotion via feature-wise transformations and local motion phases,” Proceedings of the ACM on Computer Graphics and Interactive Techniques, vol. 5, no. 1, may 2022.arxiv.org/abs/2201.04439

Paper details

vlnc:W_vzzmph8p
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 4

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents