betaFeedback

WorksComputer Science

MAPO: Mixture of Experts with Alternating Policy Optimization for Universal Humanoid Motion Tracking

Abstract

Universal humanoid motion tracking is essential for enabling humanoid robots to perform diverse tasks in human environments. To capture the wide diversity of human motions, mixture-of-experts (MoE) policies have been widely adopted, leveraging dynamic routing to enhance policy expressiveness. However, we identify that divergence between old and updated policies can be misestimated during policy-gradient updates of MoE policies, due to their dynamic routing. Because most policy-gradient algorithms regulate policy updates based on the divergence, this misestimation can lead to either overly conservative or excessively large updates, which can yield a suboptimal policy. In this paper, we propose Mixture of Experts with Alternating Policy Optimization (MAPO), which alternately optimizes the router and the experts to avoid the divergence misestimation. MAPO consistently tracks 182 motions that PPO with MoE fails to track, whereas PPO with MoE tracks only one motion that MAPO fails to track. MAPO also attains the highest tracking success rate among the compared methods and the lowest tracking error across most error metrics, demonstrating its superior performance. Videos can be found on the project website: https://mapo-corl2026.github.io/MAPO/

Introduction

Humanoid robots have the potential to perform various tasks in human environments, given their human-like morphology and whole-body dexterity for locomotion and manipulation. To achieve such potential, a humanoid should be able to perform diverse motions ranging from basic locomotion to complex whole-body behaviors. Reinforcement learning (RL)-based approaches have enabled humanoids to perform robust walking [1, 2, 3, 4], parkour [5, 6], and fall recovery [7, 8]. However, extending such approaches to universal humanoid motion tracking requires physically feasible and diverse reference motions during training. Large-scale human motion datasets [9, 10, 11] have become promising sources of such reference motions [12, 13, 14, 15, 16, 17, 18].

Nevertheless, learning a universal motion tracker with a monolithic policy can be challenging [19, 20]. Because diverse reference motions require distinct control strategies, this task can be cast as multi-task learning, where monolithic policies suffer from inter-task interference [21, 22]. Moreover, different motion modes often share overlapping state regions [19], likely exacerbating the interference [23]. In contrast, the mixture of experts (MoE) architecture [24] has multiple experts and a router as its components, with the router computing mixing weights over the experts for a given state. This architecture has shown effectiveness in such multi-task learning problems [25, 26, 27, 28, 29]. The MoE architecture has also been explored for motion tracking tasks [19, 20, 30, 31]. However, these studies primarily focus on architectural design, and the problems that arise when MoE policies are trained with widely used policy-gradient algorithms [32, 33] remain largely unexplored.

In this work, we identify a phenomenon termed divergence misestimation where the divergence between the data-collection and updated policies does not accurately reflect the magnitude of MoE policy updates. Specifically, the phenomenon becomes problematic for algorithms that constrain policy updates via this divergence, including proximal policy optimization (PPO) [33], which is widely adopted in robot control tasks. For monolithic policies, the magnitude of the policy update is directly reflected in the divergence. However, the distribution of an MoE policy is a weighted sum of expert outputs, with mixing weights determined by the router. Therefore, the divergence can change not only with updates to the experts but also with shifts in the mixing weights. It can thus either overestimate or underestimate the changes in each component, leading to overly conservative or aggressive policy updates, respectively.

To address the divergence misestimation, we propose Mixture of Experts with Alternating Policy Optimization (MAPO). A naive approach to resolve the divergence misestimation is to freeze the mixing weights at their data-collection values during the update. Although this approach can avoid the misestimation, it prevents the router from being optimized via the RL objective. MAPO resolves this trade-off by symmetrically alternating updates of the two components. During the expert update, the mixing weights are held fixed at their rollout values, and vice versa for the router update. In this way, MAPO measures the divergence only for the component under update, rather than over the entire MoE policy as in existing policy-gradient algorithms. Such separation avoids the misestimation, and the alternating updates allow both components to be optimized with the RL objective.

Our contributions are summarized as follows. (i) We identify divergence misestimation in MoE policy optimization. Because most policy-gradient algorithms regulate policy divergence, the misestimation can lead to overly conservative or aggressive updates. (ii) We propose MAPO, a simple but efficient remedy for the divergence misestimation that alternates expert and router updates, measuring divergence per component. In our experiments on humanoid motion tracking, the proposed method achieves substantially higher reward and better tracking performance than a PPO-trained MoE policy by avoiding the divergence misestimation (see Figure 1).

Humanoid motion tracking comparison between MAPO and PPO with MoE

Figure 1. MAPO resolves the divergence misestimation problem that hinders policy-gradient training of MoE policies, enabling effective humanoid motion tracking. Consequently, MAPO successfully tracks a wide variety of human motions, including complex whole-body motions, such as (a) sitting in mid-air, (b) abrupt full-body turns, (c) deep crouching, and (d) one-footed spinning. Red spheres denote the reference motion targets.

Related Work

Mixture of Experts in Reinforcement Learning: Originally proposed by Jacobs et al. [24], MoE has gained increasing attention for scaling up large language models [34, 35, 36, 37] and vision models [38]. MoE has also been explored in RL [39, 40, 41], particularly in multi-task settings [25, 26, 27, 28, 29]. Many of these works adopt a sparse MoE, which activates only a subset of experts for each input, enabling the network to scale efficiently. However, sparse routing is non-differentiable, hindering end-to-end optimization [39]. In contrast, dense MoE [42, 43, 44, 45] uses a weighted combination of all experts with continuous mixing weights, making it fully differentiable [42]. Recent empirical studies [25, 46] have analyzed MoE in RL, reporting that differentiable MoE shows better performance than sparse MoE. Despite these advances, the divergence misestimation phenomenon has remained unaddressed, which we identify and resolve in this work.

Universal Whole-body Control for Humanoids: RL-based approaches have achieved robust whole-body control for quadrupeds [47, 48] and legged manipulators [49, 50, 51, 52, 53]. Such approaches require sampling motion targets during training, which is straightforward when motion targets can be abstracted as low-dimensional states, such as body velocities or end-effector poses. However, this approach is hard to extend to humanoids, where such sampling rarely yields physically feasible whole-body motions. Instead, DeepMimic [12] demonstrates that human motion data can serve as feasible and diverse reference motions for various simulated avatars. This paradigm has been scaled to achieve universal humanoid whole-body control in the real world [12, 13, 14, 15, 16, 17, 18, 54, 55, 56, 57, 58, 59]. However, the increasing scale and diversity of motion data raise the need for more expressive policy networks. MoE architectures have been explored to improve expressiveness via expert specialization [19, 20, 30, 31]. CLONE [31] and KungfuBot2 [30] adopt a sparse MoE, whose non-differentiable routing hinders end-to-end RL optimization. In contrast, GMT [19] and FAST [20] employ a fully differentiable dense MoE. However, these works primarily focus on architectural design, leaving the effective optimization of MoE policies under widely used policy-gradient algorithms [32, 33] underexplored.

Background: Mixture of Experts with Policy Gradient

Reinforcement Learning

RL considers a sequential decision-making problem within a given environment [60]. The environment is typically modeled as a Markov decision process defined by the tuple ⟨S,A,P,R,γ⟩\langle\mathcal{S}, \mathcal{A}, P, R, \gamma\rangle, where S\mathcal{S} is the state space, A\mathcal{A} is the action space, P(s′∣s,a)P(s' \mid s, a) is the transition probability from state ss to s′s' given action aa, R(s,a)R(s, a) is the reward function, and γ∈[0,1]\gamma\in[0, 1] is the discount factor. A stochastic policy π(a∣s)\pi(a \mid s) defines a distribution over actions conditioned on the current state. Because the desired behavior is encoded in the reward function, the goal of RL is to find an optimal policy π∗\pi^* that maximizes the expected return, Eπ[∑t=0∞γtR(s,a)]\mathbb{E}_{\pi}[\sum_{t=0}^{\infty} \gamma^t R(s, a)].

Policy Gradient Methods

Policy gradient methods directly optimize the policy to maximize the expected return. One of the key challenges in policy gradient methods is that aggressive updates can destabilize training. To address this challenge, Schulman et al. [32] proposed trust region policy optimization (TRPO), which constrains the Kullback–Leibler (KL) divergence between the data-collection policy πold\pi^{\mathrm{old}} and the new policy πnew\pi^{\mathrm{new}}. When the divergence remains under the trust region, TRPO theoretically guarantees monotonic improvement of the policy performance. However, TRPO is computationally expensive because it requires second-order optimization. To address this limitation, PPO [33] was proposed as an efficient first-order alternative, using the following clipped surrogate objective LPPOL^{\mathrm{PPO}}:

LPPO=E^t[min⁡(rtA^t,clip⁡(rt,1−ϵ,1+ϵ)A^t)],(1)L^{\mathrm{PPO}} = \hat{\mathbb{E}}_t \left[\min\left(r_t \hat{A}_t, \operatorname{clip}(r_t, 1-\epsilon, 1+\epsilon)\hat{A}_t\right)\right], \tag*{(1)}

where rt=πnew(at∣st)/πold(at∣st)r_t = \pi^{\mathrm{new}}(a_t \mid s_t)/\pi^{\mathrm{old}}(a_t \mid s_t) is the probability ratio between the new and old policies, A^t\hat{A}_t is an advantage estimate, and ϵ\epsilon controls the clipping range. Notably, the PPO clipping induces an implicit trust region by constraining the KL divergence on the order of ϵ2\epsilon^2 (see Supplementary A).

Divergence Misestimation in Mixture-of-Experts Policy Optimization

The divergence of MoE policies can be misestimated. Such misestimation is problematic for policy-gradient algorithms [32, 33] that regulate the divergence. To formalize the misestimation, we consider a Gaussian policy whose mean is parameterized by a dense MoE. Let μ1,…,μN\mu_1,\ldots,\mu_N denote the outputs of NN experts. The MoE policy is then parameterized as π(a∣s)=N(m,σ2I)=N(∑e=1Nweμe,σ2I)\pi(a\mid s)=\mathcal{N}(m,\sigma^2 I)=\mathcal{N}\left(\sum_{e=1}^{N}w_e\mu_e,\sigma^2 I\right), where w=[w1,…,wN]\mathbf{w}=[w_1,\ldots,w_N] are obtained by applying a softmax to the router’s output, ensuring their sum to one. For simplicity, σ\sigma is assumed to be fixed within each update step.

The measured KL divergence between the old and new policies is as follows:

KLmeasured=D(πold∥πnew)=∥mold−mnew∥22σ2.(2)\mathrm{KL}_{\mathrm{measured}}=D\left(\pi^{\mathrm{old}}\mathbin{\|}\pi^{\mathrm{new}}\right)=\frac{\left\|m^{\mathrm{old}}-m^{\mathrm{new}}\right\|^2}{2\sigma^2}. \tag*{(2)}

where mold=∑weoldμeoldm^{\mathrm{old}}=\sum w_e^{\mathrm{old}}\mu_e^{\mathrm{old}} and mnew=∑wenewμenewm^{\mathrm{new}}=\sum w_e^{\mathrm{new}}\mu_e^{\mathrm{new}} denote the means of the old and new policies, respectively. The difference mold−mnewm^{\mathrm{old}}-m^{\mathrm{new}} yields the decomposition as follows:

mold−mnew=∑weold(μeold−μenew)⏟dμ+∑(weold−wenew)μenew⏟dw.(3)m^{\mathrm{old}}-m^{\mathrm{new}}=\underbrace{\sum w_e^{\mathrm{old}}\left(\mu_e^{\mathrm{old}}-\mu_e^{\mathrm{new}}\right)}_{d_\mu}+\underbrace{\sum\left(w_e^{\mathrm{old}}-w_e^{\mathrm{new}}\right)\mu_e^{\mathrm{new}}}_{d_w}. \tag*{(3)}

where dμd_\mu and dwd_w isolate the changes attributable to the experts and the router, respectively. Then, the measured KL divergence can be expressed as follows:

KLmeasured=12σ2(∥dμ∥2+2dμ⋅dw+∥dw∥2).(4)\mathrm{KL}_{\mathrm{measured}}=\frac{1}{2\sigma^2}\left(\left\|d_\mu\right\|^2+2d_\mu\cdot d_w+\left\|d_w\right\|^2\right). \tag*{(4)}

We now show that the measured KL divergence can misestimate the true changes in each component. To measure the true changes in each component, we decompose the policy update into two sequential steps that update the experts first and then the router. Consider a partially updated policy, π′=N(∑weoldμenew,σ2I)\pi'=\mathcal{N}\left(\sum w_e^{\mathrm{old}}\mu_e^{\mathrm{new}},\sigma^2 I\right), in which only the experts are updated while the router remains unchanged. The fully updated policy πnew\pi^{\mathrm{new}} is then obtained by updating only the router from π′\pi'. The two-step path is as follows: πold→expert updateπ′→router updateπnew\pi^{\mathrm{old}}\xrightarrow{\text{expert update}}\pi'\xrightarrow{\text{router update}}\pi^{\mathrm{new}}.

Then we can define the KL divergence for each update step in the path as follows:

KLexpert=D(πold∥π′)=∥dμ∥22σ2,KLrouter=D(π′∥πnew)=∥dw∥22σ2.(5)\mathrm{KL}_{\mathrm{expert}}=D\left(\pi^{\mathrm{old}}\mathbin{\|}\pi'\right)=\frac{\left\|d_\mu\right\|^2}{2\sigma^2},\qquad\mathrm{KL}_{\mathrm{router}}=D\left(\pi'\mathbin{\|}\pi^{\mathrm{new}}\right)=\frac{\left\|d_w\right\|^2}{2\sigma^2}. \tag*{(5)}

Therefore, the measured KL divergence can be rewritten as follows:

KLmeasured=KLrouter+KLexpert+dw⋅dμσ2.(6)\mathrm{KL}_{\mathrm{measured}}=\mathrm{KL}_{\mathrm{router}}+\mathrm{KL}_{\mathrm{expert}}+\frac{d_w\cdot d_\mu}{\sigma^2}. \tag*{(6)}

An analogous decomposition holds when the router is updated first, with dwd_w and dμd_\mu defined along that path.

Definition 1 (Divergence Misestimation). Divergence misestimation refers to the discrepancy between the measured divergence KLmeasured\mathrm{KL}_{\mathrm{measured}} and the sum of component-wise divergences KLrouter+KLexpert\mathrm{KL}_{\mathrm{router}}+\mathrm{KL}_{\mathrm{expert}}. Specifically,

dw⋅dμ>0⟹KLmeasured>KLrouter+KLexpert(overestimation),d_w\cdot d_\mu>0\quad\Longrightarrow\quad\mathrm{KL}_{\mathrm{measured}}>\mathrm{KL}_{\mathrm{router}}+\mathrm{KL}_{\mathrm{expert}}\quad\text{(overestimation)},
dw⋅dμ<0⟹KLmeasured<KLrouter+KLexpert(underestimation).d_w\cdot d_\mu<0\quad\Longrightarrow\quad\mathrm{KL}_{\mathrm{measured}}<\mathrm{KL}_{\mathrm{router}}+\mathrm{KL}_{\mathrm{expert}}\quad\text{(underestimation)}.

When overestimation occurs, the policy update is overly constrained by the divergence regularization, which can lead to conservative updates and suboptimal performance. Underestimation has the opposite effect. It makes the divergence regularization fail to constrain aggressive updates of each component, potentially leading to unstable training. Notably, the sign of dw⋅dμd_w\cdot d_\mu is difficult to control explicitly during policy updates, as the gradient directions of the router and experts are not guaranteed to align or oppose each other. Therefore, simple solutions such as adjusting the clipping range of PPO cannot effectively address the divergence misestimation.

Figure 2 shows the ratio of non-trivial divergence misestimation during the training of an MoE policy with vanilla policy-gradient algorithms. A divergence misestimation is considered non-trivial when the difference between the measured KL divergence and the sum of component KL divergences exceeds the threshold. Because PPO typically constrains the KL divergence on the order of ϵ2\epsilon^2, we set the threshold to ϵ2\epsilon^2 with the default value of ϵ\epsilon. Overestimation becomes more prevalent as the number of experts increases, while underestimation dominates with fewer experts. Therefore, divergence misestimation is inherent to combining MoE with policy-gradient optimization; tuning the number of experts only switches its dominant mode between underestimation and overestimation. Please refer to the Figure D.2 for the distribution of the misestimation and further analysis of its effect under training dynamics.

Figure 2. Ratio of non-trivial policy divergence misestimation. NN is the number of experts. Overestimation becomes more frequent with more experts, whereas underestimation becomes more prevalent with fewer experts. Triviality threshold is ϵ2\epsilon^2, where ϵ\epsilon is the default PPO clipping ratio.

Mixture of Experts for Universal Humanoid Motion Tracking

In this study, we aim to train a humanoid robot to imitate diverse human motions [9, 10, 11]. Following previous works [19, 20, 30, 31], we formulate this task as an RL problem and adopt an MoE architecture to enhance the expressiveness of the policy network. To avoid the divergence misestimation issue, we propose MAPO, a policy optimization algorithm for MoE policies.

MAPO: Mixture of Experts with Alternating Policy Optimization

A simple solution to policy divergence misestimation is to store the mixing weights during data collection and use them to compute the probability ratio during policy optimization. However, in this case, the router is no longer part of the gradient computation and thus cannot be optimized with the RL objective. We thus face a trade-off between preventing divergence misestimation and optimizing the router with the RL objective.

To address this trade-off, we propose MAPO. A key idea of MAPO is to alternate between expert and router updates, with the other component held fixed at its data-collection value. The corresponding probability ratios for each update are

rtexpert=N(∑weoldμe,σ2I)N(∑weoldμeold,σ2I),rtrouter=N(∑weμeold,σ2I)N(∑weoldμeold,σ2I).(7)r_t^{\mathrm{expert}} = \frac{\mathcal{N}\left(\sum w_e^{\mathrm{old}}\mu_e,\sigma^2 I\right)}{\mathcal{N}\left(\sum w_e^{\mathrm{old}}\mu_e^{\mathrm{old}},\sigma^2 I\right)}, \qquad r_t^{\mathrm{router}} = \frac{\mathcal{N}\left(\sum w_e\mu_e^{\mathrm{old}},\sigma^2 I\right)}{\mathcal{N}\left(\sum w_e^{\mathrm{old}}\mu_e^{\mathrm{old}},\sigma^2 I\right)}. \tag*{(7)}

Following Equation (1), the surrogate loss is then computed separately for each component using the clipped probability ratio, yielding gradients for the corresponding parameters. Because each ratio can directly measure the amount of change in the corresponding component, the policy divergence misestimation can be prevented. Additionally, because both components are used to compute the gradient, the router and experts can be optimized with the RL objective. Notably, alternating refers to separate gradient computations for the experts and the router, rather than an interleaved update schedule. Both surrogate losses are computed from the same rollout data within a single update step.

Figure 3 illustrates the differences in update mechanisms between MAPO and a standard policy gradient algorithm, PPO. Specifically, PPO measures the KL divergence of the mixed policy, while MAPO measures each component separately for clipping. When overestimation occurs, the measured divergence of the mixed policy can exceed the implicit trust region even though each component’s update is admissible. In this case, MAPO still updates both components properly, as shown in fig:3a (a), whereas PPO blocks these admissible updates, as shown in fig:3b (b). When underestimation occurs, the mixed policy can stay within the trust region while individual components violate their own trust regions. MAPO properly clips each component, as shown in Figure 3 (c), whereas PPO fails to clip the aggressive updates, as shown in Figure 3 (d). Consequently, MAPO avoids the divergence misestimation in both cases, enabling more effective policy optimization.

Figure 3. Comparison of update mechanisms between MAPO and PPO. Pink, blue, and blue-green indicate the router, experts, and mixed policy, respectively. The bars in the center show the magnitudes of each component and the mixed policy. The shaded regions show the implicit trust regions induced by clipping. (a, c) MAPO measures the KL divergence of each component separately, allowing proper regulation of the updates. (b, d) In contrast, because PPO measures the KL divergence of the mixed policy, the clipping mechanism may fail to regulate the updates properly.

Although MAPO avoids the misestimation issue, each update step optimizes a partially updated policy rather than the fully updated policy, introducing a gradient bias. For the expert parameters, the bias preserves the stationary points of the true gradient. For the router parameters, the gradient direction may deviate, but the bias depends only on the relative inter-expert deviations and vanishes when experts update coherently. While the bias persists when experts update incoherently, our experiments show that resolving the misestimation yields a favorable net effect. See Supplementary B for a detailed analysis.

Teacher-Student Training Framework

We adopt the teacher-student training framework [14, 55, 56, 57], in which a teacher is trained with privileged information and then distilled into a student for real-world deployment.

Teacher Policy Training: The teacher policy is implemented as an MoE and trained with MAPO. For a fair comparison of the effect of MAPO, we closely follow prior humanoid motion tracking works [15, 17, 56, 58] for the observation space, reward function, and other training details [12]. The policy outputs target joint positions, which a PD controller converts into joint torques. The observation space consists of two parts. The first part, the proprioception state, contains the joint angles and velocities, root angular velocity, gravity vector, previous action, and per-link position, orientation, linear velocity, and angular velocity. The second part, the reference motion state, contains (i) future reference link positions and heading direction, (ii) current linear and angular velocities of each link, and (iii) per-link differences in current position and orientation between the robot and the reference. The reward function consists of tracking and regularization terms. The tracking term encourages the robot to imitate the reference motion, while the regularization term penalizes undesirable behaviors such as joint position limit violations. See Table C.1 for the full reward specification.

Student Policy Distillation: Because some privileged observations, such as the linear velocity of the robot, are unavailable in the real world [55], the teacher policy cannot be directly deployed. Therefore, we distill it into a student policy that excludes privileged observations. We use DAgger [61] for the distillation, where the student policy rolls out in the environment, and the teacher policy provides action labels for supervision. Because the student is trained with simple supervised learning, we adopt a monolithic network for it. The student policy’s observation and domain randomization follow the settings from TWIST [16].

Experiments

Experiment Setup

We evaluate MAPO on a humanoid motion tracking task. Because MAPO focuses on the training of MoE teacher policies, all evaluations in this section are conducted on the teacher policy, except for the real-world deployment. We train both the teacher and student policies in the Isaac Lab simulator [62], with 4096 parallel environments on a single NVIDIA A6000 GPU. The student policy is deployed on a real humanoid robot, Unitree G1 (29DoF) [63], for real-world demonstration. We use AMASS [9] and LAFAN1 [10] datasets for training and OMOMO [11] for evaluation. All motions are retargeted to the humanoid using GMR [59] to account for morphological differences. The MoE policy uses 6 experts, following prior works [19, 30]. For all methods, we use the standard PPO hyperparameters; the clipping parameter ϵ\epsilon is set to 0.2, and the learning rate is adaptively adjusted following Rudin et al. [64]. The other hyperparameters can be found in Table C.2.

We compare MAPO against the following baselines. PPO with MoE: A dense MoE policy trained with PPO. It directly captures how divergence misestimation affects MoE policy training. Monolithic PPO: A single monolithic policy trained with PPO. It serves as a baseline to quantify the benefit of the MoE architecture. MAPO with fixed router: A variant of MAPO that skips the router update, keeping the parameters of the router fixed. Although divergence misestimation is reduced by fixing the router, end-to-end router optimization is prevented. HOVER [56]: A universal motion tracker based on a teacher-student distillation framework. We re-implement their teacher policy on our humanoid platform and train it on the same dataset as our method. Notably, we scale the penalty terms of HOVER to 20% of the original values. The penalty adjustment is the minimum required to enable training without unrealistic motion. TWIST 2 [58]: A state-of-the-art humanoid motion tracking method. We evaluate it using its released checkpoint.

Experimental Results

Tracking Performance: To evaluate the tracking performance, we measure the success rate and tracking errors in diverse metrics across all motions in the training and evaluation datasets, without any motion curation. A motion is considered successfully tracked if the average position tracking error is lower than 0.5 m throughout the motion, following previous works [54, 55]. For fairness, we freeze the tracking error at its failure-time value after a failure, because excluding post-failure errors would underestimate the error of harder motions that fail early. The results are summarized in Table 1. MAPO achieves the highest success rate and the lowest tracking errors across most metrics on both the training and evaluation datasets. Additionally, we ablate the number of experts in the MoE policy. As shown in Table 1, MAPO consistently achieves better performance than PPO with MoE across all tested expert counts, except for the root roll and pitch errors.

Table 1: Tracking metrics across methods. Eg-posE_{\mathrm{g\text{-}pos}} and El-posE_{\mathrm{l\text{-}pos}} are the global and root-relative (local) body position errors (mm), EaccE_{\mathrm{acc}} and EvelE_{\mathrm{vel}} are the body acceleration (mm/frame2^{2}) and velocity (mm/frame) errors. Ert-hE_{\mathrm{rt\text{-}h}}, Ert-velE_{\mathrm{rt\text{-}vel}}, Ert-rE_{\mathrm{rt\text{-}r}}, and Ert-pE_{\mathrm{rt\text{-}p}} denote the root height (m), linear velocity (m/s), and roll and pitch errors (rad), respectively. NN denotes the number of experts. The best performance for each metric is highlighted in bold, and the second best is <u>underlined</u>.

MethodSuccess ↑Eg-posE_{\mathrm{g\text{-}pos}} ↓El-posE_{\mathrm{l\text{-}pos}} ↓EaccE_{\mathrm{acc}} ↓EvelE_{\mathrm{vel}} ↓Ert-hE_{\mathrm{rt\text{-}h}} ↓Ert-velE_{\mathrm{rt\text{-}vel}} ↓Ert-rE_{\mathrm{rt\text{-}r}} ↓Ert-pE_{\mathrm{rt\text{-}p}} ↓
(a) Training set (AMASS [9] + LAFAN1 [10])
Monolithic PPO81.78 %165.796.505.78716.350.0600.5320.1690.136
PPO with MoE83.38 %153.187.985.49914.300.0580.5010.1200.105
MAPO with fixed router<u>83.78 %</u><u>152.7</u><u>86.65</u><u>5.250</u>14.42<u>0.052</u><u>0.487</u>0.1440.122
MAPO (Ours)86.60 %147.478.104.57611.780.0480.406<u>0.138</u><u>0.117</u>
N=4N = 4PPO with MoE83.96 %156.585.015.30113.750.0540.4900.1310.119
MAPO86.87 %143.674.614.67211.600.0500.4070.1190.108
N=8N = 8PPO with MoE82.28 %164.493.475.84115.510.0640.5380.1260.110
MAPO87.85 %134.871.654.12010.060.0450.3600.1380.115
N=16N = 16PPO with MoE82.95 %159.694.086.17515.540.0610.5280.1180.107
MAPO87.73 %136.972.584.54911.020.0480.3750.1280.110
(b) Test set (OMOMO [11])
Monolithic PPO84.36 %227.2<u>86.43</u>6.01412.440.0350.3560.1570.134
PPO with MoE82.78 %241.493.906.45113.770.0410.3970.1400.130
MAPO with fixed router85.21 %221.687.015.93712.30<u>0.031</u><u>0.331</u>0.1530.136
MAPO (Ours)85.45 %<u>225.8</u>85.295.76411.760.0300.326<u>0.151</u>0.129
HOVER [56]83.39 %272.2122.816.00112.460.0580.3820.1930.163
TWIST 2 [58]57.85 %386.6104.724.661<u>11.87</u>0.0420.3550.0800.074

Table 1. Tracking metrics across methods. Eg-posE_{\mathrm{g\text{-}pos}} and El-posE_{\mathrm{l\text{-}pos}} are the global and root-relative (local) body position errors (mm), EaccE_{\mathrm{acc}} and EvelE_{\mathrm{vel}} are the body acceleration (mm/frame2^{2}) and velocity (mm/frame) errors. Ert-hE_{\mathrm{rt\text{-}h}}, Ert-velE_{\mathrm{rt\text{-}vel}}, Ert-rE_{\mathrm{rt\text{-}r}}, and Ert-pE_{\mathrm{rt\text{-}p}} denote the root height (m), linear velocity (m/s), and roll and pitch errors (rad), respectively. NN denotes the number of experts. The best performance for each metric is highlighted in bold, and the second best is underlined.

Effectiveness of MAPO Training: Figure 4 (a) shows the effect of divergence misestimation on training performance. MAPO achieves a significantly higher reward than PPO with MoE, demonstrating that avoiding misestimation substantially improves the training performance of MoE policies. Furthermore, MAPO with fixed router achieves slightly higher reward than PPO with MoE, suggesting that the performance loss from misestimation is comparable to that from fixing the router. Additionally, all MoE-based methods achieve higher reward than Monolithic PPO, confirming the effectiveness of the MoE architecture in humanoid motion tracking.

Figure 4. (a) Reward learning curves. MAPO achieves substantially higher rewards than the baselines. (b) Sorted mixing weights of MAPO and PPO, averaged at each rank across inputs. The misestimation of PPO prevents the mixing weights from sharpening. (c) Difference in pairwise expert output distances (MAPO − PPO). Positive entries indicate higher expert diversity in MAPO.

Expert Specialization: The specialization of dense MoE arises from differences in the mixing weights, which induce distinct gradients for each expert [24]. Therefore, sharper mixing weights are generally preferred, provided that all experts remain utilized across inputs. However, although sharpened routing is desirable for specialization, it can exacerbate divergence overestimation: sharpening induces a large router-side change dwd_w, which can amplify the cross term dw⋅dμd_w \cdot d_\mu. Consequently, PPO with MoE tends to clip such updates, yielding less sharp routing than MAPO (Figure 4 (b)). The sharper routing of MAPO promotes better expert specialization, reflected in the generally larger pairwise expert output distances (Figure 4 (c)). Each entry is the average L2L2 distance between the outputs of expert pairs on the same input, reported as (MAPO − PPO). Figure D.1 in the Supplementary shows per-expert activation across different motion types.

Ablation of Confounding Factors: We examine potential confounding factors through the ablation studies in Figure 5. First, we test whether the performance gain of MAPO results from a larger update budget due to separate clipping of the expert and router updates (Figure 5 (a)). We scale the default clipping parameter ϵ\epsilon by 0.5 for MAPO and by 1.5 and 2 for PPO with MoE. Increasing ϵ\epsilon reduces the performance of PPO with MoE. In contrast, MAPO with the smaller ϵ\epsilon retains higher performance than PPO with MoE across all tested values. The results suggest that the performance gain of MAPO does not arise solely from a larger update budget.

Figure 5. (a) Clipping parameter ϵ\epsilon sweep. (b) Entropy coefficient sweep. (c) Fixed learning rate ablation. (d) Variants of PPO with MoE using a default learning rate of 10−410^{-4}.

Second, the policy standard deviation σ\sigma in MAPO is updated by both surrogate losses computed from the ratios in Equation (7). However, the entropy bonus contributes to the update of σ\sigma only once. Therefore, the effective entropy coefficient for σ\sigma can be halved. To separate the effect of the reduced effective entropy coefficient from the effect of reduced divergence misestimation, we introduce Averaged-MAPO. Averaged-MAPO scales each surrogate loss by 1/21/2, thereby keeping the ef-

fective entropy coefficient for σ\sigma unchanged. Under matched entropy coefficients, Averaged-MAPO consistently achieves higher reward than PPO with MoE across all tested values (Figure 5 (b)).

Consistency of Performance Gains: We examine whether the performance gain of MAPO persists under fixed learning rates. We compare MAPO and PPO with MoE at learning rates of 10−410^{-4} and 2×10−42 \times10^{-4} (Figure 5 (c)). MAPO consistently outperforms PPO with MoE at both learning rates. Because training becomes unstable for both methods at 2×10−42 \times10^{-4}, we set 2×10−42 \times10^{-4} as the maximum learning rate in the comparison. Additionally, we compare MAPO with variants of PPO with MoE (Figure 5 (d)). We test separate learning rates for the router and remaining parameters. We also evaluate alternative regularization techniques, including a KL penalty and early stopping. However, the tested variants do not substantially improve on the default PPO with MoE and fail to match the performance of MAPO.

Qualitative Results: Figure 1 visualizes representative motions where MAPO succeeds but PPO with MoE fails, under the same training setup. The selected motions require complex whole-body coordination, including sitting in mid-air, abrupt full-body turns, deep crouching, and one-footed spinning. These motions are selected from 182 cases on which MAPO consistently succeeds while PPO with MoE consistently fails across 5 independent trials, whereas the reverse occurs for only 1 motion. Notably, the five independent trials refer to evaluation rollouts of each trained policy. Additionally, we demonstrate that the MAPO-trained policy can serve as an effective teacher for distillation. Figure 1 also shows a real-world deployment of the student policy. The robot successfully tracks a wide range of reference motions, including challenging ones that require precise balance, such as squatting and bowing. Videos can be found on our project website.

Conclusion

In this study, we identified the divergence misestimation phenomenon that arises when MoE policies are trained with policy-gradient algorithms. To avoid this misestimation, we proposed MAPO, which alternately optimizes the router and experts, enabling more effective training and better tracking performance for humanoid whole-body control. The resulting policy performs complex whole-body motions well and is distilled into a deployable student policy on a real humanoid robot.

Limitations

While MAPO enables effective training of MoE policies with policy-gradient algorithms, further encouraging expert specialization remains an open question. One potential direction is to introduce an auxiliary loss that promotes discriminability between expert actions, inspired by prior works on unsupervised skill discovery [65, 66, 67, 68, 69]. Another limitation is that our analysis assumes dense softmax routing with σ\sigma fixed within each update step. The analysis does not cover learned per-expert variances or top-kk sparse routing. Nevertheless, divergence misestimation may also occur in these settings.

Supplementary Material

Implicit Trust Region from clipping

Let rt=1+δr_t = 1+\delta (∣δ∣≤ϵ|\delta| \le\epsilon), then the KL divergence between the new and old policies can be expressed as,

D(πold∥πnew)=E[log⁡1rt]=E[−log⁡(1+δ)].(8)D(\pi^{\mathrm{old}} \mathbin\Vert\pi^{\mathrm{new}}) = \mathbb{E}\left[\log\frac{1}{r_t}\right] = \mathbb{E}[-\log(1+\delta)]. \tag*{(8)}

Because log⁡(1+δ)≈δ−δ2/2\log(1+\delta) \approx\delta-\delta^2/2 for small δ\delta, we have,

D(πold∥πnew)≈E[−δ+δ2/2].(9)D(\pi^{\mathrm{old}} \mathbin\Vert\pi^{\mathrm{new}}) \approx\mathbb{E}\left[-\delta+\delta^2/2\right]. \tag*{(9)}

Because Ea∼πold(⋅∣s)[δ]=∫πΘ(a∣s) da−1=0\mathbb{E}_{a\sim\pi^{\mathrm{old}}(\cdot|s)}[\delta] = \int\pi_\Theta(a|s)\,da-1=0, the KL divergence can be approximated as follows:

D(πold∥πnew)≈E[δ2/2]=O(δ2)≤O(ϵ2).(10)D(\pi^{\mathrm{old}} \mathbin\Vert\pi^{\mathrm{new}}) \approx\mathbb{E}\left[\delta^2/2\right] = O(\delta^2) \leq O(\epsilon^2). \tag*{(10)}

It means the clipping of PPO implicitly constructs a trust region by constraining the KL divergence to O(ϵ2)O(\epsilon^2).

Bias Analysis of MAPO

Recall that the dense MoE policy can be modeled as πΘ(a∣s)=N(mΘ,σ2I)\pi_\Theta(a|s)=\mathcal{N}(m_\Theta,\sigma^2 I), where mΘ=∑eweμem_\Theta=\sum_e w_e\mu_e and Θ={ψ,θ1:N}\Theta=\{\psi,\theta_{1:N}\}. The mixing weights w=[w1,w2,…,wN]\mathbf{w}=[w_1,w_2,\ldots,w_N] are obtained by applying a softmax to the output of the router. The ee-th expert outputs the deterministic mean μe=μe(s;θe)\mu_e=\mu_e(s;\theta_e). For simplicity, we assume that all experts share the same variance σ2\sigma^2.

Score Function Decomposition

Consider the score function decomposition for the policy gradient algorithm. Define h(s,a)h(s,a) as,

∇mlog⁡π(a∣s)=a−mσ2:=h(s,a).(11)\nabla_m\log\pi(a|s)=\frac{a-m}{\sigma^2}\mathrel{:=}h(s,a). \tag*{(11)}

Then,

∇Θlog⁡πΘ(a∣s)=∇mlog⁡π(a∣s)∇ΘmΘ(12)\nabla_\Theta\log\pi_\Theta(a|s)=\nabla_m\log\pi(a|s)\nabla_\Theta m_\Theta \tag*{(12)}
=h(s,a)∇ΘmΘ.(13)=h(s,a)\nabla_\Theta m_\Theta. \tag*{(13)}

The ∇Θ\nabla_\Theta consists of gradients for the parameters of the router and the experts, which can be expressed as,

∇ψmΘ=∑e∂we∂ψμe, and ∇θkmΘ=wk∂μk∂θk.(14)\nabla_\psi m_\Theta=\sum_e\frac{\partial w_e}{\partial\psi}\mu_e,\ \text{and}\ \nabla_{\theta_k}m_\Theta=w_k\frac{\partial\mu_k}{\partial\theta_k}. \tag*{(14)}

Therefore,

∇ψlog⁡πΘ(a∣s)=h∑e∂we∂ψμe, and ∇θklog⁡πΘ(a∣s)=hwk∂μk∂θk.(15)\nabla_\psi\log\pi_\Theta(a|s)=h\sum_e\frac{\partial w_e}{\partial\psi}\mu_e,\ \text{and}\ \nabla_{\theta_k}\log\pi_\Theta(a|s)=hw_k\frac{\partial\mu_k}{\partial\theta_k}. \tag*{(15)}

PPO Gradient

Because the gradient of PPO when it is clipped is zero, we only consider the case when it is not clipped. In this case, the PPO gradient is identical to the gradient of true importance sampling.

Recall,

LPPO=E^[min⁡(r(s,a)A^,clip⁡(r(s,a),1−ϵ,1+ϵ)A^)](16)L^{\mathrm{PPO}}=\hat{\mathbb{E}}\left[\min\left(r(s,a)\hat{A},\operatorname{clip}(r(s,a),1-\epsilon,1+\epsilon)\hat{A}\right)\right] \tag*{(16)}

where r(s,a)=πΘ(a∣s)/πold(a∣s)r(s,a)=\pi_\Theta(a|s)/\pi^{\mathrm{old}}(a|s).

Notably,

∇ΘLPPO=E^[∇Θr(s,a)A^]=E^[r(s,a)h(s,a)∇Θm(s;Θ)A^](17)\nabla_\Theta L^{\mathrm{PPO}}=\hat{\mathbb{E}}\left[\nabla_\Theta r(s,a)\hat{A}\right]=\hat{\mathbb{E}}\left[r(s,a)h(s,a)\nabla_\Theta m(s;\Theta)\hat{A}\right] \tag*{(17)}

MAPO Gradient

Let, m1(s;ψ)=∑ewe(s,ψ)μeold(s)m_1(s;\psi)=\sum_e w_e(s,\psi)\mu_e^{\mathrm{old}}(s) and m2(s;θ1:N)=∑eweold(s)μe(s;θ1:N)m_2(s;\theta_{1:N})=\sum_e w_e^{\mathrm{old}}(s)\mu_e(s;\theta_{1:N}). Then, we can define the following quantities,

π1=N(m1,σ2I),r1=π1πold,h1=(a−m1)σ2(18)\pi_1=\mathcal{N}(m_1,\sigma^2 I),\quad r_1=\frac{\pi_1}{\pi^{\mathrm{old}}},\quad h_1=\frac{(a-m_1)}{\sigma^2} \tag*{(18)}
π2=N(m2,σ2I),r2=π2πold,h2=(a−m2)σ2.(19)\pi_2=\mathcal{N}(m_2,\sigma^2 I),\quad r_2=\frac{\pi_2}{\pi^{\mathrm{old}}},\quad h_2=\frac{(a-m_2)}{\sigma^2}. \tag*{(19)}

Similar to the PPO case, we only consider the case when the MAPO gradient is not clipped. Then, the MAPO loss function can be expressed as,

LMAPO=E^[r1(s,a)A^]+E^[r2(s,a)A^].(20)L^{\mathrm{MAPO}}=\hat{\mathbb{E}}\left[r_1(s,a)\hat{A}\right]+\hat{\mathbb{E}}\left[r_2(s,a)\hat{A}\right]. \tag*{(20)}

Then the gradient of MAPO with respect to each parameter can be expressed as,

∇ψLMAPO=E^[r1h1(s,a)∇ψm1(s;ψ)A^](21)\nabla_{\psi}L^{\mathrm{MAPO}}=\hat{\mathbb{E}}\left[r_1h_1(s,a)\nabla_{\psi}m_1(s;\psi)\hat{A}\right] \tag*{(21)}
∇θkLMAPO=E^[r2h2(s,a)∇θkm2(s;θ1:N)A^],(22)\nabla_{\theta_k}L^{\mathrm{MAPO}}=\hat{\mathbb{E}}\left[r_2h_2(s,a)\nabla_{\theta_k}m_2(s;\theta_{1:N})\hat{A}\right], \tag*{(22)}

where ∇ψm1(s;ψ)=∑e∂we∂ψμeold\nabla_{\psi}m_1(s;\psi)=\sum_e\frac{\partial w_e}{\partial\psi}\mu_e^{\mathrm{old}} and ∇θkm2(s;θ1:N)=wkold∂μk∂θk\nabla_{\theta_k}m_2(s;\theta_{1:N})=w_k^{\mathrm{old}}\frac{\partial\mu_k}{\partial\theta_k}.

Effect of Clipping

Because the clipping implicitly constrains the KL divergence induced by each of the two intermediate ratios r1,r2r_1,r_2 to O(ϵ2)O(\epsilon^2), it yields the following inequalities:

∥m1−mold∥σ=1σ∥∑e(we−weold)μeold∥≤ϵ(23)\frac{\lVert m_1-m_{\mathrm{old}}\rVert}{\sigma}=\frac{1}{\sigma}\left\lVert\sum_e(w_e-w_e^{\mathrm{old}})\mu_e^{\mathrm{old}}\right\rVert\leq\epsilon \tag*{(23)}
∥m2−mold∥σ=1σ∥∑e(μe−μeold)weold∥≤ϵ.(24)\frac{\lVert m_2-m_{\mathrm{old}}\rVert}{\sigma}=\frac{1}{\sigma}\left\lVert\sum_e(\mu_e-\mu_e^{\mathrm{old}})w_e^{\mathrm{old}}\right\rVert\leq\epsilon. \tag*{(24)}

Bias Analysis

There are two sources of bias in the MAPO gradient. The first is the bias in the mean gradient term, ∇m\nabla m. The second is the bias in the importance sampling ratio, rr, and score factor hh. As we will show, the second source is of the same order as the surrogate approximation error of PPO and does not change the qualitative picture; we focus first on the bias in the mean gradient term.

Mean Gradient Bias in Expert Parameters: We have,

∇θkmΘ=wk∂μk∂θk(for PPO)(25)\nabla_{\theta_k}m_{\Theta}=w_k\frac{\partial\mu_k}{\partial\theta_k}\qquad\text{(for PPO)} \tag*{(25)}
∇θkm2=wkold∂μk∂θk(for MAPO).(26)\nabla_{\theta_k}m_2=w_k^{\mathrm{old}}\frac{\partial\mu_k}{\partial\theta_k}\qquad\text{(for MAPO)}. \tag*{(26)}

Therefore, the MAPO gradient on each expert block is the true importance sampling gradient on that block scaled by a per-expert positive factor wkold/wkw_k^{\mathrm{old}}/w_k. As these scalars differ across experts, the aggregate gradient direction in θ1:N\theta_{1:N}-space is not preserved in general; however, each per-expert block vanishes exactly when its true counterpart does, therefore the stationary points of MAPO coincide with those of the true importance sampling gradient.

Mean Gradient Bias in Router Parameters: The bias in the mean gradient term for the router parameters can be expressed as,

bψmean=∑e∂we∂ψ(μe−μeold).(27)b_{\psi}^{\mathrm{mean}}=\sum_e\frac{\partial w_e}{\partial\psi}(\mu_e-\mu_e^{\mathrm{old}}). \tag*{(27)}

Notably, unlike the expert parameters, the MAPO gradient with respect to the router parameters can have a directional bias. The following cancellation shows that this bias depends only on the relative deviations between experts, not on their absolute magnitudes.

Because the sum of the mixing weights is 1, we have,

∑e∂we∂ψ=∂∂ψ∑ewe=0.(28)\sum_{e}\frac{\partial w_e}{\partial\psi} = \frac{\partial}{\partial\psi}\sum_{e}w_e = 0. \tag*{(28)}

Then, for any constant ξˉ\bar{\xi} independent of the expert index ee, we have

∑e∂we∂ψ(μe−μeold)=∑e∂we∂ψ[(μe−μeold)−ξˉ],(29)\sum_{e}\frac{\partial w_e}{\partial\psi}(\mu_e-\mu_e^{\mathrm{old}})=\sum_{e}\frac{\partial w_e}{\partial\psi}\left[(\mu_e-\mu_e^{\mathrm{old}})-\bar{\xi}\right], \tag*{(29)}

because ∑e∂we∂ψξˉ=ξˉ∑e∂we∂ψ=0\sum_{e}\frac{\partial w_e}{\partial\psi}\bar{\xi}=\bar{\xi}\sum_{e}\frac{\partial w_e}{\partial\psi}=0.

We define

ξe=(μe−μeold),andξˉ=1N∑eξe=1N∑e(μe−μeold),(30)\xi_e=(\mu_e-\mu_e^{\mathrm{old}}),\quad\text{and}\quad\bar{\xi}=\frac{1}{N}\sum_{e}\xi_e=\frac{1}{N}\sum_{e}(\mu_e-\mu_e^{\mathrm{old}}), \tag*{(30)}

then, we have

bψmean=∑e∂we∂ψ(ξe−ξˉ).(31)b_{\psi}^{\mathrm{mean}}=\sum_{e}\frac{\partial w_e}{\partial\psi}(\xi_e-\bar{\xi}). \tag*{(31)}

Therefore, the bias depends on the relative difference between each expert, rather than the absolute value of the difference. In particular, when all the experts are updated coherently, ξe−ξˉ≈0\xi_e-\bar{\xi}\approx0 and the bias vanishes.

Bound on the Rotation Angle on the Router Gradient: We now show that the directional discrepancy between the MAPO router gradient g^:=∇ψm1\hat{g}:=\nabla_{\psi}m_1 and the true mean gradient g:=∇ψmΘg:=\nabla_{\psi}m_{\Theta} is also controlled by the inter-expert variance. Within a single update step, the trust region in ψ\psi keeps the router parameters close to their old values, and the router Jacobian is approximately unchanged: ∂we/∂ψ≈∂weold/∂ψ\partial w_e/\partial\psi\approx\partial w_e^{\mathrm{old}}/\partial\psi. Under this approximation, the gradient discrepancy reduces to the mean-gradient bias,

g−g^≈bψmean.(32)g-\hat{g}\approx b_{\psi}^{\mathrm{mean}}. \tag*{(32)}

Decomposing bψmeanb_{\psi}^{\mathrm{mean}} into components parallel and perpendicular to g^\hat{g}, the angle θ\theta between gg and g^\hat{g} satisfies

sin⁡θ≲∥bψmean∥∥g^∥,(33)\sin\theta\lesssim\frac{\left\|b_{\psi}^{\mathrm{mean}}\right\|}{\left\|\hat{g}\right\|}, \tag*{(33)}

where the ≲\lesssim absorbs only the first-order Jacobian approximation above.

Applying the Cauchy–Schwarz inequality to (31),

∥bψmean∥≤∑e∥∂we∂ψ∥2⋅∑e∥ξe−ξˉ∥2.(34)\left\|b_{\psi}^{\mathrm{mean}}\right\|\leq\sqrt{\sum_{e}\left\|\frac{\partial w_e}{\partial\psi}\right\|^2}\cdot\sqrt{\sum_{e}\left\|\xi_e-\bar{\xi}\right\|^2}. \tag*{(34)}

Because g^=∑e(∂weold/∂ψ) μeold\hat{g}=\sum_{e}(\partial w_e^{\mathrm{old}}/\partial\psi)\,\mu_e^{\mathrm{old}} shares the same router Jacobian as the first factor above, that factor cancels to leading order in the ratio ∥bψmean∥/∥g^∥\left\|b_{\psi}^{\mathrm{mean}}\right\|/\left\|\hat{g}\right\|. The rotation angle is therefore governed primarily by the inter-expert variance ∑e∥ξe−ξˉ∥2\sum_{e}\left\|\xi_e-\bar{\xi}\right\|^2, not by the absolute expert deviations or the router sensitivity. Whether this variance remains small in practice depends on the training dynamics and is not guaranteed by the trust-region constraint ∥∑eweoldξe∥≤ϵσ\left\|\sum_{e}w_e^{\mathrm{old}}\xi_e\right\|\leq\epsilon\sigma from Equation (24) alone. Figure B.1 shows the changes in the router-gradient angle during training. The router-gradient angle remains below 90∘90^\circ throughout training.

Figure B.1. Router-gradient angle.

Table C.1: Reward functions for teacher training. Superscripts pp and gg denote proprioception and goal states, respectively. oo, pp, vv, ω\omega, aa, qq, and FF are orientation, position, linear velocity, angular velocity, policy action, joint position, and contact force. Subscripts specify the reference point: root for the pelvis link, body for tracked body links in B\mathcal{B}, rel-body for body link quantities expressed relative to the root frame, and last for the previous timestep. I[⋅]\mathbb{I}[\cdot] is the indicator function.

Reward termEquationWeight
Root orientationexp⁡(−∥orootp−orootg∥22/0.42)\exp\left(-\lVert o_{\mathrm{root}}^{p}-o_{\mathrm{root}}^{g}\rVert_{2}^{2}/0.4^{2}\right)1.0
Relative body-link positionexp⁡(−1∣B∣∑b∈B∥prel−body,bp−prel−body,bg∥22/0.052)\exp\left(-\frac{1}{|\mathcal{B}|}\sum_{b\in\mathcal{B}}\lVert p_{\mathrm{rel-body},b}^{p}-p_{\mathrm{rel-body},b}^{g}\rVert_{2}^{2}/0.05^{2}\right)20.0
Relative body-link orientationexp⁡(−1∣B∣∑b∈B∥orel−body,bp−orel−body,bg∥22/0.42)\exp\left(-\frac{1}{|\mathcal{B}|}\sum_{b\in\mathcal{B}}\lVert o_{\mathrm{rel-body},b}^{p}-o_{\mathrm{rel-body},b}^{g}\rVert_{2}^{2}/0.4^{2}\right)1.0
Body-link linear velocityexp⁡(−1∣B∣∑b∈B∥vbody,bp−vbody,bg∥22/1.0)\exp\left(-\frac{1}{|\mathcal{B}|}\sum_{b\in\mathcal{B}}\lVert v_{\mathrm{body},b}^{p}-v_{\mathrm{body},b}^{g}\rVert_{2}^{2}/1.0\right)1.0
Body-link angular velocityexp⁡(−1∣B∣∑b∈B∥ωbody,bp−ωbody,bg∥22/3.142)\exp\left(-\frac{1}{|\mathcal{B}|}\sum_{b\in\mathcal{B}}\lVert\omega_{\mathrm{body},b}^{p}-\omega_{\mathrm{body},b}^{g}\rVert_{2}^{2}/3.14^{2}\right)1.0
Action rate∥a−alast∥22\lVert a-a_{\mathrm{last}}\rVert_{2}^{2}-0.2
Joint limit∑jI[qj∉[qjmin⁡,qjmax⁡]]\sum_{j}\mathbb{I}\left[q_{j}\notin[q_{j}^{\min},q_{j}^{\max}]\right]-10.0
Undesired contact∑c∉{ankles, wrists}I[∥Fc∥>1.0N]\sum_{c\notin\{\mathrm{ankles},\ \mathrm{wrists}\}}\mathbb{I}[\lVert F_{c}\rVert>1.0\mathrm{N}]-0.1

Table C.1.

Difference in Importance Sampling Ratio and Score Factor: Because of the clipping, the deviations in the importance sampling ratio and score factor are of order O(ϵ)O(\epsilon),

∣ri−r∣≈O(ϵ), and ∣hi−h∣=∣mi−m∣/σ2≈O(ϵ)(i=1,2).(35)|r_i-r|\approx O(\epsilon),\ \text{and}\ |h_i-h|=|m_i-m|/\sigma^2\approx O(\epsilon)\quad(i=1,2). \tag*{(35)}

Decomposing the gradient difference as

rihi∇mi−rh∇m=rh(∇mi−∇m)+(rihi−rh)∇mi,r_i h_i\nabla m_i-rh\nabla m=rh\left(\nabla m_i-\nabla m\right)+\left(r_i h_i-rh\right)\nabla m_i,

the first term is the mean-gradient bias analyzed above, while the second contributes an additional O(ϵ)O(\epsilon) correction because rihi−rh=O(ϵ)r_i h_i-rh=O(\epsilon). Importantly, within the trust region r≈1r\approx1, therefore the leading rhrh pre-factor is itself close to the score hh, and the residual error from the (ri,hi)(r_i,h_i) deviation is on the same order as PPO’s surrogate approximation error and does not alter the qualitative conclusions of the mean-gradient analysis.

Training Details

Reward Function

The reward function for the teacher training is almost similar to the prior works [15, 17]. The specific reward functions are listed in Table C.1. We find that the sharper relative body link position reward encourages the robot to fully complete the reference motion, particularly for deep squatting and bowing. Without it, the robot tends to follow the reference motion only partially, stopping halfway through such motions, as shown in Figure C.1 (b). Notably, MAPO also achieves higher reward than PPO with MoE under the BeyondMimic reward setting (Figure C.1 (c)).

Comparison of reward settings and training performance

Figure C.1. Comparison of reward settings and training performance. (a) The reward setting used in this work. (b) The reward setting of BeyondMimic [15]. The reward settings differ mainly in the scale and sharpness of the relative body-link position reward. (c) Reward learning curves of MAPO and PPO with MoE under the BeyondMimic reward setting.

HyperparameterValue
Num parallel environments4096
Num steps per environment24
Num mini-batches4
Learning epochs5
Discount γ\gamma0.99
GAE λ\lambda0.95
Clip parameter ϵ\epsilon0.2
Entropy coefficient0.01
Learning rateAdaptive
Max grad norm1.0
Init noise std1.0
Shared actor encoder hidden layers[512, 256]
Shared encoder final feature dimension256
Router hidden layers[256, 128]
Expert hidden layers[128, 128]
Value function hidden layers[512, 256, 128]

Table C.2. Teacher policy training hyperparameters.

Training Hyperparameters

The training hyperparameters for the teacher policy are listed in Table C.2. The Adam [70] optimizer is used for optimization, and exponential linear units (ELUs) [71] are used as the activation function in all networks. Additionally, the teacher uses a shared encoder that extracts features from the observation, which are fed into both the router and experts [72, 73]. Notably, for the alternating optimization, we detach the gradients from the router to the shared encoder. For the student policy distillation, a fixed learning rate of 2×10−42 \times10^{-4} is used, and the student policy network adopts hidden layers of sizes [512, 256, 128].

Domain Randomization

To improve the robustness of the trained policy, we apply domain randomization during training. Details of the randomized parameters are listed in Table C.3.

Additional Experimental Results

Per-Expert Activation: Figure D.1 shows per-expert activation across different motion types. The router assigns higher mixing weights to different experts depending on the motion type, indicating expert specialization. Expert 2 (E2) is primarily activated during walking and upper-body motions

Domain Randomization ParameterValue
Joint P gain scaleU(0.75,1.25)×\mathcal{U}(0.75, 1.25) \times default
Joint D gain scaleU(0.75,1.25)×\mathcal{U}(0.75, 1.25) \times default
Root massU(0.75,1.25)+\mathcal{U}(0.75, 1.25) + default kg
Root center of mass offset (x, y, z)U([−0.1,0.1],[−0.1,0.1],[−0.1,0.1])\mathcal{U}([-0.1, 0.1], [-0.1, 0.1], [-0.1, 0.1]) m
Control delayU(0,60)\mathcal{U}(0, 60) ms
Push robot magnitudeU(−1.0,1.0)\mathcal{U}(-1.0, 1.0) m/s
Push robot interval5 s
Reference motion offset (x, y, z)U([−0.02,0.02],[−0.02,0.02],[−0.1,0.1])\mathcal{U}([-0.02, 0.02], [-0.02, 0.02], [-0.1, 0.1]) m

Table C.3. Domain randomization parameters. U(a,b)\mathcal{U}(a,b) denotes a uniform distribution over the interval [a,b][a,b].

Per-expert activation according to the motion type

Figure D.1. Per-expert activation according to the motion type.

while standing (Figure D.1 (a)). E4 and E5 support E2 during different phases of walking. Although E1 appears redundant in Figure 4 (c), E1 is activated for motions that require a squatting posture (Figure D.1 (b)). E3 and E6 are activated during balance recovery and unavoidable falls, respectively (Figure D.1 (c, d)). Because we do not manually curate the dataset, some reference motions may be infeasible for the robot. Concentrating activation during unavoidable falls in E6 may reduce interference from infeasible motions during the training of the remaining experts.

Effect of Divergence Misestimation: We measure divergence misestimation during training with vanilla policy gradient because clipping can obscure the magnitude of misestimation. Figure D.2 shows the mean and percentiles of overestimation and underestimation magnitudes. We also evaluate the fractions of updates that would be clipped if the PPO or MAPO criteria were applied during training with vanilla policy gradient (Figure D.3 (a)). The MAPO router criterion would clip only about 5% of updates. In contrast, if the PPO criterion were applied to the same updates, approximately 50% of updates would be clipped. The higher clipping fraction for router updates is consistent with the weaker expert specialization of PPO with MoE (Figure 4 (b, Figure 4c)).

Mean and percentiles of divergence misestimation magnitudes

Figure D.2. Mean and percentiles of divergence misestimation magnitudes. (a) Overestimation. (b) Underestimation.

A similar pattern appears during MAPO training (Figure D.3 (c, d)). The MAPO router criterion clips only about 2% of updates, compared with approximately 24% if the PPO criterion were applied. Notably, the PPO criterion yields similar clipping fractions during MAPO training and during training of PPO with MoE (Figure D.3 (d)). The results suggest that excessive clipping of router updates is prevented during MAPO training.

Seed Variance: The results in Table 1 and Figure 4 are based on a single training seed. We further evaluate MAPO and PPO with MoE across three training seeds.

Figure D.3. (a) Fractions of updates that would be clipped under the MAPO and PPO criteria during training with vanilla policy gradient. (b, c) Clipped update fractions under the MAPO expert and router criteria, respectively, during MAPO training. (d) Clipping fractions under the PPO criterion during MAPO training (hypothetical clipping) and PPO with MoE training (actual clipping).

The learning curves are shown in Figure D.4. Across the three seeds, MAPO achieves a success rate of (87.03±0.64)%(87.03 \pm0.64)\% and a global body position error Eg-posE_{\mathrm{g\text{-}pos}} of 147.9±1.5147.9 \pm1.5 mm. The corresponding results for PPO with MoE are (82.89±0.43)%(82.89 \pm0.43)\% and 154.1±1.7154.1 \pm1.7 mm. Success rates and tracking errors are reported as mean ±\pm standard deviation across the three training seeds.

Figure D.4. Learning curves of MAPO and PPO with MoE across three training seeds.

Additionally, we examine whether the difference in tracking success between MAPO and PPO with MoE is consistent across training seeds. For each seed, we count motions on which MAPO succeeds and PPO with MoE fails in all five evaluation rollouts. The counts are 182, 155, and 191 for the three training seeds, respectively. Conversely, PPO with MoE consistently succeeds while MAPO consistently fails on only 1, 1, and 0 motions, respectively.

References

  1. [1]I. Radosavovic, T. Xiao, B. Zhang, T. Darrell, J. Malik, and K. Sreenath. Real-world humanoid locomotion with reinforcement learning. Science Robotics, 9(89):eadi9579, 2024.DOI
  2. [2]J. Long, J. Ren, M. Shi, Z. Wang, T. Huang, P. Luo, and J. Pang. Learning humanoid locomotion with perceptive internal model. In Proc. IEEE International Conference on Robotics and Automation, 2025.arxiv.org/abs/2411.14386
  3. [3]X. Gu, Y.-J. Wang, X. Zhu, C. Shi, Y. Guo, Y. Liu, and J. Chen. Advancing humanoid locomotion: Mastering challenging terrains with denoising world model learning. arXiv preprint arXiv:2408.14472, 2024.arxiv.org/abs/2408.14472
  4. [4]I. Radosavovic, S. Kamat, T. Darrell, and J. Malik. Learning humanoid locomotion over challenging terrain. arXiv preprint arXiv:2410.03654, 2024.arxiv.org/abs/2410.03654
  5. [5]Z. Zhuang, S. Yao, and H. Zhao. Humanoid parkour learning. In Proc. Conference on Robot Learning, 2025.arxiv.org/abs/2406.10759
  6. [6]S. Zhu, Z. Zhuang, M. Zhao, K.-Y. Lee, and H. Zhao. Hiking in the wild: A scalable perceptive parkour framework for humanoids. arXiv preprint arXiv:2601.07718, 2026.arxiv.org/abs/2601.07718
  7. [7]T. Huang, J. Ren, H. Wang, Z. Wang, Q. Ben, M. Wen, X. Chen, J. Li, and J. Pang. Learning humanoid standing-up control across diverse postures. arXiv preprint arXiv:2502.08378, 2025.arxiv.org/abs/2502.08378
  8. [8]X. He, R. Dong, Z. Chen, and S. Gupta. Learning getting-up policies for real-world humanoid robots. arXiv preprint arXiv:2502.12152, 2025.arxiv.org/abs/2502.12152
  9. [9]N. Mahmood, N. Ghorbani, N. F. Troje, G. Pons-Moll, and M. J. Black. AMASS: Archive of motion capture as surface shapes. In Proc. IEEE/CVF International Conference on Computer Vision, 2019.arxiv.org/abs/1904.03278
  10. [10]F. G. Harvey, M. Yurick, D. Nowrouzezahrai, and C. Pal. Robust motion in-betweening. ACM Transactions on Graphics, 39(4):60:1–60:12, 2020.DOI
  11. [11]J. Li, J. Wu, and C. K. Liu. Object motion guided human motion synthesis. ACM Transactions on Graphics, 42(6):1–11, 2023.arxiv.org/abs/2309.16237
  12. [12]X. B. Peng, P. Abbeel, S. Levine, and M. Van de Panne. DeepMimic: Example-guided deep reinforcement learning of physics-based character skills. ACM Transactions on Graphics, 37 (4):1–14, 2018.
  13. [13]X. Cheng, Y. Ji, J. Chen, R. Yang, G. Yang, and X. Wang. Expressive whole-body control for humanoid robots. arXiv preprint arXiv:2402.16796, 2024.arxiv.org/abs/2402.16796
  14. [14]M. Ji, X. Peng, F. Liu, J. Li, G. Yang, X. Cheng, and X. Wang. ExBody2: Advanced expressive humanoid whole-body control. In Proc. Robotics: Science and Systems Workshop on Whole-body Control and Bimanual Manipulation: Applications in Humanoids and Beyond, 2025.arxiv.org/abs/2412.13196
  15. [15]Q. Liao, T. E. Truong, X. Huang, Y. Gao, G. Tevet, K. Sreenath, and C. K. Liu. Beyond-Mimic: From motion tracking to versatile humanoid control via guided diffusion. arXiv preprint arXiv:2508.08241, 2025.arxiv.org/abs/2508.08241
  16. [16]Y. Ze, Z. Chen, J. P. Araujo, Z.-a. Cao, X. B. Peng, J. Wu, and K. Liu. TWIST: Teleoperated whole-body imitation system. In Proc. Conference on Robot Learning, 2025.arxiv.org/abs/2505.02833
  17. [17]Z. Luo, Y. Yuan, T. Wang, C. Li, S. Chen, F. Castaneda, Z.-A. Cao, J. Li, D. Minor, Q. Ben, et al. SONIC: Supersizing motion tracking for natural humanoid whole-body control. arXiv preprint arXiv:2511.07820, 2025.DOI
  18. [18]T. He, Z. Luo, W. Xiao, C. Zhang, K. Kitani, C. Liu, and G. Shi. Learning human-to-humanoid real-time whole-body teleoperation. In Proc. IEEE/RSJ International Conference on Intelligent Robots and Systems, 2024.arxiv.org/abs/2403.04436
  19. [19]Z. Chen, M. Ji, X. Cheng, X. Peng, X. B. Peng, and X. Wang. GMT: General motion tracking for humanoid whole-body control. arXiv preprint arXiv:2506.14770, 2025.arxiv.org/abs/2506.14770
  20. [20]Z. Wang, J. Wang, S. Yao, Y. Zhang, Z. Ding, M. Yang, Y. Wang, H. Jiang, C. Ma, X. Shi, et al. General humanoid whole-body control via pretraining and fast adaptation. arXiv preprint arXiv:2602.11929, 2026.arxiv.org/abs/2602.11929
  21. [21]T. Yu, S. Kumar, A. Gupta, S. Levine, K. Hausman, and C. Finn. Gradient surgery for multi-task learning. In Proc. Advances in Neural Information Processing Systems, 2020.arxiv.org/abs/2001.06782
  22. [22]B. Liu, X. Liu, X. Jin, P. Stone, and Q. Liu. Conflict-averse gradient descent for multi-task learning. In Proc. Advances in Neural Information Processing Systems, 2021.arxiv.org/abs/2110.14048
  23. [23]T. Schaul, D. Borsa, J. Modayil, and R. Pascanu. Ray interference: a source of plateaus in deep reinforcement learning. arXiv preprint arXiv:1904.11455, 2019.arxiv.org/abs/1904.11455
  24. [24]R. A. Jacobs, M. I. Jordan, S. J. Nowlan, and G. E. Hinton. Adaptive mixtures of local experts. Neural Computation, 3(1):79–87, 1991.DOI
  25. [25]T. Willi, J. S. O. Ceron, J. N. Foerster, G. K. Dziugaite, and P. S. Castro. Mixture of experts in a mixture of RL settings. Reinforcement Learning Journal, 3:1072–1105, 2024.arxiv.org/abs/2406.18420
  26. [26]A. Hendawy, J. Peters, and C. D’Eramo. Multi-task reinforcement learning with mixture of orthogonal experts. In Proc. International Conference on Learning Representations, 2024.arxiv.org/abs/2311.11385
  27. [27]G. Cheng, L. Dong, W. Cai, and C. Sun. Multi-task reinforcement learning with attention-based mixture of experts. IEEE Robotics and Automation Letters, 8(6):3812–3819, 2023.DOI
  28. [28]R. Yang, H. Xu, Y. Wu, and X. Wang. Multi-task reinforcement learning with soft modularization. In Proc. Advances in Neural Information Processing Systems, 2020.arxiv.org/abs/2003.13661
  29. [29]R. Huang, S. Zhu, Y. Du, and H. Zhao. MoE-Loco: Mixture of experts for multitask locomotion. In Proc. IEEE/RSJ International Conference on Intelligent Robots and Systems, 2025.arxiv.org/abs/2503.08564
  30. [30]J. Han, W. Xie, J. Zheng, J. Shi, W. Zhang, T. Xiao, and C. Bai. KungfuBot2: Learning versatile motion skills for humanoid whole-body control. arXiv preprint arXiv:2509.16638, 2025.arxiv.org/abs/2509.16638
  31. [31]Y. Li, Y. Lin, J. Cui, T. Liu, W. Liang, Y. Zhu, and S. Huang. CLONE: Closed-loop whole-body humanoid teleoperation for long-horizon tasks. In Proc. Conference on Robot Learning, 2025.arxiv.org/abs/2506.08931
  32. [32]J. Schulman, S. Levine, P. Abbeel, M. Jordan, and P. Moritz. Trust region policy optimization. In Proc. International Conference on Machine Learning, 2015.arxiv.org/abs/1502.05477
  33. [33]J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.arxiv.org/abs/1707.06347
  34. [34]N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. Le, G. Hinton, and J. Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In Proc. International Conference on Learning Representations, 2017.arxiv.org/abs/1701.06538
  35. [35]D. Lepikhin, H. Lee, Y. Xu, D. Chen, O. Firat, Y. Huang, M. Krikun, N. Shazeer, and Z. Chen. GShard: Scaling giant models with conditional computation and automatic sharding. In Proc. International Conference on Learning Representations, 2021.
  36. [36]W. Fedus, B. Zoph, and N. Shazeer. Switch Transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1–39, 2022.arxiv.org/abs/2101.03961
  37. [37]Y. Zhou, T. Lei, H. Liu, N. Du, Y. Huang, V. Zhao, A. M. Dai, Q. V. Le, J. Laudon, et al. Mixture-of-Experts with expert choice routing. In Proc. Advances in Neural Information Processing Systems, 2022.arxiv.org/abs/2202.09368
  38. [38]C. Riquelme, J. Puigcerver, B. Mustafa, M. Neumann, R. Jenatton, A. Susano Pinto, D. Keysers, and N. Houlsby. Scaling vision with sparse mixture of experts. In Proc. Advances in Neural Information Processing Systems, 2021.arxiv.org/abs/2106.05974
  39. [39]J. Ren, Y. Li, Z. Ding, W. Pan, and H. Dong. Probabilistic mixture-of-experts for efficient deep reinforcement learning. arXiv preprint arXiv:2104.09122, 2021.arxiv.org/abs/2104.09122
  40. [40]R. Akrou​r, D. Tateo, and J. Peters. Continuous action reinforcement learning from a mixture of interpretable experts. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44 (10):6795–6806, 2022.
  41. [41]X. B. Peng, G. Berseth, and M. Van de Panne. Terrain-adaptive locomotion skills using deep reinforcement learning. ACM Transactions on Graphics, 35(4):1–12, 2016.
  42. [42]J. Puigcerver, C. R. Ruiz, B. Mustafa, and N. Houlsby. From sparse to soft mixtures of experts. In Proc. International Conference on Learning Representations, 2024.arxiv.org/abs/2308.00951
  43. [43]B. Yang, G. Bender, Q. V. Le, and J. Ngi​am. CondConv: Conditionally parameterized convolutions for efficient inference. In Proc. Advances in Neural Information Processing Systems, 2019.arxiv.org/abs/1904.04971
  44. [44]Z. Tian, C. Shen, and H. Chen. Conditional convolutions for instance segmentation. In Proc. European Conference on Computer Vision, 2020.arxiv.org/abs/2003.05664
  45. [45]M. Muqeeth, H. Liu, and C. Raffel. Soft merging of experts with adaptive routing. Transactions on Machine Learning Research, 2024.arxiv.org/abs/2306.03745
  46. [46]J. Obando-Ceron, G. Sokar, T. Willi, C. Lyle, J. Farebrother, J. Foerster, K. Dziugaite, D. Precup, and P. S. Castro. Mixtures of experts unlock parameter scaling for deep RL. In Proc. International Conference on Machine Learning, 2024.
  47. [47]S. Jeon, M. Jung, S. Choi, B. Kim, and J. Hwangbo. Learning whole-body manipulation for quadrupedal robot. IEEE Robotics and Automation Letters, 9(1):699–706, 2024.arxiv.org/abs/2308.16820
  48. [48]Z. He, K. Lei, Y. Ze, K. Sreenath, Z. Li, and H. Xu. Learning visual quadrupedal locomotion from demonstrations. In Proc. IEEE/RSJ International Conference on Intelligent Robots and Systems, 2024.arxiv.org/abs/2403.20328
  49. [49]Z. Fu, X. Cheng, and D. Pathak. Deep whole-body control: Learning a unified policy for manipulation and locomotion. In Proc. Conference on Robot Learning, 2023.arxiv.org/abs/2210.10044
  50. [50]T. Portela, A. Cramariuc, M. Mittal, and M. Hutter. Whole-body end-effector pose tracking. In Proc. IEEE International Conference on Robotics and Automation, 2025.
  51. [51]S. Lim, H. Lee, S. Lee, I. M. A. Nahrendra, and H. Myung. OpenHEART: Opening heterogeneous articulated objects with a legged manipulator. In Proc. IEEE International Conference on Robotics and Automation, 2026.arxiv.org/abs/2603.05830
  52. [52]S. Lim, H. Lee, S. Lee, and H. Myung. Learning robust whole-body control for legged manipulation via learned terrain estimation. In Proc. International Conference on Control, Automation, and Systems, 2024.DOI
  53. [53]A. E. Vijayan, A. Cramariuc, M. Risiglione, C. Gehring, and M. Hutter. Multi-critic learning for whole-body end-effector twist tracking. In Proc. Conference on Robot Learning, 2025.arxiv.org/abs/2507.08656
  54. [54]Z. Luo, J. Cao, A. Winkler, K. Kitani, and W. Xu. Perpetual humanoid control for real-time simulated avatars. In Proc. IEEE/CVF International Conference on Computer Vision, 2023.arxiv.org/abs/2305.06456
  55. [55]T. He, Z. Luo, X. He, W. Xiao, C. Zhang, W. Zhang, K. M. Kitani, C. Liu, and G. Shi. OmniH2O: Universal and dexterous human-to-humanoid whole-body teleoperation and learning. In Proc. Conference on Robot Learning, 2025.arxiv.org/abs/2406.08858
  56. [56]T. He, W. Xiao, T. Lin, Z. Luo, Z. Xu, Z. Jiang, J. Kautz, C. Liu, G. Shi, X. Wang, et al. HOVER: Versatile neural whole-body controller for humanoid robots. In Proc. IEEE International Conference on Robotics and Automation, 2025.arxiv.org/abs/2410.21229
  57. [57]J. Li, X. Cheng, T. Huang, S. Yang, R.-Z. Qiu, and X. Wang. AMO: Adaptive motion optimization for hyper-dexterous humanoid whole-body control. arXiv preprint arXiv:2505.03738, 2025.arxiv.org/abs/2505.03738
  58. [58]Y. Ze, S. Zhao, W. Wang, A. Kanazawa, R. Duan, P. Abbeel, G. Shi, J. Wu, and C. K. Liu. TWIST2: Scalable, portable, and holistic humanoid data collection system. arXiv preprint arXiv:2511.02832, 2025.arxiv.org/abs/2511.02832
  59. [59]J. P. Araujo, Y. Ze, P. Xu, J. Wu, and C. K. Liu. Retargeting matters: General motion retargeting for humanoid motion tracking. arXiv preprint arXiv:2510.02252, 2025.arxiv.org/abs/2510.02252
  60. [60]R. S. Sutton and A. G. Barto. Reinforcement Learning: An Introduction. The MIT Press, 1998.DOI
  61. [61]S. Ross, G. Gordon, and D. Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Proc. International Conference on Artificial Intelligence and Statistics, 2011.arxiv.org/abs/1011.0686
  62. [62]M. Mittal, P. Roth, J. Tigue, A. Richard, O. Zhang, P. Du, A. Serrano-Munoz, X. Yao, R. Zurbrügg, N. Rudin, et al. Isaac Lab: A GPU-accelerated simulation framework for multi-modal robot learning. arXiv preprint arXiv:2511.04831, 2025.arxiv.org/abs/2511.04831
  63. [63]Unitree Robotics. Unitree G1. https://www.unitree.com/g1. Accessed: 2026-05-22.
  64. [64]N. Rudin, D. Hoeller, P. Reist, and M. Hutter. Learning to walk in minutes using massively parallel deep reinforcement learning. In Proc. Conference on Robot Learning, 2022.arxiv.org/abs/2109.11978
  65. [65]B. Eysenbach, A. Gupta, J. Ibarz, and S. Levine. Diversity is all you need: Learning skills without a reward function. In Proc. International Conference on Learning Representations, 2019.arxiv.org/abs/1802.06070
  66. [66]S. Park, O. Rybkin, and S. Levine. METRA: Scalable unsupervised RL with metric-aware abstraction. In Proc. International Conference on Learning Representations, 2024.arxiv.org/abs/2310.08887
  67. [67]D. Strouse, K. Baumli, D. Warde-Farley, V. Mnih, and S. S. Hansen. Learning more skills through optimistic exploration. In Proc. International Conference on Learning Representations, 2022.arxiv.org/abs/2107.14226
  68. [68]J. Hu, Z. Wang, P. Stone, and R. Martín-Martín. Disentangled unsupervised skill discovery for efficient hierarchical reinforcement learning. In Proc. Advances in Neural Information Processing Systems, 2024.arxiv.org/abs/2410.11251
  69. [69]R. Cathomen, M. Mittal, M. Vlastelica, and M. Hutter. Divide, discover, deploy: Factorized skill learning with symmetry and style priors. In Proc. Conference on Robot Learning, 2025.
  70. [70]D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In Proc. International Conference on Learning Representations, 2015.arxiv.org/abs/1412.6980
  71. [71]D.-A. Clevert, T. Unterthiner, and S. Hochreiter. Fast and accurate deep network learning by exponential linear units (ELUs). In Proc. International Conference on Learning Representations, 2016.arxiv.org/abs/1511.07289
  72. [72]S. Lee, S. Purushwalkam, M. Cogswell, D. Crandall, and D. Batra. Why M heads are better than one: Training a diverse ensemble of deep networks. arXiv preprint arXiv:1511.06314, 2015.arxiv.org/abs/1511.06314
  73. [73]C. D’Eramo, D. Tateo, A. Bonarini, M. Restelli, and J. Peters. Sharing knowledge in multi-task deep reinforcement learning. In Proc. International Conference on Learning Representations, 2020.

Paper details

vlnc:W_tkvzcxvk
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents