betaFeedback

WorksComputer Science

QTOP: Scaling Humanoid Motion Tracking with Quantile Critics

Abstract

Learning human motion tracking policies through reinforcement learning (RL) has emerged as a core capability that underpins a majority of recent progress in humanoid whole-body control. However, achieving generalization to unseen motions at high fidelity requires jointly scaling training to thousands of GPU and dataset hours. This work hypothesizes that motion tracking is inherently compute-bound because training on a diverse set of motions introduces learning interference, similar to multi-task RL. To tackle this, we propose Quantile Tracking Optimization (QTOP), which improves sample efficiency by introducing a distributional advantage estimate conditioned on state and reference pairs. QTOP outperforms on and off-policy RL baselines both in simulation and real-world tracking scenarios on multiple datasets by up to 10%. Furthermore, we establish a scaling law for QTOP and find that, aside from scaling predictably, dataset size has minimal impact on performance for moderately sized GPU hour regimes.

Introduction

Humanoid motion tracking aims to learn policies conditioned on reference motions, allowing humanoid robots to generate motor commands that faithfully reproduce diverse behaviors in real time. Over the past few years, motion tracking has rapidly evolved from learning individual skills [1], [2] to training generalist policies on thousands of motions [3], [4], [5], [6] with remarkable results on real robot hardware. A key enabler of modern trackers is the utilization of human motion datasets as a rich source of behavioral supervision as well as massively parallelized RL in simulation to converge to a stable and general controller.

As a result, the central challenge is no longer learning individual motions, but optimizing a single policy over a heterogeneous collection of tracking tasks. While the desired behavior is explicitly specified through reference frames at high frequency, different motions can induce distinct state visitation and return distributions, making optimization challenging [7]. Conventional RL algorithms, such as PPO, typically aggregate these diverse learning signals through a single scalar value function, which can lead to overly conservative policies [8]. The resulting tracking errors, although often small, can accumulate over the course of a rollout and become particularly detrimental for loco-manipulation tasks, where accurate global position, orientation and timing are required to grasp objects with high dexterity. A common strategy to eliminate such tracking errors is to scale both motion datasets and simulation throughput [4], [6], [9]. Larger datasets expose policies to a broader range of behaviors, while increased compute enables faster convergence to robust imitation. This scaling recipe has proven highly effective, but it requires substantial compute resources of over 10410^{4} GPU hours, which is out of reach for most robotics researchers. Secondly, it does not address whether additional data and compute contributes equally to performance. We hypothesize in this work that general motion tracking is inherently multi-task, which makes the task compute-bound even at small dataset scales. To address this, we introduce Quantile Tracking Optimization (QTOP), a PPO-based training recipe for large-scale humanoid motion tracking that combines Muon optimization with a distributional value critic. Instead of assigning credit using only a scalar return residual, QTOP estimates the return distribution conditioned on each observation-reference pair and reinforces rollouts according to how favorable their realized returns are relative to this local distribution. This converts tracking difficulty across motions and contact regimes into a calibrated advantage signal, enabling more effective policy improvement.

Using QTOP, we study how motion-tracking performance scales jointly with training compute and motion-dataset size. We construct nested subsets of the same motion corpus for multiple seeds and train policies over a grid of dataset sizes and optimization budgets, allowing us to separate the marginal benefit of reference data from additional RL compute. Across the low-to-moderate data regime considered in our experiments, we find that increasing the dataset size improves performance only when paired with sufficient compute, while additional training compute remains the dominant bottleneck. Together with existing results at substantially larger data scales [6], our findings suggest that an important understudied direction is not only how to collect larger motion corpora, but how to design learning algorithms that can effectively exploit them.

Finally, we evaluate QTOP against strong PPO baselines and representative off-policy alternatives on both academic held-out tracking benchmarks and our own motion capture dataset. QTOP consistently achieves the best tracking performance and transfers to improved real-world tracking on a Unitree G1 under a fixed data and training budget. This points to learning algorithms that can more effectively exploit large reference-conditioned motion datasets as an important direction for future research.

We summarize our contributions as follows:

  • We identify multitask optimization as a central bottleneck in large-scale humanoid motion tracking.

  • We introduce QTOP, combining PPO, Muon optimization and a custom distributional critic for advantage estimation using return quantiles for scalable training.

  • We characterize the joint scaling of tracking performance with dataset size and compute. We show that QTOP continues to benefit from additional compute, whereas increasing dataset size yields comparatively smaller gains in practical regimes.

  • We find that QTOP consistently outperforms well tuned on- and off-policy RL methods on held-out datasets.

Motion Tracking

Learning physical controllers of humanoid characters is a deeply studied problem in computer graphics and robotics. Foundational works quickly evolved from basic RL formulation and single clip learning [10], [11], [12], to larger scale imitation [13]. This has sparked great interest in improving and deploying trackers for real robot hardware, yielding expressive and dynamic replay across a wide variety of motions and robots [14], [1], [15], [16], [3], [17], [5].

Nonetheless, learning a unified tracker that accurately imitates novel, unseen motions remains challenging. Recent works have increasingly turned to scale as a solution. SONIC [4] demonstrates the benefits of scaling up data collection and compute budgets, and Humanoid-GPT [6] further expands on this philosophy by spending spending 75% of a 15,000 GPU hour budget on training 384 separate PPO experts before distilling it into a single model. More closely related to ours, concurrent work Scaling BFM [9] studies how tracking performance scales with increased GPU budgets and dataset sizes. In contrast, we study the algorithmic foundations of scaling in limited compute regimes of up to 1,300 GPU hours, introducing QTOP and fitting explicit scaling relationships to disentangle how tracking performance depends jointly on dataset size, training compute, and the learning algorithm.

Massively Parallelized RL

Massively parallel RL utilizes multiple simulation environments to speed up rollout collection alongside a learning policy [18], [19]. Across most tasks, PPO is the most commonly chosen algorithm due to its stability, scaling properties [20], and highly optimized implementations [21]. While, on the other hand, off-policy methods struggle to scale out of the box [22], there has been a greater effort to revisit off-policy methods for parallel simulated robotics tasks [23], [24], [25], [26] due to its sample efficiency.

Akin to motion tracking, multitask RL introduces an additional problem of task interference [7]. However, while PPO is the dominant paradigm for motion tracking, most multi-task RL works focus on using off-policy algorithms such as SAC [27] and combine it with additional strategies such as gradient surgery [28], [29] or return normalization [30]. TOPPO [31] similarly builds on on-policy PPO, but targets discrete multitask RL benchmarks, whereas we address interference within motion tracking, where a continuously changing reference induces a smooth continuum of objectives rather than explicitly separated tasks.

Method

Tracking setup

For the tracker formulation, we largely follow the Beyond-Mimic recipe [1]. We therefore only define the necessary variables and omit details such as reward design, joint impedance and domain randomization, which have remained unchanged in our work. The goal of motion tracking is to optimize a tracking reward rtrackingr_{\mathrm{tracking}}, which consists of position and orientation errors for selected robot bodies with respect to the reference motion over a fixed horizon of N=500N = 500 steps. The tracking controller runs at 50 Hz and, at every timestep, receives an observation

ot=[ct,ϕtref,ωtbase,qt−qˉ,q˙t,at−1].\mathbf{o}_t = \left[\mathbf{c}_t,\phi_t^{\mathrm{ref}},\boldsymbol{\omega}_t^{\mathrm{base}},\mathbf{q}_t-\bar{\mathbf{q}},\dot{\mathbf{q}}_t,\mathbf{a}_{t-1}\right].

Here, ct=[qtref,q˙tref]\mathbf{c}_t = [\mathbf{q}_t^{\mathrm{ref}},\dot{\mathbf{q}}_t^{\mathrm{ref}}] is the reference joint position and velocity command, consisting of retargeted motion sequences which we obtain via GMR [32]. Furthermore, ϕtref∈R6\phi_t^{\mathrm{ref}} \in\mathbb{R}^{6} is a 6D orientation error between the robot torso and reference torso, represented by the first two columns of the relative rotation matrix, while the remaining terms are proprioceptive: base angular velocity ωtbase\boldsymbol{\omega}_t^{\mathrm{base}}, joint positions relative to the nominal pose qt−qˉ\mathbf{q}_t-\bar{\mathbf{q}}, joint velocities q˙t\dot{\mathbf{q}}_t, and the previous action at−1\mathbf{a}_{t-1}. We omit including the base robot position and linear velocities in ot\mathbf{o}_t for consistency between simulation and real-world experiments, as in the latter case an external state estimation module would be required. Our embodiment of choice is the 29-DoF Unitree G1 humanoid with Dex3-1 hands, which yields ot∈R154\mathbf{o}_t \in\mathbb{R}^{154} as finger joints are not modeled explicitly.

Quantile Advantage Function

Motion tracking can be formulated as a reference-conditioned Markov Decision Process. At the start of each training episode we sample a reference motion segment c1:N∼pdata\mathbf{c}_{1:N} \sim p_{\mathrm{data}}. The policy πθ\pi_\theta is conditioned on the observation ot\mathbf{o}_t and aims to maximize

J(θ)=Ec1:N∼pdataEπθ[∑t=0N−1γtrtrack(st,ct)].J(\theta) = \mathbb{E}_{\mathbf{c}_{1:N}\sim p_{\mathrm{data}}}\mathbb{E}_{\pi_\theta}\left[\sum_{t=0}^{N-1}\gamma^t r_{\mathrm{track}}(\mathbf{s}_t,\mathbf{c}_t)\right].

where st\mathbf{s}_t is the simulator state and γ∈[0,1)\gamma\in[0,1) is the discount factor. The expectation over reference motions makes large-scale tracking a continuous multi-task optimization problem. Unlike many multi-task RL settings, where each task is specified by a discrete identifier, the task in motion tracking is indirectly specified by the reference command sequence c1:N\mathbf{c}_{1:N}. Moreover, the conditioning changes at every timestep through ct\mathbf{c}_t, while the actor observes only a compact local command rather than an explicit motion identity and future trajectory. Thus, similar observations can correspond to different future command continuations, contact regimes, and return targets. This results in substantially different state visitation and return distributions across samples of c1:N\mathbf{c}_{1:N}, while sharing the same reward function and policy parameters. Prior work has aimed to address this issue by adding future reference frames into the observation at the cost of latency [4], or decomposing training across motion subsets or expert policies and subsequently distilling them into a single deployable policy [1, 8, 6]. In this work, on the other hand, we aim to study a more robust learning algorithm that can directly alleviate optimization issues when sampling over large command spaces. We

build on top of PPO due to its success in motion tracking and existing highly optimized baselines, which optimizes J(θ)J(\theta) using rollout samples and an advantage estimate A^t\hat{A}_t. Typically, the policy update is driven by a clipped surrogate objective whose learning signal is proportional to A^t\hat{A}_t, which is computed from a learned critic VψV_\psi, e.g., through Generalized Advantage Estimation (GAE) [33]. Inspired by previous work [7, 31], which indicates that critic learning is a key bottleneck in actor-critic multi-task settings, we first analyze cosine similarities of actor and critic gradients respectively, and find that the same holds for motion tracking in Figure 2. This encourages rethinking how the critic is used for policy updates. Standard GAE compresses the critic into a scalar mean value and forms advantages from scalar temporal difference residuals. When value learning is difficult, these residuals can be detrimental for updating the actor in the right direction.

Gradient cosine similarity during training

Figure 2. Per-motion actor and critic gradient cosine similarities during training. At each point, gradients are computed for individual reference motions at a common parameter snapshot. Shaded bands denote the 5th–95th percentile over motion pairs, and dotted lines denote the median. Critic gradients exhibit substantially greater variation, including a stronger negative tail, indicating more severe disagreement in value learning.

Rather than attempting to align these critic gradients directly, QTOP changes how critic predictions are converted into policy learning signals. QTOP learns a distributional critic and computes a distributional advantage using the relative position of returns with respect to the predicted return distribution. This reduces sensitivity to motion-dependent return scales and critic errors, resulting in more balanced contributions to the aggregate actor update. Formally, QTOP replaces the scalar critic Vψ(ot)V_\psi(\mathbf{o}_t) with a categorical distributional critic over a fixed return support [34]

Z={z1,…,zK},z1<⋯<zK.\mathcal{Z} = \{z_1,\ldots,z_K\}, \qquad z_1 < \cdots< z_K.

Given an observation ot\mathbf{o}_t, the critic outputs logits ℓψ(ot)∈RK\ell_\psi(\mathbf{o}_t) \in\mathbb{R}^{K}, which define probabilities

pψi(ot)=exp⁡(ℓψi(ot))∑j=1Kexp⁡(ℓψj(ot)).p_\psi^i(\mathbf{o}_t) = \frac{\exp(\ell_\psi^i(\mathbf{o}_t))}{\sum_{j=1}^{K}\exp(\ell_\psi^j(\mathbf{o}_t))}.

The predicted return distribution is

Zψ(ot)=∑i=1Kpψi(ot) δzi,Z_\psi(\mathbf{o}_t) = \sum_{i=1}^{K}p_\psi^i(\mathbf{o}_t)\,\delta_{z_i},

and its expectation recovers a scalar value estimate

Vψ(ot)=E[Zψ(ot)]=∑i=1Kpψi(ot)zi.V_\psi(\mathbf{o}_t) = \mathbb{E}[Z_\psi(\mathbf{o}_t)] = \sum_{i=1}^{K}p_\psi^i(\mathbf{o}_t)z_i.

We use this expected value wherever PPO requires a scalar critic, e.g., for bootstrapping returns in non-terminal rollouts. To compute the policy advantage, QTOP evaluates where the realized return lies under the predicted local return distribution. Let

Fψ(x∣ot)=∑i:zi≤xpiψ(ot)F_{\psi}(x \mid o_t) = \sum_{i:z_i\leq x} p_i^{\psi}(o_t)

denote the CDF of the predicted return distribution, with linear interpolation between adjacent support atoms. For each rollout sample with (bootstrapped) returns G^t\hat{G}_t, we compute

ut=Fψ(G^t∣ot),u_t = F_{\psi}(\hat{G}_t \mid o_t),

and define the distributional advantage

A^tdist=2(ut−12).\hat{A}^{\mathrm{dist}}_t = 2\left(u_t-\frac{1}{2}\right).

Thus, A^tdist>0\hat{A}^{\mathrm{dist}}_t > 0 when the realized return is above the median of the critic’s predicted local return distribution, and A^tdist<0\hat{A}^{\mathrm{dist}}_t < 0 when it is below. In practice, we use a warmup phase with standard GAE advantages and then blend the standardized scalar and distributional advantages:

A^tQTOP=(1−α)Norm⁡(A^tGAE)+αNorm⁡(A^tdist),\hat{A}^{\mathrm{QTOP}}_t = (1-\alpha)\operatorname{Norm}\left(\hat{A}^{\mathrm{GAE}}_t\right) + \alpha\operatorname{Norm}\left(\hat{A}^{\mathrm{dist}}_t\right),

where α∈[0,1]\alpha\in[0,1] controls the contribution of the quantile advantage. The resulting advantage is normalized over the rollout batch and used in the standard PPO surrogate objective, which is unchanged. Unless otherwise stated, we use α=0.5\alpha= 0.5 and enable the quantile advantage after a 500-iteration warmup. Finally, we combine this distributional advantage with the Muon optimizer [35], as we empirically find it to provide more efficient optimization than Adam when combined with distributional critics.

Experiments

We evaluate QTOP along three dimensions. Firstly, we compare tracking performance in simulation against strong on-policy and off-policy RL baselines under a fixed compute budget. Secondly, we establish a scaling law with respect to motion dataset and RL compute hours. Finally, we deploy the resulting policies on a Unitree G1 humanoid and evaluate real-world tracking behavior. For our training dataset, we use the union of AMASS [36] and LAFAN1 [37]. Although this corpus is significantly smaller than those used by recent large-scale systems [4], [6], our objective is not to build a state-of-the art foundation model, but to isolate the effect of the learning algorithm under limited data and compute budgets. As shown in Section IV-B, even within this comparatively small data regime, tracking performance requires substantial compute to approach saturation.

Simulation Evaluation

Baselines and training budget. We train QTOP and several baselines for a fixed compute budget of 200 GPU hours, corresponding to around 24 hours of parallel training on a single node of eight A100 GPUs. Wherever possible, we share the exact network size and architecture. We follow

previous work [4] to choose an appropriate number of learnable parameters and implement an MLP with 4 hidden layers, equating to 11.5 million parameters in total. Furthermore, we use mjlab [38] as our simulator due to its open-source availability and throughput advantages.

Our on-policy comparisons isolate the contributions of the optimizer, distributional critic, and quantile advantage. PPO (Adam) is the standard motion tracking baseline, whereas PPO (Muon) replaces the optimizer Muon [35] while retaining the scalar critic. QTOP (α=0\alpha= 0) uses Muon and the categorical distributional critic, but computes GAE using the expected value of the predicted return distribution. Finally, QTOP (α=0.5\alpha= 0.5) adds the distributional advantage with equal weight. Additionally, we compare against SAC using the recently released optimized RSL-RL implementation [23] and FastTD3 [24] due to its focus on humanoid tasks and inclusion of a distributional critic. We also experimented with MT-GRPO [7], which was motivated by the impaired critic gradients of Figure 2, but found its reliance on complete Monte Carlo rollouts prohibitively expensive for our 10 second (500-step) tracking horizon.

Evaluation Datasets. We evaluate generalization performance on unseen motion sequences. We use the 3.1-hour optical motion-capture subset of MOSAIC [39], whose motions produce consistently higher-quality retargeting than MOSAIC’s inertial motion capture subset. To broaden evaluation beyond this, we additionally collect our own inertial motion-capture dataset of diverse human motions including everyday activities such as walking, crouching, jumping, cooking, shelf restocking and sports. For each policy, we evaluate randomly sampled reference motions using 8192 parallel environments and report the mean and standard error across two independently trained seeds.

Results. Table 1 reports mean Per-Keybody Position Error (MPKPE), root relative MPKPE (R-MPKPE), global end-effector position (Global EE Pos.) as well as global end-effector orientation (EE Ori.) errors. Following Beyond-Mimic, we define key bodies as 14 representative rigid links spanning the pelvis, torso, arms, and legs, rather than all kinematic joints, such that MPKPE measures anchor-aligned position error over these key bodies and R-MPKPE measures body positions relative to the torso. The end effector metrics isolate the robot’s hands and feet positions, which are particularly relevant for loco-manipulation. QTOP (α=0.5\alpha= 0.5) proves to have the strongest overall generalization performance, achieving the lowest MPKPE, R-MPKPE and end-effector orientation error, whereas QTOP (α=0\alpha= 0) obtains the lowest global end-effector position error. We note that the global end-effector position errors accumulate over time and therefore are comparatively large, since they are measured in the world frame and the lack of global root position as policy inputs. This indicates that while the distributional critic is useful in itself, utilizing a mix of GAE and distributional advantage estimation can further improve tracking performance overall, with QTOP (α=0.5\alpha= 0.5) consistently yielding about 10% improvement over the standard PPO baseline.

MethodMPKPE (mm) ↓R-MPKPE (mm) ↓Global EE Pos. (cm) ↓EE Ori. (degrees) ↓
MOSAIC (optical mocap, 3.1h)
QTOP (α=0.5\alpha=0.5)41.84±0.9041.84\pm0.9038.12±0.90\mathbf{38.12}\pm0.9036.66±1.8236.66\pm1.8211.44±0.00\mathbf{11.44}\pm0.00
QTOP (α=0\alpha=0)43.17±0.2243.17\pm0.2239.53±0.2739.53\pm0.2732.71±0.87\mathbf{32.71}\pm0.8713.37±1.4413.37\pm1.44
PPO (Muon)44.99±0.1444.99\pm0.1441.70±0.0841.70\pm0.0835.75±0.8735.75\pm0.8711.86±0.4911.86\pm0.49
PPO (Adam)45.68±0.5645.68\pm0.5641.96±0.5541.96\pm0.5541.88±0.1941.88\pm0.1913.12±0.1113.12\pm0.11
SAC54.56±0.8954.56\pm0.8951.48±0.7351.48\pm0.7377.90±8.4777.90\pm8.4714.64±0.3914.64\pm0.39
FastTD368.79±0.9768.79\pm0.9764.32±0.7464.32\pm0.7450.33±2.1450.33\pm2.1420.54±0.0620.54\pm0.06
Ours (inertial mocap, 1h)
QTOP (α=0.5\alpha=0.5)39.76±1.58\mathbf{39.76}\pm1.5837.70±1.55\mathbf{37.70}\pm1.5545.33±2.3645.33\pm2.3613.10±0.10\mathbf{13.10}\pm0.10
QTOP (α=0\alpha=0)41.79±0.2541.79\pm0.2540.07±0.6240.07\pm0.6239.83±3.05\mathbf{39.83}\pm3.0514.32±0.9714.32\pm0.97
PPO (Muon)44.65±0.4644.65\pm0.4642.20±0.0042.20\pm0.0041.47±1.5141.47\pm1.5114.08±1.2014.08\pm1.20
PPO (Adam)45.72±0.1045.72\pm0.1043.59±0.4143.59\pm0.4145.61±0.0845.61\pm0.0814.78±0.1914.78\pm0.19
SAC56.42±2.0056.42\pm2.0055.30±2.4755.30\pm2.4786.74±0.8886.74\pm0.8815.97±0.4415.97\pm0.44
FastTD373.55±0.4973.55\pm0.4970.76±0.1070.76\pm0.1065.01±3.8465.01\pm3.8424.52±0.3824.52\pm0.38

Table 1. Algorithm comparison on two datasets. Position errors are reported in millimeters or centimeters and orientation errors in degrees. Lower is better for all metrics.

Interestingly, we find off-policy algorithms to perform underwhelmingly in motion tracking despite significant tuning efforts. While SAC worked well out of the box with similar hyperparameters as PPO, its performance saturates well before that of PPO. For FastTD3, on the other hand, we employed an automated agent based on GPT 5.5 with a budget of 100 training runs to derive both optimized hyperparameters as well as having the option to change the implementation. The agent’s starting point was the original FastTD3 codebase as well as our distributional critic support hyperparameters from QTOP. Despite these significant tuning efforts, its performance lags behind drastically, suggesting that motion tracking poses additional challenges beyond those introduced by learning high DoF humanoid controllers.

Scaling Law

Experiment Setup. The previous experiments evaluate QTOP under a fixed compute budget. However, an effective learning algorithm should not only perform well at a single operating point, but should also continue to improve as additional resources become available. Inspired by the vast literature on scaling laws for language modeling [40] and, more recently, reinforcement learning [20, 41], we study how the tracking performance of QTOP scales jointly with respect to training compute and motion data. Specifically, we fit a predicted error surface as

E(C,D)=E0+ADγC−α+BD−β,E(C,\mathcal{D}) = E_0 + A\mathcal{D}^{\gamma}C^{-\alpha} + B\mathcal{D}^{-\beta},

where (C)(C) denotes GPU-hours and (D)(\mathcal{D}) denotes dataset hours. The constant E0E_0 represents an irreducible error floor. Traditionally, scaling laws do not feature the interaction component Dγ\mathcal{D}^{\gamma}, but we include it due to providing a better fit that better captures optimization error. Overall, ADγC−αA\mathcal{D}^{\gamma}C^{-\alpha} always decreases with compute, but increases with dataset size as a larger motion corpus can require more optimization, while BD−βB\mathcal{D}^{-\beta} captures the typical behavior of data scaling.

We focus on compute and dataset size while holding the model architecture fixed, allowing us to isolate which resource provides the larger marginal improvement. Prior motion-tracking work has generally increased data and compute simultaneously, making their individual contributions difficult to disentangle. SONIC [4], for example, reports separate curves for increasing dataset size and increasing compute, but does not evaluate an interaction over a complete compute and data grid. Our study instead varies both quantities systematically, enabling us to characterize their joint effect and compare the benefits of allocating additional resources to either data or optimization.

Training and evaluation split. We construct three training sets containing approximately 4, 12, and 24 hours of motion data, where the 24-hour set corresponds to the full training corpus used in the previous experiments. For the 4- and 12-hour settings, we independently sample three subsets of the corresponding size from the full corpus to eliminate sampling bias. We train QTOP three times for different seeds at each dataset size and evaluate checkpoints at log-spaced compute intervals. At each checkpoint, we then measure R-MPKPE on the held-out MOSAIC test set and average the metric across the three runs. This produces an evaluation grid indexed by dataset size and accumulated GPU-hours.

To reduce sensitivity to noisy checkpoint evaluations, we estimate the scaling-law parameters using a robust softℓ1\ell_1 loss [40]. We then visualize the resulting surface using iso-error contours and local compute-data improvement directions and additionally perform leave-one-dataset-size-out validation to assess whether the fitted relationship generalizes across dataset scales rather than merely interpolating individual measurements.

Interpolation Results. Figure 3 visualizes the fitted compute-data scaling law for QTOP. The robust fit achieves R2=0.981R^2 = 0.981 on the log-spaced evaluation grid, with fitted exponents α=0.592\alpha= 0.592, β=0.0277\beta= 0.0277, γ=0.0915\gamma= 0.0915. We use the fitted surface E(C,D)E(C, D) to compare two local resource-allocation strategies: doubling the dataset size, yielding E(C,2D)E(C, 2D), or doubling the training compute E(2C,D)E(2C, D). The right panel plots the difference E(C,2D)−E(2C,D)E(C, 2D) - E(2C, D), such that positive values indicate that doubling compute is predicted to produce lower tracking error than doubling dataset hours. The left panel shows iso-error contours of the same fitted surface, with arrows indicating the locally preferred resource-allocation direction.

QTOP scaling law and compute-data tradeoff plots

Figure 3. Left: Fitted scaling law surface to log-spaced compute points for three different dataset sizes. Right: R-MPKPE difference heatmap of doubling compute versus doubling dataset hour budgets. For reasonable compute budgets up to 700 GPU hours, dataset size has a marginal impact on performance.

Across most of the evaluated low-to-moderate compute regime, extending to approximately 700 GPU-hours, the fitted law predicts that increasing training compute is more beneficial than increasing dataset size. Thus, even when using QTOP on a comparatively small motion corpus, policy learning remains strongly compute-constrained. These results suggest that simply collecting more reference motions may provide limited benefits unless accompanied by a sufficient compute budget, and motivate further research into RL algorithms that exploit larger datasets more compute-efficiently.

Extrapolation Results. Figure 4 analyzes the generalization capabilities by fitting our law to a subset of the available points. Taking the same training and evaluation setup, we now omit one of the three dataset sizes to fit the parameters

Extrapolating scaling law predictions to each dataset size

Figure 4. Extrapolating scaling law predictions to each dataset size yields reliable predictions of tracking error. Bottom right: Predicted versus observed R-MPKPE for each held-out dataset size when fit to the remaining two.

SystemInterventionCC (GPUh)DD (hours)ErrorGain/doubling
OursCompute13.7–130325.055→.0355.59%
OursData13035–25.037→.0353.31%
SONICCompute2k–21k556.027→.0243.10%
SONICData21k22–556.024→.0240.53%

Table 2. Relative error reduction per budget doubling when scaling compute or data. Frames are converted to hours at 50 Hz.

α,β,γ,E0\alpha, \beta, \gamma, E_0 on the remaining 12 points and evaluate on a respective held-out test set of 6 points. For each dataset, the dashed line represents predicted R-MPKPE behavior as GPU hours increase, whereas solid points indicate empirical observations. Overall, we see that despite the small training set, our law is able to consistently predict the behavior for different dataset sizes with an overall MAE of 0.9mm.

We want to note that this accuracy should be interpreted within the measured regime: dataset size spans only 4–24 hours, whereas compute varies over nearly two orders of magnitude, and the fitted surface is comparatively flat along the data axis. We therefore do not claim that the law extrapolates to substantially larger motion corpora. Nevertheless, Figure 3 indicates that even these small datasets require considerable compute before additional data becomes beneficial. Table II quantifies this imbalance and compares it with the trends reported by SONIC [4]. We compute the average gain per doubling by dividing the (unrounded) relative endpoint error reduction by log⁡2(Bmax⁡/Bmin⁡)\log_2(B_{\max}/B_{\min}), where BB denotes the compute or data budget. Compute scaling provides larger gains in both settings, while SONIC observes particularly small improvements from substantially increasing its dataset. Our larger data-scaling gain is consistent with operating in a smaller-data regime. Because SONIC evaluates its data sweep at a fixed maximum compute budget and uses a different error metric, we treat this comparison as qualitative evidence rather than a controlled benchmark.

Motion Contribution Balancing

To understand why QTOP improves tracking, we compare GAE and QTOP on identical rollouts from 32 motions while keeping the policy fixed. Figure 5 shows how unevenly the resulting learning signals are distributed across motions.

Advantage magnitude and actor-gradient norm across training iterations

Figure 5. QTOP balances policy updates across motions. Using identical rollouts and frozen policy checkpoints, we compare the across-motion coefficient of variation of advantage magnitudes and actor-gradient norms.

Real-Robot Motion Tracking results for Cartesian and joint-space tracking

Figure 6. Real world tracking results for converged PPO and QTOP checkpoints in terms of R-MPKPE and joint MAE. QTOP outperforms PPO at the same compute and dataset budgets in unseen motion replay.

QTOP reduces the variation in advantage magnitude by 68% and in actor-gradient norm by 39%, averaged across checkpoints. Above, we used gradient cosine similarity as evidence of heterogeneous value learning, rather than as a quantity that QTOP directly optimizes, as QTOP does not explicitly align motion-conditioned gradients. Instead, it limits how strongly differences in return scale and critic error affect their contribution to the actor update. This mechanism is complementary to gradient-surgery methods [31], which directly modify gradient directions but require additional per-task gradient computation. Our method, on the other hand, retains the scalability and minibatch structure of PPO.

Real World Evaluation

Finally, we investigate whether QTOP’s simulation gains translate to improved tracking on physical hardware. To do so, we deploy our PPO and QTOP policies on a Unitree G1 humanoid, with policy inference running at 50Hz on an external workstation. We evaluate on 32 diverse motions from our inertial motion-capture dataset, ranging from 5 to 15s in duration. The selected motions form a representative and diverse subset including walking in different styles, shelf restocking, whiteboard cleaning, playing pool and cooking. For each trial, the robot is initialized at the first reference frame and tracks the motion until completion while its achieved pose is recorded. Each motion is repeated three times for both methods using checkpoints trained on our 24-hour motion dataset with 10310^{3} GPU-hours of compute.

Figure 6 summarizes the resulting real-world tracking accuracy in terms of R-MPKPE and absolute joint error (joint MAE), averaged across trials. We note that joint MAE is computed over all actuated joints, whereas R-MPKPE is evaluated only on the key bodies included in the training reward. We find that QTOP significantly improves both metrics over PPO, reducing R-MPKPE by approximately (8%), with a smaller but consistent improvement in joint MAE. These results indicate that the tracking improvements obtained by

QTOP in simulation transfer to physical hardware under the same data and training-compute budget.

Finally, we qualitatively evaluate both policies by replaying challenging out-of-distribution motions, such as martial arts sequences in Figure 7. While both policies produce robust behavior overall, we find that the PPO policy tends to rely on additional stabilization steps during high-velocity upper body movements, leading to a substantial positional drift over time. QTOP, on the other hand, follows the commanded motion more faithfully. We also test a whiteboard cleaning task as shown in Figure 1. In this scenario, the differences are more visually subtle, but we find that the marginal tracking gains by QTOP can significantly improve the positioning of the robot end effectors. As a result, we conclude that even modest improvements in tracking accuracy can translate into meaningful gains for real-world loco-manipulation tasks.

QTOP overview showing quantile critic, scaling law, and real-world tracking

Figure 1. Achieving high-fidelity real-world tracking with QTOP. We modify PPO to include advantage estimates coming from a distributional critic (A). Using a scaling law in terms of compute (GPU hours) and number of reference motion frames (dataset hours), we analyze the scaling behavior of QTOP and find that motion tracking is heavily compute-bound (B). Finally, we show that QTOP performs well on real-world tracking with minimal data (C).

QTOP and PPO real-world motion tracking comparison

Figure 7. QTOP faithfully replays challenging out-of-distribution motions such as martial arts, whereas PPO relies on multiple stabilization steps.

Discussion

We introduce QTOP, a motion tracking algorithm that consistently improves over strong on- and off-policy baselines in simulation and transfers these gains to real hardware. Our scaling analysis further shows that, within the measured regime of 4–24 dataset hours and up to approximately 1,300 GPU-hours, tracking remains primarily compute-limited, as increasing optimization compute generally yields larger gains than adding reference data. The main limitation of our work is that, due to limited resources, our experimental regime is one magnitude smaller in compute than state-of-the-art trackers. We therefore cannot yet determine whether the fitted relationship or QTOP’s advantages persist at substantially larger data and compute scales. In particular, the importance of dataset size may increase once policies receive sufficient compute to exploit larger and more diverse motion corpora. Rather than advocating for scaling data and compute indiscriminately, our results motivate greater attention to algorithms that handle the continuous multi-task structure of motion tracking more efficiently. We view QTOP and our compute-data analysis as a starting point for developing such methods and for studying scaling behavior at larger regimes.

Acknowledgements

Generative AI tools were used for coding assistance (GPT 5.5, GPT 5.6-sol), literature search (GPT 5.5), language editing (GPT 5.5), and as conversational aids for brainstorming scientific hypotheses (GPT 5.5). We note that all hypotheses were evaluated and selected on our own by conducting the experiments, interpreting the results and contextualizing them within the literature independently.

References

  1. [1]Q. Liao, T. E. Truong, X. Huang, Y. Gao, G. Tevet, K. Sreenath, and C. K. Liu, “Beyondmimic: From motion tracking to versatile humanoid control via guided diffusion,” arXiv preprint arXiv:2508.08241, 2025.arxiv.org/abs/2508.08241
  2. [2]Z. Zhang, J. Guo, C. Chen, J. Wang, C. Lin, Y. Lian, H. Xue, Z. Wang, M. Liu, J. Lyu et al., “Track any motions under any disturbances,” arXiv preprint arXiv:2509.13833, 2025.arxiv.org/abs/2509.13833
  3. [3]Z. Chen, M. Ji, X. Cheng, X. Peng, X. B. Peng, and X. Wang, “Gmt: General motion tracking for humanoid whole-body control,” arXiv preprint arXiv:2506.14770, 2025.arxiv.org/abs/2506.14770
  4. [4]Z. Luo, Y. Yuan, T. Wang, C. Li, F. Castaneda, S. Chen, Z.-A. Cao, J. Li, D. Minor, Q. Ben et al., “Sonic: Supersizing motion tracking for natural humanoid whole-body control,” arXiv preprint arXiv:2511.07820, 2025.DOI
  5. [5]K. Yin, W. Zeng, K. Fan, M. Dai, Z. Wang, Q. Zhang, Z. Tian, J. Wang, J. Pang, and W. Zhang, “Unitracker: Learning universal whole-body motion tracker for humanoid robots,” IEEE Robotics and Automation Letters, 2026.
  6. [6]Z. Qi, X. Chen, J. Wang, C. Lin, Y. Lian, W. Zhang, X. Yu, H. Wang, and L. Yi, “Humanoid generative pre-training for zero-shot motion tracking,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2026, pp. 20 834–20 844.
  7. [7]V. Joshi, Z. Xu, B. Liu, P. Stone, and A. Zhang, “Benchmarking massively parallelized multi-task reinforcement learning for robotics tasks,” arXiv preprint arXiv:2507.23172, 2025.arxiv.org/abs/2507.23172
  8. [8]Y. Wang, S. Zhu, P. Zhi, Y. Li, J. Y.-L. Li, Y. Xiao, X. Wang, B. Jia, and S. Huang, “Omnixtreme: Breaking the generality barrier in high-dynamic humanoid control,” arXiv preprint arXiv:2602.23843, 2026.arxiv.org/abs/2602.23843
  9. [9]W. Zeng, K. Yin, X. Niu, S. Lu, W. Zhong, J. Chen, F. Jia, X. Chen, Z. Wang, F. Xu et al., “Scaling behavior foundation model for humanoid robots,” arXiv preprint arXiv:2607.15163, 2026.arxiv.org/abs/2607.15163
  10. [10]X. B. Peng, P. Abbeel, S. Levine, and M. Van de Panne, “Deepmimic: Example-guided deep reinforcement learning of physics-based character skills,” ACM Transactions On Graphics (TOG), vol. 37, no. 4, pp. 1–14, 2018.
  11. [11]T. Wang, Y. Guo, M. Shugrina, and S. Fidler, “Unicon: Universal neural controller for physics-based character motion,” arXiv preprint arXiv:2011.15119, 2020.arxiv.org/abs/2011.15119
  12. [12]N. Wagner, A. Kolobov, F. Vieira Frujeri, R. Loynd, C.-A. Cheng, and M. Hausknecht, “Mocapact: A multi-task dataset for simulated humanoid control,” Advances in Neural Information Processing Systems, vol. 35, pp. 35 418–35 431, 2022.
  13. [13]Z. Luo, J. Cao, K. Kitani, W. Xu et al., “Perpetual humanoid control for real-time simulated avatars,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 10 895–10 904.arxiv.org/abs/2305.06456
  14. [14]Z. Fu, Q. Zhao, Q. Wu, G. Wetzstein, and C. Finn, “Humanplus: Humanoid shadowing and imitation from humans,” arXiv preprint arXiv:2406.10454, 2024.arxiv.org/abs/2406.10454
  15. [15]Y. Xue, W. Dong, M. Liu, W. Zhang, and J. Pang, “A unified and general humanoid whole-body controller for versatile locomotion,” arXiv preprint arXiv:2502.03206, 2025.arxiv.org/abs/2502.03206
  16. [16]T. He, W. Xiao, T. Lin, Z. Luo, Z. Xu, Z. Jiang, J. Kautz, C. Liu, G. Shi, X. Wang et al., “Hover: Versatile neural whole-body controller for humanoid robots,” in 2025 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2025, pp. 9989–9996.
  17. [17]Y. Ze, Z. Chen, J. P. Araújo, Z.-a. Cao, X. B. Peng, J. Wu, and C. K. Liu, “Twist: Teleoperated whole-body imitation system,” arXiv preprint arXiv:2505.02833, 2025.arxiv.org/abs/2505.02833
  18. [18]A. Nair, P. Srinivasan, S. Blackwell, C. Alcicek, R. Fearon, A. De Maria, V. Panneershelvam, M. Suleyman, C. Beattie, S. Petersen et al., “Massively parallel methods for deep reinforcement learning,” arXiv preprint arXiv:1507.04296, 2015.arxiv.org/abs/1507.04296
  19. [19]N. Rudin, D. Hoeller, P. Reist, and M. Hutter, “Learning to walk in minutes using massively parallel deep reinforcement learning,” in Conference on robot learning. PMLR, 2022, pp. 91–100.arxiv.org/abs/2109.11978
  20. [20]J. Hilton, J. Tang, and J. Schulman, “Scaling laws for single-agent reinforcement learning,” arXiv preprint arXiv:2301.13442, 2023.arxiv.org/abs/2301.13442
  21. [21]S. Huang, R. F. J. Dossa, A. Raffin, A. Kanervisto, and W. Wang, “The 37 implementation details of proximal policy optimization,” in IC LR Blog Track, 2022, https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/. [Online]. Available: https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/
  22. [22]S. Park, K. Frans, D. Mann, B. Evsenbach, A. Kumar, and S. Levine, “Horizon reduction makes rl scalable,” Advances in Neural Information Processing Systems, vol. 38, pp. 8350–8389, 2026.arxiv.org/abs/2506.04168
  23. [23]G. Sabatini, C. Li, and M. Hutter, “Bridging the gap: Enabling soft actor critic for high performance legged locomotion,” arXiv preprint arXiv:2605.24975, 2026.arxiv.org/abs/2605.24975
  24. [24]Y. Seo, C. Sferrazza, H. Geng, M. Nauman, Z.-H. Yin, and P. Abbeel, “Fasttd3: Simple, fast, and capable reinforcement learning for humanoid control,” arXiv preprint arXiv:2505.22642, 2025.arxiv.org/abs/2505.22642
  25. [25]D. Kim, Y. Lee, M. Park, K. Kim, I. Nahendra, T. Senо, S. Min, D. Palenicek, F. Vogt, D. Kragic et al., “Flashsac: Fast and stable off-policy reinforcement learning for high-dimensional robot control,” arXiv preprint arXiv:2604.04539, 2026.arxiv.org/abs/2604.04539
  26. [26]C. Voelcker, A. Brunnbauer, M. Hussing, M. Nauman, P. Abbeel, R. Grosu, E. Eaton, A.-m. Farahmand, and I. Gilitschenski, “Relative entropy pathway policy optimization,” in International Conference on Learning Representations, vol. 2026, 2026, pp. 35 761–35 790.arxiv.org/abs/2507.11019
  27. [27]T. Yu, D. Quillen, Z. He, R. Julian, K. Hausman, C. Finn, and S. Levine, “Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning,” in Conference on robot learning. PMLR, 2020, pp. 1049–1100.arxiv.org/abs/1910.10897
  28. [28]Z. Chen, V. Badrinarayanan, C.-Y. Lee, and A. Rabinovich, “Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks,” in International conference on machine learning. PMLR, 2018, pp. 794–803.
  29. [29]B. Liu, Y. Feng, P. Stone, and Q. Liu, “Famo: Fast adaptive multitask optimization,” Advances in Neural Information Processing Systems, vol. 36, pp. 57 226–57 243, 2023.arxiv.org/abs/2306.03792
  30. [30]M. Cho, J. Park, J. Kim, and Y. Sung, “Ars: Adaptive reward scaling for multi-task reinforcement learning,” in Forty-second International Conference on Machine Learning, 2025.
  31. [31]Y. Li, G. Lin, A. Qu, and R. Miao, “Toppo: Rethinking ppo for multi-task reinforcement learning with critic balancing,” arXiv preprint arXiv:2605.11473, 2026.arxiv.org/abs/2605.11473
  32. [32]J. P. Araujo, Y. Ze, P. Xu, J. Wu, and C. K. Liu, “Retargeting matters: General motion retargeting for humanoid motion tracking,” arXiv preprint arXiv:2510.02252, 2025.arxiv.org/abs/2510.02252
  33. [33]J. Schulman, P. Moritz, S. Levine, M. Jordan, and P. Abbeel, “High-dimensional continuous control using generalized advantage estimation,” arXiv preprint arXiv:1506.02438, 2015.arxiv.org/abs/1506.02438
  34. [34]M. G. Bellemare, W. Dabney, and R. Munos, “A distributional perspective on reinforcement learning,” in International conference on machine learning. Pmlr, 2017, pp. 449–458.arxiv.org/abs/1707.06887
  35. [35]K. Jordan, Y. Jin, V. Boza, J. You, F. Cesista, L. Newhouse, and J. Bernstein, “Muon: An optimizer for hidden layers in neural networks,” 2024. [Online]. Available: https://keljordant.github.io/posts/muon/
  36. [36]N. Mahmood, N. Ghorbani, N. F. Troje, G. Pons-Moll, and M. J. Black, “AMASS: Archive of motion capture as surface shapes,” in International Conference on Computer Vision, Oct. 2019, pp. 5442–5451.
  37. [37]E. G. Harvey, M. Yurick, D. Nowrouzezahrai, and C. Pal, “Robust motion in-betweening,” ACM Transactions on Graphics (TOG), vol. 39, no. 4, pp. 60–1, 2020.DOI
  38. [38]K. Zakka, Q. Liao, B. Xia, Y. L. L. Lay, K. Sreenath, and P. Abbeel, “mjlab: A lightweight framework for gpu-accelerated robot learning,” 2026. [Online]. Available: https://arxiv.org/abs/2601.22074
  39. [39]Z. Sun, B.-S. Huang, Y. Peng, X. Li, J. Ma, Y. Sun, Z. Li, H. Jiang, B. Gao, Z. Bing et al., “Mosaic: Bridging the sim-to-real gap in generalist humanoid motion tracking and teleoperation with rapid residual adaptation,” arXiv preprint arXiv:2602.08594, 2026.arxiv.org/abs/2602.08594
  40. [40]J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. d. L. Casas, L. A. Hendricks, J. Welbl, A. Clark et al., “Training compute-optimal large language models,” arXiv preprint arXiv:2203.15556, 2022.arxiv.org/abs/2203.15556
  41. [41]P. Fu, O. Rybkin, Z. P. Zhou, M. Nauman, P. Abbeel, S. Levine, and A. Kumar, “Compute-optimal scaling for value-based deep rl,” Advances in Neural Information Processing Systems, vol. 38, pp. 160 363–160 402, 2026.DOI

Paper details

vlnc:W_qwhbkuhs
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

GPT-5.5 (bare), GPT-5.6 Sol (bare)

View statement
Looking for related papers…
Contents