WorksComputer Science
MuCHMDPoser: Multiview RGB-Camera Full-Body Pose Tracking for Virtual Reality
Abstract
Accurate full-body tracking in Virtual Reality (VR) is heavily underconstrained when relying solely on consumer-grade sparse sensors, such as Head-Mounted Displays (HMDs) and handheld controllers. Indeed, the self-avatar full-body pose must be inferred based on the 3 available upper-body joints tracked by these devices. This leads to significant tracking error, especially in lower-body pose. To overcome this issue, we propose to extend the tracking setup with external multiview RGB cameras. We show that integrating RGB-based pose estimation this framework is able to significantly decrease lower-body tracking errors, even under common artifacts in RGB-based estimates. Additionally, we evaluate a only-once HMD-camera calibration procedure that does not require any physical markers. We finally propose an implementation of the overall framework, achieving an end-to-end animation at 60 FPS on a Nvidia RTX3080 GPU. Codes are available at https://github.com/antmaio/MuCHMDPoser.
Introduction
In Virtual Reality (VR), a key component of immersive experience is the concept of the self-avatar i.e., a digital representation that embodies the user within the virtual environment. A fundamental requirement for effective self-avatars is their ability to accurately follow the user’s real-world motions with high fidelity. Many works highlight the importance of an accurate full-body tracking to maintain user experience, especially regarding user-centered qualitative metrics such as Sense-of-Embodiment (SoE) or Sense-of-Agency (SoA) [17, 51].
To track user’s motion, immersive applications often rely on consumer grade VR hardware i.e., Head-Mounted Display (HMD) and handheld controllers. These devices integrate IMU-based trackers to accurately capture head and hands motion. In this context, animating the full body of the articulated self-avatar is not straightforward. The system must infer the complete body pose from the motion signals of the head and hands. This problematic is even more challenging regarding lower-body pose reconstruction. Indeed, this configuration does not involve any lower-body tracking as input to animate the lower-body of the self-avatar with an high fidelity.
To address these limitations, various approaches incorporate additional tracking modalities. Some methods use supplementary IMUs or wearable sensors attached to the lower body [35, 33], while others leverage RGB cameras [49, 32, 15, 28]. Camera-based systems offer a non-intrusive alternative: they require no user-worn hardware and avoid per-user calibration.
In this context, we propose a framework for fast full-body pose estimation in large-scale VR environments. Our setup relies on multiple RGB cameras strategically placed around the scene to capture additional motion cues, particularly for the lower body, helping to resolve pose ambiguities that arise when only head and hand signals are available. The resulting 3D joint positions are then integrated with the VR motion features to estimate the full-body avatar pose.
Related Work
Full-body Pose Estimation from Sparse IMUs Tracking in VR
Estimating the full-body pose of an articulated avatar from 18-DoF motion data is a major challenge in Extended Reality (XR), primarily due to the complete absence of lower-body tracking in consumer-grade XR hardware. Early solutions relying on Inverse Kinematics [8], Machine Learning [1] or Motion Matching [2] [34] have increasingly been superseded by Deep Learning techniques.
Modern generative techniques such as diffusion [9, 12, 13] and normalizing flows [4] have been introduced to synthesize more natural motion by modeling the underlying data distribution. Building on this generative philosophy, Starke et al. recently introduced codebook matching techniques capable of generating full-body motion across diverse scenarios, such as sports or dance, while enabling character control via traditional inputs like joysticks [40].
Sequence modeling approaches are now widely used to capture temporal and spatial dependencies and predict poses end-to-end [22, 53, 48, 11, 35]. Following this trend, HMD-Poser further optimizes this by employing a lightweight temporal-spatial network for on-device inference, while incorporating online body shape estimation to improve accuracy across diverse user morphologies [10]. To further address the computational bottleneck of Transformer-based architectures, SSD-Poser introduces a State Space Attention Encoder, leveraging State Space Duality to achieve high-precision pose reconstruction with significantly lower latency and a smaller memory footprint [52].
While efficient, these techniques often rely on reliable hand tracking, making it difficult to adapt when motion signals are lost. Egocentric video-based methods have been introduced to tackle this problem by modeling hand visibility within the HMD field-of-view [5] [21]. Similarly, ReliaAvatar employs an auto-regressive architecture to ensure robustness against both short- and long-term tracking signal loss [36], while DragPoser utilizes a variational latent space to maintain pose sampling efficiency and adherence to real motion distributions during such outages [33].
Multimodal Full-body Pose Estimation in VR
Despite recent advancements, a common limitation shared across methods presented in § 2.1 is the reduced accuracy of synthesized lower-body motion compared to upper-body gestures. Furthermore, even the most recent models struggle to generalize across a wide variety of static poses due to inherent ambiguities in estimating lower-body configurations from sparse input signals [40].
To address the inherent ambiguity of lower-body pose estimation, several full-body animation approaches rely on complementary sensing modalities to recover additional motion cues. Optical motion capture systems provide highly accurate motion measurements at high frame rates and have been successfully used in VR contexts [45, 19]. However, these systems require expensive infrastructure and cumbersome body-worn markers or sensors, which limits their scalability and practical deployment outside controlled environments. Consequently, more affordable body-tracking solutions have been proposed that combine inertial measurement units (IMUs) with additional sensing technologies. In particular, integrating IMUs with Ultra-Wideband (UWB) localization has been shown to mitigate the drift and jitter inherent to inertial tracking [6, 24, 46]. Despite improving robustness, these approaches still rely on body-worn hardware and typically operate with a single instrumented user per sensor setup. Furthermore, they require per-user calibration procedures to align the different sensing modalities. In contrast, vision-based approaches using RGB cameras enable passive motion capture without body-worn hardware, and require only a one-time extrinsic calibration rather than per-user calibration. Several works fuse IMU measurements with 2D pose estimates from RGB images to alleviate occlusion and depth ambiguity [32, 49, 7, 50]. FusePose [7] combines inertial and visual observations through kinematic modeling and state-space fusion, while VR-oriented systems such as HybridTrak [49] and PoseVR [50] augment headset and controller tracking with additional RGB views to estimate lower-body motion. However, these approaches are typically designed around single-view configurations, which limits their scalability to larger environments and multi-user scenes.
Camera-based systems are, moreover, restricted to the space they cover. To overcome this constraint, recent works estimate full-body motion from head-mounted sensors only, such as egocentric cameras and IMUs, leveraging stereo or temporal visual cues [23, 47, 41]. Yet, egocentric views suffer from severe self-occlusion and a narrow field of view that often leaves the lower body out of sight, making the kinematics of invisible joints highly ambiguous. These systems are also prone to “false positives,” where limbs of other people are attributed to the user.
Observing the user from the outside avoids these issues. Multi-view setups reduce self-occlusion, since a joint hidden in one view usually remains visible in another, and per-view 2D detections can be lifted to 3D by triangulation, typically through the Direct Linear Transform (DLT) [20]. Yet, external cameras alone are not sufficient to animate a self-avatar: their 3D estimates remain noisy and are expressed in the camera frame rather than in the HMD tracking space, whereas the HMD and controllers already provide accurate, low-latency head and hand measurements. The two modalities are therefore complementary.
Building on this, we propose a framework that fuses sparse HMD and controller signals with 3D keypoints from external multiview RGB cameras. Our contributions are:
A deep learning-based model that estimates the full-body pose from 18-DoF HMD and controller and RGB-based 3D keypoints, significantly reducing lower-body tracking errors, even under common artifacts in RGB-based estimates.
A one-time, marker-free HMD-camera calibration procedure that expresses both modalities in a common coordinate system and supports a variable number of cameras.
An implementation of the complete pipeline achieving end-to-end animation at 60 FPS on an Nvidia RTX3080 GPU.
Full-Body Pose Estimation
Problem Formulation
This work addresses the challenge of accurately animating full-body self-avatars using sparse, multimodal body tracking for immersive applications. The core objective is to define a mapping function that transforms motion data from consumer-grade VR hardware and off-the-shelf RGB video-based multiview pose estimations into an articulated full-body avatar that follows accurately user’s movements. The overview of this framework is shown in Figure 1.

Figure 1. Overview of the proposed framework for self-avatar full-body tracking from an HMD and external RGB cameras. First, the HMD and cameras are calibrated into a common coordinate system, so the 18-DoF HMD and controller motion data and the 3D keypoints from video-based pose estimation are expressed in the same frame. Then, a deep learning-based model takes these sparse tracking signals and 3D keypoints as input and estimates the full-body pose. The pipeline runs end-to-end at 60 FPS and supports a variable number of cameras, calibrated into a common coordinate system regardless of their placement.
We assume that head and hands nodes are tracked by consumer-grade level devices i.e., HMD and handheld controllers. For each timestep , we collect motion features , respectively, for head, left and right hand node. we define these motion features as global position and rotation , as well as their respective linear and angular velocities, and . Rotations are expressed in 6D, particularly useful in deep learning due to the continuous mapping of this representation in Euclidean space [54]. Following previous works [10], these motion features are also expressed in head space coordinate, leading to the set of motion features for HMD and hands defined in Equation 1. denotes motion features expressed in HMD space.
The second motion data stream consists of global 3D Cartesian positions of body joints . These positions are estimated using off-the-shelf, multiview 2D pose estimation on videos combined with a 3D pose lifting method , as stated by Equation 2.
and are respectively 2D positions in image space and the estimation confidence score for joint in camera , where is the set of involved calibrated cameras.
Hence, this configuration leads to an input vector formed by the concatenation of the sensor data and the estimated 3D positions . The predicted full-body motion of the self-avatar is given by
where and represent the predicted global body displacement and orientation, while denotes the predicted local orientation for each bone.
Input Data Acquisition
We use AMASS [27] as ground truth motion data source. It unifies multiple optical motion capture datasets via the SMPL body model [25]. From this dataset, we extract head and hands motion to simulate the tracking signals from a HMD and controllers.
The 3D Cartesian coordinates of key body joints are collected through off-the-shelf multiview 2D pose estimations on videos rendering of the motion dataset. These videos are rendered from virtual cameras strategically positioned in the scene. A sample of the scene rendering with the virtual cameras is shown as matter of example in Figure 2. The extrinsic parameters of these virtual cameras are optimized to ensure the entire range of motion within the dataset remains within the field of view. The distribution of the root positions on the ground plane within the dataset is displayed in Figure 3. Moreover, the cameras intrinsic parameters are fixed to match the specifications of a real-world ZED2 camera setup.

Figure 2. Virtual configuration that imitates hardware setup. We define 3 virtual cameras, whose the intrinsic and extrinsic parameters are known. Off-the-shelf 2D pose estimation is applied in every rendered view to capture lower-body motion cues.

Figure 3. Root global position distribution on the ground surface under a standard evaluation protocol using AMASS dataset. This delimitates an approximative interaction area.
From these rendered videos, we extract 2D joint keypoints position and the related confidence scores using an off-the-shelf 2D pose estimation model. A significant challenge is that such models, while fast, are prone to estimation artifacts e.g., occlusions or jitter, which severely degrade the quality of 3D poses recovered through standard pose lifting methods.
To tackle this effect, from Equation 2 combines the Direct Linear Triangulation (DLT) method with Kalman filter to estimate smooth 3D positions of full-body keypoints from the multiple 2D poses estimated in each camera . For each frame and joint , we apply DLT using the 2D detections from the two cameras with the highest confidence scores . If either of these two selected cameras yields a confidence score , where is our confidence threshold, we consider the joint as occluded and set its 3D position .
Architecture
As defined in related work [28], we rely on an extended version of HMD-Poser [10] to estimate the full-body pose of the self-avatar, suitable for its lightweight, yet effective avatar animation framework. The overview of the proposed architecture can be found in Figure 4. First, the motion features from HMD and hand-held controllers are projected into an embedding space in a channel-wise manner. We extend this embedding layer to incorporate 3D positions from the pose estimation framework . Then, the spatial-temporal encoder consists in LSTM and Transformer blocks, to capture respectively the temporal dynamic and spatial relationship in motion data. Finally, we estimate the full state of the articulated avatar i.e., global translation and orientation as well as local bone orientation, using a fully-connected-based layer as pose estimation from previously encoded information.
Figure 4. Full-body Pose Estimation model. We extend the architecture from HMD-Poser [10] to feed the filtered positions of body joints acquired by the multiview pose estimation setup. Extending the channel-wise embedding, we are able to provide embeddings from body joints outside standard head and hands tracking.
HMD and External Camera Rig Calibration
Fusing HMD motion data with video-based 3D keypoints requires both streams to be expressed in a common coordinate system. The two tracking systems, however, are independent. The HMD estimates its own 6-DoF pose through inside-out tracking in a device-centric frame , whereas the external cameras only know the physical world frame defined by their calibration (Figure 6). Neither system observes the other’s frame, so the rigid transform between them must be estimated explicitly. However, this estimation is non-trivial. Indeed, is initialized arbitrarily at each application startup, so a transform estimated once is valid for one session only. Then, a session-invariant frame must therefore be provided by the runtime, allowing a one-time calibration procedure between HMD and cameras. Nevertheless, relocalization errors or tracking failures of can make it deviate unpredictably. Since every calibrated pose is expressed relative to , such deviations propagate directly into the user’s pose in , without any warning to the application.

Figure 6. Reference frames involved in HMD pose estimation. The device-centric frame , re-initialized arbitrarily at each VR application startup, is mapped to the session-invariant frame . Both and live within the VR software domain, while additionally serves as the bridge to the physical world domain: the rigid-body transform , obtained from the multi-camera calibration, relates to the shared physical world frame .
Session-Invariant Reference Estimation
We define using Meta Mixed Reality Utility Kit (MRUK)1. MRUK provides tools to perform a semantic scan of the room (walls, floor, ceiling, furniture) configured at the OS level before the application starts and exposes the resulting geometry in Unity. We build a deterministic reference axis system anchored to the ground and aligned with the longest detected wall. The rigid transform mapping coordinates from axis system to is then known.
Calibration approach
Inspired by the standard multiview calibration procedure [20], we use a planar checkerboard of known geometry as the physical reference shared by both tracking devices. Let , , be the homogeneous 3D positions of its inner corners, defined in with the board in the plane. For each external camera , the corners are detected and the extrinsics are recovered by solving the
Perspective--Point (PnP) problem [14]:
where denotes the perspective division, the intrinsics provided by the manufacturer and the corners detected in the image captured by the camera . The transform represents the camera pose in world reference frame .
Then, the HMD is registered with a single snapshot of the same checkerboard, taken with its left passthrough camera , while we record the camera pose in , denoted . The same PnP solve (Equation (4)) yields the HMD left camera pose in world reference . The transform between the virtually and physically anchored reference frames, respectively and , is retrieved by transform composition:
Figure 5. Overview of the calibration setup used to register the multi-camera system, the HMD, and the handheld controllers into a common reference axis system. Left: Schematic of the full setup, showing the subject wearing an HMD inside the capture volume observed by the external RGB cameras. Right: (a) Standard multiview camera calibration protocol [20]. (b) The Meta Mixed Reality Utility Kit (MRUK) provides a room scan from passthrough, allowing a physically-anchored virtual tracking space . (c) Snapshot from the HMD camera with corner detection and pose registration . This setup closes the loop between the HMD tracking space, the floor-anchored frame within the virtual application, and the external camera system.
Finally, the full pipeline transform between initial tracking space for the 18DoF motion data from consumer-grade VR devices and world reference axis system shared across external cameras
Because is assumed session-invariant, this registration is performed once. With that procedure, we are then able to express the 6DoF motion data from the HMD in the world reference frame shared with the external RGB cameras. Similarly, we can also express the 3D positions estimated via pose estimation and DLT applied to the camera videos in the virtual tracking spaces and .
Experiments
Full-Body Pose Estimation
From RGB video inputs, we estimate the 3D positions of body joints, including the feet, ankles, hips, elbows, shoulders, and wrists. The proposed approach is evaluated on the AMASS dataset [27] following two standard evaluation protocols introduced in prior work [53].
Protocol 1 This protocol follows a commonly used training and evaluation split constructed from the CMU [44], HDM05 [30], and BMLrub [42] motion capture subsets. In total, the dataset contains approximately 20 hours of human motion capture data. The training and testing partitions are generated by mixing motion sequences from each subset, following the standard split strategy adopted in recent self-avatar full-body tracking methods [13, 22].
Protocol 2 In this protocol, the model is trained on a larger set of AMASS subsets, including ACCAD [16], BMLrub [42], BMLmovi [18], CMU [44], EKUT [29], Eyes Japan Dataset [26], KIT [29], HDM05 [30], MPI Limits [3], SFU [31], and TotalCapture [43]. Evaluation is performed on the Transitions Mocap [27] and HumanEva [38] subsets, which are excluded from training.
Protocol 1 comprises approximately 20 hours of high-fidelity motion capture sequences, whereas Protocol 2 spans roughly 43 hours of recorded motion data. For both protocols, training is performed using a temporal window of 40 frames. We consider both motion streams i.e., 3D positions and IMUs trackers synchronized at constant framerate 60 Hz. The model is trained for 400 epochs on a single RTX 4090 GPU with a batch size of 128. The model is optimized under the same loss function as in related work [10]. This optimization is performed using the Adam optimizer with a one-cycle learning rate policy [39], reaching a maximum learning rate of . To estimate 2D positions in each camera view, we rely on the nano model of YOLOv8 framework for a fast data acquisition.
Finally, full-body pose estimation performance is evaluated through standard objective metrics: Mean Per Joint Positional Error (MPJPE), Mean Per Joint Rotational Error (MPJRE), Mean Per Joint Velocity Error (MPJVE), and the jitter ratio computed between the jitter in predicted and ground-truth global positions of avatar joints.
Calibration
To assess the stability of a candidate frame without external ground-truth equipment, we use a cross-reprojection protocol, shown by Figure 7. We capture snapshots of the board from different viewpoints, each pairing a passthrough image with the simultaneously recorded camera pose in . For every snapshot , we compute by Equation (4), so the control points can be expressed in the candidate axis system following Equation (5). In a round-robin fashion, the points obtained from a reference snapshot are projected into every other snapshot using the

Figure 7. Protocol to assess the consistency and stability of tracking space. We take several snapshots from left camera of the HMD, register its pose relatively to . We use this transform to reproject the control points into the image plane of each snapshot and measure the discrepancy between these 2D points and the corners detected by OpenCV.
pose recorded for . We report two metrics:
with .
The two metrics capture complementary aspects of frame quality. isolates the intrinsic stability of the candidate frame itself, independently of any camera- or projection-related error. Since the checkerboard is rigid and its corners are fixed and known once and for all from the multiview calibration, any discrepancy between and can only originate from an inconsistency between and . Thus, directly measures whether the candidate frame’s pose relative to the world remains consistent across snapshots, or drifts over time and viewpoint. It is expressed in physical units (mm) and does not depend on external camera intrinsics or on the accuracy of the HMD-to-camera transform.
In contrast, is an end-to-end, application-facing metric. It evaluates whether a 3D point expressed in from a reference snapshot correctly reprojects onto the corresponding checkerboard corner when using the camera pose recorded in a different snapshot . Unlike , this metric additionally incorporates the accuracy of the HMD’s own tracking relative to , the camera intrinsics , and the corner-detection noise in . Expressed in pixels, it answers a practical question for augmented-reality applications: if virtual content is anchored in during one session or viewpoint, how many pixels of misalignment will be observed when revisiting it from another viewpoint?
Reporting both metrics allows the source of error to be localized: a low combined with a high indicates that the candidate frame itself is stable but that HMD tracking or intrinsic calibration is the limiting factor, whereas a high indicates that the frame itself drifts, which necessarily propagates into as well. This comparison is particularly relevant when contrasting the MRUK-based frame (sceneSpace) with the baseline (trackingSpace), as it reveals whether room-scan-based floor anchoring reduces drift () relative to the native tracking origin, and whether this improvement translates into a user-perceptible reduction in reprojection error (). Note that and are reported in different units (mm and pixels, respectively) and are therefore not directly comparable in magnitude, only in relative trend across the two candidate frames.
As hardware tracking devices, we use two ZED2 as external RGB cameras (1280px720px), capturing videos from their left lens. We employ a checkerboard with inner corners and 25 cm squares. We use MetaQuest 3 as consumer-grade VR devices, and the HMD calibration relies on its left passthrough camera (1280px1280px).
Results
Calibration
Table 1 reports the cross-reprojection error and the 3D consistency error for each reference snapshot. The MRUK-based frame and the baseline tracking space yield identical results, with for every snapshot. Across snapshots, ranges from 2.37 to 3.86 px (mean 3.19 px) and from 0.62 to 1.03 cm (mean 0.81 cm). Defining from the room scan therefore provides the same stability as the raw tracking frame, and both remain subject to the drift of the headset’s visual-inertial odometry. This residual drift, below 1 cm on average, is smaller than the error
| Ref. snapshot | (px) | (cm) | ||||
| S₁ | 2.37 | 2.37 | 0.00 | 0.62 | 0.62 | 0.00 |
| S₂ | 3.55 | 3.55 | 0.00 | 0.82 | 0.82 | 0.00 |
| S₃ | 3.58 | 3.58 | 0.00 | 1.03 | 1.03 | 0.00 |
| S₄ | 2.82 | 2.82 | 0.00 | 0.77 | 0.77 | 0.00 |
| S₅ | 3.86 | 3.86 | 0.00 | 0.91 | 0.91 | 0.00 |
| S₆ | 2.93 | 2.93 | 0.00 | 0.71 | 0.71 | 0.00 |
| Mean | 3.19 | 3.19 | 0.00 | 0.81 | 0.81 | 0.00 |
Table 1. Cross-reprojection errors for each reference snapshot, comparing the standard tracking space with the MRUK-based frame . The reprojection procedure results in sub-centimer precision, in average. indicates that both tracking space axis systems share similar stability properties. Lower is better.
Table 1: Cross-reprojection errors for each reference snapshot, comparing the standard tracking space with the MRUK-based frame . The reprojection procedure results in sub-centimer precision, in average. indicates that both tracking space axis systems share similar stability properties. Lower is better.
Table 2: Multiview 2D-to-3D pose lifting with direct linear triangulation (DLT), evaluated on the two protocols. : lower is better.
typically reported for video-based 3D keypoint estimation, so we consider it compatible with our target application of fusing RGB-based keypoints with HMD data.
Multiview 2D to 3D positions lifting
Table 2 reports the reconstruction quality of direct linear triangulation (DLT), which we use as a fast, lightweight baseline for multiview 2D-to-3D lifting. For a fair comparison, we remove frames where one occlusion occurs for at least one joint, as we zeroed the positions of occluded joints. DLT requires no learned component and runs per frame, which makes it suitable for building the 3D pose data at scale. Its output is nevertheless imperfect: the MPJPE is 12.94 cm on Protocol 1 and 21.31 cm on Protocol 2, and the velocity error (155.78 and 206.80 cm/s) and jitter ratio (51.12 and 72.53) show that the reconstructed trajectories contain frame-to-frame noise. We do not attempt to remove these inaccuracies at the reconstruction stage. Since the DLT poses are part of the training set, the avatar animation network is exposed to this noise during training and learns to produce smooth, plausible motion from it.
Full-body Pose Estimation
Table 3 presents the quantitative evaluation of our proposed full-body pose estimation pipeline across Protocol 1 and Protocol 2. We compare our fused approach against a baseline relying exclusively on the three standard sparse trackers (the HMD and two handheld controllers). This baseline is similar to the 3-tracker configuration in HMD-Poser [10]. We provide samples of generated poses with ground truth in Figure 8.

Figure 8. Samples of full-body pose estimated from (top to bottom): 3 trackers; 3 trackers + Pose estimation; ground truth. Incorporating 3D positions from external tracking improves the of the full-body pose estimation, especially at lower-body parts.
The 3-tracker baseline exhibits significant ambiguity for untracked joints, with the lower-body position error (L-PE) reaching 9.73 cm in the dynamic Protocol 2. Integrating external 3D pose observations markedly improves spatial accuracy: MPJPE drops by 36% in Protocol 1, while L-PE is reduced to 5.48 cm in Protocol 2. Beyond positional accuracy, the fusion also refines rotational and temporal dynamics. MPJRE and MPJVE decrease substantially across both protocols, indicating a more anatomically and kinematically plausible pose reconstruction. The upper-body position error (U-PE) sees more marginal gains, as it is already highly constrained by the HMD and controller hardware. Overall, the improvement across all metrics highlights the effectiveness of the external observations in resolving the degrees of freedom that sparse tracking alone cannot capture.
The external 3D observations also introduce some high-frequency noise into the kinematic chain: the jitter ratio increases from 1.74 to 2.40 in Protocol 1, and from 2.10 to 3.44 in Protocol 2. This increase remains moderate compared with the gains in spatial and rotational accuracy.
Figure 9 shows sample self-avatar poses estimated by our proposed framework implemented in game engine. Extending sparse head and hands tracking with RGB video-based tracking enhances the accuracy of the avatar full-body pose, especially at lower-body parts. The inference of our full-body pose estimation framework provides 60FPS animation using Unity Inference Engine 2 on a Nvidia RTX3080.
Limitations and Future Work
Our quantitative evaluation relies on AMASS sequences rendered from virtual cameras with ZED2 intrinsics (Section 3.2), which lets us study full-body accuracy against reliable ground truth. The real-world setup, in contrast, is only assessed through the calibration metrics and qualitative results (Figure 9), since we have no synchronized ground-truth lower-body motion for these recordings. Off-the-shelf 2D detectors may behave differently on real video than on rendered avatars (lighting, clothing, motion blur), which can create a domain gap. Future work should record a real multiview dataset with synchronized optical motion capture, so that real-world pose accuracy can be measured directly.

Figure 9. Poses estimated by our proposed framework (top) and from baseline, relying on HMD and controllers solely (bottom). The green spheres represent the positions of body joints estimated from RGB videos (except for the axis system attached to the green sphere, which corresponds to ). Our model is able to follow the user’s motion and improves lower-body tracking compared to full-body pose estimation from sparse tracking alone.
A second limitation concerns the quality of the 3D observations. We use direct linear triangulation from the two most confident cameras as a fast, lightweight lifting step, and we zero the position of occluded joints. As Table 2 shows, the resulting 3D positions are imprecise, with an MPJPE of 12.94 to 21.31 cm, and the avatar network is trained on this noisy input. This choice explains part of the jitter increase observed in Table 3 (1.74 to 2.40 in Protocol 1 and 2.10 to 3.44 in Protocol 2), even though spatial and rotational accuracy improve. More robust lifting, such as confidence-weighted triangulation over all available views or a learned multiview lifting module, could provide cleaner inputs at a modest additional cost. The influence of the number and placement of cameras on accuracy also remains to be studied.
| Protocol | Occlusion rate [%] | MPJPE [cm] | MPJVE [cm/s] | Jitter ratio |
| Protocol 1 | 0.95 | 12.94 | 155.78 | 51.12 |
| Protocol 2 | 0.58 | 21.31 | 206.80 | 72.53 |
Table 2. Multiview 2D-to-3D pose lifting with direct linear triangulation (DLT), evaluated on the two protocols. : lower is better.
| Configuration | MPJPE ↓ | U-PE ↓ | L-PE ↓ | MPJRE ↓ | MPJVE ↓ | Jitter ↓ |
| [cm] | [cm] | [cm] | [°] | [cm/s] | ratio | |
| Protocol 1 | ||||||
| 3 trackers | 3.67 | 1.57 | 6.71 | 5.43 | 17.79 | 1.74 |
| 3 trackers + 3D pose (ours) | 2.35 | 1.23 | 3.97 | 4.62 | 16.20 | 2.40 |
| Protocol 2 | ||||||
| 3 trackers | 5.28 | 2.19 | 9.73 | 9.45 | 26.60 | 2.10 |
| 3 trackers + 3D pose (ours) | 3.26 | 1.73 | 5.48 | 8.12 | 23.73 | 3.44 |
Table 3. Full-body pose estimation accuracy on both protocols, comparing the 3-tracker baseline (HMD and two controllers) with our method, which adds external 3D pose observations. Position errors are reported for the full body (MPJPE), upper body (U-PE) and lower body (L-PE), together with the joint rotation error (MPJRE), joint velocity error (MPJVE) and jitter ratio. ↓: lower is better; best results in bold.
The calibration has its own limits. The cross-reprojection protocol (Section 5.2) is run within a single session and shows a residual drift of 0.81 cm on average, which comes from the headset’s visual-inertial odometry. In this setting, the MRUK-based frame and the native tracking space give identical results, so the benefit of room-scan anchoring is not visible. Its expected advantage is session invariance, which we assume but have not tested across application restarts, relocalization events, or longer sessions. Our implementation also depends on the Meta Quest 3 and MRUK, and a checkerboard is still needed once to register the HMD with the cameras. Evaluating the frame across sessions and adapting the procedure to other headsets are natural next steps.
Moreover, an important open question is how the remaining jitter and spatial inaccuracies affect user-centered metrics, such as the Sense of Embodiment (SoE) [37]. While related work highlights the importance of animation fidelity and control in SoE [51, 17], it remains to be determined whether the accuracy gains of our pipeline outweigh the added jitter for the user. Future work must therefore include subjective user evaluations to validate the perceptual benefit of these hybrid tracking methods.
Finally, providing lower-body 3D positions is compatible with a wide range of 3-point tracking architectures, so recent lightweight state-space models such as SSD-Poser [52] could be extended with this approach. A comprehensive benchmark against other sparse tracking methods, including DragPoser [33] and SparsePoser [35], would further contextualize our results. In addition, our system currently tracks a single user in a large scene. Extending it to multi-user environments, still without additional wearable lower-body trackers, is a promising direction, and its main challenges are user (re-)identification and fast multi-user pose estimation from RGB videos.
Conclusion
This paper presented a framework for full-body self-avatar tracking in VR that fuses the sparse signals of an HMD and handheld controllers with 3D joint positions estimated from external multiview RGB cameras. We proposed a one-time calibration procedure that expresses both modalities in a common coordinate system through a single checkerboard snapshot and supports a variable number of cameras. We also introduced a deep learning model that estimates the full-body pose from these inputs, trained on noisy triangulated positions. The calibration shows a residual error of 0.81 cm on average, which is compatible with fusing RGB-based keypoints with HMD data. On AMASS, external 3D observations reduce the MPJPE by 36 % and 38 % on Protocols 1 and 2, and the lower-body position error by 41 % and 44 %, at the cost of a moderate increase in jitter. The complete pipeline runs end-to-end at 60 FPS on a consumer GPU. These results suggest that combining consumer-grade VR devices with outside-in RGB tracking is a practical way to improve avatar full-body pose estimation in large-scale scenes, and that its perceptual benefit for the user should be examined in future work.
References
- [1]K. Ahuja, E. Ofek, M. Gonzalez-Franco, C. Holz, and A. D. Wilson. CoolMoves: User Motion Accentuation in Virtual Reality. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 5(2):1–23, June 2021. doi: 10.1145/3463499 2DOI
- [2]AIIDE 2016. Motion Matching for Realistic Animation in “For Honor” (Simon Clavet), Oct. 2016. 2
- [3]I. Akhter and M. J. Black. Pose-conditioned joint angle limits for 3D human pose reconstruction. In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1446–1455, June 2015. ISSN: 1063-6919. doi: 10.1109/CVPR.2015.7298751 6DOI
- [4]S. Aliakbarian, P. Cameron, F. Bogo, A. Fitzgibbon, and T. J. Cashman. FLAG: Flow-based 3D Avatar Generation from Sparse Observations. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 13243–13252. IEEE, New Orleans, LA, USA, June 2022. doi: 10.1109/CVPR52688.2022.01290 2DOI
- [5]S. Aliakbarian, F. Saleh, D. Collier, P. Cameron, and D. Cosker. HMD-NeMo: Online 3D Avatar Motion Generation From Sparse Observations. 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 9588–9597, Oct. 2023. Conference Name: 2023 IEEE/CVF International Conference on Computer Vision (ICCV)arxiv.org/abs/2308.11261

