PCHMR: Empowering and Benchmarking Human Mesh Recovery in Privacy-Constrained Real-World Settings

Reviewed on OpenReview: https://openreview.net/forum?id=4XHzBaL0qA against DePoser-generated pseudo-ground truth. This benchmark aims to support future work on privacy-constrained HMR models and their real-world deployment and evaluation.

Introduction

Advancements in human mesh recovery (HMR) methods ([58, 14, 18, 50, 59, 49]) are relevant to a wide range of real-world applications, such as virtual reality ([4, 74] (VRChat 2024)), augmented reality ([75]), human behavior analysis ([63, 32, 114, 115]), and virtual try-on ([25]). Accurate human mesh recovery is always desired for those human-centric applications.

For general learning-based applications like medical image computing ([28, 2, 89]) or on-device speech recognition ([10, 5 b]), fine-tuning the model directly on user-side data is one of the most effective ways of enhancing the model performance by minimizing the domain gap ([69, 47]). To the best of our knowledge, existing HMR frameworks ([42, 58, 18, 14, 15, 49, 51, 50, 56]) are typically developed and evaluated under the setting where all training data are on a single device, which we refer to as the centralized setting. As a result, HMR model training using existing frameworks usually assumes the collection and aggregation of sensitive data from user devices to a central training server. This raises privacy and compliance concerns and is increasingly challenging, if not infeasible, under strict privacy regulations such as the General Data Protection Regulation (GDPR) ([23]). A naïve raw-data-local solution would be to train the HMR model on each edge device using only the scarce data available there. However, since the mapping from a monocular image to a 3D human’s pose and shape is highly complex, training such a model with limited real-world data collected from only a single edge device would result in significantly inferior performance compared to centralized training; see Appendix § 13. This poses a performance-privacy dilemma: the desire to effectively utilize user data on the edge to enhance the model’s performance, conflicting with the principle of avoiding raw-data centralization.

Despite extensive work on HMR ([42, 58, 18, 14, 15, 49, 51, 50, 56, 24]) where models are trained and evaluated centrally under ideal conditions, one practical question remains unexplored: How do state-of-the-art methods perform under privacy-constrained real-world settings, where centralized aggregation of data from user devices is infeasible due to privacy concerns?

This work presents a systematic benchmark study of HMR models ([18, 14])³ under a comprehensive series of privacy-constrained, real-world settings, answering the aforementioned question. In all of our experiments, the training is conducted with a client-server model, where each client holds a part of the training data, which is never shared. Under this framework, we then scale up to a large number of clients, where we find that the HMR models converge with minimal accuracy loss even with 1,000 clients. In addition, since each HMR-equipped device user could have a different usage pattern, we simulate real-world data non-uniformity with three schemes of varying complexity. When data heterogeneity is pronounced, collaboratively trained HMR models can match, and in some settings slightly exceed, their centrally trained counterparts. We interpret this as evidence that federated training need not degrade HMR accuracy under the tested protocols; we discuss possible causes of the occasional gains in Appendix § 8. Furthermore, we evaluate a secure-aggregation variant with MPC ([45]). This variant protects the aggregation channel under an honest-but-curious server assumption and prevents the server from directly observing individual client updates, but it should not be read as an attack-based privacy evaluation of the released global HMR model. With less than $2 \times$ the runtime in our small-scale measurement, MPC appears feasible enough to merit further system-level study.

In privacy-constrained HMR, raw images cannot be transmitted to a central server, so locally annotated data becomes essential both for collaborative training and for model personalization. Because user- and environment-specific scenarios vary widely, the annotation pipeline must be robust; optimization-based annotators suit this need because they are stable and accurate. Unlike existing annotators ([8, 86]) that rely on sparse cues such as 2D keypoints or masks, whose performance can be limited by detector quality and monocular ambiguity, we introduce DePoser, a depth-aware HMR annotation pipeline.

DePoser leverages the large human vision foundation model Sapiens ([44]), a 2B-parameter model trained on 300M+ in-the-wild images, to provide body-part relative depth cues, and then aligns these predictions to metric scale via a lightweight linear regressor trained with a general absolute depth estimator, preserving Sapiens’ structure. At the same time, we exploit Sapiens’ accurate and robust 2D joint predictions to provide stronger optimization signals. This yields 2.5D keypoints that couple precise localization with scale-invariant depth. DePoser is fully monocular, deployable on common devices from a single RGB image input. We validate our pipeline against Pavlakos et al. [86] and benchmark downstream HMR models on a VR dataset of internet-sourced and in-house footage annotated by DePoser.

We show that locally fine-tuning models on such DePoser-generated annotations enhances HMR performance in privacy-constrained collaborative training setups, enabling on-device models to self-collect pseudo-supervision. In summary, our contributions are:

• We identify the centralized-data assumption in state-of-the-art HMR training pipelines and evaluate how these models perform without it, which is essential for deploying them on private real-world data.

• We provide a comprehensive study of state-of-the-art HMR models, investigating their performance under privacy-constrained training settings of varying complexity and assessing the feasibility of collaborative training for HMR models.

• Local annotators are essential for privacy-constrained HMR training. Motivated by the depth ambiguity issue of previous optimization-based HMR annotators, we further propose DePoser, a local HMR annotation pipeline enhanced by depth foundation models, which can be integrated with the proposed collaborative training framework while making the pseudo-label dependency explicit.

Related Work

Many recent works study 3D representations of the human body or body parts, such as 3D skeleton ([70, 72, 110]), 3D whole-body mesh ([49, 50, 18, 14, 15, 58]), 3D hand/face recovery ([105, 93, 78]), and 3D human motion generation ([30, 97, 12, 31, 54, 106, 37, 116]). Due to the large amount of prior work, we focus on 3D whole-body mesh recovery in the following section for its high relevance.

Human Mesh Recovery

Optimization-based Human Mesh Recovery. This approach estimates the 3D mesh that is consistent with the 2D observation under various constraints by directly optimizing with standard body model parameters. After the seminal work SMPLify ([8]), follow-up work extends the fitting objective with additional terms that keep the 3D mesh consistent with the 2D image observation. In addition, factors such as shape, pose, and segmentation provide additional priors to the optimization problem, such as HuMoR ([90]), Pose-NDF ([98]) and others ([29, 43]). This type of method has also been expanded to regress detailed human body, hand, and face at the same time ([86, 121, 57]). Optimization-based methods apply gradient-based optimization with a neural network and perform iteratively to obtain accurate mesh results such as HUND ([113]) and LVD ([16]). These methods generally yield a higher quality of mesh compared with the regression-based approach; however, due to their iterative optimizing nature, these methods are often incapable of real-time inference.

Regression-based Human Mesh Recovery. This method generates a 3D human mesh directly from pixels with an end-to-end network. Various methods use intermediate representations such as joints ([107]), segmentation ([91]), UV maps ([111]), heatmaps ([77]), keypoints and silhouettes ([85]), dense correspondences ([108]), or texture consistency ([87]) to supply additional information for the mesh regression. In addition, forward kinematics and inverse kinematics may be considered at the same time as additional constraints for the neural network, such as HybrIK ([49]), HybrIK-X ([51]), and NIKI ([50]). This representation is then passed into a decoder network, the output of which could be classified into the following two categories: parametric output and non-parametric output. The former is in the parameter space of parametric models such as SMPL (Poper et al., 2015), while the latter is in 3D space, such as joints and mesh vertex coordinates. SOTA performance has been achieved by transformer-based methods such as METRO ([58]), MeshGraphormer ([59]), and FastMETRO ([14]), which are non-parametric approaches excelling at modeling long-range dependency, making them robust even under occlusions. Transformer efficiency is optimized with token pruning methods such as in TORE ([18]).

Federated Learning

Federated Learning (FL) ([71]) is a family of distributed learning paradigms in which raw training data never leave the clients; we refer to the conventional alternative as centralized training. Each client instead trains a local model and sends its updates to a central server, which aggregates them into a global model. FL reduces raw-data exposure but does not by itself guarantee privacy: model updates and the released global model can still leak information ([41]), which is why FL is often combined with secure aggregation or differential privacy.

FL has been applied to recommender systems [19, 20, 79], finance [13], edge computing [112, 38, 104, 61], industrial applications [96, 117, 48], and healthcare [83, 17, 67, 12] thanks to its collaborative and raw-data-local nature. FL also demonstrates strong performance in computer vision [64, 53], natural language processing [109, 55], and automatic speech recognition [6, 119]. However, to our knowledge, it has not yet been used to address data grounding challenges in real-world monocular HMR. Existing FL-based computer vision solutions cover tasks such as object detection [64] and segmentation [53], and human-related FL work has addressed pose estimation and person re-identification [120] as well as posture recognition [103], tasks that differ significantly from monocular 3D HMR. In contrast, our approach directly regresses the 3D human mesh from standard RGB images, offering a more versatile and practical solution without the need for specialized hardware.

Other distributed training paradigms are also relevant. Split learning trains different model segments across client and server and exchanges intermediate activations rather than complete model updates [100], while FedSplit studies operator-splitting-based federated optimization [84]. These approaches may be useful for future HMR systems with different communication, compute, and leakage trade-offs. In this paper, we focus on federated parameter sharing because it directly tests whether existing HMR backbones can be trained without centralizing raw user images, and because it is readily supported by mature benchmark infrastructure.

Method

Central to the PCHMR framework is an HMR model $\mathcal{M}$ with parameters $w$, which takes a monocular image $I \in\mathbb{R}^{H \times W \times3}$ as input and outputs a 3D human mesh $\mathcal{V} \in\mathbb{R}^{6890 \times3}$ in the SMPL [65] topology.

Under our privacy-constrained setting, the HMR model is trained on a distributed HMR dataset, divided into disjoint subsets $\mathcal{D} = \bigcup_{k=1}^{n} \mathcal{D}_k$ and $\mathcal{D}_i \cap\mathcal{D}_j = \varnothing\ \forall i \ne j$, where each subset $\mathcal{D}_k$ is locally collected and annotated on device $k$. Each local dataset consists of $n_k$ locally annotated image-human mesh pairs, $\mathcal{D}_k = \{(\mathcal{I}_i,\mathcal{V}_i)\}_{i=1}^{n_k}$. An overview of the PCHMR framework is shown in Figure 2.

We first conduct a comprehensive benchmark of HMR in privacy-constrained real-world settings in section 3.1. Then, in section 3.2, we introduce and evaluate a local data annotator, named DePoser, which enables human mesh annotation for privacy-constrained HMR training.

Privacy-Constrained Benchmark Construction

We create a client-server framework to train HMR models in a raw-data-local way to construct our benchmark. Specifically, we use the Flower [7] library to simulate a federated training setup. While Secure Multi-Party Computation (MPC) [45] and Differential Privacy (DP) [21, 1] are also considered for privacy-preserving machine learning methods, due to the computation and communication overhead of completely setting up MPC training and the performance degradation of DP under large-scale models (such as SOTA HMR models), we primarily focus on benchmarking under the Federated Learning [71] approach. We consider the basic setup for federated training and the secure-aggregation variant enabled by MPC. Additional details on MPC and DP are discussed in Appendix F.

Privacy Scope and Threat Model. Our benchmark evaluates utility when raw images and mesh annotations remain on client devices. In vanilla FL, the server still receives individual model updates, so the protocol reduces raw-data centralization but does not by itself rule out update-level or released-model leakage. In the secure-aggregation variant, we assume an honest-but-curious server and use MPC to prevent the server from observing individual client updates before aggregation. We do not claim formal differential privacy and do not perform membership-inference, gradient-inversion, or model-inversion attacks in this work; these attack-based privacy evaluations are important future extensions.

We conduct a series of comprehensive experiments on two popular HMR models (Cho et al., [14]; Dou et al., [18]) in various settings that closely reflect the complexities of the real-world training environment:

Large Number of Participating Devices. In a typical real-world privacy-respecting setup, a large number of user devices are often involved (Huba et al., [35]; Bonawitz et al., [9]). In addition, due to varied usage patterns, it is impossible for each user to participate in every global fitting round. Based on these observations, we perform experiments with 10, 100, and 1000 simulated user devices (clients), with data randomly partitioned for each device to simulate locally collected and stored private training data. The number of our clients is limited by the total training dataset size ($\sim$500,000 samples), giving roughly 500 (1,000 clients) to 50,000 (10 clients) samples per client. In each global training round, 10 clients are randomly selected as participants, each of which trains for one local epoch ($E = 1$), a single pass over its own shard, using the locally partitioned data. Then, the model weights are uploaded to the central server and aggregated, while the local dataset remains local for raw-data locality.

Heterogeneous Data Distribution. In the privacy-constrained HMR training setup, the training data are collected and annotated locally. Thus, each user’s locally annotated dataset can differ substantially in data distribution and volume from another user’s local dataset due to variation in human identity, body shape, attire, and usage environment. We employ multiple methods to simulate this highly non-uniform data distribution:

1. We use the Dirichlet-based label partition commonly called Latent Dirichlet Allocation (LDA) in FL (Hsu et al., [34]), with the parameter $\alpha$ to control the degree of heterogeneity of the distribution. See Appendix § 15.§ 1 for a visualization.

2. We create a mixed dataset comprising a series of datasets (Ionescu et al., [36]; Lassner et al., [46]; Lin et al., [60]; Andriluka et al., [3]; Mehta et al., [73]) with 3DPW (Von Marcard et al., [101]) and assign clients according to datasets. This creates a more heterogeneous setup, benchmarking the SOTA HMR model’s capability under challenging training settings.

3. To simulate real-world fine-tuning, where a centrally pre-trained HMR model is fine-tuned with user-side data, we perform privacy-constrained fine-tuning on the in-the-wild 3DPW (Von Marcard et al., [101]) dataset. We further adopt the challenging setup of assigning each scene to a participating client, which resembles real-world use cases where each client uses their edge device in a different environment.

Privacy-Constrained Personalized Fine-Tuning. Data distribution alignment is essential in real-world deployment of HMR models, which could be effectively achieved by directly fine-tuning on the end user’s data distribution. To evaluate the effectiveness of model personalized fine-tuning, where the federated-trained HMR model is further fine-tuned on each user’s local dataset, we collect a dataset of over 3000 images from the internet and in-house VR usage footage, comprising 4 different scenes with users using VR and AR devices. The internet subset is drawn from publicly accessible videos, and the in-house subset is recorded with participant consent for research use. We further randomly partition the dataset into training and validation sets, fine-tuning the model for 2 epochs over the user’s local training split and then evaluating on the validation sets.

DePoser: Depth-Aware Local Annotator

Personalizing an HMR model requires annotated user-side data, but privacy constraints and the sensitivity of human images rule out collecting such images centrally. Privacy-constrained HMR training therefore needs an accurate annotation pipeline that runs on the device.

Optimization-based HMR methods ([8, 86]) are suitable candidates for local annotation due to their robustness and accuracy. These methods often use 2D keypoints to guide the optimization process of the human mesh. However, relying solely on 2D keypoints introduces monocular ambiguity, often leading to solutions that produce plausible 2D re-projections but suffer from poor 3D accuracy. Inspired by SMPLify-X ([86]) and DICE ([105]), we devise a depth-aware monocular local annotation pipeline named DePoser, where the mesh is supervised not only by 2D keypoints but also by scale-invariant depth information in the z-direction. First, we extract a depth map and 2D keypoints from a large human foundation model, Sapiens ([44]). By taking advantage of a model pre-trained on vast in-the-wild human data, our model benefits from a strong human prior, and the fitting objective is driven by 2.5D rather than 2D cues for in-the-wild user-side scenarios. Second, our method does not require expensive hardware such as RGB-D sensors, LiDAR sensors, or synchronized multi-view cameras, making it possible to deploy DePoser to ordinary consumer-grade devices with only monocular cameras.

Specifically, given an RGB image, we first use Sapiens-pose ([44]) to extract 2D human keypoints. Then, the corresponding pseudo-depth of each keypoint is extracted from a depth map obtained using Sapiens-depth ([44]). Subsequently, we minimize the following objective function with the L-BFGS optimizer ([81]):

where $E_J$ is the 2D joint projection loss, $E_{\theta_b}$ is the pose prior ([86]) loss, $E_{\alpha} = \sum_{i\in(\mathrm{elbows,knees})} \exp(\theta_i)$ is the elbow/knee bending regularizer, $E_{\beta} = \lVert\beta\rVert^2$ is the shape regularizer, $E_C$ the interpretation penalty, and $E_d$ the scale-invariant logarithmic (SILog) ([22]) loss:

where $d_i = \log z_i - \log\hat{z}_i$ and $\hat{z}_i, z_i$ are the predicted and pseudo-GT depth values of the $i$-th joint, $\mathcal{J}$ is the set of joints, and $N = |\mathcal{J}|$. Rescaling all predicted depths shifts every $d_i$ by a common constant. Since $E_d^2 = \operatorname{Var}(d) + (1-\lambda_{\mathrm{si}})\bar{d}^{\,2}$, the variance term is unaffected and only the down-weighted mean term changes, so $E_d$ reduces sensitivity to the global depth scale and emphasizes the relative depth structure across joints; it is strictly scale-invariant only for $\lambda_{\mathrm{si}} = 1$. We use the standard setting $\lambda_{\mathrm{si}} = 0.5$ of Eigen et al. [22].

Evaluation

Experimental Setup

Datasets. Following previous works ([14, 18, 58]), we use the Human3.6M ([36]), 3DPW ([101]), UP-3D ([46]), COCO ([60]), MPII ([3]), and MuCo-3DHP ([73]) datasets to train our models. We evaluate our model using the Human3.6M [36] and 3DPW [101] datasets.

In addition, we also curate a dataset from internet video footage and in-house VR usage. This dataset consists of over 3000 image-mesh pairs with 4 different scenes and identities. We conduct local fine-tuning and testing experiments on this dataset. For the internet-sourced subset, we use publicly accessible videos and retain source metadata for reproducibility. For the in-house subset, participants consented to research use of the recordings. Since the mesh labels are generated by DePoser rather than captured from an independent multi-view or marker-based system, we report these experiments as pseudo-labeled personalization studies and do not claim independent ground-truth coverage or demographic representativeness. Annotation quality is controlled through the quantitative DePoser comparison on 3DPW in § 4.3 and by excluding cases where the local fitting pipeline fails to produce a valid body fit.

Evaluation Metrics. Following previous works [14, 18, 58], we use Mean Per Joint Position Error (MPJPE) [36, 118] and Procrustes-Aligned Mean Per Joint Position Error (PA-MPJPE) [85] to evaluate our models’ accuracy. Mean Per Vertex Position Error (MPVPE) [85] is also calculated when a ground-truth 3D mesh is available. All the error metrics are in millimeters (mm).

Simulation. Following previous works that adopted the federated learning paradigm [40, 66, 68, 26, 39], we conduct our experiments in a simulated environment using Flower [7] to avoid dealing with low-level implementations such as communication and resource scheduling.

Reproducibility and Release Plan. All reported client partitions are generated with fixed random seeds and fixed partition metadata. Upon publication we will release code, but no data. The release covers both contributions: the Flower-based PCHMR harness and the DePoser implementation, including their training, evaluation, and deterministic data-splitting code together with the hyperparameter configurations of every reported run. Because the datasets we train and evaluate on [36, 46, 60, 3, 73, 101] are the property of their respective rights holders, we redistribute none of them; users obtain them directly from their owners under the original terms, and we supply a guide for preparing them and replicating our results with our code. The VR footage we curated for the personalization study is not released in any form (see the Broader Impact Statement).

Privacy-Constrained Evaluations

We choose two representative SOTA HMR models [14, 18] for their robustness and accuracy. Note that, due to the well-modularized nature of the HMR community, other models can be easily incorporated into our benchmark pipeline. For the chosen models, we perform extensive experiments in a series of simulated settings that mimic scenarios encountered in real-world privacy-constrained training. We group the experiments into categories and outline each category in the following sections.

Performance on a Large Number of Clients

We evaluate the performance of HMR models when trained under a simulated Federated Learning environment with 10, 100, and 1000 participating devices, where 10 devices are randomly selected to participate at each global round. The environment has a client-server configuration, where each client trains the model for one local epoch ($E = 1$), i.e. a single pass over its own shard of the data, and sends the model weights to a central server for aggregation using the FedAvg [71] algorithm.

As shown in Table 1, both FastMETRO [14] and TORE [18] trained under the FedAvg optimizer maintain competitive accuracy even in the most challenging setting of 1000 clients, with roughly a 2–4 mm drop in MPJPE and PA-MPJPE. In the 10-client setting, the HMR models even exhibit numerically higher accuracy than their centrally trained counterparts. This underscores the robustness of the benchmarked state-of-the-art HMR models under the complex federated setting. Qualitative results for HMR models trained with the 100-client configuration are shown in Figure 4.

#Clients (Total / Per Round)	Model	MPJPE	PA-MPJPE	Raw Data Local?
Centralized	FM-S	57.98	40.62	✗
10 / 10	FM-S	56.85	40.52	✓
100 / 10	FM-S	59.31	41.48	✓
1000 / 10	FM-S	60.82	44.53	✓
Centralized	TORE-S	63.88	41.99	✗
10 / 10	TORE-S	61.27	41.60	✓
100 / 10	TORE-S	62.92	43.11	✓
1000 / 10	TORE-S	66.04	44.04	✓

caption: tbl:3: Mixed Dataset Training. Comparison between FastMETRO-S [14] and TORE-S [18] trained under centralized or privacy-constrained settings on a mixed dataset composed of Human3.6M, UP-3D, COCO, MuCo-3DHP, and MPII [36, 46, 60, 73, 3], evaluated on Human3.6M.

:::

While the HMR models achieve performance close to centrally trained models, one caveat is that they train much more slowly: they require many more global rounds to converge, with the most complex 1000-client setting requiring 6000 rounds. Please refer to Appendix § 7 for more details, including the terminology we use to describe the training budget of each regime.

Performance on Skewed Data Distribution

In the real world, user data from user devices may vary in data distribution and sample size due to differences in identity, environment, and habits. Therefore, we manually skew the data distribution in various configurations to simulate this heterogeneity and measure its effect on HMR accuracy. We investigate three types of heterogeneous data distributions: Latent Dirichlet Allocation (LDA), Mixed Dataset, and Natural Partition Data. More details on the experiment setup are in Appendix § 9.

Latent Dirichlet Allocation. As shown in Figure 3, the models under a highly heterogeneous LDA setup perform the best under the most skewed distribution, $\alpha= 0.1$, with both MPJPE and PA-MPJPE errors lower than the centralized baseline for both configurations.

A plausible explanation is that local updates from highly non-uniform clients introduce an aggregation-induced regularization effect akin to meta-learning [80], but our current experiments do not isolate this hypothesis from centralized-run variance or remaining training-budget differences. In addition, across a wide range of $\alpha$, the model performance is consistently competitive compared to the centralized baseline. These results support the robustness of the benchmarked HMR models [14, 18] under highly heterogeneous setups close to real-world scenarios.

Mixed Dataset. As shown in Tab. 3, HMR models trained with PCHMR on the mixed dataset match their centralized counterparts on the majority dataset (Human3.6M): MPJPE improves by 3.1 mm for FastMETRO-S and 2.3 mm for TORE-S, and PA-MPJPE stays within 1.2 mm of the centralized result. This suggests that HMR models trained under the privacy-constrained federated training framework can achieve competitive performance on the overall data distribution compared to the traditional centralized training framework.

Natural Partition Data. As shown in Table 4, under both partition schemes, FastMETRO and TORE stay within 2.1 mm of their centralized counterparts on every metric; the largest gap is FastMETRO under the natural partition (+1.9 mm MPJPE, +2.1 mm MPVPE). The results show that privacy-constrained distributed fine-tuning of HMR models can achieve performance comparable to centralized fine-tuning, even under the challenging natural partition scheme.

Setting	Model	MPJPE	PA-MPJPE	MPVPE	Raw Data Local?
Centralized	FM-S	84.92	54.69	97.60	✗
PCHMR	FM-S	84.66	54.78	97.42	✓
PCHMR (Natural Part.)	FM-S	86.83	55.61	99.66	✓
Centralized	TORE-S	87.97	55.35	101.88	✗
PCHMR	TORE-S	87.55	54.08	101.86	✓
PCHMR (Natural Part.)	TORE-S	88.00	55.07	102.50	✓

Comparison between Federated Optimizers

In the previous section, experiment results confirmed the robustness of FastMETRO (Cho et al., [14]) and TORE (Dou et al., [18]) under different forms of data skewness. We now benchmark the HMR models on three popular federated learning algorithms, namely FedAvg (McMahan et al., [71]), FedOpt (Reddi et al., [88]), and FedProx (Li et al., [52]). The latter two more sophisticated federated optimizers offer better adaptiveness and robustness than FedAvg.

As shown in Table 2, in most of the settings evaluated, the three optimizers performed similarly in terms of both MPJPE and PA-MPJPE, with no clear trend of superiority or inferiority. This suggests the robustness of HMR models coupled with the simplest FedAvg (McMahan et al., [71]) under a highly skewed data setting (LDA $\alpha= 0.5$).

Optimizer	FM-S Model — MPJPE	FM-S Model — PA-MPJPE	TORE-S Model — MPJPE	TORE-S Model — PA-MPJPE
IID Distribution
FedAvg	59.31	41.48	62.92	43.11
FedOpt	58.49	41.43	62.82	42.34
FedProx	57.71	41.26	63.41	43.17
LDA $\alpha= 0.5$ Distribution
FedAvg	57.80	39.82	58.91	40.94
FedOpt	55.41	39.36	59.34	41.92
FedProx	56.46	39.58	61.57	41.10
LDA $\alpha= 1$ Distribution
FedAvg	59.63	39.80	62.39	41.76
FedOpt	58.10	40.10	59.57	42.10
FedProx	56.45	39.28	58.96	41.97
LDA $\alpha= 10$ Distribution
FedAvg	56.09	39.84	60.96	42.25
FedOpt	58.25	40.47	58.83	41.29
FedProx	57.33	40.27	58.96	41.57

Personalized Fine-tuning

Although HMR models could maintain competitive performance under challenging real-world multi-device collaborative training settings, further personalized fine-tuning is often desired to tailor the model to a specific user’s data distribution. Therefore, we benchmark further fine-tuning a collaboratively pre-trained FastMETRO (Cho et al., [14]) model on additional in-the-wild datasets that reflect real-world use cases, such as VR gaming and sports. We use our local data annotator, DePoser, to annotate pseudo-ground truth for the collected dataset. For local fine-tuning, we divide each scene of our dataset into an 80/20 random split and fine-tune the pre-trained model for two epochs over the user’s local training split.

As shown in Figure 5, our personalized fine-tuned model reconstructs human meshes that more accurately resemble the human pose and shape after fine-tuning on the pseudo-GT. Quantitatively, as shown in Table 5, MPVPE drops by up to 110.7 mm after local fine-tuning, with the largest improvement in VR-game-1, a challenging home VR gaming setting. Because the evaluation labels in this table are generated by DePoser, these numbers measure consistency with the local pseudo-label distribution rather than accuracy against independent ground truth. In addition, we perform the personalized fine-tuning using the 3DPW dataset and observe a similar performance improvement. Details are shown in Appendix § 16. This demonstrates that the strong prior of an HMR model pre-trained on a large amount of data can enable effective personalization with only two epochs of fine-tuning, significantly improving end-user performance through model personalization.

Dataset	Before/After	Model	MPVPE	MPJPE	PA-MPJPE
VR-runner	Before	PCHMR	FM-S	109.50	114.51	60.07
VR-runner	After	PCHMR	FM-S	93.23	58.24	43.16
VR-game-1	Before	PCHMR	FM-S	176.24	172.57	78.21
VR-game-1	After	PCHMR	FM-S	65.57	62.30	43.82
VR-game-2	Before	PCHMR	FM-S	150.64	161.82	84.36
VR-game-2	After	PCHMR	FM-S	82.53	69.49	53.57
Oculus	Before	PCHMR	FM-S	103.92	102.22	61.62
Oculus	After	PCHMR	FM-S	56.76	57.92	39.94

Local Annotation Pipeline

Privacy-constrained HMR training depends on accurate local annotation. We therefore compare DePoser with SMPLify-X (Pavlakos et al., [86]) to measure the effect of the Sapiens (Khirodkar et al., [44]) pose and depth priors. The comparison evaluates the full DePoser pipeline and does not ablate its components (keypoint source, depth term, metric-scale alignment). More details are available in Appendix § 10.

Quantitative Analysis. We compare the human reconstruction accuracy of our DePoser and SMPLify-X (Pavlakos et al., [86]) on a 300-image randomly sub-sampled set of the in-the-wild dataset 3DPW (Von Marcard et al., [101]), which resembles complex user-side conditions. As shown in Table 6, DePoser demonstrates improved accuracy compared to SMPLify-X, with more than 30 mm accuracy improvement in MPVPE and 15 mm improvement in PA-MPJPE, supporting the effectiveness of the combined local annotation pipeline under complex conditions.

Model	MPVPE	MPJPE	PA-MPJPE
SMPLify-X (Pavlakos et al., [86])	198.66	181.07	87.17
DePoser (Ours)	162.60	151.19	71.14

Qualitative Analysis. As shown in Figure 6, although the front-view results of DePoser and SMPLify-X (Pavlakos et al., [86]) are similar, DePoser places the feet and arms more accurately along the $z$-axis (depth) than SMPLify-X, which is evident in the side view. This visual evidence is consistent with the quantitative improvement. The figure compares two annotators that differ in keypoint source, depth term, and depth alignment simultaneously.

Limitations and Future Work

HMR-aware Aggregation Strategy. Our LDA results suggest that aggregating updates from heterogeneous clients may regularize HMR training. We have not verified this mechanism; matched-budget controlled studies are needed to separate it from optimization variance and schedule differences.

Device-specific and Personalized Learning. In real-world human mesh recovery, each device would introduce unique data perturbation into the client data distribution, such as smearing, tearing, and fuzziness. However, there is no existing solution for simulating this issue. Having a personalized model adapted to device perturbations would be a promising direction for future work.

Privacy Evaluation Scope. Our experiments measure HMR utility with raw images kept local and the runtime cost of a secure-aggregation variant. We do not evaluate privacy attacks or provide formal guarantees (see the Broader Impact Statement); both are important future work.

Local Annotation Bias. DePoser produces SMPL-parametric pseudo-annotations, which are then used to supervise non-parametric HMR regressors. The personalization gains therefore measure agreement with DePoser’s pseudo-labels and are not validated against independent ground truth. The DePoser comparison also evaluates the full pipeline (Sapiens pose, Sapiens depth, metric alignment) without ablating each component.

Conclusion

In this work, we introduced PCHMR, a benchmark for evaluating human mesh recovery (HMR) models under privacy-constrained, raw-data-local training settings. Across client scales and data-heterogeneity settings, we showed that state-of-the-art HMR models can be collaboratively trained with a large number of clients and heterogeneous data while incurring only minimal accuracy loss. Our study also reveals remaining challenges, including slower convergence and degraded performance when generalizing to out-of-distribution data, which mark remaining gaps between centralized and privacy-constrained training regimes. To address the data availability bottleneck in privacy-constrained HMR, we proposed DePoser, a local data annotation pipeline that conditions not only on 2D keypoints but also on pseudo-depth information. Comparisons against SMPLify-X demonstrate that DePoser improves pseudo-annotation quality under the combined Sapiens pose and depth priors. Using DePoser, we curated and annotated an in-the-wild dataset of VR use cases, and showed that privacy-constrained fine-tuning on this dataset yields substantial gains for HMR model personalization on end-user devices when evaluated against the same pseudo-label source. We hope that our benchmark, analysis, and the DePoser pipeline will serve as a foundation for future research on privacy-constrained human mesh recovery, including more robust collaborative training algorithms, stronger privacy guarantees, and better on-device personalization strategies.

Broader Impact Statement

This work trains and evaluates human mesh recovery models on images of people. Two aspects therefore warrant explicit discussion: the provenance of the in-the-wild dataset we collected for the personalization study, and how the term privacy-constrained, used throughout the paper, should and should not be read.

Data Provenance and Consent. The benchmark experiments use only established public HMR datasets [36, 46, 60, 3, 73, 101] under their original terms, none of which we redistribute. The VR dataset used for personalization (section 4.1) is new and has two subsets. The in-house subset was recorded from participants who gave informed consent to the research use of their footage. This footage is used for training and evaluation only and is not redistributed. The internet subset is drawn from publicly accessible videos of people using consumer VR and AR devices. These individuals are identifiable and were not asked for consent to be included in a research dataset, which we regard as the most sensitive aspect of this work. We use the footage solely to fit and evaluate body meshes: no face recognition, identity linkage, or demographic attribute inference is performed, and none is released. With four scenes and four identities, the dataset is a personalization stress test rather than a demographic coverage benchmark, and models fine-tuned on it should not be assumed to behave equitably across body shapes, skin tones, or mobility profiles.

Release Plan. We release code, not data (section 4.1). Upon publication we will make the PCHMR harness and the DePoser implementation available, covering the training, evaluation, and data-splitting code for both, so that our experiments can be reproduced end to end. We redistribute no imagery. The public HMR datasets we train and evaluate on remain the property of their respective rights holders and must be obtained from them under their original terms; for these we provide a guide to preparing the data and replicating our results rather than a copy of the data. The VR footage curated for the personalization study is not released in any form, neither the internet-sourced videos nor the pseudo-annotations derived from them. This is the most conservative option open to us, and we prefer it to the alternative: it costs reproducibility, since the personalization experiments cannot be re-run from our release without collecting comparable footage, but it avoids redistributing imagery of identifiable people who never consented to research use, and avoids attaching body-mesh annotations to those people in a public artifact.

The Limits of “Privacy-Constrained”. We use privacy-constrained to name a training setting, one in which raw images and mesh annotations never leave the client device, and not to assert a privacy guarantee. Raw-data locality is not privacy: individual model updates, and the released global model, may still leak information about the local data that produced them [41]. We perform no attack-based privacy evaluation in this work, and therefore do not claim protection against membership-inference, gradient-inversion, or model-inversion attacks. The secure-aggregation variant assumes an honest-but-curious server and prevents it from observing individual client updates, but we report it as a small-scale runtime feasibility check rather than a security analysis, and we do not claim formal differential privacy for any model we release. Our results should accordingly not be read as establishing regulatory compliance, nor as warrant for training HMR models on user data without the additional safeguards, such as formal guarantees and empirical attack-based audits, that a real deployment would require.

This supplementary material covers: More Experiment Details (section A); Training Budget of the Centralized and Federated Regimes (section B); More Details on Skewed Data Distribution (section C); Local Data Annotation (section D); More Implementation Details (section E); Discussion on MPC and DP (section F); Discussion on Single-Device Training (section G); Discussion on Multiple Local Epochs (section H); Discussion on Performance of 3DPW Fine-tuning (section J); Discussion on Data Distribution (section I); Additional Experiments for On-Device Performance (section K).

Experiment Hyperparameters

Terminology. We follow the standard federated-learning vocabulary [71, 9, 41] and reserve each term for exactly one quantity. A global round (or simply round) is one server-side aggregation cycle: the server samples a set of clients, each of them trains on its own data, and the resulting model updates are aggregated into a new global model; $T$ denotes the number of rounds. $K$ is the total number of clients and $m$ the number sampled uniformly at random per round, reported throughout as $K/m$. A local epoch ($E$) is one pass a sampled client makes over its own shard of $n_i$ samples within a round. A local step, equivalently an optimizer step, is a single gradient update on one mini-batch, so a client performs $\tau_i = E\lceil n_i/B\rceil$ of them per round at local batch size $B$. A client update is the payload a client returns to the server, not a gradient update. The unqualified word epoch always means one pass over the full training set and is used only for centralized training; federated budgets are quoted in rounds. To compare the two regimes on a common axis we count sample-visits (one image passed through the model once) and convert them to epoch-equivalents by dividing by the size of the training set.

Fixing $E$ fixes the number of local passes but not the amount of local work, because $\tau_i$ grows with the shard size $n_i$. Under the IID partitions all shards are the same size and the distinction is immaterial, but under the skewed partitions of section C the shards differ by orders of magnitude, so a single local epoch amounts to very different numbers of optimizer steps across clients. We therefore report both quantities where the distinction matters.

In Table 7 we present the hyperparameters for the experiments in § 4 of the main paper. The Epochs column applies only to the centralized runs, which have no rounds; the federated runs are instead specified by their local epochs $E$ and their global rounds $T$.

Setting	Model	Clients	Learning Rate	Epochs	Local Epochs ($E$)	Global Rounds ($T$)
Centralized Baseline	FM-S	-	0.001	60	–	–
Human3.6M naïve on-device training	FM-S	-	0.001	60	–	–
3DPW Centralized Fine-tuning	FM-S	-	0.0001	20	–	–
PCHMR + IID Data 10 / 10	FM-S	10 / 10	0.001	–	1	200
PCHMR + IID Data 100 / 10	FM-S	100 / 10	0.001	–	1	2000
PCHMR + IID Data 1000 / 10	FM-S	1000 / 10	0.001	–	1	6000
PCHMR + LDA Distribution	FM-S	100 / 10	0.001	–	1	2000
PCHMR + CNN Cluster	FM-S	100 / 10	0.001	–	1	2000
PCHMR + 3DPW Fine-tuning	FM-S	100 / 10	0.0001	–	1	600
PCHMR + 3DPW Natural Part.	FM-S	20 / 5	0.0001	–	1	200
Centralized Baseline	TORE-S	-	0.001	60	–	–
Human3.6M naïve on-device training	TORE-S	-	0.001	60	–	–
3DPW Centralized Fine-tuning	TORE-S	-	0.0001	20	–	–
PCHMR + IID Data 10 / 10	TORE-S	10 / 10	0.0005	–	1	200
PCHMR + IID Data 100 / 10	TORE-S	100 / 10	0.0005	–	1	2000
PCHMR + IID Data 1000 / 10	TORE-S	1000 / 10	0.001	–	1	6000
PCHMR + LDA Distribution	TORE-S	100 / 10	0.0005	–	1	2000
PCHMR + CNN Cluster	TORE-S	100 / 10	0.0005	–	1	2000
PCHMR + 3DPW Fine-tuning	TORE-S	100 / 10	0.0001	–	1	600
PCHMR + 3DPW Natural Part.	TORE-S	20 / 5	0.0001	–	1	200

Training Budget of the Centralized and Federated Regimes

Because several of our comparisons place a federated run next to a centrally trained baseline, we report here, for every regime in the paper, the quantities that determine how much training each of them actually received: total optimizer steps, total data passes, wall-clock cost, checkpoint-selection rule and validation protocol. Table 8 collects them.

Regime	Budget	Eff. batch	Optimizer steps	Sample-visits	Epoch-equiv.	Critical path	FM-S	TORE-S	Val. evals
Centralized (as published)	60 epochs	512	55,800	28,563,720	60.0	–	18.9	18.7	60
Centralized, budget-matched	200 epochs	512	186,000	95,212,400	200.0	–	63.0	62.3	200
Centralized, step-matched	200 epochs	64	1,487,800	95,212,400	200.0	–	≈ 500†	≈ 500†	200
PCHMR, IID 10/10	200 rounds	64	1,488,000	95,212,400	200.0	148,800	44.3	47.1	200
PCHMR, IID 100/10	2000 rounds	64	1,500,203	95,212,580	200.0	150,203	60.5	55.9	200
PCHMR, IID 1000/10	6000 rounds	64	480,060	28,563,726	60.0	48,060	56.3	60.6	600
PCHMR, LDA $\alpha= 0.1$	2000 rounds	64	1,496,597	95,124,910	199.8	707,920	87.4	77.7	200
PCHMR, LDA $\alpha= 0.5$	2000 rounds	64	1,496,536	95,151,465	199.9	437,122	65.2	61.5	200
PCHMR, LDA $\alpha= 1$	2000 rounds	64	1,497,281	95,207,630	200.0	348,?	65.9	59.0	200
PCHMR, LDA $\alpha= 10$	2000 rounds	64	1,497,717	95,228,649	200.0	204,363	55.5	56.0	200
PCHMR, LDA $\alpha= 100$	2000 rounds	64	1,497,416	95,215,663	200.0	166,057	59.4	57.6	200
PCHMR, LDA $\alpha= 1000$	2000 rounds	64	1,497,387	95,213,672	200.0	154,767	59.2	57.4	200

The quantities in table 8 follow from a few fixed properties of our setup. The training corpus contains 476,062 images, so one epoch-equivalent is 476,062 sample-visits. Every federated client trains with a local batch of $B = 64$ on a single device, whereas the centralized baselines are trained with data parallelism over eight devices and therefore have an effective batch of 512, i.e. $\lceil476{,}062/512 \rceil= 930$ optimizer steps per epoch. This asymmetry is the reason the two regimes cannot be matched in data passes and in gradient steps by the same run, and it is why we also list a hypothetical single-device centralized run at $B = 64$. For the federated rows we additionally report the critical path: the number of optimizer steps summed over rounds of the slowest client sampled in each round, which is the update depth a deployment would actually wait through, since a round cannot close until its last client reports. Total steps and sample-visits are backbone-independent and are obtained from the partition sizes and the sampling rule; wall-clock is the one quantity that depends on the backbone, so we report FastMETRO-S and TORE-S separately. The step-matched configuration, which we did not run to completion, is extrapolated from 200-epoch runs.

Validation protocol and checkpoint selection. Both regimes are validated on exactly the same data, the Human3.6M protocol-2 split of 27,558 images, with a validation batch of 256, and the reported metrics are computed on that split as well, so no separate held-out test set is used; the numbers should be read accordingly. Federated evaluation is server-side and centralized: the aggregated global model, not any client model, is evaluated on the full validation set, and client-side validation is disabled. Centralized runs evaluate and checkpoint once per epoch; federated runs evaluate and checkpoint once per round for the 200-round setting and once every ten rounds for the 2000- and 6000-round settings, which yields the evaluation counts in the last column of table 8. In every regime the value we report is the best evaluation on that validation set over the course of the run, i.e. best epoch for centralized runs and best round for federated ones.

Centralized versus federated. The comparison is not budget-matched, and the mismatch goes in opposite directions depending on which quantity one counts. In data passes, the 10/10, 100/10 and all LDA runs receive 200 epoch-equivalents against the published baseline’s 60, a factor of 3.33; the 1000/10 run is the only configuration matched in data passes, and it is also the configuration that trails the centralized baseline on every metric. In gradient steps the ordering reverses: the baseline takes only 55,800 optimizer steps because data parallelism raises its effective batch to 512, whereas the federated clients train at batch 64 and take between 480,060 and 1,500,203, so even the data-matched 1000/10 run performs 8.6× more updates than the baseline it is compared against. No single centralized re-run can close both gaps at once unless the batch size is matched too: a 200-epoch run at batch 512 equalizes sample-visits only, while a 200-epoch run at batch 64 equalizes both, at 1,487,800 steps against 10/10’s 1,488,000 on identical sample-visits. Wall-clock is not matched either, and not in the federated regime’s favor: the published baseline costs 18.7 h for TORE-S and about 18.9 h for FastMETRO-S, against 44.3–87.4 h for the federated runs, so every federated configuration is between two and five times as expensive to train as the baseline it is compared against, on both backbones. We therefore do not attribute any accuracy the federated runs gain over the centralized baseline solely to the federated protocol itself: the comparisons in which PCHMR matches or slightly exceeds centralized training were obtained with more data passes and far more gradient updates than the baseline received, and the one configuration that is matched in data passes (1000/10) does not win in accuracy.

Learning curves. Figure 7 plots the validation error of the budget-matched centralized re-run and the federated runs against each of these budget measures. Read against sample-visits, the two regimes are directly comparable, and the 200-epoch centralized run reaches the same number of visits as the 10/10, 100/10 and LDA runs. Read against optimizer steps, their positions on the axis reverse, since the centralized run covers the same data in 8–27× fewer, larger steps. Three features of the curves matter for the comparison. First, the centralized runs reach their best validation error early — epoch 26 for TORE-S and epoch 27 for FastMETRO-S, both well inside the published 60-epoch budget — and then flatten and slowly degrade, so the centralized baseline is not under-trained and the extra data passes are spent past the point of convergence. This is the convergence-matched reading of the comparison, and it holds independently of the sample-visit matching. Second, the federated curves are still descending where the centralized curve has flattened, so the two regimes are at different points of their trajectories at the moment they are compared. Third, the federated curves are visibly noisier from evaluation to evaluation, because each round evaluates a model aggregated from a different sample of clients, and that noise grows as the partition becomes more skewed.

The effect of $\alpha$ on the budget. Changing the Dirichlet concentration does not change how much training the model receives. Across $\alpha\in[0.1,1000]$ the LDA runs all take ≈1.497M optimizer steps and see 199.8–200.0 centralized epoch-equivalents, within 0.5% of the IID 100/10 run at the same number of rounds. This is structural rather than coincidental: the Dirichlet partition assigns all 476,062 images regardless of $\alpha$, and clients are sampled uniformly, so the expected amount of data processed per round is the same at every skew, and only its round-to-round variance grows as $\alpha$ falls. What skew does change is how that fixed amount of work is distributed across the clients of a round, which is captured by the critical path: it grows 4.6× from 154,767 steps at $\alpha= 1000$ to 707,920 at $\alpha= 0.1$, because at high skew nearly every round samples at least one client holding a very large shard and must wait for it. Wall-clock follows the critical path rather than the total step count, and does so on both backbones: it rises from 57.4 h at $\alpha= 1000$ to 77.7 h at $\alpha= 0.1$ for TORE-S and from about 59.2 h to 87.4 h for FastMETRO-S, increases of 35% and 48% for the same number of gradient updates. Relative to the IID run at the same number of rounds the penalty is of the same size, 39% for TORE-S and 44% for FastMETRO-S. The same straggler effect would appear, more severely, in a real cross-device deployment, where the slowest device is slow for reasons other than shard size. The practical consequence for this paper is that training budget cannot explain our most surprising observation, that the most skewed partitions perform comparably to or slightly better than centralized training: the IID 100/10 run has an identical budget and does not show the effect, so the difference between them lies in the partition and the aggregation dynamics it induces, not in the amount of training. Budget does, however, remain a live explanation for the federated-versus-centralized comparison itself, as discussed above.

Skewed Data Distribution

We present the details of the experimental setup for the three skewed data distributions discussed in § 4.2.2: Latent Dirichlet Allocation (LDA), Mixed Dataset, and Natural Partition Data.

Latent Dirichlet Allocation. To test the HMR model’s performance under a heterogeneous data setting without any prior partition information available, we follow Tang et al. [95]; Hsu et al. [34]; Grativol

et al. (2023) in using Dirichlet-based label partitioning, referred to as Latent Dirichlet Allocation (LDA) in the FL literature, for generating skewed training datasets to approximate real-world data distributions. The LDA partition scheme uses the $\alpha$ parameter to control the skewness of the generated partition with the label. A smaller $\alpha$ corresponds to a more skewed distribution, while a larger $\alpha$ indicates that the distribution is close to IID (independent and identically distributed). More details on LDA are discussed in Appendix § 15.§ 1. For LDA, we investigate FastMETRO and TORE’s performance with 100 clients and 10 clients per round, trained with datasets of varying skewness controlled by LDA.

Mixed Dataset. In addition to simulated label skewness, we also implement a mixed dataset which blends the training datasets [36, 46, 60, 3, 73]. We allocate each client to a source dataset according to the dataset size ratio, which creates a heterogeneous training corpus across clients. The experiment is performed under the 100-client setting with 10 clients fitted per round.

Natural Partition Data. We also present a study on 3DPW [101], an in-the-wild human mesh dataset collected in various indoor and outdoor scenes, which provides a natural way to partition the dataset by scenes, resulting in a highly heterogeneous per-client data distribution. Under this data partitioning, we explore the performance of the HMR model trained using a privacy-constrained method. Further information on the 3DPW natural partition is available in Appendix § 15.§ 3. As a baseline, we first fine-tune pre-trained HMR models on the 3DPW dataset under the federated framework using an IID partitioning scheme. Then, to explore the effect of this challenging partitioning scheme, we compare the IID-trained model to a model trained on the same dataset with natural partitioning. Qualitative results of the IID fine-tuned model are shown in Figure 4.

Local Data Annotation

Local data annotation, as described in § 3.1 of the main paper, plays a critical role in improving the performance of PCHMR with client-side user data. Although there are many existing annotation approaches [8, 86, 121, 57], we choose a popular method, SMPLify-X [86], as the backbone of our local data annotation. SMPLify-X is an optimization-based human mesh regressor, where a high-quality SMPL [65] mesh of 6890 vertices can be obtained as the pseudo-ground-truth mesh. The input 2D keypoints and depth supervision information for DePoser optimization are obtained through Sapiens [44].

The model is pre-trained using the PCHMR framework for 2000 global rounds with 100 clients and 10 clients per round under an IID partition. We then fine-tune the model with the locally annotated data using a learning rate of $1 \times10^{-4}$ for 100 epochs over the user’s local training split, with other training settings such as optimizers consistent with the centralized baseline.

Remark. Notably, we currently employ an optimization-based approach to obtain pseudo-ground truth from monocular images. However, end devices possessing multi-view cameras could benefit from the additional 3D spatial information extracted by multiple cameras or depth sensors, thereby enhancing the local data annotation process using actual depth information with higher accuracy.

Implementation Details

In our experiments, we develop and evaluate our model based on two backbones including FastMETRO [14] and TORE [18] using the Transformer encoder-decoder architecture. The details of this architecture are shown in Figure 8. Please refer to the original papers for further details.

Discussion on MPC and DP

We present a simple ablation setting comparing MPC and DP for the HMR model. For simplicity, we select the simplest 10/10 setting, i.e. data partitioned across 10 clients with all 10 sampled each round, with

the TORE model using an IID-partitioned Human3.6M training set. For MPC, we choose to use secure aggregation, which implements MPC at the aggregation stage and hides individual client updates from an honest-but-curious server before aggregation. This acts as a baseline setting to demonstrate the computation and communication overhead of the MPC protocol.

For DP, we choose to account with Rényi Differential Privacy [76]. We implement this with fixed clipping noise injection techniques. With the size of the Human3.6M dataset, we set $\delta= 1 \times10^{-6}$ and calculate the respective $\epsilon$ privacy budget together with the noise multiplier set. Following the suggestion on setting the L2 norm [11], we set the L2 norm to a small value, starting with $1 \times10^{-6}$.

The results are presented in Table 9. As demonstrated by the average round time, which is almost double that of FL, the MPC approach shows significant communication and computation overhead. However, the evaluation metrics (MPJPE, PA-MPJPE) still show similar performance with MPC, suggesting that MPC may be feasible to implement in HMR real-world settings with future improvement in hardware and communication. Even worse, although we attempted an extensive search over hyperparameters, we could not get our large HMR model to converge even with a small noise scale multiplier ($\eta= 1$), resulting in a large $\epsilon= 199.7$ that offers a relatively small example-level DP guarantee. Due to the size of SOTA models such as TORE, the curse of dimensionality likely makes convergence hard, as the injected DP noise grows with the number of trained parameters [62], and how to pre-train a large-scale model with DP efficiently remains an open question for future research.

Methods	Model	Round Time	Convergence
FL	TORE	139.8s	Converge
MPC	TORE	265.5s	Converge
DP	TORE	139.8s	Non-Converge

Discussion on Single-Device Training

We investigate the performance of naïve on-device training compared with privacy-constrained collaborative training under PCHMR. We simulate single-device training with a small portion of the training dataset, and compare the accuracy with models trained with full data under centralized and privacy-constrained frameworks.

In Table 10, we pre-train the model using 5, 10, 20% of the Human3.6M dataset, which results in a large performance gap for both FastMETRO and TORE implementations. With 5% of the data, naïve training is 23–26 mm worse than centralized training in MPJPE and PA-MPJPE for FastMETRO, and 38–41 mm worse for TORE; even with 20% of the data, the gap remains 6–14 mm. Compared to the naïve single-device training setting, experiment results have confirmed the superiority of a larger training corpus, i.e. the full Human3.6M dataset.

Settings	Model	MPJPE	PA-MPJPE	Raw Data Local?
Centralized Baseline	FM-S	57.98	40.62	✗
Naïve 5% Data Training	FM-S	83.78	63.65	-
Naïve 10% Data Training	FM-S	73.04	53.36	-
Naïve 20% Data Training	FM-S	65.10	47.01	-
PCHMR 100% Data Training	FM-S	59.31	41.48	✓
Centralized Baseline	TORE-S	63.88	41.99	✗
Naïve 5% Data Training	TORE-S	101.59	82.82	-
Naïve 10% Data Training	TORE-S	84.82	63.52	-
Naïve 20% Data Training	TORE-S	77.87	52.37	-
PCHMR 100% Data Training	TORE-S	62.92	43.11	✓

Discussion on Multiple Local Epochs

We also conduct an ablation on model performance with multiple local epochs, i.e. letting each sampled client make more than one pass over its own shard before its update is aggregated. The experiment is presented with the 100/10 IID setting for the H36M dataset, i.e. data partitioned across 100 clients with 10 sampled each round.

As shown in Table 11, we found that, with more local epochs, the local model tends to overfit and results in less generalization capability and worse performance.

Local Epochs ($E$)	Model	MPJPE	PA-MPJPE	Raw Data Local?
1	FM-S	59.31	41.48	✓
2	FM-S	59.28	42.42	✓
3	FM-S	60.49	42.43	✓
5	FM-S	59.52	42.07	✓
10	FM-S	60.03	42.96	✓
1	TORE-S	62.92	43.11	✓
2	TORE-S	61.84	43.27	✓
3	TORE-S	63.53	42.74	✓
5	TORE-S	63.41	43.18	✓
10	TORE-S	63.88	43.39	✓

Discussion on Data Distribution

Heterogeneous data distributions are common in real-world privacy-constrained model training since each user’s local data distribution may be different due to user identity, environment, and user habits. We therefore evaluate PCHMR under heterogeneous data distributions. This section discusses each distribution used in § 4.2.2 in more detail.

LDA Data Partition

One of the most utilized methodologies in FL to generate non-IID data is Latent Dirichlet Allocation (LDA) [34]. Specifically, LDA works by drawing the distribution $\mathbf{q}$ over $N$ category classes of labels from a Dirichlet distribution, i.e. $\mathbf{q} \sim\operatorname{Dir}(\alpha\mathbf{p})$ where $\alpha$ is the concentration parameter controlling the skewness of the distribution and $\mathbf{p}$ is the vector characterizing the prior distribution over $N$ classes. This creates skewed datasets with an approximation to real-world non-IID distribution.

However, in our case, the label category for the training set is hard to determine. In particular, each human mesh is, in fact, a unique label by definition, i.e. they all represent unique annotations of their character. Hence, two alternative methods for considering pseudo-labeling are labeling by subjects and scenes or labeling by the dataset composition. As many datasets only have one scene per object (in particular, pseudo-3D labeled datasets), labeling by scene and subjects seems infeasible. Therefore, we opted to label it with dataset composition instead and consider the category class as the sub-dataset from which the image originated.

On an IID dataset, LDA with high $\alpha$ yields a distribution similar to uniform sampling, whereas low $\alpha$ yields a much more skewed non-IID distribution. Our training dataset is already non-IID by composition, so LDA partitioning only increases its skewness: low $\alpha$ makes it much more skewed, whereas high $\alpha$ keeps it close to the original dataset composition (Figure 9).

In addition, as shown in Figure 10, LDA also creates the client dataset size distribution following the Dirichlet distribution, i.e. lower $\alpha$ creates a much more skewed long-tail distribution in the number of samples assigned to each client. This variation in client dataset size mirrors real deployments, where clients hold very different amounts of data.

CNN Activation Clustering

Another approach to pseudo-partitioning is considering natural clustering from the features. We use the same CNN feature extractor (ResNet-50; [33]) from the PCHMR model to extract the feature from the last-layer activation. The feature $x_{\mathrm{out}}$ extracted from the last-layer activation is then clustered into 100 pseudo-partitions using MiniBatchKMeans ([92]). An example of the effects of clustering is illustrated in Figure 11.

Upon inspection, we find that the feature extractor clusters also mimic the natural dataset composition relationship of the underlying dataset: the large groups of the clusters on the left consist mostly of the Human3.6M dataset, whereas the large cluster on the right consists of the pseudo-3D labeled dataset within the training dataset mixture. Clustering on CNN features therefore yields a partition that differs from IID sampling.

Under this setting, as shown in Table 12, our TORE [18] model achieves the same level of accuracy as the centralized model, with a lower MPJPE error and slightly higher (+0.26 mm) PA-MPJPE error. The FastMETRO [14] variant also performs on par with the centralized model, achieving a lower MPJPE and slightly higher PA-MPJPE (+0.21 mm).

Settings	Model	MPJPE	PA-MPJPE	Raw Data Local?
Centralized	FM-S	57.98	40.62	✗
$\alpha= 0.1$	FM-S	51.11	37.24	✓
$\alpha= 0.5$	FM-S	57.80	39.82	✓
$\alpha= 1$	FM-S	59.63	39.80	✓
$\alpha= 10$	FM-S	56.09	39.84	✓
$\alpha= 100$	FM-S	57.53	40.69	✓
$\alpha= 1000$	FM-S	59.80	41.80	✓
CNN Cluster	FM-S	56.84	40.83	✓
Centralized	TORE-S	63.88	41.99	✗
$\alpha= 0.1$	TORE-S	57.02	39.10	✓
$\alpha= 0.5$	TORE-S	58.91	40.94	✓
$\alpha= 1$	TORE-S	62.39	41.76	✓
$\alpha= 10$	TORE-S	60.96	42.25	✓
$\alpha= 100$	TORE-S	63.08	42.59	✓
$\alpha= 1000$	TORE-S	63.74	42.50	✓
CNN Cluster	TORE-S	63.14	42.25	✓

Natural Partitioned In-the-wild Dataset

In addition, the centralized HMR model is usually trained on synthetic and indoor datasets due to the ease of gathering, not on the sparsely collected data from in-the-wild applications. These data usually exhibit a much different data distribution. We aim to further evaluate PCHMR on this real-world dataset with naturally partitioned heterogeneous distribution labels. We utilize 3DPW [101] as an example of such real-world collected in-the-wild data that presents much more challenging settings and is naturally partitioned with scene labels, which is demonstrated in Figure 12. This makes them a good representation of the natural heterogeneous distribution that PCHMR deployment is likely to face in real-world applications.

Heterogeneous Distribution Results

Table 12 lists the full results of the heterogeneous experiments.

Performance of 3DPW Fine-tuning

One potential use case of the PCHMR framework is performing privacy-constrained on-device fine-tuning to align the model with real-world data. We demonstrate this by pre-training our model on a series of datasets [36, 46, 60, 3, 73] and using PCHMR to fine-tune on 3DPW [101] with 100 clients and 10 clients randomly sampled per round. As shown in Table 13, both fine-tuned models reduce MPJPE by about 32 mm compared to the model without fine-tuning. Figure 13 also gives qualitative evidence for the on-device fine-tuning, where the reconstructed mesh accuracy visibly improves. Privacy-constrained fine-tuning stays within 1.3 mm of centralized fine-tuning on every metric while keeping raw data on the device.

Setting	Model	MPJPE	PA-MPJPE	MPVPE	Raw Data Local?
No Fine-tuning	FM-S	116.35	70.20	132.96	✗
Centralized Fine-tuning	FM-S	84.92	54.69	97.6	✗
PCHMR Fine-tuning	FM-S	84.66	54.78	97.42	✓
No Fine-tuning	TORE-S	119.64	68.35	138.93	✗
Centralized Fine-tuning	TORE-S	87.97	55.35	101.88	✗
PCHMR Fine-tuning	TORE-S	87.55	54.08	101.86	✓

On-Device Performance

We further assess the feasibility of the pipeline by measuring the performance of our PCHMR on a real edge device. We selected one of the representative embedded devices, Nvidia AGX Orin [82], to run 5 rounds as one client in the 100- and 1000-client settings, following the experiment setting as outlined in the previous sections. This single-device measurement indicates the runtime scale of edge deployment; a full multi-client deployment may behave differently. With more capable hardware being introduced in applications such as VR and AR, we expect the runtime profile to improve on newer hardware in the coming years.

The runtime and throughput results are shown in Table 14. In the 1000-partition configuration, the edge device would need to train for 85 hours under the 50W power configuration. Nonetheless, the communication cost compared with both figures is relatively insignificant (42 seconds per round), assuming a 37.4 Mb/s broadband upload speed, which follows the median uplink speed in the US [94] between June and August 2024. PCHMR’s runtime is therefore within the range of practical cross-device FL systems, whose training can take hundreds of hours [35, 9].

Model	Device	100 Partitions		1000 Partitions
		Throughput	Time	Throughput	Time
FM-S	Jetson AGX - 15W	2.46	1629.33	2.42	165.33
FM-S	Jetson AGX - 50W	8.01	499.67	7.81	51.23
FM-S	RTX A6000 Ada	56.59	70.7	48.0	8.33
TORE-S	Jetson AGX - 15W	3.43	1166.06	3.35	119.29
TORE-S	Jetson AGX - 50W	8.65	461.5	8.42	47.49
TORE-S	RTX A6000 Ada	50.87	78.67	44.40	9.02