betaFeedback

WorksComputer Science

Modeling Uncertainty Propagation in 3D Morphable Models

Abstract

We study the problem of mapping parameter-space variance of a 3D Morphable Model (3DMM) to per-vertex variance. Traditionally, per-vertex variance has been used to visually interpret the learnt parameter uncertainty. However, mapping parameter-space variance to per-vertex using sampling or error propagation is computationally expensive. We address this limitation with Closed-form Local Uncertainty Estimation (CLUE), which transforms the parameter-space distribution of 3DMMs to per-vertex through a closed-form approximation. CLUE emerges from applying the law of total variance to the 3DMM formulation and is modular, computationally inexpensive, and accurate. Subsequently, we study integrating CLUE for in-the-wild human mesh recovery training and inference. We note that CLUE uses the geometry of the mesh as an inductive bias when predicting the variance, leading to improved accuracy. Through extensive qualitative and quantitative experiments, we show that when used in training, CLUE (1) outpeforms probabilistic and deterministic baselines on metrics evaluating joints, mesh and uncertainty, and (2) has the highest uncertainty-to-error spearman correlation, with applications including data mining and interpretable uncertainty. Finally, we share SAM 3D Body (CLUE) which augments the regressor with high resolution uncertainty maps. We make our code available at https://github.com/vita-epfl/CLUE

Introduction

3D Morphable Models (3DMMs) Loper et al. [34]; Pavlakos et al. [39]; Li et al. [30] have been critical to the success of digital humans in real world applications. A popular application of 3DMMs is in-the-wild Human Mesh Recovery (HMR) [21, 6, 56], which reconstructs humans by estimating their 3DMM parameters from in-the-wild images. However, in-the-wild images often include factors such as occlusions, poor image quality and challenging poses. To address these uncertainties, methods have been developed to predict a distribution over the 3DMM parameters using the the Gaussian Negative Log-Likelihood (GNLL) and its variants Sengupta et al. (2021c,b); Dwivedi et al. (2024). The parameter variance is then mapped to the mesh to obtain interpretable, spatially localized per-vertex uncertainty. However, mapping the parameter distribution onto the mesh is non-trivial.

Popular methods in human mesh regression Sengupta et al. [46] compute per-vertex variances through repeated sampling. A large (≈100\approx100) number of samples are drawn from the parameter distribution which are then used to obtain a set of meshes. The per-vertex variance is computed empirically from this set of meshes. This approach is, however, computationally intensive and sensitive to the number of samples. In statistics, error propagation for Guides in Metrology (2008) is a powerful tool to map uncertainty across a smooth function. This typically involves computing the jacobian of the output w.r.t the input and projecting the input variance through it. While error propagation provides a closed-form approximation of the per-vertex uncertainty, the technique is computationally expensive. Moreover, operators such as the Jacobian may lack compatibility with popular on-device machine learning frameworks such as XNNPACK Google [17].

CLUE transformation, runtime comparison, and in-the-wild uncertainty maps

Figure 1. a) CLUE: Closed-form Local Uncertainty Estimation is a novel transformation that maps parameter-space distribution to local, per-vertex distribution for 3D Morphable Models. b) Accurate and Real-time: CLUE closely approximates sampling-based methods, while being ∼40×\sim40\times faster. c) In-the-wild: We study in-the-wild training and inference, and share SAM 3D Body (CLUE), augmenting a frozen SAM 3D Body model with high-resolution uncertainty maps.

To address this limitation, we present Closed-form Local Uncertainty Estimation (CLUE), a novel formulation to map the parameter-space variance to per-vertex. CLUE uses the law of total variance and error propagation to analyse the 3DMM formulation, disentangling variance from pose, shape and expression. As a result, CLUE is interpretable not only by formulation but also visually. By making pragmatic assumptions, we drastically increase computational efficiency while retaining accuracy on par with linear error propagation. The reduced compute footprint coupled with accuracy of the transformation allow us to integrate CLUE in training probabilistic in-the-wild human mesh regressors. Recent advances in optimizing under uncertainty [49] highlight the use of inductive bias when modeling the predicted variance. In a similar vein, constrains the predicted parameter variance to explain the geometry of the 3DMM, resulting in improved accuracy of the HMR model. CLUE is also agnostic to the architecture of the HMR model, allowing for a seamless integration in training and inference.

We study three popular 3DMMs: (1) FLAME [30], (2) SMPL-X [39], and (3) MHR [14], which are also representative of SMPL Loper et al. [34] and STAR Osman et al. [37]. Next, we perform two sets of experiments: (a) statistical experiments to evaluate CLUE as a transformation from parameter to per-vertex distribution, and (b) real-world experiments to qualitatively and quantitatively evaluate the use of CLUE in training HMR models. Our experiments confirm that CLUE is accurate and computationally inexpensive, as well as out-performs architecture agnostic probabilistic baselines over the joints, mesh, negative log-likelihood and correlation metrics. Furthermore, we discuss applying CLUE for data mining and interpretable uncertainty for downstream decision making. Finally, we share SAM 3D Body (CLUE), which augments a frozen SAM 3D Body model with probabilistic vertices for uncertainty based reasoning. We share example visualizations in Figure 10, Figure 11. We make our code publicly available at https://github.com/vita-epfl/CLUE

Augmenting SAM 3D Body with probabilistic reasoning using CLUE (Part 1)

Figure 10. Augmenting SAM 3D Body with probabilistic reasoning using CLUE (Part 1).

Augmenting SAM 3D Body with probabilistic reasoning using CLUE (Part 2)

Figure 11. Augmenting SAM 3D Body with probabilistic reasoning using CLUE (Part 2).

Related Work

3D Morphable Model. 3DMMs are data-driven statistical models that represent different variations in human identity, expression and pose. They are typically constructed by applying Principal Component Analysis (PCA) to a registered collection of 3D meshes. These models represent geometry in rest-pose through a linear combination of shape, expression and pose-corrective bases. The rest-pose geometry can further be articulated through Linear Blend Skinning (LBS).

vi′=∑j∈Jwi,jGj(θ,J)(vˉi+BS,i(β)+BE,i(ψ)+KP,i(θ))(1)v'_i = \sum_{j \in\mathcal{J}} w_{i,j} G_j(\theta, J) \left(\bar{v}_i + B_{S,i}(\beta) + B_{E,i}(\psi) + K_{P,i}(\theta)\right) \tag*{(1)}

Following Eq. (1), a 3DMM modifies a mean template vˉi\bar{v}_i through a series of additive offsets. In this formulation, BSB_S, BEB_E, and KPK_P are the linear bases representing shape, expression, and pose correctives, respectively. To articulate the mesh, GjG_j represents the rigid transformation of each joint JJ, while wi,jw_{i,j} denote the skinning weights that determine how these transformations influence each vertex. We kindly refer to FLAME Li et al. [30], SMPL-X Pavlakos et al. [39], Basel Face Model (IEE, 2009) for a more detailed description of how 3DMMs are built.

Deterministic mesh regression. There have been significant works which leverage 3DMMs for HMR such as Kanazawa et al. [21], SPIN Kolotouros et al., [25], PARE Kocabas et al., [24], Mesh Graphormer Lin et al., [32], CLIFF Li et al., [31], SAM 3D Body Yang et al., [56] and Fast SAM 3D Body Yang et al., [55]. For faces, the classical 3DMM Blanz & Vetter [4] underpins a large family of analysis-by-synthesis fitters, and deep regressors Deng et al. [8] extend this to in-the-wild imagery. Yet, deterministic methods produce a point estimate and do not capture ambiguities in their prediction.

Probabilistic approaches. Under strong occlusion, ProHMR Kolotouros et al., [26] explicitly encodes the posterior as a multimodal distribution captured using normalizing flows. The use of normalizing flows is further refined in HuManiFlow Sengupta et al., [47] and Biggs et al. [3]. Diffusion and score-based generative models Cho & Kim, [7]; Stathopoulos et al. [52] explore its use in improved HMR. Similarly, for faces under occlusion, Diverse3DFace Dey & Boddeti, [9] and OFER Selvaraju et al., [43] use generative modeling over the FLAME parameters. While multimodal approaches effectively capture diverse modes through sampling, they do not explicitly model the confidence of individual modes. Unimodal methods focus on modeling ambiguity due to occlusion, lighting, noise and partial visibility. One line of work models pose parameters which live on SO(3)SO(3), requiring that pose distributions must respect the rotation manifold Sengupta et al., [45]; Dünkel et al. [10]; Shetty et al. [48]. Methods such as Egger et al. [12, 13]; Li et al. [28] focus on inference under occlusion and do not explicitly model the uncertainty. Along similar lines as this work, Sengupta et al. [44] uses error propagation, but the solution is applicable only in the rest pose with no rotations. When predicting Gaussians over both the shape and pose parameters, Sengupta et al. [46] defaults to using Monte Carlo (MC) sampling. To eliminate Monte Carlo sampling, POCO Dwivedi et al., [11] regresses uncertainty in a single forward pass. However, per-part uncertainty is derived heuristically and does not address the mapping of parameter distribution to the target distribution.

Uncertainty Estimation. Several approaches exist Nix & Weigend, [36]; Gal & Ghahramani [16]; Lakshminarayanan et al. [27]; Blundell et al. [5] to quantify uncertainty in the prediction. Among them, Kendall & Gal [22]; Nix & Weigend [36] jointly optimize the mean and variance of the prediction using GNLL. In our context, this translates to learning the 3DMM parameters and its variance. Transforming the parameter distribution to the mesh involves error propagation (for Guides in Metrology, 2008; Schaefer et al., 2023), with the linear variant being Σy=JΣxJT\Sigma_y = J\Sigma_xJ^T, where y=f(x)y = f(x). As we show later, computing the full Jacobian is computationally expensive. Sengupta et al. [44] uses error propagation but assumes that there is no rotation w.r.t the canonical pose. TIC-TAC Shukla et al., [49] extends error propagation to general nonlinear networks, but remains computationally expensive and abstracts away the interpretability in the formulation. This leaves a gap in recent works on a closed-form transformation which is not only agnostic to any 3DMM formulation but is also accurate, fast and interpretable. With CLUE, we propose an efficient solution for error propagation tailored for 3DMMs.

Closed-form Local Uncertainty Estimation

Let XX be the random variable denoting a vertex after applying additive offsets, and YY be the random variable denoting the rigid transformation of the vertex. We derive CLUE by asking what is Var⁡(Y)\operatorname{Var}(Y)? Our key insight is the law of total variance, which ties the uncertainty in YY as a function of an explanatory variable XX. We provide brief insights here and detail in the appendix (Sec. A).

Assumptions. We inherit our assumptions from the standard 3DMM formulation and related literature (Loper et al., 2015; Sengupta et al., 2021c;b; Schaefer et al., 2023; Dwivedi et al., 2024): the pose θ\theta, identity β\beta, and expression ψ\psi parameters are independent. Next, we drop the pose correctives KP,i(θ)\mathcal{K}_{P,i}(\theta) to allow for a simpler formulation, since the magnitude of change induced by the pose is much larger than those from pose correctives (15 v.s. 0.5cm) ([37]). Dropping pose correctives allows for a simpler formulation; yet for completeness we discuss incorporating pose correctives in the appendix (Sec: A.6).

Derivation

Additive Offset Variance. Let XiX_i be the random variable denoting vertex ii after the additive offsets, Xi=vˉi+β⃗BS,i+ψ⃗BE,iX_i = \bar{v}_i + \vec{\beta}B_{S,i} + \vec{\psi}B_{E,i}, where the shape and expression parameters have variances σβ2\sigma_{\beta}^{2} and σψ2\sigma_{\psi}^{2}. Because XiX_i is linear in these parameters, we have,

ΣXi=Var⁡[Xi]=BS,iTdiag⁡(σβ2)BS,i+BE,iTdiag⁡(σψ2)BE,i,(2)\Sigma_{X_i} = \operatorname{Var}[X_i] = B_{S,i}^{T}\operatorname{diag}(\sigma_{\beta}^{2})B_{S,i} + B_{E,i}^{T}\operatorname{diag}(\sigma_{\psi}^{2})B_{E,i}, \tag*{(2)}

We denote the entries of ΣXi\Sigma_{X_i} by ΣXi(kl)\Sigma_{X_i}^{(kl)}, k,l∈{1,2,3}k,l \in\{1,2,3\}.

Law of Total Variance. The skinned vertex is denoted as vi′=∑j∈Jwi,jGj(θ,J)Xiv_i^{\prime} = \sum_{j \in\mathcal{J}} w_{i,j}G_j(\theta,\mathcal{J})X_i, where J\mathcal{J} is the set of joints and wi,jw_{i,j} are the skinning weights. We have two sources of randomness, (1) the rest-pose vertex XiX_i, and (2) the pose θ\theta. The law of total variance splits Var⁡(vi′)\operatorname{Var}(v_i^{\prime}) into the variance due to the pose, shape and expression:

Var⁡(vi′)=EX[Var⁡θ(vi′∣Xi)]+Var⁡X(Eθ[vi′∣Xi]).(3)\operatorname{Var}(v_i^{\prime}) = \mathbb{E}_{X}\left[\operatorname{Var}_{\theta}(v_i^{\prime} \mid X_i)\right] + \operatorname{Var}_{X}\left(\mathbb{E}_{\theta}[v_i^{\prime} \mid X_i]\right). \tag*{(3)}

Intuitively, the rotational variance quantifies how much a vertex moves when the pose varies for an expected body shape. The shape and expression variance quantifies the movement due to the shape and expression given the expected pose.

Rotational Variance. Let us consider the rest-pose vertices deterministic and let the pose vary. Because GjG_j is composed along the kinematic chain, a small rotation δωa\delta\omega_a of joint aa swings all descendent joints D(a)\mathcal{D}(a) rigidly about the mean posed location tˉa\bar{t}_a of that joint. A point at xx thus moves by δωa×(x−tˉa)\delta\omega_a \times(x-\bar{t}_a), a small angle times a lever arm. For the skinned vertex, the lever arm is averaged over the joints that carry it and lie below aa, and the displacements of all joints add:

vi′≈∑j∈Jwi,jGˉjXi+∑a∈Jδωa×ℓi,a,ℓi,a=∑j∈D(a)wi,j(GˉjXi−tˉa).v_i^{\prime} \approx\sum_{j \in\mathcal{J}} w_{i,j}\bar{G}_jX_i + \sum_{a \in\mathcal{J}} \delta\omega_a \times\ell_{i,a}, \qquad\ell_{i,a} = \sum_{j \in\mathcal{D}(a)} w_{i,j}\left(\bar{G}_jX_i-\bar{t}_a\right).

A joint contributes only if a descendent of the joint skins vertex ii. For instance uncertainty travels down the chain, from the shoulder to the hand, but not across it. The lever arm grows with the distance from the joint, so the same angular uncertainty at the shoulder produces a larger positional uncertainty at the fingertip than at the elbow. For each joint, we have a Gaussian on its axis-angle coordinates ωa\omega_a in the axes of the parent joint, with independent per-axis variances σθa,x2,σθa,y2,σθa,z2\sigma_{\theta_a,x}^{2},\sigma_{\theta_a,y}^{2},\sigma_{\theta_a,z}^{2}. An increment ϵa\epsilon_a of these coordinates is the small rotation δa=Jl(ωa)ϵa\delta_a = J_l(\omega_a)\epsilon_a composed with the mean rotation, where JlJ_l is the left Jacobian of SO(3)SO(3). Thus, the covariance of the perturbation is therefore Σθa=Rˉp(a)Jl(ωˉa)diag⁡(σθa,x2,σθa,y2,σθa,z2)Jl(ωˉa)TRˉp(a)T\Sigma_{\theta_a} = \bar{R}_{p(a)}J_l(\bar{\omega}_a)\operatorname{diag}(\sigma_{\theta_a,x}^{2},\sigma_{\theta_a,y}^{2},\sigma_{\theta_a,z}^{2})J_l(\bar{\omega}_a)^{T}\bar{R}_{p(a)}^{T}, where Rˉp(a)\bar{R}_{p(a)} is the mean global rotation of the parent. The variance of the displacement comes from the cross product. Intuitively, a rotation about yy moves the point in xx by its zz lever arm, and likewise for the other axes. Evaluating the lever arm at the mean vertex, with components (ℓi,a(x),ℓi,a(y),ℓi,a(z)\ell_{i,a}^{(x)}, \ell_{i,a}^{(y)}, \ell_{i,a}^{(z)}), and writing Σθa(pq)\Sigma_{\theta_a}^{(pq)} for the entries of Σθa\Sigma_{\theta_a}, the rotational variance is

EX[Var⁡θ(vi′∣Xi)]≈∑a∈J[Σθa(yy)(ℓi,a(z))2+Σθa(zz)(ℓi,a(y))2−2Σθa(yz)ℓi,a(y)ℓi,a(z)Σθa(xx)(ℓi,a(z))2+Σθa(zz)(ℓi,a(x))2−2Σθa(xz)ℓi,a(x)ℓi,a(z)Σθa(xx)(ℓi,a(y))2+Σθa(yy)(ℓi,a(x))2−2Σθa(xy)ℓi,a(x)ℓi,a(y)].(4)\mathbb{E}_{X}\left[\operatorname{Var}_{\theta}(v_i' \mid X_i)\right] \approx\sum_{a \in\mathcal{J}} \begin{bmatrix} \Sigma_{\theta_a}^{(yy)}\left(\ell_{i,a}^{(z)}\right)^2 + \Sigma_{\theta_a}^{(zz)}\left(\ell_{i,a}^{(y)}\right)^2 - 2\Sigma_{\theta_a}^{(yz)}\ell_{i,a}^{(y)}\ell_{i,a}^{(z)} \\ \Sigma_{\theta_a}^{(xx)}\left(\ell_{i,a}^{(z)}\right)^2 + \Sigma_{\theta_a}^{(zz)}\left(\ell_{i,a}^{(x)}\right)^2 - 2\Sigma_{\theta_a}^{(xz)}\ell_{i,a}^{(x)}\ell_{i,a}^{(z)} \\ \Sigma_{\theta_a}^{(xx)}\left(\ell_{i,a}^{(y)}\right)^2 + \Sigma_{\theta_a}^{(yy)}\left(\ell_{i,a}^{(x)}\right)^2 - 2\Sigma_{\theta_a}^{(xy)}\ell_{i,a}^{(x)}\ell_{i,a}^{(y)} \end{bmatrix}. \tag*{(4)}

Shape and Expression Variance. Let us now fix the pose at its mean and let the rest-pose offsets vary. The expected posed vertex is Eθ[vi′∣Xi]=(∑j∈Jwi,jG‾j)Xi\mathbb{E}_{\theta}[v_i' \mid X_i] = \left(\sum_{j \in\mathcal{J}} w_{i,j}\overline{G}_j\right)X_i. The offset covariance of Eq. (2) is simply rotated into the posed body by the vertex’s blended rotation Rieff=∑j∈Jwi,jR‾j\mathbf{R}_{i}^{\mathrm{eff}} = \sum_{j \in\mathcal{J}} w_{i,j}\overline{\mathbf{R}}_j. Using our identity Var⁡[AZ]=AΣZAT\operatorname{Var}[AZ] = A\Sigma_Z A^{T} and keeping the diagonal of RieffΣXi(Rieff)T\mathbf{R}_{i}^{\mathrm{eff}}\Sigma_{X_i}(\mathbf{R}_{i}^{\mathrm{eff}})^{T}, we have

Var⁡X(Eθ[vi′∣Xi])=[∑k,l=13Ri,1,keffRi,1,leffΣXi(kl)∑k,l=13Ri,2,keffRi,2,leffΣXi(kl)∑k,l=13Ri,3,keffRi,3,leffΣXi(kl)].(5)\operatorname{Var}_{X}\left(\mathbb{E}_{\theta}[v_i' \mid X_i]\right) = \begin{bmatrix} \sum_{k,l=1}^{3} R_{i,1,k}^{\mathrm{eff}}R_{i,1,l}^{\mathrm{eff}}\Sigma_{X_i}^{(kl)} \\ \sum_{k,l=1}^{3} R_{i,2,k}^{\mathrm{eff}}R_{i,2,l}^{\mathrm{eff}}\Sigma_{X_i}^{(kl)} \\ \sum_{k,l=1}^{3} R_{i,3,k}^{\mathrm{eff}}R_{i,3,l}^{\mathrm{eff}}\Sigma_{X_i}^{(kl)} \end{bmatrix}. \tag*{(5)}

Substituting Eq. (4) and Eq. (5) into Eq. (3) gives the per-vertex variance of the 3DMM formulation.

In contrast to first order error propagation, CLUE leverages the 3DMM formulation without needing to compute the full Jacobian. The per-vertex variance cleanly disentangles the different sources of uncertainty and combines them through a simple summation. As a result, estimating this quantity is inexpensive in terms of compute time and memory while remaining accurate. We provide pseudo-code in the appendix (Sec. C) as well as the code implementation for FLAME, SMPL-X and MHR in the repository.

Training: CLUE with SAM 3D Body

Methods such as Sengupta et al. [46]; Dwivedi et al. [11] which rely on the negative log-likelihood to train the variance may suffer from suboptimal convergence. This is because the variance head lacks direct supervision and may take on arbitrary values that minimize the objective sub-optimally [53, 19, 49, 50]. In Shukla et al. [49], a solution was proposed which used inductive bias when modeling the predicted variance. CLUE presents one such inductive bias by incorporating the geometry of the 3DMM when learning the parameter variance.

Variance Head. CLUE is architecture agnostic, hence we make no changes to the architecture of SAM 3D Body. We use minimal additional parameters to create a variance head. The variance head takes as input the pose token from SAM 3D Body which is also consumed by the MHR head. In addition, the variance head also takes in the mean parameters output by the MHR head. Both the pose token and the mean parameters are concatenated before being fed to a two layer feedforward network with one hidden layer of width 1024 and ReLU activation. The output of this network is the variance for the shape and pose parameters, constrained by sigmoid activation but scaled by learnable scalars. These predicted variances are consumed by CLUE. Note that all our probabilistic baselines share the same variance head architecture.

Losses. By defining a transformation between the parameter, vertex and joints space, CLUE defines an equivalency between them. This implies that the parameter variance can be mapped to vertex and joint space without any additional learnable parameters. This is particularly useful because while the joints are traditionally inferred from the vertices, CLUE allows inferring the joints as well as their variances from the vertices trivially. Thus, this enables CLUE to use the joint-space negative log-likelihood (joints-NLL), replacing the traditional deterministic joint-space loss. Therefore, the geometry of the 3DMM guides the optimization of the parameter variance, which has to explain both: the parameter-NLL as well as joints-NLL. To compute joints-NLL, we assume that neighboring vertices are correlated. Hence the joint standard deviation is the weighted average of the vertex standard deviation, σjoint=∑vwvσv\sigma_{\mathrm{joint}} = \sum_v w_v \sigma_v. In contrast, assuming independence meant that the joint variance was underestimated by 5 to 13×.

Limitations

CLUE like jacobian based error propagation is a linear approximation, and overestimates the per-vertex variance when used as a statistical transformation. This is particularly visible for extreme body poses as shown in Figure 2. Furthermore, CLUE leverages the structure of the 3DMM and does not extend to neural shape representation approaches. Finally, CLUE makes several approximations to drastically reduce the number of computations needed without sacrificing accuracy, such as first-order analysis and dropping covariances between the vertices.

FLAME error maps comparing ground truth and different variance estimation baselines

Figure 2. FLAME Error Map: We visualize the per-vertex error between the ground truth variance (computed with N=10,000N = 10{,}000 samples) and the estimated variance from different baselines. To do so, we synthesize different faces by setting randomly chosen parameters and their associated variance to non-zero. We observe that sampling with N=1000N=1000, Jacobian based error propagation and CLUE have low errors. Unlike sampling, Jacobian and CLUE may overestimate the variance due to first order assumptions. Additional visualizations in Figure 7.

Complete error maps for FLAME, SMPLX, MHR

Figure 7. Complete error maps for FLAME, SMPLX, MHR. Both, Jacobian based linear error propagation and CLUE overestimate the variance due to first order assumptions. However when used in training, the learnt parameter variances compensate for them.

Experiments

We perform two sets of experiments, designed to evaluate the accuracy of CLUE as a transformation from parameter-space to vertex-space, as well as the role of CLUE to improve optimization when training in-the-wild human mesh recovery models.

Statistical Evaluation

The first set of experiments evaluates CLUE as a transformation from parameter-space to the space of meshes, studying accuracy and compute cost. These experiments rely on synthesizing parameter variance, and comparing how CLUE propagates these variances in comparison to sampling with N={5,10,100,1000}N = \{5, 10, 100, 1000\} and error propagation. We perform these studies across FLAME, SMPL-X and Momentum Human Rig (MHR).

MethodTime (ms)Mem. (MB)GFLOPs
Sampling N=100N=100138.714.21.43
Sampling N=1000N=10001379.4140.914.26
Jacobian226.4410.46.01
CLUE (ours)3.44.30.039

Table 1. FLAME compute. Median of 10 trials after 3 warm-ups. Additional results in Table 5.

Accuracy (Figure 2). We compute the ground truth variance by simulating parameter variance, drawing N=10,000N = 10{,}000 samples, and computing per-vertex variance empirically from the resulting meshes. For each baseline, we compute the absolute difference between its variance and that of the ground truth. We visualize these as error maps by randomly setting different parameter groups and their associated variance. For sampling based methods, as the value of NN increases, there is a drop in the error across all parameter groups. We also observe that N=1000N = 1000 and jacobian error propagation closely approximates the ground truth. While CLUE similarly has low errors across different parameter groups, we highlight the intricate pattern on the forehead, which also comes up in the error map for CLUE. We hypothesize that since CLUE does not model pose correctives, it does not capture intricate changes in the mesh as the jaw joint moves.

Compute. We evaluate on runtime, peak incremental memory, and GFLOPs. We implement the Jacobian in forward mode in PyTorch [38]. We use the CPU for computing the transformations to reflect on-device computing constraints, where the GPU would be prioritized for batched inferring from the incoming data stream. CLUE has the fastest runtime, with error propagation via the Jacobian consuming the largest memory. CLUE requires ≈4\approx4 MB of additional memory because the tensors needed to compute the variance are already on-device. For the larger SMPL-X and MHR rigs (Table 5), our implementation exploits the sparsity of the kinematic tree, computing lever arms only for the joints that skin a vertex and their ancestors. The low computational costs as well as accuracy of the transformation makes CLUE suitable for on-device uncertainty estimation.

MethodFLAMESMPL-XMHR
msMBGFLOPsmsMBGFLOPsmsMBGFLOPs
Sampling N=100N=100141.614.21.43368.529.55.42681.744.51.71
Sampling N=1000N=10001404.7140.914.333733.6293.654.206802.4442.917.09
Jacobian228.6410.46.08177.0383.610.233272.21984.46.40
CLUE (ours)3.54.30.0394.09.50.01510.916.90.055

Table 5. Compute cost across body models. Time (ms), memory (MB) and GFLOPs of one variance evaluation. Median of 10 trials after 3 warm-ups on one CPU (Intel i7-8700).

Probabilistic In-the-wild Human Mesh Recovery

The second set of experiments studies CLUE when used to supervise the training of unimodal in-the-wild human mesh recovery models such as SAM 3D Body. We compare our method with deterministic training, and architecture-agnostic probabilistic losses for uncertainty such as 3DMM Parameter-NLL, Matrix Fisher, and Normalizing Flows (Sengupta et al., 2021c;b; Dwivedi et al., 2024; Kolotouros et al., 2021). We randomly initialize the SAM 3D Body Yang et al. [56] architecture with frozen DINOv3 Simeoni et al. [51] backbone as our HMR model. We describe the losses, datasets and training process in the appendix (Sec. B).

Quantitative Study

3DPW and EMDB. (Table 2) We observe that all probabilistic losses outperform the deterministic loss. By introducing inductive bias when modeling the mesh, CLUE results in improved optimization and is the strongest baseline over all metrics. We also report the evaluation of SAM 3D Body on our pipeline which we recall is trained on a significantly larger dataset. Our numbers differ from those published due to possible differences with the SAM 3D Body evaluation pipeline which is not public. While CLUE is competitive with SAM 3D Body on 3DPW, it falls short on EMDB. We hypothesize that probabilistic training can reason better when dealing with ambiguities, but cannot model poses that are out-of-distribution accurately. This suggests a strategy of probabilistic modeling coupled with a diverse dataset.

Method#Training Images3DPW MPJPE3DPW PA-MPJPE3DPW PVE3DPW PA-PVEEMDB MPJPEEMDB PA-MPJPEEMDB PVEEMDB PA-PVE
SAM 3D Body (released)3.8M (w/ multi-view)56.035.167.147.263.238.274.747.1
Deterministic Training1.2M (single-view)62.737.272.949.771.943.485.552.6
Parameter NLL”57.935.568.948.366.441.279.550.9
Matrix-Fisher”58.035.768.848.168.842.581.751.8
Normalizing Flow”58.335.668.948.068.041.480.750.8
CLUE”56.134.967.047.265.440.877.950.7

Table 2. 3DPW and EMDB (MoCap SMPL). All metrics in mm (↓\downarrow). The first row measures SAM 3D Body on our evaluation pipeline. The other rows compare deterministic and probabilistic losses under identical training and evaluation protocol.

Pseudo Ground Truth datasets. Table 3 scores each baseline against the released SAM 3D Body as ground truth on five image datasets without motion capture, in MHR space. By having the prediction and ground truth in MHR space, we can compute uncertainty related metrics. CLUE is the only probabilistic objective that improves every group of metrics. Figure 4 visualizes the spearman correlation between the vertex error after rigid alignment of the ground truth and predicted mesh. We observe that CLUE has the highest correlation across all groups.

Correlations across vertex groups

Figure 4. Correlations across vertex groups. We compute the Spearman correlation between the predicted per-vertex variance and the rigidly aligned vertex error for each of the 19 MHR vertex groups. We observe that CLUE has a higher spearman correlation for each of the vertex groups, enabling applications in data mining and uncertainty based decision making.

MethodMPJPEPA-MPJPEPVEPA-PVEvNLLρ\rhoAUSE
Deterministic31.619.735.424.2–––
Param-NLL32.018.535.022.32.350.680.078
Matrix-Fisher35.920.538.924.42.740.560.129
Normalizing Flow32.018.135.322.23.080.690.077
CLUE27.517.730.421.72.130.750.067

Table 3. Pseudo-ground-truth evaluation, averaged over five in-the-wild datasets (H36M, MPI-INF-3DHP, PoseTrack, CrowdPose, OCHuman): joint and surface error (mm, ↓\downarrow), vertex NLL (↓\downarrow), Pearson correlation between predicted uncertainty and error (↑\uparrow), and the area under the sparsification error curve (AUSE, ↓\downarrow; per person, ranking by the mean predicted vertex sigma). Per-dataset results are in the appendix (Table 4).

H36MMPI-3DPoseTrackCrowdPoseOCHumanAverage
Deterministic19.213.525.417.624.916.733.220.055.230.531.619.7
Param-NLL22.613.227.917.025.415.531.317.853.128.832.018.5
Matrix-Fisher26.215.033.719.932.518.835.319.851.628.935.920.5
Normalizing Flow22.313.330.117.328.116.131.717.547.826.432.018.1
CLUE18.212.724.816.224.615.729.117.740.526.127.517.7
H36MMPI-3DPoseTrackCrowdPoseOCHumanAverage
Deterministic22.617.228.721.528.320.635.924.261.237.335.424.2
Param-NLL25.516.330.220.128.118.933.521.557.635.035.022.3
Matrix-Fisher28.418.335.923.034.621.637.823.857.735.538.924.4
Normalizing Flow25.116.932.720.730.719.434.121.353.732.735.322.2
CLUE20.615.627.019.426.818.831.121.546.433.230.421.7
H36MMPI-3DPoseTrackCrowdPoseOCHumanAverage
Deterministic––––––––––––
Param-NLL2.100.682.300.712.220.692.320.692.830.662.350.68
Matrix-Fisher2.580.362.710.512.700.612.730.622.960.662.740.56
Normalizing Flow2.920.673.110.732.900.682.990.673.480.683.080.69
CLUE1.820.712.000.722.070.742.150.772.610.792.130.75
H36MMPI-3DPoseTrackCrowdPoseOCHumanAverage
Deterministic––––––
Param-NLL0.0810.0720.0870.0720.0800.078
Matrix-Fisher0.1590.1420.1170.1170.1110.129
Normalizing Flow0.0800.0680.0840.0740.0800.077
CLUE0.0650.0670.0790.0660.0590.067

Table 4. Pseudo-ground-truth evaluation on five in-the-wild datasets: (a) joint accuracy, (b) surface accuracy (mm, ↓\downarrow), (c) uncertainty quality — per-vertex NLL (↓\downarrow) and the Pearson correlation between predicted σ\sigma and vertex error (↑\uparrow), and (d) the area under the sparsification error curve (AUSE, ↓\downarrow; per person, ranking by the mean predicted vertex σ\sigma, normalised by each dataset’s full-set error). (a) Joints: MPJPE / PA-MPJPE. (b) Surface: PVE / PA-PVE. (c) Uncertainty: vertex-NLL / σ\sigma–error correlation. (d) Uncertainty: AUSE.

Qualitative Study

Error Maps. Figure 3 colours each prediction by its per-vertex error against the reference mesh, on one shared scale. We observe two patterns. First, the baselines struggle with limbs that are occluded or have complex poses. CLUE is resistant to occlusion due to the inductive bias of the mesh in modeling the variance and allows for probabilistic reasoning under occlusion. Second, in complex body poses such as floor exercises and splits, the parameter-space baselines spread error over the torso and head as well, whereas CLUE confines the error to the extremities.

Per-vertex error maps

Figure 3. Per-vertex error maps. Each row shows the input, the reference mesh (yellow), and the predictions of different baselines, coloured by the per-vertex error. The largest baseline errors sit on limbs that are occluded or having complex poses. CLUE uses the mesh as an inductive bias when predicting the variance, allowing it to have lower errors when dealing with ambiguity. Additional visualizations are shown in Figure 8.

Error maps for in-the-wild human mesh recovery

Figure 8. Error Maps for in-the-wild Human Mesh Recovery. We show that CLUE has higher accuracy when dealing with occlusions or complex poses.

Uncertainty based data mining. Since CLUE predicts fine-grained uncertainty maps (evidenced by high spearman correlations), we can answer questions that parameter-space uncertainty cannot: which vertex groups of the body is the model unsure about? For this study, we look for a limb whose uncertainty stands out from the rest of the body. However, raw uncertainty values are not comparable as they differ in scale between methods and vertices. We therefore convert each vertex’s uncertainty into a percentile, i.e. how unusual that value is for that vertex and that method, measured against the SA-1B training set. We also recall that per-vertex variances are computed empirically for all baselines except CLUE. In Figure 5 we observe that CLUE correctly distinguishes vertex groups which are occluded or undergo motion blur, while the rest of the body remains confident. The baselines saturate uncertainty across the whole body. While they can say that an image is hard, they cannot localize where.

Uncertainty based data-mining

Figure 5. Uncertainty based data-mining. Per-vertex variance is computed empirically by sampling N=100N=100 for all other baselines, and using CLUE for itself. Similar to spearman correlation, for each method we normalize a vertex’s uncertainty based on its per-vertex percentile for that method. These images were retrieved algorithmically by querying for images with the highest uncertainty discrepancy between a limb and the body. In contrast to CLUE, the baselines’ uncertainty on the same images is body-wide or diffused and lacks fine-grained resolution. Additional visualizations are shown in Figure 9.

Uncertainty based data-mining, with columns labelled Input, Parameter NLL, Matrix-Fisher, Normalizing Flow, and CLUE, and a percentile colour scale

Figure 9. Uncertainty based data-mining.

Uncertainty based decision making. We present a use-case in egocentric face tracking. This study uses an internal dataset and GNM Ploumpis et al. [40] as the 3DMM. The goal is to regress a 3D mesh that reproduces the lower-face expression of a person from egocentric viewpoints. This setup is typical of VR-headsets such as Apple Vision Pro, Meta Quest Pro, and Galaxy XR, which further impose strict runtime and memory restrictions to enable realtime on-device operation. We employ a similar training protocol as described earlier to train the probabilistic regressor. We observe in Figure 6 that as the degree of occlusion increases, the uncertainty is localized and scaled correctly. Given this uncertainty signal, a system no longer has to track blindly through an occlusion: it can pause face tracking, lower its weight in the fusion, or hand over to another sensor until the face is visible again.

Frames from a headset’s egocentric camera with progressively occluded mouth and tracked 3DMM per-vertex standard deviation

Figure 6. Uncertainty for downstream decision making. Top: frames from a headset’s egocentric camera with the mouth progressively occluded. Bottom: the tracked 3DMM per-vertex standard deviation (mm). The uncertainty concentrates on the occluded region, rising monotonically with the degree of occlusion. With this signal, a system can suspend or down-weight vision-based face tracking during occlusion, or defer to another modality, instead of tracking blindly.

Conclusion

We proposed CLUE, a novel formulation which transforms the parameter-space distribution of a 3DMM onto the mesh in closed-form. Through statistical experiments across multiple 3DMMs, we showed that CLUE is an accurate transformation while being computationally efficient. We also noted that in comparison to a direct application of linear error propagation, CLUE is interpretable in terms of formulation and visually. We studied integrating CLUE for in-the-wild human mesh recovery training and inference. Because CLUE carries the predicted parameter variance through the geometry of the 3DMM, it serves as an inductive bias to train the variance head. Subsequently, we show that CLUE outperforms deterministic training as well as architecture agnostic unimodal uncertainty losses. Next, we analyze the variance learnt by CLUE, showing that it has the highest spearman correlation for all vertex groups, and discuss applications of the learnt uncertainty in data mining and downstream decision making. Finally, we share SAM 3D Body (CLUE) to make per-vertex uncertainty a standard output of in-the-wild mesh recovery.

AI USE STATEMENT

In this work, we used generative AI tools for minor edits in writing. This includes finding ways to convey our thoughts in a compact fashion. GenAI tools were also used for grammatical checks or minor corrections. Bulk of the paper was written by humans, for humans. No AI was used in writing this statement. We additionally used GenAI tools for code implementation and documentation. However, there was always a human-in-the-loop to verify code correctness. Finally, GenAI was used to double-check the math in this paper. However, the authors developed the idea and formulated the derivation. We take responsibility for the final content of this work, including text, claims or artifacts produced with the aid of generative AI.

REPRODUCIBILITY STATEMENT

We provide significant implementation details: the pseudo-code for CLUE, and the variance head when using CLUE for in-the-wild human mesh recovery. We also use the open-weight SAM 3D Body model as well as publicly available datasets. Barring our study on downstream decision making, all other experiments can be reproduced with publicly available data, models, and the implementation details provided in the paper.

REFERENCES

Derivation Details

We provide a detailed, annotated version of the derivation for convenience. We also provide code in the accompanying repository for 3DMM specific modifications, and present our general analysis here.

Assumptions and approximations. CLUE uses the law of total variance and first-order error propagation specialised to the skinning equation. We make several assumptions that allow this approximation to be real-time yet accurate in comparison to large-scale sampling and jacobian based error propagation.

  1. Independence and diagonal covariance. Following literature, β\beta, ψ\psi and θ\theta are mutually independent with diagonal covariances, and the perturbations of different joints are independent with zero mean. The pose uncertainty of joint aa is a Gaussian on its axis-angle coordinates ωa\omega_a and is expressed in the axes of the parent of aa. The predicted per-axis variances σθa2\sigma_{\theta_a}^{2} and its effect on the rotation is a small rotation composed with the mean rotation. We keep only per-vertex variances and not covariances for simplicity.

  1. First order in the pose. We drop terms of second or higher order in the pose perturbations. In particular Eθ[Gj]=G‾j+O(σθ2)\mathbb{E}_{\theta}[G_j] = \overline{G}_j + O(\sigma_{\theta}^{2}), and we drop the O(σθ2σX)O(\sigma_{\theta}^{2}\sigma_X) cross term. This simplicity allows us to have a lower computational cost in relative to full error propagation.

  1. Joint locations at the mean shape. When analyzing the shape and expression variance, the joint locations J\mathbf{J} and hence t‾j\overline{\mathbf{t}}_j are held at the mean.

  1. No pose correctives. We don’t model pose correctives, but show how they can be added if needed in this derivation.

Preliminary: 3DMM Formulation

Rest-pose vertex. Vertex ii in the rest pose is the template vertex with additive linear offsets of shape and expression,

Xi=v‾i+β⃗BS,i+ψ⃗BE,i.X_i = \overline{\mathbf{v}}_i + \vec{\beta} B_{S,i} + \vec{\psi} B_{E,i}.

Here, β∈R∣β∣\beta\in\mathbb{R}^{|\beta|} and BS,i∈R∣β∣×3B_{S,i} \in\mathbb{R}^{|\beta| \times3} and similar for the expression component.

Kinematic chain. Each joint kk has a parent p(k)p(k), a rest-pose location Jk\mathbf{J}_k, an offset jk=Jk−Jp(k)\mathbf{j}_k = \mathbf{J}_k - \mathbf{J}_{p(k)} from its parent, and a local rotation exp⁡(ωk)\exp(\omega_k) given by the pose θ={ωk}k∈J\theta= \{\omega_k\}_{k \in\mathcal{J}}, where J\mathcal{J} is the set of joints. Let A(j)A(j) be the ordered set of joints on the path from the root to jj, including jj. The global transformation of joint jj is the chain product

Gj(θ,J)=∏k∈A(j)[exp⁡(ωk)jk01]=[Rjtj01].G_j(\theta,\mathbf{J}) = \prod_{k \in A(j)} \begin{bmatrix} \exp(\omega_k) & \mathbf{j}_k \\ 0 & 1 \end{bmatrix} = \begin{bmatrix} \mathbf{R}_j & \mathbf{t}_j \\ 0 & 1 \end{bmatrix}.

where Rj\mathbf{R}_j is the global rotation of joint jj and tj\mathbf{t}_j its location in the posed body. The chain can be represented as a recursive relation between a joint and its parent, Rj=Rp(j)exp⁡(ωj)\mathbf{R}_j = \mathbf{R}_{p(j)}\exp(\omega_j) and tj=Rp(j)jj+tp(j)\mathbf{t}_j = \mathbf{R}_{p(j)}\mathbf{j}_j + \mathbf{t}_{p(j)}.

We use two properties. First, ωa\omega_a appears in GjG_j if and only if a∈A(j)a \in A(j); we write D(a)={j:a∈A(j)}D(a) = \{j : a \in A(j)\} for the joints at or below aa. Second, each factor is a translation followed by a rotation,

[exp⁡(ωk)jk01]=T(jk)Rot⁡(exp⁡(ωk)),T(u)=[Iu01],Rot⁡(R)=[R001].\begin{bmatrix} \exp(\omega_k) & \mathbf{j}_k \\ 0 & 1 \end{bmatrix} = T(\mathbf{j}_k)\operatorname{Rot}(\exp(\omega_k)), \qquad T(\mathbf{u}) = \begin{bmatrix} I & \mathbf{u} \\ 0 & 1 \end{bmatrix}, \qquad \operatorname{Rot}(\mathbf{R}) = \begin{bmatrix} \mathbf{R} & 0 \\ 0 & 1 \end{bmatrix}.

Pivot removal. SMPL applies GjG_j after removing the rest-pose transformation, Gj=Gj(θ,J)Gj(θ∗,J)−1G_j = G_j(\theta,\mathbf{J})G_j(\theta^*,\mathbf{J})^{-1}. In the rest pose all rotations are the identity, so Gj(θ∗,J)=T(Jj)G_j(\theta^*,\mathbf{J}) = T(\mathbf{J}_j) and

Gjv=Rj(v−Jj)+tj,in particularGjJj=tj.G_j\mathbf{v} = \mathbf{R}_j(\mathbf{v}-\mathbf{J}_j) + \mathbf{t}_j, \qquad\text{in particular}\qquad G_j\mathbf{J}_j = \mathbf{t}_j.

Skinning. The posed vertex is a weighted average of the vertex transformed by each joint,

vi′=∑j∈Jwi,jGjXi.(6)\mathbf{v}'_i = \sum_{j \in\mathcal{J}} w_{i,j}G_jX_i. \tag*{(6)}

with wi,jw_{i,j} the skinning weights, which are non-zero only for a few joints near vertex ii.

Additive Variance.

We return to the formulation of the rest pose vertex. The shape and expression parameters have variances σβ2\sigma_{\beta}^{2} and σψ2\sigma_{\psi}^{2}. Since XiX_i is linear in these parameters and using Var⁡[AZ]=AΣZAT\operatorname{Var}[AZ] = A\Sigma_Z A^{T}, the covariance of the rest-pose vertex is

ΣXi=Var⁡[Xi]=BS,iTdiag⁡(σβ2)BS,i+BE,iTdiag⁡(σψ2)BE,i.(7)\Sigma_{X_i} = \operatorname{Var}[X_i] = B_{S,i}^{T}\operatorname{diag}(\sigma_{\beta}^{2})B_{S,i} + B_{E,i}^{T}\operatorname{diag}(\sigma_{\psi}^{2})B_{E,i}. \tag*{(7)}

We denote the entries of ΣXi\Sigma_{X_i} by ΣXi(kl)\Sigma_{X_i}^{(kl)}, k,l∈{1,2,3}k,l \in\{1,2,3\}.

Law of Total Variance.

We have two independent sources of randomness in the posed vertex vi′v_i^{\prime}: namely the shape and expression parameters, through XiX_i, and the pose θ\theta, through the transformations GjG_j. The law of total variance separates them,

Var⁡(vi′)=EX[Var⁡θ(vi′∣Xi)]+Var⁡X(Eθ[vi′∣Xi]).(8)\operatorname{Var}(v_i^{\prime}) = \mathbb{E}_{X}\left[\operatorname{Var}_{\theta}(v_i^{\prime}\mid X_i)\right] + \operatorname{Var}_{X}\left(\mathbb{E}_{\theta}[v_i^{\prime}\mid X_i]\right). \tag*{(8)}

The first term asks that for a fixed rest-pose vertex, how much does the vertex move when the pose varies? The second asks whether for the expected pose, how much does the vertex move when the shape and expression vary? We derive the two terms in turn.

Rotation Variance.

The intuition is to fix the rest-pose vertex XiX_i and let only the pose vary.

Step 1: Effect of one joint's perturbation (joint a) on one transformation (j)

Let a∈A(j)a \in A(j). Splitting the chain product of GjG_j at joint aa and using the translate-then-rotate form of the atha^{\text{th}} factor,

Gj=Gp(a)T(ja)Rot⁡(exp⁡(ωa))∏k∈A(j), k below a[exp⁡(ωk)jk01],Ha=[Rp(a)ta01].G_j = G_{p(a)}T(j_a)\operatorname{Rot}(\exp(\omega_a))\prod_{\substack{k\in A(j),\ k\ \text{below }a}}\begin{bmatrix}\exp(\omega_k) & j_k \\ 0 & 1\end{bmatrix}, \qquad H_a = \begin{bmatrix}R_{p(a)} & t_a \\ 0 & 1\end{bmatrix}.

where the block form of HaH_a follows from the recursion ta=Rp(a)ja+tp(a)t_a = R_{p(a)}j_a + t_{p(a)}. HaH_a is the rigid transformation that places the frame of joint aa in the world: rotation Rp(a)R_{p(a)}, origin at the posed joint location tat_a.

Now let us perturb the rotation of joint aa. We have a Gaussian on the axis-angle coordinates of the joint, ωa+εa\omega_a + \varepsilon_a with εa∼N(0,diag⁡(σθa2))\varepsilon_a \sim\mathcal{N}(0,\operatorname{diag}(\sigma_{\theta_a}^{2})), in the axes of the parent of aa. The perturbed rotation is the unperturbed one composed with a small rotation,

exp⁡(ωa+εa)=(I+[δa]×)exp⁡(ωa),δa=Jl(ωa)εa,\exp(\omega_a+\varepsilon_a) = (I+[\delta_a]_{\times})\exp(\omega_a), \qquad\delta_a = J_l(\omega_a)\varepsilon_a,
Jl(ω)=I+1−cos⁡ϑϑ2[ω]×+ϑ−sin⁡ϑϑ3[ω]×2,ϑ=∥ω∥,J_l(\omega) = I + \frac{1-\cos\vartheta}{\vartheta^{2}}[\omega]_{\times} + \frac{\vartheta-\sin\vartheta}{\vartheta^{3}}[\omega]_{\times}^{2}, \qquad\vartheta= \lVert\omega\rVert,

where JlJ_l is the left Jacobian of SO(3)SO(3), which maps an increment of the axis-angle coordinates to the equivalent small rotation. Substituting it in GjG_j we have,

Gjpert=HaRot⁡(I+[δa]×)Ha−1Gj.G_j^{\mathrm{pert}} = H_a\operatorname{Rot}(I+[\delta_a]_{\times})H_a^{-1}G_j.

the perturbed transformation is the unperturbed one followed by an extra factor. To see what this factor does, follow a point xx through its three matrices, from right to left.

(i) Ha−1H_a^{-1} expresses xx in the frame of joint aa. The inverse of a rigid transformation subtracts the translation and undoes the rotation,

y=Ha−1x=Rp(a)T(x−ta).y = H_a^{-1}x = R_{p(a)}^{T}(x-t_a).

(ii) Rot⁡(I+[δa]×)\operatorname{Rot}(I + [\delta_a]_{\times}) applies the small rotation in that frame,

y′=(I+[δa]×)y=y+δa×y.y' = (I + [\delta_a]_{\times})y = y + \delta_a \times y .

(iii) HaH_a brings the result back to world coordinates,

x′=Hay′=Rp(a)y′+ta=Rp(a)y+ta⏟=x+Rp(a)(δa×y).x' = H_a y' = R_{p(a)} y' + t_a = \underbrace{R_{p(a)} y + t_a}_{=x} + R_{p(a)}(\delta_a \times y).

Using the property, R(u×v)=(Ru)×(Rv)\mathbf{R}(\mathbf{u} \times\mathbf{v}) = (\mathbf{R}\mathbf{u}) \times(\mathbf{R}\mathbf{v}) and (i), the last term equals (Rp(a)δa)×(Rp(a)y)=(Rp(a)δa)×(x−ta)(R_{p(a)}\delta_a) \times(R_{p(a)}y) = (R_{p(a)}\delta_a) \times(x - t_a). Writing δωa=Rp(a)δa\delta\omega_a = R_{p(a)}\delta_a for the perturbation expressed in world coordinates, the extra factor acts as

x↦x+δωa×(x−ta).x \mapsto x + \delta\omega_a \times(x - t_a).

Intuitively, perturbing the rotation of joint aa rotates every point carried by the chain below aa by the small angle δωa\delta\omega_a about the posed location tat_a of joint aa.

From the joint transformation to the vertex. We write Fa=HaRot⁡(I+[δa]×)Ha−1F_a = H_a \operatorname{Rot}(I + [\delta_a]_{\times}) H_a^{-1} for the extra factor, so that Gjpert=FaGjG_j^{\mathrm{pert}} = F_a G_j and Fax=x+δωa×(x−ta)F_a x = x + \delta\omega_a \times(x - t_a) for any point xx. To accurately locate the vertex, we need to adjust for the pivot. However, pivot removal for GjpertG_j^{\mathrm{pert}} and GjG_j is the same since this step does not depend on the pose, perturbed or not. Therefore, Gjpert=FaGjG_j^{\mathrm{pert}} = F_a G_j. Hence, FaF_a acts on the posed vertex GjXiG_jX_i like on any other point. For every joint jj at or below aa,

GjpertXi=Fa(GjXi)=GjXi+δωa×(GjXi−ta),j∈D(a).G_j^{\mathrm{pert}} X_i = F_a(G_jX_i) = G_jX_i + \delta\omega_a \times(G_jX_i - t_a), \qquad j \in\mathcal{D}(a).

For j∉D(a)j \notin\mathcal{D}(a), ωa\omega_a does not appear in the chain of GjG_j, so there is no factor FaF_a and GjXiG_jX_i is unchanged.

Step 2: Effect of perturbing all joints

Let every joint be perturbed. Each a∈A(j)a \in A(j) contributes an extra perturbation. Expanding the product and dropping products of two or more small terms, the displacements of Step 1 add up to,

GjpertXi≈GˉjXi+∑a∈A(j)δωa×(GˉjXi−tˉa).G_j^{\mathrm{pert}} X_i \approx\bar{G}_jX_i + \sum_{a \in A(j)} \delta\omega_a \times(\bar{G}_jX_i - \bar{t}_a).

where a bar denotes evaluation at the mean pose μθ\mu_\theta. Substituting into the skinning equation (Eq. 6),

vi′≈∑j∈Jwi,jGˉjXi+∑j∈J∑a∈A(j)wi,jδωa×(GˉjXi−tˉa).v_i' \approx\sum_{j \in\mathcal{J}} w_{i,j}\bar{G}_jX_i + \sum_{j \in\mathcal{J}}\sum_{a \in A(j)} w_{i,j}\delta\omega_a \times(\bar{G}_jX_i - \bar{t}_a).

The double sum runs over the pairs (j,a)(j,a) with a∈A(j)a \in A(j), which is the same set of pairs as j∈D(a)j \in\mathcal{D}(a). Exchanging the order of summation therefore gives ∑j∈J∑a∈A(j)=∑a∈J∑j∈D(a))\sum_{j \in\mathcal{J}}\sum_{a \in A(j)} = \sum_{a \in\mathcal{J}}\sum_{j \in\mathcal{D}(a))}, and since δωa\delta\omega_a does not depend on jj it factors out of the inner sum

vi′≈∑j∈Jwi,jGˉjXi+∑a∈Jδωa×ℓi,a,ℓi,a=∑j∈D(a)wi,j(GˉjXi−tˉa).v_i' \approx\sum_{j \in\mathcal{J}} w_{i,j}\bar{G}_jX_i + \sum_{a \in\mathcal{J}} \delta\omega_a \times\ell_{i,a}, \qquad\ell_{i,a} = \sum_{j \in\mathcal{D}(a)} w_{i,j}(\bar{G}_jX_i - \bar{t}_a).

For each joint aa it collects exactly those skinning terms whose joint lies at or below aa, the only terms that ωa\omega_a affects. Since tˉa\bar{t}_a does not depend on jj,

ℓi,a=∑j∈D(a)wi,jGˉjXi−Wi,atˉa,Wi,a=∑j∈D(a)wi,j,\ell_{i,a} = \sum_{j \in\mathcal{D}(a)} w_{i,j}\bar{G}_jX_i - W_{i,a}\bar{t}_a, \qquad W_{i,a} = \sum_{j \in\mathcal{D}(a)} w_{i,j},

with Wi,aW_{i,a} the total skinning weight of vertex ii on the joints at or below aa. We call ℓi,a\ell_{i,a} the lever arm of vertex ii about joint aa. It vanishes when no joint in D(a)\mathcal{D}(a) skins vertex ii, so only the joints on the chains above vertex ii contribute.

Step 3: variance

The first sum in Step 2 is deterministic whereas the second is the random. Writing δωa×ℓi,a=−ℓi,a×δωa=−[ℓi,a]×δωa\delta\omega_a \times\ell_{i,a} = -\ell_{i,a} \times\delta\omega_a = -[\ell_{i,a}]_{\times}\delta\omega_a and assuming the perturbations of different joints to be independent with zero mean,

Var⁡θ(vi′∣Xi)=∑a∈J[ℓi,a]×Σθa[ℓi,a]×T,\operatorname{Var}_{\theta}(v'_i \mid X_i) = \sum_{a \in\mathcal{J}} [\ell_{i,a}]_{\times}\Sigma_{\theta_a}[\ell_{i,a}]_{\times}^{T},
Σθa=Cov⁡(δωa)=Rˉp(a)Var⁡(δa)Rˉp(a)T,\Sigma_{\theta_a} = \operatorname{Cov}(\delta\omega_a) = \bar{R}_{p(a)}\operatorname{Var}(\delta_a)\bar{R}_{p(a)}^{T},
Var⁡(δa)=Jl(ωˉa)diag⁡(σθa,x2,σθa,y2,σθa,z2)Jl(ωˉa)T.\operatorname{Var}(\delta_a) = J_l(\bar{\omega}_a)\operatorname{diag}(\sigma_{\theta_a,x}^{2},\sigma_{\theta_a,y}^{2},\sigma_{\theta_a,z}^{2})J_l(\bar{\omega}_a)^{T}.

Here, σθa,⋅2\sigma_{\theta_a,\cdot}^{2} are the predicted per-axis variances of the axis-angle coordinates of joint aa, in the axes of its parent, Jl(ωˉa)J_l(\bar{\omega}_a) is the left Jacobian of Step 1 evaluated at the mean pose, and Rˉp(a)\bar{R}_{p(a)} is the mean global rotation of the parent. Σθa\Sigma_{\theta_a} is the covariance in world axes, Σθa(pq)\Sigma_{\theta_a}^{(pq)}, p,q∈{x,y,z}p,q \in\{x,y,z\}. The outer expectation over XiX_i in Eq. (8) is taken to first order by evaluating the lever arm at the mean rest-pose vertex μXi\mu_{X_i},

ℓˉi,a=∑j∈D(a)wi,jGˉjμXi−Wi,atˉa=(ℓi,a(x),ℓi,a(y),ℓi,a(z))T.\bar{\ell}_{i,a} = \sum_{j \in\mathcal{D}(a)} w_{i,j}\bar{G}_j\mu_{X_i} - W_{i,a}\bar{t}_a = \left(\ell_{i,a}^{(x)},\ell_{i,a}^{(y)},\ell_{i,a}^{(z)}\right)^{T}.

Writing [ℓˉi,a]×[\bar{\ell}_{i,a}]_{\times} as the cross-product matrix of this vector, the rotation variance is the diagonal of [ℓˉi,a]×Σθa[ℓˉi,a]×T[\bar{\ell}_{i,a}]_{\times}\Sigma_{\theta_a}[\bar{\ell}_{i,a}]_{\times}^{T}. This gives us

EX[Var⁡θ(vi′∣Xi)]≈∑a∈J[Σθa(yy)(ℓi,a(z))2+Σθa(zz)(ℓi,a(y))2−2Σθa(yz)ℓi,a(y)ℓi,a(z)Σθa(xx)(ℓi,a(z))2+Σθa(zz)(ℓi,a(x))2−2Σθa(xz)ℓi,a(x)ℓi,a(z)Σθa(xx)(ℓi,a(y))2+Σθa(yy)(ℓi,a(x))2−2Σθa(xy)ℓi,a(x)ℓi,a(y)].(9)\mathbb{E}_{X}\left[\operatorname{Var}_{\theta}(v'_i \mid X_i)\right] \approx\sum_{a \in\mathcal{J}} \begin{bmatrix} \Sigma_{\theta_a}^{(yy)}(\ell_{i,a}^{(z)})^2+\Sigma_{\theta_a}^{(zz)}(\ell_{i,a}^{(y)})^2-2\Sigma_{\theta_a}^{(yz)}\ell_{i,a}^{(y)}\ell_{i,a}^{(z)}\\ \Sigma_{\theta_a}^{(xx)}(\ell_{i,a}^{(z)})^2+\Sigma_{\theta_a}^{(zz)}(\ell_{i,a}^{(x)})^2-2\Sigma_{\theta_a}^{(xz)}\ell_{i,a}^{(x)}\ell_{i,a}^{(z)}\\ \Sigma_{\theta_a}^{(xx)}(\ell_{i,a}^{(y)})^2+\Sigma_{\theta_a}^{(yy)}(\ell_{i,a}^{(x)})^2-2\Sigma_{\theta_a}^{(xy)}\ell_{i,a}^{(x)}\ell_{i,a}^{(y)} \end{bmatrix}. \tag*{(9)}

Shape and Expression Variance.

Now let the pose take its expected value and let only the shape and expression vary. To first order in the pose perturbation (A2), the expected transformation of each joint is the transformation at the mean pose, Gˉj\bar{G}_j. The expected vertex is then

Eθ[vi′∣Xi]=(∑j∈Jwi,jGˉj)Xi.\mathbb{E}_{\theta}[v'_i \mid X_i] = \left(\sum_{j \in\mathcal{J}} w_{i,j}\bar{G}_j\right)X_i.

We note that XiX_i is random only in its first three components (the offsets). The joint locations J\mathbf{J} and the translations tˉj\bar{t}_j are evaluated at the mean shape and interact only with the homogeneous coordinate of 1 and do not contribute to the variance. What remains is the skinning-blended rotation

Rieff=∑j∈Jwi,jRˉjR_i^{\mathrm{eff}} = \sum_{j \in\mathcal{J}} w_{i,j}\bar{R}_j

applied to the rest-pose offsets. Using Var⁡[AZ]=AΣZAT\operatorname{Var}[AZ] = A\Sigma_ZA^{T} with the covariance ΣXi\Sigma_{X_i} of Eq. (7), the shape and expression variance is the diagonal of RieffΣXi(Rieff)TR_i^{\mathrm{eff}}\Sigma_{X_i}(R_i^{\mathrm{eff}})^{T},

Var⁡X(Eθ[vi′∣Xi])=[∑k,l=13Ri,1,keffRi,1,leffΣXi(kl)∑k,l=13Ri,2,keffRi,2,leffΣXi(kl)∑k,l=13Ri,3,keffRi,3,leffΣXi(kl)].(10)\operatorname{Var}_{X}\left(\mathbb{E}_{\theta}[v'_i \mid X_i]\right) = \begin{bmatrix} \sum_{k,l=1}^{3}R_{i,1,k}^{\mathrm{eff}}R_{i,1,l}^{\mathrm{eff}}\Sigma_{X_i}^{(kl)}\\ \sum_{k,l=1}^{3}R_{i,2,k}^{\mathrm{eff}}R_{i,2,l}^{\mathrm{eff}}\Sigma_{X_i}^{(kl)}\\ \sum_{k,l=1}^{3}R_{i,3,k}^{\mathrm{eff}}R_{i,3,l}^{\mathrm{eff}}\Sigma_{X_i}^{(kl)} \end{bmatrix}. \tag*{(10)}

Intuitively, the rest-pose offset covariance of the vertex is rotated into the posed body by the vertex’s own blended rotation.

Finally, substituting Eq. (9) and Eq. (10) into Eq. (8) gives the per-vertex variance

Var⁡(vi′)≈∑a∈J[Σθa(yy)(ℓi,a(z))2+Σθa(zz)(ℓi,a(y))2−2Σθa(yz)ℓi,a(y)ℓi,a(z)Σθa(xx)(ℓi,a(z))2+Σθa(zz)(ℓi,a(x))2−2Σθa(xz)ℓi,a(x)ℓi,a(z)Σθa(xx)(ℓi,a(y))2+Σθa(yy)(ℓi,a(x))2−2Σθa(xy)ℓi,a(x)ℓi,a(y)]+[∑k,l=13Ri,1,keffRi,1,leffΣXi(kl)∑k,l=13Ri,2,keffRi,2,leffΣXi(kl)∑k,l=13Ri,3,keffRi,3,leffΣXi(kl)].(11)\operatorname{Var}(v'_i) \approx\sum_{a \in\mathcal{J}} \begin{bmatrix} \Sigma_{\theta_a}^{(yy)}(\ell_{i,a}^{(z)})^2+\Sigma_{\theta_a}^{(zz)}(\ell_{i,a}^{(y)})^2-2\Sigma_{\theta_a}^{(yz)}\ell_{i,a}^{(y)}\ell_{i,a}^{(z)}\\ \Sigma_{\theta_a}^{(xx)}(\ell_{i,a}^{(z)})^2+\Sigma_{\theta_a}^{(zz)}(\ell_{i,a}^{(x)})^2-2\Sigma_{\theta_a}^{(xz)}\ell_{i,a}^{(x)}\ell_{i,a}^{(z)}\\ \Sigma_{\theta_a}^{(xx)}(\ell_{i,a}^{(y)})^2+\Sigma_{\theta_a}^{(yy)}(\ell_{i,a}^{(x)})^2-2\Sigma_{\theta_a}^{(xy)}\ell_{i,a}^{(x)}\ell_{i,a}^{(y)} \end{bmatrix} + \begin{bmatrix} \sum_{k,l=1}^{3}R_{i,1,k}^{\mathrm{eff}}R_{i,1,l}^{\mathrm{eff}}\Sigma_{X_i}^{(kl)}\\ \sum_{k,l=1}^{3}R_{i,2,k}^{\mathrm{eff}}R_{i,2,l}^{\mathrm{eff}}\Sigma_{X_i}^{(kl)}\\ \sum_{k,l=1}^{3}R_{i,3,k}^{\mathrm{eff}}R_{i,3,l}^{\mathrm{eff}}\Sigma_{X_i}^{(kl)} \end{bmatrix}. \tag*{(11)}

Pose Correctives

We use STAR as the 3DMM to analyse pose correctives. Following STAR’s notation convention, PjP_j is the per-joint pose corrective function which regresses corrective offsets given the joint and its direct neighbors’ quaternion values.

Pj(qne⁡(j),β2;Kj)=Kj((qne⁡(j)−qne⁡(j)∗)T∣β2)TP_j(q_{\operatorname{ne}(j)}, \beta_2; K_j) = K_j\left((q_{\operatorname{ne}(j)} - q^*_{\operatorname{ne}(j)})^T \mid\beta_2\right)^T

Here, the second principal component β2\beta_2 as well as qne⁡(j)q_{\operatorname{ne}(j)} are stochastic, while qne⁡(j)∗q^*_{\operatorname{ne}(j)} and KjK_j are deterministic. We note that the variance of this equation follows the familiar Var⁡[AZ]=AΣZAT\operatorname{Var}[AZ] = A\Sigma_ZA^T format where AA is a deterministic matrix and ZZ is random.

BPj(qne⁡(j);Aj,Kj)=ϕ(Aj)∘Pj(qne⁡(j),β2;Kj)(12)B^j_P(q_{\operatorname{ne}(j)}; A_j, K_j) = \phi(A_j) \circ P_j(q_{\operatorname{ne}(j)}, \beta_2; K_j) \tag*{(12)}

Since the activation weights AjA_j are fixed, ϕ(Aj)\phi(A_j) is a deterministic vector. The variance of a hadamard product is Var⁡[A∘Z]i=ai2σZi2\operatorname{Var}[A \circ Z]_i = a_i^2\sigma^2_{Z_i}. This allows us to compute the variance from pose correctives.

Training Details

Training. All methods share the same training recipe and differ only in the probabilistic modeling. The DINOv3 ViT-H/16+ image encoder is initialised and kept frozen. All other modules of the SAM 3D Body architecture are randomly initialized and trained from scratch. We optimise with AdamW (learning rate 10−310^{-3}, β=(0.9,0.999)\beta= (0.9, 0.999), weight decay 0.05), a linear warm-up over the first 1k steps followed by cosine decay to 5% of the peak learning rate, and gradient clipping at norm 1.0. Each step processes 32 images per GPU on 4 GPUs (128 images per step); we train for 30k steps. To train SAM 3D Body (CLUE), we freeze SAM 3D Body and only keep the variance head learnable. We train this head for 10k steps.

Losses. All methods share the same deterministic terms: an L2 loss on the 2D keypoints using the predicted perspective camera, an L2 loss on the 3D joints, and a joint-limit penalty on body joint angles. The methods differ only on how the MHR parameters are supervised. Deterministic training uses L2 loss on all MHR parameters. Parameter-space NLL replaces these deterministic losses on the MHR parameters. Normalizing flow applies NLL and RealNVP flow NLL on the joint parameters, keeping shape loss deterministic. Matrix-Fisher uses a matrix-Fisher NLL on each supervised joint rotation, with concentrations predicted hierarchically along the kinematic chain, and the Gaussian NLL for lengths and shape. CLUE uses the same losses as parameter-space NLL, but replaces L2 loss over the joints with NLL. All loss terms have unit weight except the joint-limit penalty (0.01).

Metrics. We follow standard practices and report (1) joint based metrics such as MPJPE and PA-MPJPE, and (2) vertex based metrics such as PVE and PA-PVE. When the prediction and (pseudo) ground truth are in the same space (MHR space), we additionally report variance based metrics such as vertex-NLL, spearman and pearson correlation AUSE. Since baselines measure uncertainty in the parameter space, we draw 100 samples per prediction to compute the mesh space variance.

Datasets. For training, we use the Segment Anything-1B (SA-1B) dataset which consists of 1.2 million+ training images Kirillov et al. [23]. Additionally, we use MPII and COCO with MHR annotations for training. For our evaluation, we use 3DPW and Mocap 3D, with 3DPW as our primary dataset containing ground truth motion capture data in SMPL format. Since our predictions are in MHR space and the ground truth is in SMPL, we use on-the-fly optimization to map MHR to SMPL as done in Yang et al. [56]. Ferguson et al. [14]. Additionally, we also use MPII Andriluka et al. [1], Human3.6M Ionescu et al. [20], COCO Lin et al. [33], MPI-INF-3D Mehta et al. [35], PoseTrack Andriluka et al. [2], CrowdPose Li et al. [29] and OCHuman Zhang et al. [57]. Unlike 3DPW, these datasets do not have 3DMM annotations, and hence we use SAM 3D Body as the pseudo ground truth.

Pseudo-Code

We present the algorithm for an efficient implementation of Closed-form Local Uncertainty Estimation in Algorithm 1.

Full Results

We gather our results and visualizations in this section.

Algorithm 1 CLUE

Require: σβ2∈RB×C\sigma_{\beta}^{2} \in\mathbb{R}^{B \times C}: shape and expression parameter variances concatenated; σθ2∈RB×J×3\sigma_{\theta}^{2} \in\mathbb{R}^{B \times J \times3}: pose variances, per axis-angle coordinate in the parent’s axes; vˉ∈RV×3\bar{v} \in\mathbb{R}^{V \times3}: template mesh; B∈RV×3×C\mathbf{B} \in\mathbb{R}^{V \times3 \times C}: blendshape basis (shape ∪\cup expression); R∈RJ×V\mathcal{R} \in\mathbb{R}^{J \times V}: joint regressor; W∈RV×J\mathbf{W} \in\mathbb{R}^{V \times J}: LBS weights; β⃗∈RB×C\vec{\beta} \in\mathbb{R}^{B \times C}: mean shape and expression coefficients concatenated; θ⃗∈RB×J×3\vec{\theta} \in\mathbb{R}^{B \times J \times3}: mean pose parameters (axis-angle); π\pi: parent map

Ensure: Var⁡(v′)∈RB×V×3\operatorname{Var}(v') \in\mathbb{R}^{B \times V \times3}: per-vertex variance

— Precompute once at initialisation to reduce runtime —

  1. Build ancestor matrix P∈{0,1}J×JP \in\{0,1\}^{J \times J}.

  2. for k=0,…,J−1k=0,\ldots,J-1 do

  3. j←kj \leftarrow k

  4. while j≥0j \ge0 do ▷ No parent is represented as -1

  5. Pk,j←1P_{k,j} \leftarrow1; j←π(j)j \leftarrow\pi(j)

  6. end while

  7. end for

  8. Bv,k,l,m⊗←Bv,k,mBv,l,m\mathbf{B}^{\otimes}_{v,k,l,m} \leftarrow\mathbf{B}_{v,k,m}\mathbf{B}_{v,l,m} ▷ (V×3×3×C)(V \times3 \times3 \times C), outer product per vertex and blendshape

  9. W~←W[:,:,None]⋅P[None,:,:]\widetilde{\mathbf{W}} \leftarrow\mathbf{W}[:, :, \text{None}] \cdot P[\text{None}, :, :] ▷ Fusing skinning weights and ancestor matrix

— Per-forward pass —

Step 1: Shape / expression covariance ▷ Eq. (7)

  1. (ΣX)b,v,k,l←∑mσβ2[b,m]Bv,k,l,m⊗(\Sigma_X)_{b,v,k,l} \leftarrow\sum_m \sigma_{\beta}^{2}[b,m]\mathbf{B}^{\otimes}_{v,k,l,m} ▷ (B×V×3×3)(B \times V \times3 \times3), offset covariance

Step 2: Forward kinematics

  1. v^←vˉ+Bβ⃗\hat{v} \leftarrow\bar{v}+\mathbf{B}\vec{\beta} ▷ (B×V×3)(B \times V \times3), shaped mesh

  2. J←Rv^\mathbf{J} \leftarrow\mathcal{R}\hat{v} ▷ (B×J×3)(B \times J \times3), joint locations

  3. {Rj,tj}j=1J←FK⁡(θ⃗,J,π)\{\mathbf{R}_j,\mathbf{t}_j\}_{j=1}^{J} \leftarrow\operatorname{FK}(\vec{\theta},\mathbf{J},\pi) ▷ rotations and translations of the pivot-removed Gj\mathbf{G}_j

  4. Tj←RjJj+tj∀j\mathbf{T}_j \leftarrow\mathbf{R}_j\mathbf{J}_j+\mathbf{t}_j \quad\forall j ▷ (B×J×3)(B \times J \times3), posed joint centres GjJj\mathbf{G}_j\mathbf{J}_j (pivots)

Step 3: Vertex transformation

Modelling effect of different joint transformations on the vertex

  1. v~←Rv^+t\tilde{v} \leftarrow\mathbf{R}\hat{v}+\mathbf{t} ▷ bjpq,bvq→bvjp\mathbf{b}_{jpq},\mathbf{b}_{vq}\to\mathbf{b}_{vjp}; (B×V×J×3)(B \times V \times J \times3), Gjv^\mathbf{G}_j\hat{v}

    Broadcast helps in parallelizing offset calculation w.r.t pivot

  1. r←v~[:,:,:,None,:]−T[:,None,None,:,:]r \leftarrow\tilde{v}[:, :, :, \text{None}, :]-\mathbf{T}[:, \text{None}, \text{None}, :, :] ▷ rotation around pivot; (B×V×K×J×3)(B \times V \times K \times J \times3)

    Reducing this w.r.t fused skinning weights + ancestor matrix

  1. rb,v,j,3←∑kW~⋅rr_{b,v,j,3} \leftarrow\sum_k \widetilde{W}\cdot r ▷ vkj,bvkjd→bvjdv_{kj},b v_{kjd}\to b v_{jd}; (B×V×J×3)(B \times V \times J \times3), lever arm ℓˉv,j\bar{\ell}_{v,j} about joint jj

Step 4: Rotational variance ▷ Eq. (9)

  1. aj,bj,cj←rv,j(x),rv,j(y),rv,j(z)a_j,b_j,c_j \leftarrow r_{v,j}^{(x)},r_{v,j}^{(y)},r_{v,j}^{(z)} ▷ (B×V×J)(B \times V \times J) each

  2. Jl(θ⃗j)←I+1−cos⁡ϑjϑj2[θ⃗j]×+ϑj−sin⁡ϑjϑj3[θ⃗j]×2,ϑj=∥θ⃗j∥J_l(\vec{\theta}_j) \leftarrow I+\frac{1-\cos\vartheta_j}{\vartheta_j^2}[\vec{\theta}_j]_{\times}+\frac{\vartheta_j-\sin\vartheta_j}{\vartheta_j^3}[\vec{\theta}_j]_{\times}^{2},\quad\vartheta_j=\lVert\vec{\theta}_j\rVert ▷ (B×J×3×3)(B \times J \times3 \times3), left Jacobian of SO(3)SO(3) at the mean pose

  3. Σj←Rπ(j)Jl(θ⃗j)diag⁡(σθ,j2)Jl(θ⃗j)TRπ(j)T\Sigma_j \leftarrow\mathbf{R}_{\pi(j)}J_l(\vec{\theta}_j)\operatorname{diag}(\sigma_{\theta,j}^{2})J_l(\vec{\theta}_j)^{T}\mathbf{R}_{\pi(j)}^{T} ▷ (B×J×3×3)(B \times J \times3 \times3), world axes, Rπ(root)=I\mathbf{R}_{\pi(\mathrm{root})}=I

  4. Var⁡θ(v′)←∑j=1J[Σj(yy)cj2+Σj(zz)bj2−2Σj(yz)bjcjΣj(xx)cj2+Σj(zz)aj2−2Σj(xz)ajcjΣj(xx)bj2+Σj(yy)aj2−2Σj(xy)ajbj]\operatorname{Var}_{\theta}(v') \leftarrow\sum_{j=1}^{J}\begin{bmatrix}\Sigma_j^{(yy)}c_j^2+\Sigma_j^{(zz)}b_j^2-2\Sigma_j^{(yz)}b_jc_j\\ \Sigma_j^{(xx)}c_j^2+\Sigma_j^{(zz)}a_j^2-2\Sigma_j^{(xz)}a_jc_j\\ \Sigma_j^{(xx)}b_j^2+\Sigma_j^{(yy)}a_j^2-2\Sigma_j^{(xy)}a_jb_j\end{bmatrix} ▷ (B×V×3)(B \times V \times3)

Step 5: Rotated shape / expression variance ▷ Eq. (10)

  1. Rveff←∑j=1Jwv,jRj\mathbf{R}^{\mathrm{eff}}_{v} \leftarrow\sum_{j=1}^{J}w_{v,j}\mathbf{R}_j ▷ (B×V×3×3)(B \times V \times3 \times3), LBS-blended rotation

  2. Var⁡β(v′)p←∑k,l=13Rv,p,keffRv,p,leff(ΣX)v,k,l∀p∈{1,2,3}\operatorname{Var}_{\beta}(v')_p \leftarrow\sum_{k,l=1}^{3}R^{\mathrm{eff}}_{v,p,k}R^{\mathrm{eff}}_{v,p,l}(\Sigma_X)_{v,k,l}\quad\forall p \in\{1,2,3\} ▷ (B×V×3)(B \times V \times3)

Step 6: Total variance ▷ Eq. (11)

  1. return Var⁡(v′)←Var⁡θ(v′)+Var⁡β(v′)\operatorname{Var}(v') \leftarrow\operatorname{Var}_{\theta}(v')+\operatorname{Var}_{\beta}(v')

References

  1. [1]Mykhaylo Andriluka, Leonid Pishchulin, Peter Gehler, and Bernt Schiele. 2d human pose estimation: New benchmark and state of the art analysis. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2014.DOI
  2. [2]Mykhaylo Andriluka, Umar Iqbal, Eldar Insafutdinov, Leonid Pishchulin, Anton Milan, Juergen Gall, and Bernt Schiele. Posetrack: A benchmark for human pose estimation and tracking. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.arxiv.org/abs/1710.10000
  3. [3]Benjamin Biggs, David Novotny, Sebastien Ehrhardt, Hanbyul Joo, Ben Graham, and Andrea Vedaldi. 3d multi-bodies: Fitting sets of plausible 3d human models to ambiguous image data. Advances in neural information processing systems, 33:20496–20507, 2020.arxiv.org/abs/2011.00980
  4. [4]Volker Blanz and Thomas Vetter. A morphable model for the synthesis of 3d faces. In Seminal Graphics Papers: Pushing the Boundaries, Volume 2, pp. 157–164. 2023.
  5. [5]Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural network. In International conference on machine learning, pp. 1613–1622. PMLR, 2015.
  6. [6]Federica Bogo, Angjoo Kanazawa, Christoph Lassner, Peter Gehler, Javier Romero, and Michael J Black. Keep it smpl: Automatic estimation of 3d human pose and shape from a single image. In European conference on computer vision, pp. 561–578. Springer, 2016.arxiv.org/abs/1607.08128
  7. [7]Hanbyel Cho and Junmo Kim. Generative approach for probabilistic human mesh recovery using diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4183–4188, 2023.arxiv.org/abs/2308.02963
  8. [8]Yu Deng, Jiaolong Yang, Sicheng Xu, Dong Chen, Yunde Jia, and Xin Tong. Accurate 3d face reconstruction with weakly-supervised learning: From single image to image set. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pp. 0–0, 2019.arxiv.org/abs/1903.08527
  9. [9]Rahul Dey and Vishnu Naresh Bodd etti. Generating diverse 3d reconstructions from a single occluded face image. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1547–1557, 2022.arxiv.org/abs/2112.00879
  10. [10]Olaf Dünkel, Tim Salzmann, and Florian Pfaff. Normalizing flows on the product space of so (3) manifolds for probabilistic human pose modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2285–2294, 2024.arxiv.org/abs/2404.05675
  11. [11]Sai Kumar Dwivedi, Cordelia Schmid, Hongwei Yi, Michael J Black, and Dimitrios Tzionas. Poco: 3d pose and shape estimation with confidence. In 2024 International Conference on 3D Vision (3DV), pp. 85–95. IEEE, 2024.arxiv.org/abs/2308.12965
  12. [12]Bernhard Egger, Andreas Schneider, Clemens Blumer, Sandro Schönborn, and Thomas Vetter. Occlusion-aware 3d morphable face models. 2016.DOI
  13. [13]Bernhard Egger, Sandro Schönborn, Andreas Schneider, Adam Kortylewski, Andreas Morel-Forster, Clemens Blumer, and Thomas Vetter. Occlusion-aware 3d morphable models and an illumination prior for face image analysis. International Journal of Computer Vision, 126(12): 1269–1287, 2018.DOI
  14. [14]Aaron Ferguson, Ahmed AA Osman, Berta Bescos, Carsten Stoll, Chris Twigg, Christoph Lassner, David Otte, Eric Vignola, Fabian Prada, Federica Bogo, et al. Mhr: Momentum human rig. arXiv preprint arXiv:2511.15586, 2025.arxiv.org/abs/2511.15586
  15. [16]Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning, pp. 1050–1059. PMLR, 2016.arxiv.org/abs/1506.02142
  16. [17]Google. XNNPACK: High-efficiency floating-point neural network inference operators for mobile, server, and web, 2026. URL https://github.com/google/XNNPACK. Accessed: 2026-02-24.
  17. [19]Alexander Immer, Emanuele Palumbo, Alexander Marx, and Julia Vogt. Effective bayesian heteroscedastic regression with deep neural networks. Advances in neural information processing systems, 36:53996–54019, 2023.DOI
  18. [20]Catalin Ionescu, Dragos Papava, Vlad Olaru, and Cristian Sminchisescu. Human3.6m: Large scale datasets and predictive methods for 3d human sensing in natural environments. IEEE Transactions on Pattern Analysis and Machine Intelligence, 36(7):1325–1339, jul 2014.DOI
  19. [21]Angjoo Kanazawa, Michael J Black, David W Jacobs, and Jitendra Malik. End-to-end recovery of human shape and pose. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7122–7131, 2018.arxiv.org/abs/1712.06584
  20. [22]Alex Kendall and Yarin Gal. What uncertainties do we need in bayesian deep learning for computer vision? In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https://proceedings.neurips.cc/paper_files/paper/2017/file/2650d6089a6d640c5e85b2b88265dc2b-Paper.pdf.
  21. [23]Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick. Segment anything. arXiv:2304.02643, 2023.arxiv.org/abs/2304.02643
  22. [24]Muhammed Kocabas, Chun-Hao P Huang, Otmar Hilliges, and Michael J Black. Pare: Part attention regressor for 3d human body estimation. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 11127–11137, 2021.arxiv.org/abs/2104.08527
  23. [25]Nikos Kolotouros, Georgios Pavlakos, Michael J Black, and Kostas Daniilidis. Learning to reconstruct 3d human pose and shape via model-fitting in the loop. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 2252–2261, 2019.arxiv.org/abs/1909.12828
  24. [26]Nikos Kolotouros, Georgios Pavlakos, Dinesh Jayaraman, and Kostas Daniilidis. Probabilistic modeling for human mesh recovery. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 11605–11614, 2021.arxiv.org/abs/2108.11944
  25. [27]Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems, 30, 2017.arxiv.org/abs/1612.01474
  26. [28]Chunlu Li, Andreas Morel-Forster, Thomas Vetter, Bernhard Egger, and Adam Kortylewski. Robust model-based face reconstruction through weakly-supervised outlier segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 372–381, 2023.arxiv.org/abs/2106.09614
  27. [29]Jiefeng Li, Can Wang, Hao Zhu, Yihuan Mao, Haoshu Fang, and Cewu Lu. Crowdpose: Efficient crowded scenes pose estimation and a new benchmark. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pp. 10863–10872. Computer Vision Foundation / IEEE, 2019. doi: 10.1109/CVPR.2019.01112. URL http://openaccess.thecvf.com/content_CVPR_2019/html/Li_CrowdPose_Efficient_Crowded_Scenes_Pose_Estimation_and_a_New_Benchmark_CVPR_2019_paper.html.DOI
  28. [30]Tianye Li, Timo Bolkart, Michael J Black, Hao Li, and Javier Romero. Learning a model of facial shape and expression from 4d scans. ACM Transactions on Graphics (TOG), 36(6):1–17, 2017.DOI
  29. [31]Zhihiao Li, Jianzhuang Liu, Zhensong Zhang, Songcen Xu, and Youliang Yan. Cliff: Carrying location information in full frames into human pose and shape estimation. In European Conference on Computer Vision, pp. 590–606. Springer, 2022.arxiv.org/abs/2208.00571
  30. [32]Kevin Lin, Lijuan Wang, and Zicheng Liu. Mesh graphormer. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 12939–12948, 2021.
  31. [33]Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740–755. Springer, 2014.arxiv.org/abs/1405.0312
  32. [34]Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J. Black. SMPL: A skinned multi-person linear model. ACM Trans. Graphics (Proc. SIGGRAPH Asia), 34(6):248:1–248:16, October 2015.
  33. [35]Dushyant Mehta, Helge Rhodin, Dan Casas, Pascal Fua, Oleksandr Sotnychenko, Weipeng Xu, and Christian Theobalt. Monocular 3d human pose estimation in the wild using improved cnn supervision. In 3D Vision (3DV), 2017 Fifth International Conference on. IEEE, 2017. doi: 10.1109/3DV.2017.00064. URL http://gvv.mpi-inf.mpg.de/3dhp_dataset.DOI
  34. [36]D.A. Nix and A.S. Weigend. Estimating the mean and variance of the target probability distribution. In Proceedings of 1994 IEEE International Conference on Neural Networks (ICNN’94), volume 1, pp. 55–60 vol.1, 1994. doi: 10.1109/ICNN.1994.374138.DOI
  35. [37]Ahmed AA Osman, Timo Bolkart, and Michael J Black. Star: Sparse trained articulated human body regressor. In European Conference on Computer Vision, pp. 598–613. Springer, 2020.DOI
  36. [38]Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelsh​ein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.arxiv.org/abs/1912.01703
  37. [39]Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed A. A. Osman, Dimitrios Tzionas, and Michael J. Black. Expressive body capture: 3D hands, face, and body from a single image. In Proceedings IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pp. 10975–10985, 2019.arxiv.org/abs/1904.05866
  38. [40]S. Ploumpis, J. Bednarik, G. Zoss, R. Guseinov, L. Prasso, P. Chandran, O. Boyne, V. Choutas, T. Bolkart, D. Wang, M. Chai, D. Qiu, S. Winberg, G. Rainer, L. Bridgeman, L. L. Helminger, E. Collins, D. Vicini, J. Riviere, Y. Boetzel, A. Koumis, S. Moschoglou, J. Busch, C. Herrera, J. Still, S. Ysebert, P. Lincoln, S. O. Escolano, C. Rehmann, E. Wood, T. Beeler, and S. Zafeiriou. Gmm head: A generative anthropometric model of the human head. 2026. URL https://arxiv.org/abs/2607.23687.
  39. [43]Pratheba Selvaraju, Victoria Fernández Abrevaya, Timo Bolkart, Rick Akkerman, Tianyu Ding, Faezeh Amjadi, and Ilya Zharkov. Ofer: Occluded face expression reconstruction. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 26985–26995, 2025.arxiv.org/abs/2410.21629
  40. [44]Akash Sengupta, Ignas Budvytis, and Roberto Cipolla. Probabilistic estimation of 3d human shape and pose with a semantic local parametric model. arXiv preprint arXiv:2111.15404, 2021a.arxiv.org/abs/2111.15404
  41. [45]Akash Sengupta, Ignas Budvytis, and Roberto Cipolla. Hierarchical kinematic probability distributions for 3d human shape and pose estimation from images in the wild. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 11219–11229, 2021b.arxiv.org/abs/2110.00990
  42. [46]Akash Sengupta, Ignas Budvytis, and Roberto Cipolla. Probabilistic 3d human shape and pose estimation from multiple unconstrained images in the wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16094–16104, 2021c.arxiv.org/abs/2103.10978
  43. [47]Akash Sengupta, Ignas Budvytis, and Roberto Cipolla. Humaniflow: Ancestor-conditioned normal-ising flows on so (3) manifolds for human pose and shape distribution estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4779–4789, 2023.
  44. [48]Karthik Shetty, Annette Birkhold, Bernhard Egger, Srikrishna Jaganathan, Norbert Strobel, Markus Kowarschik, and Andreas Maier. Propliks: Probabilistic 3d human body pose estimation. arXiv preprint arXiv:2412.04665, 2024.arxiv.org/abs/2412.04665
  45. [49]Megh Shukla, Mathieu Salzmann, and Alexandre Alahi. TIC-TAC: A framework for improved covariance estimation in deep heteroscedastic regression. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 45244–45257. PMLR, 2024. URL https://proceedings.mlr.press/v235/shukla24a.html.arxiv.org/abs/2310.18953
  46. [50]Megh Shukla, Aziz Shameem, Mathieu Salzmann, and Alexandre Alahi. Towards self-supervised covariance estimation in deep heteroscedastic regression. In International Conference on Learning Representations, volume 2025, pp. 93359–93381, 2025.arxiv.org/abs/2502.10587
  47. [51]Oriane Simeoni, Huy V. Vo, Maximilian Seitzer, Federico Baldassarre, Maxime Oquab, Cijo Jose, Vasil Khalidov, Marc Szafraniec, Seung Eun Yi, Michael Ramamonjisoa, Francesco Massa, Daniel HAZIZA, Luca Wehrstedt, Jianyuan Wang, Timothée Darcet, Théo Moutakanni, Leonel Santana, Claire Roberts, Andrea Vedaldi, Jamie Tolan, John Brandt, Camille Couprie, Julien Mairal, Herve Jegou, Patrick Labatut, and Piotr Bojanowski. DINOv3. Transactions on Machine Learning Research, 2026. ISSN 2835-8856. URL https://openreview.net/forum?id=2NlGyqNjns. Featured Certification.
  48. [52]Anastasis Stathopoulos, Ligong Han, and Dimitris Metaxas. Score-guided diffusion for 3d human recovery. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 906–915, 2024.arxiv.org/abs/2403.09623
  49. [53]Andrew Stirn, Harm Wessels, Megan Schertzer, Laura Pereira, Neville Sanjana, and David Knowles. Faithful heteroscedastic regression with neural networks. In International conference on artificial intelligence and statistics, pp. 5593–5613. PMLR, 2023.arxiv.org/abs/2212.09184
  50. [55]Timing Yang, Sicheng He, Hongyi Jing, Jiawei Yang, Zhijian Liu, Chuhang Zou, and Yue Wang. Fast sam 3d body: Accelerating sam 3d body for real-time full-body human mesh recovery. arXiv preprint arXiv:2603.15603, 2026a.arxiv.org/abs/2603.15603
  51. [56]Xitong Yang, Devansh Kukreja, Don Pinkus, Taosha Fan, Jinhyung Park, Soyong Shin, Jinkun Cao, Jia-Wei Liu, Nicolás Ugrinovic, Anushka Sagar, et al. Sam 3d body: Robust full-body human mesh recovery. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7209–7219, 2026b.
  52. [57]Song-Hai Zhang, Ruilong Li, Xin Dong, Paul Rosin, Zixi Cai, Xi Han, Dingcheng Yang, Haozhi Huang, and Shi-Min Hu. Pose2Seg: Detection free human instance segmentation. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 889–898. IEEE, 2019.arxiv.org/abs/1803.10683

Paper details

vlnc:W_kvvng4xx
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY-SA 4.0
Usage

views · 10

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

generative AI tools

View statement
Looking for related papers…
Contents