WorksComputer Science
XM4D: 4D Reconstruction through Certifiable Geometric Optimization
Abstract
Accurate 4D reconstruction remains challenging: jointly estimating camera motion, scene geometry, and non-rigid deformation entails highly nonconvex optimization, making global optimality difficult to guarantee. We present XM4D, a framework for accurate and efficient 4D reconstruction built upon certifiably globally optimal geometric optimization. By separating the estimation of camera motion from non-rigid deformation, we formulate both stages as structured quadratically constrained quadratic programs (QCQP) that admit semidefinite programming (SDP) relaxations and verifiable global optimality certificates. For camera motion estimation, we lift static point tracks into 3D using foundational depth prediction models and formulate a scaled bundle adjustment problem that jointly estimates camera poses, depth scales, and static landmarks. By analytically eliminating translations and landmarks, we reduce the problem to a QCQP over scaled rotations. For non-rigid reconstruction, we represent dynamic motion using an embedded deformation graph (EDG) and show that its tracking, rigidity, and temporal regularization objectives similarly reduce to a rotation-only QCQP. Both problems admit SDP relaxations that we solve efficiently with a GPU-accelerated Burer-Monteiro solver based on Riemannian optimization. The recovered geometry further initializes dynamic Gaussian splatting for photometric refinement. Extensive experiments across seven benchmarks demonstrate accurate camera motion estimation on both static and dynamic scenes, efficient optimization that scales to thousands of frames, and state-of-the-art novel-view rendering quality.
Website: https://computationalrobotics.seas.harvard.edu/XM4D/

Figure 1. XM4D overview. From (a) 2D tracks and monocular depth, a solver with certified global optimality followed by refinement recovers (b) the cameras, the static geometry (points), and the 3D trajectories of dynamic points (colored curves), which initialize (c) 4D Gaussian splatting for rendering. The skeleton in (c) is a subsampled embedded deformation graph (EDG).
Introduction
4D reconstruction aims to recover the time-varying 3D structure of a scene from visual observations. It supports applications including dynamic novel-view rendering [36, 42], deformable and articulated object reconstruction [24, 48, 74], digital humans and digital twins [25, 53, 70], and motion retargeting for humanoid control [3, 21, 40]. Despite its broad utility, reconstructing a dynamic scene from monocular images or videos remains highly challenging [17, 18]. Camera and object motion are entangled in the visual observations [15], while dynamic geometry itself is weakly constrained and may undergo complex non-rigid deformation. Long sequences introduce further difficulties, including fragmented point tracks, noisy and scale-inconsistent monocular depth, and temporally incomplete observations of the dynamic scene [35, 68].
Recent 4D reconstruction methods commonly rely on coupled nonconvex optimization over camera poses, scene geometry, deformation, and appearance [35, 68]. Such formulations have achieved impressive results, but their strong coupling makes the optimization difficult: the objective is highly nonconvex and local optimization may therefore be sensitive to initialization and converge to undesired local minima. As illustrated in Figure 3, local optimization of the motion of an embedded deformation graph (EDG) can converge to an inferior solution whereas the certifiable solver presented in this work achieves an accurate reconstruction. Similar failures of local optimization in static reconstruction were demonstrated in XM [20] (Fig. 10). This raises a natural question:
Figure 3. Certified versus local optimization. Results on a real DAVIS video [54] (top) and a synthetic deformation graph (bottom). Each panel shows the EDG nodes, their estimated rotations (red, green, and blue axes), and the graph edges at different frames. The local solver converges to a local minimum with wrong node rotations (orange boxes), whereas the certified XM solution recovers consistent rotations.
Can we achieve accurate and efficient 4D reconstruction through globally certifiable geometric optimization?
In this work, we present XM4D, a framework for accurate and efficient 4D reconstruction based on globally certifiable geometric optimization. Our approach separates camera motion estimation from non-rigid deformation, allowing both stages to be formulated as structured (nonconvex) quadratically constrained quadratic programs (QCQPs) and solved through (convex) semidefinite programming (SDP) relaxations. Figure 2(a) illustrates this decomposition as a view graph and Figure 2(b) instantiates it on a real scene: static and dynamic point tracks share the same cameras and predicted depth, but static tracks constrain time-invariant landmarks and camera poses, while dynamic tracks constrain the motion of an EDG. In the following, we overview the problem formulation and solution approach for both stages.
Figure 2. The XM4D view graph. (a) Schematic view graph. Camera poses (red) observe static landmarks (blue triangles) and dynamic points (pink circles) through the depth-lifted measurements and (blue squares). Each dynamic keypoint is attached by skinning edges (dashed) to its neighboring nodes of the embedded deformation graph (EDG) (purple capsules), whose nodes are connected by EDG edges (thick solid lines). (b) The view graph instantiated on a dynamic scene: a train moves along a circular track (black arrow), colored curves show the dynamic point tracks, and each part of the train is an EDG node with a free pose.
I: Static stage. As shown in Figure 2, let index video frames and index static points. The visibility set contains the tracked point-frame pairs, and denotes the corresponding 2D observation lifted to 3D using predicted monocular depth. Because monocular depth predictions are noisy and their scale drifts from frame to frame, we introduce a per-frame scale that corrects this frame-dependent scale error in the predicted depth. Weighting each observation by an optional confidence , we jointly estimate camera poses, depth scales, and static landmarks via
The objective enforces that observations of the same static point, after correcting the predicted depth scale and transforming them into the world frame, agree on a common 3D landmark. Without a scale constraint, the objective admits a degenerate solution as all variables shrink toward zero. The original XM formulation [20] prevents this collapse by anchoring the first-frame scale to , making the reconstruction overfit onto the first frame. We instead impose a uniform sphere constraint, , which fixes the global scale without privileging any frame. We formulate this problem as a quadratically constrained quadratic program (QCQP), relax it to a semidefinite program (SDP), and solve it efficiently using Burer–Monteiro (BM) factorization together with a Riemannian staircase, following the certifiable optimization framework of XM and SE-Sync [20, 60].
II: Dynamic stage. After fixing the recovered camera trajectory, we transform dynamic observations into the world frame and model non-rigid motion with an EDG. Each dynamic point is warped by the rigid transformations of its neighboring EDG nodes, where denotes the rest position of node and the skinning weight. Let denote the corresponding depth-lifted observation in the world frame. The tracking loss is
We optimize the EDG node transformations with tracking loss, as-rigid-as-possible (ARAP) and temporal smoothness regularization. Unlike MoSca [35], which models deformation through relative transformations between frames, we express all node transformations relative to a fixed rest frame. With fixed EDG structure and skinning weights, this choice makes the tracking, ARAP, and temporal smoothness objectives quadratic in the node transformations. Eliminating node translations then yields a rotation-only QCQP that admits the same SDP-based certifiable optimization framework as the static stage. The recovered geometry then initializes dynamic Gaussian splatting for rendering.
In summary, XM4D makes the following contributions:
For static reconstruction, we build on the QCQP–SDP–BM framework of our conference version, XM [20], retaining its efficient optimization and global optimality certificates. We introduce a uniform sphere constraint that resolves the scale ambiguity without anchoring any individual frame.
For dynamic reconstruction, we design an EDG-based non-rigid reconstruction formulation that incorporates tracking, rigidity, and temporal smoothness into a structured QCQP. This formulation admits the same globally certifiable optimization framework.
We build XM4D, an accurate and efficient 4D reconstruction pipeline that combines certifiable geometric optimization with dynamic Gaussian splatting. Extensive experiments across seven benchmarks demonstrate leading camera accuracy, fast and scalable optimization, and the best average novel-view rendering performance on all three dynamic benchmarks (DyCheck, NVIDIA, and Syn4D).
Outline. The remainder of this paper is organized as follows. § 2 reviews related work. § 3 presents the static stage: the rotation-only QCQP and SDP relaxation, the BM solver with its optimality certificate, and the sphere constraint. § 4 extends this machinery to the EDG-based dynamic reconstruction and shows that it admits the same certifiable solver. § 6 describes the complete pipeline. § 7 evaluates camera accuracy, runtime, and novel-view rendering on seven benchmarks. Proofs and implementation details are given in the appendices. Visualizations and code are available on the project page:
https://computationalrobotics.seas.harvard.edu/XM4D/.
Related Work
Dynamic Scene Reconstruction
Existing dynamic scene reconstruction methods can be grouped by how they constrain the reconstruction. Rendering-based methods fit time-varying scene representations to images, while geometry-driven methods use measurements such as depth and point tracks to estimate structure and motion. Rendering-based methods represent dynamics using time-varying (volumetric) fields [36, 42], deformable canonical radiance fields and their higher-dimensional extensions [51, 52, 57], or moving Gaussians [44, 62, 72]. Many of them incorporate geometric regularization, and some jointly estimate camera motion [41]. However, recovering object or part level motion for downstream scene understanding generally requires additional motion estimation. Moreover, many methods rely on accurate camera poses, carefully designed initialization, and heavy per-scene optimization.
Geometry-based methods use geometric measurements and motion priors to constrain dynamic reconstruction. Earlier non-rigid structure-from-motion methods recover 3D shape from monocular point tracks using low-rank or subspace priors [19, 88], while RGB-D methods use depth observations with deformation graphs or volumetric models to track motion and reconstruct geometry [9, 24, 48]. More recently, MoSca [35] lifts learned depth and point-track priors into a motion scaffold and anchors Gaussians to it for photometric reconstruction. Shape of Motion [68] represents scene motion using shared motion bases, with per-point weights combining the bases to recover persistent trajectories from monocular depth and long-range 2D tracks. XM4D also uses explicit local transformations as an EDG. Unlike them, XM4D estimates camera poses and temporally coupled graph transformations from depth and tracks with a certifiable geometric solver, making local rigid motions explicit without involving Gaussians.
Learning-Assisted Geometric Reconstruction
Learned geometric priors have become important components of reconstruction pipelines. Monocular depth models [5, 23, 38, 55, 79] predict dense scene depth, while point trackers [16, 30, 32, 73, 85] establish correspondences across frames. Together, depth and correspondence estimates provide complementary constraints for geometric reconstruction: depth informs scene structure, while tracks connect observations over time.
Recent feed-forward methods predict richer 3D scene representations directly from images. DUSt3R [69] predicts pointmaps for image pairs and aligns them in a shared coordinate frame, and MUSt3R [13] extends it to many views, and VGGT [67] jointly predicts camera parameters, depth maps, pointmaps, and 3D point tracks. In XM4D, learned depth and tracks are measurements for an explicit geometric optimization.
Certifiable Geometric Optimization
Certifiable geometric optimization solves nonconvex estimation problems, such as bundle adjustment, while verifying global optimality [33, 78, 84]. Semidefinite relaxations provide such certificates for pose synchronization [60, 82], outlier-robust registration [77], and general polynomial formulations of robust estimation [76].
Our prior conference work, XM [20], introduced depth-assisted scaled bundle adjustment with a scalable solver based on BM factorization [11] and Riemannian optimization [8], which XM4D extends to dynamic scenes.
Static Reconstruction with XM
This section presents the static stage of XM4D. Section 3.1 reviews scaled bundle adjustment and its QCQP reformulation (Propositions 1–2); Section 3.2 relaxes the QCQP to a convex SDP (Proposition 3); Section 3.3 solves the SDP with certified global optimality via the Riemannian staircase method (Algorithm 1). Section 3.4 highlights our main contribution compared to XM [20]: a uniform sphere constraint on the depth scales that replaces the first-frame anchor.
Bundle Adjustment with Learned Depth
The static observations originate from 2D correspondences, obtained by point trackers [16, 30, 85] or by classical keypoint detection and matching [43, 61], and denotes the normalized image coordinate of static point in frame obtained with the camera intrinsics, which are either calibrated or estimated by a foundation model [38, 67]. Each observation is lifted to a noisy 3D measurement using its predicted depth ,
which constitutes the depth-lifted observation of Eq. (SBA).
The translations and landmarks in Eq. (SBA) are unconstrained variables and thus can be eliminated analytically to get a reduced scaled-rotation-only problem.
Proposition 1 (Scaled-Rotation-Only Formulation, adapted from [20]). Problem (SBA) is equivalent to
where , , is a data matrix built from the lifted measurements through the Moore–Penrose pseudo-inverse of the view-graph Laplacian. In particular, the optimal values coincide: , where denotes the optimal value of (SBA). The closed form of , the proof, and the recovery of the optimal translations and landmarks are given in Appendix § 11.§ 1.
The objective of (SRO) vanishes as , a collapse of the global scale. XM excludes it by anchoring the first frame; we instead exclude it with the uniform sphere constraint motivated in § 3.4.
By defining the set of scaled orthogonal matrices through the quadratic constraint
which contains the scaled rotation of each frame, (SRO) reduces to the following QCQP.
Proposition 2 (QCQP Formulation, adapted from [20]). Consider the QCQP
and let be a global optimizer. If for all , then is a global minimizer of (SRO) under the same sphere constraint. The proof is given in Appendix § 11.§ 2.
Otherwise, rotations in SO(3) are recovered by projecting each block onto the scaled rotations through its singular value decomposition (Appendix § 12.§ 1).
Convex SDP Relaxation
Since local solvers for the QCQP may stall in suboptimal minima, we use a semidefinite relaxation [75 Chapter 3]. The basic idea is fairly simple: creating a matrix variable
which is quadratic in the original variable , makes the objective of (QCQP) linear in , namely . The convex relaxation then enforces the matrix variable through convex constraints. Partition into blocks with . Since each ,
All diagonal blocks play identical roles, and the sphere constraint becomes the linear equality . Ignoring the rank-3 constraint on yields Shor’s SDP relaxation.
Proposition 3 (SDP Relaxation, adapted from [20]). The semidefinite program
where each scalar is not a separate variable but the common diagonal entry of the block , is a convex relaxation of (QCQP), i.e., . If a global minimizer has , then it factorizes as with , unique up to multiplication by , and is a global optimizer of (QCQP). In this case, the relaxation is said to be tight or exact (proof in Appendix § 11.§ 3).
In practice the relaxation is empirically tight [20]; when a minimizer has rank above three, it is projected to rank three through its three largest eigenpairs, a process known as rounding (Appendix § 12.§ 1).
Burer–Monteiro Optimization
Although (SDP) is convex, explicitly storing and optimizing the matrix becomes prohibitive for long videos with a large : interior-point solvers already struggle beyond a few thousand frames [20]. When the relaxation is tight, however, the optimal has rank three (Proposition 3), so its effective dimension is only . BM factorization [11] exploits this low-rank structure: for a chosen rank , we substitute
with blocks into (SDP), which yields the rank- BM problem
At first glance, (BM) looks like a nonconvex constrained problem, but its constraints define a smooth manifold.
Proposition 4 (BM Riemannian Optimization, adapted from [20]). Problem (BM) is equivalent to the unconstrained Riemannian optimization problem
over the product manifold
where is the radius- sphere, is the Stiefel manifold, and . The proof is given in Appendix § 11.§ 4.
We solve (7) with the Riemannian trust-region method with truncated conjugate gradients (RTR-TCG) [7], using the C++/CUDA implementation from XM [20].
Three observations are in order [11, 75]. First, holds by construction, so the PSD constraint disappears. Second, at , problem (BM) is exactly (QCQP) and hence nonconvex. Third, increasing the rank cannot increase the optimal objective value, since any feasible rank- factor remains feasible at rank by appending a zero row; hence
with equality as soon as , the minimum rank among optimal solutions of (SDP) [11, 75]. We emphasize that (9) is a statement about global minima: each is the optimal value of the nonconvex problem (BM), which a local Riemannian solver alone is not guaranteed to attain. Attaining it—and thereby —is precisely the role of the Riemannian staircase [8, 60] developed below.
Global optimality certification. Write (SDP) in standard primal form,
The associated dual standard SDP is
Since Slater’s condition holds ( is feasible for (SDP)), strong duality holds between (10) and (11) [71].
Theorem 5 (Dual Optimality Certificate, adapted from [20]). Let be a locally optimal solution of (BM) with for all . Then there exists a unique multiplier satisfying the first-order condition
If, furthermore, , then is a global optimizer of (BM), and are optimal for the primal–dual pair (10)–(11). The proof is given in Appendix B.5.
Theorem 5 gives a simple certification recipe: solve the linear system (12) for (recall is linear in ), form , and check its positive semidefiniteness. If the check fails, the next theorem provides an escape route.
Theorem 6 (Negative-Curvature Direction, adapted from [20]). Let be a local minimizer of (BM) with multiplier , and suppose . Let be an eigenvector of associated with a negative eigenvalue. Then, for the rank- problem,
along the feasible retraction curve through in the direction , the objective satisfies and . The proof is given in Appendix B.6.
Since the objective decreases to second order along , the halving line search in Algorithm 1 terminates with a strictly lower objective, and the optimization escapes the rank- local minimizer by climbing one step of the rank “staircase”.
Theorem 7 (Global Convergence, adapted from [20]). Algorithm 1 returns a globally optimal solution of the SDP pair (10)–(11), regardless of the initialization (proof in Appendix B.7).
Theorem 7 makes the approach initialization-free. In practice, Algorithm 1 terminates at or 4 [20], so the worst case is never approached and the scalability advantage of the low-rank factorization is retained.
Anchoring via a Uniform Sphere Constraint
As discussed in § 3.1, the original XM formulation removes the scale and rotation ambiguities of (SBA) by anchoring the first frame (, , ). This pins the reconstruction to the first depth prediction, so the solver overfits to it when the first-frame observations are noisy, and XM’s scale-regularization adds a hyperparameter that must be tuned for each video. We instead fix the global scale with a constraint rather than a penalty: the uniform Algorithm 1: Riemannian Staircase
Input: data matrix
Output: certified optimizer of (BM)
while true do
\hfill // local minimizer of (BM)
solve (12) for and form ;
if then
\hspace{1em}return \hfill // certified global optimum
end
eigenvector of the most negative eigenvalue of ;
while do
\hspace{1em}
end
end
sphere constraint
excludes the collapse toward zero while treating every frame symmetrically. It is quadratic in the scales, so (QCQP) remains a QCQP, and after relaxation it is the trace constraint already in (SDP) and (BM).
Suboptimality gap. Since the staircase runs with finite tolerances, we report the relative gap
where is the objective of the solution rounded to (SRO) under the sphere constraint (Appendix § 12.§ 1), and are defined in (10)–(11b), and solves (12). By Theorem 11 (Appendix § 12), , so certifies that is numerically globally optimal for (SRO). Table 6 reports per dataset: all static solves satisfy , where for the partial sphere constraint uses the trace-restricted bound of Appendix § 12.
| Dataset | Static | Dynamic (EDG) | ||||
| scenes | median | scenes | median | |||
| Sintel | 23 | 23/23 | – | – | – | |
| TUM | 6 | 6/6 | – | – | – | |
| Replica | 8 | 8/8 | – | – | – | |
| BAL | 3 | 3/3 | – | – | – | |
| IMC | 6 | 6/6 | – | – | – | |
| DyCheck | 7 | 7/7 | 7 | 700/700 | ||
| NVIDIA | 7 | 7/7 | 7 | 511/511 | ||
| Syn4D | 8 | 8/8 | 8 | 800/800 |
Table 6. Median relative suboptimality of (21) over every scene of each dataset. counts solves with . When is PSD and the solution has rank three with all blocks of positive determinant, the solve is certified optimal for the original problem, and we report , the numerical resolution of the eigenvalue solve. Otherwise, the static uses the trace-restricted bound of Appendix § 12. For the dynamic solver, each entry is one segment solve (its final certificate), and the median and are over all segment solves.
Summary
The static stage lifts 2D point tracks to 3D with learned depth, eliminates translations and landmarks to obtain (SRO), and relaxes it through (QCQP) to the convex program (SDP), solved by the Riemannian staircase with a certificate of global optimality and a suboptimality gap (15). Unlike XM, a sphere constraint fixes the global scale without anchoring any frame. The recovered camera poses define the world frame in which dynamic tracks are lifted for the non-rigid stage.
Dynamic Reconstruction with XM
We extend the globally certifiable optimization framework of § 3 to the dynamic part of the view graph. § 5 introduces the EDG motion model and dynamic objective; § 5.1 analytically eliminates the unconstrained variables to obtain a rotation-only QCQP, as in the static case; and § 5.2 describes how the XM solver of § 3.3 solves the resulting problem.
EDG Model and the Dynamic Objective
EDG Model and the Dynamic Objective
Dynamic motion is represented by an EDG [65], , consisting of sparse control nodes (Eq. (2)). Each node has a rest position and a rigid transformation describing its motion at frame . Dynamic points are attached to nearby nodes and deformed using linear blend skinning. In this section, the graph (its nodes, edges, rest positions, and skinning weights) is taken as given; § 6.3 describes how it is built from the lifted dynamic tracks.
We first describe the EDG model for a single temporal window . Within this window, we choose a reference frame , typically the frame with the most observed EDG nodes, and define the rest positions in that frame, which keeps the skinning offsets and ARAP constraints consistent. We anchor the node transformations at by
so that is the identity. This is a local reference-frame convention for one solve; it does not require a single rest frame for the entire video (§ 6.3).
Since the cameras and depth scales are fixed at their static-stage estimates, the lifted observations are constants, and the dynamic problem estimates the node transformations alone,
Here is the tracking loss ((2)), which asks the warped rest positions to match the lifted observations; the as-rigid-as-possible (ARAP) energy over the EDG edges,
asks the transformation of node to also explain the motion of its neighbor , and the temporal smoothness energy
discourages large inter-frame deformation.
Rotation-Only QCQP
Every residual in (DYN) is affine: the tracking residual is affine in for fixed skinning weights and rest positions, the ARAP residual is affine in with the constant offset , and the smoothness residuals are linear. The objective is thus quadratic in the stacked variables, and (DYN) is a QCQP whose only constraints are the orthogonality of the EDG blocks.
As in the static case, the node translations are unconstrained Euclidean variables and can be eliminated analytically, leaving a problem over the EDG rotations alone.
Proposition 8 (Rotation-Only Formulation of the Dynamic Problem). Under the rest-frame convention ((22)), Problem (DYN) is equivalent to
where , , , and are built from the lifted observations, the skinning weights, and the rest positions through the Moore–Penrose pseudo-inverse of a node-translation Laplacian. In particular, the optimal values coincide: , where denotes the optimal value of (DYN). The closed forms of , , and , and the proof, and the recovery of the node translations are in Appendix § 11.8.
Two differences from the static problem (SRO) are worth noting. First, a linear term appears: the ARAP and tracking residuals carry the constant offsets , , and , which survive the elimination. Second, every block is a unit-scale rotation, whose relaxed block constraint is rather than , so no sphere normalization is needed.
Homogenization. The XM solver operates on a homogeneous quadratic objective, so we absorb the linear term and the constant of (DYN-RO) into a single matrix by introducing one auxiliary rotation and defining
The auxiliary block adds no physical degree of freedom; after solving, the EDG rotations are taken relative to it.
Proposition 9 (Homogeneous Formulation of the Dynamic Problem). Problem (DYN-RO) is equivalent to
(DYN-H)
in the following sense: the optimal values coincide, ; if is optimal for (DYN-H), then is optimal for (DYN-RO); and if is optimal for (DYN-RO), then is optimal for (DYN-H) (proof in Appendix B.9).
Solving with the XM Solver
As in Proposition 2, we drop the determinant constraints of (DYN-H), replacing by . The resulting QCQP has orthogonal blocks only, so Section 3.3 carries over with unit-scale blocks, , and no sphere constraint.
Proposition 10 (Certifiable Optimization of the Dynamic Problem). The SDP
is a convex relaxation of (DYN-H). Its rank- BM factorization is equivalent to the unconstrained Riemannian problem
Theorems 5–7 still hold for (DYN-SDP)–(DYN-BM), so Algorithm 1 returns a certified global optimum of (DYN-SDP), and hence of (DYN-H) whenever the relaxation is tight and the recovered relative rotation blocks lie in . The proof is given in Appendix B.10.
Global optimality matters in practice. Figure 3 compares the certified solution with that of a local solver on the same input: the local solver converges to a local minimum in which some nodes are rotated incorrectly, whereas the certified solution recovers consistent node rotations.
Suboptimality gap. The gap (15) applies verbatim to (DYN-SDP), with replaced by , and now encoding , and replaced by (Appendix C). Every segment solve on DyCheck, NVIDIA, and Syn4D is certified (Table 6).
Summary. The dynamic stage models non-rigid motion within each temporal segment with an EDG whose transformations are expressed relative to a single rest frame, which makes the tracking, ARAP, and temporal smoothness objective (DYN) quadratic. Eliminating the node translations yields the rotation-only problem (DYN-RO), whose linear term is absorbed by one auxiliary rotation into the homogeneous problem (DYN-H). Its SDP relaxation is solved with the same BM staircase as the static stage, with a certificate of global optimality and a suboptimality gap, whereas a local solver can stall in a wrong local minimum.
The XM4D Pipeline
This section describes the end-to-end system that recovers both the static and the dynamic structure of the view graph in Figure 2. We present it for a monocular video, but the formulation only requires that each image carry a time tag, and an image collection without time tags reduces the pipeline to static 3D reconstruction. The stages are preprocessing and tracking (Section 5.1), the static stage (Section 5.2), EDG construction, segment solving and completion (Section 5.3), and the final photometric stage (Section 5.4).
Preprocessing: Depth, Intrinsics, Masks, and Tracks
Every frame is processed by a monocular depth foundation model, with Depth Anything 3 (DA3) [38] as the default, which provides the metric depths used in the lift (2) and, for uncalibrated videos, the camera intrinsics. Per-frame masks of the moving objects are obtained with SAM 3 [14], prompted either by a text description or by a few hand-selected keypoints on the dynamic object and then tracked across the video. Finally, both static and dynamic 2D correspondences are produced by point trackers, with TAPNext [85] as the default. The trackers are operated in an online fashion: as each frame arrives, new query points are sampled only at locations farther than a threshold from existing tracks. Unreliable observations are discarded (Appendix E). Optionally, dynamic tracks can be assembled into long-range trajectories for the EDG (Appendix E.5).
Static Stage: Certified Solve
The static stage solves (SBA) with the solver of Section 3, augmented with two robustness steps. First, we run XM [20]: the solver is applied twice, with an outlier filter in between that keeps only the observations with the smallest residuals under the first solution, so tracking failures or inaccurate depth predictions do not corrupt the final estimate.
Second, the result is refined with Ceres [2], warm-started at the certified XM outputs and minimizing the classical 2D reprojection error. Further implementation details of the static stage are collected in Appendix § 14.
Partial sphere constraint. In practice, some videos contain a small fraction of outlier frames—strong blur or large inter-frame motion, or depth predictions whose scale is far off—and forcing such frames onto the sphere lets them distort the global scale. We therefore use a mixed-manifold variant of the solver: frames are ranked by the trace of their diagonal block of , the top frames are constrained to the sphere of § 3.4, and the scales of the remaining frames are optimized freely. Formally, with the set of these frames, the sphere constraint becomes
i.e.,
in (SDP), which is still linear, so the SDP relaxation and the certificate of Theorem 5 carry over. The suboptimality gap (21) is adapted accordingly in Appendix § 12. In the same spirit, we optionally normalize the observation weights per frame: with the total weight of frame , the solver uses the raw weights , equalizes every frame’s influence , or applies a softer normalization . Table 1 compares eight such configurations; configuration (6) is our default.
EDG Construction, Segment Solve, and Completion
Starting from the lifted dynamic tracks of § 4, we first build a time-independent graph connecting tracks that remain spatially close over time. We then subsample this graph to obtain sparse EDG nodes and inherit its connectivity to define the EDG edges. Because no single frame observes all nodes in long videos, we solve the EDG motion independently on short overlapping segments using the certified solver of § 5.2, and fuse the segment solutions. Finally, missing EDG poses are filled by a completion step.
Track graph. The track graph records which tracks stay on the same part of the object. A single-frame distance cannot decide this, since points on different parts may pass close to each other in some frames. We therefore measure the distance of two tracks over their whole common lifetime. Let be the frames in which track is reliable, the frames in which tracks and are co-visible, and the pixel position of track in frame . We define the percentile distance of two tracks as the -th percentile of their per-frame distance over co-visible frames,
in the world frame and in the image, respectively. We write if
Photometric Stage
that is, the tracks overlap in time and stay close in 3D or 2D, and connect them in ; the thresholds and are proportional to the median nearest-neighbor distance of the tracks in 3D and in the image. Each track keeps at most edges with the smallest .
EDG nodes and edges. The same criterion selects the EDG nodes. Tracks are ranked by their length , and the longest remaining track becomes a node and suppresses every remaining track that stays on the same part of the object during their common lifetime, i.e., with in (30), with thresholds and overlap fraction that differ from those of the track graph, until every track is either a node or suppressed by one. If this yields more than nodes, of them are kept by farthest-point sampling under the same percentile distance. Two nodes , are connected if they are within hops on the track graph, which keeps independently moving parts apart. A node left with fewer than three edges is connected to its nearest nodes that are reachable on the track graph.
Skinning. Every other dynamic track is attached to nodes chosen among the nodes within two hops of it on the track graph; an isolated track with no node within two hops falls back to its nearest node in percentile distance. The skinning weights are the normalized inverse distances from the track to its nodes at the rest frame. The graph and weights are fixed before solving, so the certificate covers the EDG motion given this graph.
Segment solve and fusion. For a long sequence, we apply the single-window formulation of § 4 to overlapping windows . Each window has its own reference frame , rest positions, and locally estimated transformations obtained by solving (DYN) to certified global optimality with Proposition 10. We fuse the estimates over overlapping windows by averaging the estimated world-space node positions for each frame. We recover globally consistent rotations by synchronizing the relative rotations estimated within each window; see Appendix § 14.§ 4.
Completion. Node-frame pairs that remain empty after fusion are filled by a local version of (DYN): over all nodes and frames, we minimize the same objective, with a robust 2D reprojection error as the tracking term, plus an initialization loss that keeps the solution close to the fused table, using a local solver initialized from it. This yields a complete trajectory for every node and frame.
Photometric Stage
The recovered geometry initializes a dynamic Gaussian-splatting representation [31]. We use gsplat [80] as the photometric training engine and add an EDG wrapper on top of it, which warps every dynamic Gaussian to frame by linear blend skinning with the EDG transformations. During training, the rest positions of the dynamic Gaussians are optimized together with the EDG transformations , initialized from the certified solve, under a photometric loss and depth supervision.
Experiments
We compare XM4D with Structure-from-Motion (SfM) and 4D reconstruction baselines on seven static and dynamic benchmarks in camera accuracy, runtime, and novel-view rendering, and extend the qualitative evaluation to additional videos (Figure 4); Figure 5 summarizes the quantitative comparison.
Figure 4. Qualitative results of XM4D. Sequences from DyCheck [18], NVIDIA [81], Syn4D, DAVIS [54], Sintel [12], EgoDexter [46], a Boston Dynamics video [6], and a generated video from Google Flow. For each sequence, from left to right: an input frame, the reconstructed static and dynamic structure, the 3D trajectories of the dynamic points over the video, a rendered novel view, and the rendered depth.
Figure 5. Performance overview. Each axis is the square-root-compressed ratio to the best method, averaged over the datasets of Table 2–Table 5. Open polygons mark axes a method does not report. Farther out is better. XM4D performs best overall, leading on four of the five axes.
Setup
Datasets. We use seven benchmarks. On four of them—IMC [28], Replica [63, 89], TUM [64], and Sintel [12]—we evaluate camera accuracy only. IMC and Replica are static, while TUM and Sintel contain moving objects. On the remaining three—Syn4D, NVIDIA [81], and the DyCheck iPhone dataset [18]—we additionally run the EDG solve and the photometric reconstruction of § 6. Syn4D is a synthetic benchmark that we assemble from Replica [63] and RoboCasa [47]. Each dataset is described in Appendix § 13. The solver-configuration study in Table 1 additionally uses three problems from BAL [1].
| Dataset | Metric | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 |
| BAL | ATE-T | 2.243 | 2.194 | 2.163 | 2.185 | 2.218 | 2.155 | 2.181 | 2.283 |
| RPE-T | 4.003 | 3.942 | 3.934 | 3.936 | 3.935 | 3.957 | 3.949 | 3.979 | |
| RPE-R | 1.674 | 1.661 | 1.650 | 1.667 | 1.655 | 1.660 | 1.712 | 1.690 | |
| Sintel | ATE-T | 53 | 53 | 54 | 51 | 52 | 56 | 49 | 74 |
| RPE-T | 72 | 71 | 75 | 69 | 71 | 74 | 69 | 105 | |
| RPE-R | 76 | 71 | 70 | 66 | 67 | 52 | 56 | 121 | |
| Sintel (Ceres) | ATE-T | 29 | 30 | 29 | 16 | 18 | 18 | 29 | 30 |
| RPE-T | 35 | 37 | 37 | 25 | 28 | 26 | 36 | 43 | |
| RPE-R | 62 | 63 | 61 | 48 | 56 | 41 | 49 | 107 | |
| TUM (Ceres) | ATE-T | 9.45 | 8.80 | 10.48 | 8.70 | 9.21 | 5.72 | 7.33 | 11.42 |
| RPE-T | 12.85 | 12.22 | 14.51 | 12.42 | 12.99 | 8.02 | 10.04 | 16.23 | |
| RPE-R | 6.63 | 6.18 | 7.69 | 6.28 | 6.64 | 5.98 | 5.87 | 7.05 |
Table 1. Mean camera accuracy () of eight solver configurations of Section 5.2, each determined by the sphere fraction and the weight normalization: (1)–(5) raw weights at , 0.3, 0.5, 0.9, and 1; (6) with the per-frame normalization ; (7) with the softer normalization ; and (8) the original XM solver [20] with raw weights. Configuration 6, our default, achieves the best or second-best accuracy in eight of the twelve rows.
Baselines. For camera estimation, we compare against the classical SfM pipeline COLMAP [61], the feed-forward geometry model VGGT [67], and MegaSAM [37], which adapts DROID-SLAM [66] for camera pose and depth estimation in casual dynamic videos. For dynamic reconstruction, we compare against the geometry-based methods MoSca [35] and Shape of Motion [68]. For novel-view rendering, we additionally compare against rendering-based methods that use time-varying fields (T-NeRF [18], NSFF [36], CTNeRF [45], DyBluRF [10], and RoDynRF [41]), deformable canonical radiance fields (Nerfies [51], HyperNeRF [52]), or moving Gaussians (Gaussian Marbles [62]), and against DynPoint [86] and PGDVS [83], which render from depth-warped points.
Hardware. All experiments run on a workstation with an AMD Ryzen Threadripper PRO 7985WX CPU (64 cores, 128 threads), 256 GB of RAM, and two 48 GB NVIDIA RTX 6000 Ada GPUs, under Ubuntu 22.04 with CUDA 12.8. Every method, including ours, runs on a single GPU.
Metrics. We report the absolute trajectory error in translation (ATE-T) and the relative pose error in translation and rotation (RPE-T, RPE-R). The estimated camera centers are first aligned to the ground-truth centers by a similarity transformation estimated robustly with TEASER++ [77]. The translation errors are normalized by , where is the mean distance of the ground-truth centers from their centroid:
where contains all frame pairs, and RPE-R is the mean angle, in radians, of the relative rotation error over the same pairs. The constant 1, in ground-truth units, prevents division by a near-zero when the camera barely moves. For novel-view synthesis on DyCheck, NVIDIA, and Syn4D we report mPSNR, mSSIM, and mLPIPS or their alternates. Following MoSca [35], we refine each test camera photometrically against its test image with the scene frozen.
Camera Accuracy
Table 2 summarizes camera accuracy across all seven benchmarks. XM4D is the most accurate method on Replica and TUM, and in translation on Sintel; on Syn4D, NVIDIA, and DyCheck it is close to the best method, MegaSAM; and on the unordered image collections of IMC, XM4D is outperformed only by the dedicated SfM pipeline COLMAP. Figure 6 shows the trajectories on one scene per benchmark.
| Dataset | Metric | COLMAP | VGGT | MegaSAM | MoSca | Ours |
| IMC | ATE-T | 0.003 | 0.075 | 0.871 | – | 0.047 |
| RPE-T | 0.005 | 0.121 | 1.221 | – | 0.074 | |
| RPE-R | 0.002 | 0.033 | 0.703 | – | 0.044 | |
| Replica | ATE-T | 0.009 | 0.014 | 0.067 | 0.060 | 0.002 |
| RPE-T | 0.012 | 0.019 | 0.087 | 0.072 | 0.002 | |
| RPE-R | 0.015 | 0.005 | 0.574 | 0.007 | 0.001 | |
| TUM | ATE-T | 0.006 | 0.008 | 0.007 | 0.009 | 0.006 |
| RPE-T | 0.009 | 0.012 | 0.009 | 0.013 | 0.008 | |
| RPE-R | 0.006 | 0.006 | 0.006 | 0.007 | 0.006 | |
| Sintel | ATE-T | 0.359 | 0.066 | 0.049 | 0.067 | 0.018 |
| RPE-T | 0.462 | 0.095 | 0.058 | 0.094 | 0.026 | |
| RPE-R | 0.081 | 0.057 | 0.019 | 0.056 | 0.041 | |
| Syn4D | ATE-T | 0.035 | 0.009 | 0.001 | 0.022 | 0.002 |
| RPE-T | 0.060 | 0.011 | 0.001 | 0.031 | 0.002 | |
| RPE-R | 0.022 | 0.012 | 0.001 | 0.057 | 0.002 | |
| NVIDIA | ATE-T | 0.150 | 0.111 | 0.024 | 0.174 | 0.038 |
| RPE-T | 0.206 | 0.168 | 0.036 | 0.259 | 0.056 | |
| RPE-R | 0.010 | 0.010 | 0.005 | 0.107 | 0.005 | |
| DyCheck | ATE-T | 0.006 | 0.019 | 0.005 | 0.048 | 0.008 |
| RPE-T | 0.008 | 0.026 | 0.007 | 0.065 | 0.011 | |
| RPE-R | 0.009 | 0.035 | 0.007 | 0.024 | 0.012 |
Table 2. Camera accuracy averaged per dataset. MoSca [35] crashes on IMC [28] (–), whose unordered photo collections are not video data. VGGT [67] is run on subsampled frames on DyCheck [18] (§ 7.3).
Figure 6. Estimated camera trajectories. Estimated camera poses (blue) after Sim(3) alignment to the ground truth (red), shown for one scene per benchmark. IMC [28] consists of unordered images, whereas all other benchmarks contain temporally ordered sequences.
COLMAP is the most accurate on IMC, the unordered setting it is designed for, and remains competitive on Replica and TUM, but degrades sharply on Sintel, Syn4D, and NVIDIA, where its ATE-T is – that of XM4D. The feed-forward VGGT is roughly three times less accurate in translation. MegaSAM, designed for dynamic videos, is the most accurate on Syn4D, NVIDIA, and DyCheck, with XM4D close behind, but is less accurate on Replica and TUM and in translation on Sintel and fails on IMC, whose images lack the temporal continuity its optical-flow front end relies on. MoSca is – less accurate than XM4D in ATE-T on the dynamic benchmarks and also fails on IMC; we attribute the gap to the XM solver, which recovers the cameras by a certifiably optimal SDP on static tracks alone, so errors in the deformation model cannot propagate into the trajectory. Overall, XM4D ranks first on Replica and TUM and in translation on Sintel, second on Syn4D, NVIDIA, and in translation on IMC.
Runtime
Static stage. Table 3 reports the runtime of camera estimation and, where available, its split into preprocessing (depth, masks, and tracking) and the solver. End to end, XM4D is faster than MoSca on every benchmark and up to faster than COLMAP on the long sequences, while the feed-forward VGGT is the fastest overall, as expected for a single network pass. Against MegaSAM the totals are mixed: XM4D is faster on IMC, NVIDIA, and DyCheck, and slower on Replica, TUM, Sintel, and Syn4D.
| Dataset | Stage | COLMAP | VGGT | MegaSAM | MoSca | Ours |
| IMC | Total | 227.7 | 38.7 | 136.8 | – | 23.2 |
| Replica | Total | 125.4 | 54.4 | 83.5 | 1054.4 | 115.1 |
| Preprocess | – | – | 49.6 | 888.5 | 100.5 | |
| Solver | – | – | 33.9 | 165.9 | 14.6 | |
| TUM | Total | 1743.7 | 52.8 | 79.5 | 403.8 | 83.7 |
| Preprocess | – | – | 50.8 | 288.3 | 73.7 | |
| Solver | – | – | 28.5 | 115.3 | 10.0 | |
| Sintel | Total | 206.0 | 29.3 | 48.4 | 410.2 | 62.0 |
| Preprocess | – | – | 34.1 | 363.2 | 47.2 | |
| Solver | – | – | 14.3 | 47.0 | 14.8 | |
| Syn4D | Total | 605.0 | 55.0 | 82.6 | 1729.3 | 110.3 |
| Preprocess | – | – | 52.0 | 724.6 | 88.4 | |
| Solver | – | – | 30.6 | 1004.6 | 21.9 | |
| NVIDIA | Total | 25 | 17 | 35 | 111 | 28 |
| Preprocess | – | – | 28 | 92 | 25 | |
| Solver | – | – | 7 | 18 | 3 | |
| DyCheck | Total | 843 | 38 | 316 | 942 | 96 |
| Preprocess | 211 | – | 149 | 555 | 78 | |
| Solver | 631 | – | 167 | 387 | 18 | |
| Replica room0 | 36.4 | 37.6 | 89.6 | 468.5 | 73.7 | |
| 225.8 | OOM | 149.0 | 892.6 | 131.3 | ||
| 1451.9 | – | 345.0 | 2140.1 | 383.5 | ||
| 3230.0 | – | 568.6 | err | 862.3 | ||
| >1h | – | err | – | 1656.2 | ||
| – | – | – | – | 3322.4 |
Table 3. Runtime of camera estimation. Top: runtime per scene in seconds, averaged per dataset and split into preprocessing (depth, masks, tracking) and the solver. Bottom: wall-clock time in seconds on Replica [63] room0 as the number of input frames grows (contiguous prefix); a run is invalid if it exceeds the GPU memory (OOM), runs longer than 1 h (>1h), or crashes (err). Best in bold, second best underlined.
The preprocessing relies on foundation models and will speed up as they improve, so we focus on the solver, where XM4D is the fastest on every benchmark but Sintel.
Dynamic stage. Table 5 reports the runtime of the full dynamic reconstruction against MoSca and Shape of Motion.
| Dataset | Method | Preprocess | Optimization | Total |
| DyCheck | Shape of Motion | 230.0 | 282.4 | 512.4 |
| DyCheck | MoSca | 9.2 | 31.3 | 40.5 |
| DyCheck | Ours | 8.8 | 25.8 | 34.6 |
| NVIDIA | Shape of Motion | 0.6 | 48.3 | 48.8 |
| NVIDIA | MoSca | 1.5 | 32.1 | 33.6 |
| NVIDIA | Ours | 1.5 | 20.4 | 21.9 |
| Syn4D | Shape of Motion | 20.0 | 50.9 | 71.0 |
| Syn4D | MoSca | 12.1 | 32.3 | 44.5 |
| Syn4D | Ours | 6.6 | 10.4 | 16.9 |
Table 5. Per-scene runtime of the full dynamic reconstruction in minutes (averages over the scenes of each dataset), split into preprocessing (depth, masks, static and dynamic tracking) and optimization (camera estimation, the motion model, and photometric training). Best in bold.
XM4D is the fastest end to end and in optimization on all three benchmarks. Shape of Motion is the slowest, mainly because its dense point tracking takes about four hours per scene on DyCheck.
Sequence length. Table 3 (bottom) reports the runtime on Replica room0 versus the number of frames . VGGT runs out of GPU memory at , MoSca fails beyond , and beyond COLMAP exceeds one hour and MegaSAM its 1024-frame buffer, whereas XM4D completes every length, reaching in 55 minutes. VGGT’s limit is also why it runs on subsampled frames on DyCheck, whose sequences exceed 100 frames.
Novel-View Rendering
Table 4 and Figure 7 evaluate novel-view synthesis on DyCheck, NVIDIA, and Syn4D. XM4D attains the best average score on every metric of all three datasets. This holds although most baselines are given the ground-truth camera poses (all except MoSca and RoDynRF, and Shape of Motion on Syn4D), and Gaussian Marbles on Syn4D additionally the ground-truth depth, whereas XM4D estimates all cameras. The margins are largest on Syn4D, where XM4D improves PSNR by more than 2 dB over the best baseline, and on DyCheck, where it has the best mSSIM and mLPIPS on all seven scenes and the best mPSNR on three. As Figure 7 shows, the baselines lose mainly because inaccurate geometry misaligns their renderings with the test view and blurs the dynamic parts. Since the recovered dynamic motion is hard to compare directly across methods, we further evaluate the geometry on Syn4D, whose moving objects have ground-truth geometry, through the Chamfer distance of the dynamic object and the error of the rendered depth (Table 7); XM4D is the most accurate on every scene.
| DyCheck | mPSNR ↑ | mSSIM ↑ | mLPIPS ↓ |
| T-NeRF (w) | 16.96 | 0.577 | 0.379 |
| NSFF (w) | 15.46 | 0.551 | 0.396 |
| Nerfies (w) | 16.45 | 0.570 | 0.339 |
| HyperNeRF (w) | 16.81 | 0.569 | 0.332 |
| DynPoint (w) | 16.89 | 0.573 | – |
| PGDVS (w) | 15.88 | 0.548 | 0.340 |
| DyBluRF (w) | 17.37 | 0.591 | 0.373 |
| CTNeRF (w) | 17.69 | 0.531 | – |
| Shape of Motion (w) | 17.32 | 0.598 | 0.296 |
| Gaussian Marbles (w) | 16.72 | – | 0.413 |
| RoDynRF (w/o) | 17.10 | 0.534 | 0.517 |
| MoSca (w/o) | 18.91 | 0.677 | 0.284 |
| Ours (w/o) | 19.51 | 0.725 | 0.163 |
| NVIDIA | PSNR ↑ | LPIPS ↓ | |
| NSFF (w) | 24.33 | 0.199 | |
| HyperNeRF (w) | 17.60 | 0.367 | |
| DynPoint (w) | 26.53 | 0.068 | |
| CTNeRF (w) | 26.13 | 0.082 | |
| RoDynRF (w) | 25.89 | 0.067 | |
| RoDynRF (w/o) | 25.38 | 0.079 | |
| MoSca (w/o) | 26.54 | 0.073 | |
| Ours (w/o) | 27.01 | 0.063 | |
| Syn4D | mPSNR ↑ | mLPIPS ↓ | |
| Gaussian Marbles (w) | 19.48 | 0.348 | |
| Shape of Motion (w/o) | 16.72 | 0.418 | |
| MoSca (w/o) | 19.49 | 0.235 | |
| Ours (w/o) | 21.73 | 0.148 |
Table 4. Novel-view rendering averaged over the scenes of each dataset; (w)/(w/o): with/without ground-truth camera poses. XM4D is the best on every metric although it estimates its camera poses.

Figure 7. Novel-view rendering on DyCheck [18], NVIDIA [81], and Syn4D. For each method, the rendered test view is shown with its error map with respect to the ground truth (brighter is larger error). (a) DyCheck: from top to bottom, block, paper-windmill, spin, and wheel, with error maps displayed only inside the co-visibility mask. The other methods score lower mainly because inaccurate 3D geometry misaligns their renderings with the test view, and because they blur the dynamic parts. (b) NVIDIA: from top to bottom, Jumping, Playground, Skating, and Truck, with error maps over the full frame. (c) Syn4D: from top to bottom, two Bear [4] and two RoboCasa [47] sequences, with the rendered depth also shown.
| Metric | Method | bear1 | bear2 | bear3 | bear4 | casa1 | casa2 | casa3 | casa4 | Mean |
| Chamfer | Ours | 6.1 | 5.4 | 6.4 | 4.6 | 11.2 | 9.2 | 7.6 | 8.6 | 7.4 |
| MoSca | 27.3 | 29.1 | 75.2 | 45.6 | 112.0 | 125.1 | 50.2 | 60.4 | 65.6 | |
| SoM | 34.4 | 214.7 | 90.8 | 172.4 | 38.3 | 33.3 | 69.3 | 2429.9 | 385.4 | |
| Train depth | Ours | 0.012 | 0.012 | 0.026 | 0.015 | 0.077 | 0.029 | 0.054 | 0.045 | 0.034 |
| MoSca | 0.031 | 0.029 | 0.035 | 0.029 | 0.123 | 0.049 | 0.081 | 0.069 | 0.056 | |
| SoM | 0.102 | 0.096 | 0.104 | 0.096 | 0.193 | 0.082 | 0.158 | 0.139 | 0.121 | |
| DA | 0.204 | 0.208 | 0.260 | 0.240 | 0.469 | 0.237 | 0.315 | 0.220 | 0.269 | |
| Novel depth | Ours | 0.050 | 0.094 | 0.094 | 0.085 | 0.101 | 0.038 | 0.055 | 0.053 | 0.071 |
| MoSca | 0.296 | 0.329 | 0.099 | 0.292 | 0.171 | 0.182 | 0.121 | 0.087 | 0.197 | |
| SoM | 0.321 | 0.322 | 0.273 | 0.334 | 0.253 | 0.143 | 0.188 | 0.210 | 0.255 | |
| DA | 0.210 | 0.286 | 0.368 | 0.296 | 0.458 | 0.255 | 0.357 | 0.262 | 0.312 |
Table 7. Geometry accuracy on Syn4D per scene: Chamfer distance () between the reconstructed and the ground-truth dynamic object, and relative error of the rendered depth in the training and novel views; lower is better. All methods are scored with the same protocol: the Chamfer distance follows the camera-based similarity alignment with one global scale per scene chosen to minimize it, and the depth uses one median scale per scene, fitted on the training views and reused for the novel views. SoM is Shape of Motion [68], and DA renders the Depth Anything 3 [38] depth lifted with its cameras.
Qualitative Results
Figure 4 shows XM4D on dynamic videos beyond the benchmarks: in-the-wild videos from DAVIS [54], Sintel [12], egocentric hand–object interaction videos from EgoDexter [46], a Boston Dynamics humanoid robot performing a flip [6], and a generated video of a walking lion from Google Flow. These videos span real, synthetic, and generated footage, third-person and egocentric viewpoints, and the motion of animals, people, hands, and a robot; XM4D handles all of them, demonstrating the robustness of the pipeline.
For videos of human body and hands, we build the EDG from keypoints instead of subsampling the track graph. For hands, WiLoR [56] predicts a MANO [59] hand in every frame; we keep only its 21 3D joints and connect them by the kinematic skeleton of the hand. For bodies, RTMPose [26] detects the 17 COCO [39] keypoints in 2D within the
bounding box of the person’s mask.
Because the EDG represents motion with explicit transformations, XM4D recovers the part-level rigid motion of objects without an extra motion-estimation step. Such part-level articulation is what simulation-ready digital twins of articulated objects require, especially in robotics [22, 27, 34, 58, 87]. Figure 8 shows a refrigerator whose four doors are opened in turn: the reconstruction yields the opening angle and displacement of every door over time and its hinge axis.
Figure 8. Articulated object reconstruction with XM4D. A person opens the four doors of a refrigerator in turn in a casual monocular video. (a) Input frames. (b) The four moving doors, color-coded. (c) Rendered views of the reconstruction at the same time instants. (d) Rotation angle and translation of each door over the video. (e) Reconstructed geometry with the four doors and their estimated hinge axes.
Summary
XM4D gives the best overall results for three reasons. First, its certified cameras make it the most accurate on Replica and TUM and in translation on Sintel, and close to the best elsewhere. Second, its solver is the fastest on nearly every benchmark, making XM4D the fastest 4D reconstruction end to end, and scales to 2000 frames. Third, its cameras and EDG motion give the photometric stage an accurate scaffold, yielding the best novel views on all three dynamic benchmarks, even against methods with ground-truth poses.
Conclusions
We presented XM4D, a 4D reconstruction framework that recovers camera motion and non-rigid deformation through certifiably globally optimal geometric optimization. Separating the two stages turns both into QCQPs over (scaled) rotations: the static stage estimates camera poses, depth scales, and static landmarks, and the dynamic stage estimates the motion of an embedded deformation graph. Both QCQPs are relaxed to SDPs and solved by a GPU solver that returns a certificate of global optimality. Across seven benchmarks, XM4D estimates accurate cameras on both dynamic videos and unordered photo collections, scales to thousands of frames, and initializes dynamic Gaussian splatting for state-of-the-art novel-view rendering.
Limitations. The certificate covers the optimization, not its inputs: XM4D relies on the predicted monocular depth and point tracks, and errors that survive the filters of Appendix § 14.§ 2 bias the certified solution. The solver stores the data matrices densely, so its memory grows quadratically with the number of frames; exploiting their sparsity [50] is a natural next step. Finally, the EDG structure and skinning weights are fixed before the solve; estimating the graph jointly with its motion is future work.
Acknowledgements
This work was partially funded by Office of Naval Research grant N00014-25-1-2322.
Additional Tables
Table 6 reports the suboptimality gaps per dataset. Table 7 compares the reconstructed geometry on Syn4D with MoSca [35], Shape of Motion [68], and Depth Anything 3 [38]. For the Chamfer distance, the dynamic object at frame is represented by the centers of the dynamic Gaussians at (for XM4D, warped by the EDG); 1000 points are drawn uniformly at random from them and from the ground-truth object point cloud, and the distance is the sum of the two mean squared nearest-neighbor distances, averaged over the 100 frames.
Table 6 Median relative suboptimality of (21) over every scene of each dataset. counts solves with . When is PSD and the solution has rank three with all blocks of positive determinant, the solve is certified optimal for the original problem, and we report , the numerical resolution of the eigenvalue solve. Otherwise, the static uses the trace-restricted bound of Appendix § 12. For the dynamic solver, each entry is one segment solve (its final certificate), and the median and are over all segment solves.
Table 7 Geometry accuracy on Syn4D per scene: Chamfer distance () between the reconstructed and the ground-truth dynamic object, and relative error of the rendered depth in the training and novel views; lower is better. All methods are scored with the same protocol: the Chamfer distance follows the camera-based similarity alignment with one global scale per scene chosen to minimize it, and the depth uses one median scale per scene, fitted on the training views and reused for the novel views. SoM is Shape of Motion [68], and DA renders the Depth Anything 3 [38] depth lifted with its cameras.
Proofs
Proof of Proposition 1
Proof. Let and denote standard basis vectors. All vectors are columns; collect the unknown translations and landmarks as the columns of the matrix
For each observation define
so that, with , each residual of Eq. (SBA) reads . The objective of Eq. (SBA) is therefore the quadratic form
with
By construction, is the weighted Laplacian of the bipartite view graph on the vertices with edge weights (cf. [20]): implies , and when is connected the nullspace of is exactly . Likewise, .
For fixed , is a convex quadratic in , bounded below by zero, so its minimizers are the solutions of the stationarity condition . This linear system is consistent: places the rows of in the range of . Its minimum-norm solution is
and every other solution differs by . Substituting (37) into (33) and using gives
the generalized Schur complement of in ; since the columns of lie in , . Minimizing both sides of (38) over proves the claimed equivalence, and (37) recovers the optimal translations and landmarks from .
Proof of Proposition 2
Writing a generic matrix as , membership in the set (5) of scaled orthogonal matrices is described by the quadratic equalities
which, together with the quadratic sphere constraint , are the constraints of (QCQP).
Proof. We first characterize the feasible set. If satisfies the quadratic equalities (39)–(40), then with . If , then and ; if , then . The quadratic constraints therefore describe exactly the set (5), including the zero matrix, and the feasible set of (QCQP) is . This set contains the feasible set of (SRO) under the sphere constraint: for and , each block lies in , and holds if and only if . The QCQP thus relaxes (SRO) in exactly two ways: reflections are admitted (SO(3) is relaxed to O(3)), and individual blocks may vanish. Since the objectives coincide on the common feasible set,
Now let be a global optimizer of (QCQP) with for all . The determinant condition rules out both relaxations at once: every block is nonzero, so with and , and
i.e., . Moreover, , so is feasible for (SRO) under the sphere constraint, which gives . Combined with (41), equality holds, and is a global minimizer of (SRO). Finally, since , is equally optimal for every , and by (37) the recovered translations and landmarks rotate with it; the normalization selects the representative that aligns the first camera with the identity.
Proof of Proposition 3
Proof. Relaxation. Let be feasible for (QCQP) and set . Then each diagonal block is by (5); and . Hence is feasible for (SDP) with the same objective value , and minimizing over gives .
Tightness. Let be a global minimizer of (SDP) with , and write its eigendecomposition as with . The matrix with rows satisfies . Partitioning into blocks, the block constraints give , so by (5), and the trace constraint gives . Thus is feasible for (QCQP), and
so equality holds and is a global optimizer of (QCQP).
Proof of Proposition 4
Proof. Let be feasible for (BM). Each block satisfies with ; set . If , then satisfies , i.e., ; if , then and any gives . The trace constraint reads , so . Conversely, every point of gives blocks with and , which is feasible for (BM) with . The map is therefore onto the feasible set of (BM), and the two problems share the objective , so they are equivalent.
Proof of Theorem 5
Proof. We first make the constraint matrices of (10) explicit. For each frame , the block constraint for some is equivalent to the five homogeneous equations , , where is zero except for its -th diagonal block, which equals
The form a basis of the traceless symmetric matrices. The trace constraint of (SDP) corresponds to with .
LICQ. The gradient of the -th constraint of the BM problem at is . Suppose for some coefficients , evaluated at . The matrix is supported on rows only, while contributes to those same rows; collecting the rows of block therefore gives
where . Since , the block has full column rank; multiplying (44) on the right by and using yields . Taking the trace gives because every is traceless, so ; linear independence of then forces for all . Hence the constraint gradients are linearly independent, and LICQ holds at .
First-order condition. By LICQ, there exists a unique Lagrange multiplier at which the Lagrangian of (BM) is stationary:
which is exactly (18).
Global optimality. Suppose in addition . Then is feasible for the dual (11), and is feasible for the primal (10). Their objective values coincide:
using primal feasibility in the first equality and (18) in the last. The duality gap is zero, so and are optimal for the primal–dual pair, and attains , hence it is a global optimizer of (BM) [11, 75].
Proof of Theorem 6
Proof of Theorem 7
Proof. In the worst case , every local optimizer of (BM) corresponds to a globally optimal solution of the SDP [29 Corollary 8]: if the local optimizer is rank-deficient, global optimality follows from [11 Proposition 4]; if it is full-rank, then (18) forces , which is positive semidefinite and again certifies global optimality.
Proof of Proposition 8
Proof. Let be the free node-frame pairs of (DYN), , and fix an ordering of , the one used in . Let denote the standard basis vector of the pair , set , and let be the Kronecker delta. Collect the unknown node translations as the columns of
in the same order as . Then, for every node , frame , and vector , the rest-frame convention (22) gives
so the fixed rest-frame blocks contribute constants only.
Residuals. By (48), every residual of (DYN) has the form
with weight equal to the coefficient of its term in (DYN). Using , the three terms give the following.
Tracking, one residual per with weight :
ARAP, one residual per edge and frame with weight (at the residual vanishes identically):
Temporal smoothness, for every node and with weight : writing
with the standard basis of , three rotation residuals () and one translation residual,
The node term of the implementation (Appendix § 14.§ 3) has the same form, with , , and , so the reduction below applies unchanged when it is included.
Quadratic form. Summing over all residuals, the objective of (DYN) is
with
Elimination. For fixed , is a convex quadratic in , bounded below by zero, so its minimizers solve the stationarity condition . This system is consistent: the rows of lie in . Its minimum-norm solution is
which is unique when , e.g., whenever , since every node’s temporal chain then reaches the fixed rest frame. Substituting (57) into (53) and using gives with
The matrix is the generalized Schur complement of in , and the columns of lie in , so . Minimizing both sides over proves the claimed equivalence, and (57) recovers the optimal node translations from .
Proof of Proposition 9
Proof. Since ,
and therefore
Let . Orthogonality of gives , so by (60)
which is the objective of (DYN-RO) evaluated at . The blocks of are , which lie in whenever and do, so is feasible for (DYN-RO). Hence every feasible of (DYN-H) attains the objective value of a feasible point of (DYN-RO), and . Conversely, every feasible of (DYN-RO) is attained by the feasible pair , so . The two optimal values therefore coincide, and the correspondence of minimizers follows from the equality of objective values along the map . The same argument applies verbatim with in place of .
Proof of Proposition 10
Proof. Identical to Propositions 3–4 and Theorems 5–7, with every block constraint fixed to and the trace constraint removed: remains strictly feasible, so strong duality holds, and each block contributes six linear equalities, so LICQ holds as in Appendix § 11.§ 6.
Suboptimality Gap for the Sphere-Constrained SDP
Algorithm 1 is run with finite gradient tolerances, so the returned may have a slightly negative minimum eigenvalue even when the solution is essentially optimal. Adapting the suboptimality analysis of XM [20] to the sphere-constrained relaxation (10), the following bound quantifies the resulting optimality gap.
Theorem 11 (Suboptimality Bound, adapted from [20]). Let be any feasible point of (10) with objective value , let collect the right-hand sides of (14), let be the dual slack of (17), and let be arbitrary. Then
:::`
Proof. The first inequality holds because is feasible. For the second, let be any feasible point of (10). Using ,
Since , we have , and the trace constraint fixes ; hence . Minimizing over all feasible yields the claim.
In the static stage we apply (62) with , the rounded solution of Appendix § 12.§ 1, and from the certification step (18); the resulting relative gap is defined in (21) (§ 3.4). The same argument applies to the dynamic relaxation (DYN-SDP), whose block constraints fix ; there the bound reads .
Partial sphere constraint. The partial sphere of § 6.2 fixes only , so is no longer constant. Restricting the partial-sphere variant of (10) to , the same proof gives , and without the restriction when . In our implementation we take , which bounds the suboptimality of among feasible points whose trace does not exceed twice its own.
Rounding to the Scaled-Rotation Problem
Let be the output of Algorithm 1. We round it to a feasible point of (SRO) under the sphere constraint in three steps.
Rank-three factor. Let be the three largest eigenpairs of and set , with blocks .
Projection. Each block is projected onto the scaled rotations: with the singular value decomposition , we set and .
Sphere. We rescale all scales by the common factor .
Under the partial sphere constraint, step 3 uses the factor , where collects the scales of the frames in . This enforces .
Dataset Details
IMC. Six scenes of the IMC PhotoTourism benchmark [28]—gate_00, gate_01, rome_00, rome_04, temple_00, and temple_01—each an unordered collection of internet photographs of a static landmark, subsampled from the original collection to about 200 images per scene.
Replica. Eight synthetic indoor sequences rendered from the Replica dataset [63] as in NICE-SLAM [89]: office1, office2, room1, and room2, two sequences each. Each sequence is subsampled from the original 2000 frames to 100 frames. The scenes are static.
TUM. Six sequences from the TUM RGB-D benchmark [64], each subsampled from the original sequence to 100 frames. The sequences contain moving objects; only camera accuracy is evaluated.
BAL. Three problems of the Bundle Adjustment in the Large dataset [1], BAL-93, BAL-394, and BAL-1936, used only for the solver-configuration study.
Sintel. Twenty-three synthetic sequences of the MPI Sintel dataset [12] with up to 50 frames each, containing fast articulated motion and large camera motion.
NVIDIA. The seven scenes of the NVIDIA dynamic scene dataset [81]—Baloon1, Baloon2, Jumping, Playground, Skating, Truck, and Umbrella—each with 12 frames.
DyCheck. The seven scenes of the DyCheck iPhone dataset [18]—apple, block, paper-windmill, space-out, spin, teddy, and wheel—with 277 to 473 frames each, captured by a handheld iPhone with realistic camera motion and evaluated from two static test cameras.
Syn4D. A synthetic benchmark of eight 100-frame sequences with ground-truth cameras and geometry, in two sub-datasets, Bear and RoboCasa. The four Bear sequences (bear1–bear4) place a scanned Christmas-bear model from Artec 3D [4] in a Replica [63] background, each with a different motion of the bear. The four RoboCasa sequences (casa1–casa4) are rendered from RoboCasa [47] scenes.
Implementation Details
Throughout, is the pixel of track at frame .
Online Tracking and Query Management
Online query sampling. At every frame, candidate queries are drawn uniformly and accepted iff
where contains all tracks visible at frame and all queries added in recent frames.
Mask filter. On the static side, a track whose observations fall inside the mask more than 20% of the time leaves the static set (meaning it lives on the moving object); otherwise only its masked observations are dropped. On the dynamic side the masks play the opposite role.
Observation Filtering
Large 2D drift (static tracks). A static track that suddenly disagrees with the motion of its neighbors should be excluded. For every transition and every track visible at both frames, define the flow and its local reference,
where collects the other tracks visible at both frames whose position lies in a window centered on , and the median is taken component-wise. A transition is bad iff the relative-motion error exceeds a threshold,
The threshold adapts to the motion speed of the frame: with the median flow magnitude over all tracks visible at the transition,
For every track with at least one bad transition, the visible timeline is split at the bad transitions and only the longest contiguous segment is kept.
Depth filters. Each surviving observation, static or dynamic, is assigned the depth of its nearest depth-map pixel when lifted to 3D. Observations with excessive depth,
are discarded, as monocular depth is unreliable far from the camera. Observations on depth discontinuities, detected by thresholding the second-order Sobel gradient magnitude of the depth map at the sample location, are discarded as well, since the nearest-pixel depth lookup is unreliable there.
XM2 residual trim. After the first solve of the static problem, the per-observation residual
is computed, observations above the -th percentile of are removed ( in our experiments), and the system is re-solved as described in § 6.2.
Node Observation Term in the Dynamic Solve
Each EDG node is itself a tracked point with lifted observations at the frames where it is observed. In the implementation, the segment solves add to (DYN) a node term that adds every node to follow its own observations,
Propositions 8 and 10 hold unchanged (Appendix § 11.8).
Fusion of Segment Solutions
Positions are averaged over all sequences that solved node at frame , each solution weighted by , where is the number of frames from to the nearest frame in which node is directly observed within that sequence, so that extrapolated poses count less than observed ones. Rotations cannot be averaged directly, since each sequence fixes its own gauge at its rest frame, but their relative motion does not depend on that choice: every sequence that contains frames and provides the relative rotation . For each node we recover all its rotations at once by rotation synchronization,
which we solve with a local solver, fixing the gauge by setting at the first frame in which node is observed. Solving (71) with XM is left to future work.
Long-Range Dynamic Tracks
Raw dynamic tracks are short and, on repetitive motion, swap the identity of similar-looking parts, so we optionally assemble them into long-range tracks in three steps. (a) Skeleton. A sparse but more reliable long-range tracker, CoWTracker [32], is restarted periodically from a grid inside the object mask and run forward and backward, so that every skeleton track spans the whole video; tracks with large 2D jumps, inconsistent color, or low coverage are filtered out. (b) Attachment. Short online tracks [85] are attached to every skeleton point whose percentile distance to them is small. (c) Averaging. Each skeleton point is replaced by the per-frame mean of its attached tracks, giving one long track per skeleton point; unattached online tracks are kept as additional shorter tracks.
GPU Solver
XM4D implements RTR-TCG and the Riemannian staircase of Algorithm 1 on the GPU in double precision. The data matrices and are obtained by eliminating the linearly entering translations, landmarks, and node translations, an instance of variable projection that stays efficient when sparsity is exploited [50]. We store densely, so each inner conjugate-gradient iteration costs one dense matrix product for the Euclidean Hessian action , computed with cuBLAS [49]. The tangent-space projections and the retractions, a batched Gram–Schmidt QR for the Stiefel blocks and a renormalization onto the sphere for the scales, are custom CUDA kernels; the dynamic problem (DYN-BM) reuses them without the scale variables.
References
- [1]Sameer Agarwal, Noah Snavely, Steven M Seitz, and Richard Szeliski. Bundle adjustment in the large. In European Conference on Computer Vision (ECCV), pages 29–42. Springer, 2010. 14, 30DOI
- [2]Sameer Agarwal, Keir Mierle, and The Ceres Solver Team. Ceres Solver, 10 2023. URL https://github.com/ceres-solver/ceres-solver. 12
- [3]Arthur Allshire, Chouki Chou, Jiunyi Zhang, David McAllister, Anthony Zhang, Chung Min Kim, Trevor Darrell, Pieter Abbeel, Jitendra Malik, and Angjoo Kanazawa. Visual imitation enables contextual humanoid control. Conference on Robot Learning (CoRL), 2025. 2arxiv.org/abs/2505.03729
- [4]Artec 3D. Christmas Bear. 3D model. URL https://www.artec3d.com/3d-models/christmas-bear. Licensed under CC BY 4.0. 18, 30
- [5]Alexey Bochkovskiy, Amaël Delaunoy, Hugo Germain, Marcel Santos, Yichao Zhou, Stephan Richter, and Vladlen Koltun. Depth pro: Sharp monocular metric depth in less than a second. In International Conference on Learning Representations (ICLR), volume 2025, pages 75602–75637, 2025. 4
- [6]Boston Dynamics. Atlas Airborne. YouTube video, February 2026. URL https://www.youtube.com/watch?v=UNorxwIZFk. 16, 17
- [7]N. Boumal, B. Mishra, P.-A. Absil, and R. Sepulchre. Manopt, a MATLAB Toolbox for Optimization on Manifolds. Journal of Machine Learning Research, 15(1):1455–1459, 2014. 6arxiv.org/abs/1308.5200
- [8]Nicolas Boumal. A Riemannian low-rank method for optimization over semidefinite matrices with block-diagonal constraints. arXiv preprint arXiv:1506.00575, 2015. 4, 7
- [9]Aljaz Bozic, Pablo Palafox, Michael Zollhofer, Justus Thies, Angela Dai, and Matthias Nießner. Neural deformation graphs for globally-consistent non-rigid reconstruction. In Conference on Computer Vision and Pattern Recognition (CVPR), pages 1450–1459, 2021. 4
- [10]Minh-Quan Viet Bui, Jongmin Park, Jihyong Oh, and Munchurl Kim. DyBluRF: Dynamic Deblurring Neural Radiance Fields for Blurry Monocular Video. arXiv preprint arXiv:2312.13528, 2023. 15
- [11]Samuel Burer and Renato D.C. Monteiro. A nonlinear programming algorithm for solving semidefinite programs via low-rank factorization. Mathematical Programming, 95(2):329–357, 2003. 4, 6, 27, 28
- [12]D. J. Butler, J. Wulff, G. B. Stanley, and M. J. Black. A naturalistic open source movie for optical flow evaluation. In A. Fitzgibbon et al. (Eds.), editor, European Conference on Computer Vision (ECCV), Part IV, LNCS 7577, pages 611–625. Springer-Verlag, October 2012. 14, 16, 17, 30
- [13]Yohann Cabon, Lucas Stoffl, Leonid Antsfeld, Gabriela Csurka, Boris Chidlovskii, Jerome Revaud, and Vincent Leroy. Must3r: Multi-view network for stereo 3d reconstruction. In Conference on Computer Vision and Pattern Recognition (CVPR), pages 1050–1060. IEEE, 2025. 4
- [14]Nicolas Carion, Genglin G. Guan, Yuan-Ting Hu, Shoubhik Debnath, Ronghang Hu, Didac Suris, Chaitanya Ryali, Kalyan Vasudev Alwala, Haitham Khedr, Andrew Huang, et al. Sam 3: Segment anything with concepts. arXiv preprint arXiv:2511.16719, 2025. 11arxiv.org/abs/2511.16719
- [15]Weirong Chen, Ganlin Zhang, Felix Wimbauer, Rui Wang, Nikita Araslanov, Andrea Vedaldi, and Daniel Cremers. Back on track: Bundle adjustment for dynamic scene reconstruction. In International Conference on Computer Vision (ICCV), pages 4951–4960. IEEE, 2025. 2
- [16]Carl Doersch, Pauline Luc, Yi Yang, Dilara Gokay, Skanda Koppula, Ankush Gupta, Joseph Heyward, Ignacio Roccco, Ross Goroshin, Joao Carreira, et al. Bootstap: Bootstrapped training for tracking-any-point. In Proceedings of the Asian Conference on Computer Vision, pages 3257–3274, 2024. 4arxiv.org/abs/2402.00847
- [17]Chen Gao, Ayush Saraf, Johannes Kopf, and Jia-Bin Huang. Dynamic view synthesis from dynamic monocular video. In International Conference on Computer Vision (ICCV), pages 5692–5701. IEEE, 2021. 2arxiv.org/abs/2105.06468
- [18]Hang Gao, Ruilong Li, Shubham Tulsiani, Bryan Russell, and Angjoo Kanazawa. Monocular dynamic view synthesis: A reality check. Conference on Neural Information Processing Systems (NeurIPS), 35:33768–33780, 2022.arxiv.org/abs/2210.13445
- [19]Ravi Garg, Anastasios Roussos, and Lourdes Agapito. Dense variational reconstruction of non-rigid surfaces from monocular video. In Conference on Computer Vision and Pattern Recognition (CVPR), pages 1272–1279, 2013.DOI
- [20]Haoyu Han and Heng Yang. Building Rome with Convex Optimization. In Robotics: Science and Systems (RSS), 2025.arxiv.org/abs/2502.04640
- [21]Tairan He, Zhengyi Luo, Xialin He, Wenli Xiao, Chong Zhang, Weinan Zhang, Kris Kitani, Changliu Liu, and Guanya Shi. Omni2o: Universal and dexterous human-to-humanoid whole-body teleoperation and learning. Conference on Robot Learning (CoRL), 2024.arxiv.org/abs/2406.08858
- [22]Luke Hollis, Tianxiang Fan, and Heng Yang. SceneAgent: An Agentic Pipeline from 3D and 4D Captures to Simulation Environments. arXiv preprint arXiv:2610.34745, 2026.arxiv.org/abs/2610.34745
- [23]Mu Hu, Wei Yin, Chi Zhang, Zhipeng Cai, Xiaoxiao Long, Hao Chen, Kaixuan Wang, Gang Yu, Chunhua Shen, and Shaojie Shen. Metric3D v2: A versatile monocular geometric foundation model for zero-shot metric depth and surface normal estimation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(12):10579–10596, 2024.DOI
- [24]Matthias Innmann, Michael Zollhöfer, Matthias Nießner, Christian Theobalt, and Marc Stamminger. Volumedeform: Real-time volumetric non-rigid reconstruction. In European Conference on Computer Vision (ECCV), pages 362–379. Springer, 2016.arxiv.org/abs/1603.08161
- [25]Hanxiao Jiang, Hao-Yu Hsu, Kaifeng Zhang, Hsin-Ni Yu, Shenlong Wang, and Yunzhu Li. Phystwin: Physics-informed reconstruction and simulation of deformable objects from videos. In International Conference on Computer Vision (ICCV), pages 7219–7230. IEEE, 2025.arxiv.org/abs/2503.17973
- [26]Tao Jiang, Peng Lu, Li Zhang, Ningsheng Ma, Rui Han, Chengqi Lyu, Yining Li, and Kai Chen. Rtmpose: Real-time multi-person pose estimation based on mmpose. arXiv preprint arXiv:2303.07399, 2023.arxiv.org/abs/2303.07399
- [27]Zhenyu Jiang, Cheng-Chun Hsu, and Yuke Zhu. Ditto: Building digital twins of articulated objects from interaction. In Conference on Computer Vision and Pattern Recognition (CVPR), pages 5606–5616. IEEE, 2022.arxiv.org/abs/2202.08227
- [28]Yuhe Jin, Dmytro Mishkin, Anastasiia Mishchuk, Jiri Matas, Pascal Fua, Kwang Moo Yi, and Eduard Trulls. Image Matching across Wide Baselines: From Paper to Practice. International Journal of Computer Vision, 2020.DOI
- [29]Michel Journée, Francis Bach, P-A Absil, and Rodolphe Sepulchre. Low-rank optimization on the cone of positive semidefinite matrices. SIAM Journal on Optimization, 20(5):2327–2351, 2010.DOI
- [30]Nikita Karaev, Yuri Makarov, Jianyuan Wang, Natalia Neverova, Andrea Vedaldi, and Christian Rupprecht. CoTracker3: Simpler and Better Point Tracking by Pseudo-Labeling Real Videos. In International Conference on Computer Vision (ICCV), pages 1–10. IEEE, 2025.DOI
- [31]Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3D Gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics, 42(4):139–1, 2023.arxiv.org/abs/2308.04079
- [32]Zihang Lai, Eldar Insafutdinov, Edgar Sucar, and Andrea Vedaldi. Cowtracker: Tracking by warping instead of correlation. arXiv preprint arXiv:2602.04877, 2026.
- [33]Jean B Lasserre. Convergent SDP-relaxations in polynomial optimization with sparsity. SIAM Journal on optimization, 17(3):822–843, 2006.DOI
- [34]Long Le, Jason Xie, William Liang, Hung-Ju Wang, Yue Yang, Yecheng Jason Ma, Kyle Vedder, Arjun Krishna, Dinesh Jayaraman, and Eric Eaton. Articulate-anything: Automatic modeling of articulated objects via a vision-language foundation model. In International Conference on Learning Representations (ICLR), volume 2025, pages 17578–17602, 2025.arxiv.org/abs/2410.13882
- [35]Jiahui Lei, Yijia Weng, Adam W Harley, Leonidas Guibas, and Kostas Daniilidis. Mosca: Dynamic gaussian fusion from casual videos via 4d motion scaffolds. In Conference on Computer Vision and Pattern Recognition (CVPR), pages 6165–6177. IEEE, 2025.arxiv.org/abs/2405.17421
- [36]Zhengqi Li, Simon Niklaus, Noah Snavely, and Oliver Wang. Neural scene flow fields for space-time view synthesis of dynamic scenes. In Conference on Computer Vision and Pattern Recognition (CVPR), pages 6498–6508, 2021.arxiv.org/abs/2011.13084
- [37]Zhengqi Li, Richard Tucker, Forrester Cole, Qianqian Wang, Linyi Jin, Vickie Ye, Angjoo Kanazawa, Alexander Holynski, and Noah Snavely. Megasam: Accurate, fast, and robust structure and motion from casual dynamic videos. In Conference on Computer Vision and Pattern Recognition (CVPR), pages 10486–10496. IEEE, 2025.arxiv.org/abs/2412.04463
- [38]Haotong Lin, Sili Chen, Junhao Liew, Donny Y Chen, Zhenyu Li, Guang Shi, Jiashi Feng, and Bingyi Kang. Depth anything 3: Recovering the visual space from any views. arXiv preprint arXiv:2511.10647, 2025.arxiv.org/abs/2511.10647
- [39]Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European Conference on Computer Vision (ECCV), pages 740–755. Springer, 2014.arxiv.org/abs/1405.0312
- [40]Yang Liu, Tong Wang, Yiming Bao, and Heng Yang. GRASP: Gradient-Guided Sampling for Contact-Rich Physics-Based Motion Retargeting. arXiv preprint arXiv:2611.05754, 2026.arxiv.org/abs/2611.05754
- [41]Yu-Lun Liu, Chen Gao, Andreas Meulman, Hung-Yu Tseng, Ayush Saraf, Changil Kim, Yung-Yu Chuang, Johannes Kopf, and Jia-Bin Huang. Robust dynamic radiance fields. In Conference on Computer Vision and Pattern Recognition (CVPR), pages 13–23. IEEE, 2023.arxiv.org/abs/2301.02239
- [42]Stephen Lombardi, Tomas Simon, Jason Saragih, Gabriel Schwartz, Andreas Lehrmann, and Yaser Sheikh. Neural volumes: Learning dynamic renderable volumes from images. ACM Trans. Graph., 38(4):65:1–65:14, July 2019.DOI
- [43]David G. Lowe. Distinctive image features from scale-invariant keypoints. International journal of computer vision, 60(2):91–110, 2004.DOI
- [44]Jonathon Luiten, Georgios Kopanas, Bastian Leibe, and Deva Ramanan. Dynamic 3d gaussians: Tracking by persistent dynamic view synthesis. In 2024 International Conference on 3D Vision (3DV), pages 800–809. IEEE, 2024.arxiv.org/abs/2308.09713
- [45]Xingyu Miao, Yang Bai, Haoran Duan, Fan Wan, Yawen Huang, Yang Long, and Yefeng Zheng. Ctnerf: Cross-time transformer for dynamic neural radiance field from monocular video. Pattern Recognition, 156:110729, 2024.DOI
- [46]Franziska Mueller, Dushyant Mehta, Oleksandr Sotnychenko, Srinath Sridhar, Dan Casas, and Christian Theobalt. Real-time hand tracking under occlusion from an egocentric rgb-d sensor. In International Conference on Computer Vision (ICCV), pages 1163–1172. IEEE, 2017.
- [47]Soroosh Norouzzadeh, Abhinav Maddikuri, Lance Zhang, Adeet Parikh, Aaron Lo, Abhishek Joshi, Ajay Mandlekar, and Yuke Zhu. RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robots. In Robotics: Science and Systems (RSS), 2024.
- [48]Richard A Newcombe, Dieter Fox, and Steven M Seitz. Dynamicfusion: Reconstruction and tracking of non-rigid scenes in real-time. In Conference on Computer Vision and Pattern Recognition (CVPR), pages 343–352, 2015.DOI
- [49]NVIDIA Corporation. cuBLAS Library User Guide, Version 12.9, 2025. NVIDIA Developer Documentation.
- [50]Alan Papalia, Nikolas Sanderson, Haoyu Han, Heng Yang, Hanumant Singh, and Michael Everett. Sparse variable projection in robotic perception: Exploiting separable structure for efficient nonlinear optimization. In IEEE International Conference on Robotics and Automation (ICRA), pages 11826–11833. IEEE, 2026.arxiv.org/abs/2512.07969
- [51]Keunhong Park, Utkarsh Sinha, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Steven M Seitz, and Ricardo Martin-Brualla. Nerfies: Deformable neural radiance fields. In International Conference on Computer Vision (ICCV), pages 5845–5854. IEEE, 2021.arxiv.org/abs/2011.12948
- [52]Keunhong Park, Utkarsh Sinha, Peter Hedman, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Ricardo Martin-Brualla, and Steven M Seitz. Hypernerf: A higher-dimensional representation for topologically varying neural radiance fields. arXiv preprint arXiv:2106.13228, 2021.arxiv.org/abs/2106.13228
- [53]Sida Peng, Qianqian Wang, Qing Shuai, Hujun Bao, and Xiaowei Zhou. Neural body: Implicit neural representations with structured latent codes for novel view synthesis of dynamic humans. In Conference on Computer Vision and Pattern Recognition (CVPR), pages 9050–9059. IEEE, 2021.arxiv.org/abs/2012.15838
- [54]Federico Perazzi, Jordi Pont-Tuset, Brian McWilliams, Luc Van Gool, Markus Gross, and Alexander Sorkine-Hornung. A benchmark dataset and evaluation methodology for video object segmentation. In Conference on Computer Vision and Pattern Recognition (CVPR), pages 724–732, 2016.DOI
- [55]Luigi Piccinelli, Yung-Hsu Yang, Christos Sakaridis, Mattia Segu, Siyuan Li, Luc Van Gool, and Fisher Yu. Unidepth: Universal monocular metric depth estimation. In Conference on Computer Vision and Pattern Recognition (CVPR), pages 10106–10116, 2024.
- [56]Rolandos Alexandros Potamias, Jinglei Zhang, Jiankang Deng, and Stefanos Zafeiriou. Wilor: End-to-end 3d hand localization and reconstruction in-the-wild. In Conference on Computer Vision and Pattern Recognition (CVPR), pages 12242–12254. IEEE, 2025.arxiv.org/abs/2409.12259
- [57]Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francesc Moreno-Noguer. D-NeRF: Neural Radiance Fields for Dynamic Scenes. arXiv preprint arXiv:2011.13961, 2020.arxiv.org/abs/2011.13961
- [58]Han Qi, Changhe Chen, and Heng Yang. Compose by Focus: Scene Graph-based Atomic Skills. In IEEE International Conference on Robotics and Automation (ICRA), 2026.arxiv.org/abs/2509.16053
- [59]Javier Romero, Dimitrios Tzionas, and Michael J Black. Embodied hands: Modeling and capturing hands and bodies together. arXiv preprint arXiv:2201.02610, 2022.DOI
- [60]David M Rosen, Luca Carlone, Alfonso S Bandeira, and John J Leonard. SE-Sync: A certifiably correct algorithm for synchronization over the special Euclidean group. International Journal of Robotics Research (IJRR), 38(2-3):95–125, 2019.
- [61]Johannes L Schonberger and Jan-Michael Frahm. Structure-from-motion revisited. In Conference on Computer Vision and Pattern Recognition (CVPR), pages 4104–4113, 2016.DOI
- [62]Colton Stearns, Adam Harley, Mikaela Uy, Florian Dubost, Federico Tombari, Gordon Wetzstein, and Leonidas Guibas. Dynamic gaussian marbles for novel view synthesis of casual monocular videos. In SIGGRAPH Asia 2024 Conference Papers, pages 1–11, 2024.DOI
- [63]Julian Straub, Thomas Whelan, Lingni Ma, Yufan Chen, Erik Wijmans, Simon Green, Jakob J. Engel, Raul Mur-Artal, Carl Ren, Shobhit Verma, Anton Clarkson, Mingfei Yan, Brian Budge, Yajie Yan, Xiaoqing Pan, June Yon, Yuyang Zou, Kimberly Leon, Nigel Carter, Jesus Briales, Tyler Gillingham, Elias Mueggler, Luis Pesqueira, Manolis Savva, Dhruv Batra, Hauke M. Strassdat, Renzo De Nardi, Michael Goesele, Steven Lovegrove, and Richard Newcombe. The Replica Dataset: A Digital Replica of Indoor Spaces. arXiv preprint arXiv:1906.05797, 2019.arxiv.org/abs/1906.05797
- [64]J. Sturm, N. Engelhard, F. Endres, W. Burgard, and D. Cremers. A Benchmark for the Evaluation of RGB-D SLAM Systems. In International Conference on Intelligent Robots and Systems (IROS), Oct. 2012.DOI
- [65]Robert W Sumner, Johannes Schmid, and Mark Pauly. Embedded deformation for shape manipulation. In ACM siggraph 2007 papers, pages 80–es, 2007. 9
- [66]Zachary Teed and Jia Deng. Droid-slam: Deep visual slam for monocular, stereo, and rgb-d cameras. Conference on Neural Information Processing Systems (NeurIPS), 34:16558–16569, 2021. 15arxiv.org/abs/2108.10869
- [67]Jianyuan Wang, Minghao Chen, Nikita Karaev, Andrea Vedaldi, Christian Rupprecht, and David Novotny. Vggt: Visual geometry grounded transformer. In Conference on Computer Vision and Pattern Recognition (CVPR), pages 5294–5306. IEEE, 2025. 4, 5, 13, 15arxiv.org/abs/2503.11651
- [68]Qianqian Wang, Vickie Ye, Hang Gao, Weijia Zeng, Jake Austin, Zhengqi Li, and Angjoo Kanazawa. Shape of motion: 4d reconstruction from a single video. In International Conference on Computer Vision (ICCV), pages 9660–9672. IEEE, 2025. 2, 4, 15, 25arxiv.org/abs/2407.13764
- [69]Shuzhe Wang, Vincent Leroy, Yohann Cabon, Boris Chidlovskii, and Jerome Revaud. Dust3r: Geometric 3d vision made easy. In Conference on Computer Vision and Pattern Recognition (CVPR), pages 20697–20709. IEEE, 2024. 4arxiv.org/abs/2312.14132
- [70]Chung-Yi Weng, Brian Curless, Pratul P Srinivasan, Jonathan T Barron, and Ira Kemelmacher-Shlizerman. Humannerf: Free-viewpoint rendering of moving people from monocular video. In Conference on Computer Vision and Pattern Recognition (CVPR), pages 16210–16220, 2022. 2arxiv.org/abs/2201.04127
- [71]Henry Wolkowicz, Romesh Saigal, and Lieven Vandenberghe, editors. Handbook of Semidefinite Programming: Theory, Algorithms, and Applications, volume 27 of International Series in Operations Research & Management Science. Kluwer Academic Publishers, Boston, MA, 2000. doi: 10.1007/978-1-4615-4381-7. 7DOI
- [72]Guanzhun Wu, Taoran Yi, Jiemin Fang, Lingxi Xie, Xiaopeng Zhang, Wei Wei, Wenyu Liu, Qi Tian, and Xinggang Wang. 4d gaussian splatting for real-time dynamic scene rendering. In Conference on Computer Vision and Pattern Recognition (CVPR), pages 20310–20320. IEEE, 2024. 4arxiv.org/abs/2310.08528
- [73]Yuxi Xiao, Qianqian Wang, Shangzhan Zhang, Nan Xue, Sida Peng, Yujun Shen, and Xiaowei Zhou. Spatialtracker: Tracking any 2d pixels in 3d space. In Conference on Computer Vision and Pattern Recognition (CVPR), pages 20406–20417. IEEE, 2024.arxiv.org/abs/2404.04319
- [74]Gengshan Yang, Deqing Sun, Varun Jampani, Daniel Vlasic, Forrester Cole, Huiwen Chang, Deva Ramanan, William T Freeman, and Ce Liu. Lasr: Learning articulated shape reconstruction from a monocular video. In Conference on Computer Vision and Pattern Recognition (CVPR), 2021. 2arxiv.org/abs/2105.02976
- [75]Heng Yang. Semidefinite Optimization and Relaxation. Lecture notes: https://hankyang.seas.harvard.edu/Semidefinite/, 2024. 5, 6, 27, 28
- [76]Heng Yang and Luca Carlone. Certifiably optimal outlier-robust geometric perception: Semidefinite relaxations and scalable global optimization. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 45(3):2816–2834, 2022. 4DOI
- [77]Heng Yang, Junheng Shi, and Luca Carlone. Teaser: Fast and certifiable point cloud registration. IEEE Transactions on Robotics (TRO), 37(2):314–333, 2020. 4, 15arxiv.org/abs/2001.07715
- [78]Heng Yang, Ling Liang, Luca Carlone, and Kim-Chuan Toh. An inexact projected gradient method with rounding and lifting by nonlinear programming for solving rank-one semidefinite relaxation of polynomial optimization. Mathematical Programming, 201(1):409–472, 2023. 4
- [79]Lihe Yang, Bingyi Kang, Zilong Huang, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything: Unleashing the power of large-scale unlabeled data. In Conference on Computer Vision and Pattern Recognition (CVPR), pages 10371–10381, 2024. 4
- [80]Vickie Ye, Ruilong Li, Justin Kerr, Matias Turkulainen, Brent Yi, Zhuoyang Pan, Otto Seiskari, Jianbo Ye, Jeffrey Hu, Matthew Tancik, and Angjoo Kanazawa. gsplat: An open-source library for Gaussian splatting. Journal of Machine Learning Research (JMLR), 26(34):1–17, 2025. 14arxiv.org/abs/2409.06765
- [81]Jae Shin Yoon, Kihwan Kim, Orazio Gallo, Hyun Soo Park, and Jan Kautz. Novel view synthesis of dynamic scenes with globally coherent depths from a monocular camera. In Conference on Computer Vision and Pattern Recognition (CVPR), pages 5335–5344. IEEE, 2020. 14, 16, 18, 30
- [82]Xihang Yu and Heng Yang. SIM-Sync: From Certifiably Optimal Synchronization over the 3D Similarity Group to Scene Reconstruction with Learned Depth. IEEE Robotics and Automation Letters (RAL), 2024. URL https://arxiv.org/abs/2309.05184. 4
- [83]Xiaoming Zhao, R Colburn, Fangchang Ma, Miguel Angel Bautista, Joshua Susskind, and Alex Schwing. Pseudo-generalized dynamic view synthesis from a video. In International Conference on Learning Representations (ICLR), volume 2024, pages 42726–42748, 2024. 15arxiv.org/abs/2310.08587
- [84]Zhenjun Zhao, Heng Yang, Bangyan Liao, Yingping Zeng, Shaocheng Yan, Yingdong Gu, Peidong Liu, Yi Zhou, Haoang Li, and Javier Civera. Advances in global solvers for 3d vision. arXiv preprint arXiv:2602.14662, 2026. 4arxiv.org/abs/2602.14662
- [85]Artem Zholus, Carl Doersch, Yi Yang, Skanda Koppula, Viorica Patraucean, Xu Owen He, Ignacio Rocco, Mehdi S. M. Sajjadi, Sarath Chandar, and Ross Goroshin. TAPNext: Tracking Any Point (TAP) as Next Token Prediction. International Conference on Computer Vision (ICCV), 2025. 4, 11, 32
- [86]Kaichen Zhou, Jia-Xing Zhong, Sangyun Shin, Kai Lu, Yiyuan Yang, Andrew Markham, and Niki Trigoni. Dynpoint: Dynamic neural point for view synthesis. Conference on Neural Information Processing Systems (NeurIPS), 36:69532–69545, 2023. 15arxiv.org/abs/2310.18999
- [87]Yifeng Zhu, Zhenyu Jiang, Peter Stone, and Yuke Zhu. Learning generalizable manipulation policies with object-centric 3d representations. arXiv preprint arXiv:2310.14386, 2023. 19arxiv.org/abs/2310.14386
- [88]Yingying Zhu, Dong Huang, Fernando De La Torre, and Simon Lucey. Complex non-rigid motion 3d reconstruction by union of subspaces. In Conference on Computer Vision and Pattern Recognition (CVPR), pages 1542–1549, 2014. 4DOI
- [89]Zihan Zhu, Songyou Peng, Viktor Larsson, Weiwei Xu, Hujun Bao, Zhaopeng Cui, Martin R. Oswald, and Marc Pollefeys. NICE-SLAM: Neural Implicit Scalable Encoding for SLAM. In Conference on Computer Vision and Pattern Recognition (CVPR), 2022. 14, 30arxiv.org/abs/2112.12130





