betaFeedback

WorksComputer Science

XM4D: 4D Reconstruction through Certifiable Geometric Optimization

Abstract

Accurate 4D reconstruction remains challenging: jointly estimating camera motion, scene geometry, and non-rigid deformation entails highly nonconvex optimization, making global optimality difficult to guarantee. We present XM4D, a framework for accurate and efficient 4D reconstruction built upon certifiably globally optimal geometric optimization. By separating the estimation of camera motion from non-rigid deformation, we formulate both stages as structured quadratically constrained quadratic programs (QCQP) that admit semidefinite programming (SDP) relaxations and verifiable global optimality certificates. For camera motion estimation, we lift static point tracks into 3D using foundational depth prediction models and formulate a scaled bundle adjustment problem that jointly estimates camera poses, depth scales, and static landmarks. By analytically eliminating translations and landmarks, we reduce the problem to a QCQP over scaled rotations. For non-rigid reconstruction, we represent dynamic motion using an embedded deformation graph (EDG) and show that its tracking, rigidity, and temporal regularization objectives similarly reduce to a rotation-only QCQP. Both problems admit SDP relaxations that we solve efficiently with a GPU-accelerated Burer-Monteiro solver based on Riemannian optimization. The recovered geometry further initializes dynamic Gaussian splatting for photometric refinement. Extensive experiments across seven benchmarks demonstrate accurate camera motion estimation on both static and dynamic scenes, efficient optimization that scales to thousands of frames, and state-of-the-art novel-view rendering quality.

Website: https://computationalrobotics.seas.harvard.edu/XM4D/

XM4D overview showing (a) Images, tracks and depths, (b) Reconstruction, and (c) Rendering, with 2D tracks, depth maps, preprocess, depth-guided 2D to 3D lifting, solve, globally certifiable optimization, render, and 4D Gaussian splatting

Figure 1. XM4D overview. From (a) 2D tracks and monocular depth, a solver with certified global optimality followed by refinement recovers (b) the cameras, the static geometry (points), and the 3D trajectories of dynamic points (colored curves), which initialize (c) 4D Gaussian splatting for rendering. The skeleton in (c) is a subsampled embedded deformation graph (EDG).

Introduction

4D reconstruction aims to recover the time-varying 3D structure of a scene from visual observations. It supports applications including dynamic novel-view rendering [36, 42], deformable and articulated object reconstruction [24, 48, 74], digital humans and digital twins [25, 53, 70], and motion retargeting for humanoid control [3, 21, 40]. Despite its broad utility, reconstructing a dynamic scene from monocular images or videos remains highly challenging [17, 18]. Camera and object motion are entangled in the visual observations [15], while dynamic geometry itself is weakly constrained and may undergo complex non-rigid deformation. Long sequences introduce further difficulties, including fragmented point tracks, noisy and scale-inconsistent monocular depth, and temporally incomplete observations of the dynamic scene [35, 68].

Recent 4D reconstruction methods commonly rely on coupled nonconvex optimization over camera poses, scene geometry, deformation, and appearance [35, 68]. Such formulations have achieved impressive results, but their strong coupling makes the optimization difficult: the objective is highly nonconvex and local optimization may therefore be sensitive to initialization and converge to undesired local minima. As illustrated in Figure 3, local optimization of the motion of an embedded deformation graph (EDG) can converge to an inferior solution whereas the certifiable solver presented in this work achieves an accurate reconstruction. Similar failures of local optimization in static reconstruction were demonstrated in XM [20] (Fig. 10). This raises a natural question:

Figure 3. Certified versus local optimization. Results on a real DAVIS video [54] (top) and a synthetic deformation graph (bottom). Each panel shows the EDG nodes, their estimated rotations (red, green, and blue axes), and the graph edges at different frames. The local solver converges to a local minimum with wrong node rotations (orange boxes), whereas the certified XM solution recovers consistent rotations.

Can we achieve accurate and efficient 4D reconstruction through globally certifiable geometric optimization?

In this work, we present XM4D, a framework for accurate and efficient 4D reconstruction based on globally certifiable geometric optimization. Our approach separates camera motion estimation from non-rigid deformation, allowing both stages to be formulated as structured (nonconvex) quadratically constrained quadratic programs (QCQPs) and solved through (convex) semidefinite programming (SDP) relaxations. Figure 2(a) illustrates this decomposition as a view graph and Figure 2(b) instantiates it on a real scene: static and dynamic point tracks share the same cameras and predicted depth, but static tracks constrain time-invariant landmarks and camera poses, while dynamic tracks constrain the motion of an EDG. In the following, we overview the problem formulation and solution approach for both stages.

Figure 2. The XM4D view graph. (a) Schematic view graph. Camera poses (Ct,ct)∈SO(3)×R3(C_t, c_t) \in\mathrm{SO}(3) \times\mathbb{R}^3 (red) observe static landmarks xis∈R3x_i^s \in\mathbb{R}^3 (blue triangles) and dynamic points (pink circles) through the depth-lifted measurements u~its\tilde{u}_{it}^s and u~itd\tilde{u}_{it}^d (blue squares). Each dynamic keypoint is attached by skinning edges (dashed) to its neighboring nodes (Rj,t,pj,t)(R_{j,t}, p_{j,t}) of the embedded deformation graph (EDG) (purple capsules), whose nodes are connected by EDG edges (thick solid lines). (b) The view graph instantiated on a dynamic scene: a train moves along a circular track (black arrow), colored curves show the dynamic point tracks, and each part of the train is an EDG node with a free SE(3)\mathrm{SE}(3) pose.

I: Static stage. As shown in Figure 2, let t∈{1,…,T}t \in\{1, \ldots, T\} index video frames and i∈{1,…,Ns}i \in\{1, \ldots, N_s\} index static points. The visibility set Es\mathcal{E}_s contains the tracked point-frame pairs, and u~its∈R3\tilde{u}_{it}^s \in\mathbb{R}^3 denotes the corresponding 2D observation lifted to 3D using predicted monocular depth. Because monocular depth predictions are noisy and their scale drifts from frame to frame, we introduce a per-frame scale st>0s_t > 0 that corrects this frame-dependent scale error in the predicted depth. Weighting each observation by an optional confidence ωit>0\omega_{it} > 0, we jointly estimate camera poses, depth scales, and static landmarks via

min⁡Ct∈SO(3), ct∈R3st>0, xis∈R3∑(i,t)∈Esωit∥Ct(stu~its)+ct−xis∥2(1)\min_{\substack{C_t \in\mathrm{SO}(3),\ c_t \in\mathbb{R}^3\\ s_t>0,\ x_i^s\in\mathbb{R}^3}} \sum_{(i,t)\in\mathcal{E}_s} \omega_{it}\left\|C_t(s_t\tilde{u}_{it}^s)+c_t-x_i^s\right\|^2 \tag*{(1)}

The objective enforces that observations of the same static point, after correcting the predicted depth scale and transforming them into the world frame, agree on a common 3D landmark. Without a scale constraint, the objective admits a degenerate solution as all variables shrink toward zero. The original XM formulation [20] prevents this collapse by anchoring the first-frame scale to s1=1s_1 = 1, making the reconstruction overfit onto the first frame. We instead impose a uniform sphere constraint, ∑tst2=T\sum_t s_t^2 = T, which fixes the global scale without privileging any frame. We formulate this problem as a quadratically constrained quadratic program (QCQP), relax it to a semidefinite program (SDP), and solve it efficiently using Burer–Monteiro (BM) factorization together with a Riemannian staircase, following the certifiable optimization framework of XM and SE-Sync [20, 60].

II: Dynamic stage. After fixing the recovered camera trajectory, we transform dynamic observations into the world frame and model non-rigid motion with an EDG. Each dynamic point xidx_i^d is warped by the rigid transformations (Rj,t,pj,t)(R_{j,t}, p_{j,t}) of its neighboring EDG nodes, where gjg_j denotes the rest position of node jj and wijw_{ij} the skinning weight. Let yitd=Ct(stu~itd)+cty_{it}^d = C_t(s_t\tilde{u}_{it}^d) + c_t denote the corresponding depth-lifted observation in the world frame. The tracking loss is

Ltrack=∑(i,t)∈Ed∥Φt(xid)−yitd∥2,\mathcal{L}_{\mathrm{track}} = \sum_{(i,t)\in\mathcal{E}_d}\left\|\Phi_t(x_i^d)-y_{it}^d\right\|^2,
Φt(xid)=∑j∈N(i)wij[Rj,t(xid−gj)+gj+pj,t].(2)\Phi_t(x_i^d) = \sum_{j\in\mathcal{N}(i)} w_{ij}\left[R_{j,t}(x_i^d-g_j)+g_j+p_{j,t}\right]. \tag*{(2)}

We optimize the EDG node transformations with tracking loss, as-rigid-as-possible (ARAP) and temporal smoothness regularization. Unlike MoSca [35], which models deformation through relative transformations between frames, we express all node transformations relative to a fixed rest frame. With fixed EDG structure and skinning weights, this choice makes the tracking, ARAP, and temporal smoothness objectives quadratic in the node transformations. Eliminating node translations then yields a rotation-only QCQP that admits the same SDP-based certifiable optimization framework as the static stage. The recovered geometry then initializes dynamic Gaussian splatting for rendering.

In summary, XM4D makes the following contributions:

  • For static reconstruction, we build on the QCQP–SDP–BM framework of our conference version, XM [20], retaining its efficient optimization and global optimality certificates. We introduce a uniform sphere constraint that resolves the scale ambiguity without anchoring any individual frame.

  • For dynamic reconstruction, we design an EDG-based non-rigid reconstruction formulation that incorporates tracking, rigidity, and temporal smoothness into a structured QCQP. This formulation admits the same globally certifiable optimization framework.

  • We build XM4D, an accurate and efficient 4D reconstruction pipeline that combines certifiable geometric optimization with dynamic Gaussian splatting. Extensive experiments across seven benchmarks demonstrate leading camera accuracy, fast and scalable optimization, and the best average novel-view rendering performance on all three dynamic benchmarks (DyCheck, NVIDIA, and Syn4D).

Outline. The remainder of this paper is organized as follows. § 2 reviews related work. § 3 presents the static stage: the rotation-only QCQP and SDP relaxation, the BM solver with its optimality certificate, and the sphere constraint. § 4 extends this machinery to the EDG-based dynamic reconstruction and shows that it admits the same certifiable solver. § 6 describes the complete pipeline. § 7 evaluates camera accuracy, runtime, and novel-view rendering on seven benchmarks. Proofs and implementation details are given in the appendices. Visualizations and code are available on the project page:

https://computationalrobotics.seas.harvard.edu/XM4D/.

Related Work

Dynamic Scene Reconstruction

Existing dynamic scene reconstruction methods can be grouped by how they constrain the reconstruction. Rendering-based methods fit time-varying scene representations to images, while geometry-driven methods use measurements such as depth and point tracks to estimate structure and motion. Rendering-based methods represent dynamics using time-varying (volumetric) fields [36, 42], deformable canonical radiance fields and their higher-dimensional extensions [51, 52, 57], or moving Gaussians [44, 62, 72]. Many of them incorporate geometric regularization, and some jointly estimate camera motion [41]. However, recovering object or part level SE(3)\mathrm{SE}(3) motion for downstream scene understanding generally requires additional motion estimation. Moreover, many methods rely on accurate camera poses, carefully designed initialization, and heavy per-scene optimization.

Geometry-based methods use geometric measurements and motion priors to constrain dynamic reconstruction. Earlier non-rigid structure-from-motion methods recover 3D shape from monocular point tracks using low-rank or subspace priors [19, 88], while RGB-D methods use depth observations with deformation graphs or volumetric models to track motion and reconstruct geometry [9, 24, 48]. More recently, MoSca [35] lifts learned depth and point-track priors into a motion scaffold and anchors Gaussians to it for photometric reconstruction. Shape of Motion [68] represents scene motion using shared SE(3)\mathrm{SE}(3) motion bases, with per-point weights combining the bases to recover persistent trajectories from monocular depth and long-range 2D tracks. XM4D also uses explicit local SE(3)\mathrm{SE}(3) transformations as an EDG. Unlike them, XM4D estimates camera poses and temporally coupled graph transformations from depth and tracks with a certifiable geometric solver, making local rigid motions explicit without involving Gaussians.

Learning-Assisted Geometric Reconstruction

Learned geometric priors have become important components of reconstruction pipelines. Monocular depth models [5, 23, 38, 55, 79] predict dense scene depth, while point trackers [16, 30, 32, 73, 85] establish correspondences across frames. Together, depth and correspondence estimates provide complementary constraints for geometric reconstruction: depth informs scene structure, while tracks connect observations over time.

Recent feed-forward methods predict richer 3D scene representations directly from images. DUSt3R [69] predicts pointmaps for image pairs and aligns them in a shared coordinate frame, and MUSt3R [13] extends it to many views, and VGGT [67] jointly predicts camera parameters, depth maps, pointmaps, and 3D point tracks. In XM4D, learned depth and tracks are measurements for an explicit geometric optimization.

Certifiable Geometric Optimization

Certifiable geometric optimization solves nonconvex estimation problems, such as bundle adjustment, while verifying global optimality [33, 78, 84]. Semidefinite relaxations provide such certificates for pose synchronization [60, 82], outlier-robust registration [77], and general polynomial formulations of robust estimation [76].

Our prior conference work, XM [20], introduced depth-assisted scaled bundle adjustment with a scalable solver based on BM factorization [11] and Riemannian optimization [8], which XM4D extends to dynamic scenes.

Static Reconstruction with XM

This section presents the static stage of XM4D. Section 3.1 reviews scaled bundle adjustment and its QCQP reformulation (Propositions 1–2); Section 3.2 relaxes the QCQP to a convex SDP (Proposition 3); Section 3.3 solves the SDP with certified global optimality via the Riemannian staircase method (Algorithm 1). Section 3.4 highlights our main contribution compared to XM [20]: a uniform sphere constraint on the depth scales that replaces the first-frame anchor.

Bundle Adjustment with Learned Depth

The static observations originate from 2D correspondences, obtained by point trackers [16, 30, 85] or by classical keypoint detection and matching [43, 61], and uits∈R2u_{it}^{s} \in\mathbb{R}^{2} denotes the normalized image coordinate of static point ii in frame tt obtained with the camera intrinsics, which are either calibrated or estimated by a foundation model [38, 67]. Each observation is lifted to a noisy 3D measurement using its predicted depth dit>0d_{it}>0,

u~its=dit[uits1]∈R3,(3)\tilde{\mathbf{u}}_{it}^{s}=d_{it}\begin{bmatrix}\mathbf{u}_{it}^{s}\\1\end{bmatrix}\in\mathbb{R}^{3}, \tag*{(3)}

which constitutes the depth-lifted observation of Eq. (SBA).

The translations {ct}\{c_t\} and landmarks {xis}\{x_i^s\} in Eq. (SBA) are unconstrained variables and thus can be eliminated analytically to get a reduced scaled-rotation-only problem.

Proposition 1 (Scaled-Rotation-Only Formulation, adapted from [20]). Problem (SBA) is equivalent to

ρSRO⋆=min⁡st>0, Ct∈SO(3)t=1,…,Ttr⁡(QUTU),U=[s1C1 ⋯ sTCT],(4)\rho_{\mathrm{SRO}}^{\star}=\min_{\substack{s_t>0,\ C_t\in\mathrm{SO}(3)\\t=1,\ldots,T}}\operatorname{tr}(QU^{\mathsf{T}}U), \qquad U=\left[s_1C_1\ \cdots\ s_TC_T\right], \tag*{(4)}

where Q∈S3TQ\in\mathbb{S}^{3T}, Q⪰0Q\succeq0, is a data matrix built from the lifted measurements through the Moore–Penrose pseudo-inverse of the view-graph Laplacian. In particular, the optimal values coincide: ρSBA⋆=ρSRO⋆\rho_{\mathrm{SBA}}^{\star}=\rho_{\mathrm{SRO}}^{\star}, where ρSBA⋆\rho_{\mathrm{SBA}}^{\star} denotes the optimal value of (SBA). The closed form of QQ, the proof, and the recovery of the optimal translations and landmarks are given in Appendix § 11.§ 1.

The objective of (SRO) vanishes as U→0U\to0, a collapse of the global scale. XM excludes it by anchoring the first frame; we instead exclude it with the uniform sphere constraint ∥U∥F2=3T\lVert U\rVert_{F}^{2}=3T motivated in § 3.4.

By defining the set of scaled orthogonal matrices through the quadratic constraint

sO⁡(3)={Rˉ∈R3×3∣RˉTRˉ=αI3, α≥0}={sR∣s≥0, R∈O(3)},(5)\operatorname{sO}(3)=\left\{\bar{R}\in\mathbb{R}^{3\times3}\mid\bar{R}^{\mathsf{T}}\bar{R}=\alpha I_3,\ \alpha\geq0\right\} =\left\{sR\mid s\geq0,\ R\in\mathrm{O}(3)\right\}, \tag*{(5)}

which contains the scaled rotation Cˉt=stCt\bar{C}_t=s_tC_t of each frame, (SRO) reduces to the following QCQP.

Proposition 2 (QCQP Formulation, adapted from [20]). Consider the QCQP

ρQCQP⋆=min⁡Cˉt∈sO⁡(3)t=1,…,Ttr⁡(QUTU),U=[Cˉ1 ⋯ CˉT],∥U∥F2=3T,(6)\rho_{\mathrm{QCQP}}^{\star}=\min_{\substack{\bar{C}_t\in\operatorname{sO}(3)\\t=1,\ldots,T}}\operatorname{tr}(QU^{\mathsf{T}}U), \qquad U=\left[\bar{C}_1\ \cdots\ \bar{C}_T\right],\qquad \lVert U\rVert_{F}^{2}=3T, \tag*{(6)}

and let U⋆=[Cˉ1⋆,…,CˉT⋆]U^{\star}=[\bar{C}_1^{\star},\ldots,\bar{C}_T^{\star}] be a global optimizer. If det⁡Cˉt⋆>0\det\bar{C}_t^{\star}>0 for all tt, then U⋆U^{\star} is a global minimizer of (SRO) under the same sphere constraint. The proof is given in Appendix § 11.§ 2.

Otherwise, rotations in SO(3) are recovered by projecting each block onto the scaled rotations through its singular value decomposition (Appendix § 12.§ 1).

Convex SDP Relaxation

Since local solvers for the QCQP may stall in suboptimal minima, we use a semidefinite relaxation [75 Chapter 3]. The basic idea is fairly simple: creating a matrix variable

X=UTU∈S3T,(7)X=U^{\mathsf{T}}U\in\mathbb{S}^{3T}, \tag*{(7)}

which is quadratic in the original variable UU, makes the objective of (QCQP) linear in XX, namely tr⁡(QUTU)=⟨Q,X⟩\operatorname{tr}(QU^{\mathsf{T}}U)=\langle Q,X\rangle. The convex relaxation then enforces the matrix variable XX through convex constraints. Partition XX into 3×33\times3 blocks Xtt′X_{tt'} with t,t′∈{1,…,T}t,t'\in\{1,\ldots,T\}. Since each Cˉt=stCt\bar{C}_t=s_tC_t,

Xtt=CˉtTCˉt=st2I3=αtI3,αt=st2.(8)X_{tt}=\bar{C}_t^{\mathsf{T}}\bar{C}_t=s_t^2I_3=\alpha_tI_3,\qquad\alpha_t=s_t^2. \tag*{(8)}

All diagonal blocks play identical roles, and the sphere constraint becomes the linear equality tr⁡(X)=3T\operatorname{tr}(X)=3T. Ignoring the rank-3 constraint on XX yields Shor’s SDP relaxation.

Proposition 3 (SDP Relaxation, adapted from [20]). The semidefinite program

ρSDP⋆=min⁡X ⟨Q,X⟩\rho_{\mathrm{SDP}}^{\star}=\min_{X}\ \langle Q,X\rangle
s.t.X⪰0,Xtt=αtI3,t=1,…,T,tr⁡(X)=3T,\begin{aligned} \text{s.t.}\quad&X\succeq0,\\ &X_{tt}=\alpha_t I_3,\quad t=1,\ldots,T,\\ &\operatorname{tr}(X)=3T, \end{aligned}

where each scalar αt\alpha_t is not a separate variable but the common diagonal entry of the block XttX_{tt}, is a convex relaxation of (QCQP), i.e., ρSDP⋆≤ρQCQP⋆\rho_{\mathrm{SDP}}^{\star}\leq\rho_{\mathrm{QCQP}}^{\star}. If a global minimizer X⋆X^{\star} has rank⁡(X⋆)=3\operatorname{rank}(X^{\star})=3, then it factorizes as X⋆=(U⋆)⊤U⋆X^{\star}=(U^{\star})^{\top}U^{\star} with U⋆=[Cˉ1⋆,…,CˉT⋆]∈sO⁡(3)TU^{\star}=[\bar{C}_1^{\star},\ldots,\bar{C}_T^{\star}]\in\operatorname{sO}(3)^T, unique up to multiplication by O⁡(3)\operatorname{O}(3), and U⋆U^{\star} is a global optimizer of (QCQP). In this case, the relaxation is said to be tight or exact (proof in Appendix § 11.§ 3).

In practice the relaxation is empirically tight [20]; when a minimizer has rank above three, it is projected to rank three through its three largest eigenpairs, a process known as rounding (Appendix § 12.§ 1).

Burer–Monteiro Optimization

Although (SDP) is convex, explicitly storing and optimizing the 3T×3T3T \times3T matrix XX becomes prohibitive for long videos with a large TT: interior-point solvers already struggle beyond a few thousand frames [20]. When the relaxation is tight, however, the optimal X⋆X^{\star} has rank three (Proposition 3), so its effective dimension is only 3×3T3 \times3T. BM factorization [11] exploits this low-rank structure: for a chosen rank r≥3r \geq3, we substitute

X=U⊤U,U=[U1⋯UT]∈Rr×3T,(9)X=U^{\top}U,\qquad U=\begin{bmatrix}U_1&\cdots&U_T\end{bmatrix}\in\mathbb{R}^{r\times3T}, \tag*{(9)}

with blocks Ut∈Rr×3U_t\in\mathbb{R}^{r\times3} into (SDP), which yields the rank-rr BM problem

ρBM,r⋆=min⁡U∈Rr×3Ttr⁡(QU⊤U)s.t.Ut⊤Ut=αtI3,t=1,…,T,∑t=1Tαt=T.\begin{aligned} \rho_{\mathrm{BM},r}^{\star}&=\min_{U\in\mathbb{R}^{r\times3T}}\operatorname{tr}(QU^{\top}U)\\ \text{s.t.}\quad&U_t^{\top}U_t=\alpha_t I_3,\quad t=1,\ldots,T,\\ &\sum_{t=1}^{T}\alpha_t=T. \end{aligned}

At first glance, (BM) looks like a nonconvex constrained problem, but its constraints define a smooth manifold.

Proposition 4 (BM Riemannian Optimization, adapted from [20]). Problem (BM) is equivalent to the unconstrained Riemannian optimization problem

min⁡(s,Y1,…,YT)∈Mtr⁡(QU⊤U),Ut=stYt,(10)\min_{(s,Y_1,\ldots,Y_T)\in\mathcal{M}}\operatorname{tr}(QU^{\top}U),\qquad U_t=s_tY_t, \tag*{(10)}

over the product manifold

M=STT−1×St⁡(r,3)T,(11)\mathcal{M}=\mathbb{S}_{\sqrt{T}}^{T-1}\times\operatorname{St}(r,3)^T, \tag*{(11)}

where STT−1={s∈RT:∥s∥22=T}\mathbb{S}_{\sqrt{T}}^{T-1}=\{s\in\mathbb{R}^{T}:\lVert s\rVert_2^2=T\} is the radius-T\sqrt{T} sphere, St⁡(r,3)={Y∈Rr×3:Y⊤Y=I3}\operatorname{St}(r,3)=\{Y\in\mathbb{R}^{r\times3}:Y^{\top}Y=I_3\} is the Stiefel manifold, and αt=st2\alpha_t=s_t^2. The proof is given in Appendix § 11.§ 4.

We solve (7) with the Riemannian trust-region method with truncated conjugate gradients (RTR-TCG) [7], using the C++/CUDA implementation from XM [20].

Three observations are in order [11, 75]. First, X=U⊤U⪰0X=U^{\top}U\succeq0 holds by construction, so the PSD constraint disappears. Second, at r=3r=3, problem (BM) is exactly (QCQP) and hence nonconvex. Third, increasing the rank cannot increase the optimal objective value, since any feasible rank-rr factor remains feasible at rank r+1r+1 by appending a zero row; hence

ρBM,3⋆≥ρBM,4⋆≥⋯≥ρSDP⋆,(12)\rho_{\mathrm{BM},3}^{\star}\geq\rho_{\mathrm{BM},4}^{\star}\geq\cdots\geq\rho_{\mathrm{SDP}}^{\star}, \tag*{(12)}

with equality ρBM,r⋆=ρSDP⋆\rho_{\mathrm{BM},r}^{\star}=\rho_{\mathrm{SDP}}^{\star} as soon as r≥r⋆r\geq r^{\star}, the minimum rank among optimal solutions of (SDP) [11, 75]. We emphasize that (9) is a statement about global minima: each ρBM,r⋆\rho_{\mathrm{BM},r}^{\star} is the optimal value of the nonconvex problem (BM), which a local Riemannian solver alone is not guaranteed to attain. Attaining it—and thereby ρSDP⋆\rho_{\mathrm{SDP}}^{\star}—is precisely the role of the Riemannian staircase [8, 60] developed below.

Global optimality certification. Write (SDP) in standard primal form,

min⁡X∈S3Ttr⁡(QX)(13)\min_{X\in\mathbb{S}^{3T}} \operatorname{tr}(QX) \tag*{(13)}
s.t.⟨Ai,X⟩=bi,i=1,…,m,(14)\text{s.t.}\quad\langle A_i,X\rangle=b_i,\quad i=1,\ldots,m, \tag*{(14)}
X⪰0.(15)X\succeq0. \tag*{(15)}

The associated dual standard SDP is

max⁡y∈Rm∑i=1mbiyi(16)\max_{y\in\mathbb{R}^{m}}\sum_{i=1}^{m}b_i y_i \tag*{(16)}
s.t.Z(y):=Q−∑i=1myiAi⪰0.(17)\text{s.t.}\quad Z(y):=Q-\sum_{i=1}^{m}y_iA_i\succeq0. \tag*{(17)}

Since Slater’s condition holds (X=I3TX=I_{3T} is feasible for (SDP)), strong duality holds between (10) and (11) [71].

Theorem 5 (Dual Optimality Certificate, adapted from [20]). Let Ur⋆U_r^\star be a locally optimal solution of (BM) with αt>0\alpha_t>0 for all tt. Then there exists a unique multiplier yr∈Rmy_r\in\mathbb{R}^{m} satisfying the first-order condition

Z(yr)(Ur⋆)T=0.(18)Z(y_r)(U_r^\star)^{\mathsf{T}}=0. \tag*{(18)}

If, furthermore, Z(yr)⪰0Z(y_r)\succeq0, then Ur⋆U_r^\star is a global optimizer of (BM), and (Ur⋆)TUr⋆,yr,Z(yr)(U_r^\star)^{\mathsf{T}}U_r^\star,y_r,Z(y_r) are optimal for the primal–dual pair (10)–(11). The proof is given in Appendix B.5.

Theorem 5 gives a simple certification recipe: solve the linear system (12) for yry_r (recall Z(y)Z(y) is linear in yy), form Z(yr)Z(y_r), and check its positive semidefiniteness. If the check fails, the next theorem provides an escape route.

Theorem 6 (Negative-Curvature Direction, adapted from [20]). Let Ur⋆U_r^\star be a local minimizer of (BM) with multiplier yry_r, and suppose Z(yr)⪰̸0Z(y_r)\not\succeq0. Let vv be an eigenvector of Z(yr)Z(y_r) associated with a negative eigenvalue. Then, for the rank-(r+1)(r+1) problem,

D=[0vT] is a second-order descent direction at Uˉ=[Ur⋆0]:(19)D=\begin{bmatrix}0\\v^{\mathsf{T}}\end{bmatrix}\text{ is a second-order descent direction at }\bar{U}=\begin{bmatrix}U_r^\star\\0\end{bmatrix}: \tag*{(19)}

along the feasible retraction curve through Uˉ\bar{U} in the direction DD, the objective ff satisfies f′(0)=0f'(0)=0 and f′′(0)=2vTZ(yr)v<0f''(0)=2v^{\mathsf{T}}Z(y_r)v<0. The proof is given in Appendix B.6.

Since the objective decreases to second order along DD, the halving line search in Algorithm 1 terminates with a strictly lower objective, and the optimization escapes the rank-rr local minimizer by climbing one step of the rank “staircase”.

Theorem 7 (Global Convergence, adapted from [20]). Algorithm 1 returns a globally optimal solution of the SDP pair (10)–(11), regardless of the initialization Ur0U_r^0 (proof in Appendix B.7).

Theorem 7 makes the approach initialization-free. In practice, Algorithm 1 terminates at r=3r=3 or 4 [20], so the worst case r=3Tr=3T is never approached and the scalability advantage of the low-rank factorization is retained.

Anchoring via a Uniform Sphere Constraint

As discussed in § 3.1, the original XM formulation removes the scale and rotation ambiguities of (SBA) by anchoring the first frame (C1=I3C_1=I_3, c1=0c_1=0, s1=1s_1=1). This pins the reconstruction to the first depth prediction, so the solver overfits to it when the first-frame observations are noisy, and XM’s scale-regularization adds a hyperparameter that must be tuned for each video. We instead fix the global scale with a constraint rather than a penalty: the uniform Algorithm 1: Riemannian Staircase

Input: data matrix Q∈S3T\mathcal{Q} \in\mathbb{S}^{3T}

Output: certified optimizer U⋆U^{\star} of (BM)

  1. r←3;Ur0←[I3 ⋯ I3];r \leftarrow3;\quad U_r^0 \leftarrow[I_3\ \cdots\ I_3];

  2. while true do

  3. Ur⋆←RTR-TCG⁡(Q,Ur0);U_r^{\star} \leftarrow\operatorname{RTR\text{-}TCG}(\mathcal{Q},U_r^0); \hfill // local minimizer of (BM)

  4. solve (12) for yry_r and form Z(yr)Z(y_r);

  5. if Z(yr)⪰0Z(y_r) \succeq0 then

  6. \hspace{1em}return U⋆=Ur⋆;U^{\star}=U_r^{\star}; \hfill // certified global optimum

  7. end

  8. v←v \leftarrow eigenvector of the most negative eigenvalue of Z(yr)Z(y_r);

  9. α←1;Uˉ←[Ur⋆0];D←[0vT];\alpha\leftarrow1;\quad\bar{U}\leftarrow\begin{bmatrix} U_r^{\star}\\ 0 \end{bmatrix};\quad D\leftarrow\begin{bmatrix} 0\\ v^{\mathsf{T}} \end{bmatrix};

  10. Ur+1←Retr⁡Uˉ(αD);U_{r+1}\leftarrow\operatorname{Retr}_{\bar{U}}(\alpha D);

  11. while tr⁡(QUr+1TUr+1)≥tr⁡(Q(Ur⋆)TUr⋆)\operatorname{tr}(\mathcal{Q}U_{r+1}^{\mathsf{T}}U_{r+1}) \ge\operatorname{tr}(\mathcal{Q}(U_r^{\star})^{\mathsf{T}}U_r^{\star}) do

  12. \hspace{1em}α←α/2;Ur+1←Retr⁡Uˉ(αD);\alpha\leftarrow\alpha/2;\quad U_{r+1}\leftarrow\operatorname{Retr}_{\bar{U}}(\alpha D);

  13. end

  14. r←r+1;Ur0←Ur+1;r\leftarrow r+1;\quad U_r^0\leftarrow U_{r+1};

  15. end

sphere constraint

∑t=1Tst2=T(20)\sum_{t=1}^{T}s_t^2=T \tag*{(20)}

excludes the collapse toward zero while treating every frame symmetrically. It is quadratic in the scales, so (QCQP) remains a QCQP, and after relaxation it is the trace constraint tr⁡(X)=3T\operatorname{tr}(X)=3T already in (SDP) and (BM).

Suboptimality gap. Since the staircase runs with finite tolerances, we report the relative gap

η=ρ^−ρd1+∣ρ^∣+∣ρd∣,ρd=bTyr+3Tλmin⁡(Z(yr)),(21)\eta=\frac{\hat{\rho}-\rho_d}{1+\lvert\hat{\rho}\rvert+\lvert\rho_d\rvert},\qquad\rho_d=\mathbf{b}^{\mathsf{T}}y_r+3T\lambda_{\min}(Z(y_r)), \tag*{(21)}

where ρ^=tr⁡(QU^TU^)\hat{\rho}=\operatorname{tr}(\mathcal{Q}\hat{U}^{\mathsf{T}}\hat{U}) is the objective of the solution U^\hat{U} rounded to (SRO) under the sphere constraint (Appendix § 12.§ 1), b=(b1,…,bm)\mathbf{b}=(b_1,\ldots,b_m) and Z(y)Z(y) are defined in (10)–(11b), and yry_r solves (12). By Theorem 11 (Appendix § 12), ρ^≥ρSRO≥ρSDP≥ρd\hat{\rho}\ge\rho_{\mathrm{SRO}}\ge\rho_{\mathrm{SDP}}\ge\rho_d, so η→0\eta\to0 certifies that U^\hat{U} is numerically globally optimal for (SRO). Table 6 reports η\eta per dataset: all static solves satisfy η<10−3\eta<10^{-3}, where for the partial sphere constraint η\eta uses the trace-restricted bound of Appendix § 12.

DatasetStaticDynamic (EDG)
scenesmedian η\etanokn_{\mathrm{ok}}scenesmedian η\etanokn_{\mathrm{ok}}
Sintel239.9×10−129.9 \times10^{-12}23/23–––
TUM63.6×10−113.6 \times10^{-11}6/6–––
Replica89.7×10−109.7 \times10^{-10}8/8–––
BAL33.9×10−53.9 \times10^{-5}3/3–––
IMC65.5×10−125.5 \times10^{-12}6/6–––
DyCheck710−1210^{-12}7/7710−1210^{-12}700/700
NVIDIA710−1210^{-12}7/773.5×10−113.5 \times10^{-11}511/511
Syn4D810−1210^{-12}8/8810−1210^{-12}800/800

Table 6. Median relative suboptimality η\eta of (21) over every scene of each dataset. nokn_{\mathrm{ok}} counts solves with η<10−3\eta< 10^{-3}. When Z(yr)Z(y_{\mathrm{r}}) is PSD and the solution has rank three with all blocks of positive determinant, the solve is certified optimal for the original problem, and we report η=10−12\eta= 10^{-12}, the numerical resolution of the eigenvalue solve. Otherwise, the static η\eta uses the trace-restricted bound of Appendix § 12. For the dynamic solver, each entry is one segment solve (its final certificate), and the median and nokn_{\mathrm{ok}} are over all segment solves.

Summary

The static stage lifts 2D point tracks to 3D with learned depth, eliminates translations and landmarks to obtain (SRO), and relaxes it through (QCQP) to the convex program (SDP), solved by the Riemannian staircase with a certificate of global optimality and a suboptimality gap (15). Unlike XM, a sphere constraint fixes the global scale without anchoring any frame. The recovered camera poses (Ct,ct,st)(C_t,c_t,s_t) define the world frame in which dynamic tracks are lifted for the non-rigid stage.

Dynamic Reconstruction with XM

We extend the globally certifiable optimization framework of § 3 to the dynamic part of the view graph. § 5 introduces the EDG motion model and dynamic objective; § 5.1 analytically eliminates the unconstrained variables to obtain a rotation-only QCQP, as in the static case; and § 5.2 describes how the XM solver of § 3.3 solves the resulting problem.

EDG Model and the Dynamic Objective

EDG Model and the Dynamic Objective

Dynamic motion is represented by an EDG [65], GEDG=(VEDG,EEDG)\mathcal{G}_{\mathrm{EDG}} = (\mathcal{V}_{\mathrm{EDG}}, \mathcal{E}_{\mathrm{EDG}}), consisting of JJ sparse control nodes (Eq. (2)). Each node jj has a rest position gj∈R3\mathbf{g}_j \in\mathbb{R}^{3} and a rigid transformation (Rj,t,pj,t)∈SO(3)×R3(R_{j,t}, \mathbf{p}_{j,t}) \in\mathrm{SO}(3) \times\mathbb{R}^{3} describing its motion at frame tt. Dynamic points are attached to nearby nodes and deformed using linear blend skinning. In this section, the graph (its nodes, edges, rest positions, and skinning weights) is taken as given; § 6.3 describes how it is built from the lifted dynamic tracks.

We first describe the EDG model for a single temporal window W\mathcal{W}. Within this window, we choose a reference frame t0∈Wt_0 \in\mathcal{W}, typically the frame with the most observed EDG nodes, and define the rest positions in that frame, which keeps the skinning offsets and ARAP constraints consistent. We anchor the node transformations at t0t_0 by

Rj,t0=I3,pj,t0=0,j=1,…,J,(22)R_{j,t_0} = I_3,\qquad\mathbf{p}_{j,t_0} = \mathbf{0},\qquad j = 1,\ldots,J, \tag*{(22)}

so that Φt0\Phi_{t_0} is the identity. This is a local reference-frame convention for one solve; it does not require a single rest frame for the entire video (§ 6.3).

Since the cameras and depth scales are fixed at their static-stage estimates, the lifted observations yitd\mathbf{y}_{it}^{d} are constants, and the dynamic problem estimates the node transformations alone,

min⁡Rj,t∈SO(3),pj,t∈R3λtrackLtrack+λARAPLARAP+λtempLtemp.(23)\min_{\substack{R_{j,t}\in\mathrm{SO}(3),\\ \mathbf{p}_{j,t}\in\mathbb{R}^{3}}}\lambda_{\mathrm{track}}\mathcal{L}_{\mathrm{track}}+\lambda_{\mathrm{ARAP}}\mathcal{L}_{\mathrm{ARAP}}+\lambda_{\mathrm{temp}}\mathcal{L}_{\mathrm{temp}}. \tag*{(23)}

Here Ltrack\mathcal{L}_{\mathrm{track}} is the tracking loss ((2)), which asks the warped rest positions to match the lifted observations; the as-rigid-as-possible (ARAP) energy over the EDG edges,

LARAP=∑t=1T∑(j,k)∈EEDG∥rjk,tarap∥2,\mathcal{L}_{\mathrm{ARAP}}=\sum_{t=1}^{T}\sum_{(j,k)\in\mathcal{E}_{\mathrm{EDG}}}\left\lVert\mathbf{r}^{\mathrm{arap}}_{jk,t}\right\rVert^{2},
rjk,tarap=Rj,t(gk−gj)+gj+pj,t−gk−pk,t,(24)\mathbf{r}^{\mathrm{arap}}_{jk,t}=R_{j,t}(\mathbf{g}_{k}-\mathbf{g}_{j})+\mathbf{g}_{j}+\mathbf{p}_{j,t}-\mathbf{g}_{k}-\mathbf{p}_{k,t}, \tag*{(24)}

asks the transformation of node jj to also explain the motion of its neighbor kk, and the temporal smoothness energy

Ltemp=∑t=1T−1∑j=1J∥Rj,t−Rj,t+1∥F2+∥pj,t−pj,t+1∥2(25)\mathcal{L}_{\mathrm{temp}}=\sum_{t=1}^{T-1}\sum_{j=1}^{J}\left\lVert R_{j,t}-R_{j,t+1}\right\rVert_{F}^{2}+\left\lVert\mathbf{p}_{j,t}-\mathbf{p}_{j,t+1}\right\rVert^{2} \tag*{(25)}

discourages large inter-frame deformation.

Rotation-Only QCQP

Every residual in (DYN) is affine: the tracking residual is affine in (Rj,t,pj,t)(R_{j,t},\mathbf{p}_{j,t}) for fixed skinning weights and rest positions, the ARAP residual is affine in (Rj,t,pj,t,pk,t)(R_{j,t},\mathbf{p}_{j,t},\mathbf{p}_{k,t}) with the constant offset gj−gk\mathbf{g}_{j}-\mathbf{g}_{k}, and the smoothness residuals are linear. The objective is thus quadratic in the stacked variables, and (DYN) is a QCQP whose only constraints are the orthogonality of the EDG blocks.

As in the static case, the node translations pj,t\mathbf{p}_{j,t} are unconstrained Euclidean variables and can be eliminated analytically, leaving a problem over the EDG rotations alone.

Proposition 8 (Rotation-Only Formulation of the Dynamic Problem). Under the rest-frame convention ((22)), Problem (DYN) is equivalent to

ρDYN−RO⋆=min⁡Rj,t∈SO(3)t≠t0tr⁡(QUTU)+2⟨P,U⟩+c0,\rho_{\mathrm{DYN-RO}}^{\star}=\min_{\substack{R_{j,t}\in\mathrm{SO}(3)\\t\ne t_0}}\operatorname{tr}(Q U^{\mathsf{T}}U)+2\langle P,U\rangle+c_0,
U=[Rj,t]t≠t0,U=[R_{j,t}]_{t\ne t_0},

where Q∈S3J(T−1)Q\in\mathbb{S}^{3J(T-1)}, Q⪰0Q\succeq0, P∈R3×3J(T−1)P\in\mathbb{R}^{3\times3J(T-1)}, and c0∈Rc_0\in\mathbb{R} are built from the lifted observations, the skinning weights, and the rest positions through the Moore–Penrose pseudo-inverse of a node-translation Laplacian. In particular, the optimal values coincide: ρDYN⋆=ρDYN−RO⋆\rho_{\mathrm{DYN}}^{\star}=\rho_{\mathrm{DYN-RO}}^{\star}, where ρDYN⋆\rho_{\mathrm{DYN}}^{\star} denotes the optimal value of (DYN). The closed forms of QQ, PP, and c0c_0, and the proof, and the recovery of the node translations are in Appendix § 11.8.

Two differences from the static problem (SRO) are worth noting. First, a linear term ⟨P,U⟩\langle P,U\rangle appears: the ARAP and tracking residuals carry the constant offsets gj−gk\mathbf{g}_j-\mathbf{g}_k, ∑jwijgj\sum_j w_{ij}\mathbf{g}_j, and yitd\mathbf{y}_{it}^{d}, which survive the elimination. Second, every block is a unit-scale rotation, whose relaxed block constraint is Xjj=I3X_{jj}=I_3 rather than αI3\alpha I_3, so no sphere normalization is needed.

Homogenization. The XM solver operates on a homogeneous quadratic objective, so we absorb the linear term and the constant of (DYN-RO) into a single matrix by introducing one auxiliary rotation V∈SO(3)V\in\mathrm{SO}(3) and defining

Uh=[VU],Qh=[c03I3PPTQ].(26)U_h=\begin{bmatrix}V & U\end{bmatrix},\qquad Q_h=\begin{bmatrix}\frac{c_0}{3}I_3 & P\\ P^{\mathsf T} & Q\end{bmatrix}. \tag*{(26)}

The auxiliary block adds no physical degree of freedom; after solving, the EDG rotations are taken relative to it.

Proposition 9 (Homogeneous Formulation of the Dynamic Problem). Problem (DYN-RO) is equivalent to

ρDYN−H⋆=min⁡V∈SO(3),Rj,t∈SO(3)t≠t0tr⁡(QhUhTUh),\rho^\star_{\mathrm{DYN-H}}= \min_{\substack{V\in\mathrm{SO}(3),\,R_{j,t}\in\mathrm{SO}(3)\\t\ne t_0}} \operatorname{tr}(Q_hU_h^{\mathsf T}U_h),

(DYN-H)

in the following sense: the optimal values coincide, ρDYN−RO⋆=ρDYN−H⋆\rho^\star_{\mathrm{DYN-RO}}=\rho^\star_{\mathrm{DYN-H}}; if (V⋆,U⋆)(V^\star,U^\star) is optimal for (DYN-H), then V⋆TU⋆V^{\star\mathsf T}U^\star is optimal for (DYN-RO); and if U~⋆\widetilde{U}^\star is optimal for (DYN-RO), then (I3,U~⋆)(I_3,\widetilde{U}^\star) is optimal for (DYN-H) (proof in Appendix B.9).

Solving with the XM Solver

As in Proposition 2, we drop the determinant constraints of (DYN-H), replacing SO(3)\mathrm{SO}(3) by O(3)\mathrm{O}(3). The resulting QCQP has orthogonal blocks only, so Section 3.3 carries over with n+1n+1 unit-scale blocks, n=J(T−1)n=J(T-1), and no sphere constraint.

Proposition 10 (Certifiable Optimization of the Dynamic Problem). The SDP

ρDYN-SDP⋆=min⁡X∈S3(n+1)⟨Qh,X⟩\rho^\star_{\mathrm{DYN\text{-}SDP}}=\min_{X\in\mathbb{S}^{3(n+1)}} \langle Q_h,X\rangle
s.t.X⪰0,Xii=I3,i=0,…,n.(27)\begin{aligned} \text{s.t.}\quad&X\succeq0,\\ &X_{ii}=I_3,\quad i=0,\ldots,n. \tag*{(27)} \end{aligned}

is a convex relaxation of (DYN-H). Its rank-rr BM factorization X=UhTUhX=U_h^{\mathsf{T}}U_h is equivalent to the unconstrained Riemannian problem

min⁡Uh∈St⁡(r,3)n+1tr⁡(QhUhTUh).(28)\min_{U_h\in\operatorname{St}(r,3)^{n+1}}\operatorname{tr}(Q_hU_h^{\mathsf{T}}U_h). \tag*{(28)}

Theorems 5–7 still hold for (DYN-SDP)–(DYN-BM), so Algorithm 1 returns a certified global optimum of (DYN-SDP), and hence of (DYN-H) whenever the relaxation is tight and the recovered relative rotation blocks lie in SO⁡(3)\operatorname{SO}(3). The proof is given in Appendix B.10.

Global optimality matters in practice. Figure 3 compares the certified solution with that of a local solver on the same input: the local solver converges to a local minimum in which some nodes are rotated incorrectly, whereas the certified solution recovers consistent node rotations.

Suboptimality gap. The gap (15) applies verbatim to (DYN-SDP), with QQ replaced by QhQ_h, AiA_i and bib_i now encoding Xii=I3X_{ii}=I_3, and 3T3T replaced by tr⁡(X)=3(n+1)\operatorname{tr}(X)=3(n+1) (Appendix C). Every segment solve on DyCheck, NVIDIA, and Syn4D is certified (Table 6).

Summary. The dynamic stage models non-rigid motion within each temporal segment with an EDG whose transformations are expressed relative to a single rest frame, which makes the tracking, ARAP, and temporal smoothness objective (DYN) quadratic. Eliminating the node translations yields the rotation-only problem (DYN-RO), whose linear term is absorbed by one auxiliary rotation into the homogeneous problem (DYN-H). Its SDP relaxation is solved with the same BM staircase as the static stage, with a certificate of global optimality and a suboptimality gap, whereas a local solver can stall in a wrong local minimum.

The XM4D Pipeline

This section describes the end-to-end system that recovers both the static and the dynamic structure of the view graph in Figure 2. We present it for a monocular video, but the formulation only requires that each image carry a time tag, and an image collection without time tags reduces the pipeline to static 3D reconstruction. The stages are preprocessing and tracking (Section 5.1), the static stage (Section 5.2), EDG construction, segment solving and completion (Section 5.3), and the final photometric stage (Section 5.4).

Preprocessing: Depth, Intrinsics, Masks, and Tracks

Every frame is processed by a monocular depth foundation model, with Depth Anything 3 (DA3) [38] as the default, which provides the metric depths ditd_{it} used in the lift (2) and, for uncalibrated videos, the camera intrinsics. Per-frame masks of the moving objects are obtained with SAM 3 [14], prompted either by a text description or by a few hand-selected keypoints on the dynamic object and then tracked across the video. Finally, both static and dynamic 2D correspondences are produced by point trackers, with TAPNext [85] as the default. The trackers are operated in an online fashion: as each frame arrives, new query points are sampled only at locations farther than a threshold from existing tracks. Unreliable observations are discarded (Appendix E). Optionally, dynamic tracks can be assembled into long-range trajectories for the EDG (Appendix E.5).

Static Stage: Certified Solve

The static stage solves (SBA) with the solver of Section 3, augmented with two robustness steps. First, we run XM2^2 [20]: the solver is applied twice, with an outlier filter in between that keeps only the observations with the smallest residuals under the first solution, so tracking failures or inaccurate depth predictions do not corrupt the final estimate.

Second, the result is refined with Ceres [2], warm-started at the certified XM2^2 outputs and minimizing the classical 2D reprojection error. Further implementation details of the static stage are collected in Appendix § 14.

Partial sphere constraint. In practice, some videos contain a small fraction of outlier frames—strong blur or large inter-frame motion, or depth predictions whose scale is far off—and forcing such frames onto the sphere lets them distort the global scale. We therefore use a mixed-manifold variant of the solver: frames are ranked by the trace of their diagonal block of QQ, the top ⌈ρT⌉\lceil\rho T \rceil frames are constrained to the sphere of § 3.4, and the scales of the remaining frames are optimized freely. Formally, with SS the set of these frames, the sphere constraint becomes

∑t∈Sst2=∣S∣,\sum_{t \in S} s_t^2 = |S|,

i.e.,

∑t∈Str⁡(Xtt)=3∣S∣\sum_{t \in S} \operatorname{tr}(X_{tt}) = 3|S|

in (SDP), which is still linear, so the SDP relaxation and the certificate of Theorem 5 carry over. The suboptimality gap (21) is adapted accordingly in Appendix § 12. In the same spirit, we optionally normalize the observation weights per frame: with wt=∑iωitw_t = \sum_i \omega_{it} the total weight of frame tt, the solver uses the raw weights ωit\omega_{it}, equalizes every frame’s influence (ωit/wt)(\omega_{it}/w_t), or applies a softer normalization (ωit/wt)(\omega_{it}/\sqrt{w_t}). Table 1 compares eight such configurations; configuration (6) is our default.

EDG Construction, Segment Solve, and Completion

Starting from the lifted dynamic tracks of § 4, we first build a time-independent graph connecting tracks that remain spatially close over time. We then subsample this graph to obtain sparse EDG nodes and inherit its connectivity to define the EDG edges. Because no single frame observes all nodes in long videos, we solve the EDG motion independently on short overlapping segments using the certified solver of § 5.2, and fuse the segment solutions. Finally, missing EDG poses are filled by a completion step.

Track graph. The track graph Gtr\mathcal{G}_{\mathrm{tr}} records which tracks stay on the same part of the object. A single-frame distance cannot decide this, since points on different parts may pass close to each other in some frames. We therefore measure the distance of two tracks over their whole common lifetime. Let Ti\mathcal{T}_i be the frames in which track ii is reliable, Tii′=Ti∩Ti′\mathcal{T}_{ii'} = \mathcal{T}_i \cap\mathcal{T}_{i'} the frames in which tracks ii and i′i' are co-visible, and πit∈R2\boldsymbol{\pi}_{it} \in\mathbb{R}^2 the pixel position of track ii in frame tt. We define the percentile distance of two tracks as the β\beta-th percentile of their per-frame distance over co-visible frames,

dii′=perc⁡β{∥yitd−yi′td∥}t∈Tii′,dii′2D=perc⁡β{∥πit−πi′t∥}t∈Tii′.(29)d_{ii'} = \operatorname{perc}_{\beta}\left\{ \left\lVert y^d_{it} - y^d_{i't} \right\rVert\right\}_{t \in\mathcal{T}_{ii'}}, \qquad d^{2\mathrm{D}}_{ii'} = \operatorname{perc}_{\beta}\left\{ \left\lVert\boldsymbol{\pi}_{it} - \boldsymbol{\pi}_{i't} \right\rVert\right\}_{t \in\mathcal{T}_{ii'}}. \tag*{(29)}

in the world frame and in the image, respectively. We write i∼(τ,τ2D)i′i \sim_{(\tau,\tau_{2\mathrm{D}})} i' if

∣Tii′∣≥max⁡(nmin⁡,γmin⁡(∣Ti∣,∣Ti′∣)),\left|\mathcal{T}_{ii'}\right| \ge\max\left(n_{\min}, \gamma\min\left(\left|\mathcal{T}_i\right|,\left|\mathcal{T}_{i'}\right|\right)\right),
dii′≤τordii′2D≤τ2D,(30)d_{ii'} \le\tau\quad\text{or} \quad d^{2\mathrm{D}}_{ii'} \le\tau_{2\mathrm{D}}, \tag*{(30)}

Photometric Stage

that is, the tracks overlap in time and stay close in 3D or 2D, and connect them in GtrG_{\mathrm{tr}}; the thresholds τ\tau and τ2D\tau_{\mathrm{2D}} are proportional to the median nearest-neighbor distance of the tracks in 3D and in the image. Each track keeps at most Dmax⁡D_{\max} edges with the smallest dii′d_{ii'}.

EDG nodes and edges. The same criterion selects the EDG nodes. Tracks are ranked by their length ∣Ti∣|\mathcal{T}_i|, and the longest remaining track jj becomes a node and suppresses every remaining track ii that stays on the same part of the object during their common lifetime, i.e., with i∼(τE,τE,2D)ji \sim_{(\tau_{\mathrm{E}},\tau_{\mathrm{E},2\mathrm{D}})} j in (30), with thresholds and overlap fraction that differ from those of the track graph, until every track is either a node or suppressed by one. If this yields more than Jmax⁡J_{\max} nodes, Jmax⁡J_{\max} of them are kept by farthest-point sampling under the same percentile distance. Two nodes jj, j′j' are connected if they are within kk hops on the track graph, which keeps independently moving parts apart. A node left with fewer than three edges is connected to its nearest nodes that are reachable on the track graph.

Skinning. Every other dynamic track ii is attached to nodes N(i)\mathcal{N}(i) chosen among the nodes within two hops of it on the track graph; an isolated track with no node within two hops falls back to its nearest node in percentile distance. The skinning weights wijw_{ij} are the normalized inverse distances from the track to its nodes at the rest frame. The graph and weights are fixed before solving, so the certificate covers the EDG motion given this graph.

Segment solve and fusion. For a long sequence, we apply the single-window formulation of § 4 to overlapping windows {Wk}\{\mathcal{W}_k\}. Each window has its own reference frame t0(k)t_0^{(k)}, rest positions, and locally estimated transformations (Rj,t(k),pj,t(k))(R_{j,t}^{(k)}, p_{j,t}^{(k)}) obtained by solving (DYN) to certified global optimality with Proposition 10. We fuse the estimates over overlapping windows by averaging the estimated world-space node positions for each frame. We recover globally consistent rotations by synchronizing the relative rotations estimated within each window; see Appendix § 14.§ 4.

Completion. Node-frame pairs that remain empty after fusion are filled by a local version of (DYN): over all nodes and frames, we minimize the same objective, with a robust 2D reprojection error as the tracking term, plus an initialization loss that keeps the solution close to the fused table, using a local solver initialized from it. This yields a complete trajectory (Rj,t,pj,t)(R_{j,t}, p_{j,t}) for every node and frame.

Photometric Stage

The recovered geometry initializes a dynamic Gaussian-splatting representation [31]. We use gsplat [80] as the photometric training engine and add an EDG wrapper on top of it, which warps every dynamic Gaussian to frame tt by linear blend skinning with the EDG transformations. During training, the rest positions of the dynamic Gaussians are optimized together with the EDG transformations (Rj,t,pj,t)(R_{j,t}, p_{j,t}), initialized from the certified solve, under a photometric loss and depth supervision.

Experiments

We compare XM4D with Structure-from-Motion (SfM) and 4D reconstruction baselines on seven static and dynamic benchmarks in camera accuracy, runtime, and novel-view rendering, and extend the qualitative evaluation to additional videos (Figure 4); Figure 5 summarizes the quantitative comparison.

Figure 4. Qualitative results of XM4D. Sequences from DyCheck [18], NVIDIA [81], Syn4D, DAVIS [54], Sintel [12], EgoDexter [46], a Boston Dynamics video [6], and a generated video from Google Flow. For each sequence, from left to right: an input frame, the reconstructed static and dynamic structure, the 3D trajectories of the dynamic points over the video, a rendered novel view, and the rendered depth.

Figure 5. Performance overview. Each axis is the square-root-compressed ratio to the best method, averaged over the datasets of Table 2–Table 5. Open polygons mark axes a method does not report. Farther out is better. XM4D performs best overall, leading on four of the five axes.

Setup

Datasets. We use seven benchmarks. On four of them—IMC [28], Replica [63, 89], TUM [64], and Sintel [12]—we evaluate camera accuracy only. IMC and Replica are static, while TUM and Sintel contain moving objects. On the remaining three—Syn4D, NVIDIA [81], and the DyCheck iPhone dataset [18]—we additionally run the EDG solve and the photometric reconstruction of § 6. Syn4D is a synthetic benchmark that we assemble from Replica [63] and RoboCasa [47]. Each dataset is described in Appendix § 13. The solver-configuration study in Table 1 additionally uses three problems from BAL [1].

DatasetMetric12345678
BALATE-T2.2432.1942.1632.1852.2182.1552.1812.283
RPE-T4.0033.9423.9343.9363.9353.9573.9493.979
RPE-R1.6741.6611.6501.6671.6551.6601.7121.690
SintelATE-T5353545152564974
RPE-T72717569717469105
RPE-R76717066675256121
Sintel (Ceres)ATE-T2930291618182930
RPE-T3537372528263643
RPE-R62636148564149107
TUM (Ceres)ATE-T9.458.8010.488.709.215.727.3311.42
RPE-T12.8512.2214.5112.4212.998.0210.0416.23
RPE-R6.636.187.696.286.645.985.877.05

Table 1. Mean camera accuracy (×10−3\times10^{-3}) of eight solver configurations of Section 5.2, each determined by the sphere fraction ρ\rho and the weight normalization: (1)–(5) raw weights at ρ=0.1\rho= 0.1, 0.3, 0.5, 0.9, and 1; (6) ρ=0.9\rho= 0.9 with the per-frame normalization ωit/wt\omega_{it}/w_t; (7) ρ=0.9\rho= 0.9 with the softer normalization ωit/wt\omega_{it}/\sqrt{w_t}; and (8) the original XM solver [20] with raw weights. Configuration 6, our default, achieves the best or second-best accuracy in eight of the twelve rows.

Baselines. For camera estimation, we compare against the classical SfM pipeline COLMAP [61], the feed-forward geometry model VGGT [67], and MegaSAM [37], which adapts DROID-SLAM [66] for camera pose and depth estimation in casual dynamic videos. For dynamic reconstruction, we compare against the geometry-based methods MoSca [35] and Shape of Motion [68]. For novel-view rendering, we additionally compare against rendering-based methods that use time-varying fields (T-NeRF [18], NSFF [36], CTNeRF [45], DyBluRF [10], and RoDynRF [41]), deformable canonical radiance fields (Nerfies [51], HyperNeRF [52]), or moving Gaussians (Gaussian Marbles [62]), and against DynPoint [86] and PGDVS [83], which render from depth-warped points.

Hardware. All experiments run on a workstation with an AMD Ryzen Threadripper PRO 7985WX CPU (64 cores, 128 threads), 256 GB of RAM, and two 48 GB NVIDIA RTX 6000 Ada GPUs, under Ubuntu 22.04 with CUDA 12.8. Every method, including ours, runs on a single GPU.

Metrics. We report the absolute trajectory error in translation (ATE-T) and the relative pose error in translation and rotation (RPE-T, RPE-R). The estimated camera centers c^t\hat{c}_t are first aligned to the ground-truth centers ctc_t by a similarity transformation (σ,R,t)(\sigma, R, t) estimated robustly with TEASER++ [77]. The translation errors are normalized by d+1d+1, where d=1T∑t∥ct−cˉ∥d=\frac{1}{T}\sum_t\lVert c_t-\bar{c}\rVert is the mean distance of the ground-truth centers from their centroid:

ATE-T=1d+11T∑t∥σR^c^t+t−ct∥,\mathrm{ATE\text{-}T}=\frac{1}{d+1}\frac{1}{T}\sum_t\lVert\sigma\hat{R}\hat{c}_t+t-c_t\rVert,
RPE-T=1d+11∣P∣∑(t,t′)∈P∥(ct−ct′)−σR(c^t−c^t′)∥,\mathrm{RPE\text{-}T}=\frac{1}{d+1}\frac{1}{|\mathcal{P}|}\sum_{(t,t')\in\mathcal{P}}\lVert(c_t-c_{t'})-\sigma R(\hat{c}_t-\hat{c}_{t'})\rVert,

where P\mathcal{P} contains all frame pairs, and RPE-R is the mean angle, in radians, of the relative rotation error over the same pairs. The constant 1, in ground-truth units, prevents division by a near-zero dd when the camera barely moves. For novel-view synthesis on DyCheck, NVIDIA, and Syn4D we report mPSNR, mSSIM, and mLPIPS or their alternates. Following MoSca [35], we refine each test camera photometrically against its test image with the scene frozen.

Camera Accuracy

Table 2 summarizes camera accuracy across all seven benchmarks. XM4D is the most accurate method on Replica and TUM, and in translation on Sintel; on Syn4D, NVIDIA, and DyCheck it is close to the best method, MegaSAM; and on the unordered image collections of IMC, XM4D is outperformed only by the dedicated SfM pipeline COLMAP. Figure 6 shows the trajectories on one scene per benchmark.

DatasetMetricCOLMAPVGGTMegaSAMMoScaOurs
IMCATE-T0.0030.0750.871–0.047
RPE-T0.0050.1211.221–0.074
RPE-R0.0020.0330.703–0.044
ReplicaATE-T0.0090.0140.0670.0600.002
RPE-T0.0120.0190.0870.0720.002
RPE-R0.0150.0050.5740.0070.001
TUMATE-T0.0060.0080.0070.0090.006
RPE-T0.0090.0120.0090.0130.008
RPE-R0.0060.0060.0060.0070.006
SintelATE-T0.3590.0660.0490.0670.018
RPE-T0.4620.0950.0580.0940.026
RPE-R0.0810.0570.0190.0560.041
Syn4DATE-T0.0350.0090.0010.0220.002
RPE-T0.0600.0110.0010.0310.002
RPE-R0.0220.0120.0010.0570.002
NVIDIAATE-T0.1500.1110.0240.1740.038
RPE-T0.2060.1680.0360.2590.056
RPE-R0.0100.0100.0050.1070.005
DyCheckATE-T0.0060.0190.0050.0480.008
RPE-T0.0080.0260.0070.0650.011
RPE-R0.0090.0350.0070.0240.012

Table 2. Camera accuracy averaged per dataset. MoSca [35] crashes on IMC [28] (–), whose unordered photo collections are not video data. VGGT [67] is run on subsampled frames on DyCheck [18] (§ 7.3).

Figure 6. Estimated camera trajectories. Estimated camera poses (blue) after Sim(3) alignment to the ground truth (red), shown for one scene per benchmark. IMC [28] consists of unordered images, whereas all other benchmarks contain temporally ordered sequences.

COLMAP is the most accurate on IMC, the unordered setting it is designed for, and remains competitive on Replica and TUM, but degrades sharply on Sintel, Syn4D, and NVIDIA, where its ATE-T is 44–20×20\times that of XM4D. The feed-forward VGGT is roughly three times less accurate in translation. MegaSAM, designed for dynamic videos, is the most accurate on Syn4D, NVIDIA, and DyCheck, with XM4D close behind, but is less accurate on Replica and TUM and in translation on Sintel and fails on IMC, whose images lack the temporal continuity its optical-flow front end relies on. MoSca is 33–10×10\times less accurate than XM4D in ATE-T on the dynamic benchmarks and also fails on IMC; we attribute the gap to the XM solver, which recovers the cameras by a certifiably optimal SDP on static tracks alone, so errors in the deformation model cannot propagate into the trajectory. Overall, XM4D ranks first on Replica and TUM and in translation on Sintel, second on Syn4D, NVIDIA, and in translation on IMC.

Runtime

Static stage. Table 3 reports the runtime of camera estimation and, where available, its split into preprocessing (depth, masks, and tracking) and the solver. End to end, XM4D is faster than MoSca on every benchmark and up to 9×9\times faster than COLMAP on the long sequences, while the feed-forward VGGT is the fastest overall, as expected for a single network pass. Against MegaSAM the totals are mixed: XM4D is faster on IMC, NVIDIA, and DyCheck, and slower on Replica, TUM, Sintel, and Syn4D.

DatasetStageCOLMAPVGGTMegaSAMMoScaOurs
IMCTotal227.738.7136.8–23.2
ReplicaTotal125.454.483.51054.4115.1
Preprocess––49.6888.5100.5
Solver––33.9165.914.6
TUMTotal1743.752.879.5403.883.7
Preprocess––50.8288.373.7
Solver––28.5115.310.0
SintelTotal206.029.348.4410.262.0
Preprocess––34.1363.247.2
Solver––14.347.014.8
Syn4DTotal605.055.082.61729.3110.3
Preprocess––52.0724.688.4
Solver––30.61004.621.9
NVIDIATotal25173511128
Preprocess––289225
Solver––7183
DyCheckTotal8433831694296
Preprocess211–14955578
Solver631–16738718
Replica room0N=100N=10036.437.689.6468.573.7
N=200N=200225.8OOM149.0892.6131.3
N=500N=5001451.9–345.02140.1383.5
N=800N=8003230.0–568.6err862.3
N=1200N=1200>1h–err–1656.2
N=2000N=2000––––3322.4

Table 3. Runtime of camera estimation. Top: runtime per scene in seconds, averaged per dataset and split into preprocessing (depth, masks, tracking) and the solver. Bottom: wall-clock time in seconds on Replica [63] room0 as the number of input frames NN grows (contiguous prefix); a run is invalid if it exceeds the GPU memory (OOM), runs longer than 1 h (>1h), or crashes (err). Best in bold, second best underlined.

The preprocessing relies on foundation models and will speed up as they improve, so we focus on the solver, where XM4D is the fastest on every benchmark but Sintel.

Dynamic stage. Table 5 reports the runtime of the full dynamic reconstruction against MoSca and Shape of Motion.

DatasetMethodPreprocessOptimizationTotal
DyCheckShape of Motion230.0282.4512.4
DyCheckMoSca9.231.340.5
DyCheckOurs8.825.834.6
NVIDIAShape of Motion0.648.348.8
NVIDIAMoSca1.532.133.6
NVIDIAOurs1.520.421.9
Syn4DShape of Motion20.050.971.0
Syn4DMoSca12.132.344.5
Syn4DOurs6.610.416.9

Table 5. Per-scene runtime of the full dynamic reconstruction in minutes (averages over the scenes of each dataset), split into preprocessing (depth, masks, static and dynamic tracking) and optimization (camera estimation, the motion model, and photometric training). Best in bold.

XM4D is the fastest end to end and in optimization on all three benchmarks. Shape of Motion is the slowest, mainly because its dense point tracking takes about four hours per scene on DyCheck.

Sequence length. Table 3 (bottom) reports the runtime on Replica room0 versus the number of frames NN. VGGT runs out of GPU memory at N=200N = 200, MoSca fails beyond N=500N = 500, and beyond N=800N = 800 COLMAP exceeds one hour and MegaSAM its 1024-frame buffer, whereas XM4D completes every length, reaching N=2000N = 2000 in 55 minutes. VGGT’s limit is also why it runs on subsampled frames on DyCheck, whose sequences exceed 100 frames.

Novel-View Rendering

Table 4 and Figure 7 evaluate novel-view synthesis on DyCheck, NVIDIA, and Syn4D. XM4D attains the best average score on every metric of all three datasets. This holds although most baselines are given the ground-truth camera poses (all except MoSca and RoDynRF, and Shape of Motion on Syn4D), and Gaussian Marbles on Syn4D additionally the ground-truth depth, whereas XM4D estimates all cameras. The margins are largest on Syn4D, where XM4D improves PSNR by more than 2 dB over the best baseline, and on DyCheck, where it has the best mSSIM and mLPIPS on all seven scenes and the best mPSNR on three. As Figure 7 shows, the baselines lose mainly because inaccurate geometry misaligns their renderings with the test view and blurs the dynamic parts. Since the recovered dynamic motion is hard to compare directly across methods, we further evaluate the geometry on Syn4D, whose moving objects have ground-truth geometry, through the Chamfer distance of the dynamic object and the error of the rendered depth (Table 7); XM4D is the most accurate on every scene.

DyCheckmPSNR ↑mSSIM ↑mLPIPS ↓
T-NeRF (w)16.960.5770.379
NSFF (w)15.460.5510.396
Nerfies (w)16.450.5700.339
HyperNeRF (w)16.810.5690.332
DynPoint (w)16.890.573–
PGDVS (w)15.880.5480.340
DyBluRF (w)17.370.5910.373
CTNeRF (w)17.690.531–
Shape of Motion (w)17.320.5980.296
Gaussian Marbles (w)16.72–0.413
RoDynRF (w/o)17.100.5340.517
MoSca (w/o)18.910.6770.284
Ours (w/o)19.510.7250.163
NVIDIAPSNR ↑LPIPS ↓
NSFF (w)24.330.199
HyperNeRF (w)17.600.367
DynPoint (w)26.530.068
CTNeRF (w)26.130.082
RoDynRF (w)25.890.067
RoDynRF (w/o)25.380.079
MoSca (w/o)26.540.073
Ours (w/o)27.010.063
Syn4DmPSNR ↑mLPIPS ↓
Gaussian Marbles (w)19.480.348
Shape of Motion (w/o)16.720.418
MoSca (w/o)19.490.235
Ours (w/o)21.730.148

Table 4. Novel-view rendering averaged over the scenes of each dataset; (w)/(w/o): with/without ground-truth camera poses. XM4D is the best on every metric although it estimates its camera poses.

Novel-view rendering on DyCheck, NVIDIA, and Syn4D

Figure 7. Novel-view rendering on DyCheck [18], NVIDIA [81], and Syn4D. For each method, the rendered test view is shown with its error map with respect to the ground truth (brighter is larger error). (a) DyCheck: from top to bottom, block, paper-windmill, spin, and wheel, with error maps displayed only inside the co-visibility mask. The other methods score lower mainly because inaccurate 3D geometry misaligns their renderings with the test view, and because they blur the dynamic parts. (b) NVIDIA: from top to bottom, Jumping, Playground, Skating, and Truck, with error maps over the full frame. (c) Syn4D: from top to bottom, two Bear [4] and two RoboCasa [47] sequences, with the rendered depth also shown.

MetricMethodbear1bear2bear3bear4casa1casa2casa3casa4Mean
ChamferOurs6.15.46.44.611.29.27.68.67.4
MoSca27.329.175.245.6112.0125.150.260.465.6
SoM34.4214.790.8172.438.333.369.32429.9385.4
Train depthOurs0.0120.0120.0260.0150.0770.0290.0540.0450.034
MoSca0.0310.0290.0350.0290.1230.0490.0810.0690.056
SoM0.1020.0960.1040.0960.1930.0820.1580.1390.121
DA0.2040.2080.2600.2400.4690.2370.3150.2200.269
Novel depthOurs0.0500.0940.0940.0850.1010.0380.0550.0530.071
MoSca0.2960.3290.0990.2920.1710.1820.1210.0870.197
SoM0.3210.3220.2730.3340.2530.1430.1880.2100.255
DA0.2100.2860.3680.2960.4580.2550.3570.2620.312

Table 7. Geometry accuracy on Syn4D per scene: Chamfer distance (×10−3\times10^{-3}) between the reconstructed and the ground-truth dynamic object, and relative L1L_1 error of the rendered depth in the training and novel views; lower is better. All methods are scored with the same protocol: the Chamfer distance follows the camera-based similarity alignment with one global scale per scene chosen to minimize it, and the depth uses one median scale per scene, fitted on the training views and reused for the novel views. SoM is Shape of Motion [68], and DA renders the Depth Anything 3 [38] depth lifted with its cameras.

Qualitative Results

Figure 4 shows XM4D on dynamic videos beyond the benchmarks: in-the-wild videos from DAVIS [54], Sintel [12], egocentric hand–object interaction videos from EgoDexter [46], a Boston Dynamics humanoid robot performing a flip [6], and a generated video of a walking lion from Google Flow. These videos span real, synthetic, and generated footage, third-person and egocentric viewpoints, and the motion of animals, people, hands, and a robot; XM4D handles all of them, demonstrating the robustness of the pipeline.

For videos of human body and hands, we build the EDG from keypoints instead of subsampling the track graph. For hands, WiLoR [56] predicts a MANO [59] hand in every frame; we keep only its 21 3D joints and connect them by the kinematic skeleton of the hand. For bodies, RTMPose [26] detects the 17 COCO [39] keypoints in 2D within the

bounding box of the person’s mask.

Because the EDG represents motion with explicit SE(3)SE(3) transformations, XM4D recovers the part-level rigid motion of objects without an extra motion-estimation step. Such part-level articulation is what simulation-ready digital twins of articulated objects require, especially in robotics [22, 27, 34, 58, 87]. Figure 8 shows a refrigerator whose four doors are opened in turn: the reconstruction yields the opening angle and displacement of every door over time and its hinge axis.

Figure 8. Articulated object reconstruction with XM4D. A person opens the four doors of a refrigerator in turn in a casual monocular video. (a) Input frames. (b) The four moving doors, color-coded. (c) Rendered views of the reconstruction at the same time instants. (d) Rotation angle and translation of each door over the video. (e) Reconstructed geometry with the four doors and their estimated hinge axes.

Summary

XM4D gives the best overall results for three reasons. First, its certified cameras make it the most accurate on Replica and TUM and in translation on Sintel, and close to the best elsewhere. Second, its solver is the fastest on nearly every benchmark, making XM4D the fastest 4D reconstruction end to end, and scales to 2000 frames. Third, its cameras and EDG motion give the photometric stage an accurate scaffold, yielding the best novel views on all three dynamic benchmarks, even against methods with ground-truth poses.

Conclusions

We presented XM4D, a 4D reconstruction framework that recovers camera motion and non-rigid deformation through certifiably globally optimal geometric optimization. Separating the two stages turns both into QCQPs over (scaled) rotations: the static stage estimates camera poses, depth scales, and static landmarks, and the dynamic stage estimates the motion of an embedded deformation graph. Both QCQPs are relaxed to SDPs and solved by a GPU solver that returns a certificate of global optimality. Across seven benchmarks, XM4D estimates accurate cameras on both dynamic videos and unordered photo collections, scales to thousands of frames, and initializes dynamic Gaussian splatting for state-of-the-art novel-view rendering.

Limitations. The certificate covers the optimization, not its inputs: XM4D relies on the predicted monocular depth and point tracks, and errors that survive the filters of Appendix § 14.§ 2 bias the certified solution. The solver stores the data matrices densely, so its memory grows quadratically with the number of frames; exploiting their sparsity [50] is a natural next step. Finally, the EDG structure and skinning weights are fixed before the solve; estimating the graph jointly with its motion is future work.

Acknowledgements

This work was partially funded by Office of Naval Research grant N00014-25-1-2322.

Additional Tables

Table 6 reports the suboptimality gaps per dataset. Table 7 compares the reconstructed geometry on Syn4D with MoSca [35], Shape of Motion [68], and Depth Anything 3 [38]. For the Chamfer distance, the dynamic object at frame tt is represented by the centers of the dynamic Gaussians at tt (for XM4D, warped by the EDG); 1000 points are drawn uniformly at random from them and from the ground-truth object point cloud, and the distance is the sum of the two mean squared nearest-neighbor distances, averaged over the 100 frames.

Table 6 Median relative suboptimality η\eta of (21) over every scene of each dataset. nokn_{\mathrm{ok}} counts solves with η<10−3\eta< 10^{-3}. When Z(yr)Z(y_{\mathrm{r}}) is PSD and the solution has rank three with all blocks of positive determinant, the solve is certified optimal for the original problem, and we report η=10−12\eta= 10^{-12}, the numerical resolution of the eigenvalue solve. Otherwise, the static η\eta uses the trace-restricted bound of Appendix § 12. For the dynamic solver, each entry is one segment solve (its final certificate), and the median and nokn_{\mathrm{ok}} are over all segment solves.

Table 7 Geometry accuracy on Syn4D per scene: Chamfer distance (×10−3\times10^{-3}) between the reconstructed and the ground-truth dynamic object, and relative L1L_1 error of the rendered depth in the training and novel views; lower is better. All methods are scored with the same protocol: the Chamfer distance follows the camera-based similarity alignment with one global scale per scene chosen to minimize it, and the depth uses one median scale per scene, fitted on the training views and reused for the novel views. SoM is Shape of Motion [68], and DA renders the Depth Anything 3 [38] depth lifted with its cameras.

Proofs

Proof of Proposition 1

Proof. Let et∈RT\mathbf{e}_t \in\mathbb{R}^{T} and eˉi∈RNs\bar{\mathbf{e}}_i \in\mathbb{R}^{N_s} denote standard basis vectors. All vectors are columns; collect the unknown translations and landmarks as the columns of the matrix

Y=[c1⋯cTx1s⋯xNss]∈R3×(T+Ns).(31)Y = [\mathbf{c}_1 \quad\cdots\quad\mathbf{c}_T \quad\mathbf{x}^{s}_1 \quad\cdots\quad\mathbf{x}^{s}_{N_s}] \in\mathbb{R}^{3 \times(T+N_s)}. \tag*{(31)}

For each observation (i,t)∈Es(i,t) \in\mathcal{E}_s define

ait=et⊗u~its∈R3T,bit=[et−eˉi]∈RT+Ns,(32)\mathbf{a}_{it} = \mathbf{e}_t \otimes\tilde{\mathbf{u}}^{s}_{it} \in\mathbb{R}^{3T}, \qquad\mathbf{b}_{it} = \begin{bmatrix} \mathbf{e}_t \\ -\bar{\mathbf{e}}_i \end{bmatrix} \in\mathbb{R}^{T+N_s}, \tag*{(32)}

so that, with U=[s1C1 ⋯ sTCT]U=[s_1C_1\ \cdots\ s_TC_T], each residual of Eq. (SBA) reads Ct(stu~is)+ct−xis=Uait+YbitC_t(s_t\tilde{u}_i^s)+c_t-x_i^s=Ua_{it}+Yb_{it}. The objective of Eq. (SBA) is therefore the quadratic form

f(U,Y)=tr⁡(Q1UTU)+2tr⁡(VTUTY)+tr⁡(LGYTY),(33)f(U,Y)=\operatorname{tr}(Q_1U^\mathsf{T}U)+2\operatorname{tr}(V^\mathsf{T}U^\mathsf{T}Y)+\operatorname{tr}(L_GY^\mathsf{T}Y), \tag*{(33)}

with

Q1=∑(i,t)∈EsωitaitaitT∈S3T,(34)Q_1=\sum_{(i,t)\in\mathcal{E}_s}\omega_{it}a_{it}a_{it}^\mathsf{T}\in\mathbb{S}^{3T}, \tag*{(34)}
V=∑(i,t)∈EsωitaitbitT∈R3T×(T+Ns),(35)V=\sum_{(i,t)\in\mathcal{E}_s}\omega_{it}a_{it}b_{it}^\mathsf{T}\in\mathbb{R}^{3T\times(T+N_s)}, \tag*{(35)}
LG=∑(i,t)∈EsωitbitbitT∈ST+Ns.(36)L_G=\sum_{(i,t)\in\mathcal{E}_s}\omega_{it}b_{it}b_{it}^\mathsf{T}\in\mathbb{S}^{T+N_s}. \tag*{(36)}

By construction, LGL_G is the weighted Laplacian of the bipartite view graph G\mathcal{G} on the T+NsT+N_s vertices {frames}∪{landmarks}\{\text{frames}\}\cup\{\text{landmarks}\} with edge weights ωit\omega_{it} (cf. [20]): bitT1=0b_{it}^\mathsf{T}\mathbf{1}=0 implies LG1=0L_G\mathbf{1}=0, and when G\mathcal{G} is connected the nullspace of LGL_G is exactly span⁡(1)\operatorname{span}(\mathbf{1}). Likewise, V1=∑ωitait(bitT1)=0V\mathbf{1}=\sum\omega_{it}a_{it}(b_{it}^\mathsf{T}\mathbf{1})=0.

For fixed UU, ff is a convex quadratic in YY, bounded below by zero, so its minimizers are the solutions of the stationarity condition YLG=−UVYL_G=-UV. This linear system is consistent: V1=0V\mathbf{1}=0 places the rows of −UV-UV in the range of LGL_G. Its minimum-norm solution is

Y⋆=−UVLG+,(37)Y^\star=-UVL_G^+, \tag*{(37)}

and every other solution differs by v1Tv\mathbf{1}^\mathsf{T}. Substituting (37) into (33) and using LG+LGLG+=LG+L_G^+L_GL_G^+=L_G^+ gives

min⁡Yf(U,Y)=tr⁡(QUTU),Q=Q1−VLG+VT,(38)\min_Y f(U,Y)=\operatorname{tr}(QU^\mathsf{T}U),\qquad Q=Q_1-VL_G^+V^\mathsf{T}, \tag*{(38)}

the generalized Schur complement of LGL_G in ∑ωit[ait;bit][ait;bit]T⪰0\sum\omega_{it}[a_{it};b_{it}][a_{it};b_{it}]^\mathsf{T}\succeq0; since the columns of VTV^\mathsf{T} lie in range⁡(LG)\operatorname{range}(L_G), Q⪰0Q\succeq0. Minimizing both sides of (38) over {st>0, Ct∈SO(3)}\{s_t>0,\ C_t\in\mathrm{SO}(3)\} proves the claimed equivalence, and (37) recovers the optimal translations and landmarks from U⋆U^\star.

Proof of Proposition 2

Writing a generic 3×33\times3 matrix as Cˉ=[a1,a2,a3]\bar{C}=[a_1,a_2,a_3], membership in the set (5) of scaled orthogonal matrices is described by the quadratic equalities

a1Ta1=a2Ta2=a3Ta3,(39)a_1^\mathsf{T}a_1=a_2^\mathsf{T}a_2=a_3^\mathsf{T}a_3, \tag*{(39)}
a1Ta2=a2Ta3=a3Ta1=0,(40)a_1^\mathsf{T}a_2=a_2^\mathsf{T}a_3=a_3^\mathsf{T}a_1=0, \tag*{(40)}

which, together with the quadratic sphere constraint ∥U∥F2=3T\lVert U\rVert_F^2=3T, are the constraints of (QCQP).

Proof. We first characterize the feasible set. If Cˉ=[a1,a2,a3]\bar{C}=[a_1,a_2,a_3] satisfies the quadratic equalities (39)–(40), then CˉTCˉ=σ2I3\bar{C}^\mathsf{T}\bar{C}=\sigma^2I_3 with σ2=a1Ta1≥0\sigma^2=a_1^\mathsf{T}a_1\geq0. If σ>0\sigma>0, then Cˉ/σ∈O(3)\bar{C}/\sigma\in\mathrm{O}(3) and Cˉ=σR\bar{C}=\sigma R; if σ=0\sigma=0, then Cˉ=0\bar{C}=0. The quadratic constraints therefore describe exactly the set (5), including the zero matrix, and the feasible set of (QCQP) is {U:Cˉt∈sO(3), ∥U∥F2=3T}\{U:\bar{C}_t\in s\mathrm{O}(3),\ \lVert U\rVert_F^2=3T\}. This set contains the feasible set of (SRO) under the sphere constraint: for st>0s_t>0 and Ct∈SO(3)C_t\in\mathrm{SO}(3), each block Cˉt=stCt\bar{C}_t=s_tC_t lies in sO(3)s\mathrm{O}(3), and ∥U∥F2=∑t3st2=3T\lVert U\rVert_F^2=\sum_t3s_t^2=3T holds if and only if ∑tst2=T\sum_ts_t^2=T. The QCQP thus relaxes (SRO) in exactly two ways: reflections are admitted (SO(3) is relaxed to O(3)), and individual blocks may vanish. Since the objectives coincide on the common feasible set,

ρQCQP⋆=tr⁡(Q(U⋆)TU⋆)≤ρSRO⋆.(41)\rho_{\mathrm{QCQP}}^\star=\operatorname{tr}(Q(U^\star)^\mathsf{T}U^\star)\leq\rho_{\mathrm{SRO}}^\star. \tag*{(41)}

Now let U⋆U^\star be a global optimizer of (QCQP) with det⁡Cˉt⋆>0\det\bar{C}_t^\star>0 for all tt. The determinant condition rules out both relaxations at once: every block is nonzero, so Cˉt⋆=st⋆Ct⋆\bar{C}_t^\star=s_t^\star C_t^\star with st⋆=∥Cˉt⋆∥F/3>0s_t^\star=\lVert\bar{C}_t^\star\rVert_F/\sqrt{3}>0 and Ct⋆∈O(3)C_t^\star\in\mathrm{O}(3), and

det⁡Ct⋆=det⁡Cˉt⋆(st⋆)3>0,(42)\det C_t^\star=\frac{\det\bar{C}_t^\star}{(s_t^\star)^3}>0, \tag*{(42)}

i.e., Ct⋆∈SO(3)C_t^\star\in\mathrm{SO}(3). Moreover, ∑t(st⋆)2=∥U⋆∥F2/3=T\sum_t(s_t^\star)^2=\lVert U^\star\rVert_F^2/3=T, so {st⋆,Ct⋆}t=1T\{s_t^\star,C_t^\star\}_{t=1}^T is feasible for (SRO) under the sphere constraint, which gives tr⁡(Q(U⋆)TU⋆)≥ρSRO⋆\operatorname{tr}(Q(U^\star)^\mathsf{T}U^\star)\geq\rho_{\mathrm{SRO}}^\star. Combined with (41), equality holds, and U⋆U^\star is a global minimizer of (SRO). Finally, since (GU)T(GU)=UTU(GU)^\mathsf{T}(GU)=U^\mathsf{T}U, GU⋆GU^\star is equally optimal for every G∈O(3)G\in\mathrm{O}(3), and by (37) the recovered translations and landmarks rotate with it; the normalization U⋆←(C1⋆)TU⋆U^\star\leftarrow(C_1^\star)^\mathsf{T}U^\star selects the representative that aligns the first camera with the identity.

Proof of Proposition 3

Proof. Relaxation. Let U=[Cˉ1,…,CˉT]U=[\bar{C}_1,\ldots,\bar{C}_T] be feasible for (QCQP) and set X=UTUX=U^{\mathsf{T}}U. Then each diagonal block is Xtt=CˉtTCˉt=αtI3X_{tt}=\bar{C}_t^{\mathsf{T}}\bar{C}_t=\alpha_t I_3 by (5); and tr⁡(X)=∥U∥F2=3T\operatorname{tr}(X)=\lVert U\rVert_F^2=3T. Hence XX is feasible for (SDP) with the same objective value ⟨Q,X⟩=tr⁡(QUTU)\langle Q,X\rangle=\operatorname{tr}(QU^{\mathsf{T}}U), and minimizing over UU gives ρSDP⋆≤ρQCQP⋆\rho_{\mathrm{SDP}}^{\star}\leq\rho_{\mathrm{QCQP}}^{\star}.

Tightness. Let X⋆X^{\star} be a global minimizer of (SDP) with rank⁡(X⋆)=3\operatorname{rank}(X^{\star})=3, and write its eigendecomposition as X⋆=∑k=13λkvkvkTX^{\star}=\sum_{k=1}^{3}\lambda_kv_kv_k^{\mathsf{T}} with λk>0\lambda_k>0. The matrix U⋆∈R3×3TU^{\star}\in\mathbb{R}^{3\times3T} with rows λkvkT\sqrt{\lambda_k}v_k^{\mathsf{T}} satisfies X⋆=(U⋆)TU⋆X^{\star}=(U^{\star})^{\mathsf{T}}U^{\star}. Partitioning U⋆=[Cˉ1⋆,…,CˉT⋆]U^{\star}=[\bar{C}_1^{\star},\ldots,\bar{C}_T^{\star}] into 3×33\times3 blocks, the block constraints give (Cˉt⋆)TCˉt⋆=Xtt⋆=αtI3(\bar{C}_t^{\star})^{\mathsf{T}}\bar{C}_t^{\star}=X_{tt}^{\star}=\alpha_tI_3, so Cˉt⋆∈SO(3)\bar{C}_t^{\star}\in\mathrm{SO}(3) by (5), and the trace constraint gives ∥U⋆∥F2=tr⁡(X⋆)=3T\lVert U^{\star}\rVert_F^2=\operatorname{tr}(X^{\star})=3T. Thus U⋆U^{\star} is feasible for (QCQP), and

ρQCQP⋆≤tr⁡(Q(U⋆)TU⋆)=⟨Q,X⋆⟩=ρSDP⋆≤ρQCQP⋆,(43)\rho_{\mathrm{QCQP}}^{\star}\leq\operatorname{tr}\left(Q(U^{\star})^{\mathsf{T}}U^{\star}\right)=\langle Q,X^{\star}\rangle=\rho_{\mathrm{SDP}}^{\star}\leq\rho_{\mathrm{QCQP}}^{\star}, \tag*{(43)}

so equality holds and U⋆U^{\star} is a global optimizer of (QCQP).

Proof of Proposition 4

Proof. Let UU be feasible for (BM). Each block satisfies UtTUt=αtI3U_t^{\mathsf{T}}U_t=\alpha_tI_3 with αt≥0\alpha_t\geq0; set st=αts_t=\sqrt{\alpha_t}. If st>0s_t>0, then Yt=Ut/stY_t=U_t/s_t satisfies YtTYt=I3Y_t^{\mathsf{T}}Y_t=I_3, i.e., Yt∈St(r,3)Y_t\in\mathrm{St}(r,3); if st=0s_t=0, then Ut=0U_t=0 and any Yt∈St(r,3)Y_t\in\mathrm{St}(r,3) gives Ut=stYtU_t=s_tY_t. The trace constraint ∑tαt=T\sum_t\alpha_t=T reads ∥s∥22=T\lVert s\rVert_2^2=T, so (s,Y1,…,YT)∈M(s,Y_1,\ldots,Y_T)\in\mathcal{M}. Conversely, every point of M\mathcal{M} gives blocks Ut=stYtU_t=s_tY_t with UtTUt=st2I3U_t^{\mathsf{T}}U_t=s_t^2I_3 and ∑tst2=T\sum_ts_t^2=T, which is feasible for (BM) with αt=st2\alpha_t=s_t^2. The map (s,Y1,…,YT)↦U(s,Y_1,\ldots,Y_T)\mapsto U is therefore onto the feasible set of (BM), and the two problems share the objective tr⁡(QUTU)\operatorname{tr}(QU^{\mathsf{T}}U), so they are equivalent.

Proof of Theorem 5

Proof. We first make the constraint matrices {Ai}i=1m\{A_i\}_{i=1}^m of (10) explicit. For each frame tt, the block constraint Xtt=αtI3X_{tt}=\alpha_tI_3 for some αt\alpha_t is equivalent to the five homogeneous equations ⟨A5(t−1)+ℓ,X⟩=0\langle A_{5(t-1)+\ell},X\rangle=0, ℓ=1,…,5\ell=1,\ldots,5, where A5(t−1)+ℓ∈S3TA_{5(t-1)+\ell}\in\mathbb{S}^{3T} is zero except for its tt-th diagonal 3×33\times3 block, which equals

B1=[1000−10000],B2=[00001000−1],B3=[010100000],B^1=\begin{bmatrix}1&0&0\\0&-1&0\\0&0&0\end{bmatrix},\quad B^2=\begin{bmatrix}0&0&0\\0&1&0\\0&0&-1\end{bmatrix},\quad B^3=\begin{bmatrix}0&1&0\\1&0&0\\0&0&0\end{bmatrix},
B^4=\begin{bmatrix}0&0&1\\0&0&0\\1&0&0\end{bmatrix},\quad B^5=\begin{bmatrix}0&0&0\\0&0&1\\0&1&0\end{bmatrix}. \] {xref}`(35)`

The {Bℓ}ℓ=15\{B^\ell\}_{\ell=1}^5 form a basis of the traceless symmetric 3×33\times3 matrices. The trace constraint tr⁡(X)=3T\operatorname{tr}(X)=3T of (SDP) corresponds to Am=I3TA_m=I_{3T} with bm=3Tb_m=3T.

LICQ. The gradient of the ii-th constraint of the BM problem at UU is 2AiUT2A_iU^{\mathsf{T}}. Suppose ∑i=1mγiAiUT=0\sum_{i=1}^m\gamma_iA_iU^{\mathsf{T}}=0 for some coefficients γ∈Rm\gamma\in\mathbb{R}^m, evaluated at U=Ur⋆U=U_r^{\star}. The matrix A5(t−1)+ℓUTA_{5(t-1)+\ell}U^{\mathsf{T}} is supported on rows 3t−2,…,3t3t-2,\ldots,3t only, while AmUT=UTA_mU^{\mathsf{T}}=U^{\mathsf{T}} contributes to those same rows; collecting the rows of block tt therefore gives

(∑ℓ=15γtℓBℓ+γmI3)UtT=0,t=1,…,T,(44)\left(\sum_{\ell=1}^{5}\gamma_{t\ell}B^\ell+\gamma_mI_3\right)U_t^{\mathsf{T}}=0,\qquad t=1,\ldots,T, \tag*{(44)}

where γtℓ=γ5(t−1)+ℓ\gamma_{t\ell}=\gamma_{5(t-1)+\ell}. Since αt>0\alpha_t>0, the block UtU_t has full column rank; multiplying (44) on the right by UtU_t and using UtTUt=αtI3U_t^{\mathsf{T}}U_t=\alpha_tI_3 yields ∑ℓγtℓBℓ+γmI3=0\sum_\ell\gamma_{t\ell}B^\ell+\gamma_mI_3=0. Taking the trace gives 3γm=03\gamma_m=0 because every BℓB^\ell is traceless, so γm=0\gamma_m=0; linear independence of {Bℓ}\{B^\ell\} then forces γtℓ=0\gamma_{t\ell}=0 for all t,ℓt,\ell. Hence the constraint gradients {2AiUr⋆}i=1m\{2A_iU_r^{\star}\}_{i=1}^m are linearly independent, and LICQ holds at Ur⋆U_r^{\star}.

First-order condition. By LICQ, there exists a unique Lagrange multiplier yr∈Rmy_r\in\mathbb{R}^m at which the Lagrangian of (BM) is stationary:

Q(Ur⋆)T−∑i=1m(yr)iAi(Ur⋆)T=Z(yr)(Ur⋆)T=0,(45)Q(U_r^{\star})^{\mathsf{T}}-\sum_{i=1}^{m}(y_r)_iA_i(U_r^{\star})^{\mathsf{T}}=Z(y_r)(U_r^{\star})^{\mathsf{T}}=0, \tag*{(45)}

which is exactly (18).

Global optimality. Suppose in addition Z(yr)⪰0Z(y_r)\succeq0. Then yry_r is feasible for the dual (11), and X⋆=(Ur⋆)TUr⋆X^{\star}=(U_r^{\star})^{\mathsf{T}}U_r^{\star} is feasible for the primal (10). Their objective values coincide:

tr⁡(QX⋆)−∑i=1mbi(yr)i=⟨Z(yr),X⋆⟩=tr⁡(Z(yr)(Ur⋆)TUr⋆)=0,(46)\begin{aligned} \operatorname{tr}(QX^{\star})-\sum_{i=1}^{m}b_i(y_r)_i=\langle Z(y_r),X^{\star}\rangle\\ &=\operatorname{tr}\left(Z(y_r)(U_r^{\star})^{\mathsf{T}}U_r^{\star}\right)=0, \tag*{(46)} \end{aligned}

using primal feasibility in the first equality and (18) in the last. The duality gap is zero, so X⋆X^{\star} and yry_r are optimal for the primal–dual pair, and Ur⋆U_r^{\star} attains ρSDP⋆≤ρBM,r⋆\rho_{\mathrm{SDP}}^{\star}\leq\rho_{\mathrm{BM},r}^{\star}, hence it is a global optimizer of (BM) [11, 75].

Proof of Theorem 6

Proof. See [11] or [75].

Proof of Theorem 7

Proof. In the worst case r=3Tr = 3T, every local optimizer of (BM) corresponds to a globally optimal solution of the SDP [29 Corollary 8]: if the local optimizer is rank-deficient, global optimality follows from [11 Proposition 4]; if it is full-rank, then (18) forces Z(yr)=0Z(y_r) = 0, which is positive semidefinite and again certifies global optimality.

Proof of Proposition 8

Proof. Let Ω={(j,t):j=1,…,J, t≠t0}\Omega= \{(j,t) : j = 1,\ldots,J,\ t \ne t_0\} be the free node-frame pairs of (DYN), n=∣Ω∣=J(T−1)n = |\Omega| = J(T-1), and fix an ordering of Ω\Omega, the one used in UU. Let ejt∈Rne_{jt} \in\mathbb{R}^{n} denote the standard basis vector of the pair (j,t)∈Ω(j,t) \in\Omega, set ejt0=0e_{jt_0} = 0, and let δtt0\delta_{tt_0} be the Kronecker delta. Collect the unknown node translations as the columns of

Y=[pj,t](j,t)∈Ω∈R3×n,(47)Y = [p_{j,t}]_{(j,t)\in\Omega} \in\mathbb{R}^{3\times n}, \tag*{(47)}

in the same order as UU. Then, for every node jj, frame tt, and vector v∈R3v \in\mathbb{R}^{3}, the rest-frame convention (22) gives

Rj,tv=U(ejt⊗v)+δtt0v,pj,t=Yejt,(48)R_{j,t}v = U(e_{jt} \otimes v) + \delta_{tt_0}v,\qquad p_{j,t} = Ye_{jt}, \tag*{(48)}

so the fixed rest-frame blocks contribute constants only.

Residuals. By (48), every residual ℓ\ell of (DYN) has the form

rℓ=Uaℓ+Ybℓ+cℓ,aℓ∈R3n, bℓ∈Rn, cℓ∈R3,(49)r_\ell= Ua_\ell+ Yb_\ell+ c_\ell,\qquad a_\ell\in\mathbb{R}^{3n},\ b_\ell\in\mathbb{R}^{n},\ c_\ell\in\mathbb{R}^{3}, \tag*{(49)}

with weight λℓ\lambda_\ell equal to the coefficient of its term in (DYN). Using ∑j∈N(i)wij=1\sum_{j\in\mathcal{N}(i)}w_{ij} = 1, the three terms give the following.

  • Tracking, one residual per (i,t)∈Ed(i,t) \in\mathcal{E}_d with weight λtrack\lambda_{\mathrm{track}}:

a=∑j∈N(i)wijejt⊗(xid−gj),a = \sum_{j\in\mathcal{N}(i)} w_{ij}e_{jt}\otimes(x_i^d-g_j),
b=∑j∈N(i)wijejt,b = \sum_{j\in\mathcal{N}(i)} w_{ij}e_{jt},
c=gˉi+δtt0(xid−gˉi)−yitd,gˉi=∑j∈N(i)wijgj.(50)c = \bar{g}_i+\delta_{tt_0}(x_i^d-\bar{g}_i)-y_{it}^d,\qquad\bar{g}_i=\sum_{j\in\mathcal{N}(i)}w_{ij}g_j. \tag*{(50)}
  • ARAP, one residual per edge (j,k)∈EEDG(j,k) \in\mathcal{E}_{\mathrm{EDG}} and frame t≠t0t \ne t_0 with weight λARAP\lambda_{\mathrm{ARAP}} (at t=t0t=t_0 the residual vanishes identically):

a=ejt⊗(gk−gj),b=ejt−ekt,c=gj−gk.(51)a=e_{jt}\otimes(g_k-g_j),\qquad b=e_{jt}-e_{kt},\qquad c=g_j-g_k. \tag*{(51)}
  • Temporal smoothness, for every node jj and t=1,…,T−1t=1,\ldots,T-1 with weight λtemp\lambda_{\mathrm{temp}}: writing

∥Rj,t−Rj,t+1∥F2=∑m=13∥(Rj,t−Rj,t+1)ϵm∥2\lVert R_{j,t}-R_{j,t+1}\rVert_F^2=\sum_{m=1}^{3}\lVert(R_{j,t}-R_{j,t+1})\epsilon_m\rVert^2

with ϵm\epsilon_m the standard basis of R3\mathbb{R}^{3}, three rotation residuals (m=1,2,3m=1,2,3) and one translation residual,

a=(ejt−ej,t+1)⊗ϵm,b=0,a=(e_{jt}-e_{j,t+1})\otimes\epsilon_m,\qquad b=0,
c=(δtt0−δt+1,t0)ϵm,c=(\delta_{tt_0}-\delta_{t+1,t_0})\epsilon_m,
a=0,b=ejt−ej,t+1,c=0.(52)a=0,\qquad b=e_{jt}-e_{j,t+1},\qquad c=0. \tag*{(52)}

The node term of the implementation (Appendix § 14.§ 3) has the same form, with a=0a=0, b=ejtb=e_{jt}, and c=gj−yjtdc=g_j-y_{jt}^d, so the reduction below applies unchanged when it is included.

Quadratic form. Summing λℓ∥rℓ∥2\lambda_\ell\lVert r_\ell\rVert^2 over all residuals, the objective of (DYN) is

f(U,Y)=tr⁡(Q1UTU)+2tr⁡(WTUTY)+tr⁡(LYTY)+2⟨P1,U⟩+2⟨PY,Y⟩+c1,(53)\begin{aligned} f(U,Y)={}&\operatorname{tr}(Q_1U^{\mathsf{T}}U)+2\operatorname{tr}(W^{\mathsf{T}}U^{\mathsf{T}}Y)+\operatorname{tr}(LY^{\mathsf{T}}Y)\\ &+2\langle P_1,U\rangle+2\langle P_Y,Y\rangle+c_1, \tag*{(53)} \end{aligned}

with

Q1=∑ℓλℓaℓaℓT∈S3n,W=∑ℓλℓaℓbℓT∈R3n×n,(54)Q_1=\sum_{\ell}\lambda_\ell a_\ell a_\ell^{\mathsf{T}}\in\mathbb{S}^{3n},\qquad W=\sum_{\ell}\lambda_\ell a_\ell b_\ell^{\mathsf{T}}\in\mathbb{R}^{3n\times n}, \tag*{(54)}
L=∑ℓλℓbℓbℓT∈Sn,P1=∑ℓλℓcℓaℓT∈R3×3n,(55)L=\sum_{\ell}\lambda_\ell b_\ell b_\ell^{\mathsf{T}}\in\mathbb{S}^{n},\qquad P_1=\sum_{\ell}\lambda_\ell c_\ell a_\ell^{\mathsf{T}}\in\mathbb{R}^{3\times3n}, \tag*{(55)}
PY=∑ℓλℓcℓbℓT∈R3×n,c1=∑ℓλℓ∥cℓ∥2.(56)P_Y=\sum_{\ell}\lambda_\ell c_\ell b_\ell^{\mathsf{T}}\in\mathbb{R}^{3\times n},\qquad c_1=\sum_{\ell}\lambda_\ell\lVert c_\ell\rVert^2. \tag*{(56)}

Elimination. For fixed UU, ff is a convex quadratic in YY, bounded below by zero, so its minimizers solve the stationarity condition YL=−(UW+PY)YL=-(UW+P_Y). This system is consistent: the rows of UW+PY=∑ℓλℓ(Uaℓ+cℓ)bℓ⊤UW+P_Y=\sum_\ell\lambda_\ell(Ua_\ell+c_\ell)b_\ell^\top lie in span⁡{bℓ}=range⁡(L)\operatorname{span}\{b_\ell\}=\operatorname{range}(L). Its minimum-norm solution is

Y⋆=−(UW+PY)L+(57)Y^\star=-(UW+P_Y)L^+ \tag*{(57)}

which is unique when L≻0L\succ0, e.g., whenever λtemp>0\lambda_{\mathrm{temp}}>0, since every node’s temporal chain then reaches the fixed rest frame. Substituting (57) into (53) and using L+LL+=L+L^+LL^+=L^+ gives min⁡Yf(U,Y)=tr⁡(QU⊤U)+2⟨P,U⟩+c0\min_Y f(U,Y)=\operatorname{tr}(QU^\top U)+2\langle P,U\rangle+c_0 with

Q=Q1−WL+W⊤,P=P1−PYL+W⊤,c0=c1−tr⁡(PYL+PY⊤).(58)\begin{aligned} Q&=Q_1-WL^+W^\top,\\ P&=P_1-P_YL^+W^\top,\\ c_0&=c_1-\operatorname{tr}(P_YL^+P_Y^\top). \tag*{(58)} \end{aligned}

The matrix QQ is the generalized Schur complement of LL in ∑ℓλℓ[aℓbℓ][aℓbℓ]⊤⪰0\sum_\ell\lambda_\ell\begin{bmatrix}a_\ell\\ b_\ell\end{bmatrix}\begin{bmatrix}a_\ell\\ b_\ell\end{bmatrix}^{\top}\succeq0, and the columns of W⊤W^\top lie in range⁡(L)\operatorname{range}(L), so Q⪰0Q\succeq0. Minimizing both sides over {Rj,t∈SO(3)}(j,t)∈Ω\{R_{j,t}\in\mathrm{SO}(3)\}_{(j,t)\in\Omega} proves the claimed equivalence, and (57) recovers the optimal node translations from U⋆U^\star.

Proof of Proposition 9

Proof. Since V⊤V=I3V^\top V=I_3,

Uh⊤Uh=[I3V⊤UU⊤VU⊤U](59)U_h^\top U_h= \begin{bmatrix} I_3 & V^\top U\\ U^\top V & U^\top U \end{bmatrix} \tag*{(59)}

and therefore

tr⁡(QhUh⊤Uh)=c03tr⁡(I3)+tr⁡(PU⊤V)+tr⁡(P⊤V⊤U)+tr⁡(QU⊤U)=tr⁡(QU⊤U)+2⟨P,V⊤U⟩+c0.(60)\begin{aligned} \operatorname{tr}(Q_hU_h^\top U_h) &=\frac{c_0}{3}\operatorname{tr}(I_3)+\operatorname{tr}(PU^\top V)\\ &\quad+\operatorname{tr}(P^\top V^\top U)+\operatorname{tr}(QU^\top U)\\ &=\operatorname{tr}(QU^\top U)+2\langle P,V^\top U\rangle+c_0. \tag*{(60)} \end{aligned}

Let U~=V⊤U\widetilde U=V^\top U. Orthogonality of VV gives U⊤U=U~⊤U~U^\top U=\widetilde U^\top\widetilde U, so by (60)

tr⁡(QhUh⊤Uh)=tr⁡(QU~⊤U~)+2⟨P,U~⟩+c0.(61)\operatorname{tr}(Q_hU_h^\top U_h)=\operatorname{tr}(Q\widetilde U^\top\widetilde U)+2\langle P,\widetilde U\rangle+c_0. \tag*{(61)}

which is the objective of (DYN-RO) evaluated at U~\widetilde U. The blocks of U~\widetilde U are V⊤Rj,tV^\top R_{j,t}, which lie in SO(3)\mathrm{SO}(3) whenever VV and Rj,tR_{j,t} do, so U~\widetilde U is feasible for (DYN-RO). Hence every feasible (V,U)(V,U) of (DYN-H) attains the objective value of a feasible point of (DYN-RO), and ρDYN−H⋆≥ρDYN−RO⋆\rho^\star_{\mathrm{DYN-H}}\ge\rho^\star_{\mathrm{DYN-RO}}. Conversely, every feasible U~\widetilde U of (DYN-RO) is attained by the feasible pair (V,U)=(I3,U~)(V,U)=(I_3,\widetilde U), so ρDYN−H⋆≤ρDYN−RO⋆\rho^\star_{\mathrm{DYN-H}}\le\rho^\star_{\mathrm{DYN-RO}}. The two optimal values therefore coincide, and the correspondence of minimizers follows from the equality of objective values along the map (V,U)↦V⊤U(V,U)\mapsto V^\top U. The same argument applies verbatim with O(3)\mathrm{O}(3) in place of SO(3)\mathrm{SO}(3).

Proof of Proposition 10

Proof. Identical to Propositions 3–4 and Theorems 5–7, with every block constraint Xii=αiI3X_{ii}=\alpha_iI_3 fixed to Xii=I3X_{ii}=I_3 and the trace constraint removed: X=IX=I remains strictly feasible, so strong duality holds, and each block contributes six linear equalities, so LICQ holds as in Appendix § 11.§ 6.

Suboptimality Gap for the Sphere-Constrained SDP

Algorithm 1 is run with finite gradient tolerances, so the returned Z(yr)Z(y_r) may have a slightly negative minimum eigenvalue even when the solution is essentially optimal. Adapting the suboptimality analysis of XM [20] to the sphere-constrained relaxation (10), the following bound quantifies the resulting optimality gap.

Theorem 11 (Suboptimality Bound, adapted from [20]). Let X^\widehat X be any feasible point of (10) with objective value ρ^=tr⁡(QX^)\hat\rho=\operatorname{tr}(Q\widehat X), let b=(b1,…,bm)b=(b_1,\ldots,b_m) collect the right-hand sides of (14), let Z(y)Z(y) be the dual slack of (17), and let y∈Rmy\in\mathbb{R}^m be arbitrary. Then

ρ^≥ρSDP⋆≥b⊤y+3Tλmin⁡(Z(y)).(62)\hat\rho\ge\rho^\star_{\mathrm{SDP}}\ge b^\top y+3T\lambda_{\min}(Z(y)). \tag*{(62)}

:::`

Proof. The first inequality holds because X^\hat{X} is feasible. For the second, let XX be any feasible point of (10). Using ⟨Ai,X⟩=bi\langle A_i,X\rangle=b_i,

tr⁡(QX)=⟨Z(y),X⟩+∑i=1myi⟨Ai,X⟩=⟨Z(y),X⟩+bTy.(63)\operatorname{tr}(QX)=\langle Z(y),X\rangle+\sum_{i=1}^{m}y_i\langle A_i,X\rangle=\langle Z(y),X\rangle+\mathbf{b}^{\mathsf{T}}y. \tag*{(63)}

Since X⪰0X \succeq0, we have ⟨Z(y),X⟩≥λmin⁡(Z(y))tr⁡(X)\langle Z(y),X\rangle\geq\lambda_{\min}(Z(y))\operatorname{tr}(X), and the trace constraint fixes tr⁡(X)=3T\operatorname{tr}(X)=3T; hence tr⁡(QX)≥bTy+3Tλmin⁡(Z(y))\operatorname{tr}(QX)\geq\mathbf{b}^{\mathsf{T}}y+3T\lambda_{\min}(Z(y)). Minimizing over all feasible XX yields the claim.

In the static stage we apply (62) with X^=U^TU^\hat{X}=\hat{U}^{\mathsf{T}}\hat{U}, the rounded solution of Appendix § 12.§ 1, and y=yry=y_r from the certification step (18); the resulting relative gap η\eta is defined in (21) (§ 3.4). The same argument applies to the dynamic relaxation (DYN-SDP), whose block constraints Xii=I3X_{ii}=I_3 fix tr⁡(X)=3(n+1)\operatorname{tr}(X)=3(n+1); there the bound reads bTy+3(n+1)λmin⁡(Z(y))\mathbf{b}^{\mathsf{T}}y+3(n+1)\lambda_{\min}(Z(y)).

Partial sphere constraint. The partial sphere of § 6.2 fixes only ∑t∈Str⁡(Xtt)\sum_{t\in S}\operatorname{tr}(X_{tt}), so tr⁡(X)\operatorname{tr}(X) is no longer constant. Restricting the partial-sphere variant of (10) to tr⁡(X)≤τ\operatorname{tr}(X)\leq\tau, the same proof gives ρτ⋆≥bTy+τmin⁡{0,λmin⁡(Z(y))}\rho_\tau^\star\geq\mathbf{b}^{\mathsf{T}}y+\tau\min\{0,\lambda_{\min}(Z(y))\}, and without the restriction when Z(y)⪰0Z(y)\succeq0. In our implementation we take τ=2tr⁡(X^)\tau=2\operatorname{tr}(\hat{X}), which bounds the suboptimality of X^\hat{X} among feasible points whose trace does not exceed twice its own.

Rounding to the Scaled-Rotation Problem

Let Ur⋆∈Rr×3TU_r^\star\in\mathbb{R}^{r\times3T} be the output of Algorithm 1. We round it to a feasible point U^=[s1C1 ⋯ sTCT]\hat{U}=[s_1C_1\ \cdots\ s_TC_T] of (SRO) under the sphere constraint ∥U^∥F2=3T\lVert\hat{U}\rVert_F^2=3T in three steps.

  1. Rank-three factor. Let (Λ3,V3)(\Lambda_3,V_3) be the three largest eigenpairs of (Ur⋆)TUr⋆(U_r^\star)^{\mathsf{T}}U_r^\star and set U~=Λ31/2V3T∈R3×3T\widetilde{U}=\Lambda_3^{1/2}V_3^{\mathsf{T}}\in\mathbb{R}^{3\times3T}, with blocks U~t∈R3×3\widetilde{U}_t\in\mathbb{R}^{3\times3}.

  1. Projection. Each block is projected onto the scaled rotations: with the singular value decomposition U~t=AtΣtBtT\widetilde{U}_t=A_t\Sigma_tB_t^{\mathsf{T}}, we set Ct=Atdiag⁡(1,1,det⁡(AtBtT))BtT∈SO⁡(3)C_t=A_t\operatorname{diag}(1,1,\det(A_tB_t^{\mathsf{T}}))B_t^{\mathsf{T}}\in\operatorname{SO}(3) and st=⟨U~t,Ct⟩/3s_t=\langle\widetilde{U}_t,C_t\rangle/3.

  1. Sphere. We rescale all scales by the common factor T/∥s∥2\sqrt{T}/\lVert s\rVert_2.

Under the partial sphere constraint, step 3 uses the factor ∣S∣/∥sS∥2\sqrt{|S|}/\lVert s_S\rVert_2, where sSs_S collects the scales of the frames in SS. This enforces ∑t∈Sst2=∣S∣\sum_{t\in S}s_t^2=|S|.

Dataset Details

IMC. Six scenes of the IMC PhotoTourism benchmark [28]—gate_00, gate_01, rome_00, rome_04, temple_00, and temple_01—each an unordered collection of internet photographs of a static landmark, subsampled from the original collection to about 200 images per scene.

Replica. Eight synthetic indoor sequences rendered from the Replica dataset [63] as in NICE-SLAM [89]: office1, office2, room1, and room2, two sequences each. Each sequence is subsampled from the original 2000 frames to 100 frames. The scenes are static.

TUM. Six sequences from the TUM RGB-D benchmark [64], each subsampled from the original sequence to 100 frames. The sequences contain moving objects; only camera accuracy is evaluated.

BAL. Three problems of the Bundle Adjustment in the Large dataset [1], BAL-93, BAL-394, and BAL-1936, used only for the solver-configuration study.

Sintel. Twenty-three synthetic sequences of the MPI Sintel dataset [12] with up to 50 frames each, containing fast articulated motion and large camera motion.

NVIDIA. The seven scenes of the NVIDIA dynamic scene dataset [81]—Baloon1, Baloon2, Jumping, Playground, Skating, Truck, and Umbrella—each with 12 frames.

DyCheck. The seven scenes of the DyCheck iPhone dataset [18]—apple, block, paper-windmill, space-out, spin, teddy, and wheel—with 277 to 473 frames each, captured by a handheld iPhone with realistic camera motion and evaluated from two static test cameras.

Syn4D. A synthetic benchmark of eight 100-frame sequences with ground-truth cameras and geometry, in two sub-datasets, Bear and RoboCasa. The four Bear sequences (bear1–bear4) place a scanned Christmas-bear model from Artec 3D [4] in a Replica [63] background, each with a different motion of the bear. The four RoboCasa sequences (casa1–casa4) are rendered from RoboCasa [47] scenes.

Implementation Details

Throughout, uit∈R2u_{it} \in\mathbb{R}^{2} is the pixel of track ii at frame tt.

Online Tracking and Query Management

Online query sampling. At every frame, candidate queries cc are drawn uniformly and accepted iff

min⁡p∈Rt∥c−p∥>r,(64)\min_{p \in\mathcal{R}_{t}} \lVert c-p\rVert> r, \tag*{(64)}

where Rt\mathcal{R}_{t} contains all tracks visible at frame tt and all queries added in recent frames.

Mask filter. On the static side, a track whose observations fall inside the mask more than 20% of the time leaves the static set (meaning it lives on the moving object); otherwise only its masked observations are dropped. On the dynamic side the masks play the opposite role.

Observation Filtering

Large 2D drift (static tracks). A static track that suddenly disagrees with the motion of its neighbors should be excluded. For every transition t−1→tt-1 \to t and every track ii visible at both frames, define the flow and its local reference,

fit=uit−ui,t−1,fˉit=median⁡k∈Nitfkt,(65)\mathbf{f}_{it}=\mathbf{u}_{it}-\mathbf{u}_{i,t-1}, \qquad\bar{\mathbf{f}}_{it}=\operatorname{median}_{k\in\mathcal{N}_{it}}\mathbf{f}_{kt}, \tag*{(65)}

where Nit\mathcal{N}_{it} collects the other tracks visible at both frames whose position lies in a w×ww \times w window centered on uit\mathbf{u}_{it}, and the median is taken component-wise. A transition is bad iff the relative-motion error exceeds a threshold,

eit=∥fit−fˉit∥>θt.(66)e_{it}=\lVert\mathbf{f}_{it}-\bar{\mathbf{f}}_{it}\rVert>\theta_t. \tag*{(66)}

The threshold adapts to the motion speed of the frame: with mt=median⁡k∥fkt∥m_t=\operatorname{median}_{k}\lVert\mathbf{f}_{kt}\rVert the median flow magnitude over all tracks visible at the transition,

θt=max⁡(θmin⁡,κmt).(67)\theta_t=\max\left(\theta_{\min},\kappa m_t\right). \tag*{(67)}

For every track with at least one bad transition, the visible timeline is split at the bad transitions and only the longest contiguous segment is kept.

Depth filters. Each surviving observation, static or dynamic, is assigned the depth ditd_{it} of its nearest depth-map pixel when lifted to 3D. Observations with excessive depth,

dit>τz(typically τz=100),(68)d_{it}>\tau_z \qquad\text{(typically }\tau_z=100\text{)}, \tag*{(68)}

are discarded, as monocular depth is unreliable far from the camera. Observations on depth discontinuities, detected by thresholding the second-order Sobel gradient magnitude of the depth map at the sample location, are discarded as well, since the nearest-pixel depth lookup is unreliable there.

XM2 residual trim. After the first solve of the static problem, the per-observation residual

rit=ωit∥Ct(stu~its)+ct−xis∥2(69)r_{it}=\omega_{it}\left\lVert C_t(s_t\tilde{\mathbf{u}}_{it}^{s})+\mathbf{c}_t-\mathbf{x}_{i}^{s}\right\rVert^{2} \tag*{(69)}

is computed, observations above the PP-th percentile of {rit}\{r_{it}\} are removed (P=95P=95 in our experiments), and the system is re-solved as described in § 6.2.

Node Observation Term in the Dynamic Solve

Each EDG node is itself a tracked point with lifted observations yjtd\mathbf{y}_{jt}^{d} at the frames (j,t)∈Enode(j,t)\in\mathcal{E}_{\mathrm{node}} where it is observed. In the implementation, the segment solves add to (DYN) a node term that adds every node to follow its own observations,

λnodeLnode,Lnode=∑(j,t)∈Enode∥gj+pj,t−yjtd∥2.(70)\lambda_{\mathrm{node}}\mathcal{L}_{\mathrm{node}}, \qquad\mathcal{L}_{\mathrm{node}}=\sum_{(j,t)\in\mathcal{E}_{\mathrm{node}}}\left\lVert\mathbf{g}_{j}+\mathbf{p}_{j,t}-\mathbf{y}_{jt}^{d}\right\rVert^{2}. \tag*{(70)}

Propositions 8 and 10 hold unchanged (Appendix § 11.8).

Fusion of Segment Solutions

Positions are averaged over all sequences that solved node jj at frame tt, each solution weighted by 1/(1+δ)1/(1+\delta), where δ\delta is the number of frames from tt to the nearest frame in which node jj is directly observed within that sequence, so that extrapolated poses count less than observed ones. Rotations cannot be averaged directly, since each sequence fixes its own gauge at its rest frame, but their relative motion does not depend on that choice: every sequence ss that contains frames tt and t′t' provides the relative rotation Rˉj,t,t′s=Rj,t′s(Rj,ts)⊤\bar{R}^{s}_{j,t,t'}=R^{s}_{j,t'}(R^{s}_{j,t})^{\top}. For each node we recover all its rotations at once by rotation synchronization,

min⁡{Rj,t}⊂SO(3)∑s∑t,t′∥Rj,t′−Rˉj,t,t′sRj,t∥F2(71)\min_{\{R_{j,t}\}\subset\mathrm{SO}(3)} \sum_{s}\sum_{t,t'} \left\lVert R_{j,t'}-\bar{R}^{s}_{j,t,t'}R_{j,t}\right\rVert_{F}^{2} \tag*{(71)}

which we solve with a local solver, fixing the gauge by setting Rj,t=I3R_{j,t}=I_{3} at the first frame in which node jj is observed. Solving (71) with XM is left to future work.

Long-Range Dynamic Tracks

Raw dynamic tracks are short and, on repetitive motion, swap the identity of similar-looking parts, so we optionally assemble them into long-range tracks in three steps. (a) Skeleton. A sparse but more reliable long-range tracker, CoWTracker [32], is restarted periodically from a grid inside the object mask and run forward and backward, so that every skeleton track spans the whole video; tracks with large 2D jumps, inconsistent color, or low coverage are filtered out. (b) Attachment. Short online tracks [85] are attached to every skeleton point whose percentile distance to them is small. (c) Averaging. Each skeleton point is replaced by the per-frame mean of its attached tracks, giving one long track per skeleton point; unattached online tracks are kept as additional shorter tracks.

GPU Solver

XM4D implements RTR-TCG and the Riemannian staircase of Algorithm 1 on the GPU in double precision. The data matrices QQ and QhQ_h are obtained by eliminating the linearly entering translations, landmarks, and node translations, an instance of variable projection that stays efficient when sparsity is exploited [50]. We store QQ densely, so each inner conjugate-gradient iteration costs one dense matrix product for the Euclidean Hessian action 2UQ˙2U\dot{Q}, computed with cuBLAS [49]. The tangent-space projections and the retractions, a batched Gram–Schmidt QR for the Stiefel blocks and a renormalization onto the sphere for the scales, are custom CUDA kernels; the dynamic problem (DYN-BM) reuses them without the scale variables.

References

  1. [1]Sameer Agarwal, Noah Snavely, Steven M Seitz, and Richard Szeliski. Bundle adjustment in the large. In European Conference on Computer Vision (ECCV), pages 29–42. Springer, 2010. 14, 30DOI
  2. [2]Sameer Agarwal, Keir Mierle, and The Ceres Solver Team. Ceres Solver, 10 2023. URL https://github.com/ceres-solver/ceres-solver. 12
  3. [3]Arthur Allshire, Chouki Chou, Jiunyi Zhang, David McAllister, Anthony Zhang, Chung Min Kim, Trevor Darrell, Pieter Abbeel, Jitendra Malik, and Angjoo Kanazawa. Visual imitation enables contextual humanoid control. Conference on Robot Learning (CoRL), 2025. 2arxiv.org/abs/2505.03729
  4. [4]Artec 3D. Christmas Bear. 3D model. URL https://www.artec3d.com/3d-models/christmas-bear. Licensed under CC BY 4.0. 18, 30
  5. [5]Alexey Bochkovskiy, Amaël Delaunoy, Hugo Germain, Marcel Santos, Yichao Zhou, Stephan Richter, and Vladlen Koltun. Depth pro: Sharp monocular metric depth in less than a second. In International Conference on Learning Representations (ICLR), volume 2025, pages 75602–75637, 2025. 4
  6. [6]Boston Dynamics. Atlas Airborne. YouTube video, February 2026. URL https://www.youtube.com/watch?v=UNorxwIZFk. 16, 17
  7. [7]N. Boumal, B. Mishra, P.-A. Absil, and R. Sepulchre. Manopt, a MATLAB Toolbox for Optimization on Manifolds. Journal of Machine Learning Research, 15(1):1455–1459, 2014. 6arxiv.org/abs/1308.5200
  8. [8]Nicolas Boumal. A Riemannian low-rank method for optimization over semidefinite matrices with block-diagonal constraints. arXiv preprint arXiv:1506.00575, 2015. 4, 7
  9. [9]Aljaz Bozic, Pablo Palafox, Michael Zollhofer, Justus Thies, Angela Dai, and Matthias Nießner. Neural deformation graphs for globally-consistent non-rigid reconstruction. In Conference on Computer Vision and Pattern Recognition (CVPR), pages 1450–1459, 2021. 4
  10. [10]Minh-Quan Viet Bui, Jongmin Park, Jihyong Oh, and Munchurl Kim. DyBluRF: Dynamic Deblurring Neural Radiance Fields for Blurry Monocular Video. arXiv preprint arXiv:2312.13528, 2023. 15
  11. [11]Samuel Burer and Renato D.C. Monteiro. A nonlinear programming algorithm for solving semidefinite programs via low-rank factorization. Mathematical Programming, 95(2):329–357, 2003. 4, 6, 27, 28
  12. [12]D. J. Butler, J. Wulff, G. B. Stanley, and M. J. Black. A naturalistic open source movie for optical flow evaluation. In A. Fitzgibbon et al. (Eds.), editor, European Conference on Computer Vision (ECCV), Part IV, LNCS 7577, pages 611–625. Springer-Verlag, October 2012. 14, 16, 17, 30
  13. [13]Yohann Cabon, Lucas Stoffl, Leonid Antsfeld, Gabriela Csurka, Boris Chidlovskii, Jerome Revaud, and Vincent Leroy. Must3r: Multi-view network for stereo 3d reconstruction. In Conference on Computer Vision and Pattern Recognition (CVPR), pages 1050–1060. IEEE, 2025. 4
  14. [14]Nicolas Carion, Genglin G. Guan, Yuan-Ting Hu, Shoubhik Debnath, Ronghang Hu, Didac Suris, Chaitanya Ryali, Kalyan Vasudev Alwala, Haitham Khedr, Andrew Huang, et al. Sam 3: Segment anything with concepts. arXiv preprint arXiv:2511.16719, 2025. 11arxiv.org/abs/2511.16719
  15. [15]Weirong Chen, Ganlin Zhang, Felix Wimbauer, Rui Wang, Nikita Araslanov, Andrea Vedaldi, and Daniel Cremers. Back on track: Bundle adjustment for dynamic scene reconstruction. In International Conference on Computer Vision (ICCV), pages 4951–4960. IEEE, 2025. 2
  16. [16]Carl Doersch, Pauline Luc, Yi Yang, Dilara Gokay, Skanda Koppula, Ankush Gupta, Joseph Heyward, Ignacio Roccco, Ross Goroshin, Joao Carreira, et al. Bootstap: Bootstrapped training for tracking-any-point. In Proceedings of the Asian Conference on Computer Vision, pages 3257–3274, 2024. 4arxiv.org/abs/2402.00847
  17. [17]Chen Gao, Ayush Saraf, Johannes Kopf, and Jia-Bin Huang. Dynamic view synthesis from dynamic monocular video. In International Conference on Computer Vision (ICCV), pages 5692–5701. IEEE, 2021. 2arxiv.org/abs/2105.06468
  18. [18]Hang Gao, Ruilong Li, Shubham Tulsiani, Bryan Russell, and Angjoo Kanazawa. Monocular dynamic view synthesis: A reality check. Conference on Neural Information Processing Systems (NeurIPS), 35:33768–33780, 2022.arxiv.org/abs/2210.13445
  19. [19]Ravi Garg, Anastasios Roussos, and Lourdes Agapito. Dense variational reconstruction of non-rigid surfaces from monocular video. In Conference on Computer Vision and Pattern Recognition (CVPR), pages 1272–1279, 2013.DOI
  20. [20]Haoyu Han and Heng Yang. Building Rome with Convex Optimization. In Robotics: Science and Systems (RSS), 2025.arxiv.org/abs/2502.04640
  21. [21]Tairan He, Zhengyi Luo, Xialin He, Wenli Xiao, Chong Zhang, Weinan Zhang, Kris Kitani, Changliu Liu, and Guanya Shi. Omni2o: Universal and dexterous human-to-humanoid whole-body teleoperation and learning. Conference on Robot Learning (CoRL), 2024.arxiv.org/abs/2406.08858
  22. [22]Luke Hollis, Tianxiang Fan, and Heng Yang. SceneAgent: An Agentic Pipeline from 3D and 4D Captures to Simulation Environments. arXiv preprint arXiv:2610.34745, 2026.arxiv.org/abs/2610.34745
  23. [23]Mu Hu, Wei Yin, Chi Zhang, Zhipeng Cai, Xiaoxiao Long, Hao Chen, Kaixuan Wang, Gang Yu, Chunhua Shen, and Shaojie Shen. Metric3D v2: A versatile monocular geometric foundation model for zero-shot metric depth and surface normal estimation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(12):10579–10596, 2024.DOI
  24. [24]Matthias Innmann, Michael Zollhöfer, Matthias Nießner, Christian Theobalt, and Marc Stamminger. Volumedeform: Real-time volumetric non-rigid reconstruction. In European Conference on Computer Vision (ECCV), pages 362–379. Springer, 2016.arxiv.org/abs/1603.08161
  25. [25]Hanxiao Jiang, Hao-Yu Hsu, Kaifeng Zhang, Hsin-Ni Yu, Shenlong Wang, and Yunzhu Li. Phystwin: Physics-informed reconstruction and simulation of deformable objects from videos. In International Conference on Computer Vision (ICCV), pages 7219–7230. IEEE, 2025.arxiv.org/abs/2503.17973
  26. [26]Tao Jiang, Peng Lu, Li Zhang, Ningsheng Ma, Rui Han, Chengqi Lyu, Yining Li, and Kai Chen. Rtmpose: Real-time multi-person pose estimation based on mmpose. arXiv preprint arXiv:2303.07399, 2023.arxiv.org/abs/2303.07399
  27. [27]Zhenyu Jiang, Cheng-Chun Hsu, and Yuke Zhu. Ditto: Building digital twins of articulated objects from interaction. In Conference on Computer Vision and Pattern Recognition (CVPR), pages 5606–5616. IEEE, 2022.arxiv.org/abs/2202.08227
  28. [28]Yuhe Jin, Dmytro Mishkin, Anastasiia Mishchuk, Jiri Matas, Pascal Fua, Kwang Moo Yi, and Eduard Trulls. Image Matching across Wide Baselines: From Paper to Practice. International Journal of Computer Vision, 2020.DOI
  29. [29]Michel Journée, Francis Bach, P-A Absil, and Rodolphe Sepulchre. Low-rank optimization on the cone of positive semidefinite matrices. SIAM Journal on Optimization, 20(5):2327–2351, 2010.DOI
  30. [30]Nikita Karaev, Yuri Makarov, Jianyuan Wang, Natalia Neverova, Andrea Vedaldi, and Christian Rupprecht. CoTracker3: Simpler and Better Point Tracking by Pseudo-Labeling Real Videos. In International Conference on Computer Vision (ICCV), pages 1–10. IEEE, 2025.DOI
  31. [31]Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3D Gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics, 42(4):139–1, 2023.arxiv.org/abs/2308.04079
  32. [32]Zihang Lai, Eldar Insafutdinov, Edgar Sucar, and Andrea Vedaldi. Cowtracker: Tracking by warping instead of correlation. arXiv preprint arXiv:2602.04877, 2026.
  33. [33]Jean B Lasserre. Convergent SDP-relaxations in polynomial optimization with sparsity. SIAM Journal on optimization, 17(3):822–843, 2006.DOI
  34. [34]Long Le, Jason Xie, William Liang, Hung-Ju Wang, Yue Yang, Yecheng Jason Ma, Kyle Vedder, Arjun Krishna, Dinesh Jayaraman, and Eric Eaton. Articulate-anything: Automatic modeling of articulated objects via a vision-language foundation model. In International Conference on Learning Representations (ICLR), volume 2025, pages 17578–17602, 2025.arxiv.org/abs/2410.13882
  35. [35]Jiahui Lei, Yijia Weng, Adam W Harley, Leonidas Guibas, and Kostas Daniilidis. Mosca: Dynamic gaussian fusion from casual videos via 4d motion scaffolds. In Conference on Computer Vision and Pattern Recognition (CVPR), pages 6165–6177. IEEE, 2025.arxiv.org/abs/2405.17421
  36. [36]Zhengqi Li, Simon Niklaus, Noah Snavely, and Oliver Wang. Neural scene flow fields for space-time view synthesis of dynamic scenes. In Conference on Computer Vision and Pattern Recognition (CVPR), pages 6498–6508, 2021.arxiv.org/abs/2011.13084
  37. [37]Zhengqi Li, Richard Tucker, Forrester Cole, Qianqian Wang, Linyi Jin, Vickie Ye, Angjoo Kanazawa, Alexander Holynski, and Noah Snavely. Megasam: Accurate, fast, and robust structure and motion from casual dynamic videos. In Conference on Computer Vision and Pattern Recognition (CVPR), pages 10486–10496. IEEE, 2025.arxiv.org/abs/2412.04463
  38. [38]Haotong Lin, Sili Chen, Junhao Liew, Donny Y Chen, Zhenyu Li, Guang Shi, Jiashi Feng, and Bingyi Kang. Depth anything 3: Recovering the visual space from any views. arXiv preprint arXiv:2511.10647, 2025.arxiv.org/abs/2511.10647
  39. [39]Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European Conference on Computer Vision (ECCV), pages 740–755. Springer, 2014.arxiv.org/abs/1405.0312
  40. [40]Yang Liu, Tong Wang, Yiming Bao, and Heng Yang. GRASP: Gradient-Guided Sampling for Contact-Rich Physics-Based Motion Retargeting. arXiv preprint arXiv:2611.05754, 2026.arxiv.org/abs/2611.05754
  41. [41]Yu-Lun Liu, Chen Gao, Andreas Meulman, Hung-Yu Tseng, Ayush Saraf, Changil Kim, Yung-Yu Chuang, Johannes Kopf, and Jia-Bin Huang. Robust dynamic radiance fields. In Conference on Computer Vision and Pattern Recognition (CVPR), pages 13–23. IEEE, 2023.arxiv.org/abs/2301.02239
  42. [42]Stephen Lombardi, Tomas Simon, Jason Saragih, Gabriel Schwartz, Andreas Lehrmann, and Yaser Sheikh. Neural volumes: Learning dynamic renderable volumes from images. ACM Trans. Graph., 38(4):65:1–65:14, July 2019.DOI
  43. [43]David G. Lowe. Distinctive image features from scale-invariant keypoints. International journal of computer vision, 60(2):91–110, 2004.DOI
  44. [44]Jonathon Luiten, Georgios Kopanas, Bastian Leibe, and Deva Ramanan. Dynamic 3d gaussians: Tracking by persistent dynamic view synthesis. In 2024 International Conference on 3D Vision (3DV), pages 800–809. IEEE, 2024.arxiv.org/abs/2308.09713
  45. [45]Xingyu Miao, Yang Bai, Haoran Duan, Fan Wan, Yawen Huang, Yang Long, and Yefeng Zheng. Ctnerf: Cross-time transformer for dynamic neural radiance field from monocular video. Pattern Recognition, 156:110729, 2024.DOI
  46. [46]Franziska Mueller, Dushyant Mehta, Oleksandr Sotnychenko, Srinath Sridhar, Dan Casas, and Christian Theobalt. Real-time hand tracking under occlusion from an egocentric rgb-d sensor. In International Conference on Computer Vision (ICCV), pages 1163–1172. IEEE, 2017.
  47. [47]Soroosh Norouzzadeh, Abhinav Maddikuri, Lance Zhang, Adeet Parikh, Aaron Lo, Abhishek Joshi, Ajay Mandlekar, and Yuke Zhu. RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robots. In Robotics: Science and Systems (RSS), 2024.
  48. [48]Richard A Newcombe, Dieter Fox, and Steven M Seitz. Dynamicfusion: Reconstruction and tracking of non-rigid scenes in real-time. In Conference on Computer Vision and Pattern Recognition (CVPR), pages 343–352, 2015.DOI
  49. [49]NVIDIA Corporation. cuBLAS Library User Guide, Version 12.9, 2025. NVIDIA Developer Documentation.
  50. [50]Alan Papalia, Nikolas Sanderson, Haoyu Han, Heng Yang, Hanumant Singh, and Michael Everett. Sparse variable projection in robotic perception: Exploiting separable structure for efficient nonlinear optimization. In IEEE International Conference on Robotics and Automation (ICRA), pages 11826–11833. IEEE, 2026.arxiv.org/abs/2512.07969
  51. [51]Keunhong Park, Utkarsh Sinha, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Steven M Seitz, and Ricardo Martin-Brualla. Nerfies: Deformable neural radiance fields. In International Conference on Computer Vision (ICCV), pages 5845–5854. IEEE, 2021.arxiv.org/abs/2011.12948
  52. [52]Keunhong Park, Utkarsh Sinha, Peter Hedman, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Ricardo Martin-Brualla, and Steven M Seitz. Hypernerf: A higher-dimensional representation for topologically varying neural radiance fields. arXiv preprint arXiv:2106.13228, 2021.arxiv.org/abs/2106.13228
  53. [53]Sida Peng, Qianqian Wang, Qing Shuai, Hujun Bao, and Xiaowei Zhou. Neural body: Implicit neural representations with structured latent codes for novel view synthesis of dynamic humans. In Conference on Computer Vision and Pattern Recognition (CVPR), pages 9050–9059. IEEE, 2021.arxiv.org/abs/2012.15838
  54. [54]Federico Perazzi, Jordi Pont-Tuset, Brian McWilliams, Luc Van Gool, Markus Gross, and Alexander Sorkine-Hornung. A benchmark dataset and evaluation methodology for video object segmentation. In Conference on Computer Vision and Pattern Recognition (CVPR), pages 724–732, 2016.DOI
  55. [55]Luigi Piccinelli, Yung-Hsu Yang, Christos Sakaridis, Mattia Segu, Siyuan Li, Luc Van Gool, and Fisher Yu. Unidepth: Universal monocular metric depth estimation. In Conference on Computer Vision and Pattern Recognition (CVPR), pages 10106–10116, 2024.
  56. [56]Rolandos Alexandros Potamias, Jinglei Zhang, Jiankang Deng, and Stefanos Zafeiriou. Wilor: End-to-end 3d hand localization and reconstruction in-the-wild. In Conference on Computer Vision and Pattern Recognition (CVPR), pages 12242–12254. IEEE, 2025.arxiv.org/abs/2409.12259
  57. [57]Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francesc Moreno-Noguer. D-NeRF: Neural Radiance Fields for Dynamic Scenes. arXiv preprint arXiv:2011.13961, 2020.arxiv.org/abs/2011.13961
  58. [58]Han Qi, Changhe Chen, and Heng Yang. Compose by Focus: Scene Graph-based Atomic Skills. In IEEE International Conference on Robotics and Automation (ICRA), 2026.arxiv.org/abs/2509.16053
  59. [59]Javier Romero, Dimitrios Tzionas, and Michael J Black. Embodied hands: Modeling and capturing hands and bodies together. arXiv preprint arXiv:2201.02610, 2022.DOI
  60. [60]David M Rosen, Luca Carlone, Alfonso S Bandeira, and John J Leonard. SE-Sync: A certifiably correct algorithm for synchronization over the special Euclidean group. International Journal of Robotics Research (IJRR), 38(2-3):95–125, 2019.
  61. [61]Johannes L Schonberger and Jan-Michael Frahm. Structure-from-motion revisited. In Conference on Computer Vision and Pattern Recognition (CVPR), pages 4104–4113, 2016.DOI
  62. [62]Colton Stearns, Adam Harley, Mikaela Uy, Florian Dubost, Federico Tombari, Gordon Wetzstein, and Leonidas Guibas. Dynamic gaussian marbles for novel view synthesis of casual monocular videos. In SIGGRAPH Asia 2024 Conference Papers, pages 1–11, 2024.DOI
  63. [63]Julian Straub, Thomas Whelan, Lingni Ma, Yufan Chen, Erik Wijmans, Simon Green, Jakob J. Engel, Raul Mur-Artal, Carl Ren, Shobhit Verma, Anton Clarkson, Mingfei Yan, Brian Budge, Yajie Yan, Xiaoqing Pan, June Yon, Yuyang Zou, Kimberly Leon, Nigel Carter, Jesus Briales, Tyler Gillingham, Elias Mueggler, Luis Pesqueira, Manolis Savva, Dhruv Batra, Hauke M. Strassdat, Renzo De Nardi, Michael Goesele, Steven Lovegrove, and Richard Newcombe. The Replica Dataset: A Digital Replica of Indoor Spaces. arXiv preprint arXiv:1906.05797, 2019.arxiv.org/abs/1906.05797
  64. [64]J. Sturm, N. Engelhard, F. Endres, W. Burgard, and D. Cremers. A Benchmark for the Evaluation of RGB-D SLAM Systems. In International Conference on Intelligent Robots and Systems (IROS), Oct. 2012.DOI
  65. [65]Robert W Sumner, Johannes Schmid, and Mark Pauly. Embedded deformation for shape manipulation. In ACM siggraph 2007 papers, pages 80–es, 2007. 9
  66. [66]Zachary Teed and Jia Deng. Droid-slam: Deep visual slam for monocular, stereo, and rgb-d cameras. Conference on Neural Information Processing Systems (NeurIPS), 34:16558–16569, 2021. 15arxiv.org/abs/2108.10869
  67. [67]Jianyuan Wang, Minghao Chen, Nikita Karaev, Andrea Vedaldi, Christian Rupprecht, and David Novotny. Vggt: Visual geometry grounded transformer. In Conference on Computer Vision and Pattern Recognition (CVPR), pages 5294–5306. IEEE, 2025. 4, 5, 13, 15arxiv.org/abs/2503.11651
  68. [68]Qianqian Wang, Vickie Ye, Hang Gao, Weijia Zeng, Jake Austin, Zhengqi Li, and Angjoo Kanazawa. Shape of motion: 4d reconstruction from a single video. In International Conference on Computer Vision (ICCV), pages 9660–9672. IEEE, 2025. 2, 4, 15, 25arxiv.org/abs/2407.13764
  69. [69]Shuzhe Wang, Vincent Leroy, Yohann Cabon, Boris Chidlovskii, and Jerome Revaud. Dust3r: Geometric 3d vision made easy. In Conference on Computer Vision and Pattern Recognition (CVPR), pages 20697–20709. IEEE, 2024. 4arxiv.org/abs/2312.14132
  70. [70]Chung-Yi Weng, Brian Curless, Pratul P Srinivasan, Jonathan T Barron, and Ira Kemelmacher-Shlizerman. Humannerf: Free-viewpoint rendering of moving people from monocular video. In Conference on Computer Vision and Pattern Recognition (CVPR), pages 16210–16220, 2022. 2arxiv.org/abs/2201.04127
  71. [71]Henry Wolkowicz, Romesh Saigal, and Lieven Vandenberghe, editors. Handbook of Semidefinite Programming: Theory, Algorithms, and Applications, volume 27 of International Series in Operations Research & Management Science. Kluwer Academic Publishers, Boston, MA, 2000. doi: 10.1007/978-1-4615-4381-7. 7DOI
  72. [72]Guanzhun Wu, Taoran Yi, Jiemin Fang, Lingxi Xie, Xiaopeng Zhang, Wei Wei, Wenyu Liu, Qi Tian, and Xinggang Wang. 4d gaussian splatting for real-time dynamic scene rendering. In Conference on Computer Vision and Pattern Recognition (CVPR), pages 20310–20320. IEEE, 2024. 4arxiv.org/abs/2310.08528
  73. [73]Yuxi Xiao, Qianqian Wang, Shangzhan Zhang, Nan Xue, Sida Peng, Yujun Shen, and Xiaowei Zhou. Spatialtracker: Tracking any 2d pixels in 3d space. In Conference on Computer Vision and Pattern Recognition (CVPR), pages 20406–20417. IEEE, 2024.arxiv.org/abs/2404.04319
  74. [74]Gengshan Yang, Deqing Sun, Varun Jampani, Daniel Vlasic, Forrester Cole, Huiwen Chang, Deva Ramanan, William T Freeman, and Ce Liu. Lasr: Learning articulated shape reconstruction from a monocular video. In Conference on Computer Vision and Pattern Recognition (CVPR), 2021. 2arxiv.org/abs/2105.02976
  75. [75]Heng Yang. Semidefinite Optimization and Relaxation. Lecture notes: https://hankyang.seas.harvard.edu/Semidefinite/, 2024. 5, 6, 27, 28
  76. [76]Heng Yang and Luca Carlone. Certifiably optimal outlier-robust geometric perception: Semidefinite relaxations and scalable global optimization. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 45(3):2816–2834, 2022. 4DOI
  77. [77]Heng Yang, Junheng Shi, and Luca Carlone. Teaser: Fast and certifiable point cloud registration. IEEE Transactions on Robotics (TRO), 37(2):314–333, 2020. 4, 15arxiv.org/abs/2001.07715
  78. [78]Heng Yang, Ling Liang, Luca Carlone, and Kim-Chuan Toh. An inexact projected gradient method with rounding and lifting by nonlinear programming for solving rank-one semidefinite relaxation of polynomial optimization. Mathematical Programming, 201(1):409–472, 2023. 4
  79. [79]Lihe Yang, Bingyi Kang, Zilong Huang, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything: Unleashing the power of large-scale unlabeled data. In Conference on Computer Vision and Pattern Recognition (CVPR), pages 10371–10381, 2024. 4
  80. [80]Vickie Ye, Ruilong Li, Justin Kerr, Matias Turkulainen, Brent Yi, Zhuoyang Pan, Otto Seiskari, Jianbo Ye, Jeffrey Hu, Matthew Tancik, and Angjoo Kanazawa. gsplat: An open-source library for Gaussian splatting. Journal of Machine Learning Research (JMLR), 26(34):1–17, 2025. 14arxiv.org/abs/2409.06765
  81. [81]Jae Shin Yoon, Kihwan Kim, Orazio Gallo, Hyun Soo Park, and Jan Kautz. Novel view synthesis of dynamic scenes with globally coherent depths from a monocular camera. In Conference on Computer Vision and Pattern Recognition (CVPR), pages 5335–5344. IEEE, 2020. 14, 16, 18, 30
  82. [82]Xihang Yu and Heng Yang. SIM-Sync: From Certifiably Optimal Synchronization over the 3D Similarity Group to Scene Reconstruction with Learned Depth. IEEE Robotics and Automation Letters (RAL), 2024. URL https://arxiv.org/abs/2309.05184. 4
  83. [83]Xiaoming Zhao, R Colburn, Fangchang Ma, Miguel Angel Bautista, Joshua Susskind, and Alex Schwing. Pseudo-generalized dynamic view synthesis from a video. In International Conference on Learning Representations (ICLR), volume 2024, pages 42726–42748, 2024. 15arxiv.org/abs/2310.08587
  84. [84]Zhenjun Zhao, Heng Yang, Bangyan Liao, Yingping Zeng, Shaocheng Yan, Yingdong Gu, Peidong Liu, Yi Zhou, Haoang Li, and Javier Civera. Advances in global solvers for 3d vision. arXiv preprint arXiv:2602.14662, 2026. 4arxiv.org/abs/2602.14662
  85. [85]Artem Zholus, Carl Doersch, Yi Yang, Skanda Koppula, Viorica Patraucean, Xu Owen He, Ignacio Rocco, Mehdi S. M. Sajjadi, Sarath Chandar, and Ross Goroshin. TAPNext: Tracking Any Point (TAP) as Next Token Prediction. International Conference on Computer Vision (ICCV), 2025. 4, 11, 32
  86. [86]Kaichen Zhou, Jia-Xing Zhong, Sangyun Shin, Kai Lu, Yiyuan Yang, Andrew Markham, and Niki Trigoni. Dynpoint: Dynamic neural point for view synthesis. Conference on Neural Information Processing Systems (NeurIPS), 36:69532–69545, 2023. 15arxiv.org/abs/2310.18999
  87. [87]Yifeng Zhu, Zhenyu Jiang, Peter Stone, and Yuke Zhu. Learning generalizable manipulation policies with object-centric 3d representations. arXiv preprint arXiv:2310.14386, 2023. 19arxiv.org/abs/2310.14386
  88. [88]Yingying Zhu, Dong Huang, Fernando De La Torre, and Simon Lucey. Complex non-rigid motion 3d reconstruction by union of subspaces. In Conference on Computer Vision and Pattern Recognition (CVPR), pages 1542–1549, 2014. 4DOI
  89. [89]Zihan Zhu, Songyou Peng, Viktor Larsson, Weiwei Xu, Hujun Bao, Zhaopeng Cui, Martin R. Oswald, and Marc Pollefeys. NICE-SLAM: Neural Implicit Scalable Encoding for SLAM. In Conference on Computer Vision and Pattern Recognition (CVPR), 2022. 14, 30arxiv.org/abs/2112.12130

Paper details

vlnc:W_jjx5p76a
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 11

downloads · 1

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents