Introduction
The blind PnP problem is to estimate a calibrated camera’s pose from unordered 2D points and 3D model points, without given correspondences [54 ] (Figure 1 ). It enables localization from geometry alone when compatible visual descriptors are deliberately omitted [82 ] or unavailable, as for laser scans [7 ] . The sets must still share scene structure: inlier 2D points are projections of model points under a common camera pose, while either set may contain unmatched points. Known pairs would give the pose by PnP [44 ] , but the number of possible pairings grows superpolynomially with the set sizes [24 ] .
Figure 1. Blind perspective-n n n -point with MomentPnP. The inputs are (a) an unordered set of 2D SIFT points [49 ] in a Sacre-Cœur image from the Phototourism dataset [37 , 69 ] , tracked by its structure-from-motion reconstruction [64 ] , (b) the unordered 3D point set of those tracks, and known camera intrinsics K K K . The outputs are (c) the camera pose, with the matched model points and their rays in pink , and (d) the correspondences between projected model points (blue crosses) and observed 2D points (orange ).
Local methods alternate pose estimation and correspondence updates, making their success sensitive to initialization [11 , 54 ] . Global methods search pose space by branch and bound [4 , 5 , 7 ] , with computational cost driven by the regions that must be examined and excluded. Learned methods predict correspondences from coordinates, using information acquired from training data [6 , 48 , 82 ] . These limitations motivate estimating the pose from the whole point sets before resolving matches.
Because a known 3D model limits how its points can appear in an image, feature averages of the projected model constrain the pose without any pairing of points [60 , 72 ] . Mean and covariance describe position and spread, but distinct projections can share them (Figure 2 ). Higher-order moments, averages of products of point coordinates [19 , 34 ] , capture shape beyond the mean and covariance [31 ] and can give the pose of planar point sets. Whether a fixed number of moments determines a perspective camera pose without pairing points was, to our knowledge, open.
Figure 2. Higher-order moments and Fourier features. Pairs (a, b) and (c, d) share centroid and covariance, and bars compare their moments M g \mathcal{M}_g M g . Third-order moments separate (a) and (b), fourth-order moments the noisy versions (c, d). (e) An outlier at distance r r r among n = 100 n=100 n = 100 points changes fourth-order moments in proportion to r 4 r^4 r 4 and Fourier feature averages by at most 2 / n 2/n 2/ n .
Contributions. We introduce MomentPnP (§ 3 ), a learning-free blind PnP solver that estimates the camera pose before the correspondences. It adjusts the six pose parameters until averages of bounded Fourier features over the projected model match those over the observations, after subtracting the expected effect of outliers in both sets [23 ] . This comparison pairs no points. On complete inputs, it matches fourteen polynomial moments, and we prove that noiseless moments determine the pose under perspective for almost every model of at least twenty points (Theorem 1). With outliers and noise, finitely many Fourier frequencies identify the pose and bound its error by the residual plus sampling and noise terms (Theorem 2).
Empirical results. On released blind PnP benchmarks, MomentPnP has a 5 to 170 times smaller median rotation error than the best learned baseline, and 82 to 97% of frames are within 1° with as many outliers as inliers (Table 1 ). On rendered ETH3D laser scans [65 ] , its median rotation error falls to 0.00007° at one million points (Figure 8 ). In full scene reconstructions, it localizes 35 of 39 ETH3D images within 1° (Table A18 ).
Table 1. Blind PnP on released MegaDepth, ModelNet40 and Data61 test sets. On each test set, we sample 1,000 outlier-free frames and 300 frames with outliers. Searches use a 30-second limit per frame. MomentPnP treats inputs as complete only when they are known to be outlier-free. Err.: median rotation error (∘ ^\circ ∘ ) / translation error (%) over returned poses. Acc.: accuracy (%). Time: range of the per-dataset median times without outliers (s). EM-ICP (ours): our EM refinement (Step 3) from random poses, without moment matching. Bold: best per column.
Figure 8. Rotation error against point count. Medians over the views of three ETH3D scans [65 ] .
Table A18. Recovery from full scene reconstructions on ETH3D. Each query contains the tracked 2D points of one image and every finite SfM model point of the scene, without the visible subset or a pose. Points is the number of model points, and Ratio is the scene extent divided by the viewing distance. Recovery columns count queries with rotation error below 1 ∘ 1^\circ 1 ∘ . The learned baselines and Tuning use the three queries on which we chose the settings, and Test uses three held-out queries with these settings fixed. DBP, BPnP and GoM denote Deep Blind PnP, BPnPNet and GoMatch, and OOM means out of memory. Bold marks the highest recovery on the tuning queries in each scene.
Related work
Blind PnP. Methods differ in how they couple pose and correspondences. Hypothesize-and-test methods such as P3P-RANSAC [18 ] sample triples of candidate pairs, but correct triples become unlikely as the sets grow. Local methods such as SoftPOSIT [11 ] and BlindPnP [54 ] jointly estimate pose and soft correspondences from an initial pose or pose prior, as do recent refinements [3 , 70 ] . Global methods such as GOPAC [7 ] and GOSMA [5 ] search pose space by branch and bound [4 , 29 , 77 ] , at a cost growing with the search domain. Learned methods such as Deep Blind PnP [48 ] and BPnPNet [6 ] predict correspondences from coordinates and then solve PnP. GoMatch [82 ] does so for localization against retrieved views, and MinCD-PnP [1 ] adds color images and surface normals. All rely on correspondences, a pose prior or pose search.
Point-set registration. Registering point sets of equal dimension also couples transformations and correspondences. Softassign [20 ] , EM-ICP [21 ] and coherent point drift [55 ] alternate between soft correspondences and a transformation, which we adopt for refinement under perspective projection. Moment-based methods avoid correspondences by comparing moments. They have been used for planar object pose [72 ] , affine registration [31 ] and planar shapes under homographies [16 ] . Sketching methods also average sines and cosines over whole datasets [23 ] . Because these features are bounded, estimators that match their averages remain accurate when a fraction of the data are outliers [8 ] . Correspondence-free pose estimation (CFPE) [60 ] comes closest to our setting by matching averages of nonlinear features under perspective projection, but its planar experiments start near the true pose. We estimate a perspective camera pose from feature averages of whole point sets, without a pose prior, with outliers in both sets, before matching points.
Kernel mean embeddings. Averages of feature maps represent a distribution as a kernel mean embedding, and maximum mean discrepancy measures the distance between two embeddings [22 ] . For characteristic kernels, equal embeddings imply equal distributions [71 ] . Our feature averages are embeddings of this kind, but we need identification only among the projections of one known model. § 3.3 proves it from fourteen moments on complete inputs and from finitely many Fourier features with outliers.
Pose verification. A fitted pose can be wrong in two ways. For planar targets, two poses can give nearly the same image even with known correspondences [58 , 66 ] . Without correspondences, a wrong pose can also place projections near many observations by chance. Desolneux et al. [14 ] judge a structure by how often it would arise at random, and Moisan and Stival [53 ] apply this test to two-view matching. We check alignment against randomly placed projections and compare each accepted pose with its half-turn alternatives (§ 3.1 ).
Problem statement
Input & camera. We are given n n n 2D points (observations) U = { u i } i = 1 n ⊂ R 2 \mathcal{U} = \{u_i\}_{i=1}^{n} \subset\mathbb{R}^{2} U = { u i } i = 1 n ⊂ R 2 , N N N model points Y = { y j } j = 1 N ⊂ R 3 \mathcal{Y} = \{y_j\}_{j=1}^{N} \subset\mathbb{R}^{3} Y = { y j } j = 1 N ⊂ R 3 and the intrinsic matrix K K K of the camera, with zero skew and positive focal lengths. No correspondences between the two point sets are given [6 , 54 ] . A pose ξ = ( R , t ) ∈ S O ( 3 ) × R 3 \xi= (R,t) \in\mathrm{SO}(3) \times\mathbb{R}^{3} ξ = ( R , t ) ∈ SO ( 3 ) × R 3 maps a model point to camera coordinates x = R y + t x = Ry + t x = R y + t . If its depth x 3 x_3 x 3 is positive, its image proj ( x ) \operatorname{proj}(x) proj ( x ) is the first two entries of K x / x 3 Kx/x_3 K x / x 3 [27 ] . We write p j ( ξ ) = proj ( R y j + t ) p_j(\xi) = \operatorname{proj}(Ry_j + t) p j ( ξ ) = proj ( R y j + t ) .
Correspondences, outliers & noise. The correspondences form an unknown map σ : { 1 , … , n } → { 1 , … , N } ∪ { ∅ } \sigma: \{1,\ldots,n\} \to\{1,\ldots,N\} \cup\{\varnothing\} σ : { 1 , … , n } → { 1 , … , N } ∪ { ∅ } . We set σ ( i ) = j \sigma(i)=j σ ( i ) = j if observation i i i is the image of model point j j j , and σ ( i ) = ∅ \sigma(i)=\varnothing σ ( i ) = ∅ when it is the image of no model point. Distinct observations correspond to distinct model points. The m m m corresponding pairs are inliers, and the remaining n − m n-m n − m observations and N − m N-m N − m model points are outliers [6 , 82 ] . The number m m m is not known. Complete inputs have n = N = m n=N=m n = N = m . At the true pose ξ ∗ \xi^* ξ ∗ , every inlier model point has positive depth, and we model each inlier observation as
u i = p σ ( i ) ( ξ ∗ ) + w i , w i ∼ N ( 0 , s 2 I 2 ) , (1) u_i = p_{\sigma(i)}(\xi^*) + w_i,\qquad w_i \sim\mathcal{N}(0,s^2 I_2), \tag*{(1)} u i = p σ ( i ) ( ξ ∗ ) + w i , w i ∼ N ( 0 , s 2 I 2 ) , (1)
with i.i.d. noise w i w_i w i of unknown standard deviation s s s pixels per coordinate. This noise also approximately absorbs model errors [27 ] .
Task. Blind PnP is the task of estimating, from U \mathcal{U} U , Y \mathcal{Y} Y and K K K , the true pose ξ ∗ \xi^* ξ ∗ and the correspondences σ \sigma σ of Eq. (1) .
MomentPnP
In blind PnP (§ 2 ), MomentPnP estimates the camera pose from observations, a 3D model and the intrinsics, without correspondences. In general, both point sets contain outliers. Complete inputs, without outliers, are an exception that MomentPnP treats separately when this is known.
Preliminaries
Polynomial Moments. Moments are statistical averages of products of point coordinates, and their order g g g is the number of factors [19 , 34 ] . Averages need no pairing of points, so earlier correspondence-free methods use moments [16 , 31 , 72 ] , and we use orders one to four, M g M_g M g . Orders three and four capture asymmetry and concentration, which separate shapes with equal centroid and covariance (Figure 2 a–Figure 2 d).
For an observation u u u in pixels, the calibrated point u c a l u_{\mathrm{cal}} u cal consists of the first two coordinates of K − 1 ( u ⊤ , 1 ) ⊤ K^{-1}(u^{\top},1)^{\top} K − 1 ( u ⊤ , 1 ) ⊤ , where K K K is the intrinsic matrix [27 ] . The affine map h ( u ) = ( u c a l − u ˉ c a l ) / r c a l h(u)=(u_{\mathrm{cal}}-\bar{u}_{\mathrm{cal}})/r_{\mathrm{cal}} h ( u ) = ( u cal − u ˉ cal ) / r cal centers calibrated points at the mean u ˉ c a l \bar{u}_{\mathrm{cal}} u ˉ cal of the calibrated observations and divides by their root-mean-square radius r c a l r_{\mathrm{cal}} r cal , which keeps the coordinates well conditioned [28 ] . We write z = h ( u ) z=h(u) z = h ( u ) for the normalized point, and the same map normalizes the projection p j ( ξ ) p_j(\xi) p j ( ξ ) of each model point at a pose ξ \xi ξ (§ 2 ).
Stacking the fourteen monomials of orders one to four of a normalized point z z z gives the feature vector ϕ 4 ( z ) = ( z 1 , z 2 , z 1 2 , z 1 z 2 , z 2 2 , … , z 2 4 ) ∈ R 14 \phi_4(z)=(z_1,z_2,z_1^2,z_1z_2,z_2^2,\ldots,z_2^4)\in\mathbb{R}^{14} ϕ 4 ( z ) = ( z 1 , z 2 , z 1 2 , z 1 z 2 , z 2 2 , … , z 2 4 ) ∈ R 14 . Its averages μ U = 1 N ∑ i ϕ 4 ( h ( u i ) ) \mu_U=\frac{1}{N}\sum_i\phi_4(h(u_i)) μ U = N 1 ∑ i ϕ 4 ( h ( u i )) over the observations and μ Y ( ξ ) = 1 N ∑ j ϕ 4 ( h ( p j ( ξ ) ) ) \mu_Y(\xi)=\frac{1}{N}\sum_j\phi_4(h(p_j(\xi))) μ Y ( ξ ) = N 1 ∑ j ϕ 4 ( h ( p j ( ξ ))) over the projections are the moments of both point sets, and on complete inputs they determine the pose (Theorem 1).
Trigonometric moments. Monomial features fail with outliers, because one distant outlier can dominate the moments (Figure 2 e). Like sketching methods [23 ] , MomentPnP therefore uses weighted cosines and sines ϕ F ( z ) = ( cos k ⊤ z , sin k ⊤ z ) k ∈ F \phi_F(z)=(\cos k^{\top}z,\sin k^{\top}z)_{k\in F} ϕ F ( z ) = ( cos k ⊤ z , sin k ⊤ z ) k ∈ F over a finite set F F F of frequencies.1 The Fourier feature averages μ U , F \mu_{U,F} μ U , F and μ Y , F ( ξ ) \mu_{Y,F}(\xi) μ Y , F ( ξ ) are defined like μ U \mu_U μ U and μ Y ( ξ ) \mu_Y(\xi) μ Y ( ξ ) , with ϕ F \phi_F ϕ F and h F h_F h F in place of ϕ 4 \phi_4 ϕ 4 and h h h . They are the trigonometric moments [51 ] of the point distribution. Because the features are bounded, one outlier has a bounded effect, and low frequencies describe the coarse layout as low orders do [35 , 38 ] .
As in the benchmark protocol [6 ] , we consider image outliers are spread uniformly over the rectangle that bounds the observations, and model outliers through the box that bounds the 3D model. On average, an image outlier adds μ U , 0 \mu_{U,0} μ U , 0 , the average of ϕ F \phi_F ϕ F over the rectangle, and a model outlier adds μ Y , 0 ( ξ ) \mu_{Y,0}(\xi) μ Y , 0 ( ξ ) , the average of ϕ F \phi_F ϕ F over the projections of points spread through the box. With a fraction η \eta η of outliers in each point set, μ U , F \mu_{U,F} μ U , F and μ Y , F ( ξ ) \mu_{Y,F}(\xi) μ Y , F ( ξ ) mix the inlier averages with μ U , 0 \mu_{U,0} μ U , 0 and μ Y , 0 ( ξ ) \mu_{Y,0}(\xi) μ Y , 0 ( ξ ) . The inlier averages agree at the true pose, so the expected difference there is η ( μ U , 0 − μ Y , 0 ( ξ ) ) \eta(\mu_{U,0}-\mu_{Y,0}(\xi)) η ( μ U , 0 − μ Y , 0 ( ξ )) .
Definition 1 (Moment matching). Moment matching minimizes
L F ( ξ , η ) = ∥ μ U , F − μ Y , F ( ξ ) − η ( μ U , 0 − μ Y , 0 ( ξ ) ) ∥ 2 (2) \mathcal{L}_F(\xi,\eta)=\lVert\mu_{U,F}-\mu_{Y,F}(\xi)-\eta(\mu_{U,0}-\mu_{Y,0}(\xi))\rVert^2 \tag*{(2)} L F ( ξ , η ) = ∥ μ U , F − μ Y , F ( ξ ) − η ( μ U , 0 − μ Y , 0 ( ξ )) ∥ 2 (2) over the pose ξ \xi ξ and η \eta η . The objective is quadratic in the unknown η \eta η , so the best η \eta η at each pose has a closed form. On complete inputs, η = 0 \eta=0 η = 0 and no outlier can dominate the moments, so we replace the Fourier feature averages, and we minimize L 4 ( ξ ) = ∥ μ U − μ Y ( ξ ) ∥ 2 \mathcal{L}_4(\xi)=\lVert\mu_U-\mu_Y(\xi)\rVert^2 L 4 ( ξ ) = ∥ μ U − μ Y ( ξ ) ∥ 2 (Theorem 1).
Expectation–maximization (EM). EM [13 , 21 , 55 ] then refines the pose with every observation, each explained as a uniform outlier over the rectangle or a noisy projection of a model point in front of the camera (Figure 3 c). A refined pose is correct only if the observation lie much closer to their nearest projections than to the next, which we test against randomly placed projections [14 , 53 ] .
Figure 3. Overview of MomentPnP. (a) Cameras are searched over centers and orientations, and observed features averaged once. (b) Moment matching refines the best cameras, raising the cutoff from 4 to 20. (c) EM refines fits with soft correspondences. (d) The alignment check accepts poses whose observations lie much closer to their nearest projections than to the next, and the most likely accepted pose or alternative is returned. Orange: observations. Blue: model points and projections. Gray: unmatched points, unselected poses.
Definition 2 (Association level). For observation i i i , let d i , 1 ≤ d i , 2 ≤ ⋯ d_{i,1}\le d_{i,2}\le\cdots d i , 1 ≤ d i , 2 ≤ ⋯ be the distances to the projections in increasing order. For projections placed as a homogeneous Poisson process of any density, each E i = − ℓ log ( 1 − d i , 1 2 / d i , 2 2 ) E_i=-\ell\log(1-d_{i,1}^2/d_{i,2}^2) E i = − ℓ log ( 1 − d i , 1 2 / d i , 2 2 ) , which compares the nearest distance with the local spacing over ℓ ≤ 8 \ell\le8 ℓ ≤ 8 neighbors, would be unit exponential [9 ] . Observation i i i is associated when E i ≤ π / 16 E_i\le\pi/16 E i ≤ π /16 , that is, when its nearest projection lies within about a quarter of the average spacing. The association level is the probability under random placement, with independent observations, of at least the observed number of associated observations.
Camera pose estimation
MomentPnP estimates the pose by matching moments and then verifies it (Figure 3 , Algorithm 1). Step 2 adjusts the six pose parameters until the feature averages of the projected model agree with those of the observations, after subtracting
Algorithm 1: MomentPnP (Steps 1–4; expanded in Alg. A1)
Input : Observations U \mathcal{U} U , 3D model Y \mathcal{Y} Y , intrinsics K K K , angular grid spacing ϵ > 0 \epsilon> 0 ϵ > 0
Output : Pose ξ ^ \hat{\xi} ξ ^ and correspondences σ ^ \hat{\sigma} σ ^
1: ( μ U , F , μ U , 0 ) ← ObservedMoments ( U , K ) (\mu_{\mathcal{U},F}, \mu_{\mathcal{U},0}) \leftarrow\operatorname{ObservedMoments}(\mathcal{U}, K) ( μ U , F , μ U , 0 ) ← ObservedMoments ( U , K ) 2: G ← ∅ G \leftarrow\varnothing G ← ∅ ; Π ← ∅ \Pi\leftarrow\varnothing Π ← ∅ 3: repeat 4: G ← WidenCameraSearch ( G , Y , ϵ ) G \leftarrow\operatorname{WidenCameraSearch}(G, \mathcal{Y}, \epsilon) G ← WidenCameraSearch ( G , Y , ϵ ) 5: Π ′ ← CameraInitialization ( G , U ) \Pi' \leftarrow\operatorname{CameraInitialization}(G, \mathcal{U}) Π ′ ← CameraInitialization ( G , U ) 6: Π ′ ← MomentMatching ( Π ′ , μ U , F , μ U , 0 ) \Pi' \leftarrow\operatorname{MomentMatching}(\Pi', \mu_{\mathcal{U},F}, \mu_{\mathcal{U},0}) Π ′ ← MomentMatching ( Π ′ , μ U , F , μ U , 0 ) 7: Π ← Π ∪ EMRefinement ( Π ′ , U ) \Pi\leftarrow\Pi\cup\operatorname{EMRefinement}(\Pi', \mathcal{U}) Π ← Π ∪ EMRefinement ( Π ′ , U ) 8: ( ξ , a ^ ) ← AlignmentCheck ( Π ) (\xi, \hat{a}) \leftarrow\operatorname{AlignmentCheck}(\Pi) ( ξ , a ^ ) ← AlignmentCheck ( Π ) // a a a : ξ ^ \hat{\xi} ξ ^ accepted 9: until a ^ \hat{a} a ^ or G G G is at its finest level 10: if a ^ \hat{a} a ^ then 11: Π ′ ← EMRefinement ( Alternatives ( ξ ^ ) , U ) \Pi' \leftarrow\operatorname{EMRefinement}(\operatorname{Alternatives}(\hat{\xi}), \mathcal{U}) Π ′ ← EMRefinement ( Alternatives ( ξ ^ ) , U ) 12: ( ξ ^ , a ^ ) ← AlignmentCheck ( { ξ ^ } ∪ Π ′ ) (\hat{\xi}, \hat{a}) \leftarrow\operatorname{AlignmentCheck}(\{\hat{\xi}\} \cup\Pi') ( ξ ^ , a ^ ) ← AlignmentCheck ({ ξ ^ } ∪ Π ′ ) 13: σ ^ ← Correspondences ( ξ ^ , U ) \hat{\sigma} \leftarrow\operatorname{Correspondences}(\hat{\xi}, \mathcal{U}) σ ^ ← Correspondences ( ξ ^ , U ) 14: return ( ξ ^ , σ ^ ) (\hat{\xi}, \hat{\sigma}) ( ξ ^ , σ ^ )
the expected effect of outliers in both sets, which pairs no points. Because the objective has many local minima [57 ] , Step 1 supplies starting poses. Because averages ignore the individual observations, Step 3 refines the matched pose until it explains each observation on its own. Because a wrong pose can still align with the observations by chance, or through an ambiguity that averages cannot resolve, Step 4 checks the refined pose before it is returned.
Step 1: Camera initialization. Without a pose prior, the image may show any part of the model, so MomentPnP first places candidate cameras throughout it. The candidates combine camera centers on a grid over an enlarged bounding box of the model with viewing directions over the sphere and roll angles about each. We score each candidate, as in GOSMA [5 ] , by how much more densely its projections cluster at the observations than around them, measured by the ratio of a narrow to a broad smoothing of the projections, and refine the best candidates by this score.
Step 2: Moment matching. Step 2 fits the pose by minimizing the moment-matching objective of Definition 1. For a candidate camera ξ \xi ξ , a taper fades out projections outside the rectangle of the observations, so only the model points that project near the observations contribute to μ Y , F ( ξ ) \mu_{\mathcal{Y},F}(\xi) μ Y , F ( ξ ) , and the average changes smoothly as points enter or leave the view [26 ] . Starting from the candidates with distinct orientations and the lowest L F \mathcal{L}_F L F , Levenberg–Marquardt [45 , 52 ] minimizes L F \mathcal{L}_F L F over the six pose parameters under perspective projection. The fit proceeds from coarse to fine. The cutoff K F K_F K F rises from 4 to 20 in five stages, and after each stage only the fits of lowest L F \mathcal{L}_F L F with distinct orientations continue. The best fits go to EM (Figure 3 b).
Step 3: Expectation–maximization refinement. For refinement we adapt EM-ICP and coherent point drift [21 , 55 ] to perspective projection. EM refines each fit with the mixture of uniform outliers and noisy projections of Sec. 4.1, and estimates the noise standard deviation and the outlier probability together with the pose (Figure 3 c). The refined poses are then ranked by the likelihood of the observations as a Poisson process of uniform outliers and Gaussian peaks at the projections [9 ] . A refined pose can still be wrong in two ways. It can place projections near many observations by chance, or it can be one of the poses that give nearly the same image as the truth. Step 4 tests for both.
Step 4: Alignment check and pose ambiguity resolution. An observation is matched when its nearest projection explains it better than the outlier term does. The alignment check accepts a refined pose when enough observations are matched, their median ratio d i , 1 / d i , 2 d_{i,1}/d_{i,2} d i , 1 / d i , 2 is small, and the association level of Definition 2 is very small. Acceptance does not settle the pose of a nearly symmetric or shallow model, because other poses give nearly the same image and the averages cannot separate them (§ 3.3 ). For a distant camera [12 ] , the poses that give the image the same mean and covariance as an accepted pose with rotation R R R are its seven alternatives R H k RH_k R H k and D R H k DRH_k D R H k in Figure 3 d, where H 1 H_1 H 1 , H 2 H_2 H 2 , H 3 H_3 H 3 turn the model by 180 ∘ 180^\circ 18 0 ∘ about the principal axes of its matched points, H 0 H_0 H 0 is the identity, and D D D turns the camera by 180 ∘ 180^\circ 18 0 ∘ about the optical axis. That these seven are all the poses the mean and covariance cannot separate is proved in App. § 8 .7.2 Which of them EM then prefers is not proved, but on the benchmarks 63 of the 64 poses that an alternative replaced end within 1 ∘ 1^\circ 1 ∘ (App. § 10 .7). MomentPnP therefore refines the most likely accepted pose and its seven alternatives by EM under perspective, checks each, and returns the most likely of them. Without an accepted pose, the search widens and then returns the most likely refined pose. At the returned pose, each matched observation corresponds to its nearest projection, and every other observation is an outlier, σ ^ ( i ) = ∅ \hat{\sigma}(i)=\varnothing σ ^ ( i ) = ∅ . When several observations share a nearest projection, only the closest keeps it, so distinct observations correspond to distinct model points. The refit of an accepted pose to its correspondences is given in App. § 9 .
Uniqueness and accuracy of the pose
Moment matching recovers the pose only if two properties hold. On ideal complete inputs, its objective vanishes at the true pose alone (uniqueness). With noise and outliers, its minimizer stays within a bounded distance of the true pose (accuracy). Theorem 1 proves uniqueness for complete inputs. Theorem 2 builds on it to prove both for L F \mathcal{L}_F L F .
Uniqueness for complete inputs. On complete noiseless inputs, L 4 \mathcal{L}_4 L 4 vanishes at the true pose and, for a symmetric model such as a cube, at a second pose. Such models are exceptional. Equal moments at a second pose impose polynomial equations on the model points, and a dimension count shows that almost no model of N ≥ 20 N \ge20 N ≥ 20 points satisfies them.
Theorem 1 (Pose uniqueness for complete inputs). Fix the true pose ξ ⋆ \xi^\star ξ ⋆ of a calibrated camera. For almost every model of N ≥ 20 N \ge20 N ≥ 20 points in front of the camera, every pose that keeps the points in front of the camera and gives their projections the same fourteen moments as ξ ⋆ \xi^\star ξ ⋆ equals ξ ⋆ \xi^\star ξ ⋆ .
Only the true pose therefore has L 4 = 0 \mathcal{L}_4=0 L 4 = 0 , which justifies minimizing L 4 \mathcal{L}_4 L 4 .3
Accuracy of the Fourier fit. On ideal inputs, zero L F \mathcal{L}_F L F over all frequencies determines the inlier projections and the outlier fraction, since the cosine and sine averages uniquely determine a distribution [38 ] . The pose is then determined by Theorem 1. Observation noise and finite sampling of the outliers generally make the loss nonzero even at the true pose. Theorem 2 bounds the error at a finite cutoff: for every cutoff K F K_F K F above a threshold, the loss and the perturbations bound the distance dist \operatorname{dist} dist between any candidate ( ξ , η ) (\xi,\eta) ( ξ , η ) and the truth ( ξ ⋆ , η ⋆ ) (\xi^\star,\eta^\star) ( ξ ⋆ , η ⋆ ) . This distance combines the rotation, translation and image-outlier fraction errors.
Theorem 2 (Accuracy of the Fourier fit). Let the outliers be drawn independently and uniformly from the rectangle of the observations and the bounding box of the model, and let a compact set Θ \Theta Θ of pairs ( ξ , η ) (\xi,\eta) ( ξ , η ) contain ( ξ ⋆ , η ⋆ ) (\xi^\star,\eta^\star) ( ξ ⋆ , η ⋆ ) , with every pose keeping the box in front of the camera and η ∈ [ 0 , 1 ) \eta\in[0,1) η ∈ [ 0 , 1 ) . For almost every model of m ≥ 20 m \ge20 m ≥ 20 inlier points and every cutoff K F K_F K F above a threshold K 0 K_0 K 0 , there is a constant C F C_F C F such that, for each δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) , with probability at least 1 − δ 1-\delta 1 − δ over the outliers, every ( ξ , η ) ∈ Θ (\xi,\eta)\in\Theta ( ξ , η ) ∈ Θ satisfies
dist ≤ C F ( L F ( ξ , η ) + Δ δ ) . \operatorname{dist} \le C_F\left(\sqrt{\mathcal{L}_F(\xi,\eta)}+\Delta_\delta\right). dist ≤ C F ( L F ( ξ , η ) + Δ δ ) .
The error term Δ δ \Delta_\delta Δ δ adds the sampling and grid errors of the outlier averages, the noise error and twice the difference of the image and model outlier fractions.
At the true pair, L F \sqrt{\mathcal{L}_F} L F is at most Δ δ \Delta_\delta Δ δ , so the minimizing pair lies within 2 C F Δ δ 2C_F\Delta_\delta 2 C F Δ δ of the truth, and L F \mathcal{L}_F L F can select fits for EM. The constant C F C_F C F is large when L F \mathcal{L}_F L F grows slowly away from the truth on ideal inputs. The error term measures the departure from this ideal, and only its sampling part, which shrinks as the point sets grow, depends on δ \delta δ .
For nearly symmetric and distant planar models, the averages barely change between a pose and its alternatives in § 3.2 , and C F C_F C F is large. Step 4 therefore compares these alternatives by EM on the individual observations. § 4 tests MomentPnP in practice, and the appendix tests it beyond the theorems’ assumptions, with structured outliers (Figure A14 ), partial views (Tab. A14) and large depth spreads (Table A19 ); proofs and further conditions in Apps. § 8 .§ 1 and A.6, further details in Apps. § 10 .§ 2 and C.7.
Table A19. Depth variation with complete point sets. Ten paired trials per condition, each with 128 points and Gaussian noise of standard deviation s = 2 s = 2 s = 2 pixels per coordinate at a focal length of 800 pixels. Depths vary about z r e f = 4.5 z_{\mathrm{ref}} = 4.5 z ref = 4.5 . The second column gives the median over trials of the ratio x 3 , max / x 3 , min x_{3,\max}/x_{3,\min} x 3 , m a x / x 3 , m i n of the largest to the smallest camera-frame depth x j 3 = ( R y j + t ) 3 x_{j3} = (R y_j + t)_3 x j 3 = ( R y j + t ) 3 . Error is the median rotation error and Time the median total runtime. Affine initialization returns the candidate with the lowest moment residual, before perspective fitting and EM. The full MomentPnP completes its search and refinement. Bold marks the lower error and runtime in each row.
Figure A14. Noise, structured outliers and acceptance. (a) Rotation error (left) and percentage of correct labels (right) of MomentPnP on synthetic inputs against N s ^ / ρ r m s N\hat{s}/\rho_{\mathrm{rms}} N s ^ / ρ rms , one curve per point count N N N . Crosses mark the pose fitted to the true correspondences (left) and minimum-cost bijective matching at the true pose (right). (b) Percentage of Data61 [56 ] frames with rotation error below 1 ∘ 1^\circ 1 ∘ for MomentPnP and BPnPNet, against the percentage of 2D points replaced by other 2D points on the scene, one curve per N N N . (c) Percentage of the 1,000 held-out frames of MegaDepth [46 ] , ModelNet40 [79 ] and Data61 at each outlier ratio, split jointly by acceptance and by rotation error below 1 ∘ 1^\circ 1 ∘ .
Experiments
Metrics. We report the median rotation error, the angle in degrees between estimated and true rotations [6 ] , and the median relative translation error, the distance between the estimated and true translations divided by the true translation length [44 ] . Accuracy is the percentage of frames with rotation error below 1 ∘ 1^\circ 1 ∘ , counting missing estimates as failures.
Blind PnP benchmarks
Benchmarks. We compare learned and learning-free blind PnP methods, including our reproduction of CFPE [60 ] , on the test sets of the released MegaDepth [46 ] , ModelNet40 [79 ] and Data61 [56 ] benchmarks. EM-ICP (ours) is our own Step 3, the EM refinement of § 3.2 , run from random poses without Steps 1 and 2 and restarted until the time limit, so Table 1 also shows what refinement alone achieves. In the benchmark protocol [6 ] , inliers are the released 2D–3D pairs, and outliers are drawn uniformly from the bounding box of each point set. We tuned the method on the test sets, so we also test on 1,000 held-out frames per dataset from the training split. Table 1 shows that no method has a lower median rotation error than MomentPnP on any test set.
Compared with the best learned baseline, MomentPnP’s median rotation error is 5 to 170 times smaller, and its median translation error is 9 to 320 times smaller. On the sampled test frames of Table 1 , its accuracy is 97.3 to 100% without outliers and 82.3 to 97.0% with as many outliers as inliers. Averaged over the three test sets without outliers, 99.1% of frames are within 1 ∘ 1^\circ 1 ∘ , against 38.3% for the best learned baseline. The curve of MomentPnP lies above those of all learned baselines at every threshold (Figure 5 , Figure A3 ). On the held-out frames, its accuracy is 98.6 to 100% without outliers and 72.4 to 91.4% with as many outliers as inliers (Table A8 ).
Figure 5. Rotation-error distribution without outliers. Each curve gives the percentage of frames below each rotation error, averaged with equal weight over the MegaDepth, ModelNet40 and Data61 test sets. The vertical line marks 1°.
Table A8. MomentPnP (ours) on every evaluated split of MegaDepth [46 ] , ModelNet40 [79 ] and Data61 [56 ] . The outlier ratio is the number of added outliers per point set divided by its inlier count. Rotation-accuracy percentages use every evaluated frame, independently of the acceptance test. Acceptance columns count frames, and timing columns give milliseconds per frame. Test rows use the full released test sets. Held-out rows use 1,000 randomly selected training-split frames per dataset that were not used for tuning.
Figure A3. Accuracy with outliers and further error distributions on MegaDepth [46 ] , ModelNet40 [79 ] and Data61 [56 ] . (a) Percentage of frames with rotation error below 1 ∘ 1^\circ 1 ∘ against the outlier ratio. (b) Percentage of frames below each rotation error, with as many outliers as inliers. (c) Percentage of outlier-free frames below each relative translation error. Vertical lines mark 1 ∘ 1^\circ 1 ∘ and 1 % 1\% 1% . EM-ICP uses 100 of the comparison frames per condition, and the other methods 300 frames with outliers and 1,000 without. CFPE appears in (b) and (c).
Further datasets. We also evaluate on Cambridge Landmarks [40 ] , Phototourism [37 , 69 ] , 7-Scenes [62 , 68 ] and ETH3D [65 ] . The tracked 2D points of each image and the model points of their tracks form complete inputs (Table 2 ). Learned methods use the MegaDepth benchmark weights, and MomentPnP uses the same settings as on the earlier benchmarks. All 169 MomentPnP rotation errors are below 1°, compared with 95% for reproduced CFPE and at most 48% for learned methods. With half as many uniform outliers as inliers in each point set, MomentPnP reaches 95% accuracy on the 126 images with at most 2,000 inliers (Table A12 ). The best learned baseline reaches 33%. On all 169 images, without the inlier limit, MomentPnP reaches 87% and the best learned baseline 29% (Table A13 ). Figure 4 shows real scenes (Figure A7 ) and Figure 6 frames of the point-only benchmarks.
Figure 4. Qualitative results on real-world scenes. Columns are images, titled by scene and number of observations, and rows are methods. Orange points are observations, and blue crosses are projections under the recovered pose. Below each panel is its rotation error, in bold where lowest in the column, and a green border marks an error below 1 ∘ 1^\circ 1 ∘ . Model points come from the released structure-from-motion reconstructions of Cambridge Landmarks [40 ] , Phototourism [37 ] , 7-Scenes [62 ] and ETH3D [65 ] , and from LiDAR for Data61 [56 ] .
Figure 6. Qualitative results on the point-only benchmarks. MegaDepth and ModelNet40 provide coordinates without photographs. Markers and labels follow Figure 4 .
Table 2. Blind PnP on structure-from-motion (SfM) inputs. Accuracy (%) on the Cambridge Landmarks (C), Phototourism (P), 7-Scenes (S, depth-assisted SfM) and ETH3D (E) datasets (errors and times in Tab. A12). Bold: highest, including ties.
Figure A7. Qualitative results on SfM inputs. Each dataset is shown in its first selected outlier-free image, with one row per method. Orange dots are the tracked 2D points, and blue crosses are the model points projected under the returned pose. Titles give the scene and the point count. Labels give rotation error, with the lowest in bold, and green borders mark errors below 1 ∘ 1^\circ 1 ∘ .
Table A12. SfM inputs per dataset, extending Table 2 . Images/scenes counts the evaluated images and scenes, and m m m gives the range of inlier counts per image. For each method, the columns give the percentage of all images with rotation error below 1 ∘ 1^\circ 1 ∘ (↑ \uparrow ↑ ), the median rotation error in degrees over returned poses (↓ \downarrow ↓ ), and the median time per image in milliseconds (↓ \downarrow ↓ ). The lower panel adds ⌊ m / 2 ⌋ \lfloor m/2 \rfloor ⌊ m /2 ⌋ uniform outliers to each point set and keeps the images with m ≤ 2,000 m \leq2{,}000 m ≤ 2 , 000 . Times of Deep Blind PnP and BPnPNet include all evaluated pose outputs, and their errors use the selected output. Without outliers, every method returns all 169 poses. With outliers, MomentPnP returns 125 of 126 poses and the other methods all 126. Bold marks the highest accuracy and the lowest error and time within each dataset and panel.
Table A12.
Table A13. SfM inputs with outliers, without the inlier limit. All 169 images of Table 2 , with ⌊ m / 2 ⌋ \lfloor m/2 \rfloor ⌊ m /2 ⌋ outliers added to each point set for m m m inliers. Inlier counts range from 128 to 5,010 per image, which gives 192 to 7,515 points in each point set. Within 1 ∘ 1^\circ 1 ∘ gives the number of images with rotation error below 1 ∘ 1^\circ 1 ∘ and their percentage of all 169. OOM means out of GPU memory, and Other exceptions counts the remaining runs without a pose. Times include failed runs and, for Deep Blind PnP and BPnPNet, all evaluated pose outputs. Bold marks the highest recovery and the lowest runtimes.
Table A13.
Large point sets and full scenes
Scaling to large point sets. Beyond the benchmark sizes of up to about 12,000 points, more points reduce the random part of the noise in the moments but crowd the projections that each correspondence must separate. We test both effects on three ETH3D laser scans [65 ] , each rendered into three views with one pixel of noise, ignoring occlusion. From 1,000 to 256,000 points, the median rotation error falls from 0.002° to 0.00009° (Figure 8 ), and at one million points it is 0.00007° (Figure 7 , Tab. A14), in about 20 minutes per frame with GPU memory linear in the point count (Figure A1 ). The learned baselines bring at most two of the nine poses within 1° and return no pose from 64,000 points (Table A17 ).
Figure 7. Recovered poses at one million points. ETH3D laser-scan model points [65 ] of three scans, shaded by depth and projected under the poses estimated from 1,000,000 2D points. Labels give rotation error. Each inset compares observations (orange) with projections under the recovered pose (blue) in a 30-pixel window.
Figure A1. Measured time and memory on rendered ETH3D scans. (a) Time per frame. Light markers are single views and lines are medians. The triangle marks the Courtyard view that returned no pose within its 1,800-second limit. (b) GPU allocator memory, which is equal across the views of each size, and the peak CPU memory of the four one-million-point runs that recorded it (gray). The horizontal line is the memory of the specified GPU.
Table A17. Learned baselines on rendered 2D points. Poses with rotation error below 1 ∘ 1^\circ 1 ∘ over the nine views of Tab. A14. OOM marks point counts at which a method runs out of memory and returns no pose. Bold marks the highest count at each point count.
At one million points, 21 to 25% of the 2D points lie nearer to another projection than to the one that generated them, up from none at 1,000 points, so a nearest-projection labeling cannot resolve them (Figure A11 ). The rotation error nevertheless falls over the same range, because moment matching and EM use every point without requiring each correspondence to be correct.
Figure A11. Incorrect labels on rendered 2D points. Median percentage, over the reference poses of each scan, of 2D points whose nearest projected model point under the recovered pose did not generate them.
Full scene reconstructions. We also localize single images against complete ETH3D scene reconstructions, where each query image observes an unknown part of the whole scene without a pose prior. MomentPnP recovers all 39 tuning queries within 1° and 35 of 39 held-out test queries (Table A18 ). On the tuning queries, the best learned baseline, Deep Blind PnP, recovers two.
Ablation study and failure modes
Role of moment matching. Three studies isolate what moment matching contributes. (i) From the same initial poses on 100 test frames per benchmark, moments through order three or four bring 53 to 78 of these poses within 1° and 1% translation error before EM (Table A9 ). The mean and covariance bring at most two, because distinct orientations can share them. (ii) EM-ICP (ours), our EM refinement from random poses restarted until a 30-second limit, needs those 30 seconds per frame to reach the median errors that MomentPnP reaches in at most 0.06 seconds without outliers (Table A4 ). With as many outliers as inliers, it reaches at most 31% accuracy (Table 1 , Figure A4 ). (iii) Given a domain that contains the true camera center and about six hours per search, GOPAC and GOSMA bring none of the thirty outlier-free frames within 1°. MomentPnP needs no such domain and brings all thirty within 1°, in about 2 seconds per frame (Table 3 , Table A10 ). Controls on the depth spread, the image noise and the kind of outliers are in App. C.7. Under noise, the rotation error follows that of the pose fitted to the true correspondences, and with outliers on scene structure, every frame with up to 30% replaced stays within 1°.
Table 3. Global search on complete inputs. On thirty outlier-free test frames, GOPAC, from a zero or an annealed initial bound, and GOSMA search by branch and bound over rotations and over camera centers in a cuboid that contains the true center. Finished counts the searches that ended on their own. The others were stopped after 5.2 to 6.0 hours and scored at their best pose so far. Errors cover all thirty frames, and Time is the median over finished searches. Bold: best per column. Further details in App. C.3.
Table A4. Median time per frame (ms, ↓) on the comparison frames of Table 1 , at each outlier ratio. Each learned method is timed with its most accurate pose output. Bold marks the lowest time in each column of each dataset.
Figure A4. Accuracy against time on the outlier-free frames of MegaDepth [46 ] , ModelNet40 [79 ] and Data61 [56 ] . Each curve gives the percentage of frames with a pose within 1 ∘ 1^\circ 1 ∘ by a given time. EM-ICP, SoftPOSIT, P3P-RANSAC and GOSMA keep their best pose so far until the 30-second limit, and the curves score it at each time. For the other methods, a frame counts once a pose within 1 ∘ 1^\circ 1 ∘ is returned.
Table A9. Moment order with shared refinement (ours) on outlier-free inputs. Entries count accurate poses before / after the same EM refinement, out of the n r n_r n r inputs in each row. Direct EM refines the initial poses without moment matching, and Degree g g g matches the average of every monomial through degree g g g . A pose is accurate when its rotation error is below 1 ∘ 1^\circ 1 ∘ and its translation error is below 1 % 1\% 1% of the reference translation norm.
Table A10. Global search on complete inputs without the released time limits, all outcomes. Each method runs on the thirty outlier-free frames of Table 3 . Columns count the frames with rotation error below 1° and 5°, the searches that ended with GOPAC’s optimality flag or at GOSMA’s queue-size limit, and the stagnated GOPAC searches that we stopped after five to six hours. We score them at their best pose so far. Times are the median and the largest wall time of the finished searches.
Table A10.
Independently selected inputs and failure modes. So far, every inlier 2D point is the image of a model point (Figure 1 ), but 2D points detected without reference to the 3D model may have no corresponding model point. With SIFT keypoints [49 ] in the query image and a COLMAP scene model [64 ] built from other photographs, the search over
camera poses still recovers Facade 26 within 1° (Figure 9 (a), Table A22 ). Such inputs can fail in two ways. In Meadow, only 255 of the 12,069 detections belong to reconstruction tracks, and many others lie on vegetation and sky absent from the model (Figure 9 (b), Table A23 ). A dense model poses a second difficulty, because projections can lie near most 2D points even at an incorrect pose. In another Facade image, 81% of the 2D points lie within 3.4 pixels of a projected laser-scan point at the reference pose, and 77% still do at a pose rotated by 2° (Figure 9 (c)); further details of every study in App. C. All code will be released.
Figure 9. Independently selected inputs. (a) Facade 26, recovered from SIFT detections and a model built from other photographs. The camera is orange, with pink rays to matched model points. (b) Meadow, where only the pink diamonds belong to reconstruction tracks. (c) Facade laser scan, where pink diamonds lie within 3.4 pixels of blue projections.
Table A22. Recovery from independently selected inputs (ours). A case is recovered when its rotation and position errors are below the listed thresholds. Position error is the camera-center error or, for the infrared LEDs, the translation error of the motion relative to frame 61. Raw evaluates the returned pose, and Symmetry applies the official T-LESS model symmetry. We count every selected case, including extraction failures and poses returned at the search limit.
Table A23. SIFT queries against independently built maps (ours). Column 2D points counts every SIFT detection in the query image, including several orientations at one location, and Locations counts distinct image locations. Center gives the camera-center error as a percentage of the scene extent (App. C.6). Recovered requires rotation error below 1 ∘ 1^\circ 1 ∘ and camera-center error below 1%, and Accepted states whether the returned pose passes the acceptance tests. Times above 3,600 s mark searches stopped at the limit.
Conclusion
In blind PnP, the camera pose need not wait for the correspondences. Finitely many feature averages of the two point sets determine it, and the correspondences then follow from nearest projections. The theory explains why this works on complete inputs and with uniform outliers, and the experiments show that it holds beyond these assumptions. Learned methods inherit the limits of their training data [63 , 73 ] , lose accuracy outside it and run out of memory on large point sets. Using neither training nor descriptors, this inference offers an alternative to both and to branch-and-bound search.
Limitations. When 2D points have no corresponding model point, the feature averages need not match even at the true pose, so independently selected inputs remain harder than tracked ones. Adversarial outliers can prevent recovery.
Future work. Missing structure and unequal sampling might be handled by reweighting the projections with unbalanced optimal transport [10 ] . Moment matching may also extend to relative pose estimation and to monocular depth estimation of unknown scale [15 , 80 ] .
LLM Usage Statement
We used OpenAI’s Codex to assist with editing and refining prose, code and implementation, figures, and formatting. No LLMs or AI tools were used to develop the research ideas, hypotheses, scientific claims, or conclusions. The authors take full responsibility for the accuracy and integrity of the paper and its accompanying artifacts.
References
App. A gives the notation, method details and proofs, App. B the implementation, and App. C the experiments. Appendix figures, tables, equations and statements carry the prefix A, and unprefixed numbers refer to the main paper.
Method details and proofs
Table A1 summarizes the notation of the main paper and the appendix.
Table A1. Notation. The first four groups follow the main-text sections, including symbols used only in their appendix analyses. The last group covers further appendix analyses. A star marks a true value, a hat an estimate and a tilde a coordinate or moment after covariance normalization. Vector norms are Euclidean, unqualified tensor and matrix norms are Frobenius, and moment pairs use the direct-sum norm. Subscripts op, th and E mark the Euclidean operator norm, constants of the threshold proof and edge quantities.
Moments & measurement error
This section bounds how much image noise changes the moments of the observations and describes the effect of errors in the model points. In the bound, the terms that are linear in the noise have zero mean, and their contribution decreases as 1 / N 1/\sqrt{N} 1/ N when the point sets grow (§ 3.3 and 5). The second part concerns the assumption that the model points are exact (§ 2 ). We first illustrate what moments distinguish and fix the coordinates in which we measure them.
A scalar example. The sets of real numbers { − 2 , − 1 , 3 } \{-2,-1,3\} { − 2 , − 1 , 3 } and { 2 , 1 , − 3 } \{2,1,-3\} { 2 , 1 , − 3 } are reflections of each other about zero. Both have mean zero and second moment 14 / 3 14/3 14/3 , so their first two moments do not distinguish them. Their third moments are 6 6 6 and − 6 -6 − 6 , because ( − 2 ) 3 + ( − 1 ) 3 + 3 3 = 18 (-2)^3+(-1)^3+3^3=18 ( − 2 ) 3 + ( − 1 ) 3 + 3 3 = 18 and the reflection changes the sign of every cube. An average does not depend on the order of the points, so the third moment distinguishes the two sets without any correspondence between their points. Moment matching uses the same property in the image, where the moments of orders three and four distinguish orientations whose projections have equal mean and covariance (Figure 2 ). The moment of order g g g of a 2D point set has several distinct entries, one for each monomial of degree g g g , and we now fix how we index them.
Moment coordinates. For a nonempty finite collection A \mathcal{A} A of points in a Euclidean space and a positive integer g g g , the tensor power a ⊗ g a^{\otimes g} a ⊗ g of a point a ∈ A a\in\mathcal{A} a ∈ A contains all products of g g g coordinate components, including repeated entries. The moment tensor is their average,
M g ( A ) = 1 ∣ A ∣ ∑ a ∈ A a ⊗ g . (3) \mathcal{M}_g(\mathcal{A})=\frac{1}{|\mathcal{A}|}\sum_{a\in\mathcal{A}}a^{\otimes g}. \tag*{(3)} M g ( A ) = ∣ A ∣ 1 a ∈ A ∑ a ⊗ g . (3)
We index the points of A ⊂ R d A \mathcal{A}\subset\mathbb{R}^{d_{\mathcal{A}}} A ⊂ R d A as a 1 , … , a ∣ A ∣ a_1,\ldots,a_{|\mathcal{A}|} a 1 , … , a ∣ A ∣ . A multi-index α = ( α 1 , … , α d A ) \alpha=(\alpha_1,\ldots,\alpha_{d_{\mathcal{A}}}) α = ( α 1 , … , α d A ) of nonnegative integers with ∣ α ∣ = α 1 + ⋯ + α d A = g |\alpha|=\alpha_1+\cdots+\alpha_{d_{\mathcal{A}}}=g ∣ α ∣ = α 1 + ⋯ + α d A = g names one distinct entry of the symmetric tensor M g ( A ) \mathcal{M}_g(\mathcal{A}) M g ( A ) of Eq. (3) , namely the average
m α ( A ) = 1 ∣ A ∣ ∑ i = 1 ∣ A ∣ ∏ r = 1 d A a i , r α r . m_\alpha(\mathcal{A})=\frac{1}{|\mathcal{A}|}\sum_{i=1}^{|\mathcal{A}|}\prod_{r=1}^{d_{\mathcal{A}}}a_{i,r}^{\alpha_r}. m α ( A ) = ∣ A ∣ 1 i = 1 ∑ ∣ A ∣ r = 1 ∏ d A a i , r α r .
We collect these entries, in a fixed order, in the vector m g ( A ) = ( m α ( A ) ) ∣ α ∣ = g m_g(\mathcal{A})=(m_\alpha(\mathcal{A}))_{|\alpha|=g} m g ( A ) = ( m α ( A ) ) ∣ α ∣ = g , which has ( d A + g − 1 g ) \binom{d_{\mathcal{A}}+g-1}{g} ( g d A + g − 1 ) entries. The tensor repeats the entry m α m_\alpha m α once for each ordering of its indices, that is, ( g α ) = g ! / ( α 1 ! ⋯ α d A ! ) \binom{g}{\alpha} =g!/(\alpha_1!\cdots\alpha_{d_{\mathcal{A}}}!) ( α g ) = g ! / ( α 1 ! ⋯ α d A !) times, so its Frobenius norm is
∥ M g ( A ) ∥ 2 = ∑ ∣ α ∣ = g ( g α ) m α ( A ) 2 . \|\mathcal{M}_g(\mathcal{A})\|^2=\sum_{|\alpha|=g}\binom{g}{\alpha}m_\alpha(\mathcal{A})^2. ∥ M g ( A ) ∥ 2 = ∣ α ∣ = g ∑ ( α g ) m α ( A ) 2 .
In 2D, the multiplicities are ( 1 , 3 , 3 , 1 ) (1,3,3,1) ( 1 , 3 , 3 , 1 ) for cubic and ( 1 , 4 , 6 , 4 , 1 ) (1,4,6,4,1) ( 1 , 4 , 6 , 4 , 1 ) for quartic moments. The fourteen averages of ϕ 4 ∘ h \phi_4\circ h ϕ 4 ∘ h in moment matching (§ 3.1 ) are the distinct entries of M 1 \mathcal{M}_1 M 1 to M 4 \mathcal{M}_4 M 4 of the mapped points, without these multiplicities. App. § 8 .§ 2 uses the form ζ ↦ ⟨ M g ( A ) , ζ ⊗ g ⟩ \zeta\mapsto\langle\mathcal{M}_g(\mathcal{A}),\zeta^{\otimes g}\rangle ζ ↦ ⟨ M g ( A ) , ζ ⊗ g ⟩ in place of the tensor. Its coefficients ( g α ) m α \binom{g}{\alpha}m_\alpha ( α g ) m α also differ from m g m_g m g . We state the noise bound below in the tensor norm, which Theorem A1 also uses.
Image noise. The bound below concerns complete inputs in pixel coordinates, before the map h h h is applied, under the measurement model of Eq. (1) . Its first term shrinks as 1 / N 1/\sqrt{N} 1/ N , and its second term, of order s 2 s^2 s 2 , does not depend on N N N .
Lemma A1 (Moments under image noise). Let the inputs be complete, with N ≥ 1 N\ge1 N ≥ 1 exact model points at positive depth under the true pose ξ ⋆ \xi^\star ξ ⋆ . For each observation i i i , let u i ⋆ = p σ ( i ) ( ξ ⋆ ) u_i^\star=p_{\sigma(i)}(\xi^\star) u i ⋆ = p σ ( i ) ( ξ ⋆ ) be its noiseless position, the projection of the model point that it images, with ∥ u i ⋆ ∥ ≤ ρ \|u_i^\star\|\le\rho ∥ u i ⋆ ∥ ≤ ρ . Let u i = u i ⋆ + w i u_i=u_i^\star+w_i u i = u i ⋆ + w i with independent noise vectors w i ∼ N ( 0 , s 2 I 2 ) w_i\sim\mathcal{N}(0,s^2I_2) w i ∼ N ( 0 , s 2 I 2 ) and 0 ≤ s ≤ 1 0\le s\le1 0 ≤ s ≤ 1 pixels, and let U ⋆ = { u i ⋆ } \mathcal{U}^\star=\{u_i^\star\} U ⋆ = { u i ⋆ } . For every order g ≥ 1 g\ge1 g ≥ 1 , the moments M g ( U ) \mathcal{M}_g(\mathcal{U}) M g ( U ) and M g ( U ⋆ ) \mathcal{M}_g(\mathcal{U}^\star) M g ( U ⋆ ) agree almost surely when s = 0 s=0 s = 0 and satisfy
E ∥ M g ( U ) − M g ( U ⋆ ) ∥ 2 ≤ g ρ g − 1 s 2 / N + C g , ρ s 2 . (4) \sqrt{\mathbb{E}\|\mathcal{M}_g(\mathcal{U})-\mathcal{M}_g(\mathcal{U}^\star)\|^2}\le g\rho^{g-1}s\sqrt{2/N}+C_{g,\rho}s^2. \tag*{(4)} E ∥ M g ( U ) − M g ( U ⋆ ) ∥ 2 ≤ g ρ g − 1 s 2/ N + C g , ρ s 2 . (4) Here C g , ρ ≥ 0 C_{g,\rho}\ge0 C g , ρ ≥ 0 depends only on g g g and ρ \rho ρ , and the expectation is over the image noise, in pixel coordinates before normalization.
Proof of Lemma A1. We fix an integer g ≥ 1 g\ge1 g ≥ 1 . The proof first shows that M g ( U ⋆ ) \mathcal{M}_g(\mathcal{U}^\star) M g ( U ⋆ ) is a quantity that moment matching can evaluate, and then separates the noise terms that averaging reduces from those that it does not.
Step 1. The projected moments do not depend on the correspondences. Since σ \sigma σ is a bijection of { 1 , … , N } \{1,\ldots,N\} { 1 , … , N } , substituting j = σ ( i ) j=\sigma(i) j = σ ( i ) reorders the sum,
M g ( U ⋆ ) = 1 N ∑ i = 1 N p σ ( i ) ( ξ ⋆ ) ⊗ g = 1 N ∑ j = 1 N p j ( ξ ⋆ ) ⊗ g . (5) \mathcal{M}_g(\mathcal{U}^\star)=\frac{1}{N}\sum_{i=1}^{N}p_{\sigma(i)}(\xi^\star)^{\otimes g}=\frac{1}{N}\sum_{j=1}^{N}p_j(\xi^\star)^{\otimes g}. \tag*{(5)} M g ( U ⋆ ) = N 1 i = 1 ∑ N p σ ( i ) ( ξ ⋆ ) ⊗ g = N 1 j = 1 ∑ N p j ( ξ ⋆ ) ⊗ g . (5)
The right-hand side depends on the pose and the model but not on σ \sigma σ . The lemma therefore compares the observed moments with the moments of the projections at the true pose, which can be computed without correspondences.
Step 2. Expansion in the noise. For s = 0 s = 0 s = 0 , the noise vanishes almost surely, so U = U ⋆ U = U^\star U = U ⋆ and the two moments agree almost surely. We assume s > 0 s > 0 s > 0 from now on. We expand ( u i ⋆ + w i ) ⊗ g (u_i^\star+ w_i)^{\otimes g} ( u i ⋆ + w i ) ⊗ g by choosing the noiseless position or the noise vector in each of its g g g factors. Subtracting ( u i ⋆ ) ⊗ g (u_i^\star)^{\otimes g} ( u i ⋆ ) ⊗ g leaves the 2 g − 1 2^g - 1 2 g − 1 products that contain at least one noise factor. Products with one noise factor have zero mean, so averaging over the N N N independent observations reduces their root mean square. Products with several noise factors can have a nonzero mean that averaging does not remove, but they are of order s 2 s^2 s 2 . We bound the two groups separately and add their root-mean-square bounds by Minkowski’s inequality.
Step 3. One noise factor. For each of the g g g positions of the noise factor, the products are linear in w i w_i w i , independent across i i i and of zero mean, with second moment ∥ u i ⋆ ∥ 2 ( g − 1 ) E ∥ w i ∥ 2 ≤ 2 s 2 ρ 2 ( g − 1 ) \lVert u_i^\star\rVert^{2(g-1)}\mathbb{E}\lVert w_i\rVert^2 \le2s^2\rho^2(g-1) ∥ u i ⋆ ∥ 2 ( g − 1 ) E ∥ w i ∥ 2 ≤ 2 s 2 ρ 2 ( g − 1 ) . Independence and zero means make the cross terms vanish when we square their average over i i i . Each average therefore has root mean square at most ρ g − 1 s 2 / N \rho^{g-1}s\sqrt{2/N} ρ g − 1 s 2/ N , and the g g g positions contribute the first term of Eq. (4) .
Step 4. Several noise factors. A product with a ≥ 2 a \ge2 a ≥ 2 noise factors has norm ∥ u i ⋆ ∥ g − a ∥ w i ∥ a ≤ ρ g − a ∥ w i ∥ a \lVert u_i^\star\rVert^{g-a}\lVert w_i\rVert^a \le\rho^{g-a}\lVert w_i\rVert^a ∥ u i ⋆ ∥ g − a ∥ w i ∥ a ≤ ρ g − a ∥ w i ∥ a . Because ∥ w i ∥ 2 / s 2 \lVert w_i\rVert^2/s^2 ∥ w i ∥ 2 / s 2 is exponentially distributed with mean two, E ∥ w i ∥ 2 a = 2 a a ! s 2 a \mathbb{E}\lVert w_i\rVert^{2a} = 2^a a! s^{2a} E ∥ w i ∥ 2 a = 2 a a ! s 2 a , so the product has root mean square at most ρ g − a 2 a a ! s a \rho^{g-a}\sqrt{2^a a!}\,s^a ρ g − a 2 a a ! s a . Minkowski’s inequality gives the same bound for its average over i i i , and there are ( g a ) \binom{g}{a} ( a g ) choices of positions for the a a a noise factors. Summing both groups gives
( E ∥ M g ( U ) − M g ( U ⋆ ) ∥ 2 ) 1 / 2 ≤ g ρ g − 1 s 2 / N + ∑ a = 2 g ( g a ) ρ g − a 2 a a ! s a . (6) \left(\mathbb{E}\lVert\mathcal{M}_g(U)-\mathcal{M}_g(U^\star)\rVert^2\right)^{1/2}
\le g\rho^{g-1}s\sqrt{2/N}+\sum_{a=2}^{g}\binom{g}{a}\rho^{g-a}\sqrt{2^a a!}\,s^a. \tag*{(6)} ( E ∥ M g ( U ) − M g ( U ⋆ ) ∥ 2 ) 1/2 ≤ g ρ g − 1 s 2/ N + a = 2 ∑ g ( a g ) ρ g − a 2 a a ! s a . (6)
Since s a ≤ s 2 s^a \le s^2 s a ≤ s 2 for a ≥ 2 a \ge2 a ≥ 2 and 0 ≤ s ≤ 1 0 \le s \le1 0 ≤ s ≤ 1 , we may take
C g , ρ = ∑ a = 2 g ( g a ) ρ g − a 2 a a ! . (7) C_{g,\rho}=\sum_{a=2}^{g}\binom{g}{a}\rho^{g-a}\sqrt{2^a a!}. \tag*{(7)} C g , ρ = a = 2 ∑ g ( a g ) ρ g − a 2 a a ! . (7)
This constant does not depend on s s s or N N N , and C 1 , ρ = 0 C_{1,\rho}=0 C 1 , ρ = 0 because the sum is empty. When ρ = 0 \rho=0 ρ = 0 , only the term a = g a=g a = g remains for g ≥ 2 g\ge2 g ≥ 2 , with ρ 0 = 1 \rho^0=1 ρ 0 = 1 . □ \square □
Errors in the model points. The lemma treats the model points as exact, as Eq. (1) does. Suppose now that model point y σ ( i ) y_{\sigma(i)} y σ ( i ) is measured with an error v i v_i v i of covariance Σ 3 D , i \Sigma_{3D,i} Σ 3 D , i and that the image error w i w_i w i has covariance Σ 2 D , i \Sigma_{2D,i} Σ 2 D , i , with v i v_i v i and w i w_i w i independent. To first order at the true pose, the difference between observation i i i and the projection of the measured model point is w i − J p r o j , i R ⋆ v i w_i-J_{\mathrm{proj},i}R^\star v_i w i − J proj , i R ⋆ v i . Here J p r o j , i J_{\mathrm{proj},i} J proj , i is
the Jacobian of the pixel projection proj \operatorname{proj} proj at the true model point x σ ( i ) x_{\sigma(i)} x σ ( i ) in camera coordinates. Independence makes the cross-covariances vanish, so this difference has covariance
Σ l i n , i = Σ 2 D , i + J p r o j , i R ⋆ Σ 3 D , i ( R ⋆ ) T J p r o j , i T . (8) \Sigma_{\mathrm{lin},i}=\Sigma_{2D,i}+J_{\mathrm{proj},i}R^\star\Sigma_{3D,i}(R^\star)^\mathsf{T}J_{\mathrm{proj},i}^\mathsf{T}. \tag*{(8)} Σ lin , i = Σ 2 D , i + J proj , i R ⋆ Σ 3 D , i ( R ⋆ ) T J proj , i T . (8)
The 3D term depends on the viewing direction through J p r o j , i J_{\mathrm{proj},i} J proj , i and is in general anisotropic. At a fixed viewing direction and a fixed covariance in model units, it scales with the inverse square of the depth. The model assumes independent errors and exact intrinsics, so it omits the correlations that shared calibration errors introduce. Because the expression is linearized, the exact projected error need not be Gaussian.
Proof of the pinhole identifiability theorem
We use calibrated image coordinates, in which a camera point x x x with x 3 > 0 x_3>0 x 3 > 0 projects to proj c a l ( x ) = x 1 : 2 / x 3 \operatorname{proj}_{\mathrm{cal}}(x)=x_{1:2}/x_3 proj cal ( x ) = x 1 : 2 / x 3 , and we identify a model Y = { y j } j = 1 N \mathcal{Y}=\{y_j\}_{j=1}^N Y = { y j } j = 1 N with the vector ( y 1 , … , y N ) ∈ R 3 N (y_1,\ldots,y_N)\in\mathbb{R}^{3N} ( y 1 , … , y N ) ∈ R 3 N . For a pose ξ = ( R , t ) \xi=(R,t) ξ = ( R , t ) under which every model point has positive depth, the raw projected moments are
P Y ( ξ ) = 1 N ∑ j = 1 N ϕ 4 ( proj c a l ( R y j + t ) ) ∈ R 14 , \mathcal{P}_{\mathcal{Y}}(\xi)=\frac{1}{N}\sum_{j=1}^{N}\phi_4\bigl(\operatorname{proj}_{\mathrm{cal}}(Ry_j+t)\bigr)\in\mathbb{R}^{14}, P Y ( ξ ) = N 1 j = 1 ∑ N ϕ 4 ( proj cal ( R y j + t ) ) ∈ R 14 ,
the fourteen moments of orders one to four of the calibrated projections, without centering or normalization.
Theorem 1 (restated). Fix a pose ξ ⋆ \xi^\star ξ ⋆ and an integer N ≥ 20 N\ge20 N ≥ 20 . There is a set Z ⊂ R 3 N \mathcal{Z}\subset\mathbb{R}^{3N} Z ⊂ R 3 N of Lebesgue measure zero with the following property. Let Y ∉ Z \mathcal{Y}\notin\mathcal{Z} Y ∈ / Z be a model whose N N N points all have positive depth under ξ ⋆ \xi^\star ξ ⋆ , and let ξ \xi ξ be a pose under which all its points have positive depth. If P Y ( ξ ) = P Y ( ξ ⋆ ) \mathcal{P}_{\mathcal{Y}}(\xi)=\mathcal{P}_{\mathcal{Y}}(\xi^\star) P Y ( ξ ) = P Y ( ξ ⋆ ) , then ξ = ξ ⋆ \xi=\xi^\star ξ = ξ ⋆ .
An invertible affine map of the image preserves the equality of its moments through order four in both directions, because composition with the map or its inverse takes every polynomial of degree at most four to another such polynomial. The restated theorem therefore also covers the moments in pixels and the moments after the map h h h of § 3.1 , which the main text compares. On complete noiseless inputs, it shows that the true pose is generically the only zero of L 4 \mathcal{L}_4 L 4 among the poses that keep all model points in front of the camera, so the minimum that Step 2 searches for is unique. The proof shows that every kind of alternative pose imposes on each model point an equation that a generic point does not satisfy. It then counts dimensions to show that the models admitting such a pose form a null set. After the proof, Corollary A1 turns uniqueness into a stability statement for the perspective moment fit.
Uniqueness cannot hold for every model. A quarter-turn of a cube about the axis through the centers of two opposite faces maps its eight corners onto themselves, so a camera pose and its composition with this rigid motion give the same image. The theorem shows that such symmetric models are exceptional.
Coordinates. We fix the true pose ξ ⋆ = ( R ⋆ , t ⋆ ) \xi^\star=(R^\star,t^\star) ξ ⋆ = ( R ⋆ , t ⋆ ) before choosing the model and express the model points in its camera coordinates as x j = R ⋆ y j + t ⋆ x_j=R^\star y_j+t^\star x j = R ⋆ y j + t ⋆ . This invertible affine change of coordinates of R 3 N \mathbb{R}^{3N} R 3 N preserves sets of Lebesgue measure zero, so it suffices to prove the theorem for the configurations X = { x j } j = 1 N \mathcal{X}=\{x_j\}_{j=1}^{N} X = { x j } j = 1 N with all x j 3 > 0 x_{j3}>0 x j 3 > 0 . The true projection is then proj c a l ( x j ) \operatorname{proj}_{\mathrm{cal}}(x_j) proj cal ( x j ) , and a competing pose ξ = ( R , t ) \xi=(R,t) ξ = ( R , t ) gives
proj Q , γ ( x j ) = proj c a l ( Q x j + γ ) , \operatorname{proj}_{Q,\gamma}(x_j)=\operatorname{proj}_{\mathrm{cal}}(Qx_j+\gamma), proj Q , γ ( x j ) = proj cal ( Q x j + γ ) ,
where Q = R ( R ⋆ ) ⊤ ∈ S O ( 3 ) Q=R(R^\star)^\top\in\mathrm{SO}(3) Q = R ( R ⋆ ) ⊤ ∈ SO ( 3 ) and γ = t − Q t ⋆ \gamma=t-Qt^\star γ = t − Q t ⋆ . The map ξ ↦ ( Q , γ ) \xi\mapsto(Q,\gamma) ξ ↦ ( Q , γ ) is a bijection from the poses onto S O ( 3 ) × R 3 \mathrm{SO}(3)\times\mathbb{R}^3 SO ( 3 ) × R 3 that sends ξ ⋆ \xi^\star ξ ⋆ to ( I 3 , 0 ) (I_3,0) ( I 3 , 0 ) . In the true camera coordinates, the competing camera center is − Q ⊤ γ -Q^\top\gamma − Q ⊤ γ , so the two cameras have the same center exactly when γ = 0 \gamma=0 γ = 0 .
Moment differences as polynomials. Let P 4 0 \mathbb{P}^{0}_{4} P 4 0 denote the fourteen-dimensional space of bivariate polynomials of degree at most four with zero constant coefficient. Each P ∈ P 4 0 P\in\mathbb{P}^{0}_{4} P ∈ P 4 0 is the inner product κ ⋅ ϕ 4 \kappa\cdot\phi_4 κ ⋅ ϕ 4 of its coefficient vector κ ∈ R 14 \kappa\in\mathbb{R}^{14} κ ∈ R 14 with ϕ 4 \phi_4 ϕ 4 . The linear combination of the fourteen moment differences with weights κ \kappa κ is therefore the average, over the points, of P ( proj Q , γ ( x j ) ) − P ( proj c a l ( x j ) ) P(\operatorname{proj}_{Q,\gamma}(x_j))-P(\operatorname{proj}_{\mathrm{cal}}(x_j)) P ( proj Q , γ ( x j )) − P ( proj cal ( x j )) . The proof distinguishes alternative poses by whether the camera center moves and whether the optical axis tilts. For each kind, the next lemma bounds the dimension of the space of polynomials whose values cannot detect the change of pose.
Lemma A2 (Polynomial features under a pinhole camera). Let Q ∈ S O ( 3 ) Q\in\mathrm{SO}(3) Q ∈ SO ( 3 ) and γ ∈ R 3 \gamma\in\mathbb{R}^3 γ ∈ R 3 , let u = ( u 1 , u 2 ) ∈ R 2 u=(u_1,u_2)\in\mathbb{R}^2 u = ( u 1 , u 2 ) ∈ R 2 be calibrated image coordinates, and let x 3 > 0 x_3>0 x 3 > 0 be the depth along the true camera ray ( x 3 u , x 3 ) (x_3u,x_3) ( x 3 u , x 3 ) . (a) If γ ≠ 0 \gamma\ne0 γ = 0 , the subspace
K Q , γ = { P ∈ P 4 0 : P ( proj Q , γ ( x 3 u 1 , x 3 u 2 , x 3 ) ) is independent of x 3 } (9) K_{Q,\gamma}=\{P\in\mathbb{P}^{0}_{4}:P(\operatorname{proj}_{Q,\gamma}(x_3u_1,x_3u_2,x_3))\text{ is independent of }x_3\} \tag*{(9)} K Q , γ = { P ∈ P 4 0 : P ( proj Q , γ ( x 3 u 1 , x 3 u 2 , x 3 )) is independent of x 3 } (9) has dimension at most four and equals { 0 } \{0\} { 0 } if γ 3 ≠ 0 \gamma_3\ne0 γ 3 = 0 . Here the composition is a rational function of ( u , x 3 ) (u,x_3) ( u , x 3 ) , and independence means that its partial derivative with respect to x 3 x_3 x 3 vanishes identically. (b) If γ = 0 \gamma=0 γ = 0 and ( Q 31 , Q 32 ) ≠ ( 0 , 0 ) (Q_{31},Q_{32})\ne(0,0) ( Q 31 , Q 32 ) = ( 0 , 0 ) , then P ( proj Q , γ ( u , 1 ) ) − P ( u ) P(\operatorname{proj}_{Q,\gamma}(u,1))-P(u) P ( proj Q , γ ( u , 1 )) − P ( u ) is a nonconstant function of u u u for every nonzero P ∈ P 4 0 P\in\mathbb{P}^{0}_{4} P ∈ P 4 0 . (c) If γ = 0 \gamma=0 γ = 0 and ( Q 31 , Q 32 ) = ( 0 , 0 ) (Q_{31},Q_{32})=(0,0) ( Q 31 , Q 32 ) = ( 0 , 0 ) , then proj Q , γ ( u , 1 ) = G u \operatorname{proj}_{Q,\gamma}(u,1)=Gu proj Q , γ ( u , 1 ) = G u for some G ∈ O ( 2 ) G\in O(2) G ∈ O ( 2 ) . If Q ≠ I 3 Q\ne I_3 Q = I 3 , then G ≠ I 2 G\ne I_2 G = I 2 , and the invariant subspace K G = { P ∈ P 4 0 : P ∘ G = P } K_G=\{P\in\mathbb{P}^{0}_{4}:P\circ G=P\} K G = { P ∈ P 4 0 : P ∘ G = P } has dimension at most eight. For P ∉ K G P\notin K_G P ∈ / K G , the polynomial u ↦ P ( G u ) − P ( u ) u\mapsto P(Gu)-P(u) u ↦ P ( G u ) − P ( u ) is nonconstant.
Proof. Part (a). We write a = Q ( u , 1 ) a=Q(u,1) a = Q ( u , 1 ) , so that Q ( x 3 u , x 3 ) + γ = x 3 a + γ Q(x_3u,x_3)+\gamma=x_3a+\gamma Q ( x 3 u , x 3 ) + γ = x 3 a + γ , and z ( u , x 3 ) = proj Q , γ ( x 3 u , x 3 ) z(u,x_3)=\operatorname{proj}_{Q,\gamma}(x_3u,x_3) z ( u , x 3 ) = proj Q , γ ( x 3 u , x 3 ) . With e 3 = ( 0 , 0 , 1 ) e_3=(0,0,1) e 3 = ( 0 , 0 , 1 ) , we have x 3 a + γ = ( x 3 Q + γ e 3 ⊤ ) ( u , 1 ) x_3a+\gamma=(x_3Q+\gamma e_3^\top)(u,1) x 3 a + γ = ( x 3 Q + γ e 3 ⊤ ) ( u , 1 ) . Because det ( x 3 Q + γ e 3 ⊤ ) = x 3 2 ( x 3 + ( Q ⊤ γ ) 3 ) \det(x_3Q+\gamma e_3^\top)=x_3^2(x_3+(Q^\top\gamma)_3) det ( x 3 Q + γ e 3 ⊤ ) = x 3 2 ( x 3 + ( Q ⊤ γ ) 3 ) , the map u ↦ z ( u , x 3 ) u\mapsto z(u,x_3) u ↦ z ( u , x 3 ) is an invertible projective transformation of the plane for all but at most one value of x 3 > 0 x_3>0 x 3 > 0 . The points z ( u , x 3 ) z(u,x_3) z ( u , x 3 ) therefore fill an open subset of R 2 \mathbb{R}^2 R 2 . We first suppose that γ 3 ≠ 0 \gamma_3\ne0 γ 3 = 0 and put u c = γ 1 : 2 / γ 3 u_c=\gamma_{1:2}/\gamma_3 u c = γ 1 : 2 / γ 3 , the image of the true camera center in the competing camera. Differentiation gives
∂ x 3 z = γ 3 x 3 ( x 3 γ 3 + γ 3 ) ( z − u c ) , \partial_{x_3}z=\frac{\gamma_3}{x_3(x_3\gamma_3+\gamma_3)}(z-u_c), ∂ x 3 z = x 3 ( x 3 γ 3 + γ 3 ) γ 3 ( z − u c ) ,
so as x 3 x_3 x 3 varies, the point z z z moves along the line through u c u_c u c . If P ∈ K Q , γ P\in K_{Q,\gamma} P ∈ K Q , γ , the chain rule gives ( z − u c ) ⋅ ∇ P ( z ) = 0 (z-u_c)\cdot\nabla P(z)=0 ( z − u c ) ⋅ ∇ P ( z ) = 0 at every point z ( u , x 3 ) z(u,x_3) z ( u , x 3 ) . This polynomial in z z z vanishes on an open set and hence identically. We expand P P P in homogeneous terms about u c u_c u c . By Euler’s identity, the operator ( z − u c ) ⋅ ∇ (z-u_c)\cdot\nabla ( z − u c ) ⋅ ∇ multiplies the term of degree k k k by k k k , so every term of positive degree is zero. Thus P P P is constant, and its zero constant coefficient gives P = 0 P=0 P = 0 . If γ 3 = 0 \gamma_3=0 γ 3 = 0 , then γ 1 : 2 ≠ 0 \gamma_{1:2}\ne0 γ 1 : 2 = 0 and
proj Q , γ ( x 3 u , x 3 ) = ( Q ( u , 1 ) ) 1 : 2 ( Q ( u , 1 ) ) 3 + γ 1 : 2 x 3 ( Q ( u , 1 ) ) 3 , \operatorname{proj}_{Q,\gamma}(x_3u,x_3)=\frac{(Q(u,1))_{1:2}}{(Q(u,1))_3}+\frac{\gamma_{1:2}}{x_3(Q(u,1))_3}, proj Q , γ ( x 3 u , x 3 ) = ( Q ( u , 1 ) ) 3 ( Q ( u , 1 ) ) 1 : 2 + x 3 ( Q ( u , 1 ) ) 3 γ 1 : 2 ,
so the point moves along the fixed direction γ 1 : 2 \gamma_{1:2} γ 1 : 2 as x 3 x_3 x 3 varies. For P ∈ K Q , γ P\in K_{Q,\gamma} P ∈ K Q , γ , the chain rule gives γ 1 : 2 ⋅ ∇ P ( z ) = 0 \gamma_{1:2}\cdot\nabla P(z)=0 γ 1 : 2 ⋅ ∇ P ( z ) = 0 on the same open set, hence as a polynomial identity. Such a P P P is a polynomial of degree at most four in the linear coordinate perpendicular to γ 1 : 2 \gamma_{1:2} γ 1 : 2 , with zero constant coefficient, and these polynomials form a space of dimension four.
Part (b). We put z Q ( u ) = ( Q ( u , 1 ) ) 3 = Q 31 u 1 + Q 32 u 2 + Q 33 z_Q(u)=(Q(u,1))_3=Q_{31}u_1+Q_{32}u_2+Q_{33} z Q ( u ) = ( Q ( u , 1 ) ) 3 = Q 31 u 1 + Q 32 u 2 + Q 33 , an affine polynomial that is nonconstant because ( Q 31 , Q 32 ) ≠ ( 0 , 0 ) (Q_{31},Q_{32})\ne(0,0) ( Q 31 , Q 32 ) = ( 0 , 0 ) . The competing camera sends the line z Q = 0 z_Q=0 z Q = 0 to infinity. A nonzero P ∈ P 4 0 P\in\mathbb{P}^{0}_{4} P ∈ P 4 0 has a degree d P ≥ 1 d_P\ge1 d P ≥ 1 and a nonzero leading homogeneous part P d P P_{d_P} P d P , and its composition with proj Q , γ \operatorname{proj}_{Q,\gamma} proj Q , γ is a quotient with denominator z Q d P z_Q^{d_P} z Q d P . On the line z Q = 0 z_Q=0 z Q = 0 , the numerator equals P d P ( ( Q ( u , 1 ) ) 1 : 2 ) P_{d_P}((Q(u,1))_{1:2}) P d P (( Q ( u , 1 ) ) 1 : 2 ) . Along this line, the two coordinates ( Q ( u , 1 ) ) 1 : 2 (Q(u,1))_{1:2} ( Q ( u , 1 ) ) 1 : 2 trace an affine line not through the origin, because otherwise the invertible matrix Q Q Q would map a vector ( u , 1 ) (u,1) ( u , 1 ) to zero. A nonzero homogeneous polynomial cannot vanish on this entire line, because scalar multiples of the line contain an open subset of R 2 \mathbb{R}^2 R 2 . The numerator is therefore not divisible by the irreducible polynomial z Q z_Q z Q , so the composition has a pole on the line z Q = 0 z_Q=0 z Q = 0 . Subtracting the polynomial P ( u ) P(u) P ( u ) cannot remove this pole, so the difference is not a polynomial and in particular is nonconstant.
Part (c). The third row of Q Q Q is ( 0 , 0 , Q 33 ) (0,0,Q_{33}) ( 0 , 0 , Q 33 ) , so orthogonality gives Q 33 = ± 1 Q_{33}=\pm1 Q 33 = ± 1 and Q 13 = Q 23 = 0 Q_{13}=Q_{23}=0 Q 13 = Q 23 = 0 . Hence G = Q 1 : 2 , 1 : 2 / Q 33 ∈ O ( 2 ) G=Q_{1:2,1:2}/Q_{33}\in O(2) G = Q 1 : 2 , 1 : 2 / Q 33 ∈ O ( 2 ) . If G = I 2 G=I_2 G = I 2 , then Q = Q 33 I 3 Q=Q_{33}I_3 Q = Q 33 I 3 , and det Q = 1 \det Q=1 det Q = 1 gives Q = I 3 Q=I_3 Q = I 3 . If G G G is a reflection, we choose orthonormal coordinates along its axis and perpendicular to it. The reflection then fixes the eight of the fourteen monomials whose exponent in the perpendicular coordinate is even and negates the other six. If G G G is a rotation by an angle θ ∉ 2 π Z \theta\notin2\pi\mathbb{Z} θ ∈ / 2 π Z , it multiplies the complex monomial ( u 1 + i u 2 ) a ( u 1 − i u 2 ) b (u_1+iu_2)^a(u_1-iu_2)^b ( u 1 + i u 2 ) a ( u 1 − i u 2 ) b by e i ( a − b ) θ e^{i(a-b)\theta} e i ( a − b ) θ . A nonidentity rotation therefore changes both linear polynomials and the two cubic polynomials with a − b = ± 1 a-b=\pm1 a − b = ± 1 . A half-turn also changes the other two cubic polynomials, and every other rotation changes the two quadratic polynomials with a − b = ± 2 a-b=\pm2 a − b = ± 2 . Thus at most eight independent polynomials are unchanged by a nonidentity G G G . If P ∉ K G P\notin K_G P ∈ / K G , then P ( G u ) − P ( u ) P(Gu)-P(u) P ( G u ) − P ( u ) is nonzero, and its zero constant coefficient makes it nonconstant. □
Proof of Theorem 1. For X = { x j } j = 1 N \mathcal{X}=\{x_j\}_{j=1}^{N} X = { x j } j = 1 N , the difference between the competing and true moment vectors is
Φ ( X , Q , γ ) = 1 N ∑ j = 1 N [ ϕ 4 ( proj Q , γ ( x j ) ) − ϕ 4 ( proj c a l ( x j ) ) ] . (10) \Phi(\mathcal{X},Q,\gamma)=\frac{1}{N}\sum_{j=1}^{N}\left[\phi_4(\operatorname{proj}_{Q,\gamma}(x_j))-\phi_4(\operatorname{proj}_{\mathrm{cal}}(x_j))\right]. \tag*{(10)} Φ ( X , Q , γ ) = N 1 j = 1 ∑ N [ ϕ 4 ( proj Q , γ ( x j )) − ϕ 4 ( proj cal ( x j )) ] . (10)
We consider only configurations and competing poses that make all true and competing depths positive, so Φ \Phi Φ is smooth on its domain. Its zeros are those of P Y ( ξ ) − P Y ( ξ ⋆ ) P_Y(\xi)-P_Y(\xi^\star) P Y ( ξ ) − P Y ( ξ ⋆ ) , expressed in calibrated image and transformed model coordinates. A zero with ( Q , γ ) ≠ ( I 3 , 0 ) (Q,\gamma)\ne(I_3,0) ( Q , γ ) = ( I 3 , 0 ) is an alternative pose. We write ∂ X Φ \partial_{\mathcal{X}}\Phi ∂ X Φ for the 14 × 3 N 14\times3N 14 × 3 N derivative of Φ \Phi Φ with respect to the point coordinates. A coefficient vector κ ∈ R 14 \kappa\in\mathbb{R}^{14} κ ∈ R 14 with κ ⊤ ∂ X Φ = 0 \kappa^{\top}\partial_{\mathcal{X}}\Phi=0 κ ⊤ ∂ X Φ = 0 defines the polynomial P = κ ⋅ ϕ 4 P=\kappa\cdot\phi_4 P = κ ⋅ ϕ 4 . The average of P ∘ proj Q , γ − P ∘ proj c a l P\circ\operatorname{proj}_{Q,\gamma}-P\circ\operatorname{proj}_{\mathrm{cal}} P ∘ proj Q , γ − P ∘ proj cal over the points then has zero derivative with respect to every point. We show that the configurations X \mathcal{X} X that admit an alternative pose form a set of dimension less than 3 N 3N 3 N .
Dimension facts. We use standard facts about semialgebraic sets [2 ] . Sets described by finitely many polynomial equalities and inequalities, and projections of such sets, are semialgebraic. Projection does not increase dimension. A semialgebraic set has dimension at most d d d if each of its points has a neighborhood in which the set lies in a smooth submanifold of dimension d d d . A family has dimension at most its number of parameters plus the largest dimension of its solution set at fixed parameters. A nonzero polynomial in k k k variables has a zero set of dimension at most k − 1 k-1 k − 1 . A semialgebraic subset of R 3 N \mathbb{R}^{3N} R 3 N of dimension less than 3 N 3N 3 N has Lebesgue measure zero. In each of the three cases below, a high rank of ∂ X Φ \partial_{\mathcal{X}}\Phi ∂ X Φ bounds the dimension directly through the implicit-function theorem. If the rank is low, Lemma A2 gives a coefficient vector κ \kappa κ whose polynomial detects the change of pose, and that polynomial yields one equation for every point.
Case 1. Distinct camera centers. We suppose γ ≠ 0 \gamma\ne0 γ = 0 , so the competing pose has six parameters. At a solution with rank ∂ X Φ ≥ 10 \partial_{\mathcal{X}}\Phi\ge10 ∂ X Φ ≥ 10 , ten of the equations have independent derivatives, so near this solution the solution set lies in a submanifold of dimension 3 N + 6 − 10 < 3 N 3N+6-10<3N 3 N + 6 − 10 < 3 N . If the rank is less than ten, the coefficient vectors κ \kappa κ with κ ⊤ ∂ X Φ = 0 \kappa^{\top}\partial_{\mathcal{X}}\Phi=0 κ ⊤ ∂ X Φ = 0 form a space of dimension greater than four. By Lemma A2(a), we can therefore choose such a κ \kappa κ whose polynomial P P P lies outside K Q , γ K_{Q,\gamma} K Q , γ , and we normalize κ \kappa κ to the unit sphere S 13 S^{13} S 13 . We write x j = ( x j 3 u j ⋆ , x j 3 ) x_j=(x_{j3}u_j^\star,x_{j3}) x j = ( x j 3 u j ⋆ , x j 3 ) with true projection u j ⋆ u_j^\star u j ⋆ and depth x j 3 > 0 x_{j3}>0 x j 3 > 0 . Moving a point along its true camera ray leaves proj c a l ( x j ) \operatorname{proj}_{\mathrm{cal}}(x_j) proj cal ( x j ) unchanged, so the zero derivative of the average along this direction gives
d d x 3 P ( proj Q , γ ( x 3 u j ⋆ , x 3 ) ) ∣ x 3 = x j 3 = 0 , j = 1 , … , N . \left.\frac{d}{dx_3}P\left(\operatorname{proj}_{Q,\gamma}(x_3u_j^\star,x_3)\right)\right|_{x_3=x_{j3}}=0,\qquad j=1,\ldots,N. d x 3 d P ( proj Q , γ ( x 3 u j ⋆ , x 3 ) ) x 3 = x j 3 = 0 , j = 1 , … , N .
For fixed ( Q , γ , κ ) (Q,\gamma,\kappa) ( Q , γ , κ ) , the left-hand side is a rational function of x j x_j x j whose denominator is a product of powers of the true and competing depths. Because P ∉ K Q , γ P\notin K_{Q,\gamma} P ∈ / K Q , γ , it is not identically zero, so its numerator is a nonzero polynomial in the three coordinates of x j x_j x j . Each point therefore lies in a set of dimension at most two. The equations involve separate points, so their joint solution set has dimension at most 2 N 2N 2 N . The six pose parameters and thirteen parameters of κ \kappa κ give total dimension at most 2 N + 19 2N+19 2 N + 19 , which is less than 3 N 3N 3 N exactly when N ≥ 20 N\ge20 N ≥ 20 . This case sets the bound on N N N in Theorem 1.
Case 2. Same center, tilted optical axis. We suppose γ = 0 \gamma=0 γ = 0 and ( Q 31 , Q 32 ) ≠ ( 0 , 0 ) (Q_{31},Q_{32})\ne(0,0) ( Q 31 , Q 32 ) = ( 0 , 0 ) . These rotations form an open subset of S O ( 3 ) \mathrm{SO}(3) SO ( 3 ) , of dimension three. When the derivative with respect to the point coordinates has full rank fourteen, the local dimension of the solution set is at most 3 N + 3 − 14 < 3 N 3N+3-14<3N 3 N + 3 − 14 < 3 N . Otherwise, we choose a unit vector κ \kappa κ with κ ⊤ ∂ X Φ = 0 \kappa^{\top}\partial_{\mathcal{X}}\Phi=0 κ ⊤ ∂ X Φ = 0 . The ray coordinates ( u j ⋆ , x j 3 ) (u_j^\star,x_{j3}) ( u j ⋆ , x j 3 ) are smooth coordinates for positive depth, and proj Q , γ ( x j 3 u j ⋆ , x j 3 ) = proj Q , γ ( u j ⋆ , 1 ) \operatorname{proj}_{Q,\gamma}(x_{j3}u_j^\star,x_{j3})=\operatorname{proj}_{Q,\gamma}(u_j^\star,1) proj Q , γ ( x j 3 u j ⋆ , x j 3 ) = proj Q , γ ( u j ⋆ , 1 ) because the two cameras share their center. The derivatives with respect to the two image coordinates therefore give
∇ u j ⋆ [ P ( proj Q , γ ( u j ⋆ , 1 ) ) − P ( u j ⋆ ) ] = 0 , j = 1 , … , N . \nabla_{u_j^\star}\left[P\left(\operatorname{proj}_{Q,\gamma}(u_j^\star,1)\right)-P(u_j^\star)\right]=0,\qquad j=1,\ldots,N. ∇ u j ⋆ [ P ( proj Q , γ ( u j ⋆ , 1 ) ) − P ( u j ⋆ ) ] = 0 , j = 1 , … , N .
By Lemma A2(b), the function in brackets is a nonconstant rational function of u j ⋆ u_j^\star u j ⋆ , so at least one of its partial derivatives is a nonzero rational function. The numerator of that partial derivative is a nonzero polynomial in two variables, so the critical set has dimension at most one in u j ⋆ u_j^\star u j ⋆ , and the free depth adds one dimension, which gives at most 2 N 2N 2 N dimensions for the point set. Adding three rotation parameters and thirteen parameters of κ \kappa κ bounds the dimension by 2 N + 16 < 3 N 2N+16<3N 2 N + 16 < 3 N .
Case 3. Same center, aligned optical axis. We suppose γ = 0 \gamma=0 γ = 0 and ( Q 31 , Q 32 ) = ( 0 , 0 ) (Q_{31},Q_{32})=(0,0) ( Q 31 , Q 32 ) = ( 0 , 0 ) , with Q ≠ I 3 Q\ne I_3 Q = I 3 . By the proof of Lemma A2(c), Q 33 = ± 1 Q_{33}=\pm1 Q 33 = ± 1 . A rotation with Q 33 = − 1 Q_{33}=-1 Q 33 = − 1 gives every point the competing depth − x j 3 < 0 -x_{j3}<0 − x j 3 < 0 and is excluded. The rotations with Q 33 = 1 Q_{33}=1 Q 33 = 1 form a family of dimension one. A rank of at least six bounds the local dimension of the solution set by 3 N + 1 − 6 < 3 N 3N+1-6<3N 3 N + 1 − 6 < 3 N . Otherwise, the coefficient vectors κ \kappa κ with κ ⊤ ∂ X Φ = 0 \kappa^{\top}\partial_{\mathcal{X}}\Phi=0 κ ⊤ ∂ X Φ = 0 form a space of dimension greater than eight, so Lemma A2(c) gives a unit κ \kappa κ whose polynomial satisfies P ∉ K G P\notin K_G P ∈ / K G . The argument of Case 2 requires each u j ∗ u_j^* u j ∗ to be a critical point of the nonconstant polynomial u ↦ P ( G u ) − P ( u ) u \mapsto P(Gu)-P(u) u ↦ P ( G u ) − P ( u ) . Its critical set has dimension at most one, which again gives at most 2 N 2N 2 N dimensions for the point set. The one rotation parameter and thirteen parameters of κ \kappa κ give total dimension at most 2 N + 14 < 3 N 2N+14<3N 2 N + 14 < 3 N .
Conclusion. All sets in these arguments are semialgebraic, because the rotation constraints, positive depths, coefficient normalization and rank conditions are polynomial equalities or inequalities after denominators are cleared. The conditions P ∉ K Q , γ P\notin K_{Q,\gamma} P ∈ / K Q , γ and P ∉ K G P\notin K_G P ∈ / K G also have this form, because each requires at least one coefficient of the corresponding rational or polynomial identity to be nonzero. The local dimension bounds hold throughout the sets of high rank. The parameter counts of the three cases cover the remaining ranks. The finitely many unions and projections of the solution sets therefore have dimension less than 3 N 3N 3 N . For the fixed true pose, we obtain a null set Z \mathcal{Z} Z of configurations outside which no other pose with positive depths has the same moments through order four. This exceptional set is semialgebraic of dimension less than 3 N 3N 3 N , so it also lies in the zero set of a nonzero polynomial in the model coordinates. Every joint density on R 3 N \mathbb{R}^{3N} R 3 N , including one for dependent points, assigns probability zero to this set. □
Perspective stability. Theorem 1 concerns exact moments, whereas Step 2 fits moments that contain error. Uniqueness together with compactness bounds the pose error of the fit by the moment errors, through a modulus that the corollary defines but does not compute. This bound is the perspective counterpart of Eq. (23) , and it concerns the moment fit before EM refinement. We hold the image map h h h fixed throughout the comparison, even if it was computed from the observations. We write μ Y ( ξ ; h ) \mu_Y(\xi;h) μ Y ( ξ ; h ) and μ U ( h ) \mu_U(h) μ U ( h ) for the averages of ϕ 4 ∘ h \phi_4\circ h ϕ 4 ∘ h over the projections at pose ξ \xi ξ and over the observations. Because h h h does not change which poses give equal moments, Theorem 1 provides the uniqueness that the corollary assumes, for a generic model of N ≥ 20 N\geq20 N ≥ 20 points and any compact set of poses with positive depths that contains ξ ∗ \xi^* ξ ∗ .
Corollary A1 (Stability of the perspective pose). We fix a 3D model and an invertible affine image map h h h , and write M ( ξ ) = μ Y ( ξ ; h ) M(\xi)=\mu_Y(\xi;h) M ( ξ ) = μ Y ( ξ ; h ) . Let E ξ E_\xi E ξ be a compact set of poses that contains ξ ∗ \xi^* ξ ∗ and places every model point at positive depth, and suppose that ξ ∗ \xi^* ξ ∗ is the unique solution of M ( ξ ) = M ( ξ ∗ ) M(\xi)=M(\xi^*) M ( ξ ) = M ( ξ ∗ ) in E ξ E_\xi E ξ . We fix a model length ℓ Y > 0 \ell_Y>0 ℓ Y > 0 and use the pose distance
d Y ( ξ , ξ ′ ) 2 = ∥ R − R ′ ∥ 2 + ∥ t − t ′ ∥ 2 / ℓ Y 2 . d_Y(\xi,\xi')^2=\lVert R-R'\rVert^2+\lVert t-t'\rVert^2/\ell_Y^2. d Y ( ξ , ξ ′ ) 2 = ∥ R − R ′ ∥ 2 + ∥ t − t ′ ∥ 2 / ℓ Y 2 .
For a ξ > 0 a_\xi>0 a ξ > 0 with { ξ ∈ E ξ : d Y ( ξ , ξ ∗ ) ≥ a ξ } \{\xi\in E_\xi:d_Y(\xi,\xi^*)\geq a_\xi\} { ξ ∈ E ξ : d Y ( ξ , ξ ∗ ) ≥ a ξ } nonempty, we define Γ ξ ( a ξ ) \Gamma_\xi(a_\xi) Γ ξ ( a ξ ) as the minimum of ∥ M ( ξ ) − M ( ξ ∗ ) ∥ \lVert M(\xi)-M(\xi^*)\rVert ∥ M ( ξ ) − M ( ξ ∗ )∥ over that set. For an observed vector μ U ( h ) \mu_U(h) μ U ( h ) and a fitted pose ξ ^ ∈ E ξ \hat{\xi}\in E_\xi ξ ^ ∈ E ξ , the observation error is ϵ o b s = ∥ μ U ( h ) − M ( ξ ∗ ) ∥ \epsilon_{\mathrm{obs}}=\lVert\mu_U(h)-M(\xi^*)\rVert ϵ obs = ∥ μ U ( h ) − M ( ξ ∗ )∥ and the fit residual is ϵ r e s = ∥ M ( ξ ^ ) − μ U ( h ) ∥ \epsilon_{\mathrm{res}}=\lVert M(\hat{\xi})-\mu_U(h)\rVert ϵ res = ∥ M ( ξ ^ ) − μ U ( h )∥ . Then Γ ξ ( a ξ ) > 0 \Gamma_\xi(a_\xi)>0 Γ ξ ( a ξ ) > 0 , and
ϵ o b s + ϵ r e s < Γ ξ ( a ξ ) implies d Y ( ξ ^ , ξ ∗ ) < a ξ . (11) \epsilon_{\mathrm{obs}}+\epsilon_{\mathrm{res}}<\Gamma_\xi(a_\xi)\quad\text{implies}\quad d_Y(\hat{\xi},\xi^*)<a_\xi. \tag*{(11)} ϵ obs + ϵ res < Γ ξ ( a ξ ) implies d Y ( ξ ^ , ξ ∗ ) < a ξ . (11)
In particular, if the residual of the fitted pose exceeds the residual ϵ o b s \epsilon_{\mathrm{obs}} ϵ obs of the true pose by at most ϵ f i t ≥ 0 \epsilon_{\mathrm{fit}}\geq0 ϵ fit ≥ 0 , so that ϵ r e s ≤ ϵ o b s + ϵ f i t \epsilon_{\mathrm{res}}\leq\epsilon_{\mathrm{obs}}+\epsilon_{\mathrm{fit}} ϵ res ≤ ϵ obs + ϵ fit , then 2 ϵ o b s + ϵ f i t < Γ ξ ( a ξ ) 2\epsilon_{\mathrm{obs}}+\epsilon_{\mathrm{fit}}<\Gamma_\xi(a_\xi) 2 ϵ obs + ϵ fit < Γ ξ ( a ξ ) suffices.
Proof. Positive depths make M M M continuous on E ξ E_\xi E ξ . The poses of E ξ E_\xi E ξ at distance at least a ξ a_\xi a ξ from ξ ∗ \xi^* ξ ∗ form a compact set, and uniqueness makes their moment difference strictly positive, so the minimum Γ ξ ( a ξ ) \Gamma_\xi(a_\xi) Γ ξ ( a ξ ) is positive. The triangle inequality gives ∥ M ( ξ ^ ) − M ( ξ ∗ ) ∥ ≤ ϵ r e s + ϵ o b s \lVert M(\hat{\xi})-M(\xi^*)\rVert\leq\epsilon_{\mathrm{res}}+\epsilon_{\mathrm{obs}} ∥ M ( ξ ^ ) − M ( ξ ∗ )∥ ≤ ϵ res + ϵ obs . If d Y ( ξ ^ , ξ ∗ ) ≥ a ξ d_Y(\hat{\xi},\xi^*)\geq a_\xi d Y ( ξ ^ , ξ ∗ ) ≥ a ξ , the definition of Γ ξ ( a ξ ) \Gamma_\xi(a_\xi) Γ ξ ( a ξ ) gives ∥ M ( ξ ^ ) − M ( ξ ∗ ) ∥ ≥ Γ ξ ( a ξ ) \lVert M(\hat{\xi})-M(\xi^*)\rVert\geq\Gamma_\xi(a_\xi) ∥ M ( ξ ^ ) − M ( ξ ∗ )∥ ≥ Γ ξ ( a ξ ) , which contradicts ϵ r e s + ϵ o b s < Γ ξ ( a ξ ) \epsilon_{\mathrm{res}}+\epsilon_{\mathrm{obs}}<\Gamma_\xi(a_\xi) ϵ res + ϵ obs < Γ ξ ( a ξ ) . Substituting ϵ r e s ≤ ϵ o b s + ϵ f i t \epsilon_{\mathrm{res}}\leq\epsilon_{\mathrm{obs}}+\epsilon_{\mathrm{fit}} ϵ res ≤ ϵ obs + ϵ fit gives the final assertion. □
On complete inputs, image noise enters through ϵ o b s \epsilon_{\mathrm{obs}} ϵ obs , which Lemma A1 bounds before the map h h h is applied. If the observations are images of only some of the model points, the difference between their average and the average over all projections also enters ϵ o b s \epsilon_{\mathrm{obs}} ϵ obs . With outliers, moment matching replaces the monomials by bounded features, which limit the influence of the outliers on the averages but need not remove their bias (Lemmas A6 and A17). When Γ ξ ( a ξ ) \Gamma_\xi(a_\xi) Γ ξ ( a ξ ) is small, a small moment discrepancy can coexist with a pose error of at least a ξ a_\xi a ξ . A large fit residual can indicate a failed search.
Orientation from normalized moments
On complete inputs, Step 1 also places initial cameras at orientations fitted under an affine camera of rank two, such as scaled orthography. This section shows that these orientations have errors at most proportional to the moment errors. We center both point sets and scale each to identity covariance, giving Y ~ \widetilde{Y} Y and U ~ \widetilde{U} U . Their means and covariances then agree, and only the orientation remains unknown. Complete noiseless affine observations satisfy U ~ = O Y ~ \widetilde{U}=O\widetilde{Y} U = O Y for an orientation O ∈ V = { O ∈ R 2 × 3 : O O T = I 2 } O\in\mathcal{V}=\{O\in\mathbb{R}^{2\times3}:OO^{\mathsf{T}}=I_2\} O ∈ V = { O ∈ R 2 × 3 : O O T = I 2 } , the first two rows of a rotation [[36 ] ]. We therefore determine the orientation from the cubic and quartic moments. Let the moment pair T ∈ R 25 T\in\mathbb{R}^{25} T ∈ R 25 collect the 10 cubic and 15 quartic moments of Y ~ \widetilde{Y} Y , and let F T ( O ) \mathcal{F}_T(O) F T ( O ) and T U T_U T U collect those of the projected model O Y ~ O\widetilde{Y} O Y and of U ~ \widetilde{U} U . For an estimate O ^ \hat{O} O ^ of the true orientation O ∗ O^* O ∗ , the residual ϵ ^ = ∥ F T ( O ^ ) − T U ∥ \hat{\epsilon}=\lVert\mathcal{F}_T(\hat{O})-T_U\rVert ϵ ^ = ∥ F T ( O ^ ) − T U ∥ measures how well the estimate fits the observed moments, and the observation error ϵ = ∥ T U − F T ( O ∗ ) ∥ \epsilon=\lVert T_U-\mathcal{F}_T(O^*)\rVert ϵ = ∥ T U − F T ( O ∗ )∥ measures their noise.
Theorem A1 (Orientation from normalized moments). Let both point sets have positive definite covariance. For every moment pair T T T outside a set of zero volume in R 25 \mathbb{R}^{25} R 25 , distinct orientations give distinct moments F T ( O ) \mathcal{F}_T(O) F T ( O ) , and there is a constant Γ T > 0 \Gamma_T>0 Γ T > 0 such that for all O ∗ , O ^ ∈ V O^*,\hat{O}\in\mathcal{V} O ∗ , O ^ ∈ V
∥ O ^ − O ∗ ∥ ≤ ϵ ^ + ϵ Γ T . \lVert\hat{O}-O^*\rVert\leq\frac{\hat{\epsilon}+\epsilon}{\Gamma_T}. ∥ O ^ − O ∗ ∥ ≤ Γ T ϵ ^ + ϵ .
A model of N ≥ 34 N \ge34 N ≥ 34 points drawn from a joint density has such a moment pair with probability one.
Here Γ T \Gamma_T Γ T is the greatest lower bound of the change of the projected moments per unit change of orientation. It is zero for symmetric models such as the cube (Lemma A5) and small for nearly symmetric ones (Corollary A2). The theorem bounds the orientation error linearly by the moment errors, with the factor 1 / Γ T 1/\Gamma_T 1/ Γ T , before any correspondence is formed. Under perspective projection, the normalized observations are in general not the image of the normalized model under any orientation, because the depths of the model points vary. The observation error ϵ \epsilon ϵ then contains this departure from an affine camera together with the image noise. Step 1 therefore uses the affine orientations as starting values, and the perspective fit of Step 2 refines them with the individual depths. The uniqueness of the perspective moment fit bounds the pose error by the moment errors, through a modulus without an explicit form (Corollary A1). In the depth study of Table A19 , the affine initial poses have median rotation errors of 9 ∘ 9^\circ 9 ∘ to 161 ∘ 161^\circ 16 1 ∘ before perspective fitting and EM, and the full solver is within 1 ∘ 1^\circ 1 ∘ on 88 of 90 poses.
Normalized point sets. We center the model points at their mean y ˉ \bar{y} y ˉ and normalize them by their empirical covariance Σ Y = N − 1 ∑ j ( y j − y ˉ ) ( y j − y ˉ ) ⊤ \Sigma_Y=N^{-1}\sum_j(y_j-\bar{y})(y_j-\bar{y})^\top Σ Y = N − 1 ∑ j ( y j − y ˉ ) ( y j − y ˉ ) ⊤ . This gives the normalized points y ~ j = Σ Y − 1 / 2 ( y j − y ˉ ) \tilde{y}_j=\Sigma_Y^{-1/2}(y_j-\bar{y}) y ~ j = Σ Y − 1/2 ( y j − y ˉ ) , which form the normalized 3D point set Y ~ \widetilde{\mathcal{Y}} Y . We normalize the calibrated observations in the same way by their own mean and covariance, which gives U ~ \tilde{U} U ~ . Both covariances are positive definite by assumption, so both normalizations are defined. The next lemma proves the reduction stated before Theorem A1, namely that, for complete noiseless affine observations, the two normalized point sets differ only by an orientation.
Lemma A3 (Affine normalization). Suppose that the 3D point set Y = { y j } j = 1 N \mathcal{Y}=\{y_j\}_{j=1}^{N} Y = { y j } j = 1 N has positive definite covariance Σ Y \Sigma_Y Σ Y , that A ∈ R 2 × 3 A\in\mathbb{R}^{2\times3} A ∈ R 2 × 3 has rank two, and that b ∈ R 2 b\in\mathbb{R}^2 b ∈ R 2 is an image translation. Let σ \sigma σ be a permutation, and let p i = A y σ ( i ) + b p_i=Ay_{\sigma(i)}+b p i = A y σ ( i ) + b be complete affine projections without measurement error, with mean p ˉ \bar{p} p ˉ . Then their covariance Σ p = A Σ Y A ⊤ \Sigma_p=A\Sigma_YA^\top Σ p = A Σ Y A ⊤ is positive definite, and the normalized projections p ~ i = Σ p − 1 / 2 ( p i − p ˉ ) \tilde{p}_i=\Sigma_p^{-1/2}(p_i-\bar{p}) p ~ i = Σ p − 1/2 ( p i − p ˉ ) satisfy
p ~ i = O y ~ σ ( i ) , O = Σ p − 1 / 2 A Σ Y 1 / 2 , O O ⊤ = I 2 . (12) \tilde{p}_i=O\tilde{y}_{\sigma(i)},\qquad O=\Sigma_p^{-1/2}A\Sigma_Y^{1/2},\qquad OO^\top=I_2. \tag*{(12)} p ~ i = O y ~ σ ( i ) , O = Σ p − 1/2 A Σ Y 1/2 , O O ⊤ = I 2 . (12)
Proof. The permutation σ \sigma σ reorders the points, so the projections have mean A y ˉ + b A\bar{y}+b A y ˉ + b and covariance A Σ Y A ⊤ A\Sigma_YA^\top A Σ Y A ⊤ , and centering removes b b b . For a nonzero vector r ∈ R 2 r\in\mathbb{R}^2 r ∈ R 2 , the rank of A A A gives A ⊤ r ≠ 0 A^\top r\ne0 A ⊤ r = 0 , and hence r ⊤ Σ p r = ( A ⊤ r ) ⊤ Σ Y ( A ⊤ r ) > 0 r^\top\Sigma_pr=(A^\top r)^\top\Sigma_Y(A^\top r)>0 r ⊤ Σ p r = ( A ⊤ r ) ⊤ Σ Y ( A ⊤ r ) > 0 . Substituting this covariance into O O ⊤ OO^\top O O ⊤ gives Σ p − 1 / 2 Σ p Σ p − 1 / 2 = I 2 \Sigma_p^{-1/2}\Sigma_p\Sigma_p^{-1/2}=I_2 Σ p − 1/2 Σ p Σ p − 1/2 = I 2 . Finally, p ~ i = Σ p − 1 / 2 A ( y σ ( i ) − y ˉ ) = Σ p − 1 / 2 A Σ Y 1 / 2 y ~ σ ( i ) = O y ~ σ ( i ) \tilde{p}_i=\Sigma_p^{-1/2}A(y_{\sigma(i)}-\bar{y})=\Sigma_p^{-1/2}A\Sigma_Y^{1/2}\tilde{y}_{\sigma(i)}=O\tilde{y}_{\sigma(i)} p ~ i = Σ p − 1/2 A ( y σ ( i ) − y ˉ ) = Σ p − 1/2 A Σ Y 1/2 y ~ σ ( i ) = O y ~ σ ( i ) .
Suppose that the observations are complete and noiseless and that the calibrated observations are u c a l , i = A y σ ( i ) + b u_{\mathrm{cal},i}=Ay_{\sigma(i)}+b u cal , i = A y σ ( i ) + b for a matrix A ∈ R 2 × 3 A\in\mathbb{R}^{2\times3} A ∈ R 2 × 3 of rank two and a vector b ∈ R 2 b\in\mathbb{R}^2 b ∈ R 2 . Lemma A3 with p i = u c a l , i p_i=u_{\mathrm{cal},i} p i = u cal , i then gives the normalized observations u ~ i = O ⋆ y ~ σ ( i ) \tilde{u}_i=O^\star\tilde{y}_{\sigma(i)} u ~ i = O ⋆ y ~ σ ( i ) for a matrix O ⋆ O^\star O ⋆ with orthonormal rows, that is, for an orientation O ⋆ ∈ V O^\star\in\mathcal{V} O ⋆ ∈ V . Moments do not depend on the order of the points, so the correspondences drop out, and the normalized observed moments are T U = F T ( O ⋆ ) T_U=\mathcal{F}_T(O^\star) T U = F T ( O ⋆ ) .
Pixel & calibrated coordinates. Theorem A1 normalizes the calibrated observations. Normalizing the pixel coordinates gives the same orientation up to an orthogonal map of the image. Removing the intrinsincs is an invertible affine map u ↦ C K u + b K u\mapsto C_Ku+b_K u ↦ C K u + b K , with an invertible matrix C K ∈ R 2 × 2 C_K\in\mathbb{R}^{2\times2} C K ∈ R 2 × 2 , that sends a pixel u u u to its calibrated point u c a l u_{\mathrm{cal}} u cal . If the pixel covariance Σ U \Sigma_U Σ U is positive definite, the normalized pixel coordinates are e ~ p i x = Σ U − 1 / 2 ( u − u ˉ ) \tilde{e}_{\mathrm{pix}}=\Sigma_U^{-1/2}(u-\bar{u}) e ~ pix = Σ U − 1/2 ( u − u ˉ ) . The calibrated covariance is C K Σ U C K ⊤ C_K\Sigma_UC_K^\top C K Σ U C K ⊤ , so normalization after calibration gives u ~ = Q K e ~ p i x \tilde{u}=Q_K\tilde{e}_{\mathrm{pix}} u ~ = Q K e ~ pix with Q K = ( C K Σ U C K ⊤ ) − 1 / 2 C K Σ U 1 / 2 Q_K=(C_K\Sigma_UC_K^\top)^{-1/2}C_K\Sigma_U^{1/2} Q K = ( C K Σ U C K ⊤ ) − 1/2 C K Σ U 1/2 , which is orthogonal because Q K Q K ⊤ = I 2 Q_KQ_K^\top=I_2 Q K Q K ⊤ = I 2 . If u ~ i = O ⋆ y ~ σ ( i ) \tilde{u}_i=O^\star\tilde{y}_{\sigma(i)} u ~ i = O ⋆ y ~ σ ( i ) , the normalized pixel coordinates of observation i i i are therefore Q K ⊤ O ⋆ y ~ σ ( i ) Q_K^\top O^\star\tilde{y}_{\sigma(i)} Q K ⊤ O ⋆ y ~ σ ( i ) , and Q K ⊤ O ⋆ Q_K^\top O^\star Q K ⊤ O ⋆ is again an orientation in V \mathcal{V} V . Perspective fitting does not need this comparison, because L 4 \mathcal{L}_4 L 4 applies one fixed map h h h to the observations and to the projections.
Separation constant. The constant of Theorem A1 is the greatest lower bound of the ratio between a change of the projected moments and the change of orientation that causes it, in the Frobenius norm,
Γ T = inf O , O ′ ∈ V O ≠ O ′ ∥ F T ( O ) − F T ( O ′ ) ∥ ∥ O − O ′ ∥ . (13) \Gamma_T=\inf_{\substack{O,O'\in\mathcal{V}\\O\ne O'}}\frac{\|\mathcal{F}_T(O)-\mathcal{F}_T(O')\|}{\|O-O'\|}. \tag*{(13)} Γ T = O , O ′ ∈ V O = O ′ inf ∥ O − O ′ ∥ ∥ F T ( O ) − F T ( O ′ ) ∥ . (13)
Outside the exceptional set defined in the proof, the derivative of F T \mathcal{F}_T F T in orientation also has rank three at every O ∈ V O\in\mathcal{V} O ∈ V . Step 4 of the proof uses this to show Γ T > 0 \Gamma_T>0 Γ T > 0 .
Outline of the proof. The proof has four parts. First, we write each moment tensor as a homogeneous polynomial, so that F T ( O ) \mathcal{F}_T(O) F T ( O ) becomes the restriction of two polynomials to the plane spanned by the rows of O O O . Second, Lemma A4 counts the independent linear conditions on T T T under which two distinct orientations have equal moments, and those under which the derivative of F T \mathcal{F}_T F T vanishes in a tangent direction. Third, the proof of Theorem A1 combines these counts with the dimensions of the families of orientation pairs and tangent directions. Steps 1 to 3 of that proof show that the exceptional moment pairs T T T form a set of dimension at most 24 in the 25-dimensional space of moment pairs. Outside this set, compactness gives Γ T > 0 \Gamma_T>0 Γ T > 0 (Step 4), and the definition of Γ T \Gamma_T Γ T gives the error bound (Step 5). Fourth, a separate proof shows that a model of N ≥ 34 N\ge34 N ≥ 34 points drawn from a joint density avoids the exceptional set with probability one. After the proof, Lemma A5 shows that Γ T \Gamma_T Γ T depends continuously on T T T and vanishes for symmetric models. Together, these two properties imply that nearly symmetric models have small Γ T \Gamma_T Γ T (Corollary A2).
Moments as forms. A symmetric tensor of order g g g on R 3 \mathbb{R}^3 R 3 is equivalent to a form of degree g g g , which is a polynomial in three variables whose monomials all have total degree g g g . We use forms because restricting a polynomial to a plane is simpler to analyze than contracting a tensor. We give each form the Frobenius norm of its symmetric tensor, and a pair of forms the direct-sum norm of the two tensors, as in Theorem A1. The value of Γ T \Gamma_T Γ T depends on this choice, because the Frobenius norm weights each distinct entry m α m_\alpha m α by its multiplicity (App. § 8 .§ 0.1 ). Unweighted norms of the distinct polynomial coefficients need not give the same value. In this representation, the normalized model moment M g ( Y ~ ) M_g(\widetilde{Y}) M g ( Y ) is the form
F g ( ζ ) = ⟨ M g ( Y ~ ) , ζ ⊗ g ⟩ = 1 N ∑ j = 1 N ( ζ ⋅ y ~ j ) g , ζ ∈ R 3 . (14) F_g(\zeta)=\langle M_g(\widetilde{Y}),\zeta^{\otimes g}\rangle=\frac{1}{N}\sum_{j=1}^{N}(\zeta\cdot\widetilde{y}_j)^g,\qquad\zeta\in\mathbb{R}^3. \tag*{(14)} F g ( ζ ) = ⟨ M g ( Y ) , ζ ⊗ g ⟩ = N 1 j = 1 ∑ N ( ζ ⋅ y j ) g , ζ ∈ R 3 . (14)
We write S g S_g S g for the space of forms of degree g g g on R 3 \mathbb{R}^3 R 3 , of dimension ( g + 2 2 ) \binom{g+2}{2} ( 2 g + 2 ) , and H g H_g H g for the space of forms of degree g g g on R 2 \mathbb{R}^2 R 2 , of dimension g + 1 g+1 g + 1 . The pair T = ( F 3 , F 4 ) T=(F_3,F_4) T = ( F 3 , F 4 ) is then a point of T = S 3 ⊕ S 4 \mathcal{T}=S_3\oplus S_4 T = S 3 ⊕ S 4 . This space has dimension 10 + 15 = 25 10+15=25 10 + 15 = 25 and is the space of moment pairs in Theorem A1.
Projected moments as restrictions. The two orthonormal rows of an orientation O ∈ V O\in V O ∈ V span a plane in R 3 \mathbb{R}^3 R 3 . We write ν \nu ν for a unit normal of this plane, so that O ν = 0 O\nu=0 O ν = 0 and O T O = I 3 − ν ν T O^{\mathsf T}O=I_3-\nu\nu^{\mathsf T} O T O = I 3 − ν ν T is the orthogonal projection onto the plane ν ⊥ \nu^\perp ν ⊥ . For image coordinates u ∈ R 2 u\in\mathbb{R}^2 u ∈ R 2 , we have u ⋅ O y ~ j = O T u ⋅ y ~ j u\cdot O\widetilde{y}_j=O^{\mathsf T}u\cdot\widetilde{y}_j u ⋅ O y j = O T u ⋅ y j . The projected moment of order g g g is therefore the restriction of F g F_g F g to the plane, written in image coordinates,
( R O F g ) ( u ) = F g ( O T u ) , u ∈ R 2 . (15) (R_OF_g)(u)=F_g(O^{\mathsf T}u),\qquad u\in\mathbb{R}^2. \tag*{(15)} ( R O F g ) ( u ) = F g ( O T u ) , u ∈ R 2 . (15)
The cubic and quartic projected moments form the pair
F T ( O ) = ( R O F 3 , R O F 4 ) ∈ H 3 ⊕ H 4 , (16) F_T(O)=(R_OF_3,R_OF_4)\in H_3\oplus H_4, \tag*{(16)} F T ( O ) = ( R O F 3 , R O F 4 ) ∈ H 3 ⊕ H 4 , (16)
in a space of dimension 4 + 5 = 9 4+5=9 4 + 5 = 9 . The map F T F_T F T is linear in T T T and polynomial in O O O , and it is defined for every T ∈ T T\in\mathcal{T} T ∈ T , whether or not T T T comes from a point set. Each restriction R O R_O R O maps S g S_g S g onto H g H_g H g , because a form f ∈ H g f\in H_g f ∈ H g has the extension ζ ↦ f ( O ζ ) \zeta\mapsto f(O\zeta) ζ ↦ f ( O ζ ) , whose restriction is f f f . Two orientations O O O and O ′ O' O ′ have equal projected moments exactly when T T T lies in the kernel of the linear operator Δ O , O ′ : T → H 3 ⊕ H 4 \Delta_{O,O'}:\mathcal{T}\to H_3\oplus H_4 Δ O , O ′ : T → H 3 ⊕ H 4 given by
Δ O , O ′ T = F T ( O ) − F T ( O ′ ) . (17) \Delta_{O,O'}T=F_T(O)-F_T(O'). \tag*{(17)} Δ O , O ′ T = F T ( O ) − F T ( O ′ ) . (17)
The rank of Δ O , O ′ \Delta_{O,O'} Δ O , O ′ is the number of independent linear conditions on T T T under which the two orientations share their moments. The following lemma computes this rank. It also computes the rank of the derivative of F T F_T F T in a tangent direction, which plays the same role for two orientations that are infinitely close.
Lemma A4 (Rank of the moment difference of two orientations). Let O O O and O ′ O' O ′ be orientations with unit normals ν \nu ν and ν ′ \nu' ν ′ , distinct in parts (a) and (b).
(a) If ν ′ ≠ ± ν \nu'\ne\pm\nu ν ′ = ± ν , let d L d_L d L be a unit vector orthogonal to both normals, and put o = O d L o=Od_L o = O d L and o ′ = O ′ d L o'=O'd_L o ′ = O ′ d L . The rank of Δ O , O ′ \Delta_{O,O'} Δ O , O ′ is 9 9 9 if o ′ ≠ ± o o'\ne\pm o o ′ = ± o . It is 7 7 7 if o ′ = o o'=o o ′ = o , and 8 8 8 if o ′ = − o o'=-o o ′ = − o .
(b) If ν ′ = ± ν \nu'=\pm\nu ν ′ = ± ν , then O ′ = G O O'=GO O ′ = GO for an orthogonal matrix G ≠ I 2 G\ne I_2 G = I 2 . The rank of Δ O , O ′ \Delta_{O,O'} Δ O , O ′ is 4 4 4 if G G G is a reflection or the rotation by π \pi π . It is 6 6 6 if G G G is a rotation by ± π / 2 \pm\pi/2 ± π /2 or ± 2 π / 3 \pm2\pi/3 ± 2 π /3 , and 8 8 8 for every other rotation G ≠ I 2 G\ne I_2 G = I 2 .
(c) Let R ⊥ R_{\perp} R ⊥ be the rotation of the plane by π / 2 \pi/2 π /2 . The tangent space of V V V at O O O consists of the matrices O ˙ = ω R ⊥ O + q ν T \dot O=\omega R_{\perp}O+q\nu^{\mathsf T} O ˙ = ω R ⊥ O + q ν T with ω ∈ R \omega\in\mathbb{R} ω ∈ R and q ∈ R 2 q\in\mathbb{R}^2 q ∈ R 2 . The derivative of F T F_T F T at O O O in the direction O ˙ \dot O O ˙ , as a linear map of T T T , has rank 9 9 9 if ω ≠ 0 ≠ q \omega\ne0\ne q ω = 0 = q . Its rank is 7 7 7 if ω = 0 ≠ q \omega=0\ne q ω = 0 = q , and 8 8 8 if q = 0 ≠ ω q=0\ne\omega q = 0 = ω .
Proof. Each operator acts on the cubic and the quartic component of T T T separately. In all three parts, we therefore compute the rank for each degree g ∈ { 3 , 4 } g\in\{3,4\} g ∈ { 3 , 4 } and add the two ranks.
Part (a). The two planes are distinct and meet in the line spanned by d L d_L d L . We first determine which pairs ( R O F , R O ′ F ) (R_OF,R_{O'}F) ( R O F , R O ′ F ) arise as F F F ranges over S g S_g S g , and then which differences between the restrictions arise. A form F ∈ S g F\in S_g F ∈ S g satisfies R O F = 0 R_OF=0 R O F = 0 exactly when it vanishes on the plane ν ⊥ \nu^\perp ν ⊥ , that is, when it is divisible by ν ⋅ ζ \nu\cdot\zeta ν ⋅ ζ . A form that vanishes on both planes is therefore divisible by ( ν ⋅ ζ ) ( ν ′ ⋅ ζ ) (\nu\cdot\zeta)(\nu'\cdot\zeta) ( ν ⋅ ζ ) ( ν ′ ⋅ ζ ) . Dividing out these factors identifies the kernel of the map F ↦ ( R O F , R O ′ F ) F\mapsto(R_OF,R_{O'}F) F ↦ ( R O F , R O ′ F ) with S g − 2 S_{g-2} S g − 2 , so this map has rank ( g + 2 2 ) − ( g 2 ) = 2 g + 1 \binom{g+2}{2}-\binom{g}{2}=2g+1 ( 2 g + 2 ) − ( 2 g ) = 2 g + 1 . The restrictions R O F R_OF R O F at o o o and R O ′ F R_{O'}F R O ′ F at o ′ o' o ′ both take the value F ( d L ) F(d_L) F ( d L ) , because O T o = d L = O ′ T o ′ O^{\mathsf T}o=d_L=O'^{\mathsf T}o' O T o = d L = O ′ T o ′ . The image of the map is therefore contained in
W g = { ( f , f ′ ) ∈ H g × H g : f ( o ) = f ′ ( o ′ ) } . (18) W_g=\{(f,f')\in H_g\times H_g:f(o)=f'(o')\}. \tag*{(18)} W g = {( f , f ′ ) ∈ H g × H g : f ( o ) = f ′ ( o ′ )} . (18)
Since W g W_g W g also has dimension 2 g + 1 2g+1 2 g + 1 , the image equals W g W_g W g .
The degree-g g g component of Δ O , O ′ \Delta_{O,O'} Δ O , O ′ subtracts the two restrictions, so its image is { f − f ′ : ( f , f ′ ) ∈ W g } \{f-f':(f,f')\in W_g\} { f − f ′ : ( f , f ′ ) ∈ W g } . Suppose first that some form of H g H_g H g takes different values at o o o and o ′ o' o ′ . The image is then all of H g H_g H g . Indeed, given f 0 ∈ H g f_0\in H_g f 0 ∈ H g , we choose f ′ ∈ H g f'\in H_g f ′ ∈ H g with f ′ ( o ′ ) − f ′ ( o ) = f 0 ( o ) f'(o')-f'(o)=f_0(o) f ′ ( o ′ ) − f ′ ( o ) = f 0 ( o ) and put f = f 0 + f ′ f=f_0+f' f = f 0 + f ′ , so that ( f , f ′ ) ∈ W g (f,f')\in W_g ( f , f ′ ) ∈ W g and f − f ′ = f 0 f-f'=f_0 f − f ′ = f 0 . Suppose now that all forms of degree g g g agree at o o o and o ′ o' o ′ . Every difference then vanishes at o o o , and every f f f with f ( o ) = 0 f(o)=0 f ( o ) = 0 is the difference between the components of ( f , 0 ) ∈ W g (f,0)\in W_g ( f , 0 ) ∈ W g . The image is then defined by one linear condition, so its dimension drops by one. Because o o o and o ′ o' o ′ are unit vectors, all forms of degree g g g agree at them exactly when o ′ = o o'=o o ′ = o , or when o ′ = − o o'=-o o ′ = − o and g g g is even. The ranks are therefore 4 + 5 4+5 4 + 5 , 3 + 4 3+4 3 + 4 and 4 + 4 4+4 4 + 4 in the three cases.
Part (b). The two orientations share their plane and differ only in the image coordinates on it. The rows of O ′ O' O ′ lie in the row space ν ⊥ \nu^\perp ν ⊥ of O O O , so O ′ = O ′ O T O = G O O'=O'O^{\mathsf T}O=GO O ′ = O ′ O T O = GO with G = O ′ O T G = O'O^{\mathsf T} G = O ′ O T . The matrix G G G is orthogonal because G G T = O ′ ( I 3 − ν ν T ) O ′ T = I 2 GG^{\mathsf T} = O'(I_3-\nu\nu^{\mathsf T})O'^{\mathsf T} = I_2 G G T = O ′ ( I 3 − ν ν T ) O ′ T = I 2 , and G ≠ I 2 G \ne I_2 G = I 2 because O ′ ≠ O O' \ne O O ′ = O . By Eq. (15) , R O ′ F = ( R O F ) ∘ G T R_{O'}F=(R_OF)\circ G^{\mathsf T} R O ′ F = ( R O F ) ∘ G T . Since R O R_O R O is onto, the degree-g g g rank of Δ O , O ′ \Delta_{O,O'} Δ O , O ′ is g + 1 g+1 g + 1 minus the dimension of the space of forms f ∈ H g f\in\mathcal H_g f ∈ H g with f ∘ G T = f f\circ G^{\mathsf T}=f f ∘ G T = f . It remains to count these invariant forms.
For a rotation G G G by an angle θ \theta θ , complex coordinates make the action diagonal. In the coordinate z = u 1 + i u 2 z=u_1+iu_2 z = u 1 + i u 2 , the monomials z j z ˉ g − j z^j\bar z^{g-j} z j z ˉ g − j with 0 ≤ j ≤ g 0\leq j\leq g 0 ≤ j ≤ g span the complex forms of degree g g g , and f ↦ f ∘ G T f\mapsto f\circ G^{\mathsf T} f ↦ f ∘ G T multiplies them by e − i ( 2 j − g ) θ e^{-i(2j-g)\theta} e − i ( 2 j − g ) θ . This operator is real, so its fixed subspaces over R \mathbb R R and over C \mathbb C C have the same dimension, namely the number of indices j j j with ( 2 j − g ) θ ∈ 2 π Z (2j-g)\theta\in2\pi\mathbb Z ( 2 j − g ) θ ∈ 2 π Z . Since G ≠ I 2 G\ne I_2 G = I 2 , the angle θ \theta θ is not a multiple of 2 π 2\pi 2 π . For g = 3 g=3 g = 3 , the integers 2 j − g 2j-g 2 j − g are ± 1 \pm1 ± 1 and ± 3 \pm3 ± 3 , so this dimension is 2 if 3 θ ∈ 2 π Z 3\theta\in2\pi\mathbb Z 3 θ ∈ 2 π Z and 0 otherwise. For g = 4 g=4 g = 4 , the integers are 0, ± 2 \pm2 ± 2 and ± 4 \pm4 ± 4 , so the dimension is 1, plus 2 if 2 θ ∈ 2 π Z 2\theta\in2\pi\mathbb Z 2 θ ∈ 2 π Z , plus 2 if 4 θ ∈ 2 π Z 4\theta\in2\pi\mathbb Z 4 θ ∈ 2 π Z . For a reflection G G G , choosing coordinate axes along and perpendicular to its axis gives ( u 1 , u 2 ) ↦ ( u 1 , − u 2 ) (u_1,u_2)\mapsto(u_1,-u_2) ( u 1 , u 2 ) ↦ ( u 1 , − u 2 ) . The invariant forms are then those even in u 2 u_2 u 2 , with dimensions 2 for g = 3 g=3 g = 3 and 3 for g = 4 g=4 g = 4 . Subtracting these dimensions from 4 + 5 4+5 4 + 5 gives the ranks of part (b).
Part (c). A tangent vector is a first-order change of an orientation that keeps its rows orthonormal. Differentiating O O T = I 2 OO^{\mathsf T}=I_2 O O T = I 2 shows that the tangent vectors are the matrices O ˙ \dot O O ˙ with O O ˙ T + O ˙ O T = 0 O\dot O^{\mathsf T}+\dot O O^{\mathsf T}=0 O O ˙ T + O ˙ O T = 0 , a space of dimension three. The matrices ω R ⊥ O + q ν T \omega R_\perp O+q\nu^{\mathsf T} ω R ⊥ O + q ν T satisfy this equation because O ν = 0 O\nu=0 O ν = 0 and R ⊥ T = − R ⊥ R_\perp^{\mathsf T}=-R_\perp R ⊥ T = − R ⊥ . They also span three dimensions, so they form the tangent space. The term ω R ⊥ O \omega R_\perp O ω R ⊥ O rotates the image coordinates within the plane, and the term q ν T q\nu^{\mathsf T} q ν T tilts the plane in R 3 \mathbb R^3 R 3 .
To differentiate the restriction, we expand F ∈ S g F\in\mathcal S_g F ∈ S g in powers of the coordinate normal to the plane,
F ( ζ ) = ∑ j = 0 g ( ν ⋅ ζ ) j F ( j ) ( O ζ ) , F ( j ) ∈ H g − j . (19) F(\zeta)=\sum_{j=0}^{g}(\nu\cdot\zeta)^jF^{(j)}(O\zeta),\qquad F^{(j)}\in\mathcal H_{g-j}. \tag*{(19)} F ( ζ ) = j = 0 ∑ g ( ν ⋅ ζ ) j F ( j ) ( O ζ ) , F ( j ) ∈ H g − j . (19)
As F F F ranges over S g \mathcal S_g S g , the pair ( F ( 0 ) , F ( 1 ) ) (F^{(0)},F^{(1)}) ( F ( 0 ) , F ( 1 ) ) ranges over H g × H g − 1 \mathcal H_g\times\mathcal H_{g-1} H g × H g − 1 . Since ( O + τ O ˙ ) T u = O T ( u + τ ω R ⊥ T u ) + τ ( q ⋅ u ) ν (O+\tau\dot O)^{\mathsf T}u=O^{\mathsf T}(u+\tau\omega R_\perp^{\mathsf T}u)+\tau(q\cdot u)\nu ( O + τ O ˙ ) T u = O T ( u + τ ω R ⊥ T u ) + τ ( q ⋅ u ) ν , the derivative of the restriction is
d d τ R O + τ O ˙ F ∣ τ = 0 = ω ∂ R ⊥ F ( 0 ) + ( q ⋅ u ) F ( 1 ) , \left.\frac{d}{d\tau}R_{O+\tau\dot O}F\right|_{\tau=0}
=\omega\partial_{R_\perp}F^{(0)}+(q\cdot u)F^{(1)}, d τ d R O + τ O ˙ F τ = 0 = ω ∂ R ⊥ F ( 0 ) + ( q ⋅ u ) F ( 1 ) ,
( ∂ R ⊥ f ) ( u ) = ∇ f ( u ) ⋅ R ⊥ T u . (20) (\partial_{R_\perp}f)(u)=\nabla f(u)\cdot R_\perp^{\mathsf T}u. \tag*{(20)} ( ∂ R ⊥ f ) ( u ) = ∇ f ( u ) ⋅ R ⊥ T u . (20)
The first term comes from the rotation within the plane and the second from the tilt of the plane. Because F ( 0 ) F^{(0)} F ( 0 ) and F ( 1 ) F^{(1)} F ( 1 ) vary independently, the image of the derivative is the sum of the images of the two terms.
Differentiating the rotation factors of part (b) at θ = 0 \theta=0 θ = 0 shows that ∂ R ⊥ \partial_{R_\perp} ∂ R ⊥ has the eigenvalues − i ( 2 j − g ) -i(2j-g) − i ( 2 j − g ) for 0 ≤ j ≤ g 0\leq j\leq g 0 ≤ j ≤ g . The operator is therefore invertible on H 3 \mathcal H_3 H 3 , and its image in H 4 \mathcal H_4 H 4 consists of the forms with zero mean on the unit circle, a space of dimension four. Multiplication by q ⋅ u q\cdot u q ⋅ u with q ≠ 0 q\ne0 q = 0 is injective from H g − 1 \mathcal H_{g-1} H g − 1 to H g \mathcal H_g H g , so the tilt term has rank g g g . If ω ≠ 0 ≠ q \omega\ne0\ne q ω = 0 = q , the cubic component is onto because ∂ R ⊥ \partial_{R_\perp} ∂ R ⊥ is invertible on H 3 \mathcal H_3 H 3 . The quartic form ( q ⋅ u ) 4 (q\cdot u)^4 ( q ⋅ u ) 4 lies in the image of the tilt term and has positive mean on the unit circle. It therefore supplies the direction missing from the image of ∂ R ⊥ \partial_{R_\perp} ∂ R ⊥ , so the quartic component is also onto and the rank is 4 + 5 4+5 4 + 5 . If ω = 0 ≠ q \omega=0\ne q ω = 0 = q , only the tilt term remains, with rank 3 + 4 3+4 3 + 4 . If q = 0 ≠ ω q=0\ne\omega q = 0 = ω , only the rotation term remains, with rank 4 + 4 4+4 4 + 4 . □ \square □
Proof of Theorem A1. Steps 1 to 3 show that the moment pairs T T T for which F T F_T F T fails to be one-to-one, or has a derivative of rank less than three, form a set of dimension at most 24. The sets involved are semialgebraic, that is, built by finite unions and intersections from polynomial equations and inequalities. As in the proof of Theorem 1, we use three facts about their dimension [2 ] . First, a semialgebraic map does not increase dimension. Second, the dimension of its domain is at most the dimension of its image plus the largest dimension of a fiber, the set of points mapped to one point. Third, a semialgebraic set of smaller dimension than its ambient Euclidean space has Lebesgue measure zero and lies in the zero set of a nonzero polynomial.
Step 1. Two orientations with the same moments. Let T i n j T_{\mathrm{inj}} T inj be the set of T ∈ T T\in\mathcal T T ∈ T with Δ O , O ′ T = 0 \Delta_{O,O'}T=0 Δ O , O ′ T = 0 for some O ≠ O ′ O\ne O' O = O ′ , so that F T F_T F T is one-to-one exactly when T ∉ T i n j T\notin T_{\mathrm{inj}} T ∈ / T inj . We divide the pairs of distinct orientations into the families of Table A2 . On each family, the rank of Δ O , O ′ \Delta_{O,O'} Δ O , O ′ is constant by Lemma A4. We consider the triples ( O , O ′ , T ) (O,O',T) ( O , O ′ , T ) with ( O , O ′ ) (O,O') ( O , O ′ ) in a family D \mathcal D D and Δ O , O ′ T = 0 \Delta_{O,O'}T=0 Δ O , O ′ T = 0 . The fiber of these triples over a fixed pair ( O , O ′ ) (O,O') ( O , O ′ ) is the kernel of Δ O , O ′ \Delta_{O,O'} Δ O , O ′ , of dimension 25 minus the rank, and the pairs contribute at most dim D \dim\mathcal D dim D further parameters. By the first two facts, the set of T T T that occur in these triples has dimension at most
Table A2. Table A2. Families of distinct orientation pairs in the proof of Theorem A1. The columns give the dimension of each family D \mathcal D D , the rank of Δ O , O ′ \Delta_{O,O'} Δ O , O ′ on the family (Lemma A4), and the bound of Eq. (21) on the dimension of the set of moment pairs T T T for which some pair of orientations in the family has equal moments.
dim D + 25 − rank Δ O , O ′ . (21) \dim\mathcal D+25-\operatorname{rank}\Delta_{O,O'}. \tag*{(21)} dim D + 25 − rank Δ O , O ′ . (21)
The families are semialgebraic. We choose the normals as the cross products of the rows of O O O and of O ′ O' O ′ , which are polynomial in the entries. Then ν ′ = ± ν \nu'=\pm\nu ν ′ = ± ν exactly when O ′ ( I 3 − O ⊤ O ) = 0 O'(I_3-O^\top O)=0 O ′ ( I 3 − O ⊤ O ) = 0 . On the families with distinct planes, the cross product d × = ν × ν ′ d_\times=\nu\times\nu' d × = ν × ν ′ is a nonzero multiple of d L d_L d L , so o ′ = ± o o'=\pm o o ′ = ± o exactly when O ′ d × = ± O d × O'd_\times=\pm Od_\times O ′ d × = ± O d × . The dimensions of the families follow from dim V = 3 \dim\mathcal{V}=3 dim V = 3 . All pairs form a set of dimension 6. A pair with o ′ = ± o o'=\pm o o ′ = ± o is determined, up to finitely many choices, by O O O , the unit vector o o o and the angle between the two planes about d L = O ⊤ o d_L=O^\top o d L = O ⊤ o , which gives dimension 3 + 1 + 1 = 5 3+1+1=5 3 + 1 + 1 = 5 . A pair with ν ′ = ± ν \nu'=\pm\nu ν ′ = ± ν has the form ( O , G O ) (O,GO) ( O , GO ) , which gives dimension 3 when G G G is one of finitely many rotations and 4 when G G G ranges over all rotations. The reflection pairs ( O , G O ) (O,GO) ( O , GO ) also have dimension 4, and Eq. (21) would give 25 for them, so we remove one redundant parameter. Rotating the image coordinates of both orientations by the same planar rotation G 1 G_1 G 1 preserves equal moments. Indeed, Eq. (15) gives Δ G 1 O , G 1 G O ⊤ = ( Δ O , G O ⊤ ) ∘ G 1 ⊤ \Delta_{G_1O,G_1GO^\top}=(\Delta_{O,GO^\top})\circ G_1^\top Δ G 1 O , G 1 G O ⊤ = ( Δ O , G O ⊤ ) ∘ G 1 ⊤ , and composition with G 1 ⊤ G_1^\top G 1 ⊤ is invertible, so the two kernels agree. As the rotation G 1 G_1 G 1 varies, the conjugate G 1 G G 1 − 1 G_1GG_1^{-1} G 1 G G 1 − 1 runs over all reflections. It therefore suffices to consider the pairs ( O , diag ( 1 , − 1 ) O ) (O,\operatorname{diag}(1,-1)O) ( O , diag ( 1 , − 1 ) O ) , a family of dimension 3 with kernels of dimension 25 − 4 25-4 25 − 4 . The largest bound in Table A2 is 24, so dim T i n j ≤ 24 \dim T_{\mathrm{inj}}\le24 dim T inj ≤ 24 .
Step 2. An orientation with a degenerate derivative. Let T i m m T_{\mathrm{imm}} T imm be the set of T T T for which, at some orientation, the derivative of F T F_T F T vanishes in some nonzero tangent direction O ˙ \dot O O ˙ . For T ∉ T i m m T\notin T_{\mathrm{imm}} T ∈ / T imm , the derivative is injective on the three-dimensional tangent space at every orientation, that is, it has rank three. Step 4 needs this property, because two orientations that approach each other produce such a direction in the limit. The condition does not change when O ˙ \dot O O ˙ is multiplied by a nonzero scalar, so only the direction of O ˙ \dot O O ˙ contributes parameters. The families of orientations and tangent directions have dimension 3 + 2 3+2 3 + 2 when ω ≠ 0 ≠ q \omega\ne0\ne q ω = 0 = q , dimension 3 + 1 3+1 3 + 1 when ω = 0 \omega=0 ω = 0 , and dimension 3 when q = 0 q=0 q = 0 . By Lemma A4(c), the corresponding derivatives have ranks 9, 7 and 8. The count of Eq. (21) gives 21, 22 and 20, so dim T i m m ≤ 22 \dim T_{\mathrm{imm}}\le22 dim T imm ≤ 22 .
Step 3. A set of measure zero. The exceptional set T e x c = T i n j ∪ T i m m T_{\mathrm{exc}}=T_{\mathrm{inj}}\cup T_{\mathrm{imm}} T exc = T inj ∪ T imm is semialgebraic of dimension at most 24 in the 25-dimensional space T \mathcal{T} T . By the third fact, it has Lebesgue measure zero and lies in a proper algebraic subset of T \mathcal{T} T . For T ∉ T e x c T\notin T_{\mathrm{exc}} T ∈ / T exc , the map F T F_T F T is one-to-one on V \mathcal{V} V and its derivative has rank three at every orientation. The first property is the first claim of Theorem A1, and Step 4 uses the second.
Step 4. The lower bound is positive. Injectivity alone does not give Γ T > 0 \Gamma_T>0 Γ T > 0 , because the ratio in Eq. (3) could tend to zero along pairs of distinct orientations that approach each other. Compactness of V \mathcal{V} V and Step 2 exclude this. We fix T ∉ T e x c T\notin T_{\mathrm{exc}} T ∈ / T exc and suppose that Γ T = 0 \Gamma_T=0 Γ T = 0 . There are then distinct orientations O i , O i ′ ∈ V O_i,O_i'\in\mathcal{V} O i , O i ′ ∈ V for i = 1 , 2 , … i=1,2,\ldots i = 1 , 2 , … such that
∥ F T ( O i ) − F T ( O i ′ ) ∥ ∥ O i − O i ′ ∥ ⟶ 0. (22) \frac{\lVert F_T(O_i)-F_T(O_i')\rVert}{\lVert O_i-O_i'\rVert}\longrightarrow0. \tag*{(22)} ∥ O i − O i ′ ∥ ∥ F T ( O i ) − F T ( O i ′ )∥ ⟶ 0. (22)
By compactness, we may pass to a subsequence with O i → O O_i\to O O i → O and O i ′ → O ′ O_i'\to O' O i ′ → O ′ . If O ≠ O ′ O\ne O' O = O ′ , the numerators tend to zero because V \mathcal{V} V is bounded, and continuity gives F T ( O ) = F T ( O ′ ) F_T(O)=F_T(O') F T ( O ) = F T ( O ′ ) , which contradicts injectivity. If O = O ′ O=O' O = O ′ , we pass to a further subsequence along which the normalized differences ( O i ′ − O i ) / ∥ O i ′ − O i ∥ (O_i'-O_i)/\lVert O_i'-O_i\rVert ( O i ′ − O i ) / ∥ O i ′ − O i ∥ converge to a matrix O ˙ \dot O O ˙ of unit norm. Subtracting the equations O i ⊤ O i = I 2 O_i^\top O_i=I_2 O i ⊤ O i = I 2 and O i ′ ⊤ O i ′ = I 2 O_i'^\top O_i'=I_2 O i ′⊤ O i ′ = I 2 and dividing by ∥ O i ′ − O i ∥ \lVert O_i'-O_i\rVert ∥ O i ′ − O i ∥ gives O ˙ ⊤ O + O ⊤ O ˙ = 0 \dot O^\top O+O^\top\dot O=0 O ˙ ⊤ O + O ⊤ O ˙ = 0 in the limit, so O ˙ \dot O O ˙ is tangent at O O O . The map F T F_T F T extends to a polynomial, hence continuously differentiable, map on R 2 × 3 \mathbb{R}^{2\times3} R 2 × 3 . The moment difference therefore equals the derivative at O O O applied to O i ′ − O i O_i'-O_i O i ′ − O i , up to a remainder of order o ( ∥ O i ′ − O i ∥ ) o(\lVert O_i'-O_i\rVert) o (∥ O i ′ − O i ∥) . By Eq. (22) , the derivative at O O O vanishes in the direction O ˙ \dot O O ˙ , which contradicts Step 2. Hence Γ T > 0 \Gamma_T>0 Γ T > 0 .
Step 5. The error bound. Let T ∉ T e x c T\notin T_{\mathrm{exc}} T ∈ / T exc , and let O ∗ O^* O ∗ and O ^ \widehat O O lie in V \mathcal{V} V . The definition of Γ T \Gamma_T Γ T gives Γ T ∥ O ^ − O ∗ ∥ ≤ ∥ F T ( O ^ ) − F T ( O ∗ ) ∥ \Gamma_T\lVert\widehat O-O^*\rVert\le\lVert F_T(\widehat O)-F_T(O^*)\rVert Γ T ∥ O − O ∗ ∥ ≤ ∥ F T ( O ) − F T ( O ∗ )∥ . The triangle inequality through T V T\mathcal{V} T V bounds the right-hand side by ϵ ^ + ϵ \widehat\epsilon+\epsilon ϵ + ϵ , and dividing by Γ T > 0 \Gamma_T>0 Γ T > 0 proves the bound
∥ O ^ − O ∗ ∥ ≤ ϵ ^ + ϵ Γ T (23) \lVert\widehat O-O^*\rVert\le\frac{\widehat\epsilon+\epsilon}{\Gamma_T} \tag*{(23)} ∥ O − O ∗ ∥ ≤ Γ T ϵ + ϵ (23)
of Theorem A1. For complete noiseless affine observations, T U = F T ( O ∗ ) T\mathcal{U}=F_T(O^*) T U = F T ( O ∗ ) by the normalization above, so ϵ = 0 \epsilon=0 ϵ = 0 . Any orientation O ^ \widehat O O that fits the observed moments exactly has ϵ ^ = 0 \widehat\epsilon=0 ϵ = 0 and therefore equals O ∗ O^* O ∗ , so these observations determine the normalized orientation. □
Proof of Theorem A1 for sampled models. Step 3 of the proof above shows that T e x c T_{\mathrm{exc}} T exc has measure zero among moment pairs. A model, however, is a point of R 3 N \mathbb{R}^{3N} R 3 N , and a set of measure zero among moment pairs need not have a preimage of measure zero among models. We therefore show that the map from point coordinates to T T T is a submersion at almost every point set, meaning that its derivative maps onto the space of moment pairs. The pair T T T depends on the point set through its 34 power sums, one sum over j j j for each monomial of degree one to four in the three coordinates, and we treat the map to the power sums and the map from them to T T T in turn.
Step 1. From the point set to its power sums. The Jacobian of the power-sum map has one row per monomial, and the row of a monomial lists its gradients at y 1 , … , y N y_1,\ldots,y_N y 1 , … , y N . These rows are linearly dependent exactly when a nonzero linear combination of the 34 monomials has zero gradient at every y j y_j y j . Such a combination is a nonconstant polynomial, so the set where its gradient vanishes is a proper algebraic subset of R 3 \mathbb{R}^3 R 3 , of dimension at most two. Up to scale, these combinations form a family of dimension 33. Allowing each of the N N N points to lie in the corresponding set gives a semialgebraic set of dimension at most 33 + 2 N 33+2N 33 + 2 N , which contains every point set at which the Jacobian loses rank. This set has measure zero in R 3 N \mathbb{R}^{3N} R 3 N because 33 + 2 N < 3 N 33+2N<3N 33 + 2 N < 3 N .
Step 2. From the power sums to T T T . The derivative of the map from power sums to moment pairs has full row rank wherever the covariance is positive definite. With the sums of degrees one and two held fixed, normalization is a fixed invertible affine map of R 3 \mathbb{R}^{3} R 3 , and ( F 3 , F 4 ) (F_{3}, F_{4}) ( F 3 , F 4 ) is an invertible affine function of the sums of degrees three and four.
Step 3. Conclusion. For N ≥ 34 N \ge34 N ≥ 34 , the covariance is positive definite with probability one. By Steps 1 and 2, the map to T T T is a submersion on an open set whose complement has measure zero. After smooth changes of coordinates, a submersion acts locally as a coordinate projection, so the preimage of the null set T e x c \mathcal{T}_{\mathrm{exc}} T exc is null [43 ] . The distribution of the point set has a density on R 3 N \mathbb{R}^{3N} R 3 N , so it assigns probability zero to this preimage. □
Corollary A2 uses two further properties of Γ T \Gamma_{T} Γ T , which the next lemma proves.
Lemma A5 (Continuity of the lower bound and symmetric models). There is a constant C C C such that ∣ Γ T − Γ T ′ ∣ ≤ C ∥ T − T ′ ∥ |\Gamma_{T}-\Gamma_{T'}| \le C\|T-T'\| ∣ Γ T − Γ T ′ ∣ ≤ C ∥ T − T ′ ∥ for all moment pairs T T T and T ′ T' T ′ . If T = ( F 3 , F 4 ) T=(F_{3},F_{4}) T = ( F 3 , F 4 ) satisfies F 3 ∘ Q s = F 3 F_{3}\circ Q_{s}=F_{3} F 3 ∘ Q s = F 3 and F 4 ∘ Q s = F 4 F_{4}\circ Q_{s}=F_{4} F 4 ∘ Q s = F 4 for an orthogonal matrix Q s ≠ I 3 Q_{s}\ne I_{3} Q s = I 3 , then Γ T = 0 \Gamma_{T}=0 Γ T = 0 and T ∈ T e x c T\in\mathcal{T}_{\mathrm{exc}} T ∈ T exc .
Proof. Continuity. The map F T ( O ) F_{T}(O) F T ( O ) is linear in T T T and polynomial in O O O , and V \mathcal{V} V is bounded. There is therefore a constant C C C such that ∥ F T ( O ) − F T ( O ′ ) ∥ ≤ C ∥ T ∥ ∥ O − O ′ ∥ \|F_{T}(O)-F_{T}(O')\| \le C\|T\|\|O-O'\| ∥ F T ( O ) − F T ( O ′ ) ∥ ≤ C ∥ T ∥∥ O − O ′ ∥ for all orientations and all T T T . Applying this bound to T − T ′ T-T' T − T ′ , and the triangle inequality to the definition of Γ T \Gamma_{T} Γ T , gives ∣ Γ T − Γ T ′ ∣ ≤ C ∥ T − T ′ ∥ |\Gamma_{T}-\Gamma_{T'}| \le C\|T-T'\| ∣ Γ T − Γ T ′ ∣ ≤ C ∥ T − T ′ ∥ . In particular, the set { T : Γ T > 0 } \{T:\Gamma_{T}>0\} { T : Γ T > 0 } is open.
Symmetry. We choose an orientation O O O with O Q s ≠ O OQ_{s}\ne O O Q s = O , which exists because Q s ≠ I 3 Q_{s}\ne I_{3} Q s = I 3 . The matrix O Q s OQ_{s} O Q s is also an orientation. The forms are invariant under Q s T = Q s − 1 Q_{s}^{\mathsf{T}}=Q_{s}^{-1} Q s T = Q s − 1 as well, so R O Q s F g ( u ) = F g ( Q s T O T u ) = F g ( O T u ) R_{OQ_{s}}F_{g}(u)=F_{g}(Q_{s}^{\mathsf{T}}O^{\mathsf{T}}u)=F_{g}(O^{\mathsf{T}}u) R O Q s F g ( u ) = F g ( Q s T O T u ) = F g ( O T u ) . Hence F T ( O Q s ) = F T ( O ) F_{T}(OQ_{s})=F_{T}(O) F T ( O Q s ) = F T ( O ) , so F T F_{T} F T is not one-to-one and Γ T = 0 \Gamma_{T}=0 Γ T = 0 . The invariance holds when Q s Q_{s} Q s maps the normalized model to itself, and a symmetry of the model gives such a matrix.
Suppose that an invertible matrix L ≠ I 3 L\ne I_{3} L = I 3 and a permutation π \pi π satisfy L ( y j − y ˉ ) = y π ( j ) − y ˉ L(y_{j}-\bar{y})=y_{\pi(j)}-\bar{y} L ( y j − y ˉ ) = y π ( j ) − y ˉ for every j j j . A mirror plane or a nontrivial rotational symmetry of the model has this form, because an isometry that maps a finite set to itself fixes its centroid. Reordering the points does not change their covariance, so L Σ Y L T = Σ Y L\Sigma_{Y}L^{\mathsf{T}}=\Sigma_{Y} L Σ Y L T = Σ Y . The conjugate Q s = Σ Y − 1 / 2 L Σ Y 1 / 2 Q_{s}=\Sigma_{Y}^{-1/2}L\Sigma_{Y}^{1/2} Q s = Σ Y − 1/2 L Σ Y 1/2 is therefore orthogonal, because Q s Q s T = Σ Y − 1 / 2 L Σ Y L T Σ Y − 1 / 2 = I 3 Q_{s}Q_{s}^{\mathsf{T}}=\Sigma_{Y}^{-1/2}L\Sigma_{Y}L^{\mathsf{T}}\Sigma_{Y}^{-1/2}=I_{3} Q s Q s T = Σ Y − 1/2 L Σ Y L T Σ Y − 1/2 = I 3 . It differs from I 3 I_{3} I 3 and maps y ~ j \tilde{y}_{j} y ~ j to y ~ π ( j ) \tilde{y}_{\pi(j)} y ~ π ( j ) . Its inverse Q s T Q_{s}^{\mathsf{T}} Q s T also permutes the normalized points, so Eq. (14) gives F g ∘ Q s = F g F_{g}\circ Q_{s}=F_{g} F g ∘ Q s = F g , and the moment pair of the model lies in T e x c \mathcal{T}_{\mathrm{exc}} T exc . □
Depth variation & affine initial poses
The affine initial poses of Step 1 replace the depth of every model point by a common reference depth z r e f > 0 z_{\mathrm{ref}}>0 z ref > 0 , which adds a feature error before perspective fitting. To bound it, we write the camera coordinates of model point j j j as ( x j l a t , z r e f ( 1 + ϑ j ) ) (x_{j}^{\mathrm{lat}},z_{\mathrm{ref}}(1+\vartheta_{j})) ( x j lat , z ref ( 1 + ϑ j )) , with lateral coordinates x j l a t ∈ R 2 x_{j}^{\mathrm{lat}}\in\mathbb{R}^{2} x j lat ∈ R 2 and relative depth deviation ϑ j \vartheta_{j} ϑ j . The affine projection of the point in calibrated coordinates is p j a f f = x j l a t / z r e f p_{j}^{\mathrm{aff}}=x_{j}^{\mathrm{lat}}/z_{\mathrm{ref}} p j aff = x j lat / z ref . We assume that ∣ ϑ j ∣ ≤ ϑ max < 1 |\vartheta_{j}|\le\vartheta_{\max}<1 ∣ ϑ j ∣ ≤ ϑ m a x < 1 and ∥ p j a f f ∥ ≤ ρ c a l \|p_{j}^{\mathrm{aff}}\|\le\rho_{\mathrm{cal}} ∥ p j aff ∥ ≤ ρ cal for every j j j , so that ρ c a l \rho_{\mathrm{cal}} ρ cal bounds the radius of the calibrated affine projections. The perspective projection of the point is p j a f f / ( 1 + ϑ j ) p_{j}^{\mathrm{aff}}/(1+\vartheta_{j}) p j aff / ( 1 + ϑ j ) , so the two are at most ρ c a l ϑ max / ( 1 − ϑ max ) \rho_{\mathrm{cal}}\vartheta_{\max}/(1-\vartheta_{\max}) ρ cal ϑ m a x / ( 1 − ϑ m a x ) apart. For a fixed feature map ϕ \phi ϕ with Lipschitz constant Lip ( ϕ ) \operatorname{Lip}(\phi) Lip ( ϕ ) on the line segments joining these pairs, Lipschitz continuity and averaging give
∥ 1 N ∑ j [ ϕ ( p j a f f 1 + ϑ j ) − ϕ ( p j a f f ) ] ∥ ≤ Lip ( ϕ ) ρ c a l ϑ max 1 − ϑ max . (24) \left\lVert\frac{1}{N}\sum_{j}\left[\phi\left(\frac{p_{j}^{\mathrm{aff}}}{1+\vartheta_{j}}\right)-\phi\left(p_{j}^{\mathrm{aff}}\right)\right]\right\rVert\le\frac{\operatorname{Lip}(\phi)\rho_{\mathrm{cal}}\vartheta_{\max}}{1-\vartheta_{\max}}. \tag*{(24)} N 1 j ∑ [ ϕ ( 1 + ϑ j p j aff ) − ϕ ( p j aff ) ] ≤ 1 − ϑ m a x Lip ( ϕ ) ρ cal ϑ m a x . (24)
Monomial features composed with a fixed affine image map, and the Fourier features of § 3.1 , have finite Lipschitz constants on every bounded region of the image, so the bound applies to both. The bound concerns the feature averages at a fixed pose and grows with the relative depth deviation ϑ max \vartheta_{\max} ϑ m a x . Perspective fitting uses the individual depths, so on complete noiseless inputs it has no error from the affine approximation at the true pose. Table A19 compares the affine initial poses with the full solver as the depth variation changes.
Near-symmetry & bounded features
This section proves two facts that the main text uses. Corollary A2 shows that Γ T \Gamma_{T} Γ T is small when the moment pair of a model is close to that of a symmetric model, so the bound of Eq. (23) is weak for nearly symmetric models. Lemma A6 proves that, at the true pose, the expected feature averages over the observations and over the projections differ by η ( μ u , 0 − μ y , 0 ) \eta(\mu_{u,0}-\mu_{y,0}) η ( μ u , 0 − μ y , 0 ) , the term that moment matching with outliers subtracts in Definition 1.
Near-symmetry. A model with a symmetry has two distinct orientations with the same projected moments, so its moment pair lies in the exception set of Theorem A1 (Lemma A5). The next corollary shows that moment pairs near such a pair have a small constant Γ T \Gamma_{T} Γ T , and hence weak orientation bounds.
Corollary A2 (Near-symmetric models). Let T 0 T_{0} T 0 be a moment pair for which two distinct orientations give identical projected moments. Then Γ T 0 = 0 \Gamma_{T_{0}}=0 Γ T 0 = 0 , and every moment pair T T T satisfies Γ T ≤ C ∥ T − T 0 ∥ \Gamma_{T}\le C\|T-T_{0}\| Γ T ≤ C ∥ T − T 0 ∥ with the constant C C C of Lemma A5.
Proof. At T 0 T_{0} T 0 , the two orientations give a zero numerator in the definition of Γ T 0 \Gamma_{T_{0}} Γ T 0 in Eq. (13) , so Γ T 0 = 0 \Gamma_{T_{0}}=0 Γ T 0 = 0 . Lemma A5 shows that ∣ Γ T − Γ T ′ ∣ ≤ C ∥ T − T ′ ∥ |\Gamma_{T}-\Gamma_{T'}|\le C\|T-T'\| ∣ Γ T − Γ T ′ ∣ ≤ C ∥ T − T ′ ∥ for all moment pairs T T T and T ′ T' T ′ , and the choice T ′ = T 0 T'=T_{0} T ′ = T 0 gives the bound. □
Bounded features with outliers. Moment matching with outliers (Definition 1) uses the fact that, at the true pose, the inliers contribute equally to the average over the observations and to the average over the projections, so the two averages differ through the outliers alone. The next lemma states this identity for the feature means of two distributions. It also bounds the change of a feature average when a fraction of the points is replaced. Without a bound on the features, one replaced point could change the average arbitrarily, so moment matching uses bounded features when outliers may be present.
Lemma A6 (Bounded features with outliers). Let ϕ : R 2 → R d ϕ \phi:\mathbb{R}^{2}\to\mathbb{R}^{d_{\phi}} ϕ : R 2 → R d ϕ be a measurable feature map with sup u ∈ R 2 ∥ ϕ ( u ) ∥ ≤ 1 \sup_{u\in\mathbb{R}^{2}}\|\phi(u)\|\le1 sup u ∈ R 2 ∥ ϕ ( u ) ∥ ≤ 1 . Replacing a fraction 0 ≤ η ≤ 1 0\le\eta\le1 0 ≤ η ≤ 1 of the points of a nonempty finite collection of 2 D 2D 2 D points by other 2 D 2D 2 D points changes the average of ϕ \phi ϕ over the collection by at most 2 η 2\eta 2 η . Consider two distributions on R 2 \mathbb{R}^{2} R 2 , each a mixture of an inlier component with weight 1 − η 1-\eta 1 − η and an outlier component with weight η \eta η . Let the two inlier components have the same feature mean ϕ i n \phi_{\mathrm{in}} ϕ in , and let the outlier components have feature means μ U , 0 \mu_{U,0} μ U , 0 and μ Y , 0 \mu_{Y,0} μ Y , 0 . Then the feature means μ 1 \mu_{1} μ 1 and μ 2 \mu_{2} μ 2 of the two distributions satisfy
μ 1 = ( 1 − η ) ϕ i n + η μ U , 0 , μ 2 = ( 1 − η ) ϕ i n + η μ Y , 0 , μ 1 − μ 2 = η ( μ U , 0 − μ Y , 0 ) . (25) \begin{aligned}
\mu_{1} &= (1-\eta)\phi_{\mathrm{in}}+\eta\mu_{U,0},\\
\mu_{2} &= (1-\eta)\phi_{\mathrm{in}}+\eta\mu_{Y,0},\\
\mu_{1}-\mu_{2} &= \eta(\mu_{U,0}-\mu_{Y,0}). \tag*{(25)}
\end{aligned} μ 1 μ 2 μ 1 − μ 2 = ( 1 − η ) ϕ in + η μ U , 0 , = ( 1 − η ) ϕ in + η μ Y , 0 , = η ( μ U , 0 − μ Y , 0 ) . (25) Proof. Each replaced point changes the feature sum by a vector of norm at most two, so division by the size of the collection gives 2 η 2\eta 2 η . Each mixture mean is the weighted average of its inlier and outlier means, and subtracting the two cancels the common inlier term.
If both point sets undergo replacements, the difference of their feature averages changes by at most twice the sum of the replaced fractions. These bounds hold for a fixed feature map and a fixed image normalization, and they do not shrink as the number of points grows. Lemma A17 gives a bound that does shrink when the outlier model points have the projected distribution of the inliers.
Poisson reference for the association level
Definition 2 compares the distance from each observation to its nearest projection with the local spacing of the projections. This section proves that, if the projections around an observation form a homogeneous Poisson process independent of it, the statistic E i E_i E i has a unit exponential distribution at every intensity. The association level compares the observed statistics with this reference, as App. § 9 .§ 0.1 specifies.
Lemma A7 (Exponential nearest-neighbor statistic). Let a 2 D 2D 2 D point be chosen independently of a homogeneous Poisson process of positive intensity on R 2 \mathbb{R}^{2} R 2 . For an integer ℓ ≥ 1 \ell\ge1 ℓ ≥ 1 , let d 1 ≤ ⋯ ≤ d ℓ + 1 d_{1}\le\cdots\le d_{\ell+1} d 1 ≤ ⋯ ≤ d ℓ + 1 be the distances from this point to its ℓ + 1 \ell+1 ℓ + 1 nearest points of the process. Then
E = − ℓ log ( 1 − d 1 2 / d ℓ + 1 2 ) ∼ Exp ( 1 ) . (26) E=-\ell\log(1-d_{1}^{2}/d_{\ell+1}^{2})\sim\operatorname{Exp}(1). \tag*{(26)} E = − ℓ log ( 1 − d 1 2 / d ℓ + 1 2 ) ∼ Exp ( 1 ) . (26) Proof. Conditional on d ℓ + 1 d_{\ell+1} d ℓ + 1 , the ℓ \ell ℓ nearer points of a homogeneous Poisson process are independent and uniform in the disk of radius d ℓ + 1 d_{\ell+1} d ℓ + 1 about the fixed point. The fraction of the area of that disk within distance d d d of its center is d 2 / d ℓ + 1 2 d^{2}/d_{\ell+1}^{2} d 2 / d ℓ + 1 2 , so the ratios d j 2 / d ℓ + 1 2 d_{j}^{2}/d_{\ell+1}^{2} d j 2 / d ℓ + 1 2 for 1 ≤ j ≤ ℓ 1\le j\le\ell 1 ≤ j ≤ ℓ are the order statistics of ℓ \ell ℓ independent uniform random variables on [ 0 , 1 ] [0,1] [ 0 , 1 ] , whatever the intensity. For x ≥ 0 x\ge0 x ≥ 0 , their minimum exceeds 1 − e − x / ℓ 1-e^{-x/\ell} 1 − e − x / ℓ with probability ( e − x / ℓ ) ℓ = e − x (e^{-x/\ell})^{\ell}=e^{-x} ( e − x / ℓ ) ℓ = e − x , and this event is { E > x } \{E>x\} { E > x } .
The lemma concerns one point chosen independently of the Poisson process. In MomentPnP, we estimate the pose from the observations, and the neighborhoods of different observations can share projections. Sums of these statistics therefore need not follow a Gamma distribution, and App. § 9 .§ 0.1 compares the association level with fixed thresholds.
Lemma A6 controls the contribution of outliers at the true pose. We now show that, under its mixture model with uniform outliers, finitely many of the method’s Fourier features determine the pose and the outlier fraction, and we bound the error caused by finitely many sampled outliers. The results concern the objective of Definition 1, before EM refinement. We first fix the setting and restate Theorem 2 in its notation.
Coordinates and features. We work in the box coordinates z z z of App. § 9 .§ 0.1 , defined by the image rectangle D F D_{F} D F of positive area. As defined there, the taper is W F ( z ) = W F , 1 ( z 1 ) W F , 2 ( z 2 ) W_{F}(z)=W_{F,1}(z_{1})W_{F,2}(z_{2}) W F ( z ) = W F , 1 ( z 1 ) W F , 2 ( z 2 ) . Each factor W F , r W_{F,r} W F , r vanishes outside the interval [ − h F , r − δ F , h F , r + δ F ] [-h_{F,r}-\delta_{F},h_{F,r}+\delta_{F}] [ − h F , r − δ F , h F , r + δ F ] given by the half-width h F , r h_{F,r} h F , r of D F D_{F} D F and the padding δ F \delta_{F} δ F , and it rises linearly over a distance τ F \tau_{F} τ F from each end. The weights are a k = exp ( − ζ F 2 ∥ k ∥ 2 / 2 ) a_{k}=\exp(-\zeta_{F}^{2}\|k\|^{2}/2) a k = exp ( − ζ F 2 ∥ k ∥ 2 /2 ) with spectral width ζ F \zeta_{F} ζ F , and F ( K F ) F(K_{F}) F ( K F ) contains one vector from each pair ± k \pm k ± k of nonzero vectors k ∈ π 2 Z 2 k\in\frac{\pi}{2}\mathbb{Z}^{2} k ∈ 2 π Z 2 with ∥ k ∥ ≤ π 2 K F \|k\|\le\frac{\pi}{2}K_{F} ∥ k ∥ ≤ 2 π K F . Throughout, D F D_{F} D F , the box normalization, the taper τ F \tau_{F} τ F and the weights a k a_{k} a k are fixed data. The method takes D F D_{F} D F to be the bounding rectangle of the calibrated observations. Holding it fixed matches the method when D F D_{F} D F is the bounding rectangle of the inlier observations, because image outliers drawn on it leave it unchanged. The same holds for the bounding box of the model points.
For a pose ξ \xi ξ , the map ρ ξ : R 3 → R 2 \rho_{\xi}:\mathbb{R}^{3}\to\mathbb{R}^{2} ρ ξ : R 3 → R 2 sends a model point at positive depth to its projection in box coordinates, that is, the pixel projection of Tab. A1 followed by calibration and the box normalization. In this section only, ρ j ( ξ ) = ρ ξ ( y j ) \rho_{j}(\xi)=\rho_{\xi}(y_{j}) ρ j ( ξ ) = ρ ξ ( y j ) denotes this box projection. For a cutoff K F K_{F} K F , let F ( K F ) F(K_{F}) F ( K F ) be the frequency set of App. § 9 .§ 0.1 , let B K F = ( ∑ k ∈ F ( K F ) a k 2 ) 1 / 2 B_{K_{F}}=(\sum_{k\in F(K_{F})}a_{k}^{2})^{1/2} B K F = ( ∑ k ∈ F ( K F ) a k 2 ) 1/2 , and let ϕ F \phi_{F} ϕ F be the feature map, read as a function of box coordinates, with n ϕ = 2 ∣ F ( K F ) ∣ n_{\phi}=2|F(K_{F})| n ϕ = 2∣ F ( K F ) ∣ real coordinates. The sets F ( K F ) F(K_{F}) F ( K F ) grow with K F K_{F} K F , every weight a k a_{k} a k is positive, and every nonzero k ∈ π 2 Z 2 k\in\frac{\pi}{2}\mathbb{Z}^{2} k ∈ 2 π Z 2 or its negative lies in F ( K F ) F(K_{F}) F ( K F ) once K F K_{F} K F is large enough. The spectral width ζ F \zeta_{F} ζ F , and hence every weight, stays fixed when K F K_{F} K F varies. The quantities below depend on K F K_{F} K F , and a subscript K F K_{F} K F marks this dependence where two cutoffs are compared.
Population model. Let y 1 , … , y m y_1,\ldots,y_m y 1 , … , y m be the inlier model points, and let B \mathcal{B} B be their bounding box, which has positive volume and is held fixed. Let U U U be the uniform distribution on D F D_F D F , expressed in box coordinates. At a pose ξ \xi ξ , the inlier projections have the empirical distribution P ξ i n P_\xi^{\mathrm{in}} P ξ in , which puts mass 1 / m 1/m 1/ m at each p j ( ξ ) p_j(\xi) p j ( ξ ) , and P ξ B P_\xi^{\mathcal{B}} P ξ B is the distribution of p ξ ( y ) p_\xi(y) p ξ ( y ) for y y y uniform in B \mathcal{B} B . Let E \mathcal{E} E be a compact set of poses that contains ξ ⋆ \xi^\star ξ ⋆ and under which every point of B \mathcal{B} B has positive depth, and let I ⊂ [ 0 , 1 ) \mathcal{I}\subset[0,1) I ⊂ [ 0 , 1 ) be a compact interval. A parameter is a pair θ = ( ξ , η ) ∈ E × I \theta=(\xi,\eta)\in\mathcal{E}\times\mathcal{I} θ = ( ξ , η ) ∈ E × I of a pose and an outlier fraction. For the true parameter θ ⋆ = ( ξ ⋆ , η ⋆ ) \theta^\star=(\xi^\star,\eta^\star) θ ⋆ = ( ξ ⋆ , η ⋆ ) with η ⋆ ∈ I \eta^\star\in\mathcal{I} η ⋆ ∈ I , we use the pose distance d Y d_Y d Y of Corollary A1 and put
d ( θ , θ ⋆ ) 2 = d Y ( ξ , ξ ⋆ ) 2 + ∣ η − η ⋆ ∣ 2 . d(\theta,\theta^\star)^2=d_Y(\xi,\xi^\star)^2+|\eta-\eta^\star|^2. d ( θ , θ ⋆ ) 2 = d Y ( ξ , ξ ⋆ ) 2 + ∣ η − η ⋆ ∣ 2 .
If both point sets contain the fraction η ⋆ \eta^\star η ⋆ of uniform outliers, the observations have the population distribution P o b s = ( 1 − η ⋆ ) P ξ ⋆ i n + η ⋆ U P^{\mathrm{obs}}=(1-\eta^\star)P_{\xi^\star}^{\mathrm{in}}+\eta^\star U P obs = ( 1 − η ⋆ ) P ξ ⋆ in + η ⋆ U , and the model points projected at ξ \xi ξ have the distribution ( 1 − η ⋆ ) P ξ i n + η ⋆ P ξ B (1-\eta^\star)P_\xi^{\mathrm{in}}+\eta^\star P_\xi^{\mathcal{B}} ( 1 − η ⋆ ) P ξ in + η ⋆ P ξ B . The population residual corresponding to Eq. (2) is therefore
r K F ( ξ , η ) = ( 1 − η ⋆ ) ( μ i n ( ξ ⋆ ) − μ i n ( ξ ) ) + ( η ⋆ − η ) ( μ U − μ B ( ξ ) ) , (27) r_{K_F}(\xi,\eta)=(1-\eta^\star)(\mu^{\mathrm{in}}(\xi^\star)-\mu^{\mathrm{in}}(\xi))+(\eta^\star-\eta)(\mu^U-\mu^{\mathcal{B}}(\xi)), \tag*{(27)} r K F ( ξ , η ) = ( 1 − η ⋆ ) ( μ in ( ξ ⋆ ) − μ in ( ξ )) + ( η ⋆ − η ) ( μ U − μ B ( ξ )) , (27)
where the feature means are μ i n ( ξ ) = ∫ ϕ F d P ξ i n \mu^{\mathrm{in}}(\xi)=\int\phi_F\,dP_\xi^{\mathrm{in}} μ in ( ξ ) = ∫ ϕ F d P ξ in for the inliers, μ U = ∫ ϕ F d U \mu^U=\int\phi_F\,dU μ U = ∫ ϕ F d U for the image outliers and μ B ( ξ ) = ∫ ϕ F d P ξ B \mu^{\mathcal{B}}(\xi)=\int\phi_F\,dP_\xi^{\mathcal{B}} μ B ( ξ ) = ∫ ϕ F d P ξ B for the model outliers. The residual vanishes at θ ⋆ \theta^\star θ ⋆ .
Geometric assumptions. We fix ξ ⋆ \xi^\star ξ ⋆ before choosing the inlier model of m ≥ 20 m\ge20 m ≥ 20 points. The model lies outside the exceptional set of Theorem 1 for N = m N=m N = m and outside the set on which the 2 m × 6 2m\times6 2 m × 6 matrix of the derivatives of the projections p j p_j p j at ξ ⋆ \xi^\star ξ ⋆ has rank less than six. Both sets have measure zero, the second by the paragraph on the rank condition in the proof of Theorem A2. The support of W F W_F W F lies strictly inside ( − 2 , 2 ) 2 (-2,2)^2 ( − 2 , 2 ) 2 , which holds for the taper of App. § 9 .§ 0.1 when h F , r + δ F < 2 h_{F,r}+\delta_F<2 h F , r + δ F < 2 for r = 1 , 2 r=1,2 r = 1 , 2 . The true projections z j ⋆ = p j ( ξ ⋆ ) z_j^\star=p_j(\xi^\star) z j ⋆ = p j ( ξ ⋆ ) are distinct, the taper is positive at each of them, and none lies on a line where a factor W F , r W_{F,r} W F , r changes slope, so W F W_F W F is differentiable at each of them. Finally, the two outlier distributions differ after weighting by W F W_F W F :
W F ( U − P ξ ⋆ B ) ≠ 0. (28) W_F(U-P_{\xi^\star}^{\mathcal{B}})\ne0. \tag*{(28)} W F ( U − P ξ ⋆ B ) = 0. (28)
Here W F μ W_F\mu W F μ denotes a measure μ \mu μ weighted by the function W F W_F W F . This condition makes the outlier fraction identifiable. Without it, the residual at the true pose stays zero when the fraction changes.
Finitely many outliers. The observations are n n n points with box coordinates z 1 , … , z n z_1,\ldots,z_n z 1 , … , z n , and the model has N N N points. For the analysis, we number the points so that observation i ≤ m i\le m i ≤ m images the inlier model point y i y_i y i , possibly with noise. The other n o u t = n − m n_{\mathrm{out}}=n-m n out = n − m observations are image outliers drawn uniformly on D F D_F D F , and the other N o u t = N − m N_{\mathrm{out}}=N-m N out = N − m model points are model outliers drawn uniformly in B \mathcal{B} B . All outliers are independent of each other and of the inlier observations, which we hold fixed. Either number of outliers may be zero. The outlier fractions are η U = n o u t / n \eta_U=n_{\mathrm{out}}/n η U = n out / n and η Y = N o u t / N \eta_Y=N_{\mathrm{out}}/N η Y = N out / N , and we set η ⋆ = η U \eta^\star=\eta_U η ⋆ = η U . With the midpoint quadratures μ U , 0 \mu_{U,0} μ U , 0 of μ U \mu^U μ U and μ Y , 0 ( ξ ) \mu_{Y,0}(\xi) μ Y , 0 ( ξ ) of μ B ( ξ ) \mu^{\mathcal{B}}(\xi) μ B ( ξ ) from App. § 9 .§ 0.1 , the empirical residual is the vector of Eq. (2) ,
r ^ K F ( θ ) = μ U , F − μ Y , F ( ξ ) − η ( μ U , 0 − μ Y , 0 ( ξ ) ) , \widehat r_{K_F}(\theta)=\mu_{U,F}-\mu_{Y,F}(\xi)-\eta(\mu_{U,0}-\mu_{Y,0}(\xi)), r K F ( θ ) = μ U , F − μ Y , F ( ξ ) − η ( μ U , 0 − μ Y , 0 ( ξ )) ,
so that ∥ r ^ K F ( θ ) ∥ 2 = L F ( ξ , η ) \|\widehat r_{K_F}(\theta)\|^2=\mathcal{L}_F(\xi,\eta) ∥ r K F ( θ ) ∥ 2 = L F ( ξ , η ) .
Error terms. The projection p ξ ( y ) p_\xi(y) p ξ ( y ) is smooth in ( ξ , y ) (\xi,y) ( ξ , y ) near the compact set E × B \mathcal{E}\times\mathcal{B} E × B , and ϕ F \phi_F ϕ F is Lipschitz, so there is a finite L ξ ≥ 0 L_\xi\ge0 L ξ ≥ 0 such that every y ∈ B y\in\mathcal{B} y ∈ B and all poses ξ , ξ ′ ∈ E \xi,\xi'\in\mathcal{E} ξ , ξ ′ ∈ E satisfy
∥ ϕ F ( p ξ ( y ) ) − ϕ F ( p ξ ′ ( y ) ) ∥ ≤ L ξ d Y ( ξ , ξ ′ ) . \|\phi_F(p_\xi(y))-\phi_F(p_{\xi'}(y))\|\le L_\xi d_Y(\xi,\xi'). ∥ ϕ F ( p ξ ( y )) − ϕ F ( p ξ ′ ( y )) ∥ ≤ L ξ d Y ( ξ , ξ ′ ) .
We fix ϵ n e t > 0 \epsilon_{\mathrm{net}}>0 ϵ net > 0 and N n e t N_{\mathrm{net}} N net poses of E \mathcal{E} E such that every pose of E \mathcal{E} E lies within d Y d_Y d Y -distance ϵ n e t \epsilon_{\mathrm{net}} ϵ net of one of them. For 0 < δ < 1 0<\delta<1 0 < δ < 1 , we put
Δ U = 2 log ( 4 n ϕ / δ ) n o u t , \Delta_U=2\sqrt{\frac{\log(4n_\phi/\delta)}{n_{\mathrm{out}}}}, Δ U = 2 n out log ( 4 n ϕ / δ ) ,
Δ Y = 2 log ( 4 n ϕ N n e t / δ ) N o u t + 2 L ξ ϵ n e t , \Delta_Y=2\sqrt{\frac{\log(4n_\phi N_{\mathrm{net}}/\delta)}{N_{\mathrm{out}}}}+2L_\xi\epsilon_{\mathrm{net}}, Δ Y = 2 N out log ( 4 n ϕ N net / δ ) + 2 L ξ ϵ net ,
Δ δ = η U Δ U + η Y Δ Y + 2 ∣ η U − η Y ∣ + η max ( q U + q Y ) + ϵ i m , (29) \Delta_\delta=\eta_U\Delta_U+\eta_Y\Delta_Y+2|\eta_U-\eta_Y|+\eta_{\max}(q_U+q_Y)+\epsilon_{\mathrm{im}}, \tag*{(29)} Δ δ = η U Δ U + η Y Δ Y + 2∣ η U − η Y ∣ + η m a x ( q U + q Y ) + ϵ im , (29)
where η max = max I \eta_{\max}=\max\mathcal{I} η m a x = max I . When n o u t = 0 n_{\mathrm{out}}=0 n out = 0 , we set η U Δ U = 0 \eta_U\Delta_U=0 η U Δ U = 0 , and likewise η Y Δ Y = 0 \eta_Y\Delta_Y=0 η Y Δ Y = 0 when N o u t = 0 N_{\mathrm{out}}=0 N out = 0 . The quadrature errors satisfy ∥ μ U , 0 − μ U ∥ ≤ q U \|\mu_{U,0}-\mu^U\|\le q_U ∥ μ U , 0 − μ U ∥ ≤ q U and ∥ μ Y , 0 ( ξ ) − μ B ( ξ ) ∥ ≤ q Y \|\mu_{Y,0}(\xi)-\mu^{\mathcal{B}}(\xi)\|\le q_Y ∥ μ Y , 0 ( ξ ) − μ B ( ξ ) ∥ ≤ q Y for every ξ ∈ E \xi\in\mathcal{E} ξ ∈ E . The image noise of the inliers enters through ϵ i m ≥ ∥ n − 1 ∑ i = 1 m ( ϕ F ( z i ) − ϕ F ( z i ⋆ ) ) ∥ \epsilon_{\mathrm{im}}\ge\|n^{-1}\sum_{i=1}^m(\phi_F(z_i)-\phi_F(z_i^\star))\| ϵ im ≥ ∥ n − 1 ∑ i = 1 m ( ϕ F ( z i ) − ϕ F ( z i ⋆ )) ∥ .
Theorem 2 (restated). Fix ξ ⋆ \xi^\star ξ ⋆ and m ≥ 20 m\ge20 m ≥ 20 , and let the inlier model lie outside the null set of the geometric assumptions. Assume the remaining geometric assumptions and the sampling model above, with η U ∈ I \eta_U\in\mathcal{I} η U ∈ I . There is a cutoff K 0 K_0 K 0 such that every cutoff K F ≥ K 0 K_F\ge K_0 K F ≥ K 0 admits a constant γ K F > 0 \gamma_{K_F}>0 γ K F > 0 with the following property. The constant depends on the inlier model, ξ ⋆ \xi^\star ξ ⋆ , η U \eta_U η U , E \mathcal{E} E , I \mathcal{I} I , D F D_F D F , B \mathcal{B} B , the taper, the weights and K F K_F K F , and not on the outlier positions, the noise or δ \delta δ . For every δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) and every net fixed before the draws, with probability at least 1 − δ 1-\delta 1 − δ over the outlier positions, every θ = ( ξ , η ) ∈ E × I \theta=(\xi,\eta)\in\mathcal{E}\times\mathcal{I} θ = ( ξ , η ) ∈ E × I satisfies
d ( θ , θ ⋆ ) ≤ L F ( ξ , η ) + Δ δ γ K F , θ ⋆ = ( ξ ⋆ , η U ) . d(\theta,\theta^\star)\le\frac{\sqrt{\mathcal{L}_F(\xi,\eta)+\Delta_\delta}}{\gamma_{K_F}},\qquad\theta^\star=(\xi^\star,\eta_U). d ( θ , θ ⋆ ) ≤ γ K F L F ( ξ , η ) + Δ δ , θ ⋆ = ( ξ ⋆ , η U ) . Here L F \mathcal{L}_F L F uses the cutoff K F K_F K F , and η U \eta_U η U is the fraction of image outliers.
Theorem A2 provides K 0 K_0 K 0 and γ K F \gamma_{K_F} γ K F for the population residual, and Theorem A3 gives the bound with probability at least 1 − δ 1-\delta 1 − δ . The constant C F C_F C F of the main text is 1 / γ K F 1/\gamma_{K_F} 1/ γ K F , its distance dist \operatorname{dist} dist is d ( θ , θ ⋆ ) d(\theta,\theta^\star) d ( θ , θ ⋆ ) , with translations in units of ℓ Y \ell_Y ℓ Y , its error term is Δ δ \Delta_\delta Δ δ of Eq. (29) , and its compact set Θ \Theta Θ lies in such a product E × I \mathcal{E} \times\mathcal{I} E × I of compact sets of poses and fractions. Only Δ U \Delta_U Δ U and Δ Y \Delta_Y Δ Y depend on δ \delta δ . The minimizer bound of the main text is the case ϵ f i t = 0 \epsilon_{\mathrm{fit}}=0 ϵ fit = 0 of Theorem A3.
Theorem A2 (Finite Fourier identification). Under the geometric assumptions, let η ⋆ ∈ I \eta^\star\in\mathcal{I} η ⋆ ∈ I . There is a finite cutoff K 0 K_0 K 0 such that every cutoff K F ≥ K 0 K_F \ge K_0 K F ≥ K 0 admits a constant γ K F > 0 \gamma_{K_F} > 0 γ K F > 0 satisfying
∥ r K F ( θ ) ∥ ≥ γ K F d ( θ , θ ⋆ ) , θ ∈ E × I . (30) \lVert r_{K_F}(\theta) \rVert\ge\gamma_{K_F} d(\theta,\theta^\star), \qquad\theta\in\mathcal{E} \times\mathcal{I}. \tag*{(30)} ∥ r K F ( θ )∥ ≥ γ K F d ( θ , θ ⋆ ) , θ ∈ E × I . (30) Consequently, θ ⋆ \theta^\star θ ⋆ is the unique zero of r K F r_{K_F} r K F on E × I \mathcal{E} \times\mathcal{I} E × I . Both K 0 K_0 K 0 and γ K F \gamma_{K_F} γ K F may depend on all fixed data, including η ⋆ \eta^\star η ⋆ .
Proof. The proof first identifies the parameter from all Fourier coefficients and then shows that finitely many suffice, both near the truth and away from it.
Fourier coefficients. The square ( − 2 , 2 ) 2 (-2,2)^2 ( − 2 , 2 ) 2 is a fundamental cell of the torus T = R 2 / 4 Z 2 \mathbb{T}=\mathbb{R}^2/4\mathbb{Z}^2 T = R 2 /4 Z 2 , so we view measures supported inside it as measures on T \mathbb{T} T . The characters of T \mathbb{T} T are χ k ( z ) = exp ( i k T z ) \chi_k(z)=\exp(\mathrm{i}k^\mathsf{T}z) χ k ( z ) = exp ( i k T z ) for k ∈ π 2 Z 2 k\in\frac{\pi}{2}\mathbb{Z}^2 k ∈ 2 π Z 2 , and a finite signed measure ν \nu ν has the coefficients ν ^ ( k ) = ∫ χ k d ν \widehat{\nu}(k)=\int\chi_k\,\mathrm{d}\nu ν ( k ) = ∫ χ k d ν . For θ ∈ E × I \theta\in\mathcal{E}\times\mathcal{I} θ ∈ E × I , we define the signed measure
ν θ = W F [ ( 1 − η ⋆ ) P ξ i n + ( η ⋆ − η ) P ξ B + η U ] , \nu_\theta=W_F\left[(1-\eta^\star)P_\xi^{\mathrm{in}}+(\eta^\star-\eta)P_\xi^B+\eta U\right], ν θ = W F [ ( 1 − η ⋆ ) P ξ in + ( η ⋆ − η ) P ξ B + η U ] ,
which is supported in the support of W F W_F W F , and ν θ ⋆ = W F P o b s \nu_{\theta^\star}=W_F P^{\mathrm{obs}} ν θ ⋆ = W F P obs . By Eq. (27) , r K F ( θ ) r_{K_F}(\theta) r K F ( θ ) is the integral of ϕ F \phi_F ϕ F against P o b s P^{\mathrm{obs}} P obs minus its integral against ( 1 − η ⋆ ) P ξ i n + ( η ⋆ − η ) P ξ B + η U (1-\eta^\star)P_\xi^{\mathrm{in}}+(\eta^\star-\eta)P_\xi^B+\eta U ( 1 − η ⋆ ) P ξ in + ( η ⋆ − η ) P ξ B + η U . For every measure μ \mu μ , the integral ∫ ϕ F d μ \int\phi_F\,\mathrm{d}\mu ∫ ϕ F d μ lists the real and imaginary parts of W F μ ^ ( k ) \widehat{W_F\mu}(k) W F μ ( k ) for k ∈ F ( K F ) k\in F(K_F) k ∈ F ( K F ) , each multiplied by a k / B K F a_k/B_{K_F} a k / B K F . Hence r K F ( θ ) r_{K_F}(\theta) r K F ( θ ) lists the scaled real and imaginary parts of W F P o b s ^ ( k ) − ν θ ^ ( k ) \widehat{W_F P^{\mathrm{obs}}}(k)-\widehat{\nu_\theta}(k) W F P obs ( k ) − ν θ ( k ) . Real measures satisfy ν ^ ( − k ) = ν ^ ( k ) ‾ \widehat{\nu}(-k)=\overline{\widehat{\nu}(k)} ν ( − k ) = ν ( k ) , so r K F ( θ ) = 0 r_{K_F}(\theta)=0 r K F ( θ ) = 0 for every cutoff exactly when ν θ ^ ( k ) = W F P o b s ^ ( k ) \widehat{\nu_\theta}(k)=\widehat{W_F P^{\mathrm{obs}}}(k) ν θ ( k ) = W F P obs ( k ) for every nonzero k k k .
Identification from all coefficients. Suppose that these coefficients agree. A finite signed measure on T \mathbb{T} T whose coefficients vanish at every nonzero frequency is a constant multiple of the uniform measure [38 ] . The difference ν θ − W F P o b s \nu_\theta-W_F P^{\mathrm{obs}} ν θ − W F P obs vanishes on the complement of the support of W F W_F W F , a nonempty open subset of T \mathbb{T} T , so the multiple is zero and ν θ = W F P o b s \nu_\theta=W_F P^{\mathrm{obs}} ν θ = W F P obs . The measures U U U and P ξ B P_\xi^B P ξ B have densities. For P ξ B P_\xi^B P ξ B , the change of camera coordinates ( u 1 , u 2 , x 3 ) ↦ ( x 3 u 1 , x 3 u 2 , x 3 ) (u_1,u_2,x_3)\mapsto(x_3u_1,x_3u_2,x_3) ( u 1 , u 2 , x 3 ) ↦ ( x 3 u 1 , x 3 u 2 , x 3 ) has volume element x 3 2 d u 1 d u 2 d x 3 x_3^2\,\mathrm{d}u_1\,\mathrm{d}u_2\,\mathrm{d}x_3 x 3 2 d u 1 d u 2 d x 3 , integration over positive depth gives a density of the calibrated projection, and the invertible box normalization preserves it. The atomic parts of the two sides therefore coincide. After division by ( 1 − η ⋆ ) / m > 0 (1-\eta^\star)/m>0 ( 1 − η ⋆ ) / m > 0 , the atomic part on the left puts mass W F ( p j ( ξ ) ) W_F(p_j(\xi)) W F ( p j ( ξ )) at each p j ( ξ ) p_j(\xi) p j ( ξ ) , with the masses of coinciding points added. The atomic part on the right puts mass W F ( z j ⋆ ) > 0 W_F(z_j^\star)>0 W F ( z j ⋆ ) > 0 at each of the m m m distinct points z j ⋆ z_j^\star z j ⋆ . The left side has at most m m m points of positive mass, so the p j ( ξ ) p_j(\xi) p j ( ξ ) are distinct, each lies where W F W_F W F is positive, and they are the points z j ⋆ z_j^\star z j ⋆ . Thus P ξ i n = P ξ ⋆ i n P_\xi^{\mathrm{in}}=P_{\xi^\star}^{\mathrm{in}} P ξ in = P ξ ⋆ in , so the inlier projections agree as multisets in calibrated coordinates as well. Theorem 1, applied to the m m m inlier points, which have positive depth under ξ \xi ξ and ξ ⋆ \xi^\star ξ ⋆ , gives ξ = ξ ⋆ \xi=\xi^\star ξ = ξ ⋆ . Then ν θ − W F P o b s = ( η ⋆ − η ) W F ( P ξ ⋆ B − U ) \nu_\theta-W_F P^{\mathrm{obs}}=(\eta^\star-\eta)W_F(P_{\xi^\star}^B-U) ν θ − W F P obs = ( η ⋆ − η ) W F ( P ξ ⋆ B − U ) , and Eq. (28) gives η = η ⋆ \eta=\eta^\star η = η ⋆ . Every point with positive mass lies in the same fundamental cell, so no point is confused with a translate by a period.
Derivative at the truth. We use local pose coordinates ( ω , τ ) ∈ R 6 (\omega,\tau)\in\mathbb{R}^6 ( ω , τ ) ∈ R 6 for the poses obtained from ξ ⋆ \xi^\star ξ ⋆ by the camera motion x ↦ exp ( [ ω ] × ) x + τ x\mapsto\exp([\omega]_\times)x+\tau x ↦ exp ([ ω ] × ) x + τ , and write a direction as θ ˙ = ( ξ ˙ , η ˙ ) ∈ R 6 × R \dot{\theta}=(\dot{\xi},\dot{\eta})\in\mathbb{R}^6\times\mathbb{R} θ ˙ = ( ξ ˙ , η ˙ ) ∈ R 6 × R . Let ∂ p j ∈ R 2 × 6 \partial p_j\in\mathbb{R}^{2\times6} ∂ p j ∈ R 2 × 6 be the derivative of p j p_j p j at ξ ⋆ \xi^\star ξ ⋆ in these coordinates. For a smooth function f f f on T \mathbb{T} T , let F ( f ) = ∫ f d ν θ F(f)=\int f\,\mathrm{d}\nu_\theta F ( f ) = ∫ f d ν θ . Its atomic term is differentiable at θ ⋆ \theta^\star θ ⋆ because W F W_F W F is differentiable at each z j ⋆ z_j^\star z j ⋆ , and its uniform term is linear in η \eta η . Its box term ( η ⋆ − η ) ∫ W F f d P ξ B (\eta^\star-\eta)\int W_F f\,\mathrm{d}P_\xi^B ( η ⋆ − η ) ∫ W F f d P ξ B is the product of a factor that vanishes at θ ⋆ \theta^\star θ ⋆ and a function of ξ \xi ξ that is continuous by dominated convergence, because W F f W_F f W F f is bounded and continuous and depths stay positive on E \mathcal{E} E . This term is therefore differentiable at θ ⋆ \theta^\star θ ⋆ with derivative − η ˙ ∫ W F f d P ξ ⋆ B -\dot{\eta}\int W_F f\,\mathrm{d}P_{\xi^\star}^B − η ˙ ∫ W F f d P ξ ⋆ B , without differentiating the box integral. With ν o u t = W F ( U − P ξ ⋆ B ) \nu^{\mathrm{out}}=W_F(U-P_{\xi^\star}^B) ν out = W F ( U − P ξ ⋆ B ) , the derivative of F f F_f F f at θ ⋆ \theta^\star θ ⋆ in the direction θ ˙ \dot{\theta} θ ˙ is
Y θ ( f ) = 1 − η ⋆ m ∑ j = 1 m ∇ ( W F f ) ( z j ⋆ ) T ∂ p j ξ ˙ + η ˙ ∫ f d ν o u t . (31) \mathcal{Y}_\theta(f)=\frac{1-\eta^\star}{m}\sum_{j=1}^{m}\nabla(W_Ff)(z_j^\star)^\mathsf{T}\partial p_j\dot{\xi}+\dot{\eta}\int f\,\mathrm{d}\nu^{\mathrm{out}}. \tag*{(31)} Y θ ( f ) = m 1 − η ⋆ j = 1 ∑ m ∇ ( W F f ) ( z j ⋆ ) T ∂ p j ξ ˙ + η ˙ ∫ f d ν out . (31)
The same argument shows that r K F r_{K_F} r K F is differentiable at θ ⋆ \theta^\star θ ⋆ . Its derivative in the direction θ ˙ \dot{\theta} θ ˙ lists, with the opposite sign and the factors a k / B K F a_k/B_{K_F} a k / B K F , the real and imaginary parts of Y θ ⋆ ( χ k ) \mathcal{Y}_{\theta^\star}(\chi_k) Y θ ⋆ ( χ k ) for k ∈ F ( K F ) k\in F(K_F) k ∈ F ( K F ) .
Injectivity. Suppose that Y θ ⋆ ( χ k ) = 0 \mathcal{Y}_{\theta^\star}(\chi_k)=0 Y θ ⋆ ( χ k ) = 0 for every nonzero k k k . Linearity then gives Y θ ⋆ ( f ) = 0 \mathcal{Y}_{\theta^\star}(f)=0 Y θ ⋆ ( f ) = 0 for every trigonometric polynomial f f f , where ⟨ f ⟩ = 1 16 ∫ ( − 2 , 2 ) 2 f ( z ) d z \langle f\rangle=\frac{1}{16}\int_{(-2,2)^2}f(z)\,\mathrm{d}z ⟨ f ⟩ = 16 1 ∫ ( − 2 , 2 ) 2 f ( z ) d z is the mean over a period. To remove the atomic terms, we use a trigonometric polynomial that vanishes together with its gradient at every true projection. On T \mathbb{T} T , let
b j ( z ) = 2 − cos ( π 2 ( z 1 − z j 1 ⋆ ) ) − cos ( π 2 ( z 2 − z j 2 ⋆ ) ) , b_j(z)=2-\cos\left(\frac{\pi}{2}(z_1-z_{j1}^\star)\right)-\cos\left(\frac{\pi}{2}(z_2-z_{j2}^\star)\right), b j ( z ) = 2 − cos ( 2 π ( z 1 − z j 1 ⋆ ) ) − cos ( 2 π ( z 2 − z j 2 ⋆ ) ) ,
f 0 ( z ) = ∏ j = 1 m b j ( z ) 2 . f_0(z)=\prod_{j=1}^{m}b_j(z)^2. f 0 ( z ) = j = 1 ∏ m b j ( z ) 2 .
Each b j b_j b j is nonnegative and vanishes on T \mathbb{T} T exactly at z j ⋆ z_j^\star z j ⋆ , so f 0 f_0 f 0 is nonnegative and vanishes, together with its gradient, exactly at the true projections. For every trigonometric polynomial f f f , the product f 0 f f_0f f 0 f also vanishes with its gradient there, so the atomic terms of Υ ˙ θ ( f 0 f ) \dot{\Upsilon}_{\theta}(f_0f) Υ ˙ θ ( f 0 f ) vanish. Hence η ˙ ∫ f 0 f d ν o u t = Υ ˙ θ ( 1 ) ⟨ f 0 f ⟩ \dot{\eta}\int f_0f\,d\nu^{\mathrm{out}}=\dot{\Upsilon}_{\theta}(1)\langle f_0f\rangle η ˙ ∫ f 0 f d ν out = Υ ˙ θ ( 1 ) ⟨ f 0 f ⟩ for every trigonometric polynomial f f f . The finite signed measures η ˙ f 0 ν o u t \dot{\eta}f_0\nu^{\mathrm{out}} η ˙ f 0 ν out and 1 16 Υ ˙ θ ( 1 ) f 0 d z \frac{1}{16}\dot{\Upsilon}_{\theta}(1)f_0\,dz 16 1 Υ ˙ θ ( 1 ) f 0 d z therefore have the same coefficients and are equal [38 ] . Outside the support of W F W_F W F , the measure ν o u t \nu^{\mathrm{out}} ν out vanishes and f 0 f_0 f 0 is positive, so Υ ˙ θ ( 1 ) = 0 \dot{\Upsilon}_{\theta}(1)=0 Υ ˙ θ ( 1 ) = 0 . Hence η ˙ f 0 ν o u t = 0 \dot{\eta}f_0\nu^{\mathrm{out}}=0 η ˙ f 0 ν out = 0 . The measure ν o u t \nu^{\mathrm{out}} ν out has a density, which is nonzero on a set of positive measure by Eq. (28) , and f 0 f_0 f 0 vanishes only at finitely many points. The density of f 0 ν o u t f_0\nu^{\mathrm{out}} f 0 ν out is therefore nonzero on a set of positive measure, and η ˙ = 0 \dot{\eta}=0 η ˙ = 0 .
It remains to show that ξ ˙ = 0 \dot{\xi}=0 ξ ˙ = 0 . We now have Υ ˙ θ ( f ) = 0 \dot{\Upsilon}_{\theta}(f)=0 Υ ˙ θ ( f ) = 0 for every trigonometric polynomial f f f , and only the atomic terms remain. For a point index i i i and a coordinate a ∈ { 1 , 2 } a\in\{1,2\} a ∈ { 1 , 2 } , let
f i , a ( z ) = sin ( π 2 ( z a − z i , a ⋆ ) ) ∏ j ≠ i b j ( z ) 2 . f_{i,a}(z)=\sin\left(\frac{\pi}{2}(z_a-z_{i,a}^{\star})\right)\prod_{j\ne i}b_j(z)^2. f i , a ( z ) = sin ( 2 π ( z a − z i , a ⋆ ) ) j = i ∏ b j ( z ) 2 .
This function vanishes at every true projection, its gradient vanishes at z j ⋆ z_j^\star z j ⋆ for j ≠ i j\ne i j = i , and its gradient at z i ⋆ z_i^\star z i ⋆ is π 2 ∏ j ≠ i b j ( z i ⋆ ) 2 \frac{\pi}{2}\prod_{j\ne i}b_j(z_i^\star)^2 2 π ∏ j = i b j ( z i ⋆ ) 2 times the a a a th coordinate vector. This factor is positive because the true projections are distinct points of the cell. Hence Υ ˙ θ ( f i , a ) \dot{\Upsilon}_{\theta}(f_{i,a}) Υ ˙ θ ( f i , a ) equals ( 1 − η ⋆ ) W F ( z i ⋆ ) / m (1-\eta^\star)W_F(z_i^\star)/m ( 1 − η ⋆ ) W F ( z i ⋆ ) / m times this factor times ( ∂ p i ξ ˙ ) a (\partial_{p_i}\dot{\xi})_a ( ∂ p i ξ ˙ ) a , and all factors except the last are positive. Thus ( ∂ p i ξ ˙ ) a = 0 (\partial_{p_i}\dot{\xi})_a=0 ( ∂ p i ξ ˙ ) a = 0 for every i i i and a a a , and the stacked derivatives have rank six, which gives ξ ˙ = 0 \dot{\xi}=0 ξ ˙ = 0 .
The complex linear functionals θ ˙ ↦ Υ ˙ θ ( χ k ) \dot{\theta}\mapsto\dot{\Upsilon}_{\theta}(\chi_k) θ ˙ ↦ Υ ˙ θ ( χ k ) on R 7 \mathbb{R}^7 R 7 , for nonzero k k k , therefore have no common nonzero kernel vector. Their kernels are subspaces of R 7 \mathbb{R}^7 R 7 with zero intersection, so finitely many of them already intersect in zero. Let K 1 K_1 K 1 be a cutoff for which F ( K 1 ) F(K_1) F ( K 1 ) contains each of these frequencies or its negative. The derivative of r K 1 r_{K_1} r K 1 at θ ⋆ \theta^\star θ ⋆ is then injective, with least singular value σ 1 > 0 \sigma_1>0 σ 1 > 0 in the coordinates ( ω , τ , η ) (\omega,\tau,\eta) ( ω , τ , η ) .
Rank condition. The rank condition on the projections is generic. At the identity camera, with u = x 1 : 2 / x 3 u=x_{1:2}/x_3 u = x 1 : 2 / x 3 for a camera point x x x , the derivatives with respect to ( ω , τ ) (\omega,\tau) ( ω , τ ) are
∂ u 1 = ( − u 1 u 2 , 1 + u 1 2 , − u 2 , x 3 − 1 , 0 , − u 1 / x 3 ) , \partial u_1=(-u_1u_2,1+u_1^2,-u_2,x_3^{-1},0,-u_1/x_3), ∂ u 1 = ( − u 1 u 2 , 1 + u 1 2 , − u 2 , x 3 − 1 , 0 , − u 1 / x 3 ) ,
∂ u 2 = ( − ( 1 + u 2 2 ) , u 1 u 2 , u 1 , 0 , x 3 − 1 , − u 2 / x 3 ) . \partial u_2=(-(1+u_2^2),u_1u_2,u_1,0,x_3^{-1},-u_2/x_3). ∂ u 2 = ( − ( 1 + u 2 2 ) , u 1 u 2 , u 1 , 0 , x 3 − 1 , − u 2 / x 3 ) .
For the camera points ( 0 , 0 , 1 ) (0,0,1) ( 0 , 0 , 1 ) , ( 2 , 0 , 2 ) (2,0,2) ( 2 , 0 , 2 ) and ( 0 , 3 , 3 ) (0,3,3) ( 0 , 3 , 3 ) , the six rows form a matrix with determinant 29 / 36 29/36 29/36 . The 6 × 6 6\times6 6 × 6 minor formed by the rows of three points is a rational function of their coordinates whose denominator is a power of their positive depths. Its numerator is a nonzero polynomial, so it vanishes only on a set of measure zero in R 9 \mathbb{R}^9 R 9 . Rank less than six forces this minor to vanish, so the configurations of m ≥ 3 m\ge3 m ≥ 3 points with rank less than six form a null set. Expressing the points in the camera coordinates of ξ ⋆ \xi^\star ξ ⋆ is an invertible affine change of the model coordinates, and the box normalization multiplies each ∂ p j \partial p_j ∂ p j by the same invertible matrix. The exceptional set of inlier models is therefore null as well.
Local bound. Differentiability at θ ⋆ \theta^\star θ ⋆ gives ∥ r K 1 ( θ ) ∥ ≥ 1 2 σ 1 ∣ θ ∣ \lVert r_{K_1}(\theta)\rVert\ge\frac{1}{2}\sigma_1\lvert\theta\rvert ∥ r K 1 ( θ )∥ ≥ 2 1 σ 1 ∣ θ ∣ for every θ \theta θ in a neighborhood of θ ⋆ \theta^\star θ ⋆ with coordinates θ ˙ \dot{\theta} θ ˙ . The chart is smooth, so d ( θ , θ ⋆ ) ≤ c 1 ∣ θ ∣ d(\theta,\theta^\star)\le c_1\lvert\theta\rvert d ( θ , θ ⋆ ) ≤ c 1 ∣ θ ∣ near θ ⋆ \theta^\star θ ⋆ for a constant c 1 c_1 c 1 . There is therefore an open neighborhood O 1 \mathcal{O}_1 O 1 of θ ⋆ \theta^\star θ ⋆ in E × I \mathcal{E}\times\mathcal{I} E × I on which ∥ r K 1 ( θ ) ∥ ≥ γ 1 d ( θ , θ ⋆ ) \lVert r_{K_1}(\theta)\rVert\ge\gamma_1d(\theta,\theta^\star) ∥ r K 1 ( θ )∥ ≥ γ 1 d ( θ , θ ⋆ ) with γ 1 = σ 1 / ( 2 c 1 ) \gamma_1=\sigma_1/(2c_1) γ 1 = σ 1 / ( 2 c 1 ) .
Away from the truth. The set O 1 c = ( E × I ) ∖ O 1 \mathcal{O}_1^c=(\mathcal{E}\times\mathcal{I})\setminus\mathcal{O}_1 O 1 c = ( E × I ) ∖ O 1 is compact. For each θ ∈ O 1 c \theta\in\mathcal{O}_1^c θ ∈ O 1 c , the identification step gives a nonzero frequency k θ k_\theta k θ with ν ^ θ ( k θ ) ≠ W F p o s ^ ( k θ ) \widehat{\nu}_{\theta}(k_\theta)\ne\widehat{W_F^{\mathrm{pos}}}(k_\theta) ν θ ( k θ ) = W F pos ( k θ ) . The coefficient ν ^ θ ( k ) \hat{\nu}_{\theta}(k) ν ^ θ ( k ) is continuous in θ \theta θ , the atomic part because W F χ k W_F\chi_k W F χ k is continuous and the box part by dominated convergence. The inequality therefore holds on an open neighborhood of θ \theta θ , and finitely many of these neighborhoods cover O 1 c \mathcal{O}_1^c O 1 c . Let K 0 ≥ K 1 K_0\ge K_1 K 0 ≥ K 1 be a cutoff for which F ( K 0 ) F(K_0) F ( K 0 ) contains each of the selected frequencies or its negative.
Every cutoff K F ≥ K 0 K_F\ge K_0 K F ≥ K 0 . The sets satisfy F ( K 1 ) ⊂ F ( K F ) F(K_1)\subset F(K_F) F ( K 1 ) ⊂ F ( K F ) , and only the normalizing constant changes, so the coordinates of r K F r_{K_F} r K F include those of r K 1 r_{K_1} r K 1 multiplied by B K 1 / B K F > 0 B_{K_1}/B_{K_F}>0 B K 1 / B K F > 0 . Hence ∥ r K F ( θ ) ∥ ≥ ( B K 1 / B K F ) γ 1 d ( θ , θ ⋆ ) \lVert r_{K_F}(\theta)\rVert\ge(B_{K_1}/B_{K_F})\gamma_1d(\theta,\theta^\star) ∥ r K F ( θ )∥ ≥ ( B K 1 / B K F ) γ 1 d ( θ , θ ⋆ ) on O 1 \mathcal{O}_1 O 1 . If O 1 c \mathcal{O}_1^c O 1 c is empty, this bound proves Eq. (30) . Otherwise, r K F r_{K_F} r K F is continuous on O 1 c \mathcal{O}_1^c O 1 c , and it is nonzero there because F ( K F ) F(K_F) F ( K F ) contains each selected frequency or its negative, so its norm has a positive minimum r K F r_{K_F} r K F there. With d max > 0 d_{\max}>0 d m a x > 0 the largest value of d ( θ , θ ⋆ ) d(\theta,\theta^\star) d ( θ , θ ⋆ ) on E × I \mathcal{E}\times\mathcal{I} E × I , every θ ∈ O 1 c \theta\in\mathcal{O}_1^c θ ∈ O 1 c satisfies ∥ r K F ( θ ) ∥ ≥ ( r K F / d max ) d ( θ , θ ⋆ ) \lVert r_{K_F}(\theta)\rVert\ge(r_{K_F}/d_{\max})d(\theta,\theta^\star) ∥ r K F ( θ )∥ ≥ ( r K F / d m a x ) d ( θ , θ ⋆ ) . Taking γ K F \gamma_{K_F} γ K F as the smaller of the two constants proves Eq. (30) . Because d d d is a metric, r K F ( θ ) = 0 r_{K_F}(\theta)=0 r K F ( θ ) = 0 forces θ = θ ⋆ \theta=\theta^\star θ = θ ⋆ . □
Theorem A3 (Finite-sample Fourier pose bound). Assume the setting above with η ⋆ = η U ∈ I \eta^\star=\eta_U\in\mathcal{I} η ⋆ = η U ∈ I , and let K F ≥ K 0 K_F\ge K_0 K F ≥ K 0 and γ K F \gamma_{K_F} γ K F be as in Theorem A2. Fix ϵ n e t > 0 \epsilon_{\mathrm{net}}>0 ϵ net > 0 , a net of N n e t N_{\mathrm{net}} N net poses as above, and δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) . With probability at least 1 − δ 1-\delta 1 − δ over the outlier draws, the empirical residual satisfies, simultaneously for every θ ∈ E × I \theta\in\mathcal{E}\times\mathcal{I} θ ∈ E × I ,
d ( θ , θ ⋆ ) ≤ ∥ r ^ K F ( θ ) ∥ + Δ δ γ K F . (32) d(\theta,\theta^\star)\le\frac{\lVert\hat{r}_{K_F}(\theta)\rVert+\Delta_\delta}{\gamma_{K_F}}. \tag*{(32)} d ( θ , θ ⋆ ) ≤ γ K F ∥ r ^ K F ( θ )∥ + Δ δ . (32) On the same event, every parameter θ \theta θ whose residual norm exceeds the infimum over E × I \mathcal{E}\times\mathcal{I} E × I by at most ϵ f i t ≥ 0 \epsilon_{\mathrm{fit}}\ge0 ϵ fit ≥ 0 satisfies d ( θ ^ , θ ⋆ ) ≤ ( 2 Δ δ + ϵ f i t ) / γ K F d(\hat{\theta},\theta^\star)\le(2\Delta_\delta+\epsilon_{\mathrm{fit}})/\gamma_{K_F} d ( θ ^ , θ ⋆ ) ≤ ( 2 Δ δ + ϵ fit ) / γ K F .
Proof. Each real coordinate of ϕ F \phi_F ϕ F belongs to one frequency k k k and lies in [ − a k / B K F , a k / B K F ] [-a_k/B_{K_F},a_k/B_{K_F}] [ − a k / B K F , a k / B K F ] , because the taper is at most one. Each frequency contributes a cosine and a sine with the same amplitude, so the squared amplitudes of all coordinates sum to 2 2 2 . Let X 1 , … , X n s X_1,\ldots,X_{n_s} X 1 , … , X n s be the values of one coordinate, with amplitude a k / B K F a_k/B_{K_F} a k / B K F , at n s n_s n s independent outliers, and let X ˉ \bar{X} X ˉ be their mean. For α > 0 \alpha>0 α > 0 , Hoeffding’s inequality [33 ] gives
Pr { ∣ X ˉ − E X 1 ∣ > α a k / B K F } ≤ 2 e − n s α 2 / 2 . \Pr\{|\bar{X}-\mathbb{E}X_1|>\alpha a_k/B_{K_F}\}\le2e^{-n_s\alpha^2/2}. Pr { ∣ X ˉ − E X 1 ∣ > α a k / B K F } ≤ 2 e − n s α 2 /2 . When every coordinate satisfies the opposite inequality, the squared norm of the vector error is at most 2 α 2 2\alpha^2 2 α 2 . Let μ ^ U \hat{\mu}^{U} μ ^ U
be the average of ϕ F \phi_F ϕ F over the image outliers and μ ^ B ( ξ ) \hat{\mu}^{B}(\xi) μ ^ B ( ξ ) the average of ϕ F ∘ p ξ \phi_F \circ p_{\xi} ϕ F ∘ p ξ over the model outliers. A union bound over the n ϕ n_{\phi} n ϕ coordinates, with 2 n ϕ e − n o u t α 2 / 2 = δ / 2 2n_{\phi}e^{-n_{\mathrm{out}}\alpha^2/2}=\delta/2 2 n ϕ e − n out α 2 /2 = δ /2 , gives ∥ μ ^ U − μ U ∥ ≤ 2 α = Δ U \lVert\hat{\mu}^{U}-\mu^{U}\rVert\le\sqrt{2}\alpha=\Delta_U ∥ μ ^ U − μ U ∥ ≤ 2 α = Δ U with probability at least 1 − δ / 2 1-\delta/2 1 − δ /2 . A union bound over the n ϕ n_{\phi} n ϕ coordinates and the N n e t N_{\mathrm{net}} N net net poses gives ∥ μ ^ B ( ξ ′ ) − μ B ( ξ ′ ) ∥ ≤ Δ Y − 2 L ξ ϵ n e t \lVert\hat{\mu}^{B}(\xi')-\mu^{B}(\xi')\rVert\le\Delta_Y-2L_{\xi}\epsilon_{\mathrm{net}} ∥ μ ^ B ( ξ ′ ) − μ B ( ξ ′ )∥ ≤ Δ Y − 2 L ξ ϵ net at every net pose ξ ′ \xi' ξ ′ with probability at least 1 − δ / 2 1-\delta/2 1 − δ /2 . The same model-outlier sample is used at every pose, and the union bound needs no independence between coordinates or between net poses. Both μ ^ B \hat{\mu}^{B} μ ^ B and μ B \mu^{B} μ B are L ξ L_{\xi} L ξ -Lipschitz in the pose, so moving from the nearest net pose to any ξ ∈ E \xi\in\mathcal{E} ξ ∈ E adds at most 2 L ξ ϵ n e t 2L_{\xi}\epsilon_{\mathrm{net}} 2 L ξ ϵ net . Both events hold with probability at least 1 − δ 1-\delta 1 − δ . An event is not needed when the corresponding number of outliers is zero.
On these events, we compare r ^ K F \hat{r}_{K_F} r ^ K F with r K F r_{K_F} r K F at η ⋆ = η U \eta^\star=\eta_U η ⋆ = η U . We split the feature average over the observations into the contributions of the inliers and the image outliers, and the feature average over the model points into those of the inliers and the model outliers. Because 1 − η U = m / n 1-\eta_U=m/n 1 − η U = m / n and 1 − η Y = m / N 1-\eta_Y=m/N 1 − η Y = m / N , this gives
r ^ K F ( θ ) − r K F ( θ ) = 1 n ∑ i = 1 m ( ϕ F ( z i ) − ϕ F ( z i ⋆ ) ) + η U ( μ ^ U − μ U ) − η Y ( μ ^ B ( ξ ) − μ B ( ξ ) ) + ( η Y − η U ) ( μ i n ( ξ ) − μ B ( ξ ) ) − η [ ( μ U , 0 − μ U ) − ( μ Y , 0 ( ξ ) − μ B ( ξ ) ) ] . \hat{r}_{K_F}(\theta)-r_{K_F}(\theta)=\frac{1}{n}\sum_{i=1}^{m}\left(\phi_F(z_i)-\phi_F(z_i^\star)\right)+\eta_U\left(\hat{\mu}^{U}-\mu^{U}\right)-\eta_Y\left(\hat{\mu}^{B}(\xi)-\mu^{B}(\xi)\right)+(\eta_Y-\eta_U)\left(\mu^{\mathrm{in}}(\xi)-\mu^{B}(\xi)\right)-\eta\left[(\mu_{U,0}-\mu^{U})-(\mu_{Y,0}(\xi)-\mu^{B}(\xi))\right]. r ^ K F ( θ ) − r K F ( θ ) = n 1 i = 1 ∑ m ( ϕ F ( z i ) − ϕ F ( z i ⋆ ) ) + η U ( μ ^ U − μ U ) − η Y ( μ ^ B ( ξ ) − μ B ( ξ ) ) + ( η Y − η U ) ( μ in ( ξ ) − μ B ( ξ ) ) − η [ ( μ U , 0 − μ U ) − ( μ Y , 0 ( ξ ) − μ B ( ξ )) ] .
The five terms have norms at most ϵ i m \epsilon_{\mathrm{im}} ϵ im , η U Δ U \eta_U\Delta_U η U Δ U , η Y Δ Y \eta_Y\Delta_Y η Y Δ Y , 2 ∣ η Y − η U ∣ 2\lvert\eta_Y-\eta_U\rvert 2 ∣ η Y − η U ∣ and η max ( q U + q Y ) \eta_{\max}(q_U+q_Y) η m a x ( q U + q Y ) , where the fourth bound uses ∥ ϕ F ∥ ≤ 1 \lVert\phi_F\rVert\le1 ∥ ϕ F ∥ ≤ 1 . The triangle inequality therefore gives
sup θ ∈ E × I ∥ r ^ K F ( θ ) − r K F ( θ ) ∥ ≤ Δ δ . \sup_{\theta\in\mathcal{E}\times\mathcal{I}}\lVert\hat{r}_{K_F}(\theta)-r_{K_F}(\theta)\rVert\le\Delta_{\delta}. θ ∈ E × I sup ∥ r ^ K F ( θ ) − r K F ( θ )∥ ≤ Δ δ .
Eq. (A28) and this inequality give γ K F d ( θ , θ ⋆ ) ≤ ∥ r K F ( θ ) ∥ ≤ ∥ r ^ K F ( θ ) ∥ + Δ δ \gamma_{K_F}d(\theta,\theta^\star)\le\lVert r_{K_F}(\theta)\rVert\le\lVert\hat{r}_{K_F}(\theta)\rVert+\Delta_{\delta} γ K F d ( θ , θ ⋆ ) ≤ ∥ r K F ( θ )∥ ≤ ∥ r ^ K F ( θ )∥ + Δ δ , which is Eq. (A30). Because r K F ( θ ⋆ ) = 0 r_{K_F}(\theta^\star)=0 r K F ( θ ⋆ ) = 0 , the empirical residual at θ ⋆ \theta^\star θ ⋆ has norm at most Δ δ \Delta_{\delta} Δ δ . A parameter θ ^ \hat{\theta} θ ^ as in the theorem therefore has residual norm at most Δ δ + ϵ f i t \Delta_{\delta}+\epsilon_{\mathrm{fit}} Δ δ + ϵ fit , and Eq. (A30) at θ ^ \hat{\theta} θ ^ gives the final bound. □
Finite quadrature bounds. We bound q U q_U q U , q Y q_Y q Y and ϵ i m \epsilon_{\mathrm{im}} ϵ im for the midpoint rules used by the method. With β k = a k / B K F \beta_k=a_k/B_{K_F} β k = a k / B K F , the feature map ϕ F \phi_F ϕ F is Lipschitz in box coordinates with constant
L ϕ ≤ 2 2 τ F + ( ∑ k β k 2 ∥ k ∥ 2 ) 1 / 2 . L_{\phi}\le\frac{2\sqrt{2}}{\tau_F}+\left(\sum_k\beta_k^2\lVert k\rVert^2\right)^{1/2}. L ϕ ≤ τ F 2 2 + ( k ∑ β k 2 ∥ k ∥ 2 ) 1/2 .
Each factor W F , r W_{F,r} W F , r is a product of two functions with values in [ 0 , 1 ] [0,1] [ 0 , 1 ] and Lipschitz constant 1 / τ F 1/\tau_F 1/ τ F , so the gradient of W F W_F W F has norm at most 2 2 / τ F 2\sqrt{2}/\tau_F 2 2 / τ F . The untapered feature vector has norm one, and the inequality ∣ e i ϑ − e i ϑ ′ ∣ ≤ ∣ ϑ − ϑ ′ ∣ \lvert e^{i\vartheta}-e^{i\vartheta'}\rvert\le\lvert\vartheta-\vartheta'\rvert ∣ e i ϑ − e i ϑ ′ ∣ ≤ ∣ ϑ − ϑ ′ ∣ for real ϑ \vartheta ϑ and ϑ ′ \vartheta' ϑ ′ bounds its Lipschitz constant by the second term. Let L p r o j L_{\mathrm{proj}} L proj bound the norm of the derivative of y ↦ p ξ ( y ) y\mapsto p_{\xi}(y) y ↦ p ξ ( y ) on E × B \mathcal{E}\times\mathcal{B} E × B , which is finite by compactness and positive depths. For an image rectangle with side lengths w U , r w_{U,r} w U , r in box coordinates and J U , r J_{U,r} J U , r midpoint cells along axis r r r , and a model box with side lengths w Y , r w_{Y,r} w Y , r and J Y , r J_{Y,r} J Y , r cells, the bounds are
q U ≤ L ϕ 2 ( ∑ r = 1 2 ( w U , r / J U , r ) 2 ) 1 / 2 , q_U\le\frac{L_{\phi}}{2}\left(\sum_{r=1}^{2}\left(w_{U,r}/J_{U,r}\right)^2\right)^{1/2}, q U ≤ 2 L ϕ ( r = 1 ∑ 2 ( w U , r / J U , r ) 2 ) 1/2 ,
q Y ≤ L ϕ L p r o j 2 ( ∑ r = 1 3 ( w Y , r / J Y , r ) 2 ) 1 / 2 . q_Y\le\frac{L_{\phi}L_{\mathrm{proj}}}{2}\left(\sum_{r=1}^{3}\left(w_{Y,r}/J_{Y,r}\right)^2\right)^{1/2}. q Y ≤ 2 L ϕ L proj ( r = 1 ∑ 3 ( w Y , r / J Y , r ) 2 ) 1/2 .
Every point of a cell lies within half the cell diagonal of its midpoint, and the box is convex, so the integrand differs from its midpoint value by at most L ϕ L_{\phi} L ϕ or L ϕ L p r o j L_{\phi}L_{\mathrm{proj}} L ϕ L proj , times that distance. Averaging over each cell and summing proves these bounds. Likewise, ϵ i m ≤ L ϕ n − 1 ∑ i = 1 m ∥ z i − z i ⋆ ∥ \epsilon_{\mathrm{im}}\le L_{\phi}n^{-1}\sum_{i=1}^{m}\lVert z_i-z_i^\star\rVert ϵ im ≤ L ϕ n − 1 ∑ i = 1 m ∥ z i − z i ⋆ ∥ .
Relation to the fitted fraction. The method sets the fraction at each pose to the least-squares value η ( ξ ) \eta(\xi) η ( ξ ) of Step 2 in App. B, clipped to [ 0 , 0.75 ] [0,0.75] [ 0 , 0.75 ] . The theorems cover this choice with I = [ 0 , 0.75 ] \mathcal{I}=[0,0.75] I = [ 0 , 0.75 ] and η max = 0.75 \eta_{\max}=0.75 η m a x = 0.75 whenever η U ≤ 0.75 \eta_U\le0.75 η U ≤ 0.75 , because their bounds hold for every θ ∈ E × I \theta\in\mathcal{E}\times\mathcal{I} θ ∈ E × I . At the population level, Eq. (A25) reads r K F ( ξ , η ) = r K F ( ξ , η ⋆ ) + ( η ⋆ − η ) μ o u t ( ξ ) r_{K_F}(\xi,\eta)=r_{K_F}(\xi,\eta^\star)+(\eta^\star-\eta)\mu^{\mathrm{out}}(\xi) r K F ( ξ , η ) = r K F ( ξ , η ⋆ ) + ( η ⋆ − η ) μ out ( ξ ) . Here μ o u t ( ξ ) = μ U − μ B ( ξ ) \mu^{\mathrm{out}}(\xi)=\mu^{U}-\mu^{B}(\xi) μ out ( ξ ) = μ U − μ B ( ξ ) is the population value of the difference μ U , 0 − μ Y , 0 ( ξ ) \mu_{U,0}-\mu_{Y,0}(\xi) μ U , 0 − μ Y , 0 ( ξ ) used by the method. For K F ≥ K 0 K_F\ge K_0 K F ≥ K 0 , the vector μ o u t ( ξ ⋆ ) \mu^{\mathrm{out}}(\xi^\star) μ out ( ξ ⋆ ) is nonzero, because − μ o u t ( ξ ⋆ ) -\mu^{\mathrm{out}}(\xi^\star) − μ out ( ξ ⋆ ) is the derivative of r K F r_{K_F} r K F with respect to η \eta η at θ ⋆ \theta^\star θ ⋆ , a column of an injective derivative. Where μ o u t ( ξ ) ≠ 0 \mu^{\mathrm{out}}(\xi)\ne0 μ out ( ξ ) = 0 , the population version of η ^ ( ξ ) \hat{\eta}(\xi) η ^ ( ξ ) is η p r ( ξ ) = η ⋆ + ⟨ r K F ( ξ , η ⋆ ) , μ o u t ( ξ ) ⟩ / ∥ μ o u t ( ξ ) ∥ 2 \eta^{\mathrm{pr}}(\xi)=\eta^\star+\langle r_{K_F}(\xi,\eta^\star),\mu^{\mathrm{out}}(\xi)\rangle/\lVert\mu^{\mathrm{out}}(\xi)\rVert^2 η pr ( ξ ) = η ⋆ + ⟨ r K F ( ξ , η ⋆ ) , μ out ( ξ )⟩ / ∥ μ out ( ξ ) ∥ 2 . The profiled residual is r p r ( ξ ) = r K F ( ξ , η p r ( ξ ) ) = Π μ o u t ( ξ ) ⊥ r K F ( ξ , η ⋆ ) r^{\mathrm{pr}}(\xi)=r_{K_F}(\xi,\eta^{\mathrm{pr}}(\xi))=\Pi_{\mu^{\mathrm{out}}(\xi)}^\perp r_{K_F}(\xi,\eta^\star) r pr ( ξ ) = r K F ( ξ , η pr ( ξ )) = Π μ out ( ξ ) ⊥ r K F ( ξ , η ⋆ ) , where Π μ o u t ⊥ \Pi_{\mu^{\mathrm{out}}}^\perp Π μ out ⊥ with a vector as subscript denotes the orthogonal projection onto the orthogonal complement of that vector. If 0 < η ⋆ < 0.75 0<\eta^\star<0.75 0 < η ⋆ < 0.75 , the continuity of η p r \eta^{\mathrm{pr}} η pr at ξ ⋆ \xi^\star ξ ⋆ , where it equals η ⋆ \eta^\star η ⋆ , makes the clipping inactive near ξ ⋆ \xi^\star ξ ⋆ . Because r K F ( ξ , η ⋆ ) r_{K_F}(\xi,\eta^\star) r K F ( ξ , η ⋆ ) vanishes and is differentiable at ξ ⋆ \xi^\star ξ ⋆ , and μ o u t \mu^{\mathrm{out}} μ out is continuous there, the profiled residual has the derivative
− ( 1 − η ⋆ ) Π μ o u t ( ξ ⋆ ) ⊥ ∂ μ i n ( ξ ⋆ ) -(1-\eta^\star)\Pi_{\mu^{\mathrm{out}}(\xi^\star)}^\perp\partial\mu^{\mathrm{in}}(\xi^\star) − ( 1 − η ⋆ ) Π μ out ( ξ ⋆ ) ⊥ ∂ μ in ( ξ ⋆ )
at ξ ⋆ \xi^\star ξ ⋆ . This derivative has rank six. Indeed, a direction ξ ˙ \dot{\xi} ξ ˙ in its kernel makes ∂ μ o u t ( ξ ⋆ ) ξ ˙ \partial\mu^{\mathrm{out}}(\xi^\star)\dot{\xi} ∂ μ out ( ξ ⋆ ) ξ ˙ a multiple of μ o u t ( ξ ⋆ ) \mu^{\mathrm{out}}(\xi^\star) μ out ( ξ ⋆ ) , and the injective derivative of r K F r_{K_F} r K F then gives ξ ˙ = 0 \dot{\xi}=0 ξ ˙ = 0 . Pose changes therefore remain distinguishable after the fraction is fitted, and the squared norm of the profiled residual grows at least quadratically in the pose error near ξ ⋆ \xi^\star ξ ⋆ . If, in addition, η p r \eta^{\mathrm{pr}} η pr is continuously differentiable with a Lipschitz derivative near ξ ⋆ \xi^\star ξ ⋆ , the local convergence theory of Gauss–Newton for zero-residual problems with an injective derivative applies [57 ] . Theorem A3 bounds the error of every returned parameter by its residual, so it does not require the search to attain the global minimum.
Scope. The theorems hold for each fixed set of weights, that is, for each fixed spectral width σ F \sigma_F σ F . The schedule of the method pairs each cutoff with its own spectral width (Table A3 ), so the theorems apply to each stage of the schedule separately. The error terms of Eq. (29) account for finite samples and the grid approximation.
Table A3. Search settings on the point-set benchmarks (ours). Each rotation-grid size gives the number of viewing directions times the number of roll angles. Moment matching and EM use every input point. Scoring the rotation grid uses at most 4,096 model points, and cross-correlation at most 1,024. The CPU search has no time limit.
Ambiguous poses
Step 4 refines seven alternatives to the accepted pose, obtained by half-turns about the principal axes of the model points and about the optical axis (§ 3.2 ). This section shows why these are the poses to test. Under scaled orthography, the accepted pose and its seven alternatives give images with the same mean and covariance for every model (Lemma A8(a)). An alternative gives the same scaled-orthographic image as the accepted pose when the model has the corresponding symmetry (part (b)). For a planar model, the alternative that mirrors the depths about the centroid does so (part (c)). A final bound shows that, under perspective projection, mirroring the depths moves each image point by at most an amount that grows with the depth variation of the model, so the two images stay close for shallow models.
Setting. We use calibrated image coordinates, with K = I 3 K = I_3 K = I 3 , and state the results for a 3D model Y Y Y . Step 4 applies them to all model points on complete inputs and to the model points matched at the accepted pose when outliers may be present. Let y ˉ \bar y y ˉ and Σ Y \Sigma_Y Σ Y be the centroid and covariance of the model points, and let e 1 , e 2 , e 3 e_1, e_2, e_3 e 1 , e 2 , e 3 be an orthonormal basis of eigenvectors of Σ Y \Sigma_Y Σ Y , the principal axes. The half-turn about the k k k th principal axis is H k = 2 e k e k T − I 3 H_k = 2e_ke_k^\mathsf{T} - I_3 H k = 2 e k e k T − I 3 for k = 1 , 2 , 3 k = 1, 2, 3 k = 1 , 2 , 3 , and H 0 = I 3 H_0 = I_3 H 0 = I 3 . The half-turn about the optical axis is D = diag ( − 1 , − 1 , 1 ) D = \operatorname{diag}(-1,-1,1) D = diag ( − 1 , − 1 , 1 ) , and − D = diag ( 1 , 1 , − 1 ) -D = \operatorname{diag}(1,1,-1) − D = diag ( 1 , 1 , − 1 ) reverses depth. A pose ( R , t ) (R,t) ( R , t ) maps y j y_j y j to x j = R y j + t x_j = Ry_j+t x j = R y j + t in camera coordinates and the centroid to x ˉ = R y ˉ + t \bar x = R\bar y+t x ˉ = R y ˉ + t , with positive depth x ˉ 3 \bar x_3 x ˉ 3 . The scaled-orthographic image of x j x_j x j is ( x j 1 , x j 2 ) / x ˉ 3 (x_{j1},x_{j2})/\bar x_3 ( x j 1 , x j 2 ) / x ˉ 3 , and its pinhole image is proj c a l ( x j ) = ( x j 1 , x j 2 ) / x j 3 \operatorname{proj}_{\mathrm{cal}}(x_j) = (x_{j1},x_{j2})/x_{j3} proj cal ( x j ) = ( x j 1 , x j 2 ) / x j 3 when x j 3 > 0 x_{j3} > 0 x j 3 > 0 . The eight poses of Step 4 are the accepted pose and its seven alternatives. For k = 0 , … , 3 k = 0,\ldots,3 k = 0 , … , 3 , they are
R k + = R H k , R k − = D R H k , R_k^+ = RH_k,\qquad R_k^- = DRH_k, R k + = R H k , R k − = D R H k ,
t k ± = x ˉ − R k ± y ˉ . (33) t_k^\pm= \bar x - R_k^\pm\bar y. \tag*{(33)} t k ± = x ˉ − R k ± y ˉ . (33)
where each translation keeps the centroid at x ˉ \bar x x ˉ . They combine the four choices of H k H_k H k with the presence or absence of D D D .
Lemma A8 (Second-order ambiguous poses and model symmetries). (a) Each R k ± R_k^\pm R k ± is a rotation, each of the eight poses maps the centroid to x ˉ \bar x x ˉ , and the scaled-orthographic images of the eight poses have the same mean and the same covariance.
(b) Suppose that the eigenvalues of Σ Y \Sigma_Y Σ Y are distinct. Then every orthogonal matrix Q s Q_s Q s that maps the centered model to itself, as a multiset, is one of ± H k \pm H_k ± H k . If H k H_k H k is such a symmetry, the pose ( R k + , t k + ) (R_k^+,t_k^+) ( R k + , t k + ) gives the same multiset of points in camera coordinates as ( R , t ) (R,t) ( R , t ) , and hence the same image. If − H k -H_k − H k is such a symmetry, the pose ( R k − , t k − ) (R_k^-,t_k^-) ( R k − , t k − ) gives the multiset { x ˉ − D ( x j − x ˉ ) } \{\bar x-D(x_j-\bar x)\} { x ˉ − D ( x j − x ˉ )} . These are the points produced by ( R , t ) (R,t) ( R , t ) with their depths mirrored about x ˉ 3 \bar x_3 x ˉ 3 , so the scaled-orthographic image is the same.
(c) If the model lies in a plane, we number the principal axes so that e 3 e_3 e 3 is its unit normal. The reflection I 3 − 2 e 3 e 3 T = − H 3 I_3-2e_3e_3^\mathsf{T}=-H_3 I 3 − 2 e 3 e 3 T = − H 3 in the plane then fixes every centered model point. The pose ( R 3 − , t 3 − ) (R_3^-,t_3^-) ( R 3 − , t 3 − ) maps each y j y_j y j to x ˉ − D ( x j − x ˉ ) \bar x-D(x_j-\bar x) x ˉ − D ( x j − x ˉ ) and hence gives the same scaled-orthographic image as ( R , t ) (R,t) ( R , t ) , whether or not the eigenvalues are distinct.
Proof. The translations keep the centroid fixed, so every part reduces to how H k H_k H k and D D D act on the centered points.
Part (a). Each H k H_k H k is either the identity or a symmetric orthogonal matrix with eigenvalues 1 , − 1 , − 1 1,-1,-1 1 , − 1 , − 1 , so it is a rotation. The matrix D D D is also a rotation, and so is every product R k ± R_k^\pm R k ± . By the choice of t k ± t_k^\pm t k ± , the pose ( R k ± , t k ± ) (R_k^\pm,t_k^\pm) ( R k ± , t k ± ) maps the model points to
x ˉ + R k ± ( y j − y ˉ ) , (34) \bar x+R_k^\pm(y_j-\bar y), \tag*{(34)} x ˉ + R k ± ( y j − y ˉ ) , (34)
with centroid x ˉ \bar x x ˉ and covariance R k ± Σ Y ( R k ± ) T R_k^\pm\Sigma_Y(R_k^\pm)^\mathsf{T} R k ± Σ Y ( R k ± ) T . A half-turn about a principal axis negates two principal coordinates and preserves their variances, because H k e i = ± e i H_ke_i=\pm e_i H k e i = ± e i and H k Σ Y H k = Σ Y H_k\Sigma_YH_k=\Sigma_Y H k Σ Y H k = Σ Y . For R k + R_k^+ R k + , the covariance is therefore R Σ Y R T R\Sigma_YR^\mathsf{T} R Σ Y R T , and for R k − R_k^- R k − it is D R Σ Y R T D DR\Sigma_YR^\mathsf{T}D D R Σ Y R T D . The scaled-orthographic image keeps the first two coordinates, divided by the common depth x ˉ 3 \bar x_3 x ˉ 3 . The matrix D D D negates both, which leaves their covariance unchanged. The image covariance is therefore the upper-left 2 × 2 2\times2 2 × 2 block of R Σ Y R T / x ˉ 3 2 R\Sigma_YR^\mathsf{T}/\bar x_3^2 R Σ Y R T / x ˉ 3 2 in all eight cases, and the image mean is ( x ˉ 1 , x ˉ 2 ) / x ˉ 3 (\bar x_1,\bar x_2)/\bar x_3 ( x ˉ 1 , x ˉ 2 ) / x ˉ 3 .
Part (b). The multiset { Q s ( y j − y ˉ ) } \{Q_s(y_j-\bar y)\} { Q s ( y j − y ˉ )} has covariance Q s Σ Y Q s T Q_s\Sigma_YQ_s^\mathsf{T} Q s Σ Y Q s T , so a symmetry satisfies Q s Σ Y Q s T = Σ Y Q_s\Sigma_YQ_s^\mathsf{T}=\Sigma_Y Q s Σ Y Q s T = Σ Y and commutes with Σ Y \Sigma_Y Σ Y . A matrix that commutes with Σ Y \Sigma_Y Σ Y maps each eigenspace of Σ Y \Sigma_Y Σ Y to itself. Distinct eigenvalues make every eigenspace a line, so the orthogonal matrix Q s Q_s Q s satisfies Q s e i = ± e i Q_se_i=\pm e_i Q s e i = ± e i . Thus Q s Q_s Q s is diagonal in the principal-axis basis with entries ± 1 \pm1 ± 1 , which gives the eight matrices ± H k \pm H_k ± H k . If H k H_k H k is a symmetry, then by Eq. (34) the pose ( R k + , t k + ) (R_k^+,t_k^+) ( R k + , t k + ) gives the multiset
{ x ˉ + R H k ( y j − y ˉ ) } = { x ˉ + R ( y j − y ˉ ) } = { x j } . (35) \{\bar x+RH_k(y_j-\bar y)\}=\{\bar x+R(y_j-\bar y)\}=\{x_j\}. \tag*{(35)} { x ˉ + R H k ( y j − y ˉ )} = { x ˉ + R ( y j − y ˉ )} = { x j } . (35)
If − H k -H_k − H k is a symmetry, the pose ( R k − , t k − ) (R_k^-,t_k^-) ( R k − , t k − ) gives the multiset
{ x ˉ + D R H k ( y j − y ˉ ) } = { x ˉ − D R ( − H k ) ( y j − y ˉ ) } = { x ˉ − D R ( y j − y ˉ ) } = { x ˉ − D ( x j − x ˉ ) } . (36) \begin{aligned}
\{\bar x+DRH_k(y_j-\bar y)\}
&=\{\bar x-DR(-H_k)(y_j-\bar y)\}\\
&=\{\bar x-DR(y_j-\bar y)\}=\{\bar x-D(x_j-\bar x)\}. \tag*{(36)}
\end{aligned} { x ˉ + D R H k ( y j − y ˉ )} = { x ˉ − D R ( − H k ) ( y j − y ˉ )} = { x ˉ − D R ( y j − y ˉ )} = { x ˉ − D ( x j − x ˉ )} . (36)
The matrix − D -D − D negates the third coordinate and keeps the first two, so the scaled-orthographic image is unchanged.
Part (c). The unit normal of the model plane is an eigenvector of Σ Y \Sigma_Y Σ Y with eigenvalue zero, so we can choose it as e 3 e_3 e 3 . For a planar model, e 3 ⋅ ( y j − y ˉ ) = 0 e_3\cdot(y_j-\bar y)=0 e 3 ⋅ ( y j − y ˉ ) = 0 for every j j j . Hence ( I 3 − 2 e 3 e 3 T ) ( y j − y ˉ ) = y j − y ˉ (I_3-2e_3e_3^\mathsf{T})(y_j-\bar y)=y_j-\bar y ( I 3 − 2 e 3 e 3 T ) ( y j − y ˉ ) = y j − y ˉ , and H 3 ( y j − y ˉ ) = − ( y j − y ˉ ) H_3(y_j-\bar y)=-(y_j-\bar y) H 3 ( y j − y ˉ ) = − ( y j − y ˉ ) . By Eq. (34) , the pose ( R 3 − , t 3 − ) (R_3^-,t_3^-) ( R 3 − , t 3 − ) maps y j y_j y j to x ˉ + D R H 3 ( y j − y ˉ ) = x ˉ − D ( x j − x ˉ ) \bar x+DRH_3(y_j-\bar y)=\bar x-D(x_j-\bar x) x ˉ + D R H 3 ( y j − y ˉ ) = x ˉ − D ( x j − x ˉ ) . As in part (b), the matrix − D -D − D leaves the scaled-orthographic image unchanged.
Part (b) requires distinct eigenvalues of Σ Y \Sigma_Y Σ Y . A model with repeated eigenvalues can have further symmetries, and parts (a) and (c) still hold.
Perspective projection. A pinhole camera divides by the depth of each point, so mirroring the depths can change the image. We bound this change when the depths of the model points vary little compared with the depth of their centroid. By parts (b) and (c), the alternative ( R k − , t ˉ k − ) (R_k^{-}, \bar{t}_k^{-}) ( R k − , t ˉ k − ) maps the model points onto the multiset of the points x j ′ = x ˉ − D ( x j − x ˉ ) x_j' = \bar{x} - D(x_j - \bar{x}) x j ′ = x ˉ − D ( x j − x ˉ ) when − H k -H_k − H k is a symmetry of the model, or when the model is planar and k = 3 k = 3 k = 3 . The point x j ′ x_j' x j ′ keeps the first two coordinates of x j x_j x j and has depth 2 x ˉ 3 − x j 3 2\bar{x}_3 - x_{j3} 2 x ˉ 3 − x j 3 . Let ζ = max j ∣ x j 3 − x ˉ 3 ∣ \zeta= \max_j |x_{j3} - \bar{x}_3| ζ = max j ∣ x j 3 − x ˉ 3 ∣ be the largest deviation of a depth from that of the centroid, and assume ζ < x ˉ 3 \zeta< \bar{x}_3 ζ < x ˉ 3 . Every x j ′ x_j' x j ′ then has positive depth, and the pinhole image of the alternative is the multiset of the points proj c a l ( x j ′ ) \operatorname{proj}_{\mathrm{cal}}(x_j') proj cal ( x j ′ ) , which satisfy
∥ proj c a l ( x j ′ ) − proj c a l ( x j ) ∥ = ∥ proj c a l ( x j ) ∥ 2 ∣ x j 3 − x ˉ 3 ∣ 2 x ˉ 3 − x j 3 ≤ ∥ proj c a l ( x j ) ∥ 2 ζ x ˉ 3 − ζ . (37) \left\|\operatorname{proj}_{\mathrm{cal}}(x_j')-\operatorname{proj}_{\mathrm{cal}}(x_j)\right\|
=
\left\|\operatorname{proj}_{\mathrm{cal}}(x_j)\right\|
\frac{2|x_{j3}-\bar{x}_3|}{2\bar{x}_3-x_{j3}}
\le
\left\|\operatorname{proj}_{\mathrm{cal}}(x_j)\right\|
\frac{2\zeta}{\bar{x}_3-\zeta}. \tag*{(37)} proj cal ( x j ′ ) − proj cal ( x j ) = ∥ proj cal ( x j ) ∥ 2 x ˉ 3 − x j 3 2∣ x j 3 − x ˉ 3 ∣ ≤ ∥ proj cal ( x j ) ∥ x ˉ 3 − ζ 2 ζ . (37)
If the accepted pose fits the observations and this displacement is small compared with the spacing of the observations in calibrated image units, the mirrored pose also places projections near the observations. A test of the accepted pose alone then cannot exclude it, so Step 4 refines both poses by EM and compares their likelihoods.
Matching under an accurate pose
At the selected pose, MomentPnP labels each observation with its nearest projection. We bound the probability that any of these labels is wrong when every projection is predicted within a tolerance ϵ p r e d \epsilon_{\mathrm{pred}} ϵ pred . The bound grows with the density of the projections and with the number of points, which makes precise the remark of § 4 that each correspondence depends on the spacing between nearby projections. It tends to zero as N s N s N s and N ϵ p r e d N\epsilon_{\mathrm{pred}} N ϵ pred tend to zero, and App. § 8 .9 compares this with the noise that a joint search over the camera and the correspondences tolerates.
Setting. We consider complete inputs with n = N ≥ 2 n = N \ge2 n = N ≥ 2 in which every model point lies at positive depth at the true pose ξ ⋆ \xi^\star ξ ⋆ . We number the observations so that the true correspondence is the identity. Then u i = p i + w i u_i = p_i + w_i u i = p i + w i with p i = p i ( ξ ⋆ ) p_i = p_i(\xi^\star) p i = p i ( ξ ⋆ ) . We assume that the projections p i ∈ R 2 p_i \in\mathbb{R}^2 p i ∈ R 2 are independent, each with a density bounded by f max f_{\max} f m a x , that they are jointly independent of the errors w 1 , … , w N w_1,\ldots,w_N w 1 , … , w N , and that E ∥ w i ∥ 2 = 2 s 2 \mathbb{E}\|w_i\|^2 = 2s^2 E ∥ w i ∥ 2 = 2 s 2 . The bound uses no other property of the projection, so it holds for any camera model. A prediction p ^ j \hat{p}_j p ^ j of p j p_j p j is any measurable function of the data.
For an estimated pose ( R ^ , t ^ ) (\hat{R},\hat{t}) ( R ^ , t ^ ) that places every model point at positive depth, the prediction is p ^ j = proj ( R ^ y j + t ^ ) \hat{p}_j = \operatorname{proj}(\hat{R}y_j+\hat{t}) p ^ j = proj ( R ^ y j + t ^ ) . The predictions have maximum error δ ^ = max j ∥ p ^ j − p j ∥ \hat{\delta} = \max_j\|\hat{p}_j-p_j\| δ ^ = max j ∥ p ^ j − p j ∥ . Nearest-projection matching assigns to observation i i i a label σ ^ ( i ) \hat{\sigma}(i) σ ^ ( i ) that minimizes ∥ u i − p ^ j ∥ \|u_i-\hat{p}_j\| ∥ u i − p ^ j ∥ over j j j , with ties resolved by a fixed measurable rule.
Lemma A9 (Matching error under an accurate pose). Under the assumptions above, every fixed tolerance ϵ p r e d ≥ 0 \epsilon_{\mathrm{pred}} \ge0 ϵ pred ≥ 0 on the prediction error satisfies
Pr [ σ ^ ≠ id and δ ^ ≤ ϵ p r e d ] ≤ 8 π f max N ( N − 1 ) ( 2 s 2 + ϵ p r e d 2 ) . (38) \Pr\left[\hat{\sigma}\ne\operatorname{id}\ \text{and}\ \hat{\delta}\le\epsilon_{\mathrm{pred}}\right]
\le8\pi f_{\max}N(N-1)(2s^2+\epsilon_{\mathrm{pred}}^2). \tag*{(38)} Pr [ σ ^ = id and δ ^ ≤ ϵ pred ] ≤ 8 π f m a x N ( N − 1 ) ( 2 s 2 + ϵ pred 2 ) . (38) Proof. We show that an observation can receive a wrong label only if another projection lies close to its own. We then bound the probability of such a close pair by the density bound.
Step 1. A well-separated observation is labeled correctly. We put d i = min j ≠ i ∥ p i − p j ∥ d_i=\min_{j\ne i}\|p_i-p_j\| d i = min j = i ∥ p i − p j ∥ , and let Ω i = { d i > 2 ( ∥ w i ∥ + ϵ p r e d ) } \Omega_i=\{d_i>2(\|w_i\|+\epsilon_{\mathrm{pred}})\} Ω i = { d i > 2 ( ∥ w i ∥ + ϵ pred )} be the event that observation i i i is well separated. On Ω i ∩ { δ ^ ≤ ϵ p r e d } \Omega_i\cap\{\hat{\delta}\le\epsilon_{\mathrm{pred}}\} Ω i ∩ { δ ^ ≤ ϵ pred } , every j ≠ i j\ne i j = i satisfies
∥ u i − p ^ j ∥ ≥ ∥ p i − p j ∥ − ∥ w i ∥ − ∥ p ^ j − p j ∥ ≥ d i − ∥ w i ∥ − ϵ p r e d > ∥ w i ∥ + ϵ p r e d ≥ ∥ u i − p ^ i ∥ . (39) \begin{aligned}
\|u_i-\hat{p}_j\| &\ge\|p_i-p_j\|-\|w_i\|-\|\hat{p}_j-p_j\| \\
&\ge d_i-\|w_i\|-\epsilon_{\mathrm{pred}} \\
&> \|w_i\|+\epsilon_{\mathrm{pred}}\ge\|u_i-\hat{p}_i\|. \tag*{(39)}
\end{aligned} ∥ u i − p ^ j ∥ ≥ ∥ p i − p j ∥ − ∥ w i ∥ − ∥ p ^ j − p j ∥ ≥ d i − ∥ w i ∥ − ϵ pred > ∥ w i ∥ + ϵ pred ≥ ∥ u i − p ^ i ∥. (39) The first line is the triangle inequality, and the last uses u i − p ^ i = w i + p i − p ^ i u_i-\hat{p}_i=w_i+p_i-\hat{p}_i u i − p ^ i = w i + p i − p ^ i . Observation i i i is therefore strictly closest to prediction i i i , and σ ^ ( i ) = i \hat{\sigma}(i)=i σ ^ ( i ) = i whatever the tie rule. Hence a wrong label with δ ^ ≤ ϵ p r e d \hat{\delta}\le\epsilon_{\mathrm{pred}} δ ^ ≤ ϵ pred requires at least one Ω i \Omega_i Ω i to fail.
Step 2. Probability of a close pair. We condition on p i p_i p i and w i w_i w i , since the other projections are independent of them. A disk of radius d d d has area π d 2 \pi d^2 π d 2 , so by the density bound each other projection falls in the disk of radius d d d about p i p_i p i with probability at most π f max d 2 \pi f_{\max}d^2 π f m a x d 2 . Taking d = 2 ( ∥ w i ∥ + ϵ p r e d ) d=2(\|w_i\|+\epsilon_{\mathrm{pred}}) d = 2 ( ∥ w i ∥ + ϵ pred ) and using ( ∥ w i ∥ + ϵ p r e d ) 2 ≤ 2 ∥ w i ∥ 2 + 2 ϵ p r e d 2 (\|w_i\|+\epsilon_{\mathrm{pred}})^2\le2\|w_i\|^2+2\epsilon_{\mathrm{pred}}^2 ( ∥ w i ∥ + ϵ pred ) 2 ≤ 2∥ w i ∥ 2 + 2 ϵ pred 2 , we obtain
Pr [ Ω i c ] ≤ ( N − 1 ) π f max E [ 4 ( ∥ w i ∥ + ϵ p r e d ) 2 ] ≤ 8 π f max ( N − 1 ) ( 2 s 2 + ϵ p r e d 2 ) . (40) \Pr[\Omega_i^c]\le(N-1)\pi f_{\max}\mathbb{E}\left[4(\|w_i\|+\epsilon_{\mathrm{pred}})^2\right]
\le8\pi f_{\max}(N-1)(2s^2+\epsilon_{\mathrm{pred}}^2). \tag*{(40)} Pr [ Ω i c ] ≤ ( N − 1 ) π f m a x E [ 4 ( ∥ w i ∥ + ϵ pred ) 2 ] ≤ 8 π f m a x ( N − 1 ) ( 2 s 2 + ϵ pred 2 ) . (40) A union bound over the N N N observations gives Eq. (38) . At the true pose, the predictions equal the projections, so δ ^ = 0 \hat{\delta}=0 δ ^ = 0 and Step 2 applies with ϵ p r e d = 0 \epsilon_{\mathrm{pred}}=0 ϵ pred = 0 and E [ 4 ∥ w i ∥ 2 ] = 8 s 2 \mathbb{E}[4\|w_i\|^2]=8s^2 E [ 4∥ w i ∥ 2 ] = 8 s 2 . Nearest-projection matching then errs with probability at most 8 π f max N ( N − 1 ) s 2 8\pi f_{\max}N(N-1)s^2 8 π f m a x N ( N − 1 ) s 2 . □ \square □
Exact affine matching threshold {#sec:A.9} Shuffled regression estimates a linear map without known input-output pairings [75 ] , and moment-based methods recover this map by comparing averages [47 , 74 ] . Exact recovery of every pairing in shuffled regression [50 , 81 ] and geometric planted matching [41 , 67 ] requires the noise to
stay below a threshold. We examine when every correspondence can be recovered. We compare estimating the correspondences after estimating the camera with a joint search over both.
MomentPnP estimates the pose first and then labels each observation with its nearest projection (§ 3.2 ). Theorem A4 concerns a joint least-squares search over an affine camera and all permutations of the model points, which differs from the perspective fitting and EM of MomentPnP. For random model points, it shows that this search recovers every correspondence with probability tending to one when N s → 0 Ns \to0 N s → 0 , and that no estimator does so when N s Ns N s does not tend to zero, even with the camera known. For fixed noise s > 0 s > 0 s > 0 , exact recovery of the correspondences therefore fails as the point sets grow. Lemma A9 gives the corresponding statement for nearest-projection matching. Under its assumptions, which do not depend on the camera model, nearest-projection matching recovers every label with probability tending to one if N s → 0 Ns \to0 N s → 0 and if the largest error δ ^ \hat{\delta} δ ^ of the predicted projections satisfies Pr [ δ ^ ≤ ϵ p r e d ] → 1 \Pr[\hat{\delta} \le\epsilon_{\mathrm{pred}}] \to1 Pr [ δ ^ ≤ ϵ pred ] → 1 for tolerances ϵ p r e d \epsilon_{\mathrm{pred}} ϵ pred with N ϵ p r e d → 0 N\epsilon_{\mathrm{pred}} \to0 N ϵ pred → 0 . Given an accurate pose, labeling after pose estimation therefore tolerates noise of the same order as the joint search. Under the random model of Theorem A4, no estimator tolerates noise of a larger order.
Theorem A4 (Exact affine matching threshold). Fix a distribution on R 3 \mathbb{R}^{3} R 3 that either has a bounded density on a bounded set or is a nondegenerate Gaussian, and fix an affine camera y ↦ A y + b y \mapsto Ay+b y ↦ A y + b with A ∈ R 2 × 3 A \in\mathbb{R}^{2 \times3} A ∈ R 2 × 3 of rank two and b ∈ R 2 b \in\mathbb{R}^{2} b ∈ R 2 . Draw N N N model points y 1 , … , y N y_{1},\ldots,y_{N} y 1 , … , y N independently from this distribution, and observe u i = A y σ ( i ) + b + w i u_i = Ay_{\sigma(i)}+b+w_i u i = A y σ ( i ) + b + w i for i = 1 , … , N i=1,\ldots,N i = 1 , … , N and an unknown permutation σ \sigma σ . The noise vectors w i ∼ N ( 0 , s 2 I 2 ) w_i \sim\mathcal{N}(0,s^{2}I_{2}) w i ∼ N ( 0 , s 2 I 2 ) are independent of each other and of the model. The noise level s = s N ≥ 0 s=s_N \ge0 s = s N ≥ 0 may depend on N N N as N → ∞ N \to\infty N → ∞ . If N s → 0 Ns \to0 N s → 0 , the correspondence that minimizes the least-squares reprojection error jointly with the unknown camera is σ \sigma σ with probability tending to one. In the bounded case, once N s Ns N s is below a constant, this probability is at least 1 − C t h ( N 2 s 2 + e − c t h N ) 1-C_{\mathrm{th}}(N^{2}s^{2}+e^{-c_{\mathrm{th}}N}) 1 − C th ( N 2 s 2 + e − c th N ) , where the constants C t h , c t h > 0 C_{\mathrm{th}},c_{\mathrm{th}}>0 C th , c th > 0 depend on the distribution and the camera. If lim sup N s > 0 \limsup Ns>0 lim sup N s > 0 , no estimator recovers a uniformly random σ \sigma σ with probability tending to one, even when the camera is known.
Scope. If the projections have unit variance per coordinate, the signal-to-noise ratio is S N R = 1 / s 2 \mathrm{SNR}=1/s^{2} SNR = 1/ s 2 . The threshold in Theorem A4 then lies at S N R \mathrm{SNR} SNR of order N 2 N^{2} N 2 , and exact recovery requires S N R / N 2 → ∞ \mathrm{SNR}/N^{2} \to\infty SNR / N 2 → ∞ . The theorem assumes a fixed model dimension, two image coordinates, Gaussian noise and outlier-free point sets under an affine camera. It concerns the event that every label is correct, whose probability Figure A15 measures. One wrong label makes this event fail. Figure A11 measures the percentage of wrong labels, which grows with N N N because each label depends on the density of the projections near it (Lemma A9). Over the same range of point counts, pose accuracy keeps improving up to one million points (Figure 8 ). This behavior is consistent with moments averaging over all points, and it motivates estimating the pose before the correspondences.
Figure A15. Exact label recovery across point counts. Percentage of synthetic frames in which every 2D point is labeled with its generating model point, for (a) MomentPnP, (b) CFPE and (c) EM-ICP. All panels place each noise level at N s ^ / ρ r m s N\hat{s}/\rho_{\mathrm{rms}} N s ^ / ρ rms computed from the noise s ^ \hat{s} s ^ fitted by MomentPnP. The gray curve in (a) is the fit e − κ f i t ( N s ^ / ρ r m s ) 2 e^{-\kappa_{\mathrm{fit}}(N\hat{s}/\rho_{\mathrm{rms}})^2} e − κ fit ( N s ^ / ρ rms ) 2 with κ f i t = 0.21 \kappa_{\mathrm{fit}}=0.21 κ fit = 0.21 .
Outline of the proof. Least squares selects the true correspondence when every incorrect correspondence fits worse after the noise is added. We therefore first show that, without noise, every incorrect correspondence leaves a loss that no affine camera removes, and then that noise rarely cancels this loss. The argument depends on how many points the correspondence moves. When it moves few points, its loss is at least half its total squared displacement, because each point has little influence on the fitted camera (Lemmas A10 and A11). When it moves many points, its loss exceeds a fixed constant, because a fixed affine map rarely fits it. A count of permutations and a finite covering of the affine maps extend this bound to all maps at once (Lemmas A12 to A14). Lemma A15 combines the two cases, and Lemma A16 bounds the probability that noise cancels the loss of one correspondence. Theorems A5 and A6 treat bounded and Gaussian model points. Each averages these probabilities over the model points and sums them over all correspondences, using the second bound of Lemma A12 for correspondences that move few points. Theorem A7 proves the converse from a close pair of projections that noise of order 1 / N 1/\sqrt{N} 1/ N confuses. The proof of Theorem A4 combines the three theorems.
Setting. The proof allows model points in R q \mathbb{R}^{q} R q for any dimension q ≥ 2 q \ge2 q ≥ 2 , with q = 3 q=3 q = 3 in Theorem A4. For N ≥ 2 N \ge2 N ≥ 2 , the model points y 1 , … , y N ∈ R q y_{1},\ldots,y_{N} \in\mathbb{R}^{q} y 1 , … , y N ∈ R q are independent and identically distributed. The camera is an affine map ( A , b ) (A,b) ( A , b ) with A ∈ R 2 × q A \in\mathbb{R}^{2 \times q} A ∈ R 2 × q of rank two and b ∈ R 2 b \in\mathbb{R}^{2} b ∈ R 2 , fixed independently of the sample. Scaled orthography is the case in which A A A is a positive multiple of a matrix with orthonormal rows. The dimension, the distribution and the camera stay fixed as N N N grows. The true projections are p i = A y i + b p_i=Ay_i+b p i = A y i + b . We relabel the observations so that the true correspondence is the identity id \operatorname{id} id , which gives
u i = p i + w i , w i ∼ N ( 0 , s 2 I 2 ) , (41) u_i = p_i + w_i,\qquad w_i \sim\mathcal{N}(0,s^2I_2), \tag*{(41)} u i = p i + w i , w i ∼ N ( 0 , s 2 I 2 ) , (41)
with noise vectors that are independent of each other and of the model. From here on, σ \sigma σ denotes any candidate correspondence, that is, any permutation of { 1 , … , N } \{1,\ldots,N\} { 1 , … , N } .
Least-squares losses. We score a candidate correspondence σ \sigma σ by its least-squares loss over affine maps,
L ( σ ) = min A ′ , b ′ ∑ i = 1 N ∥ u i − A ′ y σ ( i ) − b ′ ∥ 2 , (42) L(\sigma)=\min_{A',b'}\sum_{i=1}^{N}\left\lVert u_i-A'y_{\sigma(i)}-b'\right\rVert^{2}, \tag*{(42)} L ( σ ) = A ′ , b ′ min i = 1 ∑ N u i − A ′ y σ ( i ) − b ′ 2 , (42)
and its noiseless loss L 0 ( σ ) L_0(\sigma) L 0 ( σ ) is the same minimum with p i p_i p i in place of u i u_i u i . The joint least-squares correspondence minimizes L L L over all permutations. The proof also covers restricted camera families. Let F c a m \mathcal{F}_{\mathrm{cam}} F cam be a family of affine maps that contains the true map ( A , b ) (A,b) ( A , b ) , such as scaled orthography when the true camera has this form. The restricted loss L F c a m ( σ ) L_F^{\mathrm{cam}}(\sigma) L F cam ( σ ) is the infimum of the sum in Eq. (42) over the family. Theorem A4 takes all affine maps as the family. To measure how far σ \sigma σ moves the points, we write supp ( σ ) = { i : σ ( i ) ≠ i } \operatorname{supp}(\sigma)=\{i:\sigma(i)\ne i\} supp ( σ ) = { i : σ ( i ) = i } for the indices that σ \sigma σ moves and k ( σ ) = ∣ supp ( σ ) ∣ k(\sigma)=|\operatorname{supp}(\sigma)| k ( σ ) = ∣ supp ( σ ) ∣ for their number. The total squared displacement of σ \sigma σ and the smallest distance between two true projections are
S σ = ∑ i = 1 N ∥ p i − p σ ( i ) ∥ 2 , S_\sigma=\sum_{i=1}^{N}\lVert p_i-p_{\sigma(i)}\rVert^2, S σ = i = 1 ∑ N ∥ p i − p σ ( i ) ∥ 2 ,
d min = min i ≠ j ∥ p i − p j ∥ . (43) d_{\min}=\min_{i\ne j}\lVert p_i-p_j\rVert. \tag*{(43)} d m i n = i = j min ∥ p i − p j ∥ . (43)
None of these quantities depends on b b b .
Definition A1 (Bounded model). The model is bounded if (i) ∥ y i ∥ ≤ Y max \lVert y_i\rVert\le Y_{\max} ∥ y i ∥ ≤ Y m a x almost surely, (ii) the covariance of y i y_i y i is positive definite and (iii) p i p_i p i has a density on R 2 \mathbb{R}^2 R 2 bounded by f max f_{\max} f m a x .
The bounded distributions of Theorem A4 satisfy these conditions. If y i y_i y i has a density bounded by a constant and supported in the ball of radius Y max Y_{\max} Y m a x , then (i) and (ii) hold. Condition (iii) then holds with f max f_{\max} f m a x the product of the bound on the density of y i y_i y i , the volume of the unit ball in R q − 2 \mathbb{R}^{q-2} R q − 2 and Y max q − 2 / det ( A A T ) 1 / 2 Y_{\max}^{q-2}/\det(AA^\mathsf{T})^{1/2} Y m a x q − 2 / det ( A A T ) 1/2 . A sample from a surface need not satisfy (iii), because its projection can have an unbounded density at the contour. In the bounds below, all constants depend on the distribution and on A A A .
Least squares in matrix form. Writing the data as matrices turns each loss into a squared distance to a column space. We put y i + = ( y i T , 1 ) T ∈ R q + 1 y_i^+=(y_i^\mathsf{T},1)^\mathsf{T}\in\mathbb{R}^{q+1} y i + = ( y i T , 1 ) T ∈ R q + 1 and let Z ∈ R N × ( q + 1 ) Z\in\mathbb{R}^{N\times(q+1)} Z ∈ R N × ( q + 1 ) have rows y i + T y_i^{+\mathsf{T}} y i + T . The N × 2 N\times2 N × 2 matrices P ∗ P^* P ∗ , U U U and W W W have rows p i T p_i^\mathsf{T} p i T , u i T u_i^\mathsf{T} u i T and w i T w_i^\mathsf{T} w i T , so that U = P ∗ + W U=P^*+W U = P ∗ + W . A subscript σ \sigma σ reorders matrix rows, so that row i i i of Z σ Z_\sigma Z σ is y σ ( i ) + T y_{\sigma(i)}^{+\mathsf{T}} y σ ( i ) + T and row i i i of P σ ∗ P_\sigma^* P σ ∗ is p σ ( i ) T p_{\sigma(i)}^\mathsf{T} p σ ( i ) T . An affine map is the matrix Θ ′ = [ A ′ b ′ ] ∈ R 2 × ( q + 1 ) \Theta'=[A'\ b']\in\mathbb{R}^{2\times(q+1)} Θ ′ = [ A ′ b ′ ] ∈ R 2 × ( q + 1 ) , and the points it predicts under σ \sigma σ are the rows of Z σ Θ ′ T Z_\sigma\Theta'^\mathsf{T} Z σ Θ ′ T . The true map Θ = [ A b ] \Theta=[A\ b] Θ = [ A b ] gives P ∗ = Z Θ T P^*=Z\Theta^\mathsf{T} P ∗ = Z Θ T . Let Π \Pi Π and Π σ \Pi_\sigma Π σ be the orthogonal projections onto the column spaces of Z Z Z and Z σ Z_\sigma Z σ . For each image coordinate, least squares projects the observations onto the column space of Z σ Z_\sigma Z σ , and the loss is the squared Frobenius norm of the residual,
L ( σ ) = ∥ ( I − Π σ ) U ∥ 2 , L 0 ( σ ) = ∥ ( I − Π σ ) P ∗ ∥ 2 . (44) \begin{aligned}
L(\sigma)&=\lVert(I-\Pi_\sigma)U\rVert^2,\\
L_0(\sigma)&=\lVert(I-\Pi_\sigma)P^*\rVert^2. \tag*{(44)}
\end{aligned} L ( σ ) L 0 ( σ ) = ∥( I − Π σ ) U ∥ 2 , = ∥( I − Π σ ) P ∗ ∥ 2 . (44)
The diagonal entries of Π \Pi Π , called leverages, measure the influence of each point on the fit. We write Π max = max i Π i i \Pi_{\max}=\max_i\Pi_{ii} Π m a x = max i Π ii for the largest, and we have ( Π σ ) i i = Π σ ( i ) σ ( i ) (\Pi_\sigma)_{ii}=\Pi_{\sigma(i)\sigma(i)} ( Π σ ) ii = Π σ ( i ) σ ( i ) because Z σ Z_\sigma Z σ reorders the rows of Z Z Z .
Constants. The bounds use the augmented second-moment matrix Σ + = E [ y i + y i + T ] \Sigma_+=\mathbb{E}[y_i^+y_i^{+\mathsf{T}}] Σ + = E [ y i + y i + T ] . It is positive definite under (ii), because for ( a t h , a t h , 0 ) ≠ 0 (a_{\mathrm{th}},a_{\mathrm{th},0})\ne0 ( a th , a th , 0 ) = 0 its quadratic form Var ( a t h ⋅ y i ) + ( a t h ⋅ E y i + a t h , 0 ) 2 \operatorname{Var}(a_{\mathrm{th}}\cdot y_i)+(a_{\mathrm{th}}\cdot\mathbb{E}y_i+a_{\mathrm{th},0})^2 Var ( a th ⋅ y i ) + ( a th ⋅ E y i + a th , 0 ) 2 is positive. We use the quantities
λ Σ = 1 2 λ min ( Σ + ) , Λ = Y max 2 + 1 , \lambda_\Sigma=\tfrac{1}{2}\lambda_{\min}(\Sigma_+),\qquad\Lambda=\sqrt{Y_{\max}^2+1}, λ Σ = 2 1 λ m i n ( Σ + ) , Λ = Y m a x 2 + 1 ,
α t h = λ Σ 2 Λ 2 , ρ = Y max ∥ A ∥ o p , \alpha_{\mathrm{th}}=\frac{\lambda_\Sigma}{2\Lambda^2},\qquad\rho=Y_{\max}\lVert A\rVert_{\mathrm{op}}, α th = 2 Λ 2 λ Σ , ρ = Y m a x ∥ A ∥ op ,
β t h = 256 π f max s 2 , d 0 = min { 1 , α t h 2 1024 e 2 π f max } . \beta_{\mathrm{th}}=256\pi f_{\max}s^2,\qquad d_0=\min\left\{1,\frac{\alpha_{\mathrm{th}}^2}{1024e^2\pi f_{\max}}\right\}. β th = 256 π f m a x s 2 , d 0 = min { 1 , 1024 e 2 π f m a x α th 2 } .
All of them except β t h \beta_{\mathrm{th}} β th , which is proportional to s 2 s^2 s 2 , are constants. With high probability, Z T Z ⪰ N λ Σ I q + 1 Z^\mathsf{T}Z\succeq N\lambda_\Sigma I_{q+1} Z T Z ⪰ N λ Σ I q + 1 , and the leverage argument then applies to permutations that move at most α t h N \alpha_{\mathrm{th}}N α th N points (Lemma A11). Also with high probability, permutations that move more points have noiseless loss above d 0 d_0 d 0 (Lemma A14). The quantity β t h \beta_{\mathrm{th}} β th enters the final sum over permutations. By construction, ∥ y i + ∥ ≤ Λ \lVert y_i^+\rVert\le\Lambda ∥ y i + ∥ ≤ Λ and ∥ p i − b ∥ ≤ ρ \lVert p_i-b\rVert\le\rho ∥ p i − b ∥ ≤ ρ . The last diagonal entry of Σ + \Sigma_+ Σ + is one and Λ ≥ 1 \Lambda\ge1 Λ ≥ 1 , so λ Σ ≤ 1 2 \lambda_\Sigma\le\frac12 λ Σ ≤ 2 1 and α t h ≤ 1 4 \alpha_{\mathrm{th}}\le\frac14 α th ≤ 4 1 .
Lemma A10 (Noiseless loss and permutation displacement). Let x + = max { x , 0 } x_+=\max\{x,0\} x + = max { x , 0 } denote the positive part of a real number x x x . For every sample and every σ \sigma σ , we have
L 0 ( σ ) ≥ ( 1 − k ( σ ) Π max ) S σ . (45) L_0(\sigma)\ge(1-k(\sigma)\Pi_{\max})S_\sigma. \tag*{(45)} L 0 ( σ ) ≥ ( 1 − k ( σ ) Π m a x ) S σ . (45) In particular, if k ( σ ) Π max ≤ 1 2 k(\sigma)\Pi_{\max}\le\frac12 k ( σ ) Π m a x ≤ 2 1 , then L 0 ( σ ) ≥ S σ / 2 ≥ k ( σ ) d min 2 / 2 L_0(\sigma)\ge S_\sigma/2\ge k(\sigma)d_{\min}^2/2 L 0 ( σ ) ≥ S σ /2 ≥ k ( σ ) d m i n 2 /2 .
Proof. The true camera maps the reordered model to the reordered projections P σ ∗ P_\sigma^* P σ ∗ , so L 0 ( σ ) L_0(\sigma) L 0 ( σ ) is the squared norm of the part of P ∗ − P σ ∗ P^*-P_\sigma^* P ∗ − P σ ∗ that remains after the affine map is refitted. The columns of P σ ∗ = Z σ Θ T P_\sigma^*=Z_\sigma\Theta^\mathsf{T} P σ ∗ = Z σ Θ T lie in the column space of Z σ Z_\sigma Z σ , so ( I − Π σ ) P σ ∗ = 0 (I-\Pi_\sigma)P_\sigma^*=0 ( I − Π σ ) P σ ∗ = 0 , and Eq. (44) gives
L 0 ( σ ) = ∥ ( I − Π σ ) ( P ∗ − P σ ∗ ) ∥ 2 = S σ − tr ( ( P ∗ − P σ ∗ ) T Π σ ( P ∗ − P σ ∗ ) ) . (46) \begin{aligned}
L_0(\sigma)&=\lVert(I-\Pi_\sigma)(P^*-P_\sigma^*)\rVert^2\\
&=S_\sigma-\operatorname{tr}\left((P^*-P_\sigma^*)^\mathsf{T}\Pi_\sigma(P^*-P_\sigma^*)\right). \tag*{(46)}
\end{aligned} L 0 ( σ ) = ∥( I − Π σ ) ( P ∗ − P σ ∗ ) ∥ 2 = S σ − tr ( ( P ∗ − P σ ∗ ) T Π σ ( P ∗ − P σ ∗ ) ) . (46)
The rows of P ∗ − P σ ∗ P^*-P_\sigma^* P ∗ − P σ ∗ are ( p i − p σ ( i ) ) T (p_i-p_{\sigma(i)})^\mathsf{T} ( p i − p σ ( i ) ) T . They vanish outside supp ( σ ) \operatorname{supp}(\sigma) supp ( σ ) , and ∥ P ∗ − P σ ∗ ∥ 2 = S σ \lVert P^*-P_\sigma^*\rVert^2=S_\sigma ∥ P ∗ − P σ ∗ ∥ 2 = S σ . Only the block of Π σ \Pi_\sigma Π σ indexed by supp ( σ ) \operatorname{supp}(\sigma) supp ( σ ) therefore contributes to the trace, which is at most the largest eigenvalue of that block times S σ S_\sigma S σ . The block is positive semidefinite, so its largest eigenvalue is at most its trace,
∑ i ∈ supp ( σ ) ( Π σ ) i i = ∑ i ∈ supp ( σ ) Π σ ( i ) σ ( i ) ≤ k ( σ ) Π max , (47) \sum_{i\in\operatorname{supp}(\sigma)}(\Pi_\sigma)_{ii}
=\sum_{i\in\operatorname{supp}(\sigma)}\Pi_{\sigma(i)\sigma(i)}
\le k(\sigma)\Pi_{\max}, \tag*{(47)} i ∈ supp ( σ ) ∑ ( Π σ ) ii = i ∈ supp ( σ ) ∑ Π σ ( i ) σ ( i ) ≤ k ( σ ) Π m a x , (47)
where the equality holds because Z σ Z_\sigma Z σ reorders the rows of Z Z Z . This bound and L 0 ( σ ) ≥ 0 L_0(\sigma)\ge0 L 0 ( σ ) ≥ 0 give Eq. (45) . Finally, each of the k ( σ ) k(\sigma) k ( σ ) rows of P ∗ − P σ ∗ P^*-P_\sigma^* P ∗ − P σ ∗ indexed by supp ( σ ) \operatorname{supp}(\sigma) supp ( σ ) has norm at least d min d_{\min} d m i n , because σ ( i ) ≠ i \sigma(i)\ne i σ ( i ) = i for these rows, so S σ ≥ k ( σ ) d min 2 S_\sigma\ge k(\sigma)d_{\min}^2 S σ ≥ k ( σ ) d m i n 2 . ▫
Lemma A11 (Design matrix concentration). For a bounded model (Definition A1), the event Ω c o n d = { Z T Z ⪰ N λ Σ I q + 1 } \Omega_{\mathrm{cond}} = \{Z^{\mathsf{T}}Z \succeq N\lambda_{\Sigma}I_{q+1}\} Ω cond = { Z T Z ⪰ N λ Σ I q + 1 } satisfies
Pr [ Ω c o n d c ] ≤ 2 ( q + 1 ) 2 exp ( − N λ Σ 2 2 ( q + 1 ) 2 Λ 4 ) . (48) \Pr[\Omega_{\mathrm{cond}}^{c}] \le2(q+1)^2 \exp\left(-\frac{N\lambda_{\Sigma}^{2}}{2(q+1)^2\Lambda^{4}}\right). \tag*{(48)} Pr [ Ω cond c ] ≤ 2 ( q + 1 ) 2 exp ( − 2 ( q + 1 ) 2 Λ 4 N λ Σ 2 ) . (48) On Ω c o n d \Omega_{\mathrm{cond}} Ω cond , we have Π max ≤ Λ 2 / ( N λ Σ ) \Pi_{\max} \le\Lambda^2/(N\lambda_{\Sigma}) Π m a x ≤ Λ 2 / ( N λ Σ ) , and every σ \sigma σ with k ( σ ) ≤ α t h N k(\sigma) \le\alpha_{\mathrm{th}}N k ( σ ) ≤ α th N has L 0 ( σ ) ≥ S σ / 2 L_0(\sigma) \ge S_{\sigma}/2 L 0 ( σ ) ≥ S σ /2 .
Proof. We compare 1 N Z T Z \frac{1}{N}Z^{\mathsf{T}}Z N 1 Z T Z with Σ + \Sigma_{+} Σ + entry by entry and then bound the leverages. Each entry of y i y i T \mathbf{y}_i\mathbf{y}_i^{\mathsf{T}} y i y i T lies in [ − Λ 2 , Λ 2 ] [-\Lambda^2,\Lambda^2] [ − Λ 2 , Λ 2 ] . By Hoeffding’s inequality [76 ] , an entry of 1 N Z T Z − Σ + \frac{1}{N}Z^{\mathsf{T}}Z-\Sigma_{+} N 1 Z T Z − Σ + exceeds λ Σ / ( q + 1 ) \lambda_{\Sigma}/(q+1) λ Σ / ( q + 1 ) in absolute value with probability at most 2 exp ( − N λ Σ 2 / ( 2 ( q + 1 ) 2 Λ 4 ) ) 2\exp(-N\lambda_{\Sigma}^{2}/(2(q+1)^2\Lambda^4)) 2 exp ( − N λ Σ 2 / ( 2 ( q + 1 ) 2 Λ 4 )) , and there are ( q + 1 ) 2 (q+1)^2 ( q + 1 ) 2 entries. If no entry exceeds this threshold, the Frobenius norm of the difference is at most λ Σ \lambda_{\Sigma} λ Σ , so λ min ( 1 N Z T Z ) ≥ 2 λ Σ − λ Σ \lambda_{\min}(\frac{1}{N}Z^{\mathsf{T}}Z) \ge2\lambda_{\Sigma}-\lambda_{\Sigma} λ m i n ( N 1 Z T Z ) ≥ 2 λ Σ − λ Σ . On Ω c o n d \Omega_{\mathrm{cond}} Ω cond , every leverage satisfies Π i i = y i T ( Z T Z ) − 1 y i ≤ Λ 2 / ( N λ Σ ) \Pi_{ii}=\mathbf{y}_i^{\mathsf{T}}(Z^{\mathsf{T}}Z)^{-1}\mathbf{y}_i \le\Lambda^2/(N\lambda_{\Sigma}) Π ii = y i T ( Z T Z ) − 1 y i ≤ Λ 2 / ( N λ Σ ) . If k ( σ ) ≤ α t h N k(\sigma)\le\alpha_{\mathrm{th}}N k ( σ ) ≤ α th N , then k ( σ ) Π max ≤ 1 2 k(\sigma)\Pi_{\max}\le\frac{1}{2} k ( σ ) Π m a x ≤ 2 1 , and Lemma A10 applies. □ \square □
Lemma A11 gives no bound when σ \sigma σ moves more than α t h N \alpha_{\mathrm{th}}N α th N points, because the factor ( 1 − k ( σ ) Π max ) + (1-k(\sigma)\Pi_{\max})_{+} ( 1 − k ( σ ) Π m a x ) + in Eq. (45) can then vanish. For these permutations, we show that a fixed affine map rarely fits an incorrect correspondence (Lemma A12), count permutations by their moved indices and cycles (Lemma A13), and cover all relevant affine maps by finitely many fixed maps (Lemma A14). For σ ≠ i d \sigma\ne\mathrm{id} σ = id , we let cyc ( σ ) \operatorname{cyc}(\sigma) cyc ( σ ) be the number of cycles of σ \sigma σ of length at least two and put r ( σ ) = k ( σ ) − cyc ( σ ) r(\sigma)=k(\sigma)-\operatorname{cyc}(\sigma) r ( σ ) = k ( σ ) − cyc ( σ ) . Every such cycle has at least two elements, so r ( σ ) ≥ k ( σ ) / 2 r(\sigma)\ge k(\sigma)/2 r ( σ ) ≥ k ( σ ) /2 .
Lemma A12 (Residual bound for a fixed incorrect permutation). Suppose that the y i \mathbf{y}_i y i are independent and that each p i \mathbf{p}_i p i has a density bounded by f max f_{\max} f m a x . Fix an affine map ( A ′ , b ′ ) (A',b') ( A ′ , b ′ ) and a permutation σ ≠ i d \sigma\ne\mathrm{id} σ = id , and let r = r ( σ ) r=r(\sigma) r = r ( σ ) . Then, for every τ t h > 0 \tau_{\mathrm{th}}>0 τ th > 0 , the residuals of ( A ′ , b ′ ) (A',b') ( A ′ , b ′ ) satisfy the first bound below, and the residuals p i − p σ ( i ) \mathbf{p}_i-\mathbf{p}_{\sigma(i)} p i − p σ ( i ) of the true map, whose squared norms sum to S σ S_{\sigma} S σ , satisfy the second:
Pr [ ∑ i = 1 N ∥ p i − A ′ y σ ( i ) − b ′ ∥ 2 ≤ τ t h ] ≤ ( π f max τ t h ) r r ! , (49) \Pr\left[\sum_{i=1}^{N}\left\|\mathbf{p}_i-A'\mathbf{y}_{\sigma(i)}-b'\right\|^2\le\tau_{\mathrm{th}}\right]\le\frac{(\pi f_{\max}\tau_{\mathrm{th}})^r}{r!}, \tag*{(49)} Pr [ i = 1 ∑ N p i − A ′ y σ ( i ) − b ′ 2 ≤ τ th ] ≤ r ! ( π f m a x τ th ) r , (49) E exp ( − S σ / τ t h ) ≤ ( π f max τ t h ) r . (50) \mathbb{E}\exp(-S_{\sigma}/\tau_{\mathrm{th}})\le(\pi f_{\max}\tau_{\mathrm{th}})^r. \tag*{(50)} E exp ( − S σ / τ th ) ≤ ( π f m a x τ th ) r . (50)
Proof. We write ϵ i = p i − A ′ y σ ( i ) − b ′ \boldsymbol{\epsilon}_i=\mathbf{p}_i-A'\mathbf{y}_{\sigma(i)}-b' ϵ i = p i − A ′ y σ ( i ) − b ′ for the residual of observation i i i . A moved model point enters two residuals, so the residuals are not independent. We therefore drop one residual from each cycle of length at least two and integrate the remaining r r r residuals one at a time, each against a model point that no later residual involves.
Step 1. Choice of residuals. We write a cycle of σ \sigma σ of length at least two as i 0 → i 1 → ⋯ → i 0 i_0\to i_1\to\cdots\to i_0 i 0 → i 1 → ⋯ → i 0 , with σ ( i a ) = i a + 1 \sigma(i_a)=i_{a+1} σ ( i a ) = i a + 1 . We discard the residual at i 0 i_0 i 0 and order the remaining residuals of the cycle as i 1 , i 2 , … i_1,i_2,\ldots i 1 , i 2 , … . Repeating this for every cycle retains r r r residuals. For a ≥ 1 a\ge1 a ≥ 1 , the point y i a \mathbf{y}_{i_a} y i a enters the residual ϵ i a \boldsymbol{\epsilon}_{i_a} ϵ i a through p i a \mathbf{p}_{i_a} p i a , the residual ϵ i a − 1 \boldsymbol{\epsilon}_{i_{a-1}} ϵ i a − 1 through A ′ y i a A'\mathbf{y}_{i_a} A ′ y i a , and no other residual. The residual at i a − 1 i_{a-1} i a − 1 is either discarded or earlier in the order. The retained residuals after ϵ i a \boldsymbol{\epsilon}_{i_a} ϵ i a therefore do not depend on y i a \mathbf{y}_{i_a} y i a .
Step 2. Integration. We denote the retained residuals, in this order, by ϵ ( 1 ) , … , ϵ ( r ) \boldsymbol{\epsilon}_{(1)},\ldots,\boldsymbol{\epsilon}_{(r)} ϵ ( 1 ) , … , ϵ ( r ) . We claim that every measurable χ ≥ 0 \chi\ge0 χ ≥ 0 on ( R 2 ) r (\mathbb{R}^{2})^r ( R 2 ) r satisfies
E χ ( ϵ ( 1 ) , … , ϵ ( r ) ) ≤ f max r ∫ ( R 2 ) r χ . (51) \mathbb{E}\chi(\boldsymbol{\epsilon}_{(1)},\ldots,\boldsymbol{\epsilon}_{(r)})\le f_{\max}^{r}\int_{(\mathbb{R}^{2})^r}\chi. \tag*{(51)} E χ ( ϵ ( 1 ) , … , ϵ ( r ) ) ≤ f m a x r ∫ ( R 2 ) r χ . (51)
We condition on all model points except the one with the first retained index. By Step 1, the residuals ϵ ( 2 ) , … , ϵ ( r ) \boldsymbol{\epsilon}_{(2)},\ldots,\boldsymbol{\epsilon}_{(r)} ϵ ( 2 ) , … , ϵ ( r ) are then fixed, and ϵ ( 1 ) \boldsymbol{\epsilon}_{(1)} ϵ ( 1 ) is a translate of the corresponding p i \mathbf{p}_i p i . The model points are independent, so the conditional density of this p i \mathbf{p}_i p i is bounded by f max f_{\max} f m a x . The conditional expectation of χ \chi χ is therefore at most f max ∫ R 2 χ ( ϵ ( 1 ) , ϵ ( 2 ) , … , ϵ ( r ) ) d ϵ ( 1 ) f_{\max}\int_{\mathbb{R}^{2}}\chi(\boldsymbol{\epsilon}_{(1)},\boldsymbol{\epsilon}_{(2)},\ldots,\boldsymbol{\epsilon}_{(r)})\,\mathrm{d}\boldsymbol{\epsilon}_{(1)} f m a x ∫ R 2 χ ( ϵ ( 1 ) , ϵ ( 2 ) , … , ϵ ( r ) ) d ϵ ( 1 ) , which by Tonelli’s theorem [61 ] is a nonnegative measurable function of the remaining residuals. Repeating the argument for the remaining retained indices in order proves Eq. (51) .
Step 3. The two bounds. The sum of the squared norms of all residuals is at least the sum over the retained residuals. Taking for χ \chi χ the indicator of the ball of radius τ t h \sqrt{\tau_{\mathrm{th}}} τ th in R 2 r \mathbb{R}^{2r} R 2 r , whose volume is π r τ t h r / r ! \pi^r\tau_{\mathrm{th}}^r/r! π r τ th r / r ! , gives Eq. (49) . For Eq. (50) , we take ( A ′ , b ′ ) = ( A , b ) (A',b')=(A,b) ( A ′ , b ′ ) = ( A , b ) , so that ϵ i = p i − p σ ( i ) \boldsymbol{\epsilon}_i=\mathbf{p}_i-\mathbf{p}_{\sigma(i)} ϵ i = p i − p σ ( i ) . We also take χ ( ϵ ) = exp ( − ∥ ϵ ∥ 2 / τ t h ) \chi(\boldsymbol{\epsilon})=\exp(-\|\boldsymbol{\epsilon}\|^2/\tau_{\mathrm{th}}) χ ( ϵ ) = exp ( − ∥ ϵ ∥ 2 / τ th ) , whose integral over R 2 r \mathbb{R}^{2r} R 2 r is ( π τ t h ) r (\pi\tau_{\mathrm{th}})^r ( π τ th ) r . □ \square □
Lemma A13 (Number of permutations by moved points and cycles). For integers 2 ≤ k ≤ N 2\le k\le N 2 ≤ k ≤ N and 1 ≤ j ≤ ⌊ k / 2 ⌋ 1\le j\le\lfloor k/2\rfloor 1 ≤ j ≤ ⌊ k /2 ⌋ , the number N N , k , j N_{N,k,j} N N , k , j of permutations with k ( σ ) = k k(\sigma)=k k ( σ ) = k and cyc ( σ ) = j \operatorname{cyc}(\sigma)=j cyc ( σ ) = j satisfies N N , k , j ≤ N k 2 k / j ! N_{N,k,j}\le N^{k}2^{k}/j! N N , k , j ≤ N k 2 k / j ! . Consequently, for 0 < ϖ ≤ 1 0<\varpi\le1 0 < ϖ ≤ 1 we have
N N , k , j ϖ k − j ( k − j ) ! ≤ ( 4 e N ϖ k ) k . (52) N_{N,k,j}\frac{\varpi^{k-j}}{(k-j)!}\le\left(\frac{4eN\sqrt{\varpi}}{k}\right)^k. \tag*{(52)} N N , k , j ( k − j )! ϖ k − j ≤ ( k 4 e N ϖ ) k . (52)
Proof. We can choose an ordered list of k k k distinct indices in at most N k N^k N k ways. We can cut the list into j j j consecutive blocks of length at least two in fewer than 2 k 2^k 2 k ways and read each block as a cycle. Every permutation with k ( σ ) = k k(\sigma)=k k ( σ ) = k and cyc ( σ ) = j \operatorname{cyc}(\sigma)=j cyc ( σ ) = j arises in this way at least j ! j! j ! times, once for each ordering of its cycles, and dividing by this multiplicity gives the first bound. For the second, we use j ! ( k − j ) ! = k ! / ( k j ) ≥ k ! / 2 k j!(k-j)!=k!/\binom{k}{j}\ge k!/2^k j ! ( k − j )! = k ! / ( j k ) ≥ k ! / 2 k , then ϖ k − j ≤ ϖ k / 2 \varpi^{k-j}\le\varpi^{k/2} ϖ k − j ≤ ϖ k /2 , which holds because k − j ≥ k / 2 k-j\ge k/2 k − j ≥ k /2 and ϖ ≤ 1 \varpi\le1 ϖ ≤ 1 , and finally k ! ≥ ( k / e ) k k!\ge(k/e)^k k ! ≥ ( k / e ) k . □ \square □
Lemma A14 (Uniform loss bound for permutations moving many points). For a bounded model, let Ω d e n s e \Omega_{\mathrm{dense}} Ω dense be the event
that every σ \sigma σ with k ( σ ) > α t h N k(\sigma)>\alpha_{\mathrm{th}}N k ( σ ) > α th N has L 0 ( σ ) > d 0 L_0(\sigma)>d_0 L 0 ( σ ) > d 0 . Then
Pr [ Ω c o n d ∖ Ω d e n s e ] ≤ 4 − α t h N N 2 N n e t , N n e t = ( 1 + 2 ( ρ + 1 ) Λ N λ Σ d 0 ) 2 ( q + 1 ) . (53) \Pr[\Omega_{\mathrm{cond}}\setminus\Omega_{\mathrm{dense}}]\le4^{-\alpha_{\mathrm{th}}N}N^2N_{\mathrm{net}},
\qquad
N_{\mathrm{net}}=\left(1+\frac{2(\rho+1)\Lambda\sqrt{N}}{\sqrt{\lambda_{\Sigma}d_0}}\right)^{2(q+1)}. \tag*{(53)} Pr [ Ω cond ∖ Ω dense ] ≤ 4 − α th N N 2 N net , N net = ( 1 + λ Σ d 0 2 ( ρ + 1 ) Λ N ) 2 ( q + 1 ) . (53)
Proof. The noiseless loss is a minimum over all affine maps, whereas Lemma A12 applies to one fixed map. We therefore first show that every map with a small residual is bounded, and then approximate each bounded map by one of finitely many fixed maps. Neither L 0 L_0 L 0 nor Ω c o n d \Omega_{\mathrm{cond}} Ω cond depends on b b b , so we take b = 0 b=0 b = 0 , which gives ∥ P ⋆ ∥ ≤ ρ N \lVert P^\star\rVert\le\rho\sqrt{N} ∥ P ⋆ ∥ ≤ ρ N .
Step 1. The minimizer is bounded. On Ω c o n d \Omega_{\mathrm{cond}} Ω cond , suppose that some σ \sigma σ with k ( σ ) > α t h N k(\sigma)>\alpha_{\mathrm{th}}N k ( σ ) > α th N has L 0 ( σ ) ≤ d 0 ≤ 1 L_0(\sigma)\le d_0\le1 L 0 ( σ ) ≤ d 0 ≤ 1 , and let Θ ′ \Theta' Θ ′ attain this minimum. Since Z σ ⊤ Z σ = Z ⊤ Z ⪰ N λ Σ I q + 1 Z_{\sigma}^{\top}Z_{\sigma}=Z^{\top}Z\succeq N\lambda_{\Sigma}I_{q+1} Z σ ⊤ Z σ = Z ⊤ Z ⪰ N λ Σ I q + 1 , we obtain
N λ Σ ∥ Θ ′ ∥ ≤ ∥ Z σ Θ ′ ⊤ ∥ ≤ ∥ P ⋆ ∥ + d 0 ≤ ρ N + 1. (54) \sqrt{N\lambda_{\Sigma}}\lVert\Theta'\rVert\le\lVert Z_{\sigma}\Theta'^{\top}\rVert\le\lVert P^\star\rVert+\sqrt{d_0}\le\rho\sqrt{N}+1. \tag*{(54)} N λ Σ ∥ Θ ′ ∥ ≤ ∥ Z σ Θ ′ ⊤ ∥ ≤ ∥ P ⋆ ∥ + d 0 ≤ ρ N + 1. (54)
so ∥ Θ ′ ∥ ≤ ( ρ + 1 ) / λ Σ \lVert\Theta'\rVert\le(\rho+1)/\sqrt{\lambda_{\Sigma}} ∥ Θ ′ ∥ ≤ ( ρ + 1 ) / λ Σ , uniformly in σ \sigma σ .
Step 2. Finite approximation of the affine maps. We choose a finite subset G \mathcal{G} G of the ball of radius ( ρ + 1 ) / λ Σ (\rho+1)/\sqrt{\lambda_{\Sigma}} ( ρ + 1 ) / λ Σ in R 2 × ( q + 1 ) \mathbb{R}^{2\times(q+1)} R 2 × ( q + 1 ) such that every point of the ball lies within d 0 / ( Λ N ) \sqrt{d_0}/(\Lambda\sqrt{N}) d 0 / ( Λ N ) of G \mathcal{G} G . Such a covering set, called a net, needs at most N n e t N_{\mathrm{net}} N net points [76 ] and can be fixed before sampling. We replace Θ ′ \Theta' Θ ′ by its nearest point Θ ′ ′ ∈ G \Theta''\in\mathcal{G} Θ ′′ ∈ G . This replacement changes the matrix of predicted points by at most ∥ Z ∥ o p ∥ Θ ′ − Θ ′ ′ ∥ ≤ Λ ⋅ d 0 / ( Λ N ) = d 0 \lVert Z\rVert_{\mathrm{op}}\lVert\Theta'-\Theta''\rVert\le\Lambda\cdot\sqrt{d_0}/(\Lambda\sqrt{N})=\sqrt{d_0} ∥ Z ∥ op ∥ Θ ′ − Θ ′′ ∥ ≤ Λ ⋅ d 0 / ( Λ N ) = d 0 , so the residual of Θ ′ ′ \Theta'' Θ ′′ has norm at most 2 d 0 2\sqrt{d_0} 2 d 0 . Hence, if Ω c o n d \Omega_{\mathrm{cond}} Ω cond holds and Ω d e n s e \Omega_{\mathrm{dense}} Ω dense fails, there are a point Θ ′ ′ ∈ G \Theta''\in\mathcal{G} Θ ′′ ∈ G and a permutation σ \sigma σ with k ( σ ) > α t h N k(\sigma)>\alpha_{\mathrm{th}}N k ( σ ) > α th N such that
∑ i = 1 N ∥ p i − Θ ′ ′ y σ ( i ) ⋆ ∥ 2 ≤ 4 d 0 . (55) \sum_{i=1}^{N}\lVert p_i-\Theta''y_{\sigma(i)}^\star\rVert^2\le4d_0. \tag*{(55)} i = 1 ∑ N ∥ p i − Θ ′′ y σ ( i ) ⋆ ∥ 2 ≤ 4 d 0 . (55)
Step 3. Union bound. The points of G \mathcal{G} G do not depend on the sample, so Lemma A12 applies to each of them. Fix Θ ′ ′ \Theta'' Θ ′′ and σ \sigma σ . By Eq. (49) with τ t h = 4 d 0 \tau_{\mathrm{th}}=4d_0 τ th = 4 d 0 , the event of Eq. (55) has probability at most ϖ r ( σ ) / r ( σ ) ! \varpi^{r(\sigma)}/r(\sigma)! ϖ r ( σ ) / r ( σ )! , where ϖ = 4 π f max d 0 \varpi=4\pi f_{\max}d_0 ϖ = 4 π f m a x d 0 . The choice of d 0 d_0 d 0 gives ϖ ≤ α t h / ( 16 e ) ≤ 1 \sqrt{\varpi}\le\alpha_{\mathrm{th}}/(16e)\le1 ϖ ≤ α th / ( 16 e ) ≤ 1 . By Eq. (52) , the permutations with k ( σ ) = k k(\sigma)=k k ( σ ) = k and cyc ( σ ) = j \operatorname{cyc}(\sigma)=j cyc ( σ ) = j contribute at most ( 4 e N ϖ / k ) k (4eN\sqrt{\varpi}/k)^k ( 4 e N ϖ / k ) k in total. For k > α t h N k>\alpha_{\mathrm{th}}N k > α th N , this is at most ( 4 e ϖ / α t h ) k ≤ 4 − k ≤ 4 − α t h N (4e\sqrt{\varpi}/\alpha_{\mathrm{th}})^k\le4^{-k}\le4^{-\alpha_{\mathrm{th}}N} ( 4 e ϖ / α th ) k ≤ 4 − k ≤ 4 − α th N . Summing over the at most N 2 N^2 N 2 pairs ( k , j ) (k,j) ( k , j ) and the at most N n e t N_{\mathrm{net}} N net points of G \mathcal{G} G gives Eq. (53) .
Lemma A15 (Lower bounds on noiseless loss). For a bounded model, let Ω s e p = Ω c o n d ∩ Ω d e n s e \Omega_{\mathrm{sep}}=\Omega_{\mathrm{cond}}\cap\Omega_{\mathrm{dense}} Ω sep = Ω cond ∩ Ω dense . Then
Pr [ Ω s e p c ] ≤ p d e s , p d e s = 2 ( q + 1 ) 2 exp ( − N λ Σ 2 2 ( q + 1 ) 2 Λ 4 ) + 4 − α t h N N 2 N n e t . (56) \Pr[\Omega_{\mathrm{sep}}^c]\le p_{\mathrm{des}},
\qquad
p_{\mathrm{des}}=2(q+1)^2\exp\left(-\frac{N\lambda_{\Sigma}^2}{2(q+1)^2\Lambda^4}\right)+4^{-\alpha_{\mathrm{th}}N}N^2N_{\mathrm{net}}. \tag*{(56)} Pr [ Ω sep c ] ≤ p des , p des = 2 ( q + 1 ) 2 exp ( − 2 ( q + 1 ) 2 Λ 4 N λ Σ 2 ) + 4 − α th N N 2 N net . (56) and p d e s ≤ C d e s e − c t h N p_{\mathrm{des}}\le C_{\mathrm{des}}e^{-c_{\mathrm{th}}N} p des ≤ C des e − c th N for constants C d e s , c t h > 0 C_{\mathrm{des}},c_{\mathrm{th}}>0 C des , c th > 0 . On Ω s e p \Omega_{\mathrm{sep}} Ω sep , every σ ≠ id \sigma\ne\operatorname{id} σ = id satisfies L 0 ( σ ) ≥ S σ / 2 ≥ k ( σ ) d min 2 / 2 L_0(\sigma)\ge S_\sigma/2\ge k(\sigma)d_{\min}^2/2 L 0 ( σ ) ≥ S σ /2 ≥ k ( σ ) d m i n 2 /2 if k ( σ ) ≤ α t h N k(\sigma)\le\alpha_{\mathrm{th}}N k ( σ ) ≤ α th N , and L 0 ( σ ) > d 0 L_0(\sigma)>d_0 L 0 ( σ ) > d 0 otherwise.
Proof. The complement of Ω s e p \Omega_{\mathrm{sep}} Ω sep is contained in the union of Ω c o n d c \Omega_{\mathrm{cond}}^c Ω cond c and Ω c o n d ∖ Ω d e n s e \Omega_{\mathrm{cond}}\setminus\Omega_{\mathrm{dense}} Ω cond ∖ Ω dense , so its probability is at most the sum of the bounds in Eqs. (48) and (53) . This sum is exponentially small because N 2 N n e t N^2N_{\mathrm{net}} N 2 N net grows polynomially in N N N . On Ω s e p \Omega_{\mathrm{sep}} Ω sep , the first claim follows from Lemmas A10 and A11 and the second from the definition of Ω d e n s e \Omega_{\mathrm{dense}} Ω dense .
Lemma A16 (Noise perturbation of the loss). Let s > 0 s>0 s > 0 . For every fixed model sample and every σ \sigma σ , the probability over the noise satisfies
Pr [ L ( σ ) − ∥ W ∥ 2 < L 0 ( σ ) / 2 ] ≤ ( 1 + 2 q + 1 ) exp ( − L 0 ( σ ) 128 s 2 ) . (57) \Pr\left[L(\sigma)-\lVert W\rVert^2<L_0(\sigma)/2\right]\le(1+2^{q+1})\exp\left(-\frac{L_0(\sigma)}{128s^2}\right). \tag*{(57)} Pr [ L ( σ ) − ∥ W ∥ 2 < L 0 ( σ ) /2 ] ≤ ( 1 + 2 q + 1 ) exp ( − 128 s 2 L 0 ( σ ) ) . (57) Proof. By Eq. (44) and U = P ⋆ + W U=P^\star+W U = P ⋆ + W , we have
L ( σ ) − ∥ W ∥ 2 = L 0 ( σ ) + 2 ⟨ ( I − Π σ ) P ⋆ , W ⟩ − ∥ Π σ W ∥ 2 . (58) L(\sigma)-\lVert W\rVert^2=L_0(\sigma)+2\langle(I-\Pi_\sigma)P^\star,W\rangle-\lVert\Pi_\sigma W\rVert^2. \tag*{(58)} L ( σ ) − ∥ W ∥ 2 = L 0 ( σ ) + 2 ⟨( I − Π σ ) P ⋆ , W ⟩ − ∥ Π σ W ∥ 2 . (58) The middle term is linear in the noise, and the last term is the reduction in loss obtained by fitting the noise. The middle term is a centered Gaussian variable with variance 4 s 2 L 0 ( σ ) 4s^2L_0(\sigma) 4 s 2 L 0 ( σ ) , so it is below − L 0 ( σ ) / 4 -L_0(\sigma)/4 − L 0 ( σ ) /4 with probability at most exp ( − L 0 ( σ ) / ( 128 s 2 ) ) \exp(-L_0(\sigma)/(128s^2)) exp ( − L 0 ( σ ) / ( 128 s 2 )) . The variable ∥ Π σ W ∥ 2 / s 2 \lVert\Pi_\sigma W\rVert^2/s^2 ∥ Π σ W ∥ 2 / s 2 is chi-square with at most 2 ( q + 1 ) 2(q+1) 2 ( q + 1 ) degrees of freedom, and its moment generating function at 1 4 \frac14 4 1 is at most 2 q + 1 2^{q+1} 2 q + 1 . By Markov's inequality, ∥ Π σ W ∥ 2 \lVert\Pi_\sigma W\rVert^2 ∥ Π σ W ∥ 2 exceeds L 0 ( σ ) / 4 L_0(\sigma)/4 L 0 ( σ ) /4 with probability at most 2 q + 1 exp ( − L 0 ( σ ) / ( 16 s 2 ) ) 2^{q+1}\exp(-L_0(\sigma)/(16s^2)) 2 q + 1 exp ( − L 0 ( σ ) / ( 16 s 2 )) . When neither deviation occurs, the loss difference is at least L 0 ( σ ) / 2 L_0(\sigma)/2 L 0 ( σ ) /2 . Since e − L 0 ( σ ) / ( 16 s 2 ) ≤ e − L 0 ( σ ) / ( 128 s 2 ) e^{-L_0(\sigma)/(16s^2)}\le e^{-L_0(\sigma)/(128s^2)} e − L 0 ( σ ) / ( 16 s 2 ) ≤ e − L 0 ( σ ) / ( 128 s 2 ) , the sum of the two probabilities is at most the bound in Eq. (57) .
Theorem A5 (Exact recovery for bounded model points). For a bounded model with s > 0 s>0 s > 0 and N β t h < 1 N\beta_{\mathrm{th}}<1 N β th < 1 , let
Ω r e c = { L F c a m ( σ ) − L F c a m ( id ) ≥ L 0 ( σ ) / 2 > 0 for all σ ≠ id and all F c a m ⊇ ( A , b ) } . (59) \Omega_{\mathrm{rec}}=\left\{L^{\mathcal{F}_{\mathrm{cam}}}(\sigma)-L^{\mathcal{F}_{\mathrm{cam}}}(\operatorname{id})\ge L_0(\sigma)/2>0\ \text{for all }\sigma\ne\operatorname{id}\text{ and all }\mathcal{F}_{\mathrm{cam}}\supseteq(A,b)\right\}. \tag*{(59)} Ω rec = { L F cam ( σ ) − L F cam ( id ) ≥ L 0 ( σ ) /2 > 0 for all σ = id and all F cam ⊇ ( A , b ) } . (59) Then Ω r e c ∩ Ω s e p \Omega_{\mathrm{rec}}\cap\Omega_{\mathrm{sep}} Ω rec ∩ Ω sep has probability at least
1 − p d e s − ( 1 + 2 q + 1 ) [ exp ( N 2 β t h 2 ( 1 − N β t h ) ) − 1 + N ! e − d 0 / ( 128 s 2 ) ] . (60) 1-p_{\mathrm{des}}-(1+2^{q+1})\left[\exp\left(\frac{N^2\beta_{\mathrm{th}}}{2(1-N\beta_{\mathrm{th}})}\right)-1+N!\,e^{-d_0/(128s^2)}\right]. \tag*{(60)} 1 − p des − ( 1 + 2 q + 1 ) [ exp ( 2 ( 1 − N β th ) N 2 β th ) − 1 + N ! e − d 0 / ( 128 s 2 ) ] . (60) Consequently, there are constants c 1 , C t h , c t h > 0 c_1,C_{\mathrm{th}},c_{\mathrm{th}}>0 c 1 , C th , c th > 0 such that, for all N N N and all 0 < s 2 ≤ c 1 / N 2 0<s^2\le c_1/N^2 0 < s 2 ≤ c 1 / N 2 , we have
Pr [ Ω r e c ∩ Ω s e p ] ≥ 1 − C t h N 2 s 2 − C t h e − c t h N . (61) \Pr[\Omega_{\mathrm{rec}}\cap\Omega_{\mathrm{sep}}]\ge1-C_{\mathrm{th}}N^2s^2-C_{\mathrm{th}}e^{-c_{\mathrm{th}}N}. \tag*{(61)} Pr [ Ω rec ∩ Ω sep ] ≥ 1 − C th N 2 s 2 − C th e − c th N . (61) On Ω r e c \Omega_{\mathrm{rec}} Ω rec , the identity is the unique minimizer of L F c a m L^{\mathcal{F}_{\mathrm{cam}}} L F cam , so the least-squares correspondence is the true one.
Proof. We bound the probability that some incorrect correspondence has a loss too close to or below the true loss. Bounding each such probability through d min d_{\min} d m i n alone would add a logarithmic factor to the required signal-to-noise ratio, giving order N 2 log N N^{2}\log N N 2 log N . We avoid this factor by averaging each error probability over the model before summing over the N ! − 1 N!-1 N ! − 1 incorrect correspondences, and by counting permutations by their cycles.
Step 1. Reduction to the unrestricted loss. Restricting the family of affine maps cannot decrease the loss, so L F c a m ( σ ) ≥ L ( σ ) L^{F_{\mathrm{cam}}}(\sigma) \ge L(\sigma) L F cam ( σ ) ≥ L ( σ ) . Evaluating at the true map gives L F c a m ( i d ) ≤ ∥ W ∥ 2 L^{F_{\mathrm{cam}}}(\mathrm{id}) \le\lVert W\rVert^{2} L F cam ( id ) ≤ ∥ W ∥ 2 . It therefore suffices to bound the probability that L ( σ ) − ∥ W ∥ 2 < L 0 ( σ ) / 2 L(\sigma)-\lVert W\rVert^{2}<L_{0}(\sigma)/2 L ( σ ) − ∥ W ∥ 2 < L 0 ( σ ) /2 for some σ ≠ i d \sigma\ne\mathrm{id} σ = id . Almost surely d min > 0 d_{\min}>0 d m i n > 0 , so on Ω s e p \Omega_{\mathrm{sep}} Ω sep every L 0 ( σ ) L_{0}(\sigma) L 0 ( σ ) with σ ≠ i d \sigma\ne\mathrm{id} σ = id is positive by Lemma A15.
Step 2. Conditioning on the model. The noise is independent of the model, so we apply Lemma A16 with the model fixed and then average the conditional bound over the model. A union bound shows that the probability that Ω s e p \Omega_{\mathrm{sep}} Ω sep fails, or that some σ ≠ i d \sigma\ne\mathrm{id} σ = id has L ( σ ) − ∥ W ∥ 2 < L 0 ( σ ) / 2 L(\sigma)-\lVert W\rVert^{2}<L_{0}(\sigma)/2 L ( σ ) − ∥ W ∥ 2 < L 0 ( σ ) /2 , is at most
p d e s + ( 1 + 2 q + 1 ) ∑ σ ≠ i d E [ 1 Ω s e p e − L 0 ( σ ) / ( 128 s 2 ) ] . (62) p_{\mathrm{des}}+\left(1+2^{q+1}\right)\sum_{\sigma\ne\mathrm{id}}\mathbb{E}\left[1_{\Omega_{\mathrm{sep}}}e^{-L_{0}(\sigma)/(128s^{2})}\right]. \tag*{(62)} p des + ( 1 + 2 q + 1 ) σ = id ∑ E [ 1 Ω sep e − L 0 ( σ ) / ( 128 s 2 ) ] . (62)
Step 3. Correspondences that move many points. For k ( σ ) > α t h N k(\sigma)>\alpha_{\mathrm{th}}N k ( σ ) > α th N , we have L 0 ( σ ) > d 0 L_{0}(\sigma)>d_{0} L 0 ( σ ) > d 0 on Ω s e p \Omega_{\mathrm{sep}} Ω sep , so the summand is at most e − d 0 / ( 128 s 2 ) e^{-d_{0}/(128s^{2})} e − d 0 / ( 128 s 2 ) . There are fewer than N ! N! N ! such σ \sigma σ .
Step 4. Correspondences that move few points. For k ( σ ) ≤ α t h N k(\sigma)\le\alpha_{\mathrm{th}}N k ( σ ) ≤ α th N , we have L 0 ( σ ) ≥ S σ / 2 L_{0}(\sigma)\ge S_{\sigma}/2 L 0 ( σ ) ≥ S σ /2 on Ω s e p \Omega_{\mathrm{sep}} Ω sep . We use this inequality, drop the indicator and apply Eq. (50) with τ t h = 256 s 2 \tau_{\mathrm{th}}=256s^{2} τ th = 256 s 2 , which gives
E [ 1 Ω s e p e − L 0 ( σ ) / ( 128 s 2 ) ] ≤ E e − S σ / ( 256 s 2 ) ≤ β t h r ( σ ) . (63) \mathbb{E}\left[1_{\Omega_{\mathrm{sep}}}e^{-L_{0}(\sigma)/(128s^{2})}\right]\le\mathbb{E}e^{-S_{\sigma}/(256s^{2})}\le\beta_{\mathrm{th}}^{r(\sigma)}. \tag*{(63)} E [ 1 Ω sep e − L 0 ( σ ) / ( 128 s 2 ) ] ≤ E e − S σ / ( 256 s 2 ) ≤ β th r ( σ ) . (63)
Step 5. Sum over cycle types. To sum β t h r ( σ ) \beta_{\mathrm{th}}^{r(\sigma)} β th r ( σ ) over all σ ≠ i d \sigma\ne\mathrm{id} σ = id , which include those of Step 4, we group the permutations by their cycle lengths. A permutation with μ a \mu_a μ a cycles of length a a a for each a ≥ 2 a\ge2 a ≥ 2 has k = ∑ a a μ a k=\sum_a a\mu_a k = ∑ a a μ a and r = ∑ a ( a − 1 ) μ a r=\sum_a(a-1)\mu_a r = ∑ a ( a − 1 ) μ a . By Cauchy’s formula for cycle types, the number of such permutations is N ! / ( ( N − k ) ! ∏ a μ a ! ) ≤ ∏ a ( N a / a ) μ a / μ a ! N!/((N-k)!\prod_a\mu_a!)\le\prod_a(N^a/a)^{\mu_a}/\mu_a! N ! / (( N − k )! ∏ a μ a !) ≤ ∏ a ( N a / a ) μ a / μ a ! . A cycle type is a choice of μ 2 , … , μ N ≥ 0 \mu_2,\ldots,\mu_N\ge0 μ 2 , … , μ N ≥ 0 , not all zero. We allow every such choice, even when the cycles would use more than N N N indices. The added terms are nonnegative, and the sum then factors over a a a . Subtracting the term of the identity, for which every μ a \mu_a μ a is zero, gives
∑ σ ≠ i d β t h r ( σ ) ≤ ∏ a = 2 N ∑ μ ≥ 0 1 μ ! ( N a β t h a − 1 a ) μ − 1 = exp ( ∑ a = 2 N N a β t h a − 1 a ) − 1 ≤ exp ( N 2 β t h 2 ( 1 − N β t h ) ) − 1. (64) \begin{aligned}
\sum_{\sigma\ne\mathrm{id}}\beta_{\mathrm{th}}^{r(\sigma)}
\le\prod_{a=2}^{N}\sum_{\mu\ge0}\frac{1}{\mu!}\left(\frac{N^{a}\beta_{\mathrm{th}}^{a-1}}{a}\right)^{\mu}-1 \\
&=\exp\left(\sum_{a=2}^{N}\frac{N^{a}\beta_{\mathrm{th}}^{a-1}}{a}\right)-1 \\
&\le\exp\left(\frac{N^{2}\beta_{\mathrm{th}}}{2(1-N\beta_{\mathrm{th}})}\right)-1. \tag*{(64)}
\end{aligned} σ = id ∑ β th r ( σ ) ≤ a = 2 ∏ N μ ≥ 0 ∑ μ ! 1 ( a N a β th a − 1 ) μ − 1 = exp ( a = 2 ∑ N a N a β th a − 1 ) − 1 ≤ exp ( 2 ( 1 − N β th ) N 2 β th ) − 1. (64)
Together with p d e s p_{\mathrm{des}} p des from Step 2 and the bound of Step 3, this bound proves Eq. (60) .
Step 6. The rate. Let c 1 = min { 1 / ( 512 π f max ) , d 0 / 256 } c_{1}=\min\{1/(512\pi f_{\max}),d_{0}/256\} c 1 = min { 1/ ( 512 π f m a x ) , d 0 /256 } and suppose that s 2 ≤ c 1 / N 2 s^{2}\le c_{1}/N^{2} s 2 ≤ c 1 / N 2 . Then N 2 β t h ≤ N β t h ≤ 1 / 4 N^{2}\beta_{\mathrm{th}}\le N\beta_{\mathrm{th}}\le1/4 N 2 β th ≤ N β th ≤ 1/4 , so the exponent in Eq. (60) is at most N 2 β t h N^{2}\beta_{\mathrm{th}} N 2 β th . Since e x − 1 ≤ 2 x e^{x}-1\le2x e x − 1 ≤ 2 x on [ 0 , 1 ] [0,1] [ 0 , 1 ] , the first term in the bracket is at most 512 π f max N 2 s 2 512\pi f_{\max}N^{2}s^{2} 512 π f m a x N 2 s 2 . For the second term, the function s 2 ↦ e − d 0 / ( 128 s 2 ) / s 2 s^{2}\mapsto e^{-d_{0}/(128s^{2})}/s^{2} s 2 ↦ e − d 0 / ( 128 s 2 ) / s 2 is increasing for s 2 ≤ d 0 / 128 s^{2}\le d_{0}/128 s 2 ≤ d 0 /128 . Its value is therefore at most its value at s 2 = c 1 / N 2 s^{2}=c_{1}/N^{2} s 2 = c 1 / N 2 , where d 0 / ( 128 s 2 ) ≥ 2 N 2 d_{0}/(128s^{2})\ge2N^{2} d 0 / ( 128 s 2 ) ≥ 2 N 2 . Hence N ! e − d 0 / ( 128 s 2 ) ≤ ( N 2 s 2 / c 1 ) exp ( N log N − 2 N 2 ) ≤ N 2 s 2 / c 1 N!e^{-d_{0}/(128s^{2})}\le(N^{2}s^{2}/c_{1})\exp(N\log N-2N^{2})\le N^{2}s^{2}/c_{1} N ! e − d 0 / ( 128 s 2 ) ≤ ( N 2 s 2 / c 1 ) exp ( N log N − 2 N 2 ) ≤ N 2 s 2 / c 1 . Eq. (60) and the bound p d e s ≤ C d e s e − C t h N p_{\mathrm{des}}\le C_{\mathrm{des}}e^{-C_{\mathrm{th}}N} p des ≤ C des e − C th N of Lemma A15 give Eq. (61) with C t h = max { C d e s , ( 1 + 2 q + 1 ) ( 512 π f max + 1 / c 1 ) } C_{\mathrm{th}}=\max\{C_{\mathrm{des}},(1+2^{q+1})(512\pi f_{\max}+1/c_{1})\} C th = max { C des , ( 1 + 2 q + 1 ) ( 512 π f m a x + 1/ c 1 )} .
Exact recovery for Gaussian model points
Theorem A6 (Exact recovery for Gaussian model points). Suppose that the y i y_i y i are independent and identically distributed Gaussian vectors with a positive definite covariance Σ \Sigma Σ , and that A A A has rank two. There are constants c 1 , C t h > 0 c_1,C_{\mathrm{th}}>0 c 1 , C th > 0 such that, for all N ≥ 2 N\ge2 N ≥ 2 and all 0 < s 2 ≤ c 1 / N 2 0<s^{2}\le c_1/N^{2} 0 < s 2 ≤ c 1 / N 2 , the event Ω r e c \Omega_{\mathrm{rec}} Ω rec of Eq. (59) has probability at least 1 − C t h N 2 s 2 − C t h / N 1-C_{\mathrm{th}}N^{2}s^{2}-C_{\mathrm{th}}/N 1 − C th N 2 s 2 − C th / N .
Proof. A Gaussian distribution has unbounded support, so Theorem A5 does not apply directly. We repeat its proof on an event on which every sampled point lies within a deterministic radius that grows slowly with N N N . The losses and d min d_{\min} d m i n do not change when the model is translated, so we suppose that the y i y_i y i have zero mean. Then p i p_i p i is Gaussian with covariance A Σ A T A\Sigma A^{\mathsf{T}} A Σ A T , its density is at most f max = ( 2 π ) − 1 det ( A Σ A T ) − 1 / 2 f_{\max}=(2\pi)^{-1}\det(A\Sigma A^{\mathsf{T}})^{-1/2} f m a x = ( 2 π ) − 1 det ( A Σ A T ) − 1/2 , and Σ + = diag ( Σ , 1 ) \Sigma_{+}=\operatorname{diag}(\Sigma,1) Σ + = diag ( Σ , 1 ) .
Step 1. Truncation. We put Y N 2 = ( 2 q + 6 log N ) ∥ Σ ∥ o p Y_N^{2}=(2q+6\log N)\lVert\Sigma\rVert_{\mathrm{op}} Y N 2 = ( 2 q + 6 log N ) ∥ Σ ∥ op , A N 2 = Y N 2 + 1 A_N^{2}=Y_N^{2}+1 A N 2 = Y N 2 + 1 , ρ N = Y N ∥ A ∥ o p \rho_N=Y_N\lVert A\rVert_{\mathrm{op}} ρ N = Y N ∥ A ∥ op , α t h , N = λ Σ / ( 2 λ max ) \alpha_{\mathrm{th},N}=\lambda_{\Sigma}/(2\lambda_{\max}) α th , N = λ Σ / ( 2 λ m a x ) , and d 0 , N = min { 1 , α t h , N 2 / ( 1024 e 2 π f max 2 ) } d_{0,N}=\min\{1,\alpha_{\mathrm{th},N}^{2}/(1024e^{2}\pi f_{\max}^{2})\} d 0 , N = min { 1 , α th , N 2 / ( 1024 e 2 π f m a x 2 )} . Let Ω c o n d , N \Omega_{\mathrm{cond},N} Ω cond , N be the event that max i ∥ y i ∥ ≤ Y N \max_i\lVert y_i\rVert\le Y_N max i ∥ y i ∥ ≤ Y N and Z T Z ⪰ N λ Σ I q + 1 Z^{\mathsf{T}}Z\succeq N\lambda_{\Sigma}I_{q+1} Z T Z ⪰ N λ Σ I q + 1 . By the chi-square tail bound of Laurent and Massart [42 ] , each ∥ y i ∥ 2 \lVert y_i\rVert^{2} ∥ y i ∥ 2 exceeds Y N 2 Y_N^{2} Y N 2 with probability at most N − 2 N^{-2} N − 2 , so the first condition fails with probability at most 1 / N 1/N 1/ N . The entries of y i y i T y_i y_i^{\mathsf{T}} y i y i T have finite moment generating functions near zero. By the Chernoff bound, each entry of 1 N Z T Z − Σ + \frac{1}{N}Z^{\mathsf{T}}Z-\Sigma_{+} N 1 Z T Z − Σ + therefore exceeds λ Σ / ( q + 1 ) \lambda_{\Sigma}/(q+1) λ Σ / ( q + 1 ) in absolute value with probability at most 2 e − c G N 2e^{-c_GN} 2 e − c G N for some c G > 0 c_G>0 c G > 0 . Hence Pr [ Ω c o n d , N c ] ≤ 1 / N + 2 ( q + 1 ) 2 e − c G N \Pr[\Omega_{\mathrm{cond},N}^{c}]\le1/N+2(q+1)^{2}e^{-c_GN} Pr [ Ω cond , N c ] ≤ 1/ N + 2 ( q + 1 ) 2 e − c G N .
Step 2. Separation. Apart from the concentration of Z T Z Z^{\mathsf T}Z Z T Z , which Step 1 replaces, the proofs of Lemmas A11 and A14 use Y max Y_{\max} Y m a x only through three bounds, namely ∥ y i + ∥ ≤ Λ \lVert y_i^+\rVert\le\Lambda ∥ y i + ∥ ≤ Λ for the leverages, ∥ P ⋆ ∥ ≤ ρ N \lVert P^\star\rVert\le\rho\sqrt{N} ∥ P ⋆ ∥ ≤ ρ N for the minimizer and ∥ Z ∥ op ≤ Λ N \lVert Z\rVert_{\operatorname{op}} \le\Lambda\sqrt{N} ∥ Z ∥ op ≤ Λ N for the net. On Ω c o n d , N \Omega_{\mathrm{cond},N} Ω cond , N , all three hold with ( Y max , Λ , α t h , d 0 , ρ ) (Y_{\max},\Lambda,\alpha_{\mathrm{th}},d_0,\rho) ( Y m a x , Λ , α th , d 0 , ρ ) replaced by the deterministic quantities ( Y N , Λ N , α t h , N , d 0 , N , ρ N ) (Y_N,\Lambda_N,\alpha_{\mathrm{th},N},d_{0,N},\rho_N) ( Y N , Λ N , α th , N , d 0 , N , ρ N ) . Lemma A12 needs only the independence of the model points and the density bound, which hold without truncation. Thus, on Ω c o n d , N \Omega_{\mathrm{cond},N} Ω cond , N , every σ \sigma σ with k ( σ ) ≤ α t h , N N k(\sigma) \le\alpha_{\mathrm{th},N}N k ( σ ) ≤ α th , N N satisfies L 0 ( σ ) ≥ S σ / 2 L_0(\sigma) \ge S_\sigma/2 L 0 ( σ ) ≥ S σ /2 . The event that Ω c o n d , N \Omega_{\mathrm{cond},N} Ω cond , N holds and some σ \sigma σ with k ( σ ) > α t h , N N k(\sigma) > \alpha_{\mathrm{th},N}N k ( σ ) > α th , N N has L 0 ( σ ) ≤ d 0 , N L_0(\sigma) \le d_{0,N} L 0 ( σ ) ≤ d 0 , N has probability at most 4 − α t h , N N N 2 N n e t , N 4^{-\alpha_{\mathrm{th},N}N}N^2N_{\mathrm{net},N} 4 − α th , N N N 2 N net , N , where N n e t , N N_{\mathrm{net},N} N net , N is N n e t N_{\mathrm{net}} N net with the same replacements. We define Ω s e p , N \Omega_{\mathrm{sep},N} Ω sep , N by removing this event from Ω c o n d , N \Omega_{\mathrm{cond},N} Ω cond , N . Since log N ≥ log 2 \log N \ge\log2 log N ≥ log 2 for N ≥ 2 N \ge2 N ≥ 2 , there is a constant a G > 0 a_G > 0 a G > 0 such that α t h , N ≥ a G / log N \alpha_{\mathrm{th},N} \ge a_G/\log N α th , N ≥ a G / log N and d 0 , N ≥ a G / log 2 N d_{0,N} \ge a_G/\log^2 N d 0 , N ≥ a G / log 2 N for every N ≥ 2 N \ge2 N ≥ 2 . Hence N n e t , N N_{\mathrm{net},N} N net , N , in which ρ N \rho_N ρ N and Λ N \Lambda_N Λ N grow like log N \sqrt{\log N} log N , is at most a polynomial in N N N , and 4 − α t h , N N ≤ exp ( − a G N log 4 / log N ) 4^{-\alpha_{\mathrm{th},N}N} \le\exp(-a_GN\log4/\log N) 4 − α th , N N ≤ exp ( − a G N log 4/ log N ) . The probability of the removed event, multiplied by N N N , therefore tends to zero and is bounded over N ≥ 2 N \ge2 N ≥ 2 . With Step 1, this gives a constant C G C_G C G such that Pr [ Ω s e p , N c ] ≤ C G / N \Pr[\Omega_{\mathrm{sep},N}^{\mathsf c}] \le C_G/N Pr [ Ω sep , N c ] ≤ C G / N for every N ≥ 2 N \ge2 N ≥ 2 .
Step 3. Sum over the correspondences. Steps 1 to 5 of the proof of Theorem A5 apply with Ω s e p , N \Omega_{\mathrm{sep},N} Ω sep , N in place of Ω s e p \Omega_{\mathrm{sep}} Ω sep and Pr [ Ω s e p , N c ] \Pr[\Omega_{\mathrm{sep},N}^{\mathsf c}] Pr [ Ω sep , N c ] in place of p d e s p_{\mathrm{des}} p des . The correspondences with k ( σ ) ≤ α t h , N N k(\sigma) \le\alpha_{\mathrm{th},N}N k ( σ ) ≤ α th , N N contribute at most ( 1 + 2 q + 1 ) [ exp ( N 2 β t h / ( 2 ( 1 − N β t h ) ) ) − 1 ] (1+2^{q+1})[\exp(N^2\beta_{\mathrm{th}}/(2(1-N\beta_{\mathrm{th}})))-1] ( 1 + 2 q + 1 ) [ exp ( N 2 β th / ( 2 ( 1 − N β th ))) − 1 ] , which is at most a multiple of N 2 s 2 N^2s^2 N 2 s 2 as in Step 6 of that proof. The others contribute at most ( 1 + 2 q + 1 ) N ! e − d 0 , N / ( 128 s 2 ) (1+2^{q+1})N!e^{-d_{0,N}/(128s^2)} ( 1 + 2 q + 1 ) N ! e − d 0 , N / ( 128 s 2 ) . We choose c 1 ≤ min { 1 / ( 512 π f max ) , inf N ≥ 2 N 2 d 0 , N / 256 } c_1 \le\min\{1/(512\pi f_{\max}),\inf_{N\ge2}N^2d_{0,N}/256\} c 1 ≤ min { 1/ ( 512 π f m a x ) , inf N ≥ 2 N 2 d 0 , N /256 } , where the infimum is positive because every term is positive and N 2 d 0 , N → ∞ N^2d_{0,N}\to\infty N 2 d 0 , N → ∞ . Then s 2 ≤ c 1 / N 2 ≤ d 0 , N / 128 s^2 \le c_1/N^2 \le d_{0,N}/128 s 2 ≤ c 1 / N 2 ≤ d 0 , N /128 , and the monotonicity argument of Step 6 gives
N ! e − d 0 , N / ( 128 s 2 ) ≤ N 2 s 2 c 1 exp ( N log N − N 2 d 0 , N 128 c 1 ) . (65) N!e^{-d_{0,N}/(128s^2)} \le\frac{N^2s^2}{c_1}\exp\left(N\log N-\frac{N^2d_{0,N}}{128c_1}\right). \tag*{(65)} N ! e − d 0 , N / ( 128 s 2 ) ≤ c 1 N 2 s 2 exp ( N log N − 128 c 1 N 2 d 0 , N ) . (65)
The exponent tends to − ∞ -\infty − ∞ because N 2 d 0 , N ≥ a G N 2 / log 2 N N^2d_{0,N} \ge a_GN^2/\log^2N N 2 d 0 , N ≥ a G N 2 / log 2 N . The exponential therefore tends to zero and is bounded over N ≥ 2 N \ge2 N ≥ 2 , so this contribution is also at most a multiple of N 2 s 2 N^2s^2 N 2 s 2 . Adding Pr [ Ω s e p , N c ] ≤ C G / N \Pr[\Omega_{\mathrm{sep},N}^{\mathsf c}] \le C_G/N Pr [ Ω sep , N c ] ≤ C G / N gives the bound of the theorem. □
Theorem A7 (No exact recovery above the noise threshold). Suppose that the true projections p 1 , … , p N p_1,\ldots,p_N p 1 , … , p N are independent with a common density f p f_p f p on R 2 \mathbb{R}^2 R 2 bounded by f max f_{\max} f m a x , and let I p = ∫ f p 2 I_p=\int f_p^2 I p = ∫ f p 2 be the squared L 2 L^2 L 2 norm of this density. The observations are u i = p σ ( i ) + w i u_i=p_{\sigma(i)}+w_i u i = p σ ( i ) + w i , where σ \sigma σ is the true correspondence and the noise vectors w i ∼ N ( 0 , s 2 I 2 ) w_i\sim\mathcal{N}(0,s^2I_2) w i ∼ N ( 0 , s 2 I 2 ) are independent of each other and of the model. Suppose that σ \sigma σ is uniformly distributed over all permutations and independent of the model and the noise, and let the estimator know the model and the camera ( A , b ) (A,b) ( A , b ) . For every c 0 > 0 c_0>0 c 0 > 0 there is an N 0 ( f p , c 0 ) N_0(f_p,c_0) N 0 ( f p , c 0 ) such that, whenever N ≥ N 0 N\ge N_0 N ≥ N 0 and N s ≥ c 0 Ns\ge c_0 N s ≥ c 0 , every measurable estimator errs with probability at least
Φ ( − 1 / 2 ) ⋅ ( π c 0 2 I p / 8 ) 2 π f max 2 c 0 2 / 2 + 1 4 π 2 f max 2 c 0 4 > 0 , (66) \Phi(-1/\sqrt{2})\cdot\frac{(\pi c_0^2I_p/8)^2}{\pi f_{\max}^2c_0^2/2+\frac{1}{4}\pi^2f_{\max}^2c_0^4}>0, \tag*{(66)} Φ ( − 1/ 2 ) ⋅ π f m a x 2 c 0 2 /2 + 4 1 π 2 f m a x 2 c 0 4 ( π c 0 2 I p /8 ) 2 > 0 , (66) where Φ \Phi Φ is the standard normal distribution function. The same bound holds for the worst case over the true correspondence.
Proof. With probability bounded away from zero, two true projections lie within c 0 / N c_0/N c 0 / N of each other. At noise level s ≥ c 0 / N s\ge c_0/N s ≥ c 0 / N , an estimator confuses this pair with positive probability even when it knows all other labels.
Step 1. A close pair occurs with probability bounded away from zero. Let N c l o s e N_{\mathrm{close}} N close be the number of unordered pairs with ∥ p i − p j ∥ ≤ c 0 / N \lVert p_i-p_j\rVert\le c_0/N ∥ p i − p j ∥ ≤ c 0 / N . For d > 0 d>0 d > 0 , let f p , d ( u d ) f_{p,d}(u_d) f p , d ( u d ) be the average of f p f_p f p over the disk of radius d d d about u d u_d u d , so that f p , d ≤ f p f_{p,d}\le f_p f p , d ≤ f p and Pr [ ∥ p 1 − p 2 ∥ ≤ d ] = π d 2 ∫ f p f p , d \Pr[\lVert p_1-p_2\rVert\le d]=\pi d^2\int f_pf_{p,d} Pr [∥ p 1 − p 2 ∥ ≤ d ] = π d 2 ∫ f p f p , d . By the Lebesgue differentiation theorem [61 ] , f p , d → f p f_{p,d}\to f_p f p , d → f p almost everywhere as d → 0 d\to0 d → 0 . Since ∫ f p f p , d ≤ f max ∫ f p \int f_pf_{p,d}\le f_{\max}\int f_p ∫ f p f p , d ≤ f m a x ∫ f p , dominated convergence gives ∫ f p f p , d → I p > 0 \int f_pf_{p,d}\to I_p>0 ∫ f p f p , d → I p > 0 . For N ≥ N 0 N\ge N_0 N ≥ N 0 , therefore,
π c 0 2 I p 8 ≤ E N c l o s e ≤ ( N 2 ) π f max c 0 2 N 2 ≤ π f max c 0 2 2 . (67) \frac{\pi c_0^2I_p}{8}\le\mathbb{E}N_{\mathrm{close}}\le{N\choose2}\pi f_{\max}\frac{c_0^2}{N^2}\le\frac{\pi f_{\max}c_0^2}{2}. \tag*{(67)} 8 π c 0 2 I p ≤ E N close ≤ ( 2 N ) π f m a x N 2 c 0 2 ≤ 2 π f m a x c 0 2 . (67) Two distinct pairs are both close with probability at most ( π f max c 0 2 / N 2 ) 2 (\pi f_{\max}c_0^2/N^2)^2 ( π f m a x c 0 2 / N 2 ) 2 . For disjoint pairs this follows from independence, and for pairs that share a point it follows by conditioning on that point. There are fewer than N 4 / 4 N^4/4 N 4 /4 ordered choices of two distinct pairs, so E N c l o s e 2 ≤ E N c l o s e + 1 4 π 2 f max 2 c 0 4 \mathbb{E}N_{\mathrm{close}}^2\le\mathbb{E}N_{\mathrm{close}}+\frac14\pi^2f_{\max}^2c_0^4 E N close 2 ≤ E N close + 4 1 π 2 f m a x 2 c 0 4 . By the second-moment method, which is the Cauchy–Schwarz inequality, Pr [ N c l o s e > 0 ] ≥ ( E N c l o s e ) 2 / E N c l o s e \Pr[N_{\mathrm{close}}>0]\ge(\mathbb{E}N_{\mathrm{close}})^2/\mathbb{E}N_{\mathrm{close}} Pr [ N close > 0 ] ≥ ( E N close ) 2 / E N close . This is at least the fraction that multiplies Φ ( − 1 / 2 ) \Phi(-1/\sqrt{2}) Φ ( − 1/ 2 ) in Eq. (66) .
Step 2. A close pair forces a positive error probability. We choose { i , j } \{i,j\} { i , j } as a closest pair of true projections, with the choice determined by the model. We reveal to the estimator the true correspondence of every other model point, together with the unordered pair of observations generated by p i p_i p i and p j p_j p j . Given the revealed labels, the two remaining assignments have equal prior probabilities. Under each assignment, the ordered pair of observations is Gaussian in R 4 \mathbb{R}^4 R 4 with covariance s 2 I 4 s^2I_4 s 2 I 4 . The two means are distinct almost surely and lie 2 d min \sqrt{2}\,d_{\min} 2 d m i n apart. With equal priors and a common isotropic covariance, the nearest-mean test is optimal. Projection onto the line through the means gives its error probability Φ ( − d min / ( 2 s ) ) \Phi(-d_{\min}/(\sqrt{2}s)) Φ ( − d m i n / ( 2 s )) , which is at least Φ ( − 1 / 2 ) \Phi(-1/\sqrt{2}) Φ ( − 1/ 2 ) when d min ≤ c 0 / N ≤ s d_{\min}\le c_0/N\le s d m i n ≤ c 0 / N ≤ s , that is, on { N c l o s e > 0 } \{N_{\mathrm{close}}>0\} { N close > 0 } . An estimator without the revealed information cannot do better, so it errs with at least this probability on { N c l o s e > 0 } \{N_{\mathrm{close}}>0\} { N close > 0 } . Integrating over the model gives Eq. (66) . The worst-case error over the true correspondence is at least the average. □
Proof of Theorem A4. We combine the three preceding theorems for a fixed distribution and camera, using the constants of each. For 0 < s 2 ≤ c 1 / N 2 0 < s^{2} \le c_{1}/N^{2} 0 < s 2 ≤ c 1 / N 2 , the probability that the least-squares correspondence is wrong is at most C t h ( N 2 s 2 + e − c t h N ) C_{\mathrm{th}}(N^{2}s^{2}+e^{-c_{\mathrm{th}}N}) C th ( N 2 s 2 + e − c th N ) for a bounded model by Theorem A5, and at most C t h ( N 2 s 2 + 1 / N ) C_{\mathrm{th}}(N^{2}s^{2}+1/N) C th ( N 2 s 2 + 1/ N ) for Gaussian model points by Theorem A6. For s = 0 s=0 s = 0 , the loss L L L equals L 0 L_{0} L 0 , which vanishes at the identity. When d min > 0 d_{\min}>0 d m i n > 0 , which holds almost surely, every σ ≠ id \sigma\ne\operatorname{id} σ = id has positive noiseless loss on Ω s e p \Omega_{\mathrm{sep}} Ω sep for a bounded model (Lemma A15) and on Ω s e p , N \Omega_{\mathrm{sep},N} Ω sep , N for Gaussian model points (proof of Theorem A6). The error probability is then at most C d e s e − c t h N C_{\mathrm{des}}e^{-c_{\mathrm{th}}N} C des e − c th N or C G / N C_{\mathrm{G}}/N C G / N , respectively. Since the proof of Theorem A5 takes C t h ≥ C d e s C_{\mathrm{th}}\ge C_{\mathrm{des}} C th ≥ C des , the bounded case gives the stated bound whenever N s ≤ c 1 Ns\le\sqrt{c_{1}} N s ≤ c 1 . If N s → 0 Ns\to0 N s → 0 , then eventually s 2 ≤ c 1 / N 2 s^{2}\le c_{1}/N^{2} s 2 ≤ c 1 / N 2 , and all these bounds tend to zero. If lim sup N s > 0 \limsup Ns>0 lim sup N s > 0 , Theorem A7 applies along a subsequence in both models, so no estimator recovers a uniformly distributed correspondence with probability tending to one, even when the camera is known. If ties among minimizing correspondences are broken uniformly, relabeling the input relabels the output distribution in the same way. The error probability of least squares is then the same for every true correspondence, so it does not recover any fixed true correspondence with probability tending to one either. □
Outliers on the structure of the inliers
Moment matching models outliers as uniform. Real outliers often lie on the same structure as the inliers (§ 4 ), and we treat this case for the model points. Suppose that the outlier model points project with the same image distribution as the inliers. Including them then changes the feature average of the projections by at most 1 / m 1/\sqrt{m} 1/ m in root mean square, which is small when the number m m m of inliers is large. App. § 10 .7 tests the method when outliers among the 2D points lie on the scene structure.
The assumption concerns the distribution of the projections in the image. Model points on the same physical surface need not have a common projected distribution, for example when visibility or sampling density differs between them.
Lemma A17 (Inliers & outliers from a common distribution). We fix a measurable feature map ϕ : R 2 → R d ϕ \phi:\mathbb{R}^{2}\to\mathbb{R}^{d_{\phi}} ϕ : R 2 → R d ϕ with ∥ ϕ ( p ) ∥ ≤ 1 \lVert\phi(p)\rVert\le1 ∥ ϕ ( p )∥ ≤ 1 for every p ∈ R 2 p\in\mathbb{R}^{2} p ∈ R 2 . We fix integers 1 ≤ m ≤ N 1\le m\le N 1 ≤ m ≤ N and draw the projections p 1 , … , p N p_{1},\ldots,p_{N} p 1 , … , p N independently from one image distribution. We designate the first m m m projections as inliers, independently of their sampled positions. Then, in expectation over these samples,
E ∥ 1 N ∑ i ≤ N ϕ ( p i ) − 1 m ∑ i ≤ m ϕ ( p i ) ∥ 2 ≤ 1 m . (68) \mathbb{E}\left\lVert\frac{1}{N}\sum_{i\le N}\phi(p_i)-\frac{1}{m}\sum_{i\le m}\phi(p_i)\right\rVert^{2}\le\frac{1}{m}. \tag*{(68)} E N 1 i ≤ N ∑ ϕ ( p i ) − m 1 i ≤ m ∑ ϕ ( p i ) 2 ≤ m 1 . (68)
Proof. We write N o u t = N − m N_{\mathrm{out}}=N-m N out = N − m for the number of outliers. When N o u t ≥ 1 N_{\mathrm{out}}\ge1 N out ≥ 1 , we define the inlier and outlier feature averages
ϕ ˉ i n = 1 m ∑ i ≤ m ϕ ( p i ) , ϕ ˉ o u t = 1 N o u t ∑ i > m ϕ ( p i ) . (69) \bar{\phi}_{\mathrm{in}}=\frac{1}{m}\sum_{i\le m}\phi(p_i),\qquad\bar{\phi}_{\mathrm{out}}=\frac{1}{N_{\mathrm{out}}}\sum_{i>m}\phi(p_i). \tag*{(69)} ϕ ˉ in = m 1 i ≤ m ∑ ϕ ( p i ) , ϕ ˉ out = N out 1 i > m ∑ ϕ ( p i ) . (69)
We write the difference in the lemma as a multiple of the difference between these two averages, which are independent and have the same expectation.
Step 1. Decomposition. If N o u t = 0 N_{\mathrm{out}}=0 N out = 0 , the two averages in the lemma coincide. Otherwise, the average over all N N N points is ( m ϕ ˉ i n + N o u t ϕ ˉ o u t ) / N (m\bar{\phi}_{\mathrm{in}}+N_{\mathrm{out}}\bar{\phi}_{\mathrm{out}})/N ( m ϕ ˉ in + N out ϕ ˉ out ) / N , so the difference in the lemma equals
1 N ∑ i ≤ N ϕ ( p i ) − ϕ ˉ i n = N o u t N ( ϕ ˉ o u t − ϕ ˉ i n ) . (70) \frac{1}{N}\sum_{i\le N}\phi(p_i)-\bar{\phi}_{\mathrm{in}}=\frac{N_{\mathrm{out}}}{N}\left(\bar{\phi}_{\mathrm{out}}-\bar{\phi}_{\mathrm{in}}\right). \tag*{(70)} N 1 i ≤ N ∑ ϕ ( p i ) − ϕ ˉ in = N N out ( ϕ ˉ out − ϕ ˉ in ) . (70)
Step 2. Two independent averages. We put μ = E ϕ ( p 1 ) \mu=\mathbb{E}\,\phi(p_1) μ = E ϕ ( p 1 ) and v = E ∥ ϕ ( p 1 ) − μ ∥ 2 ≤ E ∥ ϕ ( p 1 ) ∥ 2 ≤ 1 v=\mathbb{E}\lVert\phi(p_1)-\mu\rVert^{2}\le\mathbb{E}\lVert\phi(p_1)\rVert^{2}\le1 v = E ∥ ϕ ( p 1 ) − μ ∥ 2 ≤ E ∥ ϕ ( p 1 ) ∥ 2 ≤ 1 . The two averages use disjoint sets of independent points, so they are independent and both have expectation μ \mu μ . The mean squares of their deviations from μ \mu μ are v / m v/m v / m and v / N o u t v/N_{\mathrm{out}} v / N out . The cross term vanishes by independence, so
E ∥ ϕ ˉ o u t − ϕ ˉ i n ∥ 2 = v ( 1 m + 1 N o u t ) ≤ N m N o u t . (71) \mathbb{E}\lVert\bar{\phi}_{\mathrm{out}}-\bar{\phi}_{\mathrm{in}}\rVert^{2}=v\left(\frac{1}{m}+\frac{1}{N_{\mathrm{out}}}\right)\le\frac{N}{mN_{\mathrm{out}}}. \tag*{(71)} E ∥ ϕ ˉ out − ϕ ˉ in ∥ 2 = v ( m 1 + N out 1 ) ≤ m N out N . (71)
Step 3. Conclusion. Multiplying by ( N o u t / N ) 2 (N_{\mathrm{out}}/N)^{2} ( N out / N ) 2 gives the exact mean square v N o u t / ( N m ) ≤ N o u t / ( N m ) ≤ 1 / m vN_{\mathrm{out}}/(Nm)\le N_{\mathrm{out}}/(Nm)\le1/m v N out / ( N m ) ≤ N out / ( N m ) ≤ 1/ m . □
Outliers from another distribution. Without the common distribution, Eq. (70) and ∥ ϕ ∥ ≤ 1 \lVert\phi\rVert\le1 ∥ ϕ ∥ ≤ 1 give the deterministic bound 2 N o u t / N 2N_{\mathrm{out}}/N 2 N out / N , as in Lemma A6. For a fixed fraction N o u t / N N_{\mathrm{out}}/N N out / N of outliers, this bound does not decrease as the number of points grows. Suppose that the projections are independent, with the inliers drawn from one distribution with feature mean μ \mu μ and the outliers from another with feature mean μ o u t \mu_{\mathrm{out}} μ out . The computation of Steps 2 and 3 then shows that the difference in the lemma lies within 1 / m 1/\sqrt{m} 1/ m of ( N o u t / N ) ( μ o u t − μ ) (N_{\mathrm{out}}/N)(\mu_{\mathrm{out}}-\mu) ( N out / N ) ( μ out − μ ) in root mean square. For a fixed outlier fraction and μ o u t ≠ μ \mu_{\mathrm{out}}\ne\mu μ out = μ , this vector is nonzero and does not change as the point sets grow. Differences in visibility or sampling density between the inlier and the outlier model points can produce this case, even when the outliers lie on the same structure as the inliers.
Camera initialization
Camera pose can be estimated by aligning point distributions [5 ] . We initialize the camera position and orientation from observations U \mathcal{U} U , a 3D model V \mathcal{V} V and known intrinsics K K K . Model points outside the observed view may have no image counterparts. We search for poses whose projections concentrate near the observations and select the model points that project into the observed image region. These points form the model for refining the pose by expectation–maximization (EM) and for the alignment check.
Camera centers and orientations. We parameterize a pose by its rotation R R R and camera center c c c , with t = − R c t=-Rc t = − R c . We form a box from the 1st and 99th percentiles of the model coordinates along the principal axes of the model. The diagonal of this box defines the scene extent, and we enlarge the box by half of each side length on both sides. Three full grids cover the enlarged box with maximum cell side lengths of 20%, 10% and 5% of the scene extent. Along a box axis of length L k L_k L k , a level with cell-size limit Δ c \Delta_c Δ c has ⌈ L k / Δ c ⌉ \lceil L_k/\Delta_c\rceil ⌈ L k / Δ c ⌉ intervals of equal length. We test the center of every resulting 3D cell. At each center, we test 11,520 rotations, using 640 viewing directions distributed over the sphere and 18 roll angles separated by 20 ∘ 20^\circ 2 0 ∘ . The box bounds the initial grid, but local refinement can move the center beyond it.
Projection densities. The calibrated observation u c a l , i u_{\mathrm{cal},i} u cal , i consists of the first two coordinates of K − 1 ( u i ⊤ , 1 ) ⊤ K^{-1}(u_i^\top,1)^\top K − 1 ( u i ⊤ , 1 ) ⊤ . At a pose ξ = ( R , t ) \xi=(R,t) ξ = ( R , t ) , let p c a l , j ( ξ ) p_{\mathrm{cal},j}(\xi) p cal , j ( ξ ) be the first two coordinates of R ( y j − c ) R(y_j-c) R ( y j − c ) divided by its depth. We use a narrow and a broad Gaussian bandwidth 0 < τ n < τ b 0<\tau_n<\tau_b 0 < τ n < τ b . Let J c a m ( ξ ) \mathcal{J}_{\mathrm{cam}}(\xi) J cam ( ξ ) contain the indices of model points at positive depth whose projections lie in the bounding rectangle of the calibrated observations, extended by 4.25 τ b 4.25\tau_b 4.25 τ b on each side. For a bandwidth τ > 0 \tau>0 τ > 0 , the smoothed projection density is
f τ ( z ; ξ ) = ∑ j ∈ J c a m ( ξ ) exp ( − ∥ z − p c a l , j ( ξ ) ∥ 2 / ( 2 τ 2 ) ) 2 π τ 2 . (72) f_\tau(z;\xi)=\sum_{j\in\mathcal{J}_{\mathrm{cam}}(\xi)}\frac{\exp\left(-\lVert z-p_{\mathrm{cal},j}(\xi)\rVert^2/(2\tau^2)\right)}{2\pi\tau^2}. \tag*{(72)} f τ ( z ; ξ ) = j ∈ J cam ( ξ ) ∑ 2 π τ 2 exp ( − ∥ z − p cal , j ( ξ ) ∥ 2 / ( 2 τ 2 ) ) . (72)
The density ratio at observation i i i is
q s , i ( ξ ) = f τ n ( u c a l , i ; ξ ) f τ b ( u c a l , i ; ξ ) + ε c a m , (73) q_{s,i}(\xi)=\frac{f_{\tau_n}(u_{\mathrm{cal},i};\xi)}{f_{\tau_b}(u_{\mathrm{cal},i};\xi)+\varepsilon_{\mathrm{cam}}}, \tag*{(73)} q s , i ( ξ ) = f τ b ( u cal , i ; ξ ) + ε cam f τ n ( u cal , i ; ξ ) , (73)
where ε c a m > 0 \varepsilon_{\mathrm{cam}}>0 ε cam > 0 prevents division by zero. Each Gaussian kernel has unit integral, so the ratio compares concentration at an observation with concentration in its neighborhood. We jointly vary the rotation and camera center to minimize
L c a m ( ξ ) = − 1 n ∑ i = 1 n log ( 1 − α + α q s , i ( ξ ) ) , (74) \mathcal{L}_{\mathrm{cam}}(\xi)=-\frac{1}{n}\sum_{i=1}^{n}\log\left(1-\alpha+\alpha q_{s,i}(\xi)\right), \tag*{(74)} L cam ( ξ ) = − n 1 i = 1 ∑ n log ( 1 − α + α q s , i ( ξ ) ) , (74)
with fixed α = 0.1 \alpha=0.1 α = 0.1 .
Numerical evaluation. We approximate the densities by bilinear rasterization, normalized Gaussian convolution and bilinear evaluation at the observations. The raster stores projected counts without division by their total, and we add ε c a m = 10 − 12 \varepsilon_{\mathrm{cam}}=10^{-12} ε cam = 1 0 − 12 to the broad density in raster-count units. At each smoothing stage, the raster covers the extended rectangle that defines J c a m \mathcal{J}_{\mathrm{cam}} J cam . We discard poses with fewer than eight projected points contributing to the raster. The coarse stage evaluates a fixed random sample of at most 65,536 model points, and the medium and fine stages evaluate one of at most 262,144.
Local refinement. At each center, we keep up to eight of the 64 rotations of lowest coarse objective, at pairwise angles above 20 ∘ 20^\circ 2 0 ∘ . We refine the kept candidates of a level in order of increasing objective, at most 2,048, 4,096 and 8,192 of them at the three full grids and 2,048 at each further level, in batches of 256. Refinement uses the quasi-Newton method of Broyden, Fletcher, Goldfarb and Shanno (BFGS) [57 ] , jointly updating the rotation and camera center through three smoothing stages. The coarse, medium and fine stages use narrow widths τ n = 0.04 \tau_n=0.04 τ n = 0.04 , 0.015 and 0.00375 and broad widths τ b = 0.16 \tau_b=0.16 τ b = 0.16 , 0.08 and 0.08 on rasters of 128 × \times × 128, 256 × \times × 256 and 1,024 × \times × 1,024, respectively. They run 15, 20 and 20 iterations, and the fine stage refines the 32 candidates of a batch with the lowest medium objective. These widths are in calibrated image coordinates, while the center grid uses model-coordinate units scaled by the scene extent.
Three-point correction. For the four candidates of a batch with the lowest fine objective, we pair every observation with its nearest projected model point in the observed image region. We take the 24 closest pairs with distinct model points and solve the three-point problem [39 ] for every triple of them. We rank the solutions by the medium objective and evaluate the best eight by the fine objective. The best of these replaces the candidate when it lowers the fine objective.
Refinement by EM. The observed image region is the rectangle spanned by the 0.5th and 99.5th percentiles of the calibrated observations, enlarged by 5% of each side length on both sides. Each candidate selects the model points at positive depth whose projections lie inside this region. At most 262,144 of these points, drawn at random, form the model for EM and the alignment check, and at least eight are required. EM in its form with outliers then refines the pose twice, from noise levels of 0.03 and then 0.01 in calibrated image coordinates, each time with the CPU refinement of Table A3 , before the alignment check of App. § 9 .§ 0.1 .
Acceptance and finer grids. After EM, we accept a pose when it passes the alignment check and at least max { 8 , 0.3 min ( n , N ) } \max\{8,0.3\min(n,N)\} max { 8 , 0.3 min ( n , N )} observations are matched, where N N N counts all model points of the scene. A pose with this many matches is also accepted when the fraction of observations within 10 − 3 10^{-3} 1 0 − 3 of a projected model point in the observed image region, in calibrated image coordinates, exceeds by at least 0.2 its largest value at 32 random poses. These random poses have centers drawn uniformly from the enlarged box and rotations drawn from the rotation grid. The search stops at the first accepted pose and otherwise moves to the next candidate. If no pose is accepted after the three full grids, up to three further levels halve the cell size but subdivide only the 512 cells with the best scan scores. If all levels are used, the search returns the pose with the most matched 2D points and reports that it was not accepted.
Coverage of the camera-center grid
Camera search supplies the initial poses that EM refines (App. § 8 .11). It places candidate camera centers at the centers of a grid of cells in a box around the 3D model. We show how fine the grid must be for some candidate to lie within a given distance of the true camera center, and how many candidates this requires.
Lemma A18 (Coverage of the camera-center grid). Let Δ c > 0 \Delta_c > 0 Δ c > 0 , and for k = 1 , 2 , 3 k = 1, 2, 3 k = 1 , 2 , 3 let a k ∈ R a_k \in\mathbb{R} a k ∈ R and L k > 0 L_k > 0 L k > 0 . We divide side k k k of the box ∏ k = 1 3 [ a k , a k + L k ] \prod_{k=1}^{3}[a_k, a_k + L_k] ∏ k = 1 3 [ a k , a k + L k ] into N k = ⌈ L k / Δ c ⌉ N_k = \lceil L_k/\Delta_c \rceil N k = ⌈ L k / Δ c ⌉ intervals of equal length L k / N k ≤ Δ c L_k/N_k \le\Delta_c L k / N k ≤ Δ c , and the products of these intervals form the cells. Then every point of the box is within 3 Δ c / 2 \sqrt{3}\,\Delta_c/2 3 Δ c /2 of a cell center, and the number of centers is
N 1 N 2 N 3 ≤ ∏ k = 1 3 ( L k Δ c + 1 ) . (75) N_1N_2N_3 \le\prod_{k=1}^{3}\left(\frac{L_k}{\Delta_c}+1\right). \tag*{(75)} N 1 N 2 N 3 ≤ k = 1 ∏ 3 ( Δ c L k + 1 ) . (75)
Proof. Since L k > 0 L_k > 0 L k > 0 and Δ c > 0 \Delta_c > 0 Δ c > 0 , every N k N_k N k is positive. The N k N_k N k intervals along side k k k each have length L k / N k L_k/N_k L k / N k and midpoints a k + ( i + 1 2 ) L k / N k a_k+(i+\frac{1}{2})L_k/N_k a k + ( i + 2 1 ) L k / N k for 0 ≤ i < N k 0 \le i < N_k 0 ≤ i < N k . A cell center consists of one midpoint per coordinate, so we can bound the distance to the nearest center one coordinate at a time.
Step 1. One coordinate. Let c c c be a point of the box. Its coordinate c k c_k c k lies in one of the intervals along side k k k , within half the interval length of the midpoint of that interval. Since N k ≥ L k / Δ c N_k \ge L_k/\Delta_c N k ≥ L k / Δ c , the interval length is at most Δ c \Delta_c Δ c .
Step 2. Three coordinates. Choosing such a midpoint in each coordinate gives a grid center c g r i d c_{\mathrm{grid}} c grid with ∥ c − c g r i d ∥ 2 = ∑ k ( c k − c g r i d , k ) 2 ≤ 3 Δ c 2 / 4 \lVert c-c_{\mathrm{grid}}\rVert^2 = \sum_k(c_k-c_{\mathrm{grid},k})^2 \le3\Delta_c^2/4 ∥ c − c grid ∥ 2 = ∑ k ( c k − c grid , k ) 2 ≤ 3 Δ c 2 /4 .
Step 3. Count. The centers are the N 1 N 2 N 3 N_1N_2N_3 N 1 N 2 N 3 triples of midpoints, and N k < L k / Δ c + 1 N_k < L_k/\Delta_c + 1 N k < L k / Δ c + 1 by the definition of the ceiling. ▫
Let r c > 0 r_c > 0 r c > 0 , and let the viewing distance d v d_v d v be the median depth of the model points observed in the image. Suppose that the true camera center c ⋆ c^\star c ⋆ lies in the box. Suppose also that the rotation grid contains a rotation R 0 R_0 R 0 such that refinement converges to the true pose from the rotation R 0 R_0 R 0 together with every camera center within distance r c d v r_cd_v r c d v of c ⋆ c^\star c ⋆ . By Lemma A18, the choice Δ c = 2 r c d v / 3 \Delta_c = 2r_cd_v/\sqrt{3} Δ c = 2 r c d v / 3 places a grid center within this distance, so refinement from this center and R 0 R_0 R 0 converges to the true pose. Whether the search refines this candidate also depends on its ranking. App. § 10 .§ 5 uses this choice of Δ c \Delta_c Δ c to bound the cost of the search.
Implementation details
Search, refinement & acceptance
Hardware. The implementation requires one Intel Xeon (Cascade Lake) central processing unit (CPU), 128 GB of random-access memory (RAM) and one RTX 4090 graphics processing unit (GPU).
This appendix details the four steps of § 3.2 and then the complete search of Algorithm A1, with its settings and complexity. Steps 1 and 2 work in calibrated image coordinates, and Steps 3 and 4 measure residuals, distances and noise in pixels. The solver also takes as input whether outliers may be present. This choice selects one of two searches, each organized in stages. In a GPU stage, the GPU generates initial camera poses and fits them by moment matching. It then runs EM on a batch of candidates, and the CPU reruns EM on at most three of them. Without outliers, a CPU search follows the GPU stages if none of their fits is accepted. App. § 9 .§ 1 adapts the steps to edge observations.
Camera initialization. When the observations cover an unknown part of a scene reconstruction, Step 1 searches over camera centers and orientations for the initial poses. Each candidate pose selects the model points that project into the observed image region, and these points form the model for Steps 3 and 4; further details in App. § 8 .11.
Monomial features. On outlier-free inputs, the monomial feature map is ϕ 4 ∘ h \phi_4 \circ h ϕ 4 ∘ h , where h h h is the fixed image normalization of § 3.1 . For a point z = ( z 1 , z 2 ) z=(z_1,z_2) z = ( z 1 , z 2 ) , the monomial vector lists the fourteen monomials of degrees one to four,
ϕ 4 ( z ) = ( z 1 , z 2 , z 1 2 , z 1 z 2 , z 2 2 , z 1 3 , z 1 2 z 2 , z 1 z 2 2 , z 2 3 , z 1 4 , z 1 3 z 2 , z 1 2 z 2 2 , z 1 z 2 3 , z 2 4 ) , (76) \phi_4(z) = (z_1,\,z_2,\,z_1^2,\,z_1z_2,\,z_2^2,\,z_1^3,\,z_1^2z_2,\,z_1z_2^2,\,z_2^3,\,z_1^4,\,z_1^3z_2,\,z_1^2z_2^2,\,z_1z_2^3,\,z_2^4), \tag*{(76)} ϕ 4 ( z ) = ( z 1 , z 2 , z 1 2 , z 1 z 2 , z 2 2 , z 1 3 , z 1 2 z 2 , z 1 z 2 2 , z 2 3 , z 1 4 , z 1 3 z 2 , z 1 2 z 2 2 , z 1 z 2 3 , z 2 4 ) , (76)
so its average over a point set contains the fourteen distinct entries of the moments M 1 \mathcal{M}_1 M 1 to M 4 \mathcal{M}_4 M 4 of Eq. (3) . The averages over the observations and over the projections at a pose ξ \xi ξ are
μ U = 1 n ∑ i = 1 n ϕ 4 ( h ( u i ) ) , μ Y ( ξ ) = 1 N ∑ j = 1 N ϕ 4 ( h ( p j ( ξ ) ) ) , \mu_U = \frac{1}{n}\sum_{i=1}^{n}\phi_4(h(u_i)), \qquad\mu_Y(\xi) = \frac{1}{N}\sum_{j=1}^{N}\phi_4(h(p_j(\xi))), μ U = n 1 i = 1 ∑ n ϕ 4 ( h ( u i )) , μ Y ( ξ ) = N 1 j = 1 ∑ N ϕ 4 ( h ( p j ( ξ ))) ,
and their squared distance is L 4 \mathcal{L}_4 L 4 of Definition 1. If all observations coincide, their root-mean-square radius is zero, and we set r c a l = 1 r_{\mathrm{cal}}=1 r cal = 1 in h h h .
Fourier features with outliers. With outliers, a few distant points can dominate the fourth-order moments, so moment matching uses bounded Fourier features. Let the bounding rectangle D F ⊂ R 2 D_F \subset\mathbb{R}^2 D F ⊂ R 2 of the calibrated observations have positive area, midpoint b F b_F b F and larger half-width q F q_F q F . We evaluate the features at the box coordinates z = ( u c a l − b F ) / q F z=(u_{\mathrm{cal}}-b_F)/q_F z = ( u cal − b F ) / q F , in which D F D_F D F becomes a rectangle with half-widths h F = ( h F , 1 , h F , 2 ) h_F=(h_{F,1},h_{F,2}) h F = ( h F , 1 , h F , 2 ) , those of D F D_F D F divided by q F q_F q F . Each feature is multiplied by the spatial taper W F ( z ) = W F , 1 ( z 1 ) W F , 2 ( z 2 ) W_F(z)=W_{F,1}(z_1)W_{F,2}(z_2) W F ( z ) = W F , 1 ( z 1 ) W F , 2 ( z 2 ) . With Clip ( t ) = min { 1 , max { 0 , t } } \operatorname{Clip}(t)=\min\{1,\max\{0,t\}\} Clip ( t ) = min { 1 , max { 0 , t }} , padding δ F \delta_F δ F and taper width τ F > 0 \tau_F>0 τ F > 0 , each factor is
W F , r ( t ) = Clip ( t + h F , r + δ F τ F ) Clip ( h F , r + δ F − t τ F ) . W_{F,r}(t)=\operatorname{Clip}\left(\frac{t+h_{F,r}+\delta_F}{\tau_F}\right)\operatorname{Clip}\left(\frac{h_{F,r}+\delta_F-t}{\tau_F}\right). W F , r ( t ) = Clip ( τ F t + h F , r + δ F ) Clip ( τ F h F , r + δ F − t ) .
Along each axis, W F , r W_{F,r} W F , r vanishes outside the padded interval [ − h F , r − δ F , h F , r + δ F ] [-h_{F,r}-\delta_F,h_{F,r}+\delta_F] [ − h F , r − δ F , h F , r + δ F ] and rises linearly over a distance τ F \tau_F τ F from each of its ends. Projections far outside D F D_F D F therefore contribute nothing, and the linear rise keeps each feature continuous as a projection crosses the ends of the interval.
For a cutoff K F K_F K F , the frequency set F \mathbb{F} F contains the vectors k ∈ π 2 Z 2 \boldsymbol{k} \in\frac{\pi}{2}\mathbb{Z}^2 k ∈ 2 π Z 2 with ∥ k ∥ ≤ π 2 K F \lVert\boldsymbol{k}\rVert\le\frac{\pi}{2}K_F ∥ k ∥ ≤ 2 π K F whose first coordinate is positive, or zero with a positive second coordinate. The set thus excludes k = 0 \boldsymbol{k}=0 k = 0 , and no frequency appears together with its negative. Each frequency contributes the cosine and sine pair a k W F ( z ) ( cos ( k T z ) , sin ( k T z ) ) a_kW_F(\boldsymbol{z})\left(\cos(\boldsymbol{k}^{\mathsf{T}}\boldsymbol{z}),\sin(\boldsymbol{k}^{\mathsf{T}}\boldsymbol{z})\right) a k W F ( z ) ( cos ( k T z ) , sin ( k T z ) ) . The amplitude a k = exp ( − ζ F 2 ∥ k ∥ 2 / 2 ) a_k=\exp(-\zeta_F^2\lVert\boldsymbol{k}\rVert^2/2) a k = exp ( − ζ F 2 ∥ k ∥ 2 /2 ) with spectral width ζ F > 0 \zeta_F>0 ζ F > 0 damps high frequencies. The concatenated pairs form a vector of norm at most B = ( ∑ k a k 2 ) 1 / 2 B=\left(\sum_k a_k^2\right)^{1/2} B = ( ∑ k a k 2 ) 1/2 . Evaluating this vector at the box coordinates of an image point u \boldsymbol{u} u and dividing it by B B B gives the feature map ϕ F ( u ) \phi_F(\boldsymbol{u}) ϕ F ( u ) , with ∥ ϕ F ∥ ≤ 1 \lVert\phi_F\rVert\le1 ∥ ϕ F ∥ ≤ 1 as Lemmas A6 and A17 require. The division scales L F L_F L F by the positive constant B − 2 B^{-2} B − 2 , so it changes neither the minimizing poses at each frequency cutoff nor the estimate η ^ \hat{\eta} η ^ below.
The Fourier feature averages of Definition 1 are μ U , F = 1 n ∑ i ϕ F ( u i ) \boldsymbol{\mu}_{U,F}=\frac{1}{n}\sum_i\phi_F(\boldsymbol{u}_i) μ U , F = n 1 ∑ i ϕ F ( u i ) and μ Y , F ( ξ ) = 1 N ∑ j ϕ F ( p j ( ξ ) ) \boldsymbol{\mu}_{Y,F}(\xi)=\frac{1}{N}\sum_j\phi_F(\boldsymbol{p}_j(\xi)) μ Y , F ( ξ ) = N 1 ∑ j ϕ F ( p j ( ξ )) . In the second average, a model point at nonpositive depth contributes a zero vector but still counts in the denominator N N N . The outlier model places uniform image outliers on D F D_F D F with density 1 / ∣ D F ∣ 1/\lvert D_F\rvert 1/ ∣ D F ∣ and model outliers on the bounding box B ⊂ R 3 \mathcal{B}\subset\mathbb{R}^3 B ⊂ R 3 of the model points with density 1 / vol ( B ) 1/\operatorname{vol}(\mathcal{B}) 1/ vol ( B ) . The box must have positive volume. We compute the image average μ U , 0 \boldsymbol{\mu}_{U,0} μ U , 0 once by midpoint quadrature on D F D_F D F . The image of a uniform distribution on B \mathcal{B} B depends on the pose and is generally not uniform, so we recompute μ Y , 0 ( ξ ) \boldsymbol{\mu}_{Y,0}(\xi) μ Y , 0 ( ξ ) at each pose from the projections of a midpoint lattice of B \mathcal{B} B . Lattice points at nonpositive depth also contribute zero. Lemma A6 gives the identity behind Eq. (2) . When the inlier observations equal their projections and both point sets contain the same fraction η \eta η of uniform outliers, the expected feature averages at the true pose ξ \xi ξ differ by η ( μ U , 0 − μ Y , 0 ( ξ ) ) \eta(\boldsymbol{\mu}_{U,0}-\boldsymbol{\mu}_{Y,0}(\xi)) η ( μ U , 0 − μ Y , 0 ( ξ )) . Noise and unequal outlier fractions add a further residual at the true pose. EM then refines the pose from the individual observations and re-estimates their outlier fraction.
Camera initialization and moment matching. On the point-set benchmarks, Step 1 places initial cameras at the orientations of a rotation grid, with translations from the centroids and spreads as described below, and scores them by L F L_F L F at the lowest cutoff. Whole-scene queries use the camera search of App. § 8 .11. Step 2 proceeds as in § 3.2 . Moment matching minimizes L F L_F L F of Definition 1 with outliers and its special case L 4 L_4 L 4 on complete inputs. To initialize the translation at a grid rotation, we match the projected centroid and root-mean-square radius to those of the observations, in closed form under scaled orthography [27 ] and then by Gauss–Newton [57 ] under perspective, keeping every model point in front of the camera. Scaled orthography provides these starting values, and all objectives and refinements use the pinhole projection of § 2 . Table A3 gives the grid sizes, the numbers of retained poses and their minimum separations.
With outliers, we set the outlier fraction at each pose to the least-squares value
η ^ ( ξ ) = ⟨ d , d 0 ⟩ ∥ d 0 ∥ 2 , d = μ U , F − μ Y , F ( ξ ) , d 0 = μ U , 0 − μ Y , 0 ( ξ ) . (77) \hat{\eta}(\xi)=\frac{\langle\boldsymbol{d},\boldsymbol{d}_0\rangle}{\lVert\boldsymbol{d}_0\rVert^2},
\qquad
\boldsymbol{d}=\boldsymbol{\mu}_{U,F}-\boldsymbol{\mu}_{Y,F}(\xi),
\qquad
\boldsymbol{d}_0=\boldsymbol{\mu}_{U,0}-\boldsymbol{\mu}_{Y,0}(\xi). \tag*{(77)} η ^ ( ξ ) = ∥ d 0 ∥ 2 ⟨ d , d 0 ⟩ , d = μ U , F − μ Y , F ( ξ ) , d 0 = μ U , 0 − μ Y , 0 ( ξ ) . (77)
clipped to [ 0 , 0.75 ] [0,0.75] [ 0 , 0.75 ] . We score each pose by the residual ∥ d − η ^ d 0 ∥ 2 \lVert\boldsymbol{d}-\hat{\eta}\boldsymbol{d}_0\rVert^2 ∥ d − η ^ d 0 ∥ 2 that remains after subtracting the estimated outlier contribution. Uniform outliers shift the centroids and root-mean-square radii from which we solve for the translation at a grid rotation. Before solving, we therefore subtract the expected contribution of a fraction η 0 \eta_0 η 0 of uniform outliers on D F D_F D F from the moments of the observations. We make the same correction to the moments of the model points using the midpoint lattice of B \mathcal{B} B . Uniform outliers fill every cell of a 5 × 5 5\times5 5 × 5 grid on D F D_F D F , so the emptiest cells contain mainly outliers. We set η 0 \eta_0 η 0 to 25 / n 25/n 25/ n times the mean count of the four emptiest cells, clipped to [ 0 , 0.75 ] [0,0.75] [ 0 , 0.75 ] . If the fitted fraction at the best pose differs from η 0 \eta_0 η 0 by more than 0.05, we repeat the search from that fraction. The fits then proceed through increasing frequency cutoffs K F K_F K F . If no fit is accepted, the search with outliers repeats this fitting from the further starting fractions of Table A3 . It finally obtains further camera poses from the rotation grid, optimizes their features and refines them by EM.
Affine initial poses. On outlier-free inputs, the CPU search begins from the affine initial poses of Theorem A1. We write U c a l = { u c a l , i } \mathcal{U}_{\mathrm{cal}}=\{\boldsymbol{u}_{\mathrm{cal},i}\} U cal = { u cal , i } for the calibrated observations, with mean u ˉ c a l \bar{\boldsymbol{u}}_{\mathrm{cal}} u ˉ cal and covariance Σ U c a l \Sigma_{\mathcal{U}_{\mathrm{cal}}} Σ U cal . The model points have mean y ˉ \bar{\boldsymbol{y}} y ˉ and covariance Σ Y \Sigma_{\mathcal{Y}} Σ Y . When both covariances are positive definite, the normalized points are u ~ i = Σ U c a l − 1 / 2 ( u c a l , i − u ˉ c a l ) \tilde{\boldsymbol{u}}_i=\Sigma_{\mathcal{U}_{\mathrm{cal}}}^{-1/2}(\boldsymbol{u}_{\mathrm{cal},i}-\bar{\boldsymbol{u}}_{\mathrm{cal}}) u ~ i = Σ U cal − 1/2 ( u cal , i − u ˉ cal ) and y ~ j = Σ Y − 1 / 2 ( y j − y ˉ ) \tilde{\boldsymbol{y}}_j=\Sigma_{\mathcal{Y}}^{-1/2}(\boldsymbol{y}_j-\bar{\boldsymbol{y}}) y ~ j = Σ Y − 1/2 ( y j − y ˉ ) . For complete noiseless affine projections, they are related by an orientation O ∈ V O\in\mathcal{V} O ∈ V (Lemma A3). We identify a normalized point u ~ = ( u ~ 1 , u ~ 2 ) \tilde{\boldsymbol{u}}=(\tilde{u}_1,\tilde{u}_2) u ~ = ( u ~ 1 , u ~ 2 ) with the complex number w = u ~ 1 + i u ~ 2 w=\tilde{u}_1+\mathrm{i}\tilde{u}_2 w = u ~ 1 + i u ~ 2 . For N c > 0 N_c>0 N c > 0 complex numbers w 1 , … , w N c w_1,\ldots,w_{N_c} w 1 , … , w N c and integers a ≥ b ≥ 0 a\ge b\ge0 a ≥ b ≥ 0 , the complex moments are c a , b = N c − 1 ∑ i = 1 N c ∣ w i ∣ 2 b w i a − b c_{a,b}=N_c^{-1}\sum_{i=1}^{N_c}\lvert w_i\rvert^{2b}w_i^{a-b} c a , b = N c − 1 ∑ i = 1 N c ∣ w i ∣ 2 b w i a − b .
We fit nine real values, the real and imaginary parts of c 3 , 0 c_{3,0} c 3 , 0 and c 2 , 1 c_{2,1} c 2 , 1 and of c 4 , 0 c_{4,0} c 4 , 0 and c 3 , 1 c_{3,1} c 3 , 1 , together with the real value c 2 , 2 c_{2,2} c 2 , 2 , the average of ∣ w ∣ 4 \lvert w\rvert^4 ∣ w ∣ 4 . These values are invertible linear combinations of the distinct entries ( m 3 , m 4 ) (m_3,m_4) ( m 3 , m 4 ) of M 3 \mathcal{M}_3 M 3 and M 4 \mathcal{M}_4 M 4 , so matching them matches the cubic and quartic moments of Theorem A1. Starting from a grid of orientations, we fit the values of the projected normalized model O Y ~ \mathcal{O}\tilde{\mathcal{Y}} O Y ~ to those of the normalized observations by batched Levenberg–Marquardt refinement [57 ] . We rank the resulting orientations by their residuals. Undoing the normalization converts an orientation O O O into the affine camera
A = Σ U c a l 1 / 2 O Σ Y − 1 / 2 , b = u ˉ c a l − A y ˉ . (78) A=\Sigma_{\mathcal{U}_{\mathrm{cal}}}^{1/2}O\Sigma_{\mathcal{Y}}^{-1/2},
\qquad
\boldsymbol{b}=\bar{\boldsymbol{u}}_{\mathrm{cal}}-A\bar{\boldsymbol{y}}. \tag*{(78)} A = Σ U cal 1/2 O Σ Y − 1/2 , b = u ˉ cal − A y ˉ . (78)
The thin singular-value decomposition of A A A is A = U A diag ( κ 1 , κ 2 ) V A T A = U_A \operatorname{diag}(\kappa_1,\kappa_2)V_A^\mathsf{T} A = U A diag ( κ 1 , κ 2 ) V A T , where U A ∈ R 2 × 2 U_A \in\mathbb{R}^{2 \times2} U A ∈ R 2 × 2 and V A ∈ R 3 × 2 V_A \in\mathbb{R}^{3 \times2} V A ∈ R 3 × 2 have orthonormal columns and κ 1 ≥ κ 2 > 0 \kappa_1 \ge\kappa_2 > 0 κ 1 ≥ κ 2 > 0 . We set O 0 = U A V A T O_0 = U_A V_A^\mathsf{T} O 0 = U A V A T and κ ˉ = ( κ 1 + κ 2 ) / 2 \bar{\kappa} = (\kappa_1+\kappa_2)/2 κ ˉ = ( κ 1 + κ 2 ) /2 . The initial rotation has O 0 O_0 O 0 as its first two rows and their cross product as its third, and the initial translation is t 0 = ( b 1 , b 2 , 1 ) T / κ ˉ t_0 = (b_1,b_2,1)^\mathsf{T}/\bar{\kappa} t 0 = ( b 1 , b 2 , 1 ) T / κ ˉ . From this initial pose, we fit the six pose parameters to L 4 \mathcal{L}_4 L 4 , with the same normalization h h h for the observations and the projections.
The fit weights the nine real values equally, whereas Theorem A1 measures moment differences in the norm of the full symmetric tensors. For two normalized 2D point sets, the differences of their tensors and the differences Δ c a , b \Delta c_{a,b} Δ c a , b of their complex moments are related by
∥ Δ M 3 ∥ 2 = 1 4 ∣ Δ c 3 , 0 ∣ 2 + 3 4 ∣ Δ c 2 , 1 ∣ 2 , \lVert\Delta M_3\rVert^2 = \frac{1}{4}\lvert\Delta c_{3,0}\rvert^2 + \frac{3}{4}\lvert\Delta c_{2,1}\rvert^2, ∥ Δ M 3 ∥ 2 = 4 1 ∣ Δ c 3 , 0 ∣ 2 + 4 3 ∣ Δ c 2 , 1 ∣ 2 ,
∥ Δ M 4 ∥ 2 = 1 8 ∣ Δ c 4 , 0 ∣ 2 + 1 2 ∣ Δ c 3 , 1 ∣ 2 + 3 8 ( Δ c 2 , 2 ) 2 . \lVert\Delta M_4\rVert^2 = \frac{1}{8}\lvert\Delta c_{4,0}\rvert^2 + \frac{1}{2}\lvert\Delta c_{3,1}\rvert^2 + \frac{3}{8}(\Delta c_{2,2})^2. ∥ Δ M 4 ∥ 2 = 8 1 ∣ Δ c 4 , 0 ∣ 2 + 2 1 ∣ Δ c 3 , 1 ∣ 2 + 8 3 ( Δ c 2 , 2 ) 2 .
The residual for the nine fitted values and the tensor residual therefore vanish at the same poses but weight nonzero differences differently. The residual ϵ ^ \hat{\epsilon} ϵ ^ , the error ϵ \epsilon ϵ and the modulus Γ T \Gamma_T Γ T of Theorem A1 use the tensor norm.
With four model points and positive definite covariances, the normalized model is a regular tetrahedron. Each of its 24 orthogonal symmetries Q s Q_s Q s maps it to itself, so the orientations O O O and O Q s OQ_s O Q s give the same moments and the affine fit cannot distinguish them. For each symmetry, we therefore undo the normalization and fit a rigid pose under the pinhole projection model. The CPU search compares all of these poses within its limits, even if an earlier pose is accepted. For any number of model points, if none of the first affine initial poses is accepted, the CPU search continues on successively finer rotation grids. At each grid rotation, the translation is fitted to the centroid and spread of the observations as in Step 1, which keeps every model point in front of the camera. It then tries the remaining affine initial poses.
Planar point models. We treat a 3D point set as numerically planar when the smallest eigenvalue of its covariance is at most 10 − 8 10^{-8} 1 0 − 8 times the largest. Normalization would then divide the coordinate along the plane normal by a near-zero standard deviation, so the affine fit uses plane coordinates. Let Q p ∈ R 3 × 2 Q_p \in\mathbb{R}^{3 \times2} Q p ∈ R 3 × 2 have orthonormal columns along the two leading covariance directions, and let ν \nu ν be their unit normal. We fit A 2 ∈ R 2 × 2 A_2 \in\mathbb{R}^{2 \times2} A 2 ∈ R 2 × 2 from centered plane coordinates to the calibrated observations. Let κ 1 \kappa_1 κ 1 be the largest singular value of A 2 A_2 A 2 , and let a p ∈ R 2 a_p \in\mathbb{R}^2 a p ∈ R 2 satisfy a p a p T = κ 1 2 I 2 − A 2 A 2 T a_p a_p^\mathsf{T} = \kappa_1^2 I_2 - A_2 A_2^\mathsf{T} a p a p T = κ 1 2 I 2 − A 2 A 2 T . We complete A 2 A_2 A 2 to the two matrices A = A 2 Q p T ± a p ν T A = A_2 Q_p^\mathsf{T} \pm a_p \nu^\mathsf{T} A = A 2 Q p T ± a p ν T . Then A A T = κ 1 2 I 2 AA^\mathsf{T} = \kappa_1^2 I_2 A A T = κ 1 2 I 2 , so both completions are scaled-orthographic cameras. They project points that lie exactly in the plane identically, so we compare them by perspective fitting and refinement on the original 3D coordinates.
EM likelihood. At a pose ξ \xi ξ , let N + ( ξ ) N_+(\xi) N + ( ξ ) be the number of model points in front of the camera with finite image projections, s ^ > 0 \hat{s}>0 s ^ > 0 the current noise standard deviation per image coordinate in pixels, p d p_d p d the probability that each of these model points is observed, and n c l n_{\mathrm{cl}} n cl the expected number of outlier observations. Step 3 models each observation u u u as an outlier, uniform on a rectangle D i m D_{\mathrm{im}} D im , with probability η \eta η , or as a projection of one of these model points, chosen uniformly, with isotropic Gaussian noise:
p ( u ) = η ∣ D i m ∣ + 1 − η N + ∑ j : f r o n t N ( u ; p j ( ξ ) , s ^ 2 I 2 ) . (79) p(u) = \frac{\eta}{\lvert D_{\mathrm{im}}\rvert} + \frac{1-\eta}{N_+} \sum_{j:\,\mathrm{front}} \mathcal{N}(u;p_j(\xi),\hat{s}^2 I_2). \tag*{(79)} p ( u ) = ∣ D im ∣ η + N + 1 − η j : front ∑ N ( u ; p j ( ξ ) , s ^ 2 I 2 ) . (79)
The uniform component has support D i m D_{\mathrm{im}} D im , the bounding rectangle of the observations with each side extended at both ends by its length divided by n \sqrt{n} n . A side of zero length is given the length 10 − 300 10^{-300} 1 0 − 300 pixels, so the fixed area ∣ D i m ∣ \lvert D_{\mathrm{im}}\rvert ∣ D im ∣ in square pixels is positive. EM evaluates only the projections near each observation. On the GPU, the neighborhood J i \mathcal{J}_i J i of observation i i i contains the projections within 4.5 s ^ 4.5\hat{s} 4.5 s ^ . Beyond this distance, the Gaussian weight k i j k_{ij} k ij defined below is less than e − 10 e^{-10} e − 10 . On the CPU, it contains the projections within 3 s ^ 3\hat{s} 3 s ^ and also the nearest projection, even when that projection lies outside this radius. For each pair in a neighborhood, let k i j = exp [ − ∥ u i − p j ( ξ ) ∥ 2 / ( 2 s ^ 2 ) ] k_{ij} = \exp[-\lVert u_i-p_j(\xi)\rVert^2/(2\hat{s}^2)] k ij = exp [ − ∥ u i − p j ( ξ ) ∥ 2 / ( 2 s ^ 2 )] . We score poses by the log-likelihood of a Poisson point process, with each inner sum restricted to J i \mathcal{J}_i J i ,
L E M = ∑ i = 1 n log ( n c l ∣ D i m ∣ + p d 2 π s ^ 2 ∑ j ∈ J i k i j ) − n c l − p d N + ( ξ ) . (80) \mathcal{L}_{\mathrm{EM}} = \sum_{i=1}^{n} \log\left(\frac{n_{\mathrm{cl}}}{\lvert D_{\mathrm{im}}\rvert} + \frac{p_d}{2\pi\hat{s}^2}\sum_{j \in\mathcal{J}_i} k_{ij}\right) - n_{\mathrm{cl}} - p_d N_+(\xi). \tag*{(80)} L EM = i = 1 ∑ n log ∣ D im ∣ n cl + 2 π s ^ 2 p d j ∈ J i ∑ k ij − n cl − p d N + ( ξ ) . (80)
The expected number of observations under this process is Λ E M = n c l + p d N + ( ξ ) \Lambda_{\mathrm{EM}} = n_{\mathrm{cl}} + p_d N_+(\xi) Λ EM = n cl + p d N + ( ξ ) . Dividing the argument of the logarithm by Λ E M \Lambda_{\mathrm{EM}} Λ EM gives the mixture density of Eq. Eq. (79) with η = n c l / ( n c l + p d N + ) \eta= n_{\mathrm{cl}}/(n_{\mathrm{cl}}+p_dN_+) η = n cl / ( n cl + p d N + ) , evaluated with the same neighborhoods J i \mathcal{J}_i J i . This division leaves the responsibilities unchanged. The Poisson score is therefore
L E M = ∑ i = 1 n log p ( u i ) + n log Λ E M − Λ E M . (81) \mathcal{L}_{\mathrm{EM}} = \sum_{i=1}^{n} \log p(u_i) + n\log\Lambda_{\mathrm{EM}} - \Lambda_{\mathrm{EM}}. \tag*{(81)} L EM = i = 1 ∑ n log p ( u i ) + n log Λ EM − Λ EM . (81)
The first term measures agreement with observation locations, and the remaining terms compare expected and observed counts. The term n log Λ E M − Λ E M n\log\Lambda_{\mathrm{EM}}-\Lambda_{\mathrm{EM}} n log Λ EM − Λ EM is constant for fixed Λ E M \Lambda_{\mathrm{EM}} Λ EM , but without outliers Λ E M \Lambda_{\mathrm{EM}} Λ EM can vary across poses. Step 3: EM updates. The expectation step assigns to each pair in a neighborhood the posterior probability that observation i i i comes from projection j j j . This responsibility is
r i j = k i j 2 π s ^ 2 n c l / ( ∣ D i m ∣ p d ) + ∑ j ′ ∈ J i k i j ′ . r_{ij} = \frac{k_{ij}}{2\pi\hat{s}^2 n_{\mathrm{cl}}/(\lvert D_{\mathrm{im}}\rvert p_d) + \sum_{j' \in\mathcal{J}_i} k_{ij'}}. r ij = 2 π s ^ 2 n cl / (∣ D im ∣ p d ) + ∑ j ′ ∈ J i k i j ′ k ij .
Pairs outside the neighborhoods have zero responsibility, and the remainder 1 − ∑ j r i j 1-\sum_j r_{ij} 1 − ∑ j r ij is the responsibility of the outlier. uniform component. The sum n m = ∑ i j r i j n_m = \sum_{ij} r_{ij} n m = ∑ ij r ij is the expected number of observations that come from projections. The maximization step updates the pose by a damped Gauss–Newton step on the weighted squared reprojection errors. With the cross-product matrix [ ⋅ ] × [\cdot]_{\times} [ ⋅ ] × , the projection Jacobians in the local pose coordinates δ = ( ω , t ) \delta= (\omega,t) δ = ( ω , t ) are J j = ∂ proj ( R y j + t ) [ − [ R y j ] × I 3 ] J_j = \partial\operatorname{proj}(Ry_j+t)[-[Ry_j]_{\times}\ I_3] J j = ∂ proj ( R y j + t ) [ − [ R y j ] × I 3 ] , and they give
H G N = ∑ i j r i j J j T J j , g G N = ∑ i j r i j J j T ( p j − u i ) . H_{\mathrm{GN}} = \sum_{ij} r_{ij}J_j^{\mathsf{T}}J_j,\qquad g_{\mathrm{GN}} = \sum_{ij} r_{ij}J_j^{\mathsf{T}}(p_j-u_i). H GN = ij ∑ r ij J j T J j , g GN = ij ∑ r ij J j T ( p j − u i ) .
With damping λ G N > 0 \lambda_{\mathrm{GN}}>0 λ GN > 0 and D G N = diag ( diag H G N ) D_{\mathrm{GN}}=\operatorname{diag}(\operatorname{diag}H_{\mathrm{GN}}) D GN = diag ( diag H GN ) , a trial step Δ δ = ( Δ ω T , Δ t T ) T \Delta\delta=(\Delta\omega^{\mathsf{T}},\Delta t^{\mathsf{T}})^{\mathsf{T}} Δ δ = ( Δ ω T , Δ t T ) T solves the damped system ( H G N + λ G N D G N ) Δ δ = − g G N (H_{\mathrm{GN}}+\lambda_{\mathrm{GN}}D_{\mathrm{GN}})\Delta\delta=-g_{\mathrm{GN}} ( H GN + λ GN D GN ) Δ δ = − g GN . It gives the rotation R ′ = exp ( [ Δ ω ] × ) R R'=\exp([\Delta\omega]_{\times})R R ′ = exp ([ Δ ω ] × ) R and, separately, the translation t ′ = t + Δ t t'=t+\Delta t t ′ = t + Δ t . On both the GPU and the CPU, we accept a trial step only if it lowers this weighted cost at fixed responsibilities. Then, at the updated pose ξ ′ \xi' ξ ′ and for n m > 0 n_m>0 n m > 0 , the noise estimate and, with outliers, p d p_d p d and n c l n_{\mathrm{cl}} n cl are
s ^ E M 2 = ∑ i j r i j ∥ u i − p j ( ξ ′ ) ∥ 2 2 n m , \hat{s}_{\mathrm{EM}}^2=\frac{\sum_{ij}r_{ij}\lVert u_i-p_j(\xi')\rVert^2}{2n_m}, s ^ EM 2 = 2 n m ∑ ij r ij ∥ u i − p j ( ξ ′ ) ∥ 2 ,
p d ′ = min { 1 , n m / N + ( ξ ′ ) } , p'_d=\min\{1,n_m/N_+(\xi')\}, p d ′ = min { 1 , n m / N + ( ξ ′ )} ,
n c l ′ = n − n m . n'_{\mathrm{cl}}=n-n_m. n cl ′ = n − n m .
Without outliers, p d = 1 p_d=1 p d = 1 and n c l = 10 − 3 n n_{\mathrm{cl}}=10^{-3}n n cl = 1 0 − 3 n stay fixed, and this small expected outlier count keeps a single gross error from dominating the likelihood. EM starts from the noise level of Tab. A3 and anneals it. While s ^ E M < s ^ \hat{s}_{\mathrm{EM}}<\hat{s} s ^ EM < s ^ , each iteration sets s ^ ′ = min { s ^ E M , 0.9 s ^ } \hat{s}'=\min\{\hat{s}_{\mathrm{EM}},0.9\hat{s}\} s ^ ′ = min { s ^ EM , 0.9 s ^ } . At the first iteration with s ^ E M ≥ s ^ \hat{s}_{\mathrm{EM}}\ge\hat{s} s ^ EM ≥ s ^ , annealing ends, and from then on s ^ ′ = s ^ E M \hat{s}'=\hat{s}_{\mathrm{EM}} s ^ ′ = s ^ EM , bounded on the GPU by the noise ceiling of Tab. A3. When a pose is given without a noise level, such as an initial pose of the CPU search or an alternative of Step 4, EM starts from the root-mean-square distance of the observations to their nearest projections divided by 2 \sqrt{2} 2 , without annealing. EM has converged when annealing has ended, when ∣ s ^ ′ − s ^ ∣ ≤ 10 − 4 s ^ \lvert\hat{s}'-\hat{s}\rvert\le10^{-4}\hat{s} ∣ s ^ ′ − s ^ ∣ ≤ 1 0 − 4 s ^ , and when no projection coordinate moves by more than 10 − 4 s ^ 10^{-4}\hat{s} 1 0 − 4 s ^ between iterations.
Valid fits & ranking. Step 4 considers only valid fits. A valid fit has finite parameters and, without outliers, positive depths for all model points. With outliers, every model point with a positive responsibility must stay in front of the camera, and at least three model points must lie in front of the camera with finite image projections. Before CPU refinement, we rank a stage’s valid fits, without outliers first by convergence and then by likelihood. With outliers, we rank them by likelihood alone and, if some of them have n m ≥ max { 8 , 0.3 min ( n , N ) } n_m\ge\max\{8,0.3\min(n,N)\} n m ≥ max { 8 , 0.3 min ( n , N )} , rank only those, because a fit that explains few observations provides little evidence about the pose. If no fit is accepted, the solver returns the valid fit of highest likelihood and reports that it was not accepted. It reports failure when no valid fit remains.
Association level. For the association level of Definition 2, we use ℓ = min { δ , N + − 1 } \ell=\min\{\delta,N_+-1\} ℓ = min { δ , N + − 1 } neighbors and measure the distances d i , k d_{i,k} d i , k in pixels. For observation i i i , the transformed distance ratio is E i = − ℓ log ( 1 − d i , 1 2 / d i , ℓ + 1 2 ) E_i=-\ell\log(1-d_{i,1}^2/d_{i,\ell+1}^2) E i = − ℓ log ( 1 − d i , 1 2 / d i , ℓ + 1 2 ) . Lemma A7 proves that each E i E_i E i has the unit exponential distribution under the Poisson model of Definition 2. Without outliers, the association level is the lower tail
p a s s o c = Pr [ G Γ ≤ ∑ i = 1 n E i ] , p_{\mathrm{assoc}}=\Pr\left[G_{\Gamma}\le\sum_{i=1}^{n}E_i\right], p assoc = Pr [ G Γ ≤ i = 1 ∑ n E i ] ,
G Γ ∼ Gamma ( n − 3 , 1 ) . G_{\Gamma}\sim\operatorname{Gamma}(n-3,1). G Γ ∼ Gamma ( n − 3 , 1 ) .
The reduced shape n − 3 n-3 n − 3 accounts for the six fitted pose parameters. With outliers, we count the observations with E i ≤ π / 16 E_i\le\pi/16 E i ≤ π /16 . When the nearest projection is much closer than the ( ℓ + 1 ) (\ell+1) ( ℓ + 1 ) th, the ratio E i E_i E i is close to π d i , 1 2 \pi d_{i,1}^2 π d i , 1 2 times the density of projections, so the condition places the nearest projection within about a quarter of the average spacing of the projections, the inverse square root of that density. Under the Poisson model, each observation meets it with probability 1 − e − π / 16 1-e^{-\pi/16} 1 − e − π /16 , and the association level is the upper tail
p a s s o c = Pr [ N B ≥ ∣ { i : E i ≤ π / 16 } ∣ ] , p_{\mathrm{assoc}}=\Pr\left[N_B\ge\left|\{i:E_i\le\pi/16\}\right|\right], p assoc = Pr [ N B ≥ ∣ { i : E i ≤ π /16 } ∣ ] ,
N B ∼ Binomial ( n , 1 − e − π / 16 ) . N_B\sim\operatorname{Binomial}(n,1-e^{-\pi/16}). N B ∼ Binomial ( n , 1 − e − π /16 ) .
These reference distributions treat the E i E_i E i as independent. Neighboring observations share projections, and the pose is fitted to the same observations, so neither distribution is exact (App. A.5). We therefore use p a s s o c p_{\mathrm{assoc}} p assoc as a score and compare it with fixed thresholds far in the tail. A pose that EM has not yet refined has no parameters fitted to the distances. The CPU search and the comparison of alternative poses therefore test such a pose by p a s s o c 0 p_{\mathrm{assoc}}^0 p assoc 0 , which uses the Gamma shape n n n without outliers and equals p a s s o c p_{\mathrm{assoc}} p assoc with outliers.
Step 4: alignment check. The empirical acceptance criteria of Algorithm A1 test whether the observations have projections of their own. When the noise is small compared with the spacing of the projections, an inlier observation at the true pose is typically much closer to its generating projection than to any other. The ratio d i , 1 / d i , 2 d_{i,1}/d_{i,2} d i , 1 / d i , 2 of the distances from observation i i i to its two nearest projections is then typically small, as is the association level. On outlier-free inputs, a further test asks whether the moment residual L 4 \mathcal{L}_4 L 4 is no larger than noise of standard deviation s ^ \hat{s} s ^ would make it at the true pose. Let z j = h ( p j ) z_j=h(p_j) z j = h ( p j ) be the normalized projections, ∂ ϕ 4 \partial\phi_4 ∂ ϕ 4 the 14 × 2 14\times2 14 × 2 Jacobian of ϕ 4 \phi_4 ϕ 4 , and f K = K 11 K 22 f_K=\sqrt{K_{11}K_{22}} f K = K 11 K 22 the geometric-mean focal length. With the radius r c a l r_{\mathrm{cal}} r cal of Sec. 4.1, we approximate the first-order expected value of L 4 \mathcal{L}_4 L 4 at the true pose by
V 4 = s ^ 2 f K 2 r c a l 2 N 2 ∑ j = 1 N ∥ ∂ ϕ 4 ( z j ) ∥ 2 . (82) V_4=\frac{\hat{s}^2}{f_K^2r_{\mathrm{cal}}^2N^2}\sum_{j=1}^{N}\lVert\partial\phi_4(z_j)\rVert^2. \tag*{(82)} V 4 = f K 2 r cal 2 N 2 s ^ 2 j = 1 ∑ N ∥ ∂ ϕ 4 ( z j ) ∥ 2 . (82)
When the focal lengths differ, this approximation uses one noise scale in place of the two calibrated-coordinate noise scales.
Without outliers, we accept a fit in either of two ways. The first requires convergence, at least 90% as many distinct nearest projections as distinct observation locations, and a median ratio d i , 1 / d i , 2 d_{i,1}/d_{i,2} d i , 1 / d i , 2 over the distinct observation locations of at most 0.25. Repeated observations at one location share their nearest projection, so both quantities count each location once. With fewer than thirty distinct observation locations, the first way requires all nearest projections to be distinct and replaces the median ratio by its 75th percentile with the same threshold. The second way accepts a pose whose correspondences are not resolved because the noise is comparable to the spacing of the projections, provided the observations lie far closer to the projections than chance would place them. It applies to converged fits with n ≥ 48 n \ge48 n ≥ 48 . It requires p a s s o c ≤ 10 − 14 p_{\mathrm{assoc}} \le10^{-14} p assoc ≤ 1 0 − 14 , relaxed to 10 − 12 10^{-12} 1 0 − 12 for n ≥ 200 n \ge200 n ≥ 200 , together with L 4 ≤ 9 V 4 L_4 \le9V_4 L 4 ≤ 9 V 4 unless p a s s o c ≤ 10 − 30 p_{\mathrm{assoc}} \le10^{-30} p assoc ≤ 1 0 − 30 .
With outliers, we accept a converged fit with n m ≥ max { 8 , 0.3 min ( n , N ) } n_{\mathrm{m}} \ge\max\{8, 0.3\min(n,N)\} n m ≥ max { 8 , 0.3 min ( n , N )} and p a s s o c ≤ 10 − 9 p_{\mathrm{assoc}} \le10^{-9} p assoc ≤ 1 0 − 9 if its median ratio d i , 1 / d i , 2 d_{i,1}/d_{i,2} d i , 1 / d i , 2 over the matched observations is at most 0.12. Matched observations are those to which the correspondences of Algorithm A1 assign a model point. For n ≥ 500 n \ge500 n ≥ 500 , where dense projections can prevent resolved correspondences, an association level of at most 10 − 100 10^{-100} 1 0 − 100 allows a median ratio of at most 0.25. With or without outliers, when the standard deviation of the image noise is given as an optional input, acceptance also requires s ^ \hat{s} s ^ to be at most three times that value.
Step 4: pose ambiguity resolution. Step 4 compares the selected pose, the accepted fit of highest likelihood, with the alternative poses described in § 3.2 and characterized by Lemma A8. We build the alternative poses from selected model points, which are all model points without outliers and, with outliers, those associated with matched observations. Each alternative keeps the centroid of the selected points fixed in camera coordinates.
The seven half-turn alternatives of Lemma A8 are the poses R H k RH_k R H k for k = 1 , 2 , 3 k=1,2,3 k = 1 , 2 , 3 and D R H k DRH_k D R H k for k = 0 , … , 3 k=0,\ldots,3 k = 0 , … , 3 , with H k H_k H k taken about the principal axes of the selected points. We form them when at least three points are selected and their principal axes can be computed. Each must have valid projections and p a s s o c 0 ≤ 10 − 3 p_{\mathrm{assoc}}^0 \le10^{-3} p assoc 0 ≤ 1 0 − 3 . Without outliers, we omit alternatives whose rotations differ by less than 3 ∘ 3^\circ 3 ∘ from the accepted pose, fit the others to the image moments, and keep those whose residual is at most the larger of the accepted pose’s residual and 9 V 4 9V_4 9 V 4 , with V 4 V_4 V 4 evaluated at the alternative.
The planar alternative uses the plane through the centroid of the selected points whose normal is a covariance eigenvector of smallest eigenvalue. It reflects the selected points through this plane, and it reflects the camera coordinates through the plane that contains the centroid and is parallel to the image plane. The two reflections compose to the rotation D R H k DRH_k D R H k , with H k H_k H k the half-turn about that eigenvector, so this alternative starts from the half-turn alternative that Lemma A8(c) identifies for planar models. It differs from that alternative in its refinement. For a planar model, the accepted pose and its alternative give nearly the same image with the same correspondences, so we first fit this alternative under perspective projection to the correspondences of the accepted pose. We then refine it by EM only if its projections are valid, its rotation differs by more than 3 ∘ 3^\circ 3 ∘ from the accepted pose, and p a s s o c 0 ≤ 10 − 3 p_{\mathrm{assoc}}^0 \le10^{-3} p assoc 0 ≤ 1 0 − 3 .
After EM, we discard alternatives of either kind whose rotation differs by less than 3 ∘ 3^\circ 3 ∘ from any accepted pose. A remaining alternative replaces the selected pose only if it satisfies the acceptance criteria and has higher likelihood. The selected pose then gives the correspondences as in Algorithm A1.
Correspondences. At the returned pose, each matched observation corresponds to its nearest projection, and every other observation is an outlier. When several observations share a nearest projection, only the closest keeps it. With outliers, an accepted pose with at least six matches is refitted to these correspondences by least squares, and the refit replaces it if the matched points stay in front of the camera.
Search stages. Algorithm A1 expands Algorithm 1 with the search stages of the point-set benchmarks, the division of work between the GPU and the CPU, the early exits and the acceptance criteria, and Table A3 gives the settings of each stage. Inputs must have finite coordinates and meet the requirements of § 2 and the point-count limits in the table. The table states noise levels in units of ρ r m s \rho_{\mathrm{rms}} ρ rms , the root-mean-square distance of the observations from their centroid, in pixels. The search stops after the first stage that yields an accepted fit. A fit that is not accepted but has an association level at most 10 − 100 10^{-100} 1 0 − 100 ends the GPU stages early, because its pose already places the observations far closer to the projections than chance would. Without outliers, the CPU search then follows. It defers initial poses with p a s s o c 0 > 10 − 3 p_{\mathrm{assoc}}^0 > 10^{-3} p assoc 0 > 1 0 − 3 and, if none of its other initial poses reaches EM, refines up to three deferred poses from a larger noise level.
Complexity. For fixed numbers of features, candidates and iterations, computing the features costs O ( n + N ) O(n+N) O ( n + N ) . A GPU EM iteration then costs O ( n + N + N p a i r ) O(n+N+N_{\mathrm{pair}}) O ( n + N + N pair ) , where N p a i r N_{\mathrm{pair}} N pair counts the observation–projection pairs that the GPU examines in a grid of square image cells and can reach O ( n N ) O(nN) O ( n N ) . Sorting the observations into these cells adds O ( n log n ) O(n\log n) O ( n log n ) , and GPU storage is O ( n + N ) O(n+N) O ( n + N ) for fixed batch size and grid limits. CPU refinement takes O ( N log N + n N ) O(N\log N+nN) O ( N log N + n N ) time per iteration and can store O ( n N ) O(nN) O ( n N ) neighbor pairs. With fixed search and iteration limits, the solver therefore has worst-case time O ( n N + ( n + N ) log ( n + N ) ) O(nN+(n+N)\log(n+N)) O ( n N + ( n + N ) log ( n + N )) and space O ( n N ) O(nN) O ( n N ) .
Algorithm A1: MomentPnP , expanding Algorithm 1; GPU lines handle all poses at once, CPU lines one at a time
Input: Observations U = { u i } i = 1 n \mathcal{U} = \{u_i\}_{i=1}^{n} U = { u i } i = 1 n in pixels, model Y = { y j } j = 1 N \mathcal{Y} = \{y_j\}_{j=1}^{N} Y = { y j } j = 1 N , intrinsics K K K , whether outliers may be present, optional noise level s s s
Output: Pose ξ ^ \hat{\xi} ξ ^ , noise estimate s ^ \hat{s} s ^ , correspondences σ ^ \hat{\sigma} σ ^ and the flag a a a , true if ξ ^ \hat{\xi} ξ ^ is accepted; no pose if no valid fit exists
// ObservedMoments (CPU); on complete inputs, the moments μ U = 1 n ∑ i ϕ 4 ( h ( u i ) ) \mu_{\mathcal{U}} = \frac{1}{n}\sum_i \phi_4(h(u_i)) μ U = n 1 ∑ i ϕ 4 ( h ( u i )) replace both averages
1: μ U , F ← 1 n ∑ i ϕ F ( h F ( u i ) ) \mu_{\mathcal{U},F} \leftarrow\frac{1}{n}\sum_i \phi_F(h_F(u_i)) μ U , F ← n 1 ∑ i ϕ F ( h F ( u i )) , μ U , 0 ← \mu_{\mathcal{U},0} \leftarrow μ U , 0 ← average of ϕ F \phi_F ϕ F over the rectangle D F D_F D F that bounds the calibrated observations
// Accepted(ξ \xi ξ ) (CPU), with d i , 1 ≤ d i , 2 ≤ ⋯ d_{i,1} \le d_{i,2} \le\cdots d i , 1 ≤ d i , 2 ≤ ⋯ the distances from u i u_i u i to the projections, E i = − ℓ log ( 1 − d i , 1 2 / d i , ℓ + 1 2 ) E_i = -\ell\log(1-d_{i,1}^2/d_{i,\ell+1}^2) E i = − ℓ log ( 1 − d i , 1 2 / d i , ℓ + 1 2 ) , n m n_m n m from Step 3 and V 4 V_4 V 4 the expected L 4 L_4 L 4 under noise s ^ \hat{s} s ^ at the true pose (App. B.1); a fit is valid when its parameters are finite and its matched points have positive depth
2: Without outliers, p a s s o c ← Pr [ Gamma ( n − 3 , 1 ) ≤ ∑ i E i ] p_{\mathrm{assoc}} \leftarrow\Pr[\operatorname{Gamma}(n-3,1) \le\sum_i E_i] p assoc ← Pr [ Gamma ( n − 3 , 1 ) ≤ ∑ i E i ] , and a valid converged fit is accepted if 90% of the observations have distinct nearest projections and median i d i , 1 / d i , 2 ≤ 0.25 \operatorname{median}_i d_{i,1}/d_{i,2} \le0.25 median i d i , 1 / d i , 2 ≤ 0.25 , or if n ≥ 48 n \ge48 n ≥ 48 , p a s s o c ≤ 10 − 14 p_{\mathrm{assoc}} \le10^{-14} p assoc ≤ 1 0 − 14 (10 − 12 10^{-12} 1 0 − 12 for n ≥ 200 n \ge200 n ≥ 200 ) and L 4 ≤ 9 V 4 L_4 \le9V_4 L 4 ≤ 9 V 4 or p a s s o c ≤ 10 − 30 p_{\mathrm{assoc}} \le10^{-30} p assoc ≤ 1 0 − 30
3: With outliers, p a s s o c ← Pr [ Binomial ( n , 1 − e − π / 16 ) ≥ ∣ { i : E i ≤ π / 16 } ∣ ] p_{\mathrm{assoc}} \leftarrow\Pr[\operatorname{Binomial}(n,1-e^{-\pi/16}) \ge|\{i:E_i \le\pi/16\}|] p assoc ← Pr [ Binomial ( n , 1 − e − π /16 ) ≥ ∣ { i : E i ≤ π /16 } ∣ ] , and a valid converged fit is accepted if n m ≥ max { 8 , 0.3 min ( n , N ) } n_m \ge\max\{8,0.3\min(n,N)\} n m ≥ max { 8 , 0.3 min ( n , N )} , p a s s o c ≤ 10 − 9 p_{\mathrm{assoc}} \le10^{-9} p assoc ≤ 1 0 − 9 and median i \operatorname{median}_i median i matched d i , 1 / d i , 2 ≤ 0.12 d_{i,1}/d_{i,2} \le0.12 d i , 1 / d i , 2 ≤ 0.12 (0.25 0.25 0.25 if n ≥ 500 n \ge500 n ≥ 500 and p a s s o c ≤ 10 − 100 p_{\mathrm{assoc}} \le10^{-100} p assoc ≤ 1 0 − 100 )
4: Any other fit is not accepted, in particular an invalid or unconverged fit, or one with s ^ > 3 s \hat{s} > 3s s ^ > 3 s when s s s is given
5: if the inputs are outlier-free then S ← S \leftarrow S ← GPU stages on the 800 × 48 800 \times48 800 × 48 and then the 3,200 × 96 3{,}200 \times96 3 , 200 × 96 rotation grid with η 0 = 0 \eta_0 = 0 η 0 = 0 , then the CPU search
6: else S ← S \leftarrow S ← GPU stages from the estimated outlier fraction η 0 \eta_0 η 0 , from η 0 = 0 , 0.15 , 0.30 , 0.45 , 0.60 \eta_0 = 0, 0.15, 0.30, 0.45, 0.60 η 0 = 0 , 0.15 , 0.30 , 0.45 , 0.60 , and on the 800 × 48 800 \times48 800 × 48 grid with outlier-corrected and cross-correlated translations
7: Π ← ∅ \Pi\leftarrow\varnothing Π ← ∅ , a ← a \leftarrow a ← false // the stages of S S S are the levels of WidenCameraSearch in Algorithm 1
8: foreach stage in S S S , while a a a is false do
9: if this is a later GPU stage and p a s s o c ( ξ ^ ) ≤ 10 − 100 p_{\mathrm{assoc}}(\hat{\xi}) \le10^{-100} p assoc ( ξ ^ ) ≤ 1 0 − 100 then skip it: ξ ^ \hat{\xi} ξ ^ already places the observations far closer to the projections than chance
10: if this is a GPU stage then
// Step 1, CameraInitialization (GPU)
11: Π ′ ← \Pi' \leftarrow Π ′ ← for each grid rotation R R R , the pose whose translation matches the projected centroid and spread of the model to those of the observations, both corrected for a fraction η 0 \eta_0 η 0 of uniform outliers; keep those of lowest L F ( ξ , η ^ ( ξ ) ) L_F(\xi,\hat{\eta}(\xi)) L F ( ξ , η ^ ( ξ )) at K F = 4 K_F = 4 K F = 4 with distinct orientations
// Step 2, MomentMatching (GPU); on complete inputs, η = 0 \eta= 0 η = 0 and L 4 ( ξ ) = ∥ μ U − μ Y ( ξ ) ∥ 2 L_4(\xi) = \|\mu_{\mathcal{U}} - \mu_{\mathcal{Y}}(\xi)\|^2 L 4 ( ξ ) = ∥ μ U − μ Y ( ξ ) ∥ 2 replaces L F L_F L F
12: for K F = 4 , 8 , … , 20 K_F = 4,8,\ldots,20 K F = 4 , 8 , … , 20 and each ξ ∈ Π ′ \xi\in\Pi' ξ ∈ Π ′ do
13: d ← μ U , F − μ Y , F ( ξ ) d \leftarrow\mu_{\mathcal{U},F} - \mu_{\mathcal{Y},F}(\xi) d ← μ U , F − μ Y , F ( ξ ) , d 0 ← μ U , 0 − μ Y , 0 ( ξ ) d_0 \leftarrow\mu_{\mathcal{U},0} - \mu_{\mathcal{Y},0}(\xi) d 0 ← μ U , 0 − μ Y , 0 ( ξ ) , η ^ ← min { 0.75 , max { 0 , ⟨ d , d 0 ⟩ / ∥ d 0 ∥ 2 } } \hat{\eta} \leftarrow\min\{0.75,\max\{0,\langle d,d_0\rangle/\|d_0\|^2\}\} η ^ ← min { 0.75 , max { 0 , ⟨ d , d 0 ⟩ /∥ d 0 ∥ 2 }}
14: ξ ← \xi\leftarrow ξ ← Levenberg–Marquardt on L F ( ξ , η ^ ) = ∥ d − η ^ d 0 ∥ 2 L_F(\xi,\hat{\eta}) = \|d-\hat{\eta}d_0\|^2 L F ( ξ , η ^ ) = ∥ d − η ^ d 0 ∥ 2 ; after each cutoff, Π ′ ← \Pi' \leftarrow Π ′ ← the fits of lowest L F L_F L F with distinct orientations
// Step 3, EM refinement (GPU, all of Π ′ \Pi' Π ′ in one batch), under the mixture
p ( u ) = η / ∣ D i m ∣ + ( 1 − η ) 1 n + ∑ j N ( u ; p j ( ξ ) , Σ j 2 ) p(u) = \eta/|D_{\mathrm{im}}| + (1-\eta) \frac{1}{n_+}\sum_j \mathcal{N}(u;p_j(\xi),\Sigma_j^2) p ( u ) = η /∣ D im ∣ + ( 1 − η ) n + 1 ∑ j N ( u ; p j ( ξ ) , Σ j 2 )
15: repeat
16: r i j ← r_{ij} \leftarrow r ij ← posterior probability that u i u_i u i comes from p j ( ξ ) p_j(\xi) p j ( ξ ) , n m ← ∑ i j r i j n_m \leftarrow\sum_{ij} r_{ij} n m ← ∑ ij r ij , for each ξ ∈ Π ′ \xi\in\Pi' ξ ∈ Π ′ // E-step
17: ξ ← \xi\leftarrow ξ ← Gauss–Newton step on ∑ i j r i j ∥ p j ( ξ ) − u i ∥ 2 + ∑ i j r i j ∥ u i − p j ( ξ ) ∥ 2 / ( 2 n m ) \sum_{ij} r_{ij}\|p_j(\xi)-u_i\|^2 + \sum_{ij} r_{ij}\|u_i-p_j(\xi)\|^2/(2n_m) ∑ ij r ij ∥ p j ( ξ ) − u i ∥ 2 + ∑ ij r ij ∥ u i − p j ( ξ ) ∥ 2 / ( 2 n m ) , update η \eta η // M-step
18: until the most likely fit of the batch has converged or 120 iterations have run, which also stops the other fits
19: Drop invalid fits and fits whose s s s reached 0.999 times its ceiling (they have not found the observations); for the rest, L E M ( ξ ) ← ∑ i L E M ( ξ ) log p ( u i ) + n log Λ − Λ \mathcal{L}_{\mathrm{EM}}(\xi) \leftarrow\sum_i \mathcal{L}_{\mathrm{EM}}(\xi) \log p(u_i) + n \log\Lambda- \Lambda L EM ( ξ ) ← ∑ i L EM ( ξ ) log p ( u i ) + n log Λ − Λ , with Λ \Lambda Λ the expected number of observations
20: Rank the fits by L E M \mathcal{L}_{\mathrm{EM}} L EM , converged fits first without outliers, and among those with n m ≥ max { 8 , 0.3 min ( n , N ) } n_m \ge\max\{8,0.3\min(n,N)\} n m ≥ max { 8 , 0.3 min ( n , N )} , if any, with outliers
// Step 3 continued, and Step 4, AlignmentCheck (CPU)
21: foreach ξ \xi ξ among the three highest-ranked fits do
22: ξ ← \xi\leftarrow ξ ← EM from ξ \xi ξ on the CPU until convergence or the iteration limit, Π ← Π ∪ { ξ } \Pi\leftarrow\Pi\cup\{\xi\} Π ← Π ∪ { ξ } ; if Accepted(ξ \xi ξ ) then leave this loop
23: else
// CPU search, on outlier-free inputs only; its fits replace the earlier ones
24: Π ← \Pi\leftarrow Π ← CPU EM refinements (Step 3) of the affine initial poses of Theorem A1, then of three rotation grids with translations as in Step 1, then of the other affine poses, deferring those with p a s s o c > 10 − 3 p_{\mathrm{assoc}} > 10^{-3} p assoc > 1 0 − 3 , until one is accepted (all 24 symmetric poses of four nonplanar points)
25: if no pose reached EM then refine up to three deferred poses, starting EM from a larger noise level
26: ξ ^ ← \hat{\xi} \leftarrow ξ ^ ← the most likely accepted pose in Π \Pi Π , or the most likely pose in Π \Pi Π if none is accepted, a ← a \leftarrow a ← Accepted(ξ ^ \hat{\xi} ξ ^ )
27: if Π = ∅ \Pi= \varnothing Π = ∅ then return no pose, because no valid fit was found
28: if a a a then
// Step 4, Alternatives (CPU); H k H_k H k and D D D are the model and camera half-turns of Sec. 4.2
29: Π ′ ← \Pi' \leftarrow Π ′ ← EM refinements (Step 3, CPU) of the poses with rotations R H 1 RH_1 R H 1 , R H 2 RH_2 R H 2 , R H 3 RH_3 R H 3 , D R DR D R , D R H 1 DRH_1 D R H 1 , D R H 2 DRH_2 D R H 2 , D R H 3 DRH_3 D R H 3 and the same centroid, for the rotation R R R of ξ ^ \hat{\xi} ξ ^ , when they have valid projections, p a s s o c 0 ≤ 10 − 3 p_{\mathrm{assoc}}^0 \le10^{-3} p assoc 0 ≤ 1 0 − 3 and rotations more than 3 ∘ 3^\circ 3 ∘ from ξ ^ \hat{\xi} ξ ^
ξ ^ ← \hat{\xi} \leftarrow ξ ^ ← the most likely accepted pose in { ξ ^ } ∪ Π ′ \{\hat{\xi}\} \cup\Pi' { ξ ^ } ∪ Π ′
// Correspondences (CPU)
31: For i = 1 , … , n i = 1,\ldots,n i = 1 , … , n , σ ^ ( i ) ← arg min j ∥ u i − p j ( ξ ^ ) ∥ \hat{\sigma}(i) \leftarrow\arg\min_j \|u_i-p_j(\hat{\xi})\| σ ^ ( i ) ← arg min j ∥ u i − p j ( ξ ^ ) ∥ if that projection explains u i u_i u i better than the outlier term, else σ ^ ( i ) ← ∅ \hat{\sigma}(i) \leftarrow\varnothing σ ^ ( i ) ← ∅ ; keep only the closest of several observations sharing a model point
32: if there are outliers, a a a holds and at least six matches remain then keep the least-squares refit of ξ ^ \hat{\xi} ξ ^ to σ ^ \hat{\sigma} σ ^ if its matched points stay in front of the camera
33: return ( ξ ^ , s ^ , σ ^ , a ) (\hat{\xi},\hat{s},\hat{\sigma},a) ( ξ ^ , s ^ , σ ^ , a ) // if a a a is false, ξ ^ \hat{\xi} ξ ^ is the most likely fit and is reported as not accepted
Measured time and memory. The complexity bounds above are worst cases. In Figure A1 , we measure time and memory on the rendered ETH3D scans of App. § 10 .§ 4 , from 1,000 to one million model points, with every model point or a quarter of them observed.
GPU memory grows linearly with the number of model points. From 64,000 points on, the GPU allocator memory grows by about 12.4 KiB per added point when every model point is observed and reaches 11.87 GiB at one million points. With a quarter observed, it holds about 115 KiB per model point, or 7.0 GiB at 64,000 points. CPU memory stays small, and the peak resident memory of the four one-million-point runs that recorded it is 0.81 to 1.11 GiB. Time grows faster than memory. Up to 16,000 points, the median time per frame grows about linearly with the number of points. From 64,000 to 256,000 points, it grows about as the square of that number. This growth is consistent with the term N p a i r N_{\mathrm{pair}} N pair , because denser projections in a fixed image place more of them in each grid cell around an observation. At one million points, the seven completed views take 689 to 8,735 s, with a median of 1,244 s.
Where the solver stops. No run of MomentPnP exhausted memory. Up to one million points, the only failure is one Courtyard view that returned no pose within its 1,800-second limit (App. § 10 .§ 4 ). The input size is therefore limited by time. Linear extrapolation of the measured memory places the limit of a 24 GiB GPU near two million points with every model point observed and near 200,000 model points with a quarter observed. The learned baselines, by contrast, run out of memory from 16,000 or 64,000 points (Table A17 ).
Edge observations
In the study of App. § 10 .8, we reduce a red–green–blue (RGB) image and a computer-aided design (CAD) mesh to straight edges. The image edges are line segments found by a detector. In the mesh, an edge is a crease when the normals of its two adjacent faces differ by at least 30 ∘ 30^\circ 3 0 ∘ or when it has a single adjacent face. An edge is a contour at a candidate pose when one adjacent face is turned toward the camera and the other away from it. At a candidate pose, the model edges are the visible creases and contours. We extract the image edges and the model edges independently. The inputs thus contain no pairing between image and model edges, and the two edge sets may sample the same visible boundary at different densities.
The method uses these edges in two forms. For moment matching, it averages the features over image arclength, the length along the edges in the image, rather than over points. Perspective projection shortens the parts of a 3D edge that are farther from the camera, so a uniform average along the 3D edge would not match an average along the detected image edge. Lemma A19 shows that the two image-arclength feature averages agree at the true pose when the detected edges coincide with the visible projected edges. For Step 3, EM needs point observations. We place the 2D observations at equal image-arclength spacing along the detected image edges. At each initial pose, we place points at equal image-arclength spacing along the visible projected model edges and map each one back to its 3D point on the edge by inverting the change of parameter in the proof of Lemma A19. These 3D points form the model points and stay fixed during refinement. Because the two sets of samples are placed independently, an observation and a model point need not correspond to the same physical point, and EM treats them as two unordered point sets.
Lemma A19 (Feature agreement along projected edges). For a pose ξ = ( R , t ) \xi=(R,t) ξ = ( R , t ) , let f ξ ( y ) = proj ( R y + t ) f_\xi(y)=\operatorname{proj}(Ry+t) f ξ ( y ) = proj ( R y + t ) be the projection of a 3D point y y y . At a candidate pose ξ \xi ξ , let { ( y j 0 , y j 1 ) } j = 1 J \{(y_j^0,y_j^1)\}_{j=1}^{J} {( y j 0 , y j 1 ) } j = 1 J be a finite family of visible straight 3D edge pieces whose endpoints lie in front of the camera. Their image endpoints are p j 0 = f ξ ( y j 0 ) p_j^0=f_\xi(y_j^0) p j 0 = f ξ ( y j 0 ) and p j 1 = f ξ ( y j 1 ) p_j^1=f_\xi(y_j^1) p j 1 = f ξ ( y j 1 ) , and their projected lengths are λ j = ∥ p j 1 − p j 0 ∥ \lambda_j=\lVert p_j^1-p_j^0\rVert λ j = ∥ p j 1 − p j 0 ∥ . For a fixed measurable image feature map ϕ : R 2 → R d ϕ \phi:\mathbb{R}^2\to\mathbb{R}^{d_\phi} ϕ : R 2 → R d ϕ that is bounded on both edge sets, and for a positive total projected length, we define
μ E ( ξ ) = ∑ j λ j ∫ 0 1 ϕ ( ( 1 − r E ) p j 0 + r E p j 1 ) d r E ∑ j λ j . (83) \mu_E(\xi)=\frac{\sum_j\lambda_j\int_0^1\phi\left((1-r_E)p_j^0+r_Ep_j^1\right)\,\mathrm{d}r_E}{\sum_j\lambda_j}. \tag*{(83)} μ E ( ξ ) = ∑ j λ j ∑ j λ j ∫ 0 1 ϕ ( ( 1 − r E ) p j 0 + r E p j 1 ) d r E . (83) Let C E C_E C E be the union of the independently detected image edges, with positive finite arclength. With the arclength element d λ \mathrm{d}\lambda d λ in the image, let μ C = ∫ C E ϕ d λ ∫ C E d λ \mu_C=\frac{\int_{C_E}\phi\,\mathrm{d}\lambda}{\int_{C_E}\mathrm{d}\lambda} μ C = ∫ C E d λ ∫ C E ϕ d λ be the average of ϕ \phi ϕ over C E C_E C E . If, at the true pose ξ ⋆ \xi^\star ξ ⋆ , the union of the visible projected edge pieces equals C E C_E C E up to a set of zero arclength, and distinct projected pieces overlap only on sets of zero arclength, then μ E ( ξ ⋆ ) = μ C \mu_E(\xi^\star)=\mu_C μ E ( ξ ⋆ ) = μ C . The equality holds for the fourteen raw monomials through degree four and for bounded Fourier features, without point assignments or shared edge identities.
Proof. We show that each term of the numerator in Eq. (83) is the arclength integral of ϕ \phi ϕ over one projected piece. For one 3D segment y ( r E ) = ( 1 − r E ) y j 0 + r E y j 1 y(r_E)=(1-r_E)y_j^0+r_Ey_j^1 y ( r E ) = ( 1 − r E ) y j 0 + r E y j 1 , we write x j 3 0 = ( R y j 0 + t ) 3 > 0 x_{j3}^0=(Ry_j^0+t)_3>0 x j 3 0 = ( R y j 0 + t ) 3 > 0 and x j 3 1 = ( R y j 1 + t ) 3 > 0 x_{j3}^1=(Ry_j^1+t)_3>0 x j 3 1 = ( R y j 1 + t ) 3 > 0 for the depths of its endpoints in camera coordinates. Its image is
f ξ ( y ( r E ) ) = ( 1 − α E ( r E ) ) p j 0 + α E ( r E ) p j 1 , f_\xi(y(r_E))=(1-\alpha_E(r_E))p_j^0+\alpha_E(r_E)p_j^1, f ξ ( y ( r E )) = ( 1 − α E ( r E )) p j 0 + α E ( r E ) p j 1 ,
α E ( r E ) = r E x j 3 1 ( 1 − r E ) x j 3 0 + r E x j 3 1 , \alpha_E(r_E)=\frac{r_Ex_{j3}^1}{(1-r_E)x_{j3}^0+r_Ex_{j3}^1}, α E ( r E ) = ( 1 − r E ) x j 3 0 + r E x j 3 1 r E x j 3 1 ,
whose derivative is α E ′ ( r E ) = x j 3 0 x j 3 1 ( ( 1 − r E ) x j 3 0 + r E x j 3 1 ) 2 > 0 \alpha_E'(r_E)=\frac{x_{j3}^0x_{j3}^1}{((1-r_E)x_{j3}^0+r_Ex_{j3}^1)^2}>0 α E ′ ( r E ) = (( 1 − r E ) x j 3 0 + r E x j 3 1 ) 2 x j 3 0 x j 3 1 > 0 . The map α E \alpha_E α E is therefore an increasing bijection of [ 0 , 1 ] [0,1] [ 0 , 1 ] , so the projected piece is the image segment from p j 0 p_j^0 p j 0 to p j 1 p_j^1 p j 1 , traversed once. Along this segment, the arclength element is λ j d α E \lambda_j\,\mathrm{d}\alpha_E λ j d α E , and the arclength integral of ϕ \phi ϕ over the piece is λ j ∫ 0 1 ϕ ( ( 1 − α E ) p j 0 + α E p j 1 ) d α E \lambda_j\int_0^1\phi((1-\alpha_E)p_j^0+\alpha_Ep_j^1)\,\mathrm{d}\alpha_E λ j ∫ 0 1 ϕ (( 1 − α E ) p j 0 + α E p j 1 ) d α E , the j j j th term of the numerator. A piece of zero projected length contributes zero. Because distinct projected pieces overlap only on sets of zero arclength, the sum of these integrals counts each point of the projected union once, except on a set of zero arclength. At the true pose, the projected union is C E C_E C E , so the numerator equals ∫ C E ϕ d λ \int_{C_E}\phi\,\mathrm{d}\lambda ∫ C E ϕ d λ and the denominator equals the length of C E C_E C E , which gives the equality. □ \square □
Detected edges in general differ from the projected ones. Lemma A20 bounds the effect of three departures. These are displacement by the detector, edges present in only one of the two sets, and quadrature error. The lemma then turns the resulting feature discrepancy into a bound on pose error near the true pose. A coupling makes the comparison precise without pairing edges. It is a joint distribution of a point on the true projected edges and a point on the observed image edges, each distributed uniformly with respect to image arclength.
Lemma A20 (Edge mismatch & local pose stability). We couple the arclength distributions on the true projected edges and the observed image edges, using a joint distribution with these two marginals. We assume that the coupling places at least a fraction 0 ≤ γ E ≤ 1 0\leq\gamma_E\leq1 0 ≤ γ E ≤ 1 of its mass on pairs at image distance at most δ E ≥ 0 \delta_E\geq0 δ E ≥ 0 . We let the fixed measurable feature map have norm bound B ≥ 0 B\geq0 B ≥ 0 and Lipschitz constant Lip ( ϕ ) ≥ 0 \operatorname{Lip}(\phi)\geq0 Lip ( ϕ ) ≥ 0 on a bounded image region containing the observed edges and the projected edges at all poses in a fixed neighborhood of ξ ⋆ \xi^\star ξ ⋆ . We let μ ^ E \hat{\mu}_E μ ^ E and μ ^ C \hat{\mu}_C μ ^ C be quadrature
approximations of μ E \mu_E μ E and μ C \mu_C μ C , and we let ϵ q \epsilon_q ϵ q bound their combined image and model quadrature error throughout that neighborhood. The empirical feature discrepancy at the true pose then satisfies
∥ μ ^ E ( ξ ⋆ ) − μ ^ C ∥ ≤ ϵ E = γ E Lip ( ϕ ) δ E + 2 B ( 1 − γ E ) + ϵ q . (84) \lVert\hat{\mu}_E(\xi^\star)-\hat{\mu}_C\rVert\le\epsilon_E=\gamma_E\operatorname{Lip}(\phi)\delta_E+2B(1-\gamma_E)+\epsilon_q. \tag*{(84)} ∥ μ ^ E ( ξ ⋆ ) − μ ^ C ∥ ≤ ϵ E = γ E Lip ( ϕ ) δ E + 2 B ( 1 − γ E ) + ϵ q . (84)
For a pose chart ξ ( δ ) \xi(\delta) ξ ( δ ) in fixed six-dimensional local coordinates, we define g E ( δ ) = μ E ( ξ ( δ ) ) g_E(\delta)=\mu_E(\xi(\delta)) g E ( δ ) = μ E ( ξ ( δ )) . We assume that g E g_E g E is continuously differentiable and that its Jacobian J ⋆ ∈ R d ϕ × 6 J^\star\in\mathbb{R}^{d_\phi\times6} J ⋆ ∈ R d ϕ × 6 at the true pose coordinates δ ⋆ \delta^\star δ ⋆ has full column rank, with least singular value κ J > 0 \kappa_J>0 κ J > 0 . An estimate δ ^ \hat{\delta} δ ^ sufficiently close to the true pose, with fitted residual ϵ ^ E = ∥ μ ^ E ( ξ ( δ ^ ) ) − μ ^ C ∥ \hat{\epsilon}_E=\lVert\hat{\mu}_E(\xi(\hat{\delta}))-\hat{\mu}_C\rVert ϵ ^ E = ∥ μ ^ E ( ξ ( δ ^ )) − μ ^ C ∥ , satisfies
∥ δ ^ − δ ⋆ ∥ ≤ 2 ( ϵ ^ E + ϵ E ) κ J . (85) \lVert\hat{\delta}-\delta^\star\rVert\le\frac{2(\hat{\epsilon}_E+\epsilon_E)}{\kappa_J}. \tag*{(85)} ∥ δ ^ − δ ⋆ ∥ ≤ κ J 2 ( ϵ ^ E + ϵ E ) . (85)
Proof. We first bound the difference of the exact averages through the coupling. On at least a fraction γ E \gamma_E γ E of the coupled mass, the two points lie within δ E \delta_E δ E of each other, so their features differ by at most min ( Lip ( ϕ ) δ E , 2 B ) \min(\operatorname{Lip}(\phi)\delta_E,2B) min ( Lip ( ϕ ) δ E , 2 B ) . On the remaining mass, the features differ by at most 2 B 2B 2 B . Since min ( Lip ( ϕ ) δ E , 2 B ) ≤ 2 B \min(\operatorname{Lip}(\phi)\delta_E,2B)\le2B min ( Lip ( ϕ ) δ E , 2 B ) ≤ 2 B , a larger γ E \gamma_E γ E can only lower this bound. Integration over the coupling therefore bounds the difference of the exact averages by ϵ l a w = γ E Lip ( ϕ ) δ E + 2 B ( 1 − γ E ) \epsilon_{\mathrm{law}}=\gamma_E\operatorname{Lip}(\phi)\delta_E+2B(1-\gamma_E) ϵ law = γ E Lip ( ϕ ) δ E + 2 B ( 1 − γ E ) . Adding the quadrature errors by the triangle inequality gives Eq. (84) . At the estimate, the triangle inequality through μ ^ E ( ξ ( δ ^ ) ) \hat{\mu}_E(\xi(\hat{\delta})) μ ^ E ( ξ ( δ ^ )) , then μ ^ C \hat{\mu}_C μ ^ C and then μ C \mu_C μ C gives ∥ g E ( δ ^ ) − g E ( δ ⋆ ) ∥ ≤ ϵ ^ E + ϵ q + ϵ l a w = ϵ ^ E + ϵ E \lVert g_E(\hat{\delta})-g_E(\delta^\star)\rVert\le\hat{\epsilon}_E+\epsilon_q+\epsilon_{\mathrm{law}}=\hat{\epsilon}_E+\epsilon_E ∥ g E ( δ ^ ) − g E ( δ ⋆ )∥ ≤ ϵ ^ E + ϵ q + ϵ law = ϵ ^ E + ϵ E .
We next show that g E g_E g E separates poses near δ ⋆ \delta^\star δ ⋆ . We write J ( δ ) J(\delta) J ( δ ) for the Jacobian of g E g_E g E , so J ⋆ = J ( δ ⋆ ) J^\star=J(\delta^\star) J ⋆ = J ( δ ⋆ ) . By continuity, J ( δ ) J(\delta) J ( δ ) stays within κ J / 2 \kappa_J/2 κ J /2 of J ⋆ J^\star J ⋆ in operator norm on some convex coordinate neighborhood of δ ⋆ \delta^\star δ ⋆ . For Δ δ = δ − δ ⋆ \Delta\delta=\delta-\delta^\star Δ δ = δ − δ ⋆ , the fundamental theorem of calculus gives
g E ( δ ) − g E ( δ ⋆ ) = J ⋆ Δ δ + ∫ 0 1 ( J ( δ ⋆ + ν Δ δ ) − J ⋆ ) Δ δ d ν . g_E(\delta)-g_E(\delta^\star)=J^\star\Delta\delta+\int_0^1\left(J(\delta^\star+\nu\Delta\delta)-J^\star\right)\Delta\delta\,\mathrm{d}\nu. g E ( δ ) − g E ( δ ⋆ ) = J ⋆ Δ δ + ∫ 0 1 ( J ( δ ⋆ + ν Δ δ ) − J ⋆ ) Δ δ d ν .
The first term has norm at least κ J ∥ Δ δ ∥ \kappa_J\lVert\Delta\delta\rVert κ J ∥ Δ δ ∥ , and the integral has norm at most ( κ J / 2 ) ∥ Δ δ ∥ (\kappa_J/2)\lVert\Delta\delta\rVert ( κ J /2 ) ∥ Δ δ ∥ . Thus ∥ g E ( δ ) − g E ( δ ⋆ ) ∥ ≥ ( κ J / 2 ) ∥ Δ δ ∥ \lVert g_E(\delta)-g_E(\delta^\star)\rVert\ge(\kappa_J/2)\lVert\Delta\delta\rVert ∥ g E ( δ ) − g E ( δ ⋆ )∥ ≥ ( κ J /2 ) ∥ Δ δ ∥ in this neighborhood. For an estimate in the neighborhood, combining this lower bound with the upper bound of the first part gives Eq. (85) .
The bound ϵ E \epsilon_E ϵ E separates the three departures. The fraction γ E \gamma_E γ E of arclength that the coupling pairs within δ E \delta_E δ E contributes at most γ E Lip ( ϕ ) δ E \gamma_E\operatorname{Lip}(\phi)\delta_E γ E Lip ( ϕ ) δ E . The remaining fraction 1 − γ E 1-\gamma_E 1 − γ E , which includes arclength present in only one of the two edge sets, contributes up to 2 B ( 1 − γ E ) 2B(1-\gamma_E) 2 B ( 1 − γ E ) . Since ϵ E ≥ 2 B ( 1 − γ E ) \epsilon_E\ge2B(1-\gamma_E) ϵ E ≥ 2 B ( 1 − γ E ) , the bound is small only when nearly all arclength is shared. Eq. (85) converts the feature discrepancy into a pose error, and the factor 2 / κ J 2/\kappa_J 2/ κ J is large when the features respond weakly to some pose change. The bound is local and applies only to an estimate that the search has already brought near the true pose.
The following example shows that the full-column-rank condition of Lemma A20 holds for a simple set of edges. Consider the normalized calibration K = I 3 K=I_3 K = I 3 , the true pose with R ⋆ = I 3 R^\star=I_3 R ⋆ = I 3 and t ⋆ = 0 t^\star=0 t ⋆ = 0 , and the seven segments ( x , a E , 1 ) (x,a_E,1) ( x , a E , 1 ) with − 1 ≤ x ≤ 1 -1\le x\le1 − 1 ≤ x ≤ 1 and a E = − 3 , … , 3 a_E=-3,\ldots,3 a E = − 3 , … , 3 . For nonnegative integers p p p and q q q , let m p , q m_{p,q} m p , q denote the image-arclength average of x p y q x^p y^q x p y q . The derivatives of ( m 0 , 1 , m 0 , 2 , m 0 , 3 , m 1 , 0 , m 1 , 1 , m 1 , 2 ) (m_{0,1},m_{0,2},m_{0,3},m_{1,0},m_{1,1},m_{1,2}) ( m 0 , 1 , m 0 , 2 , m 0 , 3 , m 1 , 0 , m 1 , 1 , m 1 , 2 ) in the local pose coordinates δ = ( ω x , ω y , ω z , t x , t y , t z ) \delta=(\omega_x,\omega_y,\omega_z,t_x,t_y,t_z) δ = ( ω x , ω y , ω z , t x , t y , t z ) of the EM updates are
J 0 = ( − 9 0 0 0 1 0 0 0 0 0 0 − 8 − 124 0 0 0 12 0 0 2 0 1 0 0 0 0 − 11 / 3 0 0 0 0 32 / 3 0 4 0 0 ) . J_0=
\begin{pmatrix}
-9 & 0 & 0 & 0 & 1 & 0 \\
0 & 0 & 0 & 0 & 0 & -8 \\
-124 & 0 & 0 & 0 & 12 & 0 \\
0 & 2 & 0 & 1 & 0 & 0 \\
0 & 0 & -11/3 & 0 & 0 & 0 \\
0 & 32/3 & 0 & 4 & 0 & 0
\end{pmatrix}. J 0 = − 9 0 − 124 0 0 0 0 0 0 2 0 32/3 0 0 0 0 − 11/3 0 0 0 0 1 0 4 1 0 12 0 0 0 0 − 8 0 0 0 0 .
We obtain these entries by differentiating the projected endpoints and lengths and integrating, using E x 2 = 1 / 3 \mathbb{E}x^2=1/3 E x 2 = 1/3 along a segment and E a E 2 = 4 \mathbb{E}a_E^2=4 E a E 2 = 4 and E a E 4 = 28 \mathbb{E}a_E^4=28 E a E 4 = 28 over the segments. The determinant is det J 0 = 11264 / 9 ≠ 0 \det J_0=11264/9\ne0 det J 0 = 11264/9 = 0 , and it stays nonzero under small endpoint perturbations because it is continuous in the endpoints. App. C.8 evaluates this edge variant on images and CAD models.
Experiments
This appendix follows the order of Sec. 5. App. C.1 gives the benchmark inputs and the learned baselines, and each later subsection gives the protocol and full results of one study in the main experiments section.
Datasets & baselines
This subsection describes the inputs of the three benchmarks in Tab. 1 and how we run the learned baselines on them.
Benchmark inputs. For MegaDepth [46 ] , ModelNet40 [79 ] and Data61 [56 ] , we read the released per-frame 2D and 3D coordinates of the blind PnP benchmarks [6 ] and keep their original point selection. MegaDepth pairs 2D points with model points that COLMAP reconstructs by structure from motion (SfM). Its test frames contain 10 to 12,125 points. ModelNet40 samples 1,000 points uniformly from each CAD model, projects them with a virtual pinhole camera and perturbs the projections with Gaussian image noise. Data61 pairs panoramas with light detection and ranging (LiDAR) point sets, and we use its released frames of 1,000 points. We remove the repeated padding entries indicated by the released counts, leave all other coordinates unchanged and shuffle both point sets. Each test set supplies 1,000 random outlier-free frames and 300 frames with at most 2,000 inliers. To the latter, we add outliers drawn uniformly in the bounding box of each point set. We call these frames the corruption frames. The outlier ratio is the number of outliers added to each point set divided by its inlier count.
Learned baselines. We run Deep Blind PnP [48 ] , BPnPNet [6 ] and GoMatch [82 ] with their released code and weights. BPnPNet returns three poses, from RANSAC on its predicted correspondences, from its declarative layer and after Levenberg–Marquardt (LM) refinement. On each dataset, we run BPnPNet checkpoints trained with the correspondence loss alone (L c L_c L c ) and with the pose loss. The Deep Blind PnP release passes the 2,000 most probable matches to the P3P-RANSAC of OpenCV. We also evaluate posest, the pose-estimation library recommended in that release, which reproduces its reported MegaDepth median rotation error. The released variant with a fixed match count fails on frames with fewer than about 45 points, because these frames have fewer than 2,000 candidate pairs. Our top-n n n variants therefore pass the 1.5 min(n n n , N N N ) most probable matches, up to 2,000. The variants marked +LM refine the RANSAC pose by LM on its inliers. For GoMatch, we apply its coordinate-only MegaDepth checkpoint to the input point sets without retrieved reference images. Table A6 lists every pose output that we evaluate, and Table 1 reports the most accurate one.
Table A6. Accuracy at every outlier ratio (↑ \uparrow ↑ ). Entries give the percentage of all evaluated frames with rotation error below 1 ∘ 1^\circ 1 ∘ (top) or 5 ∘ 5^\circ 5 ∘ (bottom), for the MegaDepth [46 ] , ModelNet40 [79 ] and Data61 [56 ] comparison frames of Table 1 . The outlier ratio is the number of outliers added to each point set divided by its inlier count. We list every evaluated pose output of the learned methods (App. § 10 .§ 0.1 ). The label L c L_c L c marks BPnPNet weights trained with the correspondence loss alone. Bold marks the highest accuracy in each column of each panel.
Table A6.
Benchmarks
This subsection completes the benchmark comparison of Table 1 . It reports error quartiles and distributions, accuracy at every outlier ratio and for every pose output, runtimes, results on held-out frames, and examples.
Errors without outliers. Table A7 gives the quartiles of rotation error and of relative translation error on the outlier-free test sets, together with the percentage of frames within both 1 ∘ 1^\circ 1 ∘ and 1 % 1\% 1% . Figure A2 plots the full rotation-error distribution on each test set, and Figure A3 (c) the distribution of relative translation error. A symmetric object can have a large rotation error at a correct alignment, so on ModelNet40 we also measure ADD-S [30 ] . It averages the distance from each model point under the estimated pose to the nearest model point under the reference pose. ADD-S is below 10% of the model diameter on 99.8% of all 2,466 outlier-free ModelNet40 test frames.
Table A7. Rotation & translation errors without outliers on the test sets of MegaDepth [46 ] , ModelNet40 [79 ] and Data61 [56 ] . Error columns give the first quartile (Q1), median and third quartile (Q3) over frames with a returned pose. Translation-vector error is divided by the norm of the reference translation. The final column gives the percentage of all evaluated frames with both rotation error below 1 ∘ 1^\circ 1 ∘ and relative translation error below 1 % 1\% 1% . Each learned method uses its most accurate pose output under the rotation criterion. Bold marks the lowest quartiles and the highest percentage in each dataset.
Table A7.
Figure A2. Rotation-error distributions without outliers on the test sets of MegaDepth [46 ] , ModelNet40 [79 ] and Data61 [56 ] . Each curve gives the percentage of frames with rotation error below each value, and the vertical line marks 1 ∘ 1^\circ 1 ∘ . EM-ICP uses 100 of the comparison frames per dataset, and the other methods the 1,000 outlier-free frames.
Outliers. Table A6 gives the accuracy of every evaluated pose output at each outlier ratio, at thresholds of 1 ∘ 1^\circ 1 ∘ and 5 ∘ 5^\circ 5 ∘ . Figure A3 (a) plots accuracy against the outlier ratio, and Figure A3 (b) the rotation-error distribution with as many outliers as inliers.
Runtime. Table A4 gives the median time per frame at each outlier ratio. Without outliers, GoMatch alone is faster than MomentPnP. With outliers, MomentPnP is the slowest of the four methods, at 0.2 to 7.1 s per frame. Figure A4 plots accuracy against time, comparing methods that return a single pose with searches that improve their best pose until the time limit. Table A5 groups the outlier-free MegaDepth test frames by their number of points. In every group, 99.8 to 100% of frames are within 1 ∘ 1^\circ 1 ∘ . The median time grows from 17 ms for 30 to 99 points to 227 ms for 3,000 points or more, and it is 800 ms for the six frames with fewer than 30 points.
Table A5. MomentPnP on MegaDepth [46 ] by point count (ours). Outlier-free test frames grouped by their number of points. Frames counts the frames in each group, < 1 ∘ 1^\circ 1 ∘ gives the percentage with rotation error below 1 ∘ 1^\circ 1 ∘ , and the last two columns give the median and 95th-percentile time per frame in milliseconds.
Held-out frames. We tuned MomentPnP on the benchmark test sets, so we also evaluate 1,000 frames per dataset from the training split not used for tuning. Table A8 gives the accuracy, acceptance outcomes and runtimes on these held-out frames and on the full test sets. On ModelNet40 with outliers, no held-out pose meets the acceptance criteria, so these rows score the most likely pose, which the solver returns without acceptance (App. § 9 .§ 0.1 ).
Examples. Figure A5 shows MegaDepth and ModelNet40 frames, which have no photographs, for the learned methods, CFPE and MomentPnP. Figure A6 shows one frame of each benchmark for P3P-RANSAC, SoftPOSIT, GOSMA, the learned methods and MomentPnP.
Figure A5. Qualitative results on point-only benchmarks. MegaDepth and ModelNet40 provide coordinates without photographs. Orange dots are the 2D points, and blue crosses are the model points projected under the returned pose. Titles give the landmark or object and the point count. Labels give rotation error, with the lowest in bold, and green borders mark errors below 1 ∘ 1^\circ 1 ∘ .
Figure A6. One benchmark frame per dataset for each method. Rows (a)–(c) show ModelNet40 [79 ] , MegaDepth [46 ] and Data61 [56 ] . Orange dots are the 2D points, and blue crosses are the model points projected under the returned pose. Labels give rotation error, with the lowest in bold, and green borders mark errors below 1 ∘ 1^\circ 1 ∘ .
Role of moment matching
This subsection gives the protocols and full results of the three comparisons in § 4 that test what moment matching contributes to MomentPnP. They match moments of different orders before the same EM refinement, restart EM from random poses, and search globally over poses. The subsection ends with point sets smaller than those covered by Theorem 1.
Moment order. To isolate the effect of moment order, every variant starts from the same initial orientations for each in-
put, uses the same search effort and fits translations from the centroid and spread of the 2D points. We use 100 outlier-free test frames from each benchmark and a separate sample of thirty independent synthetic models of 64 points. Their projections receive Gaussian image noise of standard deviation 0.000625 per calibrated coordinate. The moment variants match the averages of all five, nine or fourteen nonconstant monomials through degree two, three or four. All orders share one image normalization. Each moment variant ranks and refines poses by its own moment residual. Direct EM takes the initial poses in a fixed random order. All variants retain four poses with the same minimum angular separation, refine them with the same EM settings and select the final pose by likelihood. For the counts before EM, we estimate the noise and mixture parameters and select a pose by likelihood without updating it. Before EM, the third- and fourth-order variants bring 53 to 78 of the 100 poses on each benchmark within 1 ∘ 1^\circ 1 ∘ rotation error and 1% relative translation error, and the second-order variant at most two (Tab. A9). After EM, both still recover more poses than direct EM and the second-order variant on every dataset. Which of the two is more accurate varies across datasets.
EM from random poses. EM-ICP in Table 1 restarts the EM refinement of MomentPnP from random poses until the 30-second limit and keeps the most likely pose. It restarts a median of 210 times per MegaDepth frame, and its 30 seconds per frame are over five hundred times the median. time of MomentPnP without outliers (Table A4 ). Figure A4 shows how its accuracy grows with time.
Global search. GOPAC [7 ] and GOSMA [5 ] search over poses by branch and bound within a given domain of camera centers. We run them and MomentPnP on thirty complete, outlier-free pairs of point sets, ten fixed frames from each of the MegaDepth, Data61 and ModelNet40 test sets, without the released time limits. We remove the external clock and the internal duration limits of the released programs, 6,000 s for GOPAC and 600 s for GOSMA, and otherwise keep their code and settings. A search then ends on its own only when GOPAC proves its optimum or GOSMA reaches its released queue-size limit. On a frame where both end on their own within the earlier limits, the modified and released programs return the same poses. We stopped the GOPAC searches still running after 5.2 to 6.0 hours and scored their best pose so far. In a first attempt, four GOPAC searches exceeded a per-process memory cap of 3 GB, and we restarted them without the cap. Both global methods receive unit bearings and the same normalized model points. We center each model at its 3D centroid y ˉ \bar{y} y ˉ and scale it to a fixed root-mean-square distance from that centroid. For GOPAC, we use the released angular tolerance of 0.017 radians (0.974 ∘ 0.974^\circ 0.97 4 ∘ ) and compare two initial lower bounds on the number of inliers, zero and an annealed bound that starts from one fewer than the number of 2D points. GOSMA minimizes a normalized squared L 2 L^2 L 2 distance between densities, and we keep its released stopping tolerance of 0.1 on the gap between the best objective value and the global lower bound.
Search domain. Both global methods need a bounded domain of camera centers that contains the true center, which we compute from the inputs. Let a max a_{\max} a m a x be the largest distance from a model point to the centroid y ˉ \bar{y} y ˉ , let c c c be the camera center, and let β > 0 \beta> 0 β > 0 be the angle between two observed unit bearings. The bound below holds for every such pair, and we take a widely separated pair because a larger angle tightens it. For noiseless complete observations, the camera distance d c = ∥ c − y ˉ ∥ d_c = \lVert c-\bar{y}\rVert d c = ∥ c − y ˉ ∥ satisfies
d c ≤ a max ( 1 + csc ( β / 2 ) ) . d_c \le a_{\max}\left(1+\csc(\beta/2)\right). d c ≤ a m a x ( 1 + csc ( β /2 ) ) .
The bound is immediate when d c ≤ a max d_c \le a_{\max} d c ≤ a m a x . When d c > a max d_c > a_{\max} d c > a m a x , the two model points observed along these bearings are at least d c − a max > 0 d_c-a_{\max}>0 d c − a m a x > 0 from the camera, so by the cosine rule they are at least 2 ( d c − a max ) sin ( β / 2 ) 2(d_c-a_{\max})\sin(\beta/2) 2 ( d c − a m a x ) sin ( β /2 ) apart. They are also at most 2 a max 2a_{\max} 2 a m a x apart, which gives the bound. Both global methods search the same cuboid centered at y ˉ \bar{y} y ˉ , with half-width twice the right-hand side of the bound to allow for measurement noise. This cuboid contains all thirty reference camera centers.
Global search results. We evaluate the final pose of each finished search and the best pose so far of each stopped one, in the original coordinates. Timings include program startup, input processing and model preparation. Table A10 reports every outcome. MomentPnP recovers all thirty frames within 1 ∘ 1^\circ 1 ∘ , and its largest errors are 0.753 ∘ 0.753^\circ 0.75 3 ∘ in rotation and 0.303 % 0.303\% 0.303% in relative translation. Neither global method brings a frame within 1 ∘ 1^\circ 1 ∘ . GOSMA ends every search at its queue-size limit, after a median of 69 s. In each setting, GOPAC finishes twelve of the thirty searches, on ten ModelNet40 and two MegaDepth frames, and none of the Data61 searches. In the two settings, the finished searches take a median of 43 and 50 minutes. Every finished GOPAC search certifies a pose at which every observation is an angular inlier, with rotation errors from 5.4 ∘ 5.4^\circ 5. 4 ∘ to 179 ∘ 179^\circ 17 9 ∘ . The number of angular inliers that GOPAC maximizes therefore does not identify the pose. On a 113-point MegaDepth frame, for example, both settings certify 113 angular inliers at rotation errors of 19.689 ∘ 19.689^\circ 19.68 9 ∘ and 8.552 ∘ 8.552^\circ 8.55 2 ∘ . For these poses, all 113 model points lie in front of the camera, and the maximum nearest-neighbor angular errors are 0.812 ∘ 0.812^\circ 0.81 2 ∘ and 0.882 ∘ 0.882^\circ 0.88 2 ∘ , below the tolerance of 0.974 ∘ 0.974^\circ 0.97 4 ∘ . An inaccurate pose can therefore make every point an angular inlier. Since every finished search is at least 5.4 ∘ 5.4^\circ 5. 4 ∘ off, the one annealed GOPAC pose within 5 ∘ 5^\circ 5 ∘ comes from a stopped search.
Small point sets. Theorem 1 assumes at least twenty points, so we also evaluate MomentPnP on smaller complete inputs. For each of N = 8 N = 8 N = 8 and N = 16 N = 16 N = 16 , we draw one model whose 3D coordinates are independent and Gaussian with mean zero and standard deviation 0.2. We then apply a fixed pose, project with identity intrinsics and independently shuffle the exact projections. The global methods keep the parameters and normalization above and search all rotations, with the camera center in the fixed box [ − 5 , 5 ] 3 [-5, 5]^3 [ − 5 , 5 ] 3 in model units. Here, unlike above, GOSMA stops at a 30-second search limit. On both models, the unchanged solver recovers the generating pose within 1° on every run, with lower median rotation error and runtime than either global method (Table A11 ). Repeated GOPAC runs can return different poses with the same maximal number of angular inliers. GOSMA reaches its search-time limit on both models.
Table A11. Small complete point sets. One noiseless model at each point count N N N . Rotation error and solver time are medians of three runs per method, and < 1° counts runs with error below 1°. Times cover GOSMA’s mixture construction and the complete MomentPnP solve, but not file input and output, process startup, GPU initialization or compilation. Stop counts runs reaching GOSMA’s 30-second search limit, which is checked between iterations. Bold marks the lowest median rotation error and runtime.
Table A11.
Further datasets
Table 2 tests whether MomentPnP, with its benchmark settings, recovers poses from structure-from-motion inputs of four further datasets. This subsection describes the reconstructions, image selection and preprocessing for these inputs. It then gives the errors and runtimes per dataset, the comparison with outliers on all images, and examples.
Reconstructions. For Cambridge Landmarks [40 ] , we use the VisualSFM reconstructions [78 ] of King’s College, Old Hospital, Shop Facade and St. Mary’s Church. For ETH3D [65 ] , we use the released reconstructions of Courtyard, Delivery Area, Electra, Facade, Office and Terrace. For Phototourism, we use the COLMAP reconstructions [64 ] of the Image Matching Challenge [37 ] for Trevi Fountain, Brandenburg Gate and Sacré-Cœur. For 7-Scenes [68 ] , we use the depth-assisted SfM models of Sarlin et al. [62 ] for Heads, Fire and Chess.
Queries. Each query contains the tracked 2D points of one image and the model points observed in that image. We undistort the 2D points, keep the tracks with positive depth and reprojection error below three pixels, and shuffle both point sets. On Cambridge Landmarks, we take the four test images with the most points in each scene and eleven further test images spread over the rest. On the other datasets, we take up to twelve landscape images per scene with 100 to 3,000 points, or up to 4,000 points on 7-Scenes and 6,000 on ETH3D. With outliers, we keep the images with at most 2,000 inliers and add half as many outliers as inliers to each point set, as in the benchmark protocol. The learned methods keep their MegaDepth weights, and we report the released output with the most poses within 1°.
Results. Table A12 extends Table 2 with the median rotation error and runtime of each method on each dataset. Table A13 removes the limit of 2,000 inliers and evaluates all 169 images with outliers. On these inputs, MomentPnP returns 167 poses, 147 of them within 1°. GoMatch runs out of memory on nine images, and the other baselines return a pose for every image.
Examples. Figure A7 shows the first selected outlier-free image of each dataset for every method. Figure A8 adds images from Cambridge Landmarks, Phototourism and Data61, and Figure A9 from 7-Scenes and ETH3D. MomentPnP is within 1° on every image and has the lowest rotation error on all but one. On the Chess image of Figure A7 , CFPE and EM-ICP are 0.01° lower. In each figure, exactly two images have a pose within 1° from at least one learned method.
Figure A8. Qualitative results on outdoor scenes from Cambridge Landmarks, Phototourism and Data61. Orange dots are the 2D points, and blue crosses are the model points projected under the returned pose. Labels give rotation error, with the lowest in bold, and green borders mark errors below 1 ∘ 1^\circ 1 ∘ . Data61 model points come from LiDAR.
Figure A9. Qualitative results on 7-Scenes and ETH3D. Orange dots are the 2D points, and blue crosses are the model points projected under the returned pose. Labels give rotation error, with the lowest in bold, and green borders mark errors below 1 ∘ 1^\circ 1 ∘ .
Scaling to large point sets
This subsection gives the protocol behind Figure 7 and Figure 8 and the results per scan and per view up to one million points. It also reports the percentage of correct labels, the runtimes and the results of the learned baselines.
Rendered 2D points. The experiment varies the number of points, and the fraction of model points that generate 2D points, on real scene geometry. We use the ETH3D laser scans [65 ] of Facade, Courtyard and Office, which hold 45.0, 37.8 and 48.6 million scan points. For each scan, three registered cameras give three reference poses, and 2.6 to 10.6 million scan points fall within the view frustum of a reference pose. Within this frustum, we sample nested subsets of the scan points with positive depth as model points and project them to form the 2D points, ignoring occlusion. We add independent isotropic Gaussian noise with standard deviation s = 1 s = 1 s = 1 pixel per coordinate in a 6,200 × 4,100 6{,}200 \times4{,}100 6 , 200 × 4 , 100 image and shuffle both point sets. In a second setting, with partial detection, a uniformly sampled quarter of the model points generates 2D points. Every method receives the same point sets, and the learned baselines use their MegaDepth weights.
Times cover complete inference runs and exclude input and model loading. MomentPnP runs a small GPU warm-up before timing. The times of the learned methods include compilation on first use where it is required. The Courtyard and Office runs have a limit of 1,800 seconds per view and point count.
Pose accuracy. Tab. A14 gives the median rotation error, percentage of correct labels and time over the three reference poses of each scan. Figure A10 (b) plots the median rotation error of MomentPnP on each scan, together with the learned baselines over all nine views. At one million points, all three Facade runs return poses, and one of them takes 8,735 seconds. For Courtyard, one view contains repeated 3D coordinates at this size and is excluded, and the solver returns no pose for a second view within the 1,800-second limit. The Courtyard markers at one million points thus show a single pose. Tab. A14 lists one million points for Facade and Office, and Table A16 gives every Courtyard and Office view through one million points. Table A15 gives one view of each scan at one million points, and Figure 7 shows these three views. For Facade 1, the estimated projections differ from the reference projections by at most 0.01 pixels. With partial detection, every pose up to 64,000 points has rotation error below 0.006 ∘ 0.006^\circ 0.00 6 ∘ .
Figure A10. Exact label recovery and pose error per scan. (a) Percentage of synthetic frames with N = 1024 N = 1024 N = 1024 in which every 2D point is labeled with its generating model point, against the noise standard deviation s s s (App. § 10 .7). (b) Median rotation error on rendered 2D points (App. § 10 .§ 4 ). Blue curves give MomentPnP on each scan, and the learned baselines use their returned poses from all nine views.
Table A15. One view of each scan at one million points (ours). Each view has 1,000,000 distinct model points and as many 2D points. Figure 7 shows these views. Relative translation is the translation error divided by the reference translation norm, and Labels is the percentage of 2D points whose nearest projected model point generated them.
Table A15.
Table A16. Every Courtyard and Office view (ours). Rotation error in 10 − 4 10^{-4} 1 0 − 4 degrees and relative translation error in 10 − 6 10^{-6} 1 0 − 6 for each view and point count, and at one million points the solve time and Labels as in Table A15 . Invalid inputs contain repeated 3D coordinates and are excluded. Timeout marks a run that returned no pose within the 1,800-second limit.
Table A14. Rendered 2D points on ETH3D scans (ours). Medians, over the three reference poses of each scan, of time per frame in seconds (↓ \downarrow ↓ ), rotation error in thousandths of a degree (↓ \downarrow ↓ ), and Labels, the percentage of 2D points whose nearest projected model point generated them (↑ \uparrow ↑ ). Points counts the model points. In the upper block, every model point generates a 2D point, and in the lower block a uniformly sampled quarter does.
Table A14.
Correspondences. Each 2D point is labeled with its nearest projected model point. When the noise is small compared with the spacing between nearby projections, nearly every label is correct. This spacing shrinks as the point sets grow, so more 2D points lie nearer to another projection than to their own. Figure A11 plots the median percentage of 2D points whose nearest projected model point under the recovered pose did not generate them. The Labels columns of Tabs. A14 to Table A16 give the percentage that did. The percentage of correct labels falls from 100% at 1,000 points to 75 to 79% at one million. Over the same range, the rotation error decreases up to 256,000 points and stays near that level at one million (Tab. A14). App. § 8 .9 relates these percentages to the probability that every label is correct.
Runtime. Figure A12 gives the median time per frame of MomentPnP on each scan and compares it with the learned baselines. Across the scans, the median time of MomentPnP is 0.09 to 0.14 s at 1,000 points and 84 to 355 s at 256,000 points (Tab. A14).
Figure A12. Inference time on rendered 2D points [65 ] . Median time per frame over the reference poses of each scan for MomentPnP, and over the three Facade poses for the learned baselines. Their curves end where they run out of memory (Table A17 ).
Learned baselines. Table A17 counts the poses of the learned baselines within 1 ∘ 1^\circ 1 ∘ over the nine views. GoMatch runs out of memory from 16,000 points and the other learned baselines from 64,000 points. Below these sizes, they bring at most two of the nine poses within 1 ∘ 1^\circ 1 ∘ . Their curves in Figure A12 use the three Facade poses.
Complete scene reconstructions
We evaluate MomentPnP on observations in one image and the complete 3D reconstruction. Neither the visible model points nor a pose prior is given. App. § 8 .11 specifies the camera search and the refinement by EM. Here we give the query protocol, the results of Table A18 and the search cost.
Queries. We use the thirteen ETH3D high-resolution training scenes of Table A18 . Each query contains the tracked 2D points of one image and every finite model point of the released sparse reconstruction [65 ] , which starts from COLMAP [64 ] , with both point sets shuffled. We exclude image detections without a triangulated track. In each scene, the tuning queries are the first three images in file-name order with at least 200 tracked 2D points, and we fix the settings on them. We test the settings on the test queries, three further images spaced evenly over the remaining eligible images. A query is recovered when the returned pose has rotation error below 1 ∘ 1^\circ 1 ∘ . App. § 8 .11 specifies the camera search and its settings.
Results. Table A18 gives the recovery in each scene for the tuning and test queries, and for the learned baselines on the tuning queries. MomentPnP recovers all 39 tuning queries within 1 ∘ 1^\circ 1 ∘ , with rotation errors of at most 0.015 ∘ 0.015^\circ 0.01 5 ∘ , and 35 of the 39 test queries. Figure A13 plots the number of tuning queries recovered by each search time. The search recovers 19 of the 39 within 43 s and the last after 3,531 s.
Figure A13. Search time on complete scene reconstructions. Number of the 39 tuning queries of Table A18 whose search has returned a pose within 1 ∘ 1^\circ 1 ∘ by a given search time.
Cost of the search. The number of grid centers that the search must visit sets its cost. Let d v > 0 d_v > 0 d v > 0 be the viewing distance, the depth of the observed part of the scene. Suppose that the true camera center lies in the enlarged box of App. § 8 .11 and that, for some fraction r c > 0 r_c > 0 r c > 0 , refinement succeeds from any grid center within r c d v r_c d_v r c d v of it. Suppose also that the rotation grid covers the basin of convergence. By Lemma A18, cells of side Δ c = 2 r c d v / 3 \Delta_c = 2r_c d_v/\sqrt{3} Δ c = 2 r c d v / 3 place some grid center within r c d v r_c d_v r c d v of the true camera center, and the grid has at most ∏ k ( 3 L k / ( 2 r c d v ) + 1 ) \prod_k(\sqrt{3}L_k/(2r_c d_v)+1) ∏ k ( 3 L k / ( 2 r c d v ) + 1 ) centers, where L k L_k L k are the side lengths of the enlarged box. The bound grows roughly as the cube of the ratio of scene extent to viewing distance. Recovery also requires that the search keep this grid center after ranking. At each center, scoring the 11,520 rotations takes work linear in the number of model points in the coarse sample, which is at most 65,536. Beyond the three full grids, the search subdivides only the best cells. For the 70 complete grids used by the tuning queries in Table A18 , the cell containing the reference camera center ranks in the top quarter in 80% of grids. Four queries, on Electro and Playground, continue to selective subdivision. For evaluation, the viewing distance is the median depth of the model points observed in the query image, and the Ratio column of Table A18 divides the scene extent by it.
Ablation study
This subsection gives the protocols and results of the controls in the ablation study of § 4 , which vary the depths of the model points, the image noise and the kind of outliers. It then reports three further controls, on the search implementation and the pose ambiguity resolution, on the selection of visible points and on the acceptance test on held-out frames.
Depth variation. The affine initial poses of Step 1 assume an affine camera (Theorem A1), and a perspective camera departs further from an affine camera as the depths of the model points spread. In each paired trial of Table A19 , we vary the depths about a reference depth z r e f = 4.5 z_{\mathrm{ref}} = 4.5 z ref = 4.5 while keeping the noise-free image bearings and the noise realization fixed. In the first set of conditions, the depths are z r e f ( 1 + f s p q s p , j ) z_{\mathrm{ref}}(1+f_{\mathrm{sp}}q_{\mathrm{sp},j}) z ref ( 1 + f sp q sp , j ) for fixed q s p , j ∈ [ − 1 , 1 ] q_{\mathrm{sp},j} \in[-1,1] q sp , j ∈ [ − 1 , 1 ] and a spread factor f s p f_{\mathrm{sp}} f sp . In the second, 13 of the 128 points move closer to the camera and the other depths stay at z r e f z_{\mathrm{ref}} z ref . We draw calibrated image coordinates independently from scaled and shifted Beta distributions. The first set draws q s p , j q_{\mathrm{sp},j} q sp , j uniformly from [ − 1 , 1 ] [-1,1] [ − 1 , 1 ] and uses f s p ∈ { 0.02 , 0.1 , 0.3 , 0.6 , 0.9 } f_{\mathrm{sp}} \in\{0.02,0.1,0.3,0.6,0.9\} f sp ∈ { 0.02 , 0.1 , 0.3 , 0.6 , 0.9 } . The second places the same 13 points at 0.8 0.8 0.8 , 0.5 0.5 0.5 , 0.2 0.2 0.2 and 0.1 0.1 0.1 times z r e f z_{\mathrm{ref}} z ref in separate trials. The affine initialization in this study tests 3,000 viewing directions with 72 rolls each, keeps the affine map with the lowest residual and converts it to an initial pose under scaled orthography, without perspective fitting or EM. The affine initial pose has rotation error above 1 ∘ 1^\circ 1 ∘ in every trial. After perspective fitting and EM, the full solver is within 1 ∘ 1^\circ 1 ∘ in 88 of the 90 trials. At the largest spread, with a median depth ratio of 18, the affine initial pose has a median error of 161 ∘ 161^\circ 16 1 ∘ , and the full solver is within 1 ∘ 1^\circ 1 ∘ in all ten. As the spread shrinks, the median error of the full solver rises from 0.05 ∘ 0.05^\circ 0.0 5 ∘ to 0.37 ∘ 0.37^\circ 0.3 7 ∘ . Its two inaccurate poses, at 1.0 ∘ 1.0^\circ 1. 0 ∘ and 1.3 ∘ 1.3^\circ 1. 3 ∘ , are accepted and occur at the smallest spread.
Image noise. These synthetic controls measure how image noise affects the pose and the labels. We draw N N N model points independently and uniformly from [ − 0.1 , 0.1 ] 3 [-0.1,0.1]^3 [ − 0.1 , 0.1 ] 3 . The rotation axis is uniform on the unit sphere, and the rotation angle is uniform in [ 0 , π ] [0,\pi] [ 0 , π ] . The translation is ( t x , t y , 1 ) T (t_x,t_y,1)^\mathsf{T} ( t x , t y , 1 ) T with t x t_x t x and t y t_y t y independent and uniform in [ − 0.15 , 0.15 ] [-0.15, 0.15] [ − 0.15 , 0.15 ] . The camera has focal lengths of 1,400 pixels and principal point ( 640 , 512 ) (640, 512) ( 640 , 512 ) in a 1280 × 1024 1280 \times1024 1280 × 1024 image. We add independent isotropic Gaussian noise with standard deviation s ∈ { 0.02 , 0.25 , 0.5 , 1 , 2 } s \in\{0.02, 0.25, 0.5, 1, 2\} s ∈ { 0.02 , 0.25 , 0.5 , 1 , 2 } pixels per coordinate. Each noise level uses 200 trials for N ∈ { 16 , 32 , 64 } N \in\{16, 32, 64\} N ∈ { 16 , 32 , 64 } , 100 for N ∈ { 128 , 256 } N \in\{128, 256\} N ∈ { 128 , 256 } and 40 for N = 1024 N = 1024 N = 1024 . In Theorem A4, exact matching depends on the noise through N s Ns N s relative to the spread of the points. Let s ^ \hat{s} s ^ be the noise standard deviation fitted by MomentPnP, and let ρ r m s = ( n − 1 ∑ i ∥ u i − u ˉ ∥ 2 ) 1 / 2 \rho_{\mathrm{rms}} = (n^{-1}\sum_i \lVert u_i-\bar{u}\rVert^2)^{1/2} ρ rms = ( n − 1 ∑ i ∥ u i − u ˉ ∥ 2 ) 1/2 be the root-mean-square radius of the 2D points about their mean u ˉ \bar{u} u ˉ , both in pixels. To compare point counts on one scale, we plot results against N s ^ / ρ r m s N\hat{s}/\rho_{\mathrm{rms}} N s ^ / ρ rms . Over this noise range, the rotation error of MomentPnP follows that of the pose fitted to the true correspondences. Its percentage of correct labels follows that of minimum-cost bijective matching at the true pose (Figure A14 (a)). The second agreement is consistent with Lemma A9, which bounds the labeling error of nearest-projection matching at an accurate pose. It is also consistent with Theorem A7, under which, for large N N N , even an estimator that knows the camera errs with probability bounded away from zero once N s Ns N s is at least a fixed positive constant.
Exact label recovery. A stricter measure is the percentage of frames in which every 2D point is labeled with its generating model point. Figure A15 plots it for MomentPnP, CFPE and EM-ICP against N s ^ / ρ r m s N\hat{s}/\rho_{\mathrm{rms}} N s ^ / ρ rms , and Figure A10 (a) against the noise at N = 1024 N = 1024 N = 1024 . EM-ICP restarts on the CPU for up to 30 s. At N = 1024 N = 1024 N = 1024 and s = 0.02 s = 0.02 s = 0.02 pixels, MomentPnP recovers every label in all 40 frames, compared with 90% of the frames for CFPE and 30% for EM-ICP. At N = 1024 N = 1024 N = 1024 and s = 0.5 s = 0.5 s = 0.5 pixels, none of these methods recovers every label in any frame.
Table A20. Search implementation and pose ambiguity resolution (ours) on all 10,795 MegaDepth, 2,466 ModelNet40 and 2,640 Data61 outlier-free test frames. The top rows give the percentage of frames with rotation error below 1° for the GPU search and for the CPU search alone, and the ratio of their median times per frame. Bold marks the more accurate search. The lower rows count the frames in which an alternative pose of Step 4 is examined, the frames in which it replaces the selected pose, and the frames with the stated accuracy and acceptance outcomes.
Structured outliers. Real outliers often lie on scene structure, which the uniform outlier model of Definition 1 does not describe. To test such outliers, we take Data61 frames with N N N model points and replace a fraction of the 2D points of each frame with as many other 2D points on the same scene structures. MomentPnP is within 1° on every frame with up to 30% replaced and on 79% to 99% of the frames at 50% (Figure A14 (b)). At every fraction, BPnPNet is within 1° on fewer frames.
Search implementation & pose ambiguity resolution. Tab. A20 measures the effect of the batched GPU search, the pose ambiguity resolution and the acceptance test on all 15,901 outlier-free test frames. The CPU search keeps moment matching, EM and acceptance, but replaces the batched GPU search with a coarser search. The GPU search
Table A14 (PDF p. 62). Acceptance outcomes by outlier ratio.
Table A21. Selection of visible points (ours). Accurate poses in 30 paired runs with n n n 2D points, 96 model points and 24 inliers, with model points chosen at random or by directional deletion. Both counts require rotation error below 1°. Column n t n_t n t also requires translation error below 1% of the reference translation norm, and n C n_C n C camera-center error below 1% of the root-mean-square radius of the model.
places 97.0 to 100% of the frames of each test set within 1°. The coarser CPU search loses at most 0.2 percentage points and takes 4.7 to 6.4 times the median GPU runtime. The rows on pose ambiguity resolution count the frames in which an alternative pose of Step 4 is examined and the frames in which it replaces the selected pose (Lemma A8). These replacements occur mainly in the CPU search, on flat MegaDepth scenes and ModelNet40 objects. Both replacements in the GPU search and 61 of the 62 in the CPU search are within 1°. Alternatives within 3° of an accepted pose are discarded, so each of these replacements corrects a pose at least 2° from the true one. Model symmetry has a related effect. Under exact symmetry, different poses give identical projections, and near-symmetry can make the moment residual less informative about orientation error (Apps. A.4 and A.7). On ModelNet40, 27 of the 31 frames with rotation error of at least 5° meet the ADD-S criterion of 10% of the model diameter (App. C.2), as do 11 of the 13 accepted with such errors. The last two rows count the errors of the acceptance test in both directions. The test accepts 13 ModelNet40 poses with rotation error of at least 5° and none on the other test sets. It declines 7 MegaDepth and 60 ModelNet40 poses within 1°.
Selection of visible points. Moment matching needs both point sets to sample the same structure, so recovery can depend on which model points are visible as well as on how many. We compare two selections in paired trials on thirty independent 3D models of 96 model points (Tab. A21). Each pair uses the same 3D point set, intrinsics, pose, measurement noise and unmatched 2D points. The noise is Gaussian with standard deviation 0.001 per calibrated coordinate. In both selections, 24 model points generate 2D points. Random selection chooses them uniformly at random, and directional deletion chooses them from one end of the model along a random image direction. Uniform unmatched 2D points then bring the 2D point set to n n n points. With n ≤ 48 n \le48 n ≤ 48 , random selection gives 26 to 29 of the 30 poses within 1°
and 1% translation error, and directional deletion at most one. Recovery thus depends on which points are visible.
Acceptance on held-out frames. Figure A14 (c) splits the 1,000 held-out frames of each dataset and outlier ratio by acceptance and by rotation error below 1°. In every setting, at most 8 frames yield an accepted pose with rotation error of at least 1°.
In every study above, each inlier 2D point comes from a model point, through a track, a released pair or rendering. In the studies of this subsection, the 2D points and the model points are selected independently, so no 2D–3D correspondences are given. The studies pair scale-invariant feature transform (SIFT) detections with a 3D model built from other photographs, image edges with a CAD model, and detected motion-capture markers or infrared light-emitting diodes (LEDs) with their 3D positions. The SIFT study uses the search over camera centers and rotations of App. C.6. The study with CAD models uses the edge features of App. B.2. The marker studies apply MomentPnP to the two point sets. Tab. A22 summarizes recovery in these studies, with a SIFT example in Figure 9 (a) and a CAD example in Figure A16 .
Figure A16. Image edges and a CAD model. For T-LESS object 5, we show detected image edges, CAD creases with hidden creases in gray, the projected creases at the recovered pose, and the estimated camera. Rotation and camera-center errors appear below the projection.
SIFT detections & reconstructed maps. We use the ETH3D scenes Office, Courtyard and Facade. In each scene, the first photograph is a tuning query, and two photographs evenly spaced through the scene’s list of photographs are test queries. We compute new SIFT detections [49 ] and triangulate the model points [64 ] from source photographs that exclude all nine queries, so no query photograph takes part in feature extraction, matching or triangulation. The Office, Courtyard and Facade maps contain 3,272, 42,714 and 96,909 model points from 23, 35 and 73 source photographs. The search uses the grids of App. § 10 .§ 5 , with three subdivision levels.
Acceptance. We tune the acceptance thresholds for these inputs on the three tuning queries and fix them before evaluating the six test queries. A pose is accepted only if at least 30% of the points in the smaller point set are matched. This applies both to the expected number n m n_m n m of 2D points matched under EM (App. § 9 ) and to the final correspondences. We require the median, over the matched 2D points, of the ratio of the distances to the nearest and second-nearest projections to be at most 0.12. For queries with at least 500 detections and an association level of at most 10 − 100 10^{-100} 1 0 − 100 , we relax this limit to 0.50. The benchmark acceptance test with outliers permits 0.25 in this case (App. § 9 .§ 0.1 ). We run the search on each test query with a time limit of 3,600 seconds, checked between completed operations. At the limit, it returns its best valid pose. A query is recovered when its rotation error is below 1 ∘ 1^\circ 1 ∘ and its camera-center error below 1% of the scene extent.
SIFT results. Table A23 gives every returned pose. Two of the six test queries are recovered. The accepted Facade 26 pose of Figure 9 (a) gives 5,666 distinct pairs, all in front of the camera. Their median and root-mean-square reprojection distances are 0.763 and 1.168 pixels. The accurate Office 18 pose is returned at the time limit without being accepted.
Why such inputs are harder. Figure 9 (b,Figure 9 c) shows two measurements that explain this difficulty. First, the two point sets can sample different structure. Of the 12,069 SIFT points detected in a Meadow image, 255 belong to reconstruction tracks, and many of the other detections lie on vegetation and sky that the reconstruction does not contain (Figure 9 (b)). Second, a dense 3D model places projections near most 2D points at more than one pose. In a Facade image, 2.6 million laser-scan points fall within the view. At the reference pose, 80.6% of the 9,754 2D points lie within 3.4 pixels of a projected scan point. At a pose rotated by 2 ∘ 2^\circ 2 ∘ , 76.8% still do (Figure 9 (c)). Distances to the nearest projections therefore barely distinguish the two poses.
Image edges & CAD models. We fix all filter, search and refinement settings on four tuning views of T-LESS [32 ] object 25. Before evaluating any pose error, we select views 72 and 360 of objects 1, 5, 9, 13, 17, 21, 26 and 29, and views 72, 216, 360 and 504 of object 25. The twenty test views thus contain sixteen views of new objects and four held-out views of the tuning object. Each run receives one RGB image, the intrinsics and a CAD mesh in its own coordinate frame, and no reference pose, visibility, model symmetries or edge identities.
Edge features & search. We use eight-pixel segments from the line segment detector (LSD) [25 ] in the image, and creases and candidate contours (App. § 9 .§ 1 ) on the CAD model. We normalize image coordinates by the arclength-weighted centroid and root-mean-square radius of the detected image edges. We integrate Fourier features analytically along each projected segment using sinc functions. We score an initial grid of poses with these features and keep the best poses for EM. Scoring on this grid ignores occlusion. For the retained poses, exact visibility then clips occluded edges, and all CAD vertices must lie in front of the camera and project inside the image. For point-based EM, we sample each projected edge once by image arclength and keep these samples during refinement. For the final ranking, we recompute visibility and sampling and score each pose by an edge-curve likelihood. An improvement of the point-based EM objective thus need not improve this likelihood.
Edge results. Table A24 gives the rotation and camera-center errors of every test view, for the returned pose and after applying the official model symmetry. A view is recovered when its rotation error is below 5 ∘ 5^\circ 5 ∘ and its camera-center error below 50 mm. To find whether a failure comes from the search or from the objective, we score the reference and estimated poses with the same final likelihood, using a Gaussian standard deviation of two pixels and a uniform outlier fraction of 0.05. Every reference pose satisfies the constraints of the search, with every CAD vertex in front of the camera and projecting inside the image. At these poses, we sample the detected image edges and the visible projected CAD edges by image arclength, at most two pixels apart. A median of 72.9% of the image-edge samples and 83.1% of the CAD-edge samples lie within three pixels of a sample of the other set. The detected and projected edges thus overlap only in part. Of the twelve views that fail under model symmetry, four return poses with lower final likelihood than the reference pose, so the search misses a more likely pose. In the other eight, the estimated pose has higher likelihood than the reference, so the objective itself favors these inaccurate estimates.
Table A24. Held-out T-LESS views (ours). Rotation error in degrees and camera-center error in millimeters for each of the twenty test views. Raw pose evaluates the returned pose, and Model symmetry applies the same official model symmetry to both errors, sampling continuous symmetries in 1 ∘ 1^\circ 1 ∘ steps. Bold marks views with rotation error below 5 ∘ 5^\circ 5 ∘ and camera-center error below 50 mm. The views of object 25 are held-out views of the tuning object.
Motion-capture markers. We use the static trial of a public running motion-capture dataset.4 For the query camera, we keep all unlabeled marker centroids, corrected for radial distortion. We tune on one camera at the first frame and evaluate all eight other cameras at frame 501, using MomentPnP with one-pixel noise and a 30-second time limit. Six of the eight returned poses are within 5 ∘ 5^\circ 5 ∘ and 5 cm, and none is within 1 ∘ 1^\circ 1 ∘ and 1 cm. One of the five accepted poses fails the 5 ∘ 5^\circ 5 ∘ and 5 cm thresholds. As a control, efficient perspective-n-point
(EPnP) [44 ] with the reference assignments is within 5 ∘ 5^\circ 5 ∘ and 5 cm on all eight cameras and within 1 ∘ 1^\circ 1 ∘ and 1 cm on one.
Infrared LEDs. We use the raw infrared images and the surveyed coordinates of four noncoplanar LEDs from a public demonstration [17 ] . A fixed filter on intensity, area and contour shape extracts the LED centroids, and the point solver runs in its outlier-free setting with one-pixel noise and a ten-second time limit. The transform between the LED model and the attached AprilTag [59 ] is unknown, so we compare motions relative to frame 61 with independent tag estimates. With Ξ i t a g \Xi_i^{\mathrm{tag}} Ξ i tag and Ξ i l e d \Xi_i^{\mathrm{led}} Ξ i led the rigid transforms from the tag frame and from the LED frame to the camera frame at frame i i i , Ξ i t a g = Ξ i l e d Ξ o b j \Xi_i^{\mathrm{tag}} = \Xi_i^{\mathrm{led}}\Xi^{\mathrm{obj}} Ξ i tag = Ξ i led Ξ obj for a fixed transform Ξ o b j \Xi^{\mathrm{obj}} Ξ obj between the object frames, so the relative transforms Ξ j t a g ( Ξ i t a g ) − 1 = Ξ j l e d ( Ξ i l e d ) − 1 \Xi_j^{\mathrm{tag}}(\Xi_i^{\mathrm{tag}})^{-1} = \Xi_j^{\mathrm{led}}(\Xi_i^{\mathrm{led}})^{-1} Ξ j tag ( Ξ i tag ) − 1 = Ξ j led ( Ξ i led ) − 1 do not depend on Ξ o b j \Xi^{\mathrm{obj}} Ξ obj (see Table A25 ). A fixed error in the camera frame does not cancel in this comparison.
Table A25. Motion-capture markers and infrared LEDs (ours). A comparison is accurate when its rotation error is below 5 ∘ 5^\circ 5 ∘ and its position error below 5 cm. For motion capture, these are the errors of the camera pose. For the LEDs, they are the errors of the motion of each of the 31 other selected frames relative to frame 61. Accepted outputs and extraction failures count frames, and inaccurate accepted comparisons count motions. Error medians use all eight motion-capture poses and the 24 LED motions for which both poses are available.