Rethinking Asymmetry in Self-Supervised 3D Geometry Learning

Website

Code

Introduction

Self-supervised learning has driven the progress in representation learning. Because it requires no manual labels, models can be pretrained on as much data as we can collect, and their performance continues to improve as data, model size, and compute scale [21]. This approach has succeeded in language [36, 10], vision [3, 13], and embodied control [32]. In contrast, many 3D foundation models are trained to reproduce geometric annotations recovered by optimization pipelines such as COLMAP [42], which are costly to run and hard to scale. Recent work [18, 64, 16] removes this dependence by making novel-view synthesis (NVS) the training objective, predicting cameras and scene structure from raw images or a video and rendering held-out views using its own predictions. These methods differ in scene representation and how they share information across views, but evaluations focus largely on synthesis quality. In self-supervised learning, however, the pretext task is only a means, and a method is judged by what it learns. We therefore treat view synthesis as the pretext and geometry as the outcome, and ask which designs make geometry learning more accurate and scalable.

We begin with the scene representation. Since the model learns about the scene only through what it renders, the form of this representation determines how closely the learned structure reflects the actual 3D geometry. Latent NVS methods [18, 31, 35] render held-out views from an implicit scene code, so geometry is never made explicit. They achieve strong rendering quality, but can do so without recovering structure that corresponds to physical 3D space, leaving geometry and camera motion only weakly constrained by the synthesis loss. We instead use an explicit representation, 3D Gaussians [22] placed in physical space, so that every rendered pixel depends directly on the predicted depth and camera pose.

Where should the asymmetry live? A self-supervised objective needs an asymmetry between what the model sees and what it must explain. Otherwise, trivial solutions satisfy it. For example, in masked image modeling [13, 59], the image itself is the only place to create this asymmetry, so masking parts of it directly defines the task. In contrast, masking a view in multi-view 3D does not create the same asymmetry. When a held-out view lies between context views (interpolative NVS setting), the 3D scene reconstructed from the context views is the same scene that all views together describe. Removing the held-out view therefore hides nothing the model must recover. It only restricts the correspondence that would have helped recover that scene more accurately. Existing methods nevertheless impose the asymmetry on the images themselves.¹ E-RayVzer [64] splits camera and geometry prediction into two backbones so that the geometry branch sees only context views, and NAS3R [16] masks view interaction between the context and held-out sets, so context views attend only among themselves. Both keep held-out views out of the reconstruction, but in doing so also withhold the correspondence those views could provide.

We instead place the asymmetry in the 3D representation space. A shared backbone lets all views exchange information when predicting depth and pose, so every view contributes correspondence. We keep the context and held-out split of prior work but apply it when constructing the 3D scene, so that a held-out view is rendered only from Gaussians predicted for context views. This is equivalent to masking the 3D representation rather than the images. Every view thus informs correspondence estimation, while the self-reconstruction path is blocked.

Asymmetric feature alignment. View density provides a second source of asymmetry. The same model predicts more accurate depth and pose when it sees more views of a scene, so a model given fewer views is a weaker version of itself. Such gaps within a single model have been used to bootstrap representations without any external teacher [17, 4]. SelfEvo [14] applies this idea to 3D, refining a model pretrained with 3D supervision [50, 54] by using the geometry and camera predictions of a dense-context teacher as pseudo labels for a sparse-context student. We instead build this asymmetry into self-supervised training, optimizing it jointly with the rendering objective without any pretrained geometric teacher. In this setting the dense-context predictions are themselves unreliable, and matching them can propagate their errors. We therefore align the features that produce the geometry rather than the geometry itself. The largest gains occur in single-view depth estimation, which the rendering objective does not supervise directly.

Semantics as a geometric prior. Both signals above are grounded in rendered pixels. The rendering loss supervises the model directly, and asymmetric context alignment does so indirectly through features shaped by rendering. Pixel agreement, however, says little about which pixels belong together, such as which regions form the same surface or object. Such high-level structure often helps constrain geometry where photometric cues are weak, such as textureless or repetitive regions. Vision foundation models capture it well, and aligning with their features has proven effective in generative modeling [62] (Wu et al. 2025) and supervised perception tasks [58, 23]. We use such alignment as a semantic prior for self-supervised learning, aligning our backbone features with those of a frozen vision foundation model [45, 59], obtaining consistent gains in geometry estimation at modest additional cost.

We combine these components into a single self-supervised geometry framework and evaluate each through controlled experiments. Our contributions are as follows.

• Asymmetry in the 3D representation. We show that imposing the asymmetry on the reconstructed scene rather than on the input images prevents trivial solutions while letting a shared backbone use every view for correspondence, improving depth and pose estimation.

• Asymmetric feature alignment. We align sparse-context features with dense-context features jointly with the rendering objective, improving geometry without a pretrained geometric teacher.

• Semantics as a geometric prior. Aligning intermediate features with a frozen vision foundation model adds structural guidance beyond pixel agreement, improving depth and pose estimation without geometric annotations.

Related Work

Self-supervised learning with novel-view synthesis. Feed-forward 3D models recover geometry directly from a set of images, but they are trained against camera and depth targets that an offline pipeline must produce first [53, 50, 51, 27]. A separate line of work has instead learned geometry from images alone. Early methods predicted depth and ego-motion from video and used warping between neighboring frames as supervision [65, 12, 2, 24], which ties this learning to short baselines and pairwise constraints. RUST [41] and RayZer [18] removed explicit geometry altogether, encoding a scene into a latent set [40] and rendering new views with a learned decoder. Several methods extend this latent formulation to dynamic data and predictive (extrapolative) objectives [6, 31, 35]. Rendering in latent space minimizes inductive bias and lets the model learn freely from data, but it also leaves the recovered geometry only loosely tied to physical structure.

Another line of work grounds the scene in 3D space and renders it by volume rendering or rasterization, jointly predicting poses and a NeRF [30] or Gaussian scene [22] under photometric losses [7, 11, 20]. Recently, E-RayZer [64] and NAS3R [16] scaled explicit self-supervised reconstruction to large transformers and data, yielding representations with stronger geometric grounding. To prevent shortcuts and information leakage, both restrict how information moves between the context and held-out views, E-RayZer by separating camera and geometry prediction into two backbones, NAS3R by masking view interaction in its single backbone. The restriction keeps the object from collapsing, but at the cost of rich correspondence. We instead investigate where this restriction should be placed, and find that the scene representation is a better location than the input.

Self-supervised representation alignment. Aligning intermediate features to an additional target has become a common way to improve representation learning beyond what the task loss provides. In generative modeling, REPA [62] and follow-up work [44] (Wu et al. 2025) align diffusion transformer features [34] with a frozen vision foundation model [33], accelerating convergence and improving sample quality. Perception models use the same idea, aligning task features with such encoders [58, 23] or distilling several of them into one [38, 67].

On the other hand, recent work derives the target from the model itself, comparing its own representations across different inputs or layers [17, 4]. SelfEvo [14] brings this to 3D, distilling a dense-context teacher into a sparse-context student, though it starts from a supervised model and aligns the two branches through their predicted depth. We exploit the same sparse-dense gap without 3D-supervised initialization, aligning features jointly with the main rendering objective during training. We also align them with a frozen vision encoder, providing complementary internal and external signals without geometric annotations.

Approach

Preliminaries

Setup. Given $N$ unposed images $\mathcal{V} = \{I_i\}_{i=1}^{N}$ of a static scene, we partition them into disjoint context and target sets, $\mathcal{C}$ and $\mathcal{T}$. A model $f_\theta$ predicts a scene representation $\mathcal{S}$ and camera poses $\hat{\pi}_i = (\hat{T}_i, \hat{K}_i)$, where $\hat{T}_i \in SE(3)$ is the (camera-to-world) extrinsics and $\hat{K}_i$ the intrinsics. A renderer $g$, either a learned neural decoder or a differentiable rasterizer, then synthesizes each target view:

Cameras are predicted relative to a reference view, and since no metric supervision is available, the reconstruction is recovered only up to scale. Estimating the pose of a target view requires observing its image, so target images must enter the network. If the representation predicted from a target view also enters $\mathcal{S}$, however, the model can render that view and satisfy Eq. (1) without recovering a scene shared across views. Therefore, target views must be excluded from the scene used to render them.

Implicit vs. explicit representation. Implicit methods encode the context views into latent scene tokens $Z$, and render a target view with a learned decoder $g_\phi$ that queries $Z$ using an encoding of the target camera. Since $Z$ is not tied to physical 3D space, a low photometric loss need not reflect a consistent 3D scene, and the decoder can rely on interpolating appearance between nearby context views. Explicit methods instead represent the scene in 3D space, for example as 3D Gaussians. For each pixel $u$ of view $i$, an output head predicts a ray depth $d_{i,u}$ along with the Gaussian attributes. The Gaussian center is obtained by lifting the depth through the predicted camera,

where $u$ is in homogeneous coordinates and $\mathcal{G}_i$ is a set of Gaussians predicted from view $i$, and $\mathcal{S}$ collects them over all context views. Rendering it with a differentiable rasterizer $g$ places both depth and pose on the rendering path. A target view is then well explained only when predictions from different views agree in a shared 3D space, so photometric error becomes a proxy for multi-view geometric consistency.

Where asymmetry is placed. Existing methods keep target views out of the reconstructed scene by restricting how information flows between views. Separate branches [64] predict cameras from all views but geometry from context views alone, using separate features without the interaction that a shared representation can provide. Masked attention [16] uses a single backbone with a view-level mask $A \in\{0,1\}^{N \times N}$, setting $A_{ik} = 0$ for $i \in\mathcal{C}$ and $k \in\mathcal{T}$, so that context tokens never read from targets. This keeps a shared backbone but removes all context-to-target interaction. Both designs thus break the trivial solution by restricting which views the model can relate in the input space.

Asymmetry in the 3D Representation

We now ask where the asymmetry should be placed when the goal is geometry rather than rendering. Prior designs keep target views out of view interaction so that the model cannot observe what it is asked to render. However, when a target view lies between context views, the context views already cover most of the geometry the target observes (Figure 3), so the target contributes correspondence rather than new scene content.

The asymmetry therefore only needs to prevent a target view from being rendered from its own prediction. It need not restrict which views interact. We test this against existing placements, holding the representation, encoder, and training setup fixed.

Controlled comparison. We compare three placements of asymmetry. All variants share a visual encoder (Yang et al., [59]) followed by a 12-layer transformer (Vaswani et al., [47]) with alternating view-wise and global attention (Wang et al., [50]). The full-view variant (ours) and the masked-attention variant predict cameras and Gaussians from this backbone. The separate-branch variant follows E-RayZer, predicting cameras from the backbone and Gaussians from a second 12-layer transformer, which gives it more capacity. All variants are trained on DL3DV (Ling et al., [28]) for 100K iterations with the same data, view sampling, and optimization settings, with the number of views varying across training batches. At evaluation, we fix the frame span at 48 frames, so fewer views provide sparser coverage of the same scene (Figure 4). Despite having less capacity than the separate-branch variant, the full-view variant achieves the most accurate depth and pose at every view count. Placing the asymmetry in the 3D representation rather than on interaction between views lets every view contribute correspondence while still blocking the self-reconstruction path.

Beyond Rendering Signals

Learning from context asymmetry. The preceding analysis shows that geometry estimates become less accurate as the view set becomes sparser. Dense and sparse predictions come from the same network on the same scene, so the gap reflects only what the extra views contribute, and this asymmetry can serve as a learning signal. Self- supervision (Huang et al., [14]) turns this gap into supervision for a pretrained 3D foundation model (Wang et al., [50]; Wang et al., [54]), using the outputs of a dense-context teacher, maintained as an EMA of the model, as pseudo labels for a sparse-context student.

We build on this idea with one key difference. Our model is trained entirely through self-supervision, with no 3D supervision to anchor its predictions. In this setting, depth predicted from different view sets shares no common scale, and the dense-context depth can itself be unreliable. In our experiments, distilling it directly degrades geometry even with a scale-invariant loss. We therefore move the alignment from the outputs to the features that produce them. These features carry the cross-view correspondence that dense context provides, but are not tied to a particular scale or depth value, making them a more stable target than the outputs they produce. We match them with a cosine similarity loss, which constrains their direction but leaves their magnitude free. An EMA copy $f_{\bar{\theta}}$ processes the full set $\mathcal{V}$, and the online network $f_{\theta}$ processes a subset $\mathcal{V}_s \subset\mathcal{V}$. We align the patch features of each view, token by token, at the layers $\mathcal{K}$ that feed the output heads,

where $F=f_{\bar{\theta}}(\mathcal{V})$ and $F_s=f_{\theta}(\mathcal{V}_s)$ are the dense and sparse features, $p$ indexes the $P$ patch tokens of each view, $h_k$ is a lightweight projection for layer $k$, $\operatorname{sg}[\cdot]$ is the stop-gradient, and $\bar{\theta}$ is an exponential moving average of $\theta$.

We train the full-view model from § 3.2 with and without asymmetric context alignment, holding all other settings fixed. Alignment improves depth and pose at every view count, with larger gains when fewer views are available and the largest on single-view depth, the limiting case of sparse context where rendering provides no direct supervision (Figure 5).

Semantic guidance. The rendering loss supervises appearance at the pixel level, so the constraints it provides are local. In textureless or repetitive regions, photometric error barely changes with depth and says little about surface boundaries. Such distinctions are what semantic features encode, so we align our encoder features with those of a frozen vision foundation model (Siméoni et al., [45]; Yang et al., [59]). For each view, the encoder features $F^{\mathrm{enc}}$ are matched to the corresponding patch features of the frozen model $\Phi$ with a cosine similarity loss,

Since $\Phi$ is itself self-supervised, this adds no annotation. Applying the alignment at the encoder output, before the multi-view transformer, leaves the later layers free to specialize for geometry. Although the encoder is initialized from the same model, initialization alone does not retain this prior under the rendering loss, and the alignment preserves it throughout training.

Training Objectives

Our model is trained with three signals. The rendering loss defines the self-supervised task, asymmetric context alignment transfers dense-context features to sparse inputs, and semantic guidance injects a prior from a frozen foundation model. The total objective is

where $\mathcal{L}_{\mathrm{nvs}}$ combines an L2 and a perceptual loss (Chen & Koltun, [5]), and the $\lambda$ terms balance the auxiliary losses against it. We set $\lambda_{\mathrm{align}}=0.1$ and $\lambda_{\mathrm{sem}}=0.2$. Since the dense-context features are only meaningful once rendering has shaped them, $\mathcal{L}_{\mathrm{align}}$ is introduced at 40% of training and ramped to full weight by 60%.

Experiments

Implementation details. Our model uses a pretrained Pixio (base) visual encoder (Yang et al., [59]) followed by a 12-layer multi-view transformer with an alternating attention pattern, a feature dimension of 768, and a $16 \times16$ patch size. All experiments use this same backbone, which predicts both cameras and Gaussians. The camera head regresses each pose relative to the middle input view and predicts one set of intrinsics shared across the scene. A DPT head [37, 51] predicts per-pixel Gaussian attributes, which are lifted to 3D with the predicted cameras. The scene is built from context-view Gaussians only, and target views are rendered at their predicted poses by differentiable rasterization [22]. Asymmetric context alignment uses the four transformer layers read by the DPT head, and semantic guidance uses the encoder output. Sparse subsets are formed by symmetrically dropping views on either side of the reference view, and the dense branch is an EMA copy of the online network with momentum 0.999.

Datasets. We train separate models on RealEstate10K (RE10K) [66] and DL3DV [28], with images resized to 256 $\times$ 256. ScanNetv2 [8] and NRGBD [1] are used only for out-of-domain evaluation.

Evaluation. We use publicly released baseline checkpoints and follow their evaluation settings. On RE10K, we adopt the view indices of SPFSplat [15], and on DL3DV, we follow E-RayZer [64] and sample 10 views across a 96-frame span. For out-of-domain evaluation, we sample 10 views at a fixed stride, spanning 64 frames on ScanNetv2 and 256 on NRGBD. We report camera pose and depth, the two geometric quantities of interest. Camera pose is measured by relative pose accuracy (RPA) at several angular thresholds and by AUC@30. Depth is measured by AbsRel and threshold accuracy $\delta< 1.25^k$ after independent per-view median alignment against ground-truth depth. Since DL3DV provides no ground-truth depth, our controlled analyses on DL3DV use MoGe-2 [52] predictions as pseudo ground truth. Multi-view depth is predicted from the full input set, and single-view depth from each image alone.

Depth conversion. All depth metrics use planar $z$-depth. Our model and E-RayZer predict depth $r$ along each unit ray (Eq. 2), which we convert as $z = r/\lVert\hat{K}^{-1}u \rVert$ using the predicted intrinsics $\hat{K}$, so focal-length errors are reflected in the depth metrics. For NAS3R, we use the $z$-depth from its depth head. For SPFSplat, we transform its canonical point map into each view with the predicted pose and take the $z$ coordinate.

Results

We compare with SPFSplat and NAS3R [16], which learn cameras and geometry from unposed images using self-supervised rendering signal. All three methods train on RE10K with varying numbers of input views. SPFSplat additionally initializes its backbone from MASt3R [25], which is trained with 3D supervision on multiple annotated datasets, and also takes ground-truth intrinsics as input. With 10 input views, our model achieves the highest pose accuracy at every reported threshold and competitive or better depth accuracy, despite using far fewer parameters. Its advantage grows on out-of-domain data (Tab. 1). We further examine how each method scales with the number of input views on an out-of-domain dataset. Our model improves steadily on every pose metric as views are added (Fig. 6).

Method	3D prior	Params	RE10K (trained)		DL3DV		ScanNetv2			NRGBD
			RPA@3 ↑	RPA@5 ↑	RPA@3 ↑	RPA@5 ↑	RPA@3 ↑	@5 ↑	AbsRel ↓	$\delta< 1.25$ ↑	RPA@3 ↑	@5 ↑	AbsRel ↓	$\delta< 1.25$ ↑
SPFSplat	MASt3R	629M	64.2	78.1	10.5	19.3	2.0	5.7	0.223	0.757	3.2	14.1	0.156	0.803
NAS3R	–	1.22B	47.4	67.2	1.5	3.5	0.1	0.5	0.381	0.604	3.7	7.2	0.214	0.630
Ours	–	183M	65.7	80.2	14.2	26.4	2.7	7.1	0.226	0.785	16.5	30.1	0.106	0.878

We also compare with E-RayZer [64], with both models trained on DL3DV. Our model improves every reported pose and depth metric both in-domain and out-of-domain datasets (Table 2). For depth, E-RayZer feeds only the context views to its geometry branch. Our backbone processes all views, but only the context views are decoded by the DPT head, so both methods are scored on the same context views. Figure 7 and Figure 8 show the recovered camera trajectories on the in-domain and out-of-domain datasets, after $\operatorname{Sim}(3)$ alignment to the ground truth with the Umeyama method [46].

Method	Params	DL3DV (trained)		RE10K		ScanNetv2				NRGBD
		RPA@3↑	@5↑	RPA@3↑	@5↑	RPA@3↑	@5↑	AbsRel↓	δ < 1.25 ↑	RPA@3↑	@5↑	AbsRel↓	δ < 1.25 ↑
E-RayZer	246M	60.3	72.1	54.6	74.5	4.9	12.3	0.168	0.819	35.6	61.5	0.095	0.908
Ours	183M	70.0	79.7	57.7	75.3	6.6	16.7	0.125	0.872	63.5	81.5	0.082	0.917

Single-image depth. We further evaluate depth when each frame is given to the model alone (Table 3). The network then receives only the reference camera token, so the pose head returns the identity pose together with a focal length predicted from the image, which we use to convert ray depth to $z$-depth. Here our model outperforms E-RayZer on both datasets. Our multi-view depth is more accurate than our single-image depth on NRGBD, where our pose estimation is accurate, but less accurate on ScanNetv2, where pose estimation is much harder. The same pattern holds across scenes within ScanNetv2. When we sort the scenes by per-scene pose error and split them into three groups, multi-view depth outperforms single-image depth in 45% of the lowest-error scenes, 41% of the middle group, and only 3% of the highest-error scenes. Across all scenes, the multi-view AbsRel also increases with the pose error (Spearman $\rho= 0.45$). E-RayZer follows the same pattern, with 94%, 88%, and 56% of scenes and $\rho= 0.61$. This suggests that pose estimation and multi-view depth depend on the same cross-view reasoning.

Input	Method	ScanNetv2		NRGBD
		AbsRel↓	δ < 1.25 ↑	AbsRel↓	δ < 1.25 ↑
Single image	E-RayZer	0.201	0.721	0.190	0.713
	Ours	0.099	0.911	0.113	0.859
Multi-view	E-RayZer	0.168	0.819	0.095	0.908
	Ours	0.125	0.872	0.082	0.917
Pose AUC@30↑	E-RayZer	39.1		83.1
	Ours	49.9		87.2

Analyses

We conduct controlled analyses on the DL3DV dataset to isolate the effect of each design choice. Unless stated otherwise, we use the full-view attention design of § 3.2 and match model size and training compute across variants. We evaluate with $\{5, 7, 9, 11\}$ views sampled from a fixed 48-frame span, reporting the average across view counts.

Self-supervised pretraining. We investigate whether self-supervised learning improves subsequent supervised training, using the same architecture and data throughout. The supervised model (SL) is trained from scratch for 200K iterations with dataset camera poses and a relative camera loss [54]. The self-supervised model (SSL) is trained without geometric supervision for 100K iterations. The fine-tuned model (SSL+SFT) starts from SSL and receives the same supervision as SL for another 50K iterations, for 150K iterations in total.

As shown in Table 4, SSL alone is comparable to SL, and is more accurate at strict pose thresholds despite using half the iterations. SSL+SFT surpasses both on every reported metric, while training for fewer total iterations than SL. Self-supervised learning thus provides an effective initialization for supervised geometry estimation.

Method	DL3DV (trained)				ScanNetv2				NRGBD
	RPA@3↑	@5↑	@10↑	AUC@30↑	RPA@3↑	@5↑	@10↑	AUC@30↑	RPA@3↑	@5↑	@10↑	AUC@30↑
SSL	66.6	81.7	91.7	88.0	1.2	5.2	17.6	29.8	16.5	37.8	74.3	72.3
SL	50.8	74.6	92.5	87.2	0.7	3.2	13.5	25.2	16.3	41.2	82.7	77.2
SSL+SFT	73.4	89.3	97.2	92.4	3.4	10.1	28.4	41.4	33.3	62.0	87.9	81.9

Asymmetric feature alignment. Eq. (3) aligns the intermediate features of a sparse-context model to those of a dense-context model of the same scene. We compare against depth alignment, which instead applies a scale-invariant loss between the two depth maps. The alignment loss is introduced at 40% of training and reaches full weight at 60%. Table 5 shows that feature alignment improves every metric over training without alignment.

Method	Camera pose		Multi-view depth			Single-view depth
	RPA@3 $\uparrow$	RPA@5 $\uparrow$	AbsRel $\downarrow$	$\delta< 1.25 \uparrow$	$\delta< 1.25^{2} \uparrow$	AbsRel $\downarrow$	$\delta< 1.25 \uparrow$	$\delta< 1.25^{2} \uparrow$
No alignment	66.6	81.7	0.195	0.758	0.901	0.230	0.638	0.850
Depth alignment	63.4	81.7	0.244	0.692	0.854	0.248	0.651	0.844
Feature alignment (ours)	69.8	84.6	0.191	0.768	0.905	0.191	0.729	0.901

In contrast, depth alignment does not help. Figure 9 examines the metrics over training. Beyond standard metrics, we measure shape, the Pearson correlation $\rho(\log\hat{z}, \log z)$ against pseudo ground truth, which is invariant to the global scale and captures how depth is ordered across the image.

Under depth alignment, shape steadily declines, so errors in the dense-context depth are passed on as wrong geometry, not just wrong scale. Feature alignment matches representations rather than depth values, leaving depth to the rendering loss.

Dynamic monocular videos. We also explore training on dynamic monocular videos from SpatialVID [49], a large-scale dataset with diverse camera motion and moving content. We train the same model on DL3DV and SpatialVID with equal sampling per epoch, keeping the architecture, objective, and training schedule unchanged. The model thus still assumes a static scene, although many clips contain moving objects. Without any architectural change, the model tends to assign low opacity to Gaussians on moving objects (Figure 10). Since no static scene can explain these regions, their photometric error offers no consistent signal for camera pose, which likely leaves the static structure to drive camera estimation.

To examine how dynamic content affects pose estimation, we construct three disjoint test sets of 100 held-out SpatialVID scenes each from the dataset’s metadata: near-static scenes (dyn_low) and highly dynamic scenes (dyn_high), taken from the lower and upper deciles of the dynamic ratio, and crowded scenes (crowded). For each scene, we sample 10 views over a 32-frame span and compare against the same model trained on DL3DV alone (Table 6). Training with SpatialVID improves pose accuracy in all three groups, including near-static scenes, suggesting that the gain may stem partly from greater camera motion diversity rather than solely from robustness to dynamic content. On DL3DV, the two models perform comparably. Figure 11 shows the recovered trajectories.

Training data	DL3DV benchmark	SpatialVID dyn_low	SpatialVID dyn_high	SpatialVID crowded
DL3DV	80.7 / 87.1	36.9 / 70.0	25.4 / 55.7	24.1 / 55.1
DL3DV + SpatialVID	79.1 / 85.8	74.7 / 86.0	73.9 / 84.0	87.4 / 89.2

Discussion

Design motivation. Prior pose-free methods restrict how held-out views enter the backbone, either by excluding them from geometry prediction or by preventing context views from attending to them. This choice is natural when rendering quality is the goal, since observing a held-out view could reveal what the model is asked to render. Our goal is geometry, with view synthesis serving only as the training signal. We keep the context and held-out split, but apply it to the 3D scene, so a held-out view is never rendered from its own prediction while still contributing correspondence in the shared backbone. For this reason, rendering metrics are not our primary basis of comparison.

All designs receive the same images during training and differ only in how they use them. In restricted designs, held-out views cannot inform the context-view predictions from which the scene is built. In ours, every view informs every other. The gain therefore comes not from more data but from using the same views more effectively.

Conclusion and future work. We study self-supervised geometry learning with rendering as the training objective and geometric accuracy as the outcome. Our central finding is that the asymmetry needed for self-supervision belongs in the 3D representation rather than in the network. Building on this, aligning sparse-context features with dense-context features turns view density into a further learning signal, and aligning with a frozen vision foundation model adds a semantic prior that pixels alone do not provide. Together, these choices form a simple design that improves camera pose and depth without geometric annotations.

A key opportunity is to learn from visual data beyond annotation pipelines. Much of this data is casually captured monocular video, where the scene itself changes between frames, while our framework assumes a static scene. Our preliminary experiment suggests that the static model already tolerates some motion by assigning low opacity to moving content, which improves camera pose estimation on dynamic videos. This does not model motion, however, and moving objects are simply left out of the reconstruction. A natural next step can be an explicit dynamic representation that separates object motion from camera motion, either with Gaussians that move over time ([56, 57]) or with a separate reconstruction at each timestep in a bullet-time formulation ([26]). Such a representation offers a path toward self-supervised geometry learning from the far larger supply of monocular video.

Appendix

Additional Details

Training details. We train the main model for 100K iterations on RE10K [66] and 140K iterations on DL3DV [28] on 8 NVIDIA B200 GPUs. The analysis variants are trained for 100K iterations on DL3DV on 2 NVIDIA RTX A6000 GPUs. Images are resized to $256 \times256$ and tokenized with a patch size of 16, giving 256 tokens per view. Each batch uses a single view count from $\{5, 7, 9, 11\}$, split into $(N+1)/2$ context and $(N-1)/2$ target views, with per-device batch sizes of $\{24, 20, 16, 12\}$ to keep the number of tokens per step roughly constant. Following RayZer [18], we widen the frame span from which views are sampled over the first 80K iterations, moving from shorter to longer baselines, and keep it fixed afterward. The span grows from 24–96 to 48–192 frames on RE10K, from 24–48 to 32–54 on DL3DV, and from 24–32 to 32–48 for the analysis variants. We use AdamW [29] with $\beta_1 = 0.9$, $\beta_2 = 0.95$, and weight decay 0.05, together with a cosine schedule and 3000 warm-up steps. The learning rate is $2 \times10^{-4}$ for the main model and $1 \times10^{-4}$ for the analysis variants. Training uses bf16 and FlashAttention-2 [9], with gradients clipped to a norm of 1.0.

Evaluation details. We evaluate on 5,601 RE10K scenes [66], 140 DL3DV scenes [28], 9 NRGBD scenes [1], and 97 ScanNetv2 scenes [8]. We evaluate SPFSplat and NAS3R with their official evaluation code, which applies no masking at test time. In this comparison, depth is predicted and scored on all input views for both the baselines and our model. For E-RayZer, poses are estimated from all images and depth from the context views only. Accordingly, when comparing with E-RayZer, our DPT head decodes only the context views, and depth is scored on the same context views for both methods.

Architecture. Figure 12 illustrates the pipeline. A visual encoder tokenizes each view, and a learnable camera token is prepended to its patch tokens. Tokens from both context (C) and held-out (H) views enter a unified estimator with alternating view-wise and global attention, so every view can interact with every other. The resulting patch tokens feed a DPT head [37], which reads four intermediate estimator layers to predict per-pixel depth and the remaining Gaussian attributes. The camera tokens feed a relative pose head, which regresses each pose relative to the middle input view. Context-view depths are unprojected with the predicted cameras to form the Gaussian scene, which is then rendered at the predicted held-out poses. Since the asymmetry is imposed in the 3D representation, attention in the estimator remains unrestricted.

Additional Analyses

View synthesis under full-view attention. Full-view attention lets target images inform context tokens at test time, so its rendering metrics are not directly comparable to those of restricted designs. To evaluate view synthesis without this leakage, we train with full-view attention for the first 80% of iterations and switch to masked attention for the remainder, then evaluate with masked attention. We compare against a model

trained with masked attention throughout under the same total schedule. The switched model renders consistently better (Figure 13), suggesting that the geometry learned under full-view attention carries over once leakage is removed.

Effect of context in the posed setting. The main experiments predict cameras jointly with geometry, so the benefit of full-view attention could come from better pose estimation rather than better geometry. To separate the two, we repeat the comparison with ground-truth cameras on DL3DV dataset. Both variants build Gaussians from context views only and differ in a single choice. FULL feeds all views, context and target, into the encoder during training, while MASK feeds only the context views. If target views offered a shortcut, the geometry predicted by FULL would become biased toward the targets and degrade. If they instead supply correspondence, FULL should recover better geometry. At test time, both models encode only the context views, so their inputs are identical. As a diagnostic, we also evaluate FULL with all views encoded, as during training, to see how much it relies on target views. Depth is always scored on the same context views.

More context improves geometry. With identical inputs at test time, FULL recovers better depth than MASK on every metric and at every view count (Table 7), and the gap holds throughout training (Figure 14). Since the two models differ only in whether target views entered the encoder during training, the gain comes from training with unrestricted interaction.

Views	FULL					MASK
	AbsRel ↓	$\delta< 1.25$ ↑	$\delta< 1.25^2$ ↑	PSNR ↑	LPIPS ↓	AbsRel ↓	$\delta< 1.25$ ↑	$\delta< 1.25^2$ ↑	PSNR ↑	LPIPS ↓
7 (4/3)	0.271 (0.274)	0.729 (0.743)	0.887 (0.892)	20.42 (21.32)	0.365 (0.328)	0.306	0.693	0.873	21.17	0.328
9 (5/4)	0.276 (0.281)	0.734 (0.746)	0.887 (0.892)	20.92 (21.74)	0.344 (0.312)	0.319	0.689	0.871	21.68	0.311
11 (6/5)	0.276 (0.286)	0.743 (0.748)	0.892 (0.894)	21.27 (21.98)	0.329 (0.301)	0.319	0.688	0.872	22.00	0.299

Semantic guidance. Table 8 isolates the effect of aligning encoder features with a frozen vision foundation model. We use either Pixio (Yang et al., [59]), a masked autoencoder, or DINOv3 (Simóni et al., [45]), each serving both to initialize the encoder and as the frozen model. Pretrained initialization already matters substantially, as training from random initialization lowers every metric by a wide margin. Semantic guidance improves on this for both encoders, even though the encoder starts from the same pretrained weights it is aligned to. The largest gain is in camera pose, particularly at strict thresholds, and depth improves as well. This suggests that initialization alone does not retain the pretrained prior under the rendering loss, and that keeping it as a training signal helps preserve it. Adding a projection before the alignment performs similarly, so we use the version without it. Pixio yields clearly better pose than DINOv3 at comparable depth, and we use it in all other experiments.

Backbone	Method	RPA @ 3 ↑	RPA @ 5 ↑	RPA @ 10 ↑	AbsRel ↓	$\delta< 1.25$ ↑	PSNR ↑	LPIPS ↓
—	w/o Pretrained init	36.2	56.6	77.3	0.247	0.652	19.82	0.438
Pixio (base)	Baseline	66.6	81.7	91.7	0.195	0.758	21.48	0.312
	w/ Sem. guidance	82.3	90.8	96.0	0.185	0.790	21.55	0.305
	w/ Sem. guidance + Proj.	81.6	90.7	96.2	0.189	0.789	21.52	0.305
DINOv3 (base)	Baseline	46.8	64.3	80.0	0.217	0.718	20.23	0.361
	w/ Sem. guidance	62.4	79.3	91.2	0.181	0.778	20.40	0.361
	w/ Sem. guidance + Proj.	60.8	78.1	90.3	0.183	0.773	20.26	0.367

3D probing. We test whether semantic guidance improves the learned representation itself. Each self-supervised model (Pixio variants) from Table 8 is frozen, and only its prediction heads are trained with 3D supervision. We train the heads on BlendedMVS (Yao et al., [60]) and evaluate on 43 held-out scenes, and train them on ScanNet++ (Yeshwanth et al., [61]) and evaluate on 97 ScanNetv2 (Dai et al., [8]) scenes. Without pretrained initialization, performance is lower on every metric. Semantic guidance improves on the self-supervised baseline on both datasets, in depth as well as pose, with the largest gains in camera pose (Table 9). Since only the heads are trained, these gains reflect a better representation rather than a better fit to the probing data.

Method	BlendedMVS					ScanNetv2
	AbsRel $\downarrow$	$\delta< 1.25$ $\uparrow$	RPA@5 $\uparrow$	RPA@15 $\uparrow$	RPA@30 $\uparrow$	AbsRel $\downarrow$	$\delta< 1.25$ $\uparrow$	RPA@5 $\uparrow$	RPA@15 $\uparrow$	RPA@30 $\uparrow$
w/o Pretrained init	0.162	0.800	4.2	16.9	26.7	0.179	0.745	0.7	9.8	28.7
Baseline	0.113	0.891	9.1	25.0	37.7	0.120	0.873	2.9	21.0	47.7
w/ Sem. guidance	0.094	0.919	12.6	33.0	44.3	0.078	0.943	6.2	39.5	67.6
w/ Sem. guidance + Proj.	0.096	0.919	11.8	28.5	41.5	0.082	0.932	6.4	35.7	63.0

Flow estimation. We also probe dense two-view correspondence. Following UFM (Zhang et al., [63]), we train only a DPT flow head on top of each frozen model on StaticThings3D (Schroppel et al., [43]), and evaluate on non-occluded pixels of 200 image pairs from 50 held-out sequences. Without pretrained initialization, performance is again lower on every metric and semantic guidance improves on the baseline (Table 10).

Method	EPE $\downarrow$	$<1$px $\uparrow$	$<3$px $\uparrow$	$<5$px $\uparrow$
w/o Pretrained init	4.05	25.0	68.0	81.9
Baseline	3.16	32.0	74.7	86.7
w/ Sem. guidance	2.74	31.4	77.4	89.6
w/ Sem. guidance + Proj.	2.66	32.3	78.7	90.2

Combining the two signals. Table 11 adds asymmetric context alignment and semantic guidance to the self-supervised baseline, separately and together. Each signal improves every metric over the baseline, but to different degrees. Semantic guidance gives the larger gain on most metrics, especially camera pose at strict thresholds, whereas the gains of asymmetric feature alignment concentrate in single-view depth, consistent with its role of transferring dense-context features to sparse inputs. The two are complementary. Enabling both yields the more accurate camera pose and the best single-view depth. Multi-view depth remains on par with semantic guidance alone, so asymmetric feature alignment adds its benefit in pose and in depth from sparse context rather than in dense multi-view setting. Figure 15 shows per-view pose evaluation results.

Sem. guid.	Asym. align.	Camera pose				Multi-view depth		Single-view depth
		RPA@3 $\uparrow$	RPA@5 $\uparrow$	RPA@10 $\uparrow$	AUC@30 $\uparrow$	AbsRel $\downarrow$	$\delta< 1.25$ $\uparrow$	AbsRel $\downarrow$	$\delta< 1.25$ $\uparrow$
		66.6	81.7	91.7	88.0	0.195	0.758	0.230	0.638
	✓	69.8	84.6	93.5	89.9	0.191	0.768	0.191	0.729
✓		82.3	90.8	96.0	93.0	0.185	0.790	0.173	0.757
✓	✓	82.8	92.2	96.9	93.7	0.188	0.790	0.165	0.799

Comparison with neural rendering methods. We compare our explicit 3D representation with methods that render target views through a learned decoder (Sajjadi et al., 2022; Jin et al., 2024) (Tab. 12). These methods train with either interpolative targets, sampled between context views ({citep}Jiang et al., 2025b; Wang et al., 2025a`), or predictive targets that may lie beyond them [31, 35]. To control for differences in data and training, we implement one variant of each within our framework. Both use the same 12-layer backbone as our model followed by a 12-layer transformer decoder, and receive the same training data and compute. For the predictive variant, we follow the random-order autoregressive training of RayDer [35]. Both variants achieve higher rendering scores, but neither predicts camera poses that correspond to physical camera motion, and neither produces explicit depth. XFactor [31] evaluates whether learned latent poses transfer across scenes, which tests their consistency rather than their physical accuracy. Since our goal is geometry, we evaluate against the ground truth.

Method	NVS		Camera pose				Depth
	PSNR ↑	LPIPS ↓	RPA@3 ↑	RPA@5 ↑	RPA@10 ↑	AUC@30 ↑	AbsRel ↓	δ < 1.25 ↑
Neural rendering (interpolative)	22.28	0.258	0.0	0.0	1.1	0.26	–	–
Neural rendering (predictive)	22.10	0.258	0.0	0.3	0.8	0.21	–	–
Ours	21.75	0.280	85.7	94.1	98.5	95.12	0.230	0.775

More evaluation on dynamic videos. We also evaluate the two models from the dynamic-video experiment, one trained on DL3DV only and one on DL3DV with SpatialVID, on ORBIT [39]. Neither model is trained on this benchmark. ORBIT renders perspective views from $360^\circ$ videos along virtual camera trajectories and provides ground-truth camera poses. Many clips also contain moving people and objects. We used 44 clips, which come from 21 video segments of 16 YouTube videos and have 203 to 1,024 frames each. We split each clip into consecutive, non-overlapping windows of $S$ frames, keep only windows that fit entirely within the clip, and evaluate each window as a separate sample. From each window we sample 10 views evenly and pass all of them to the model. This gives 330 windows for $S=64$ and 220 for $S=96$.

Training with SpatialVID improves pose accuracy at both spans and at every threshold (Table 13). Figure 16 shows example trajectories. The model trained on DL3DV only drifts on scenes with moving people, while the model trained with SpatialVID stays close to the ground truth. Accuracy at the strictest thresholds are low for both, showing that the large rotations in ORBIT remain challenging.

$S$	Training data	$N$	RPA@3 ↑	RPA@5 ↑	RPA@10 ↑	RPA@15 ↑	RPA@30 ↑	AUC@30 ↑
64	DL3DV	330	5.5	13.6	33.7	48.1	71.6	42.5
	DL3DV + SpatialVID		7.5	17.3	39.2	55.9	76.7	47.9
96	DL3DV	220	5.3	12.5	31.4	46.2	68.4	40.3
	DL3DV + SpatialVID		7.3	16.4	36.7	51.9	71.7	44.7

More qualitative results. Figure 17 shows additional camera trajectories recovered by each method.