WorksComputer Science
Infer3D: Inverse Generative Modeling Enables Robust 3D Reconstruction
Abstract
Humans can understand the 3D structure of scenes even from highly unfamiliar or extreme viewpoints. Cognitive science suggests this generalization stems from a top-down generative inference strategy. In contrast, state-of-the-art 3D reconstruction neural networks typically rely on amortized, feedforward inference. While fast, this pattern-matching approach degrades when faced with out-of-distribution (OOD) scene conditions unseen during training. Inspired by the generative nature of human vision, we propose Infer3D, an inverse generative modeling framework that recovers the latent variables of a generative model to best match the monocular input. To prevent degenerate flat outputs, we couple the generative image prior with a frozen 3D lifting network that constrains the optimization space to a valid, multi-view-consistent 3D manifold. Across a variety of controlled synthetic and real-world tasks, Infer3D demonstrates superior OOD reconstruction accuracy compared to feedforward networks trained on equal or larger datasets. Furthermore, mirroring how human perception requires more time to process confusing stimuli, we introduce an adaptive inference strategy that expends more test-time compute on ambiguous OOD examples while efficiently routing in-distribution inputs. Project website: https://amanikiruga.github.io/infer3d-site/
Introduction
The human visual system exhibits a remarkable ability to parse the 3D world from 2D projections. Crucially, this capability is not brittle: humans can readily understand the 3D structure of objects and scenes even when viewed from highly unconventional angles ([79]), under strange lighting ([2]), or under different compositional arrangements ([35]). A central challenge in perception theory is explaining how human vision can be so robust. A rich body of literature in cognitive science and developmental psychology suggests that this robustness arises because human vision is not merely a passive, bottom-up filter but also includes an active, top-down generative process. By maintaining an internal mental model of the physical world, the mind can continuously predict and match sensory inputs, allowing it to reliably resolve ambiguities in unseen environments ([55, 16, 25, 62]).
This cognitive perspective heavily influenced classical computational theories, which formalized perception as “vision as inverse graphics” ([81]). In this paradigm, visual understanding is cast as Bayesian inference over a generative model of image formation and scene structure ([33, 32]). Rather than simply recognizing 2D patterns, the visual system performs analysis-by-synthesis, actively searching for the underlying 3D scene representation that best explains the observed 2D image.
In contrast, modern monocular 3D reconstruction systems have largely abandoned this generative top-down inference in favor of amortized, feedforward architectures ([58, 80, 14, 72, 28]) that directly map a 2D input image to a 3D output. While this pattern-matching approach enables very fast inference, it fundamentally lacks the robustness of generative inference. As we demonstrate in this paper, feedforward networks degrade in unpredictable ways when faced with out-of-distribution (OOD) scene conditions unseen during training, such as unconventional camera viewpoints (see Figure 2).
Figure 2. Failure under Viewpoint Shift. Left: Heatmap of Splatter Image (Szymanowicz et al., [58]) PSNR evaluated on input at different elevations (-axis) and rolls (-axis). Performance collapses rapidly in regions of OOD elevation or roll. Right: Comparison of extracted meshes (Marching Cubes) for an OOD view; the baseline exhibits strong artifacts, whereas our method recovers coherent 3D structure.
Inspired by the generative nature of human vision and classic analysis-by-synthesis methods ([6, 81]), we propose Infer3D. Rather than trusting a feedforward encoder on OOD data, Infer3D formulates perception as an inverse generative problem. We search for the scene geometry and rendering parameters that maximize the explanatory likelihood of the input observation. To ensure our optimization remains constrained to plausible, multi-view-consistent 3D structures, we couple a 2D generative image prior with a frozen 3D lifting network. This encourages the optimized latent variables to always decode to a valid representation on the training manifold, effectively synthesizing an internal hypothesis that best matches the confusing sensory input. Furthermore, mirroring how human perception requires more time to process ambiguous stimuli ([29, 60, 66]), we introduce an adaptive inference strategy. Because our generative framework leverages an encoder for initialization, we show that the initial reconstruction error provides a very reliable and efficient signal for OOD detection. This allows Infer3D to dynamically route standard inputs through fast feedforward passes while reserving expensive test-time optimization exclusively for OOD images.
Overall, our work makes the following contributions: (1) we introduce Infer3D, a modular analysis-by-synthesis framework that combines a learned 2D generative model with a feedforward 3D lifter to enable multi-view-consistent, OOD-robust inference from single images; (2) we propose an adaptive inference and OOD detection strategy that leverages the initial reconstruction error of the generative prior, requiring no external classifiers; and (3) through a controlled experimental setup, we demonstrate that Infer3D successfully generalizes to unseen viewpoints, non-linear lens distortions, and unobserved textures where state-of-the-art feedforward methods collapse (see Figure 1).
Figure 1. Robust 3D reconstruction under unseen conditions. Out-of-distribution (OOD) inputs categorized by domain shifts: unseen camera viewpoints (top), unseen sensors (middle), and real-world texture and lighting (bottom). Infer3D infers the underlying scene () and rendering parameters () despite these challenging OOD conditions.
Prior Work
Feedforward Inference. There has been a lot of success in models that can reconstruct 3D scenes from monocular images. Popular choices of 3D representation are meshes ([41, 63, 15]), or neural radiance fields ([80, 65, 57, 28, 74]). Modern methods rely more on 3D Gaussian representations that also use volume rendering, but are significantly more efficient than neural radiance fields ([14, 58, 75, 61]). In addition, point-based representations are also emerging as a popular choice [69, 67]. These approaches use neural networks to directly transform the input image to the output 3D reconstruction and thus struggle to reason about out-of-distribution settings.
Several papers demonstrate lack of generalization with feedforward inference in out-of-distribution settings [83, 43, 3, 1]. Mitigating such failures is also a topic of active research. Several works have introduced equivariant neural network models that aim to generalize in OOD conditions. However, none of these approaches is as general as our method, e.g., we show that our method can be used for images captured under OOD camera poses as well as under OOD textures. In particular, equivariant approaches either assume 3D or multiview inputs [76, 19, 24], or only include equivariance in parts of their network, e.g., equivariant intermediate representations without equivariant transformations between images and the corresponding representations [23, 45]. Equivariant models that take 2D image inputs currently only guarantee equivariance under 2D or color transformations [17, 78]. There is also a line of work in using 3D models for data augmentation [84, 77, 86]. However, no such method has been proposed for monocular 3D reconstruction.
Analysis-by-Synthesis. Rather than directly transforming an image into the corresponding 3D output with feedforward inference, analysis-by-synthesis solves an optimization problem for inference that searches for the optimal 3D solution that can explain the input observation. Because of the ambiguities in the problem, a good prior over 3D models is necessary. Many approaches use 3D training data, or pre-captured object geometries, to learn these priors [7, 81, 46, 27]. Recent papers instead rely on learning priors from 2D data, e.g., using a monocular image generative model as a prior over renderings of the 3D reconstruction [49, 73, 44, 71]. However, monocular priors are not sufficient to enforce multi-view consistency in output. Other approaches use multi-view priors [39, 75, 42, 59, 26]; however, these methods do not generalize to OOD images as they process input images with neural networks. The Bayesian approach of Anciukevicius et al. [5] is somewhat related to us, but it makes strong assumptions by assuming known scene parameters for the OOD inputs. Recent work on inverse generative modeling for scene understanding [70] is also closely related in spirit: it formulates perception as inverting a learned generative model and introduces compositional structure to handle complex 2D scenes. We are inspired by this view of inference as solving an inverse generative problem, but focus on multi-view-consistent 3D reconstruction and explicit control over camera parameters, rather than 2D compositional layouts.
Hybrid Inference. Hybrid inference methods have been explored in the literature [64, 4, 37]. However, they assume that the initial prediction is a reliable initialization—an assumption that breaks down under OOD conditions. In contrast, we propose an adaptive inference strategy that explicitly detects OOD inputs and selectively invokes robust optimization-based inference only when needed.
Optimization-Based Generative Inversion. -GAN [11] fits the latent conditioning of a 3D-aware GAN to an image but does not infer an unknown camera. BRFI [48] and 3D GAN Inversion [34] reconstruct 3D shape by jointly optimizing a generative latent code and camera pose. Both refine a single initialization obtained from learned pose or latent predictors, making them sensitive to initialization errors under distribution shift. FINV [56] filters multiple hypotheses using provided camera poses and additional observations, whereas Infer3D searches multiple joint latent and pose hypotheses from a single unposed image. These methods couple appearance and geometry within a monolithic 3D GAN. Infer3D instead searches a powerful 2D generative prior for appearance while a separately trained lifter supplies the 3D constraint, allowing the prior and lifter to be selected independently. We directly compare against BRFI and 3D GAN Inversion in Table 5 and Table 10.
Table 5. 3D reconstruction accuracy on unseen texture and lighting. Chamfer Distance (, ) on ARKit car scenes from the RealCars dataset.
| Method | ID SRN Cars PSNR ↑ | OOD SO(3) PSNR ↑ |
| BRFI | 19.85 | 14.30 (17.70) |
| 3D GAN Inversion | 16.22 | 15.23 (17.37) |
| Infer3D (StyleGAN2) | 21.43 | 20.29 |
Table 10. Viewpoint generalization on ShapeNet Cars. We compare our method’s performance against other 3D inversion methods: BRFI [48] and 3D GAN Inversion [34]. All 3 methods are given unposed input. Performance with oracle pose is reported in parentheses. Oracle-pose variants receive the ground-truth test camera pose.
Method
We provide an overview of our method in Figure 3. We first describe the 3D scene representation and our general analysis-by-synthesis framework in § 3.1. We then instantiate this framework across three distinct out-of-distribution (OOD) applications in § 3.3: unseen camera poses, unseen sensor distortions, and unseen appearance. Finally, we discuss efficient OOD detection in § 3.5.
Figure 3. Analysis-by-synthesis inference paradigm. To handle OOD inputs, we employ a test-time optimization loop (red arrow). We freeze the Image Generator and 2D to 3D Lifter (indicated by lock icons) to preserve the learned prior. Instead, we optimize the Latent Code and Rendering Parameters (indicated by flame icons) to minimize the reconstruction loss between the rendered reconstruction and the OOD input. This ensures the input to the lifter remains a valid predicted ID image while matching the OOD observation.
Scene Representation and Differentiable Renderer
The 3D reconstructions are represented using a scene representation which describes the underlying geometry and appearance of objects in the scene. This representation can be rendered into an image using a differentiable forward model , with rendering parameters . Depending on the scenario, is parameterized to represent unknown extrinsic camera poses (viewpoint shifts) or expanded to include uncalibrated intrinsic sensor characteristics (e.g., non-linear fisheye lens distortions).
3D Gaussians. In practice, we use 3D Gaussians [31] as our 3D representation, where the scene is described using a collection of 3D Gaussians that represent its geometry and appearance. Each Gaussian is described with parameters that determine its shape, opacity, and color. We render the 3D Gaussians using the differentiable splatting-based renderer from 3DGS [31], which takes camera parameters as additional input.
Inference via Generative Decoder Inversion
Standard feedforward reconstruction networks function as amortized encoders, mapping observations directly to 3D representations. While efficient, these encoder-based methods lack the inductive biases required for generalization to out-of-distribution (OOD) scene conditions [9]. To achieve robustness, we formulate perception as an inverse generative problem via maximum a posteriori (MAP) estimation. Given an OOD observation , we seek the scene parameters and rendering parameters that maximize the posterior:
Directly modeling a 3D prior is intractable. Instead, we leverage the manifold of a 2D generative model trained on in-distribution images, composed with a deterministic 2D-to-3D lifter . This implicitly defines a structural prior where valid scenes are those generated by lifting valid in-distribution images: , where are samples from the latent space of the generative model. By substituting this parameterization into Eq. (1), the inference reduces to a search over the latent code and parameters :
Algorithm 1 Analysis-by-Synthesis via Generative Inversion
1: Input: OOD Image , Generator , Lifter , Renderer 2: Input: Schedule , Learning rate 3: Initialize latent and camera populations 4: 5: 6: if Encoder is available then 7: Warm-start the first particle 8: 9: end if 10: Compute regularization anchor 11: for do 12: 13: Render hypotheses and compute loss for all particles 14: 15: 16: Gradient update 17: 18: Prune worst-performing particles 19: Keep the top particles in with the lowest 20: end for 21: return Best scene and pose
In practice, the (unscaled) negative log likelihood is implemented as , where is the differentiable renderer from 3DGS and is a reconstruction loss. We approximate the prior term as a quadratic penalty centered on the mean of the initial latent population, serving to discourage the optimization from drifting too far from the valid manifold initialization. The final term defines the prior over rendering parameters, described further in § 3.3.
Optimization Dynamics. We solve Eq. (2) via gradient descent. To avoid local minima in the non-convex energy landscape, we employ a multi-start strategy with particle filtering. We initialize a population of latent codes and parameters . The population is iteratively refined, with high-error particles culled periodically, yielding a set of plausible 3D interpretations for the ambiguous OOD input. This procedure is detailed in Algorithm 1.
The formulation in Eq. (2) ensures that the input to the lifting network is always an in-distribution image , strictly adhering to the manifold where the feedforward model is valid. It is also highly flexible: we can adapt to distinct domain shifts merely by redefining the optimized rendering parameters , their associated prior , and the likelihood. We instantiate this approach for three specific OOD applications below.
Adapting to Out-of-Distribution Shifts
Out-of-Distribution Camera Poses. For unseen camera viewpoint shifts, we jointly optimize the scene representation alongside the extrinsic camera parameters. Specifically, we optimize the latent code and the camera pose . We use a uniform prior over the camera pose parameters, continuously projecting rotation matrices onto the manifold via SVD Levinson et al., [38] to ensure stable gradient descent. The likelihood is computed as a combination of MSE and LPIPS losses Zhang et al., [82] to align low-frequency geometry and high-frequency textures.
Out-of-Distribution Sensor Shifts. Standard feedforward models assume strict pinhole projection and inherently fail under severe intrinsic parameter shifts, misinterpreting non-linear fisheye lens warping as curved scene geometry. To address this, we optimize the latent code , camera intrinsics, i.e., focal lengths and principal points (), and complex non-linear distortion parameters . The distortion prior is parameterized by a Hybrid Neural Distortion Field [21], which is a learnable polynomial fisheye model coupled with an iResNet on a sparse 2D grid. To evaluate the likelihood term, we first warp the perspective-rendered hypothesis generated by our generative model into the fisheye space using the learned forward distortion field (). We then compute MSE and LPIPS losses between this distorted rendering and the input OOD fisheye observation, effectively combining 3D Gaussian Splatting with a hybrid fisheye forward model.
Out-of-Distribution Appearance. When transitioning from synthetic training data to real-world captures, standard pixel-level reconstruction collapses due to large domain gaps caused by complex reflections, unobserved lighting, and novel materials. We can bridge this synthetic-to-real domain gap by leveraging the inherent modularity of our inverse framework. We replace the pixel-level likelihood in the previous cases with a deep semantic feature loss, maximizing the cosine similarity of DINOv2 [47] patch-level features extracted from both the OOD input and the rendered 3D hypothesis. As semantic features lack fine-grained spatial precision, we couple this with an explicit MSE depth consistency loss to constrain the geometric scale, all rendered via standard pinhole 3D Gaussian Splatting renderer. Similar to the OOD camera poses setting, we search for the latent code and camera pose .
Lifting and Generative Networks
3D Lifting Network (). Our framework is agnostic to the specific architecture of the network . In our experiments, we instantiate using either:
Splatter Image [58], which predicts pixel-aligned 3D Gaussians using a U-Net architecture.
CATSplat [53], which utilizes transformer-based context aggregation for more complex geometries.
Crucially, is frozen during inference; we rely on the optimization of to adapt the geometry.
Generative Image Priors (). We explore two classes of generative models to parameterize the image manifold, see appendix Fig. 8. The choice of dictates the structure of the latent space :
GAN-based: We employ StyleGAN2 [30]. The latent space is defined as , where refers to the intermediate 1D representation in the StyleGAN architecture. This generator provides a compact, smooth manifold that is amenable to gradient-based optimization.
Diffusion-based: We also utilize Diffusion Autoencoders (DiffAE) [50]. DiffAE encodes an image into a semantic latent and a stochastic latent . This offers a distinct advantage: unlike GANs, DiffAE includes an explicit encoder that can initialize for in-distribution data, significantly accelerating convergence (see § 3.5).
OOD Detection via Generative Prior
A key capability of our method is that it naturally exposes signals for out-of-distribution (OOD) detection, without requiring the training of additional classifier models.
Inference dynamics as an OOD signal. Given an input image, analysis-by-synthesis optimizes latent variables and rendering parameters to minimize a reconstruction loss. For in-distribution inputs , the generative encoder provides an initialization that lies close to an optimal solution. Consequently, the initial reconstruction error of the generated hypothesis is inherently low. In contrast, OOD inputs lie completely outside the support of the learned image distribution. The encoder fails to provide a near-optimal initialization, resulting in a severe structural mismatch and a high initial reconstruction loss.
We leverage this asymmetry to build an efficient OOD detector. Specifically, we run one forward pass through our method and use the initial reconstruction error (MSE and LPIPS) against the input image as our classification score. If this initial loss exceeds a calibrated threshold, the sample is flagged as OOD and routed to the full optimization loop (Algorithm 1); otherwise, we bypass the optimization and trust the feedforward lifter.
Automatic Threshold Calibration. Importantly, this detection signal requires no additional trained classifiers. To set the decision threshold, we perform an automatic calibration step on a small held-out validation set (5 ID and 5 OOD objects), selecting the threshold that maximizes the F1 score. As we batch all objects and run a single encode-decode pass, this entire one-time calibration process is fast, taking a maximum of 500 milliseconds on CO3D. At test time, we can accurately detect OOD inputs in roughly 286 milliseconds without constructing expensive backpropagation graphs. We quantitatively evaluate our fast OOD detection and calibration on CO3D and RE10K (Zhou et al., [85]) in § 4.3.
Experiments
We evaluate Infer3D’s capabilities across three core dimensions: (1) generalization to geometric and camera-level domain shifts, (2) robustness to appearance and environmental domain gaps, and (3) efficacy of our adaptive inference strategy. We report Peak Signal-to-Noise Ratio (PSNR, ), Structural Similarity Index Measure (SSIM, ), and Learned Perceptual Image Patch Similarity (LPIPS, ) (Zhang et al., [82]) for novel-view fidelity, Chamfer Distance (CD, ) for 3D geometry error, and the Area Under the Receiver Operating Characteristic Curve (AUROC, ) and F1 score () for OOD detection.
Geometric and Sensor Shift Generalization
We first evaluate how well our approach recovers consistent 3D geometry under extreme geometric perturbations, specifically out-of-distribution camera viewpoints and non-linear lens distortions.
Viewpoint Robustness. We evaluate on Multi-Category ShapeNet (Chang et al., [13]) (trained on a fixed elevation ring) and CO3D (Reizenstein et al., [52]) (Hydrants and Vases), see appendix Figure 9. We test under random rotations and translations that severely deviate from the training manifold. We compare against the state-of-the-art feedforward models Splatter Image (Szymanowicz et al., [58]),

Figure 9. Dataset splits for viewpoint robustness evaluation. Left: In-distribution training images from CO3D (Hydrants, Vases) and ShapeNet. These observations are captured from standard, constrained viewpoints, such as fixed elevation rings. Right: Out-of-distribution (OOD) test images featuring severe camera pose shifts. These include random translations and rotations over the full viewing sphere, resulting in unconventional angles, extreme elevations, and partial occlusions that challenge standard feedforward priors.
CATSplat [53], and Viewset Diffusion [57]. As shown in Table 1 and Table 2, feedforward baselines degrade sharply under OOD shifts, often hallucinating or flattening geometry (Figure 4a). By explicitly inverting generative models, Infer3D successfully matches the observation while maintaining view-consistent structure, recovering performance levels comparable to ID inference.
| Dataset | Method | In-distribution | OOD SE(3) | ||||||
| CD | PSNR | SSIM | LPIPS | CD | PSNR | SSIM | LPIPS | ||
| Hydrants | Splatter Image | 0.052 | 21.48 | 0.785 | 0.168 | 0.733 | 15.73 | 0.665 | 0.266 |
| Hydrants | Viewset Diffusion | 0.145 | 20.95 | 0.724 | 0.216 | 0.782 | 14.88 | 0.631 | 0.321 |
| Hydrants | CATSplat | 0.287 | 18.78 | 0.645 | 0.285 | 1.120 | 12.65 | 0.589 | 0.384 |
| Hydrants | Infer3D (DiffAE) | 0.102 | 20.55 | 0.761 | 0.182 | 0.461 | 18.46 | 0.714 | 0.220 |
| Hydrants | Infer3D (StyleGAN) | 0.220 | 17.73 | 0.722 | 0.224 | 0.637 | 17.08 | 0.694 | 0.251 |
| Vases | Splatter Image | 0.061 | 20.82 | 0.751 | 0.179 | 0.775 | 15.01 | 0.661 | 0.315 |
| Vases | Viewset Diffusion | 0.138 | 19.98 | 0.684 | 0.245 | 0.802 | 14.12 | 0.628 | 0.368 |
| Vases | CATSplat | 0.295 | 17.85 | 0.612 | 0.305 | 0.945 | 13.65 | 0.552 | 0.412 |
| Vases | Infer3D (DiffAE) | 0.119 | 19.65 | 0.715 | 0.215 | 0.552 | 17.65 | 0.678 | 0.261 |
| Vases | Infer3D (StyleGAN) | 0.245 | 16.92 | 0.675 | 0.258 | 0.725 | 16.27 | 0.658 | 0.292 |
Table 1. CO3D Evaluation (Hydrants and Vases). Metrics: CD , PSNR , SSIM , LPIPS . Bold = best.
| Setting | Method | CD | PSNR | SSIM | LPIPS |
| In-dist. | Splatter Img. | 0.122 | 24.27 | 0.911 | 0.107 |
| In-dist. | Infer3D (SG) | 0.147 | 21.61 | 0.878 | 0.133 |
| OOD SO(3) | Splatter Img. | 0.181 | 18.12 | 0.822 | 0.208 |
| OOD SO(3) | Infer3D (SG) | 0.137 | 20.67 | 0.865 | 0.148 |
| OOD SE(3) | Splatter Img. | 0.323 | 14.27 | 0.771 | 0.308 |
| OOD SE(3) | Infer3D (SG) | 0.306 | 16.30 | 0.795 | 0.238 |
Table 2. ShapeNet (11 Categories). Comparison of Splatter Image vs. Infer3D (Ours) (StyleGAN) across difficulty settings.
We further evaluate viewpoint robustness against BRFI [48] and 3D GAN Inversion [34] on ShapeNet Cars, for which pretrained generators are available. We use BRFI’s released model, which is trained on the full SRN pose distribution, whereas Infer3D’s prior and lifter only see restricted elevation rings of ShapeNet Cars. We adapt 3D GAN Inversion using the official EG3D [12] Cars checkpoint, following its initialization protocol to train latent and pose encoders on SRN training renders with known poses. On cars with OOD SO(3) viewpoints, Infer3D achieves 20.29 dB, outperforming BRFI (14.30 dB) and 3D GAN Inversion (15.23 dB), even when the baselines receive ground-truth poses (17.70 and 17.37 dB, respectively). Infer3D also preserves more of its ID performance than BRFI, with a drop of 1.14 versus 5.55 dB (Table 10).
For CO3D where scale-depth ambiguity is significant, we also incorporate a geometric consistency loss to constrain the optimization to plausible absolute depth scales. We minimize the Mean Squared Error (MSE) between the rendered predicted depth map and the ground-truth depth as . Without extra guidance, the optimization struggles to resolve absolute scale in CO3D (as also reported in related works [58, 65, 57]).
Sensor and Field-of-View Shift. Standard feedforward models assume strict pinhole camera projection. We evaluate robustness to intrinsic parameter shifts by simulating realistic wide-angle and radial/tangential fisheye distortions [36, 21] on RealEstate10K (RE10K). To handle this, we augment our optimization loop with a Hybrid Neural Distortion Field. As shown in Table 3 and Figure 4b, the baseline (CATSplat) misinterprets lens warping as scene geometry (e.g., curved walls). As a stronger baseline, Equidistant is given privileged access to the ground-truth source and target focal lengths and undistorts the input under an equidistant fisheye model before applying CATSplat. Because this model cannot capture the non-linear distortion in our inputs, residual warping remains. Infer3D explicitly models the forward imaging process, successfully disentangling the non-linear lens characteristics from the 3D geometry to recover a clean structural output.

Figure 4. Robustness to geometric and sensor domain shifts. (a) Viewpoint shift. Columns show the OOD input, Splatter Image baseline, Infer3D (Ours), and ground truth. Unlike the baseline, our method recovers view-consistent geometry without collapsing or hallucinating structure. (b) Sensor shift (fisheye distortion). Left to right: (1) OOD fisheye input, (2) our learned displacement field, (3) CATSplat baseline (incorrectly modeling distortion as curved geometry), and (4) our disentangled 3D geometry.
| Setting | Method | PSNR | SSIM | LPIPS |
| In-dist. | CATSplat | 25.41 | 0.840 | 0.149 |
| In-dist. | Infer3D (DAE) | 24.03 | 0.801 | 0.186 |
| OOD Fisheye | CATSplat | 16.25 | 0.610 | 0.327 |
| OOD Fisheye | Equidistant | 19.24 | 0.688 | 0.253 |
| OOD Fisheye | Infer3D (DAE) | 23.33 | 0.775 | 0.198 |
Table 3. RE10K Evaluation. Comparison of CATSplat vs. Infer3D (DAE) across in-distribution and extreme focal distortion
Appearance Shift and In-the-Wild Generalization
Unseen Texture and Lighting. We evaluate on the RealCars dataset (Du et al., [22]) (in-the-wild captures with complex reflections), restricting the training of our generative prior exclusively to synthetic SRN Cars (Sitzmann et al., [54]). For the 3D lifter, we use official Splatter Image weights pretrained on SRN Cars. Because standard pixel-level reconstruction fails under this synthetic-to-real gap, we integrate DINOv2 semantic feature optimization into Infer3D. While Splatter Image collapses (Chamfer Distance: 0.219) and generative baselines like Large Multi-View Gaussian Model (LGM) (Tang et al., [61]), trained on a much larger Objaverse (Deitke et al., [18]) dataset, hallucinate artifacts, our analysis-by-synthesis bypasses the texture nuisance and reconstructs the correct 3D pose and geometry, achieving a significantly lower mean Chamfer Distance of 0.131 (Table 5, Figure 5). For fairness, we also include a finetuned Splatter Image baseline where we provide it with the same DINO features we use as input (PCA-reduced features of channel-32) and pseudo-ground-truth depth from Depth Anything 3 (Lin et al., [40]). This improves the robustness of Splatter Image (Mean CD of 0.152 from 0.219) but still significantly underperforms Infer3D, demonstrating that inverse generative modeling remains essential for generalization.

Figure 5. Robustness to Unseen Texture and Lighting. Despite training exclusively on SRN Cars ([54]), our method (Infer3D) integrates semantic features (DINOv2) to infer plausible 3D structures unlike feedforward baseline Splatter Image (2nd column) or generative baseline LGM (3rd column) trained on equal or larger (Objaverse) synthetic datasets.
The improvement also holds against optimization-based inversion methods trained on synthetic SRN Cars. Infer3D achieves a mean Chamfer Distance of 0.131, compared with 0.491 for BRFI (Pavllo et al., [48]) and 0.387 for 3D GAN Inversion (Ko et al., [34]), reducing error by 73% and 66%, respectively (Table 5).
Zero-Shot In-the-Wild Captures. Figure 6 demonstrates Infer3D on casual smartphone photos with varying lighting, partial occlusions, and non-standard cropping. Feedforward baselines fail catastrophically as these images violate strict training preprocessing pipelines. By searching for the camera parameters that best align the generative prior to the noisy image, our compositional approach robustly recovers plausible shapes.

Figure 6. Reconstruction from in-the-wild real input images. Each row shows: the original photo; the binary mask fed to the model; the Splatter Image feed-forward prediction; and our analysis-by-synthesis reconstruction. Our method is robust to inputs of varying quality, including low-resolution photos, cluttered backgrounds, and masks that partially occlude the object.
Adaptive Inference and System Analysis
While optimization-based inference is highly robust, it requires more compute than feedforward mapping. We analyze the initial reconstruction error from our generative prior to validate an adaptive system that selectively routes inputs to reduce the effective amortized time per sample.
OOD Detection and Adaptive Routing. We follow the method of § 3.5 to perform OOD detection. We use the encoder of Diffusion Autoencoders (DiffAE) [50] to provide a near-optimal initialization for ID inputs, which exhibits a low initial reconstruction error. As shown in Table 6, our initial reconstruction error metric achieves strong detection performance with an AUROC of 0.972 on CO3D Hydrants, 0.961 on Vases, and 0.997 on RE10K. We compare this with an image alignment baseline: the OOD image is fed directly to Splatter Image and the output 3D Gaussians are rendered at identity pose I. We then build a classification score using the reconstruction accuracy of this render and the OOD image. Because encoder-based 3D reconstruction has a bias of copying the image at identity pose, this performs poorly and our method significantly outperforms it while requiring just roughly 286ms of latency. We also apply this adaptive strategy to a mixed stream of ID and OOD inputs (Table 4). In a stream with 10% OOD data, our system routes standard inputs through the fast feedforward filter and reserves optimization for outliers, achieving better performance than Splatter Image alone, or optimization alone. The amortized runtime is just 19 seconds per sample. Finally, calibration of our adaptive threshold is sample-efficient and stable across sample sizes. Fitting on merely 10 held-out objects takes 500ms and yields an F1 score of 0.965 (see Table 7).
| ID/OOD Split | F1 | Splatter Image | Opt. Only | Adaptive | Avg. Time |
| 99% / 1% | 0.943 | 21.42 | 20.53 | 21.45 | 2.1s |
| 90% / 10% | 0.955 | 20.91 | 20.34 | 21.16 | 19s |
| 70% / 30% | 0.971 | 19.76 | 19.92 | 20.54 | 55s |
| 50% / 50% | 0.986 | 18.61 | 19.50 | 19.94 | 1.4m |
Table 4. Adaptive Inference in Mixed ID/OOD Streams. Performance (PSNR) and amortized runtime using our adaptive strategy across simulated real-world distributions on CO3D Hydrants.
| Method | Hydrants | Vases | RE10K |
| Infer3D | 0.972 | 0.961 | 0.997 |
| Image alignment | 0.554 | 0.621 | 0.748 |
Table 6. Out-of-Distribution Detection AUROC. We evaluate the ability to distinguish between ID and OOD camera shifts.
| Num Val Objects | Threshold (ID %) | F1 Score |
| 10 | 97.7 | 0.965 |
| 20 | 97.1 | 0.944 |
| 30 | 97.3 | 0.955 |
Table 7. Calibration Sensitivity. The detection threshold is highly stable and requires very few validation objects to calibrate (CO3D Hydrants).
Ablations on Compute Tradeoffs. We run inference for 783 iterations, initializing 600 hypotheses (30 rotations 20 latent codes) and pruning to 32, 10, and 5 at iterations 3, 122, and 302. Where available, an encoder provides one warm-start hypothesis. Full optimization settings are given in Section B.3.
We find that the computational cost of Infer3D can be explicitly scaled to prioritize speed without sacrificing robustness. By parallelizing particle optimization (), full convergence is achieved efficiently in roughly 2.8 minutes. For applications requiring even faster turnaround, early stopping the optimization loop after just 100 steps yields a rapid 1.2-minute runtime per OOD sample. Notably, this early-stopping configuration still achieves a PSNR of 16.98 which still outperforms the baseline feedforward model Splatter Image (15.73) on OOD CO3D Hydrants. In practice, data streams mix standard inputs with occasional anomalies; pairing rapid OOD detection with analysis-by-synthesis (without early stopping) allows Infer3D to process mixed streams efficiently without bottlenecking pipelines (Table 4).
Objective and Optimization Ablations. Finally, we validate our optimization objective and the necessity of our constrained approach. As detailed in Table 8, relying solely on pixel-wise MSE for CO3D yields poor geometry. Sequentially integrating perceptual penalties (), latent regularization (), and depth consistency () is critical for recovering accurate out-of-distribution structures, improving Chamfer Distance from 0.954 to 0.458. Furthermore, we test whether simply updating the weights of the feedforward lifter on the OOD input (trivial test-time training) could achieve similar robustness. Table 13 demonstrates that this naive adaptation collapses the geometry and performs worse than the frozen baseline (12.41 vs. 15.73 PSNR), indicating that optimizing over an internal generative model is more effective than naive test-time adaptation of the lifter.
| Loss Function Configuration | PSNR | LPIPS | CD |
| 14.85 | 0.482 | 0.954 | |
| 16.20 | 0.315 | 0.725 | |
| (Mean Latent Dist. Reg.) | 17.45 | 0.255 | 0.580 |
| (Full Objective) | 18.51 | 0.218 | 0.458 |
Table 8. Ablation of Inference Losses. Evaluation of our objective components on CO3D Hydrants. Combining perceptual losses with latent and depth regularization is critical for recovering accurate OOD geometry.
| Method | PSNR | SSIM | LPIPS |
| Splatter Image | 15.73 | 0.665 | 0.266 |
| Test-Time Training | 12.41 | 0.569 | 0.382 |
| Infer3D (DAE) | 18.46 | 0.714 | 0.220 |
Table 13. Comparison with direct test-time training on OOD CO3D Hydrants. Best results are shown in bold.
We further examine how pose and latent optimization contribute to our method. On OOD SO(3) ShapeNet Cars, removing pose or latent updates reduces PSNR by 3.10 and 2.27 dB, respectively, while removing both reduces it by 4.08 dB (Table 9). We do not change the initialization in these ablations, only whether the target variable is optimized further or selected among initialization candidates. These results support jointly refining shape and viewpoint during inference and show the gain is spread across the two optimization regimes.
| Configuration | PSNR | PSNR |
| Full Infer3D (StyleGAN 2) | 20.29 | – |
| Without pose optimization | 17.19 | |
| Without latent optimization | 18.02 | |
| Without both | 16.21 |
Table 9. Pose and latent optimization ablation. Evaluation on OOD SO(3) ShapeNet Cars using the StyleGAN2 prior. Changes are relative to full Infer3D.
The optimization also progressively improves the fit to the input image, reducing MSE by 86% and LPIPS by 67% over the reported checkpoints on OOD ShapeNet Cars (Figure 7).

Figure 7. Optimization convergence on OOD ShapeNet Cars. Unweighted MSE and LPIPS reconstruction terms on OOD ShapeNet Cars objects rapidly decrease by 86% and 67%, respectively.
Limitations and Conclusion
Limitations. While Infer3D significantly improves robustness to OOD shifts, several limitations remain. While our method excels at OOD recovery, purely optimization-based inference can result in slightly lower quantitative metrics on in-distribution data compared to a specialized feedforward network. However, our proposed adaptive inference strategy effectively neutralizes this issue by classifying inputs based on convergence dynamics, ensuring that optimization is only triggered when necessary. We further discuss this drop in appendix Section C.6. Additionally, the visual sharpness of our OOD reconstructions is bound by the capabilities of the underlying 2D generative prior. In this work, we utilized lighter-weight generators (e.g., DiffAE, StyleGAN2) which can sometimes struggle with high-frequency details. We believe that integrating state-of-the-art diffusion models as the generative backbone would resolve this, albeit at a higher computational cost.
Conclusion. We presented a framework for robust monocular 3D reconstruction that treats inference as an inverse generative problem. We demonstrated that standard feedforward networks overfit to the training distribution, failing catastrophically under viewpoint, sensor, or appearance shifts. By combining a learned 3D lifter with a test-time optimization loop that enforces a 2D generative prior, we recover accurate geometry in scenarios where baselines produce collapsed artifacts. Furthermore, we showed that the dynamics of this optimization process serve as a strong signal for OOD detection, enabling an adaptive system that is both efficient on familiar data and robust to the unexpected. We hope this work encourages a shift from purely amortized inference toward hybrid methods for reliable 3D perception.
Additional Figures
Here we provide some additional figures that we reference in the main body text.
Figure 8. Generative Backbone Architectures. (a) StyleGAN2: Optimization occurs in the latent space for semantic consistency. (b) Diffusion Autoencoder (DiffAE): Optimization leverages both a semantic latent and a stochastic latent . Its architecture allows for an encoder-based warm-start.
Additional Method Details
In-the-wild generalization
In order to support running our 3D reconstruction inference pipeline on in-the-wild real input images, we first acquire pseudo-GT segmentation masks using Segment-Anything 2 [51] by initializing multiple point prompts, and selecting the largest mask that is close to the center. We then follow the method of Out-of-Distribution appearance to recover the geometry and camera extrinsics (See § 3.3).
Compute Resources
To run all our experiments we use 2 80GB A100 NVIDIA GPUs. At proposal batch size = 10, our full inference occupies a Peak VRAM of 48GB GPU memory.
Optimization Settings
Table 11 summarizes the optimization schedule. Iterations are indexed from 0 to 782. Each iteration updates all active hypotheses, processing at most simultaneously. Thus, controls memory usage rather than the total population size.
| Parameter | Value or schedule | Meaning |
| Initial rotation–latent hypotheses | ||
| 783 | Total iterations | |
| 10 | Maximum simultaneous hypotheses | |
| Pruning at | ||
| Changes at | ||
| Change at | ||
| (DiffAE) | 0.001 | Constant latent-prior weight |
| 0.01 with ramp-up and ramp-down | First 5% and final 25% of iterations | |
| 0.01 | Constant rendering-parameter learning rate |
Table 11. Optimization schedule. Population sizes, reconstruction-loss weights, and learning rates used in the reported inference procedure.
Initialization. When an encoder is available, we replace one random joint hypothesis with and identity camera pose. This hypothesis undergoes the same optimization and pruning as the remaining hypotheses. Without an encoder, all hypotheses are randomly initialized.
Prior-specific regularization. For DiffAE, we apply a quadratic latent penalty centered on the mean of the encoder-embedded initial latent population, with weight 0.001. For StyleGAN2, we instead use noise-map regularization to penalize spatial correlations in the optimized noise maps.
Learning rates and batching. The latent learning rate uses a base value of 0.01 with a cosine ramp-up over the first 5% of iterations and a ramp-down over the final 25%. The rendering-parameter learning rate remains 0.01. When the active population exceeds , we process it in successive batches, completing an iteration after all active hypotheses have been updated.
Additional Experiments
Component Ablations
We evaluate optimization variables, loss terms, and initialization counts on 25 OOD ShapeNet Cars using the StyleGAN2 prior. Component-removal experiments use identical examples and initializations. Unless specified otherwise, all settings follow the full configuration.
Removing MSE or LPIPS reduces PSNR by 1.20 and 0.53 dB, respectively, supporting their complementary contributions. Reducing the number of pose initializations from 30 to 10 costs 0.32 dB, while reducing latent initializations from 20 to 2 costs 0.80 dB. Noise-map regularization has a smaller effect of 0.12 dB.
| Configuration | PSNR | PSNR |
| Full Infer3D | 20.29 | – |
| Without pose optimization | 17.19 | |
| Without latent optimization | 18.02 | |
| Without both | 16.21 | |
| Without MSE | 19.09 | |
| Without LPIPS | 19.76 | |
| Without noise-map regularization | 20.17 | |
| Pose initializations: 30 10 | 19.97 | |
| Latent initializations: 20 2 | 19.49 |
Table 12. Component ablations on OOD ShapeNet Cars. Changes are relative to full Infer3D. The full configuration uses 30 pose and 20 latent initializations.
Comparison with test-time training
We evaluate direct test-time training (TTT) on OOD CO3D Hydrants by updating the feedforward lifter’s weights to match the input image. As shown in Table 13, this update scheme degrades novel-view reconstruction quality relative to the frozen lifter, whereas Infer3D improves all three metrics. For this setting, optimizing the generative latent while keeping the networks frozen is more effective than directly adapting the lifter’s weights.
Generative priors trained on diverse categories
To test compatibility with components trained on diverse categories, we combine a frozen class-conditional Equilibrium Matching (EqM) model (Wang & Du, [68]) trained on ImageNet (Deng et al., [20]) with a frozen Objaverse-trained Splatter Image lifter (Szymanowicz et al., [58]; Deitke et al., [18]). We evaluate on 25 partially occluded objects from a held-out set spanning six categories, using the same prior–lifter pair throughout. Infer3D-EqM reduces Chamfer Distance in every category and by approximately 30% overall (Table 14). Figure 10 illustrates improvements in object completeness, including the recovery of missing handles and body regions.
| Category | Direct lifter CD | Infer3D-EqM CD | Reduction |
| Teapot | 0.214 | 0.157 | 27% |
| Mug | 0.225 | 0.171 | 24% |
| Vase | 0.278 | 0.180 | 35% |
| Pineapple | 0.147 | 0.096 | 35% |
| Wine bottle | 0.145 | 0.109 | 25% |
| Helmet | 0.214 | 0.199 | 7% |
| All | 0.206 | 0.145 | 30% |
Table 14. Reconstruction with an Equilibrium Matching prior. Chamfer Distance (CD) and its relative reduction compared with applying the frozen lifter directly.

Figure 10. Qualitative reconstruction with an EqM prior. Left: partially occluded inputs. Middle: direct reconstructions from the frozen Splatter Image lifter. Right: Infer3D-EqM reconstructions. Two views are shown per reconstruction.
Comparison with SAM 3D
Table 15 compares novel-view synthesis quality on ShapeNet across 11 categories. Infer3D achieves the best results on all three metrics under OOD SO(3) input poses, while Splatter Image performs best on in-distribution inputs. Figure 11 presents selected SAM 3D failure cases, illustrating flattened or distorted geometry and missing structural components under challenging input orientations. We find that even in in-distribution settings, SAM3D does not accurately reconstruct the colors well and so the in-distribution NVS performance is also low. But the shape is not distorted in-distribution as it is OOD.
| Setting | Method | PSNR | SSIM | LPIPS |
| In-distribution | SAM 3D | 16.00 | 0.79 | 0.181 |
| In-distribution | Splatter Image | 24.27 | 0.911 | 0.107 |
| In-distribution | Infer3D (SG) | 21.61 | 0.878 | 0.133 |
| OOD SO(3) | SAM 3D | 16.18 | 0.79 | 0.194 |
| OOD SO(3) | Splatter Image | 18.12 | 0.822 | 0.208 |
| OOD SO(3) | Infer3D (SG) | 20.67 | 0.865 | 0.148 |
Table 15. Comparison with SAM 3D on ShapeNet (11 categories). Best results within each setting are shown in bold.

Figure 11. SAM 3D failure cases under OOD input poses. Each row shows an input image (left), two views of the ground-truth shape (middle), and two views of the SAM 3D reconstruction (right).
Potential application to robotic manipulation
Figure 12 illustrates a potential perception pipeline for robotic manipulation, where wrist-camera observations can involve unusual viewpoints and gripper occlusions. We estimate depth using Depth Anything 3 [40] and segment the target object using SAM 3 [10]. Given the masked RGB image and estimated depth, Infer3D jointly optimizes the generative latent and camera parameters to recover the bowl geometry. This example demonstrates reconstruction from a partially occluded wrist-camera observation; its utility for downstream manipulation remains to be evaluated.
Figure 12. Reconstruction pipeline for a robotic manipulation scene. A wrist-camera observation undergoes depth estimation and object segmentation, followed by Infer3D optimization.
In-Distribution Trade-off
Run on every input, the optimizer alone is less accurate than the feedforward lifter on ID data: by 0.93 dB on CO3D Hydrants, 2.66 dB on ShapeNet, and 1.38 dB on RE10K (Table 1 to Table 3). We perform analysis about this drop and find that the performance gap is concentrated near the input view. On ID CO3D Hydrants, the feedforward advantage is 5.7 dB for views close to the input but only 0.9 dB for the most occluded views. Unlike the feedforward lifter, the lifter inside Infer3D never receives the input image directly, so it cannot copy visible details into nearby views; for views that require inferring occluded 3D structure, Infer3D is nearly as accurate. In practice, an image-conditioned feedforward model is preferable for narrow-baseline rendering close to the input view, whereas Infer3D is more robust for wide-baseline views and 3D geometry. The detector-routed system (Table 4) avoids this cost on inputs classified as ID.
References
- [1]Amro Abbas and Stéphane Deny. Progress and limitations of deep networks to recognize objects in unusual poses. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 160–168, 2023.
- [2]Edward H Adelson. Lightness perception and lightness illusions. In Michael S Gazzaniga (ed.), The new cognitive neurosciences, pp. 339–351. MIT press, 2000.
- [3]Michael A Alcorn, Qi Li, Zhitao Gong, Chengfei Wang, Long Mai, Wei-Shinn Ku, and Anh Nguyen. Strike (with) a pose: Neural networks are easily fooled by strange poses of familiar objects. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4845–4854, 2019.arxiv.org/abs/1811.11553
- [4]Thiemo Alldieck, Marcus Magnor, Bharat Lal Bhatnagar, Christian Theobalt, and Gerard Pons-Moll. Learning to reconstruct people in clothing from a single rgb camera. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1175–1186, 2019.arxiv.org/abs/1903.05885
- [5]Titas Anciukevicius, Patrick Fox-Roberts, Edward Rosten, and Paul Henderson. Unsupervised causal generative understanding of images. Advances in Neural Information Processing Systems, 35:37037–37054, 2022.DOI
- [6]Volker Blanz and Thomas Vetter. A morphable model for the synthesis of 3d faces. In Seminal Graphics Papers: Pushing the Boundaries, Volume 2, pp. 157–164. 2023a.
- [7]Volker Blanz and Thomas Vetter. A Morphable Model For The Synthesis Of 3D Faces. Association for Computing Machinery, New York, NY, USA, 1 edition, 2023b. ISBN 9798400708978. URL https://doi.org/10.1145/3596711.3596730.
- [8]Mark Boss, Zixuan Huang, Aaryaman Vashista, and Varun Jampani. Sf3d: Stable fast 3d mesh reconstruction with uv-unwrapping and illumination disentanglement. arXiv preprint arXiv:2408.00653, 2024.arxiv.org/abs/2408.00653
- [9]Jack Brady, Bernhard Schölkopf, Thomas Kipf, Simon Buchholz, and Wieland Brendel. Generation is required for data-efficient perception. arXiv preprint arXiv:2512.08854, 2025.arxiv.org/abs/2512.08854
- [10]Nicolas Carion, Laura Gustafson, Yuan-Ting Hu, Shoubhik Debnath, Ronghang Hu, Didac Surís, Chaitanya Ryali, Kalyan Vasudev Alwala, Haitham Khedr, Andrew Huang, Jie Lei, Tengyu Ma, Baishan Guo, Arpit Kalla, Markus Marks, Joseph Greer, Meng Wang, Peize Sun, Roman Rädle, Triantafyllos Afouras, Effrosyni Mavroudi, Katherine Xu, Tsung-Han Wu, Yu Zhou, Liliane Momeni, Rishi Hazra, Shuangrui Ding, Sagar Vaze, Francois Porcher, Feng Li, Siyuan Li, Aishwarya Kamath, Ho Kei Cheng, Piotr Dollár, Nikhila Ravi, Kate Saenko, Pengchuan Zhang, and Christoph Feichtenhofer. SAM 3: Segment anything with concepts. arXiv preprint arXiv:2511.16719, 2025.arxiv.org/abs/2511.16719
- [11]Eric R Chan, Marco Monteiro, Petr Kellnhofer, Jiajun Wu, and Gordon Wetzstein. pi-gan: Periodic implicit generative adversarial networks for 3d-aware image synthesis. In Proc. CVPR, 2021.arxiv.org/abs/2012.00926
- [12]Eric R Chan, Connor Z Lin, Matthew A Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas J Guibas, Jonathan Tremblay, Sameh Khamis, et al. Efficient geometry-aware 3d generative adversarial networks. In Proc. CVPR, pp. 16123–16133, 2022.arxiv.org/abs/2112.07945
- [13]Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. Shapenet: An information-rich 3d model repository. arXiv preprint arXiv:1512.03012, 2015.arxiv.org/abs/1512.03012
- [14]David Charatan, Sizhe Lester Li, Andrea Tagliasacchi, and Vincent Sitzmann. pixelsplat: 3d gaussian splats from image pairs for scalable generalizable 3d reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19457–19467, 2024.arxiv.org/abs/2312.12337
- [15]Xingyu Chen, Fu-Jen Chu, Pierre Gleize, Kevin J Liang, Alexander Sax, Hao Tang, Weiyao Wang, Michelle Guo, Thibaut Hardin, Xiang Li, et al. Sam 3d: 3dfy anything in images. arXiv preprint arXiv:2511.16624, 2025.arxiv.org/abs/2511.16624
- [16]Andy Clark. Whatever next? predictive brains, situated agents, and the future of cognitive science. Behavioral and brain sciences, 36(3):181–204, 2013.DOI
- [17]Taco S Cohen and Max Welling. Steerable cnns. arXiv preprint arXiv:1612.08498, 2016.arxiv.org/abs/1612.08498
- [18]Matt Deitke, Ruoshi Liu, Matthew Wallingford, Huong Ngo, Oscar Michel, Aditya Kusupati, Alan Fan, Christian Laforte, Vikram Voleti, Samir Yitzhak Gadre, et al. Objaverse-xl: A universe of 10m+ 3d objects. Advances in Neural Information Processing Systems, 36:35799–35813, 2023.arxiv.org/abs/2307.05663
- [19]Congyue Deng, Or Litany, Yueqi Duan, Adrien Poulenard, Andrea Tagliasacchi, and Leonidas J Guibas. Vector neurons: A general framework for so (3)-equivariant networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 12200–12209, 2021.arxiv.org/abs/2104.12229
- [20]Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 248–255, 2009.
- [21]Youming Deng, Wenqi Xian, Guandao Yang, Leonidas Guibas, Gordon Wetzstein, Steve Marschner, and Paul Debevec. Self-calibrating gaussian splatting for large field of view reconstruction. arXiv preprint arXiv:2502.09563, 2025.arxiv.org/abs/2502.09563
- [22]Xiaobiao Du, Yida Wang, Haiyang Sun, Zhuojie Wu, Hongwei Sheng, Shuyun Wang, Jiaying Ying, Ming Lu, Tianqing Zhu, Kun Zhan, and Xin Yu. 3DRealCar: An in-the-wild RGB-D car dataset with 360-degree views. In Proc. ICCV, 2025.arxiv.org/abs/2406.04875
- [23]Emilien Dupont, Miguel Bautista Martin, Alex Colburn, Aditya Sankar, Josh Susskind, and Qi Shan. Equivariant neural rendering. In International Conference on Machine Learning, pp. 2761–2770. PMLR, 2020.arxiv.org/abs/2006.07630
- [24]Carlos Esteves, Christine Allen-Blanchette, Ameesh Makadia, and Kostas Daniilidis. Learning so (3) equivariant representations with spherical cnns. In Proceedings of the european conference on computer vision (ECCV), pp. 52–68, 2018.
- [25]Karl Friston. The free-energy principle: a unified brain theory? Nature reviews neuroscience, 11(2):127–138, 2010.DOI
- [26]Ruiqi Gao, Aleksander Holynski, Philipp Henzler, Arthur Brussee, Ricardo Martin-Brualla, Pratul P. Srinivasan, Jonathan T. Barron, and Ben Poole. Cat3d: Create anything in 3d with multi-view diffusion models. Advances in Neural Information Processing Systems, 2024.arxiv.org/abs/2405.10314
- [27]Nishad Gothoskar, Marco Cusumano-Towner, Ben Zinberg, Matin Ghavamizadeh, Falk Pollok, Austin Garrett, Josh Tenenbaum, Dan Gutfreund, and Vikash Mansinghka. 3dp3: 3d scene perception via probabilistic programming. Advances in Neural Information Processing Systems, 34:9600–9612, 2021.arxiv.org/abs/2111.00312
- [28]Yicong Hong, Kai Zhang, Jiuxiang Gu, Sai Bi, Yang Zhou, Difan Liu, Feng Liu, Kalyan Sunkavalli, Trung Bui, and Hao Tan. Lrm: Large reconstruction model for single image to 3d. arXiv preprint arXiv:2311.04400, 2023. 1, 2arxiv.org/abs/2311.04400
- [29]Kohitij Kar, Jonas Kubilius, Kailyn Schmidt, Elias B Issa, and James J DiCarlo. Evidence that recurrent circuits are critical to the ventral stream’s execution of core object recognition. Nature neuroscience, 22(6): 976–983, 2019. 2DOI
- [30]Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8110–8119, 2020. 6arxiv.org/abs/1912.04958
- [31]Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Trans. Graph., 42(4):139–1, 2023. URL https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/. 4arxiv.org/abs/2308.04079
- [32]Daniel Kersten, Pascal Mamassian, and Alan Yuille. Object perception as bayesian inference. Annual review of psychology, 55:271–304, 2004. 1pubmed.ncbi.nlm.nih.gov/14744217
- [33]David C Knill and Whitman Richards. Perception as Bayesian inference. Cambridge University Press, 1996. 1
- [34]Jaehoon Ko, Kyusun Cho, Daewon Choi, Kwangrok Ryoo, and Seungryong Kim. 3D GAN inversion with pose optimization. In WACV, 2023. 3, 8, 9, 11arxiv.org/abs/2210.07301
- [35]Brenden M Lake, Tomer D Ullman, Joshua B Tenenbaum, and Samuel J Gershman. Building machines that learn and think like people. Behavioral and Brain Sciences, 40:e253, 2017. 1arxiv.org/abs/1604.00289
- [36]Lensfun Developers. Lensfun: A library for lens distortion correction. https://lensfun.github.io/, 2024. 8
- [37]Brandon Leung, Chih-Hui Ho, and Nuno Vasconcelos. Black-box test-time shape REFINement for single view 3D reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pp. 4080–4090, June 2022. 3arxiv.org/abs/2108.09911
- [38]Jake Levinson, Carlos Esteves, Kefan Chen, Noah Snavely, Angjoo Kanazawa, Afshin Rostamizadeh, and Ameesh Makadia. An analysis of SVD for deep rotation estimation. arXiv preprint arXiv:2006.14616, 2020. 5arxiv.org/abs/2006.14616
- [39]Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming-Yu Liu, and Tsung-Yi Lin. Magic3d: High-resolution text-to-3d content creation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 300–309, 2023. 3arxiv.org/abs/2211.10440
- [40]Haotong Lin, Sili Chen, Junhao Liew, Donny Y. Chen, Zhenyu Li, Guang Shi, Jiashi Feng, and Bingyi Kang. Depth anything 3: Recovering the visual space from any views. arXiv preprint arXiv:2511.10647, 2025. 9, 22arxiv.org/abs/2511.10647
- [41]Shichen Liu, Shunsuke Saito, Weikai Chen, and Hao Li. Learning to infer implicit surfaces without 3d supervision. Advances in Neural Information Processing Systems, 32, 2019. 2arxiv.org/abs/1911.00767
- [42]Xiaoxiao Long, Yuan-Chen Guo, Cheng Lin, Yuan Liu, Zhiyang Dou, Lingjie Liu, Yuexin Ma, Song-Hai Zhang, Marc Habermann, Christian Theobalt, et al. Wonder3d: Single image to 3d using cross-domain diffusion. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9970–9980, 2024. 3arxiv.org/abs/2310.15008
- [43]Spandan Madan, Timothy Henry, Jamell Arthur Dozier, Helen Ho, Nishchal Bhandari, Tomotake Sasaki, Fredo Durand, Hanspeter Pfister, and Xavier Boix. On the capability of cnns to generalize to unseen category-viewpoint combinations. arXiv preprint arXiv:2007.08032, 2020. 3
- [44]Luke Melas-Kyriazi, Christian Rupprecht, Iro Laina, and Andrea Vedaldi. RealFusion: 360° reconstruction of any object from a single image. Proc. CVPR, 2023.
- [45]Thomas Mitchel, Michael J Taylor, and Vincent Sitzmann. Neural isometries: Taming transformations for equivariant ml. Advances in Neural Information Processing Systems, 37:7311–7338, 2024.
- [46]Norman Müller, Yawar Siddiqui, Lorenzo Porzi, Samuel Rota Bulò, Peter Kontschieder, and Matthias Nießner. Diffr: Rendering-guided 3d radiance field diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4328–4338, 2023.arxiv.org/abs/2212.01206
- [47]Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaeldin El-Nouby, Mahmoud Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po-Yao Huang, Shang-Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Hervé Jegou, Julien Mairal, Patrick Labatut, and Armand Joulin. Dinov2: Learning robust visual features without supervision. arXiv, 2023. doi: 10.48550/arxiv.2304.07193.
- [48]Dario Pavllo, David Joseph Tan, Marie-Julie Rakotosaona, and Federico Tombari. Shape, pose, and appearance from a single image via bootstrapped radiance field inversion. In CVPR, 2023.
- [49]Ben Poole, Ajay Jain, Jonathan T. Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. Proc. ICLR, 2023.
- [50]Konpat Preechakul, Nattanat Chatthee, Suttisak Wizadwongsa, and Supasorn Suwajanakorn. Diffusion autoencoders: Toward a meaningful and decodable representation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10619–10629, 2022.arxiv.org/abs/2111.15640
- [51]Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, Eric Mintun, Junting Pan, Kalyan Vasudev Alwala, Nicolas Carion, Chao-Yuan Wu, Ross Girshick, Piotr Dollár, and Christoph Feichtenhofer. SAM 2: Segment anything in images and videos. arXiv preprint arXiv:2408.00714, 2024.arxiv.org/abs/2408.00714
- [52]Jeremy Reizenstein, Roman Shapovalov, Philipp Henzler, Luca Sbordone, Patrick Labatut, and David Novotny. Common objects in 3d: Large-scale learning and evaluation of real-life 3d category reconstruction. In ICCV, 2021.
- [53]Wonseok Roh, Hwanhee Jung, Jong Wook Kim, Seungwon Lee, Innfarn Yoo, Andreas Lugmayr, Seunggeun Chi, Karthik Ramani, and Sangpil Kim. Catsplat: Context-aware transformer with spatial guidance for generalizable 3d gaussian splatting from a single-view image. arXiv preprint arXiv:2412.12906, 2024.
- [54]Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein. Scene representation networks: Continuous 3d-structure-aware neural scene representations. Proc. NeurIPS, 2019.
- [55]Alfredo Spagna, Zoe Heidenry, Michelle Miselevich, Chloe Lambert, Benjamin E. Eisenstadt, Laura Tremblay, Zixin Liu, Jianghao Liu, and Paolo Bartolomeo. Visual mental imagery: Evidence for a heterarchical neural architecture. Physics of Life Reviews, 48:113–131, 2024.
- [56]Fan-Yun Sun, Jonathan Tremblay, Valts Blukis, Kevin Lin, Danfei Xu, Boris Ivanovic, Peter Karkus, Stan Birchfield, Dieter Fox, Ruohan Zhang, Yunzhu Li, Jiajun Wu, Marco Pavone, and Nick Haber. Partial-view object view synthesis via filtering inversion. In International Conference on 3D Vision (3DV), 2024.
- [57]Stanislaw Szymanowicz, Christian Rupprecht, and Andrea Vedaldi. Viewset diffusion:(0-) image-conditioned 3d generative models from 2d data. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 8863–8873, 2023.
- [58]Stanislaw Szymanowicz, Christian Rupprecht, and Andrea Vedaldi. Splatter image: Ultra-fast single-view 3d reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10208–10217, 2024.
- [59]Stanislaw Szymanowicz, Jason Y Zhang, Pratul Srinivasan, Ruiqi Gao, Arthur Brussee, Aleksander Holynski, Ricardo Martin-Brualla, Jonathan T Barron, and Philipp Henzler. Bolt3d: Generating 3d scenes in seconds. arXiv preprint arXiv:2503.14445, 2025.arxiv.org/abs/2503.14445
- [60]Hanlin Tang, Martin Schirmpf, William Lotter, Charlotte Moerman, Ana Paredes, Josue Ortega Caro, Walter Hardesty, David Cox, and Gabriel Kreiman. Recurrent computations for visual pattern completion. Proceedings of the National Academy of Sciences, 115(35):8835–8840, 2018.DOI
- [61]Jiaxiang Tang, Zhaoxi Chen, Xiaokang Chen, Tengfei Wang, Gang Zeng, and Ziwei Liu. Lgm: Large multi-view gaussian model for high-resolution 3d content creation. In European Conference on Computer Vision, pp. 1–18. Springer, 2024.arxiv.org/abs/2402.05054
- [62]Joshua B Tenenbaum, Charles Kemp, Thomas L Griffiths, and Noah D Goodman. How to grow a mind: Statistics, structure, and abstraction. Science, 331(6022):1279–1285, 2011.DOI
- [63]Ayush Tewari, Michael Zollhofer, Hyeongwoo Kim, Pablo Garrido, Florian Bernard, Patrick Perez, and Christian Theobalt. Mofa: Model-based deep convolutional face autoencoder for unsupervised monocular reconstruction. In Proceedings of the IEEE international conference on computer vision workshops, pp. 1274–1283, 2017.
- [64]Ayush Tewari, Michael Zollhofer, Florian Bernard, Pablo Garrido, Hyeongwoo Kim, Patrick Perez, and Christian Theobalt. High-fidelity monocular face reconstruction based on an unsupervised model-based face autoencoder. IEEE transactions on pattern analysis and machine intelligence, 42(2):357–370, 2018.DOI
- [65]Ayush Tewari, Tianwei Yin, George Cazenavette, Semon Rezchikov, Josh Tenenbaum, Frédo Durand, Bill Freeman, and Vincent Sitzmann. Diffusion with forward models: Solving stochastic inverse problems without direct supervision. Advances in Neural Information Processing Systems, 36:12349–12362, 2023.arxiv.org/abs/2306.11719
- [66]Alexander Tschantz, Beren Millidge, Anil K Seth, and Christopher L Buckley. Hybrid predictive coding: Inferring, fast and slow. PLOS Computational Biology, 19(8):e1011280, 2023.arxiv.org/abs/2204.02169
- [67]Jianyuan Wang, Minghao Chen, Nikita Karaev, Andrea Vedaldi, Christian Rupprecht, and David Novotny. Vggt: Visual geometry grounded transformer. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 5294–5306, 2025a.arxiv.org/abs/2503.11651
- [68]Runqian Wang and Yilun Du. Equilibrium matching: Generative modeling with implicit energy-based models. arXiv preprint arXiv:2510.02300, 2025.arxiv.org/abs/2510.02300
- [69]Shuzhe Wang, Vincent Leroy, Yohann Cabon, Boris Chidlovskii, and Jerome Revaud. Dust3r: Geometric 3d vision made easy. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20697–20709, 2024.arxiv.org/abs/2312.14132
- [70]Yanbo Wang, Justin Dawes, and Yilun Du. Compositional scene understanding through inverse generative modeling. In Proceedings of the 2025 International Conference on Machine Learning (ICML), 2025b. URL https://arxiv.org/abs/2505.21780. arXiv preprint arXiv:2505.21780.
- [71]Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. Prolificdreamer: High-fidelity and diverse text-to-3d generation with variational score distillation. Advances in Neural Information Processing Systems, 36:8406–8441, 2023.arxiv.org/abs/2305.16213
- [72]Christopher Wewer, Kevin Raj, Eddy Ilg, Bernt Schiele, and Jan Eric Lenssen. latentsplat: Autoencoding variational gaussians for fast generalizable 3d reconstruction. arXiv preprint arXiv:2403.16292, 2024.arxiv.org/abs/2403.16292
- [73]Dejia Xu, Yifan Jiang, Peihao Wang, Zhiwen Fan, Yi Wang, and Zhangyang Wang. Neurallift-360: Lifting an in-the-wild 2d photo to a 3d object with 360deg views. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4479–4489, 2023a.arxiv.org/abs/2211.16431
- [74]Yinghao Xu, Hao Tan, Fujun Luan, Sai Bi, Peng Wang, Jiahao Li, Zifan Shi, Kalyan Sunkavalli, Gordon Wetzstein, Zexiang Xu, et al. Dmv3d: Denoising multi-view diffusion using 3d large reconstruction model. arXiv preprint arXiv:2311.09217, 2023b.arxiv.org/abs/2311.09217
- [75]Yinghao Xu, Zifan Shi, Wang Yifan, Hansheng Chen, Ceyuan Yang, Sida Peng, Yujun Shen, and Gordon Wetzstein. Grm: Large gaussian reconstruction model for efficient 3d reconstruction and generation. In European Conference on Computer Vision, pp. 1–20. Springer, 2024. 2, 3
- [76]Yinshuang Xu, Jiahui Lei, and Kostas Daniilidis. Equivariant light field convolution and transformer. arXiv preprint arXiv:2212.14871, 2022. 3arxiv.org/abs/2212.14871
- [77]Sizhe Yang, Wenye Yu, Jia Zeng, Jun Lv, Kerui Ren, Cewu Lu, Dahua Lin, and Jiangmiao Pang. Novel demonstration generation with gaussian splatting enables robust one-shot manipulation. arXiv preprint arXiv:2504.13175, 2025. 3
- [78]Yulong Yang, Felix O’Mahony, and Christine Allen-Blanchette. Learning color equivariant representations. arXiv preprint arXiv:2406.09588, 2024. 3arxiv.org/abs/2406.09588
- [79]Ilker Yildirim, Tejas D Kulkarni, Winrich A Freiwald, and Joshua B Tenenbaum. Efficient analysis-by-synthesis in vision: A computational framework, behavioral tests, and comparison with neural representations. In Proceedings of the 37th annual conference of the cognitive science society, pp. 2727–2732, 2015. 1
- [80]Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa. pixelnerf: Neural radiance fields from one or few images. In Proc. CVPR, 2021. 1, 2
- [81]Alan Yuille and Daniel Kersten. Vision as bayesian inference: analysis by synthesis? Trends in cognitive sciences, 10(7):301–308, 2006. 1, 2, 3pubmed.ncbi.nlm.nih.gov/16784882
- [82]Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proc. CVPR, 2018. 5, 7
- [83]Bingchen Zhao, Jiahao Wang, Wufei Ma, Artur Jesslen, Siwei Yang, Shaozuo Yu, Oliver Zendel, Christian Theobalt, Alan Yuille, and Adam Kortylewski. Ood-cv-2: An extended benchmark for robustness to out-of-distribution shifts of individual nuisances in natural images. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024. 3
- [84]Allan Zhou, Moo Jin Kim, Lirui Wang, Pete Florence, and Chelsea Finn. Nerf in the palm of your hand: Corrective augmentation for robotics via novel-view synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 17907–17917, 2023. 3
- [85]Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely. Stereo magnification: Learning view synthesis using multiplane images. ACM Trans. Graph. (Proc. SIGGRAPH), 37, 2018. URL https://arxiv.org/abs/1805.09817. 7
- [86]Eric Zhu, Mara Levy, Matthew Gwilliam, and Abhinav Shrivastava. Nerf-aug: Data augmentation for robotics with neural radiance fields. arXiv preprint arXiv:2411.02482, 2024. 3




