WorksComputer Science
OmniRay: Camera-Agnostic Monocular 3D Reconstruction with Composite Ray Fields
Abstract
Monocular models recover 3D geometry from a single image in one pass, but most assume a pinhole camera and produce distorted geometry on the wide-angle imagery produced by fisheye and panoramic lenses. Camera-aware models remove this assumption by predicting a per-pixel ray field. However, they represent different camera geometries using a single fixed basis to capture both the overall camera projection and its finer variations. We argue that camera geometry can be viewed as having a hierarchy of complexity for representing rays. We present OmniRay, a feed-forward model whose Ray Module is composed of three components: a monotone radial base that captures radially symmetric structure, a spherical-harmonic residual for smooth deviations from it, and a free-form residual for remaining local variation. These components are composed as successive rotations on the sphere, allowing the representation to accommodate diverse camera geometries. Given a single image, OmniRay predicts a ray direction and an up-to-scale radial distance for every pixel, from which a 3D point cloud is recovered without needing camera information at inference time. Extensive experiments across different camera configurations demonstrate that OmniRay establishes a new state-of-the-art in camera-agnostic monocular 3D reconstruction.
Project Page: https://nam1410.github.io/omniray/
Introduction
Monocular 3D reconstruction recovers a 3D point for each pixel in an image. We consider central cameras, in which all viewing rays pass through a single optical center, so each 3D point splits into a ray direction and a distance along that ray. The ray direction depends only on the camera geometry, while the distance depends on the scene under observation. Recent feed-forward models perform well on monocular 3D estimation [33, 31], but their architectures or training data focus on pinhole cameras. When applied to wide-angle images, these assumptions often lead to distorted reconstructed geometry. Wide-angle cameras are common in robotics [34], augmented and virtual reality [4], and 360° capture [15], making this a practical limitation. One way to handle these images is to calibrate the camera and rectify them to a pinhole view before reconstruction, but a pinhole cannot represent rays at or beyond 90° from its optical axis. As such, undistortion typically leads to cropping away part of the field of view or strongly stretching the image near its periphery, leading to low-confidence predictions from conventional depth estimation or 3D foundation models [7].
Camera-aware models instead account for camera geometry directly [3, 26, 21]. DeepFocal [32] and DeepCalib [3] first use convolutional neural networks (CNNs) to estimate focal length and distortion of fixed camera models. Follow-up works generalize camera parameter estimation to arbitrary camera models by estimating ray fields directly. In particular, UniK3D [21] predicts a per-pixel ray field using a spherical-harmonic (SH) basis and a radial distance along each ray. It can therefore reconstruct pinhole, fisheye, and panoramic images without camera metadata. In this formulation, the same basis must describe both the overall camera projection and its finer variations. We argue that there exists a hierarchy of complexity in camera geometry estimation, which can lead to more efficient and performant design. Many camera projections have a simple structure: for approximately radially symmetric lenses, a smooth radial function is sufficient to describe how viewing directions change across the image. Spherical harmonics can be used to complement radial functions.
We therefore propose OmniRay – that takes a single image and predicts a dense, up-to-scale 3D reconstruction without intrinsics or camera-type input (Figure 1). A shared image encoder feeds two modules. The Ray Module predicts a monotone radial base and the two residuals, each of which refines the preceding ray field. The Distance Module predicts the radial distance of each pixel, and the corresponding 3D point is obtained by scaling its predicted ray. We train OmniRay on cameras ranging from pinhole to panorama and evaluate it on held-out scenes, datasets, and camera configurations. An oracle analysis examines which ray fields each representation can express, while benchmark experiments and ablations test the performance of the proposed components.

Figure 1. OmniRay reconstructs 3D geometry from a single image across diverse camera geometries. A single model predicts an up-to-scale, camera-frame point cloud without camera intrinsics or camera-type input. Columns show inputs and reconstructions for pinhole, GoPro, fisheye, catadioptric, and panoramic cameras, respectively. Note that the GoPro and catadioptric camera configurations are withheld from training.
In summary, our contributions are:
A composite ray representation built from a monotone radial base, a spherical-harmonic residual, and a free-form residual.
OmniRay, a feed-forward model for single-image 3D reconstruction across central camera geometries, without needing intrinsics or camera-type input.
An oracle analysis of ray representations, and experiments across seen and held-out scenes, datasets, and camera configurations show that OmniRay improves over the evaluated baselines in ray field estimation and monocular 3D reconstruction.
Related Work
Camera calibration. Camera calibration determines the mapping from image pixels to rays in the camera frame. Classical methods parameterize this mapping with a low-dimensional model whose parameters are estimated from geometric observations [35, 24]. Different camera geometries are handled by different parametric models: pinhole model with radial and tangential distortion for conventional lenses, Kannala & Brand (2006) model for wide-angle optics, and unified camera models [6, 18, 14, 27] for omnidirectional cameras. Each of these models fixes a particular functional form, so a lens whose behavior departs from that form introduces residual error. Generic models relax these assumptions with more degrees of freedom and capture complex lens behavior [25] at the cost of a higher-dimensional calibration problem.
Learning-based methods offer an alternative to this high-dimensional calibration process by learning camera geometry directly from images without input calibration. DeepCalib [3] regresses camera parameters, while Perspective Fields [11], GeoCalib [29], and AnyCalib [26] move toward dense geometric cues: a per-pixel up-vector and latitude field [11, 29], or per-pixel rays [26], each of which describes the camera more generally than a parameter vector. Neural Ray Surfaces [28] take it further by learning a free-form ray surface from video observations and directly representing the pixel-to-ray mapping. These calibration approaches shift from explicit parameter estimation toward learning the pixel-to-ray mapping, but leave open how this mapping should be represented to remain both compact and expressive simultaneously.
Monocular 3D. Recent foundation models have improved depth estimation through large-scale training and distillation [33], but do not explicitly model the camera geometry. Depth Anything 3 [16] instead predicts depth with per-pixel rays by modeling camera geometry alongside scene structure. Camera-aware models specifically account for camera geometry in monocular depth estimation. UniK3D [21] represents a per-pixel ray field with spherical-harmonics and predicts radial distance along each ray to handle diverse camera geometries without camera metadata. Depth Any Camera [8] and UniDAC [5] instead transform images into a common equirectangular representation to process different camera types before depth estimation, but require input camera intrinsics for the transformation. CAM3R (Gurupur̄asad et al., 2026) and Wid3R [12] extend the spherical-harmonic ray representation to multi-view feed-forward 3D reconstruction. Together, these works highlight the value of explicitly accounting for the pixel-to-ray mapping for camera-agnostic 3D geometry, while leaving open whether a single ray representation is necessary for all camera geometries.
Method
Given a single image from an arbitrary central camera, with no intrinsics, distortion parameters or camera-type label, OmniRay predicts for every pixel a unit ray and a radial distance .
A DINOv2 backbone (Oquab et al. [20]) encodes the image, and its features drive two branches (Figure 2): the Ray Module composes from radial, spherical-harmonic, and free-form components, while the Distance Module predicts . Their combination gives the 3D point in the camera frame:

Figure 2. OmniRay Overview. Given a single image, a DINOv2 ViT-L/14 encoder produces global and dense features that drive two branches. The Ray Module progressively builds a camera-agnostic ray field from a monotone radial base, a spherical-harmonic residual, and a free-form residual. The Distance Module predicts a positive radial distance for each pixel. Combining the final ray and distance yields the local pointmap , recovering 3D geometry up to scale without camera intrinsics or a camera-type label.
We predict radial distance rather than perpendicular depth , related by , since as the ray approaches from the optical axis and changes sign beyond it; therefore remains well-defined across wide fields of view.
Notation. The DINOv2 backbone produces a dense feature map and a pooled global token . We use the normalized image coordinate , where is the image center and is the image half-diagonal in pixels. This normalization places the image corners at for any resolution and aspect ratio. Image scale is provided separately to through and the logarithm of the image diagonal. Ground-truth quantities are marked with a star, and , and denotes the set of pixels at which they are defined.
Ray Module
The module comprises three stages, , in which the radial base and the spherical-harmonic residual are predicted from the global token , and the free-form residual from the dense feature map .
Radial base models the approximately radially symmetric structure of conventional central cameras with a monotone mapping from image radius to incidence angle.
Monotonicity restricts the base to injective radius-to-angle maps, so no two radii in the modeled range receive the same incidence angle, and the profile is evaluated directly rather than inverted numerically as a parametric projection model would require. A head maps to values (Figure 2b): increments between fixed knot radii , with a principal-point offset , an axis-angle rotation , and three parameters forming an upper-triangular map for anisotropic scaling and shear. These give the transformed coordinate , with radius and azimuth . The increments are passed through a softplus to ensure positivity, accumulated into knot heights, and linearly interpolated,
so that is strictly increasing within the knot span and constant outside it. We use knots spanning in half-diagonal units. The corresponding ray is
where is the rotation matrix of . The largest angle the base can produce is the sum of its increments, so nothing caps at , and the base already represents lenses whose rays pass the equator.
Spherical-harmonic residual. The radial base restricts the incidence angle to a monotone function of image radius, which becomes insufficient when the image covers a large portion of the viewing sphere (Figure 3b). A spherical-harmonic residual supplies an angular correction to it (Figure 2b). A head maps to a single coefficient matrix for a real spherical-harmonic basis of maximum degree , shared across all pixels. Thus, the correction field is represented by predicted coefficients rather than an independent vector at each pixel. Each pixel evaluates the basis at an address on obtained from its normalized image coordinate through the inverse-stereographic lift,

Figure 3. Why combine the radial construction with the residual corrections? (a) The radial representation gives the same incidence angle to pixels that lie at the same radius in the base coordinates. (b) In the centered 2:1 equirectangular projection, point A is closer to the center of the image than point B, yet has a higher incidence angle. A monotone radial function cannot accommodate such ordering under fixed coordinate transformation and orientation. (c) The spherical-harmonic and free-form residuals improve upon the base direction via iterative correction on the unit sphere.
where stacks the basis values. We use , giving 192 coefficients per image.
The correction is applied through the exponential map on . For any ray , only the component of orthogonal to is a tangent direction, so the remaining component is discarded, leaving . Its norm is the rotation angle in radians, and the ray is transported that far along the great circle through in the direction of ,
Free-form residual. The spherical-harmonic residual provides a compact global correction but remains limited by the finite-order basis. The free-form residual captures the remaining local variation using a lightweight convolutional decoder on the dense feature map . The resulting correction map is interpolated at each pixel to obtain , so its spatial detail is limited by the resolution of . The two residuals are applied in turn through equation 5, giving the composite ray field
with given by equation 3, and with and the tangential parts of at and of at . Both residual heads are zero-initialized, so the composition starts from the radial base, and the norms and are the rotation angles each residual applies, measuring how far a camera departs from the representation beneath it and letting the representation adapt without model selection or routing.
Distance Module
The Ray Module determines the direction along which each pixel observes the scene, but not how far along that ray the surface lies. The Distance Module predicts this remaining quantity: the up-to-scale radial distance from the optical center. It reuses the same encoder pass, fusing four intermediate blocks with a Dense Prediction Transformer (DPT) decoder Ranftl et al. [22] into a dense prediction sampled at each pixel, where a softplus gives a positive distance . Scaling each predicted ray by its distance as in equation 1 produces the camera-frame point cloud in a single forward pass.
Training objective
Angular ray loss supervises the final ray field and the two intermediate fields with the mean cosine residual ,
with stage weights . This deep supervision of and keeps each stage responsible for the structure it represents, so a correct final ray cannot be reached through an arbitrary base.
weights the same per-pixel error more heavily where it is largest.
With denoting the mean error over a pixel set , it is over an inner region , an outer region , and tail of the 5% of pixels with the largest error, with the largest of the three since error concentrates at the periphery for wide fields of view. The regions are thresholds on the image radius relative to its maximum, so the split follows the imaged region of a circular fisheye as well as a rectangular frame and does not move with the predicted principal point.
point cloud loss supervises rays and distances jointly. Each cloud is normalized by its own mean point norm, for the prediction and for the ground truth, which removes the global scale that a single image leaves undetermined,
It is evaluated on with the ray attached, so a ray error displaces the point even for an exact distance and the gradient reaches both modules.
Overall objective. . Both modules and the shared encoder are trained end-to-end under this objective. Sec. A.1 gives the head architectures and the full training schedule.
Experiments
Oracle test: representation ceilings
Before training anything, we measure what each candidate representation can express at all, by fitting its parameters directly to analytic ground-truth rays; Sec. A.2 gives the fitting protocol. Table 1 shows that compact parametric models succeed only on the geometries their assumptions match, and that increasing the number of knots in the monotone radial profile reduces the catadioptric error but leaves the equirectangular error at 22.8°, the ceiling identified in § 3.1. The spherical-harmonic field removes that ceiling but remains two orders of magnitude behind the radial profile on the fisheye. Among the compact representations, only the composition is below 0.25° on all three geometries, and since it reduces to the radial base when the residual vanishes, its capacity is bounded below by that of the base.
| Representation | #Params | Fisheye | Equirectangular | Catadioptric |
| Pinhole | 6 | 5.731 | 48.080 | 50.434 |
| Double sphere | 8 | 0.021 | 22.932 | 41.876 |
| Radial () | 24 | 0.002 | 22.802 | 0.104 |
| Radial () | 40 | 0.002 | 22.801 | 0.023 |
| SH () | 195 | 0.276 | 0.641 | 0.295 |
| Radial₁₆ + SH₇ residual (ours) | 216 | 0.075 | 0.217 | 0.174 |
| Free field | 76.8k | 0.005 | 0.013 | 0.015 |
Table 1. Oracle representation fitting. Mean angular error in degrees after fitting each representation directly to ground-truth rays; lower is better. #Params denotes parameters per camera; the SH row adds three rotation parameters to the coefficients for the base direction, which the residual inherits from the radial base. Best results among compact representations are bold. The free field is shown separately as a high-capacity reference.
Datasets and Camera Types
We train on two panoramic datasets, 2D3DS ([1]), of indoor offices and corridors, and 360Loc ([10]), of larger indoor and outdoor environments, rendering perspective, fisheye, and panoramic views from each panorama; the 2D3DS perspective images are the real captures provided with the dataset. We evaluate on held-out scenes of both, and on two unseen datasets, PanoCity (Guo et al. [7]), of outdoor street-level captures, and CO3Dv2 (Reizenstein et al. [23]), of object-centric images. Two camera configurations are withheld from training entirely, a GoPro fisheye and a Nayar catadioptric camera (Nayar (1997); Baker & Nayar [2]), both satisfying our central-camera assumption. Sec. A.2 gives the splits and the camera specifications.
Random-lens Augmentation. During training we augment each panorama with randomly sampled synthetic central cameras, each defined by a viewing direction, a monotone radial profile drawn from positive spline increments, and an imaged sensor region. This single family covers filled and annular domains, from pinhole to wide-angle lenses. Draws whose profile falls close to a held-out geometry are rejected, so neither withheld camera nor its neighborhood appears in training. Sec. A.2 gives the sampling ranges and the rejection criteria, and Figure 8 shows sampled examples.

Figure 8. Random-lens augmentation. Images rendered from a single panorama under cameras drawn from the sampling distribution of § 4.2. A camera is a viewing direction, a monotone radial profile and an imaged sensor region, so one family spans narrow perspective crops, wide-angle and fisheye views, off-center and anisotropic sensors, and annular domains. The projection varies continuously from sample to sample, so no fixed map from image position to ray direction fits more than one of them. Draws falling close to either held-out geometry are rejected and never enter training.
Benchmark and Ablation
Baselines. UniK3D (Piccinelli et al. [21]) and Depth Anything 3 (DA3) (Lin et al. [16]) jointly predict camera geometry and scene structure and are evaluated on all metrics; we fine-tune UniK3D on the 2D3DS and 360Loc panoramas so that it sees the same source imagery we train on. GeoCalib (Veicht et al. [29]) and AnyCalib (Tirado-Garín & Civera [26]) calibrate the camera without predicting scene geometry. We convert their estimated cameras to dense ray fields for evaluation so all methods are scored identically.
Metrics. We evaluate the predicted ray fields, radial distance, and 3D point clouds using the following metrics: Ray accuracy is the mean angular error in degrees, and radial distance is evaluated with absolute relative error (AbsRel) and . For point clouds, accuracy and completeness are the median nearest-neighbor distances normalized by the scene’s mean point norm, and their mean is the Chamfer distance (CD). Predictions in different depth gauges are aligned per image by , and perpendicular depth is converted to radial distance by using the predicted rays. All metrics are computed over pixels with valid ground truth.
Results. Table 2 summarizes performance across camera configurations and domains. On the seen camera types, OmniRay achieves a mean ray error of and relative Chamfer distance of , compared with and for UniK3D and and for Depth Anything 3. On the held-out configurations, for the Nayar catadioptric camera, all baselines exceed while ours achieves and CD; on GoPro, ours achieves versus for UniK3D. On PanoCity, ours achieves ray error versus and for UniK3D and DA3, respectively. On CO3Dv2, our ray error remains within a degree of both baselines.
| Method | Seen cameras: Aggregate Ang. () | Seen cameras: Aggregate CD | Held-out cameras: Catadioptric Ang. () | Held-out cameras: Catadioptric CD | Held-out cameras: GoPro Ang. () | Held-out cameras: GoPro CD | Held-out domains: PanoCity Ang. () | Held-out domains: PanoCity CD | Held-out domains: CO3Dv2 Ang. () | Held-out domains: CO3Dv2 CD |
| GeoCalib [29] | 30.997 | – | 119.662 | – | 14.126 | – | 47.132 | – | 3.713 | – |
| AnyCalib [26] | 28.655 | – | 107.129 | – | 10.031 | – | 42.335 | – | 2.711 | – |
| DA3 [16] | 33.028 | 0.1999 | 121.566 | 1.0412 | 16.637 | 0.1097 | 50.533 | 0.3677 | 2.041 | 0.0288 |
| UniK3D [21] | 6.336 | 0.0760 | 75.702 | 0.5948 | 6.342 | 0.0454 | 21.220 | 0.2120 | 2.676 | 0.0277 |
| OmniRay w/o AUG | 1.385 | 0.0592 | 30.867 | 0.3440 | 7.965 | 0.0641 | 6.352 | 0.0984 | 2.825 | 0.0330 |
| OmniRay + RG | 0.356 | 0.0205 | 1.118 | 0.0113 | 5.180 | 0.0375 | 1.592 | 0.0277 | 2.935 | 0.0411 |
| OmniRay | 0.140 | 0.0179 | 1.120 | 0.0131 | 5.533 | 0.0392 | 1.350 | 0.0240 | 2.967 | 0.0380 |
Table 2. Main benchmark. Results on seen cameras, held-out cameras, and held-out domains. Ang. denotes mean ray error in degrees; CD denotes median relative Chamfer distance, normalized by each scene’s mean point norm. Metrics are macro-averaged over test images, and all methods use per-image median scale alignment. Lower is better; best results are bold. AUG denotes random-lens augmentation, and RG denotes ray guidance. denotes exclusion from our training data; a dash indicates that the method does not predict scene geometry. Full per-camera and per-dataset results and additional metrics are provided in Sec. A.3.
The w/o AUG row isolates the random-lens sampling of § 4.2. Without it the seen cameras remain strong at , but the held-out catadioptric and GoPro errors rise to and ; restoring it recovers and and improves the seen cameras to . Figure 5a shows the same effect on a catadioptric example, where the unaugmented model exhibits larger ray-field and geometric errors. These results indicate that random-lens sampling is important for extending the composite representation beyond the camera configurations encountered during training. Adding ray guidance to each DPT scale following Lin et al. [17] (+ RG) does not change the metrics consistently. The injection is zero-initialized, so the guided head could exploit the prompt if it were beneficial, and we therefore omit it from the final model.

Figure 5. Ablation. (a) Held-out catadioptric camera: predicted ray field and point cloud for UniK3D, OmniRay trained without random-lens augmentation, and the full model, against the ground truth. (b) Panoramic image: the radial base alone covers only part of the sphere (orange) and leaves the reconstruction torn, while the spherical-harmonic and free-form residuals complete the field (blue) and close the geometry.
Figure 4 compares ray fields and camera-frame point clouds on unseen PanoCity and CO3Dv2 images, with each ray field colored by its angular error. On the panorama, OmniRay keeps the road continuous while UniK3D breaks it across the image and DA3 fails to recover the ground structure. On the fisheye and GoPro rows, OmniRay keeps the road, pavement, and lane markings straight where UniK3D bends them and DA3 also distorts the surrounding buildings. On the held-out catadioptric camera both baselines produce degraded reconstructions. On the CO3Dv2 pinhole example, OmniRay remains comparable to the baselines, demonstrating that the additional flexibility for wide-angle cameras does not come at the expense of pinhole imagery. On in-the-wild images, OmniRay also recovers coherent 3D structure, as shown in Figure 6.

Figure 4. Qualitative reconstructions on unseen datasets. We compare OmniRay with Depth Anything 3 and UniK3D on PanoCity and CO3Dv2 across five camera configurations. Small visualizations show viewing-ray fields; larger renderings show the corresponding point clouds, with ground truth in the rightmost column. OmniRay better preserves scene structure in the panoramic and catadioptric examples without camera intrinsics or camera-type input.

Figure 6. In-the-wild reconstructions. Qualitative reconstructions on photographs collected from the internet, shown in order as a panorama, a fisheye image, and a pinhole image. Although these camera families are represented during training, the images come from different camera instances with unknown intrinsics and lens parameters. OmniRay requires no camera metadata at inference.
Ablation. Table 3 isolates and analyzes different ray representations. The radial base performs well on fisheye and pinhole cameras ( and ) but gives on panoramas, where a monotone radial profile assigns one incidence angle per image radius, an ordering that a panoramic projection does not follow (Figure 3b). The spherical base reduces panoramic error to , while the free-form field performs best among single representations on panoramas, fisheye, and pinhole, but reaches 11.335° on the held-out catadioptric configuration. These results motivate combining a structured radial base with flexible residuals.
| Ray representation | Training | Panorama | Fisheye | Pinhole | Catadioptric | GoPro |
| Radial base | – | 21.825 | 1.761 | 0.788 | 6.039 | 5.377 |
| Spherical base | – | 5.088 | 1.540 | 0.790 | 13.327 | 5.962 |
| Free-form field | – | 0.783 | 1.508 | 0.621 | 11.335 | 5.191 |
| Radial + SH residual | Joint | 0.796 | 1.784 | 0.581 | 1.879 | 6.566 |
| Radial + SH residual | Staged | 0.610 | 1.545 | 0.675 | 1.680 | 3.393 |
| Radial + free-form | Joint | 1.018 | 1.394 | 0.403 | 3.038 | 3.667 |
| Radial + free-form | Staged | 1.111 | 1.605 | 0.386 | 1.797 | 4.248 |
| SH + free-form | Joint | 1.114 | 1.747 | 1.250 | 2.568 | 3.739 |
| SH + free-form | Staged | 1.026 | 2.567 | 0.553 | 2.316 | 5.142 |
| Radial + SH + free-form | Joint | 0.728 | 1.963 | 0.688 | 2.525 | 3.636 |
| Radial + SH + free-form | Joint, | 0.451 | 1.864 | 0.705 | 1.838 | 3.604 |
| Radial + SH + free-form | Staged, VGGT init | 0.486 | 1.630 | 0.330 | 2.576 | 5.096 |
| Radial + SH + free-form | Staged | 0.399 | 1.322 | 0.418 | 1.538 | 3.462 |
Table 3. Ray representation ablation. Mean angular ray error in degrees (↓) at native resolution. Seen camera families are evaluated on held-out configurations; † denotes configurations excluded from training. Only the Ray Module is trained and evaluated. Best results are bold.
Adding an SH residual to the radial base improves panoramic and held-out accuracy; staged training achieves 0.610° and 1.680°, respectively, and the best GoPro result (3.393°). Adding the free-form residual further improves the panorama (0.399°), fisheye (1.322°), and catadioptric results (1.538°) in the full staged model, which also outperforms both jointly trained three-term variants across all five configurations.
On pinhole cameras, the full model remains competitive at 0.418°. The additional spherical residual offers little benefit for this narrow-field geometry, while potentially providing greater flexibility across wide-field cameras. The VGGT-initialized variant (Wang et al. [30]) illustrates a similar trade-off: it achieves the best pinhole error (0.330°) but performs worse than the full model on the other four configurations. Since VGGT is pretrained for multi-view geometry on perspective imagery, its initialization may favor narrow-field projections over the broader range of camera geometries considered here. As the backbone is fine-tuned end-to-end in both settings, this comparison suggests an effect of initialization rather than a limitation in model capacity.
Overall, these results support the staged radial → SH → free-form composition as a balance between geometric structure and representational flexibility.
Conclusion
We began with two questions: how the pixel-to-ray mapping can remain both compact and expressive, and whether a single representation is necessary for every camera geometry. OmniRay explores both by composing the ray field. A monotone radial base captures the structure of conventional lenses, a spherical-harmonic residual models smooth deviations that a radial profile cannot express, and a free-form residual accounts for remaining local variation. Each component is applied as a rotation on , allowing the representation to adapt to different camera geometries without camera-type input or model selection. A shared distance module scales the predicted rays into a point cloud in a single forward pass, and sampling lenses continuously during training helps the composition generalize to held-out camera configurations.
Limitation. A qualitative zero-shot example reveals a limitation on panoramic views that differ from the training distribution. The training images are captured near ground level, with horizons approximately aligned across images. As shown in Figure 7, on an in-the-wild aerial panorama with the horizon near the top of the frame, the predicted rays cover only a bounded, forward-facing region rather than the full viewing sphere. This example suggests that generalization depends not only on camera-projection diversity but also on the distribution of viewing altitudes represented during training.

Figure 7. Limitation: zero-shot in-the-wild reconstruction of an aerial panorama sourced from the internet. With the horizon near the top of the image, the predicted rays cover a bounded, forward-facing region.
Future work. A natural next step is to extend OmniRay to multi-view reconstruction and broaden the training distribution to include views with more varied camera orientations and altitudes.
Acknowledgments
This research is based upon work supported by the Office of the Director of National Intelligence (ODNI), Intelligence Advanced Research Projects Activity (IARPA), via IARPA R&D Contract No. 140D0423C0076. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of the ODNI, IARPA, or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright annotation thereon.
AI Use Statement
In this work, we used generative AI tools to help with the implementation code and to polish grammar in the paper to improve readability. We have reviewed all AI-assisted work. We reviewed LLM-generated text to ensure it was consistent with our findings. We take responsibility for the final content of this work, including text, claims, or artifacts produced with the aid of generative AI.
Appendix
Implementation and training
Architecture. The encoder is DINOv2 ViT-L/14 [20] with 1024-dimensional features. The radial head is a two-layer multilayer perceptron (MLP) with width 256 operating on the global token augmented with (log aspect, log diagonal), while the spherical head is a one-layer MLP of the same width on the same input. The radial head predicts values with knots spanning in half-diagonal units; the spherical head predicts coefficients with . These coefficients are shared across all pixels of an image. The free head is a convolutional decoder over the dense feature map. The Distance Module uses a width-256 DPT decoder [22] over encoder blocks to predict the radial distance. Both residual heads are zero-initialized, while the radial head is initialized to a canonical camera, so training starts from a single radial lens.
Training. Images range from equirectangular panoramas to lens crops. We train with a token cap of 8192 using a sampler for camera-type balancing. Within each image, we supervise 4096 pixels drawn round-robin over the occupied cells of a grid, ensuring coverage across the image. Training proceeds in two phases. We first train the Ray Module alone, activating its three components in order, and then attach the Distance Module at the final stage and train the full model end-to-end. In this second phase we use AdamW with weight decay 0.01 and cosine decay after a warm-up, with learning rates of for the encoder and ray heads and for the distance decoder. The encoder runs in bfloat16, while the heads and losses remain in full precision. Training uses 8 NVIDIA A40 GPUs with distributed data parallelism.
Experimental Details
Oracle fitting protocol. Each representation is fitted to the analytic ground-truth rays of a single camera by minimizing the mean cosine residual in § 3.3, with a multi-start sweep over the axis sign, roll and, where the representation uses one, the focal length. Every row is an independent fit under a fixed optimization budget, so each entry upper-bounds that representation’s ceiling. The fitted error is the representation’s approximation capacity for that camera and lower-bounds the error of any predictor built on it.
Splits and Camera sampling. For 2D3DS we train on areas 1-4 and evaluate on areas 5-6; for 360Loc we train on the Atrium and Piatrium scenes and evaluate on Concourse and Hall. Each synthetic camera samples a viewing orientation, an aspect ratio, a principal-point offset, an optional central blind spot, a rectangular or circular sensor boundary, and inner and outer incidence angles; panoramic cameras instead sample a vertical field of view, an azimuth and a small latitude stretch, with the horizon kept gravity-aligned as in the source captures. Draws near a held-out geometry are rejected: for the GoPro, any profile within of its incidence-angle curve at every radius; for the catadioptric camera, any sample with an outer angle in and a rim angle in . Note: The Nayar catadioptric camera (Nayar, 1997; Baker & Nayar, 1999) is a paraboloid mirror with orthographic projection, imaging a – annulus about its axis; the mirror and camera share a single viewpoint. For PanoCity and CO3Dv2 we use the calibration provided with each dataset to obtain ground-truth rays. Ground-truth rays and distances follow analytically from each sampled camera. Figure 8 shows images rendered under cameras drawn from this distribution.
Additional Results
Table 4, Table 5 and Table 6 report every metric of § 4.3 broken out per camera type, per dataset, and over the aggregate seen and held-out splits. Figure 9 and Figure 10 show reconstructions on the 2D3DS and 360Loc test scenes, which share the camera families used in training but none of their scenes, complementing the unseen-dataset comparison of Figure 4. Figure 11 provides additional visualizations of OmniRay on in-the-wild examples.

Figure 9. Qualitative reconstructions on seen dataset: 2D3DS. Ray field: predicted rays on the unit sphere, coloured by angular error to the ground-truth ray (turbo, θ=0°–30°); the ground-truth ray field is coloured by image RGB. Crops: enlarged view of the dashed region, same processing for every column.

Figure 10. Qualitative reconstructions on _seen_ dataset: 360Loc

Figure 11. Additional in-the-wild reconstructions.
| Camera | Method | Ray ang | Radial Dist. AbsRel | CD | Acc | Comp | F@10cm | |
| pinhole | GeoCalib | 2.196 | – | – | – | – | – | – |
| AnyCalib | 1.043 | – | – | – | – | – | – | |
| DA3 | 6.604 | 0.1508 | 0.862 | 0.0701 | 0.0862 | 0.0540 | 0.300 | |
| UniK3D | 3.619 | 0.1371 | 0.871 | 0.1058 | 0.1802 | 0.0315 | 0.356 | |
| ours | 0.155 | 0.0769 | 0.925 | 0.0242 | 0.0355 | 0.0130 | 0.533 | |
| fisheye | GeoCalib | 30.001 | – | – | – | – | – | – |
| AnyCalib | 22.134 | – | – | – | – | – | – | |
| DA3 | 28.727 | 0.3225 | 0.562 | 0.1888 | 0.1111 | 0.2665 | 0.066 | |
| UniK3D | 7.055 | 0.2155 | 0.768 | 0.0543 | 0.0569 | 0.0517 | 0.198 | |
| ours | 0.154 | 0.0906 | 0.924 | 0.0159 | 0.0197 | 0.0121 | 0.531 | |
| panoramic | GeoCalib | 60.795 | – | – | – | – | – | – |
| AnyCalib | 62.788 | – | – | – | – | – | – | |
| DA3 | 63.752 | 0.3352 | 0.547 | 0.3408 | 0.1956 | 0.4859 | 0.009 | |
| UniK3D | 8.336 | 0.2154 | 0.774 | 0.0678 | 0.0723 | 0.0633 | 0.174 | |
| ours | 0.110 | 0.0593 | 0.949 | 0.0137 | 0.0163 | 0.0111 | 0.585 | |
| catadioptric | GeoCalib | 119.662 | – | – | – | – | – | – |
| AnyCalib | 107.129 | – | – | – | – | – | – | |
| DA3 | 121.566 | 0.1617 | 0.771 | 1.0412 | 0.7232 | 1.3592 | 0.000 | |
| UniK3D | 75.702 | 0.2523 | 0.656 | 0.5948 | 0.5047 | 0.6849 | 0.004 | |
| ours | 1.120 | 0.0396 | 0.969 | 0.0131 | 0.0130 | 0.0131 | 0.633 | |
| GoPro | GeoCalib | 14.126 | – | – | – | – | – | – |
| AnyCalib | 10.031 | – | – | – | – | – | – | |
| DA3 | 16.637 | 0.1657 | 0.840 | 0.1097 | 0.0811 | 0.1383 | 0.079 | |
| UniK3D | 6.342 | 0.1665 | 0.857 | 0.0454 | 0.0449 | 0.0458 | 0.235 | |
| ours | 5.533 | 0.0842 | 0.932 | 0.0392 | 0.0354 | 0.0429 | 0.215 |
Table 4. Per-camera results, all metrics. Macro-averaged over test images. geometry withheld from training. Baselines are median-scale-aligned per image. Acc, Comp and CD are medians of the relative distance, divided by each scene’s mean point norm. is . F@10cm is the point-cloud F-score at a 10 cm distance threshold, the harmonic mean of precision and recall over nearest-neighbor distances. A dash means the method predicts no scene geometry.
| Scenes | Method | Ray ang°↓ | Radial Dist. AbsRel↓ | ↑ | CD↓ | point cloud Quality Acc↓ | Comp↓ | F@10cm↑ |
| 2D3DS | GeoCalib | 40.713 | – | – | – | – | – | – |
| 2D3DS | AnyCalib | 37.520 | – | – | – | – | – | – |
| 2D3DS | DA3 | 43.775 | 0.2124 | 0.747 | 0.3614 | 0.1744 | 0.5483 | 0.224 |
| 2D3DS | UniK3D | 19.432 | 0.1272 | 0.864 | 0.1487 | 0.0981 | 0.1994 | 0.411 |
| 2D3DS | ours | 1.619 | 0.0568 | 0.958 | 0.0236 | 0.0226 | 0.0246 | 0.767 |
| 360Loc | GeoCalib | 48.223 | – | – | – | – | – | – |
| 360Loc | AnyCalib | 42.021 | – | – | – | – | – | – |
| 360Loc | DA3 | 48.064 | 0.2872 | 0.629 | 0.3212 | 0.2121 | 0.4303 | 0.048 |
| 360Loc | UniK3D | 19.979 | 0.2709 | 0.676 | 0.1601 | 0.1455 | 0.1747 | 0.154 |
| 360Loc | ours | 1.274 | 0.0713 | 0.947 | 0.0160 | 0.0160 | 0.0160 | 0.590 |
| PanoCity† | GeoCalib | 47.132 | – | – | – | – | – | – |
| PanoCity† | AnyCalib | 42.335 | – | – | – | – | – | – |
| PanoCity† | DA3 | 50.533 | 0.1819 | 0.774 | 0.3677 | 0.3318 | 0.4037 | 0.001 |
| PanoCity† | UniK3D | 21.220 | 0.1940 | 0.815 | 0.2120 | 0.2719 | 0.1522 | 0.015 |
| PanoCity† | ours | 1.350 | 0.0823 | 0.914 | 0.0240 | 0.0333 | 0.0147 | 0.141 |
| CO3Dv2† | GeoCalib | 3.713 | – | – | – | – | – | – |
| CO3Dv2† | AnyCalib | 2.711 | – | – | – | – | – | – |
| CO3Dv2† | DA3 | 2.041 | 0.0573 | 0.950 | 0.0288 | 0.0199 | 0.0377 | 0.282 |
| CO3Dv2† | UniK3D | 2.676 | 0.0478 | 0.963 | 0.0277 | 0.0233 | 0.0322 | 0.293 |
| CO3Dv2† | ours | 2.967 | 0.0889 | 0.914 | 0.0380 | 0.0291 | 0.0469 | 0.173 |
Table 5. Per-dataset results, all metrics. Macro-averaged over test images. † unseen in training. Conventions as in Table 4. CO3Dv2 contains perspective and fisheye imagery only, so its zero-shot axis is the domain rather than the camera, and its object-masked COLMAP ground truth covers roughly 12% of pixels.
Table 6: Aggregate results over the 4500-sample benchmark. Conventions as in Table 4. Seen cameras are perspective, fisheye and panoramic; the held-out geometries are the catadioptric mirror and the GoPro fisheye.
| Split | Method | Ray ang | Radial Dist. AbsRel | CD | point cloud Quality Acc | Comp | F@10cm | |
| all | GeoCalib | 45.356 | – | – | – | – | – | – |
| AnyCalib | 40.625 | – | – | – | – | – | – | |
| DA3 | 47.457 | 0.2272 | 0.716 | 0.3501 | 0.2394 | 0.4608 | 0.091 | |
| UniK3D | 20.211 | 0.1974 | 0.785 | 0.1736 | 0.1718 | 0.1754 | 0.193 | |
| ours | 1.414 | 0.0701 | 0.940 | 0.0212 | 0.0240 | 0.0184 | 0.499 | |
| seen | GeoCalib | 30.997 | – | – | – | – | – | – |
| AnyCalib | 28.655 | – | – | – | – | – | – | |
| DA3 | 33.028 | 0.2695 | 0.657 | 0.1999 | 0.1310 | 0.2688 | 0.125 | |
| UniK3D | 6.336 | 0.1893 | 0.804 | 0.0760 | 0.1031 | 0.0488 | 0.242 | |
| ours | 0.140 | 0.0756 | 0.933 | 0.0179 | 0.0238 | 0.0120 | 0.550 | |
| held-out | GeoCalib | 66.894 | – | – | – | – | – | – |
| AnyCalib | 58.580 | – | – | – | – | – | – | |
| DA3 | 69.102 | 0.1637 | 0.806 | 0.5754 | 0.4021 | 0.7488 | 0.040 | |
| UniK3D | 41.022 | 0.2094 | 0.756 | 0.3201 | 0.2748 | 0.3654 | 0.120 | |
| ours | 3.326 | 0.0619 | 0.951 | 0.0261 | 0.0242 | 0.0280 | 0.424 |
Table 6.
References
- [1]Iro Armeni, Sasha Sax, Amir R Zamir, and Silvio Savarese. Joint 2d-3d-semantic data for indoor scene understanding. arXiv preprint arXiv:1702.01105, 2017.arxiv.org/abs/1702.01105
- [2]Simon Baker and Shree K Nayar. A theory of single-viewpoint catadioptric image formation. International journal of computer vision, 35(2):175–196, 1999.DOI
- [3]Oleksandr Bogdan, Viktor Eckstein, Francois Rameau, and Jean-Charles Bazin. Deepcalib: A deep learning approach for automatic intrinsic calibration of wide field-of-view cameras. In Proceedings of the 15th ACM SIGGRAPH European conference on visual media production, pp. 1–10, 2018.
- [4]Jakob Engel, Kiran Somasundaram, Michael GoeSele, Albert Sun, Alexander Gamino, Andrew Turner, Arjang Talattot, Arnie Yuan, Bilal Souti, Brighid Meredith, et al. Project aria: A new tool for egocentric multi-modal ai research. arXiv preprint arXiv:2308.13561, 2023.arxiv.org/abs/2308.13561
- [5]Girish Chandar Ganesan, Yuliang Guo, Liu Ren, and Xiaoming Liu. Unidac: universal metric depth estimation for any camera. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 26953–26963, 2026.
- [6]Christopher Geyer and Kostas Daniilidis. A unifying theory for central panoramic systems and practical implications. In European conference on computer vision, pp. 445–461. Springer, 2000.DOI
- [7]Yijing Guo, Mengjun Chao, Luo Wang, Tianyang Zhao, Haizhao Dai, Yingliang Zhang, Jingyi Yu, and Yujiao Shi. Panovggt: Feed-forward 3d reconstruction from panoramic imagery. arXiv preprint arXiv:2603.17571, 2026.arxiv.org/abs/2603.17571
- [8]Yuliang Guo, Sparsh Garg, S Mahdi H Miangoleh, Xinyu Huang, and Liu Ren. Depth any camera: Zero-shot metric depth estimation from any camera. In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 26996–27006. IEEE, 2025.arxiv.org/abs/2501.02464
- [10]Huajian Huang, Changkुन Liu, Yipeng Zhu, Hui Cheng, Tristan Braud, and Sai-Kit Yeung. 360loc: A dataset and benchmark for omnidirectional visual localization with cross-device queries. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 22314–22324, 2024.arxiv.org/abs/2311.17389
- [11]Linyi Jin, Jianming Zhang, Yannick Hold-Geoffroy, Oliver Wang, Kevin Blackburn-Matzen, Matthew Sticha, and David Fouhey. Perspective fields for single image camera calibration. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 17307–17316. IEEE, 2023.arxiv.org/abs/2212.03239
- [12]Dongki Jung, Jaehoon Choi, Adil Qureshi, Somi Jeong, Dinesh Manocha, and Suyong Yeon. Wid3r: Wide field-of-view 3d reconstruction via camera model conditioning. In European Conference on Computer Vision, pp. 665–684. Springer, 2026.
- [14]Bogdan Khomutenko, Gaëtan Garcia, and Philippe Martinet. An enhanced unified camera model. IEEE Robotics and Automation Letters, 1(1):137–144, 2015.
- [15]Yiyi Liao, Jun Xie, and Andreas Geiger. Kitti-360: A novel dataset and benchmarks for urban scene understanding in 2d and 3d. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45 (3):3292–3310, 2022.DOI
- [16]Haotong Lin, Sili Chen, Junhao Liew, Donny Y Chen, Zhenyu Li, Guang Shi, Jiashi Feng, and Bingyi Kang. Depth anything 3: Recovering the visual space from any views. arXiv preprint arXiv:2511.10647, 2025a.arxiv.org/abs/2511.10647
- [17]Haotong Lin, Sida Peng, Jingxiao Chen, Songyou Peng, Jiaming Sun, Minghuan Liu, Hujun Bao, Jiashi Feng, Xiaowei Zhou, and Bingyi Kang. Prompting depth anything for 4k resolution accurate metric depth estimation. In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 17070–17080. IEEE, 2025b.arxiv.org/abs/2412.14015
- [18]Christopher Mei and Patrick Rives. Single view point omnidirectional camera calibration from planar grids. In Proceedings 2007 IEEE International Conference on Robotics and Automation, pp. 3945–3950. IEEE, 2007.DOI
- [20]Maxime Oquab, Timothée Darcet, Théó Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaeddin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023.arxiv.org/abs/2304.07193
- [21]Luigi Piccinelli, Christos Sakaridis, Mattia Segu, Yung-Hsu Yang, Siyuan Li, Wim Abbeloos, and Luc Van Gool. Unik3d: Universal camera monocular 3d estimation. In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1028–1039. IEEE, 2025.arxiv.org/abs/2503.16591
- [22]René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. Vision transformers for dense prediction. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 12159–12168. IEEE, 2021.arxiv.org/abs/2103.13413
- [23]Jeremy Reizenstein, Roman Shapovalov, Philipp Henzler, Luca Sbordone, Patrick Labatut, and David Novotny. Common objects in 3d: Large-scale learning and evaluation of real-life 3d category reconstruction. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 10901–10911, 2021.arxiv.org/abs/2109.00512
- [24]Davide Scaramuzza, Agostino Martinelli, and Roland Siegwart. A toolbox for easily calibrating omnidirectional cameras. In 2006 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5695–5701. IEEE, 2006.
- [25]Thomas Schops, Viktor Larsson, Marc Pollefeys, and Torsten Sattler. Why having 10,000 parameters in your camera model is better than twelve. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2535–2544, 2020.arxiv.org/abs/1912.02908
- [26]Javier Tirado-Garín and Javier Civera. Anycalib: On-manifold learning for model-agnostic single-view camera calibration. In 2025 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 8044–8055. IEEE, 2025.
- [27]Vladyslav Usenko, Nikolaus Demmel, and Daniel Cremers. The double sphere camera model. In 2018 International Conference on 3D Vision (3DV), pp. 552–560. IEEE, 2018.DOI
- [28]Igor Vasiljevic, Vitor Guizilini, Rares Ambrus, Sudeep Pillai, Wolfram Burgard, Greg Shakhnarovich, and Adrien Gaidon. Neural ray surfaces for self-supervised learning of depth and ego-motion. In 2020 International Conference on 3D Vision (3DV), pp. 1–11. IEEE, 2020.arxiv.org/abs/2008.06630
- [29]Alexander Veicht, Paul-Edouard Sarlin, Philipp Lindenberger, and Marc Pollefeys. Geocalib: Learning single-image calibration with geometric optimization. In European Conference on Computer Vision, pp. 1–20. Springer, 2024.arxiv.org/abs/2409.06704
- [30]Jianyuan Wang, Minghao Chen, Nikita Karaev, Andrea Vedaldi, Christian Rupprecht, and David Novotny. Vggt: Visual geometry grounded transformer. In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5294–5306. IEEE, 2025a.arxiv.org/abs/2503.11651
- [31]Ruicheng Wang, Sicheng Xu, Cassie Dai, Jianfeng Xiang, Yu Deng, Xin Tong, and Jiaolong Yang. Moge: Unlocking accurate monocular geometry estimation for open-domain images with optimal training supervision. In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5261–5271. IEEE, 2025b.arxiv.org/abs/2410.19115
- [32]Scott Workman, Connor Greenwell, Menghua Zhai, Ryan Baltenberger, and Nathan Jacobs. Deep-focal: A method for direct focal length estimation. In 2015 IEEE International Conference on Image Processing (ICIP), pp. 1369–1373. IEEE, 2015.DOI
- [33]Lihe Yang, Bingyi Kang, Zilong Huang, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything: Unleashing the power of large-scale unlabeled data. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10371–10381. IEEE, 2024.arxiv.org/abs/2401.10891
- [34]Senthil Yogamani, Ciarán Hughes, Jonathan Horgan, Ganesh Sistu, Padraig Varley, Derek O’Dea, Michal Uricár, Stefan Milz, Martin Simon, Karl Amende, et al. Woodscape: A multi-task, multi-camera fisheye dataset for autonomous driving. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 9308–9318, 2019.arxiv.org/abs/1905.01489
- [35]Zhengyou Zhang. A flexible new technique for camera calibration. IEEE Transactions on pattern analysis and machine intelligence, 22(11):1330–1334, 2000.DOI