betaFeedback

WorksComputer Science

OrTeN: Neural Inverse Graphics for Joint Terrain and Line-scan Camera Trajectory Reconstruction

Abstract

Digital terrain maps (DTMs) are used for landing site selection, terrain-relative navigation, and geological analysis in planetary exploration. Existing state-of-practice reconstruction techniques for creating DTMs from satellite imagery use multi-view stereo, requiring complex pipelines for dealing with camera error, changing illumination, and images with limited disparity. This work introduces Orbital Terrain Networks (OrTeN), a DTM construction method that uses neural inverse rendering to jointly optimize a textured height map and orbital camera corrections from line-scan satellite images. OrTeN operates directly on calibrated line-scan images with pixel values expressed as radiance factor (I/F) and corrects both broad trajectory pose inconsistencies and high-frequency pointing jitter jointly with surface reconstruction during training. Surface-projected patch normalized cross-correlation (NCC) enforces consistency between observations of the same terrain, refining an initial rough reconstruction. A semi-empirical reflectance model and shadow rendering are used to fit surface appearance from multi-date imagery with differing solar illumination. OrTeN requires only calibrated line-scan images and initial camera poses and does not use any structural priors including ranging/depth measurements or initial rough DTMs. OrTeN is evaluated on four planetary imagery datasets. Aligned terrain RMSEs are 2.35 m at Jezero Crater, 2.82 m at Gale Crater, 16.18 m at Olympus Mons, and 0.43 m at McGetchin Crater.

Figure 1. Summary of OrTeN DTM reconstructions. OrTeN produces high-quality digital terrain maps (DTMs) from multi-view imagery without structural priors. Combining bundle adjustment, normalized cross-correlation, a semi-empirical photometric model, and geometric shadows allows OrTeN to reconstruct terrain reliably across multiple different instruments, planetary bodies, and lighting conditions.

Introduction

Digital terrain maps (DTMs) are an important supporting data product in space exploration. Accurate DTMs enable landing site selection and terrain-relative navigation (TRN) during entry, descent, and landing for modern autonomous space exploration missions such as Mars 2020 [11, 21]. They also provide the topographic context needed to interpret planetary landforms and their geological history [29].

Current pipelines for orbital DTM production used in the space science community rely on classical multi-view stereo (MVS) techniques to infer terrain structure—features are detected and matched between radiometrically calibrated images, bundle adjustment (BA) is performed to refine camera poses, and stereo correspondences are triangulated to produce a 3D terrain model [7, 22]. MVS can produce high-quality maps suitable for mission operations, but a number of challenges inherent to both MVS itself and its application to satellite datasets complicate the process. These challenges include the difficulty of getting reliable feature matches in the presence of shadows or low-texture regions, the line-scan geometry of most satellite images not admitting the epipolar constraints assumed by traditional MVS, and transients like dust storms or seasonal changes that alter the appearance of the terrain. Correcting for these effects can require labor-intensive preprocessing and careful manual intervention, making MVS pipelines cumbersome to use.

Neural inverse graphics provides a methodology for scene reconstruction that can address many of the challenges faced by classical MVS pipelines. For instance, satellite adaptations of neural radiance fields (NeRFs) model changing illumination and acquisition appearance while recovering scene geometry [15, 26, 27]. However, full 3D volumetric representations admit degenerate solutions—for instance, all scene density concentrated in front of cameras, explaining the observed images without recovering the true geometry—which can be particularly problematic when working with low-disparity orbital imagery where depth cues are weak. While structural priors can help mitigate these degeneracies [36, 38], these measurements are not always available or reliable.

OrTeN mitigates the shortcomings of classical MVS and volumetric neural reconstruction by optimizing a textured height map directly. Critically, OrTeN requires no structural priors to achieve accurate terrain reconstruction from images and approximate camera poses. It builds on Neural Terrain Maps (NTM) [8], adding direct bundle adjustment of line-scan camera extrinsics jointly with training, a semi-empirical photometric model, and geometric shadow modeling. The contributions of this work are:

  1. A pipeline for creating accurate DTMs through differentiable volume rendering of textured height maps including a first-order approximation of distance to the height map along the ray to improve performance at oblique angles.

  2. Joint terrain and line-scan camera optimization, including smooth trajectory adjustment and high-frequency jitter removal, and a semi-empirical, spatially varying photometric model including geometric shadow modeling with an image-space prior.

  3. An evaluation on four Mars and Moon scenes with varied relief and illumination, using the Context Camera (CTX) and Lunar Reconnaissance Orbiter Camera (LROC) Narrow Angle Camera (NAC).

OrTeN achieves aligned height errors below the ground sampling distance (GSD) in three scenes and improves on the tested Ames Stereo Pipeline (ASP) reconstruction in those scenes.

Related Work

DTM creation with MVS. A number of well-established pipelines exist for creating DTMs from orbital imagery using MVS, including the Ames Stereo Pipeline (ASP), satellite stereo pipeline (s2p), and COLMAP [7, 13, 33, 34]. All of these methods use feature extraction and matching to constrain camera alignment before performing dense stereo matching to triangulate terrain. Orbital imagery is often taken by linescan cameras, and therefore integration into standard MVS pipelines requires particular care because each detector line is acquired at a different pose. This precludes the standard epipolar geometry assumptions used in perspective-camera MVS to make searching for correspondences straightforward. Linescan epipolar loci are generally curved, so the global rectification used for ideal perspective cameras must be replaced by local or sensor-specific constructions [14, 19]. ASP can also refine time-varying camera states to correct high-frequency jitter [4].

Illumination changes, common in multi-date satellite data acquisitions, create an additional challenge for traditional MVS approaches. Shadows and surface reflectance can change between acquisitions even when terrain is static and make feature selection and matching more difficult. Therefore, masking shadows and other illumination-dependent artifacts is often necessary to improve the robustness of MVS matching. Shadows can also provide important geometric information which can be exploited in reconstruction, and shape-from-shading methods exploit this to further refine existing DTMs [3]. OrTeN performs trajectory bundle adjustment, including jitter removal, jointly with training, and uses a semi-empirical photometric model with explicit geometric shadow handling to improve terrain reconstruction.

Neural reconstruction of orbital imagery. S-NeRF introduced illumination-aware neural rendering for satellite photogrammetry, and Sat-NeRF added rational polynomial coefficient (RPC) camera support and transient appearance modeling. EO-NeRF built on these previous methods to establish a baseline framework for general-purpose scenes on Earth by incorporating geometrically consistent shadows with learned radiometric and camera corrections and transient appearance embeddings [15, 26, 27]. Season-NeRF separated seasonal appearance from Sun and viewing geometry [17], while a semi-empirical bidirectional reflectance distribution function (BRDF) was incorporated into BRDF-NeRF [39]. Other methods address the geometry directly through depth supervision, surface constraints, or gravity-aligned and coarse-to-fine regularization [31, 36, 38]. Gaussian splatting for satellite imagery has also emerged as an alternative to methods utilizing volume rendering, offering efficient, high-quality neural scene representations [2, 9].

More recent works have begun to focus on large-scale terrain reconstruction using neural inverse graphics. Both NTM and LNEM introduce direct rendering of linescan imagery for terrain reconstruction [8, 23]. For descent imagery, de Almeida et al. incorporate a neural height field into reconstruction from wide-angle cameras [12]. OrTeN eschews structural priors and retains the explicitly parameterized terrain representation, as is done in NTM, and adds ray-distance-based opacity and improved regularization with normalized cross-correlation (NCC). Its coarse-to-fine optimization follows the principle established by BARF: low-frequency structure should stabilize camera alignment before fine appearance is exposed [24].

Methodology

OrTeN represents the scene with a height field hθ:R2→Rh_\theta: \mathbb{R}^{2} \to\mathbb{R} in a local east–north–up (ENU) frame and learned surface appearance (Figure 2). The surface is x=(x,y,hθ(x,y))\mathbf{x} = (x,y,h_\theta(x,y)), and unit-direction camera rays are given by r(t)=o+td\mathbf{r}(t) = \mathbf{o} + t\mathbf{d}. The origin is constructed as the common look point of all cameras, the horizontal bounds are set by the image footprints on the horizontal plane, enlarged by a factor of 1.5, and the vertical bounds are set by global planetary height limits.

Key elements of the OrTeN pipeline

Figure 2. Key elements of the OrTeN pipeline. Two linescan acquisitions i,ji, j observe overlapping terrain swaths. The dashed trajectory indicates the original extrinsics, which are corrected during bundle adjustment. Bundle adjustment refines position and attitude, including angular jitter. Terrain networks predict height, albedo, material mixture, and a Sun-dependent ambient factor. Gray-to-red camera-ray samples illustrate interval opacity αj\alpha_j, with transition width controlled by β\beta. Sun-directed opacity integration determines visibility VV. Source image patches are lifted onto the current height field and compared with neighbor observations through normalized cross-correlation ρ\rho, constraining terrain and cameras jointly.

Rendering a height field

OrTeN follows the lineage of NeuS, which derives opacity from the decrease of a logistic cumulative distribution function (CDF) along a ray through an implicit surface [37]. Applying this construction to distance along the ray rather than vertical height residual gives

αj=[1−Φβ(q(tj+1))Φβ(q(tj))]+,(1)\alpha_j = \left[1-\frac{\Phi_\beta(q(t_{j+1}))}{\Phi_\beta(q(t_j))}\right]_+, \tag*{(1)}
Tj=∏k<j(1−αk),wj=Tjαj.(2)T_j = \prod_{k<j}(1-\alpha_k), \qquad w_j = T_j\alpha_j. \tag*{(2)}

Here Φβ(q)=(1+e−q/β)−1\Phi_\beta(q) = (1+e^{-q/\beta})^{-1}, [z]+=max⁡(z,0)[z]_+ = \max(z,0), αj\alpha_j is interval opacity, TjT_j is transmittance to that interval, and wjw_j is its rendering weight. The CDF ratio is evaluated on a log scale to improve numerical stability. Sampling locations are chosen with a proposal network as is done in Nerfacto [35].

To obtain qq, the effect of terrain slope and viewing direction on the height residual is considered:

f(t)=rz(t)−hθ(rxy(t)),(3)f(t) = r_z(t) - h_\theta(r_{xy}(t)), \tag*{(3)}
f′(t)=dz−∇hθ(rxy(t))Tdxy,(4)f'(t) = d_z - \nabla h_\theta(r_{xy}(t))^\mathsf{T}d_{xy}, \tag*{(4)}
q(t)=f(t)max⁡(−f′(t),10−3).(5)q(t) = \frac{f(t)}{\max(-f'(t),10^{-3})}. \tag*{(5)}

Subscripts xyxy and zz select horizontal and vertical components. For a plane, q(t)=t⋆−tq(t) = t_\star- t when the denominator clamp is inactive, where t⋆t_\star is the intersection distance. This formulation ensures the learned width β>0\beta> 0 has the same metric meaning across terrain slopes and viewing angles. Height derivatives are calculated by finite differences using a first-order formulation.

Calibrated I/F image formation

Training uses calibrated scalar radiance factor I/F=πL/F⊙I/F = \pi L/F_\odot images, where LL is observed radiance and F⊙F_\odot is solar irradiance at the body [6, 32]. Albedo aψa_\psi is aggregated over the ray samples:

A=∑jwj,aˉ=∑jwjaψ(rxy(tˉj))max⁡(A,10−8),(6)A = \sum_j w_j, \qquad\bar{a} = \frac{\sum_j w_j a_\psi(r_{xy}(\bar{t}_j))}{\max(A,10^{-8})}, \tag*{(6)}

where tˉj\bar{t}_j is the interval midpoint. A second network head predicts the spatial Lunar–Lambert mixture mϕ(x,y)m_\phi(x,y), integrated in the same way to give mˉ\bar{m}. The normalized, weighted terrain normal n\mathbf{n} is used to compute positive incidence and emission cosines μ0=[nTs]+\mu_0 = [\mathbf{n}^\mathsf{T}\mathbf{s}]_+ and μ=[nT(−d)]+\mu= [\mathbf{n}^\mathsf{T}(-\mathbf{d})]_+, with s\mathbf{s} pointing toward the Sun. The direct-light factor is

F(μ0,μ,m)=(1−m)μ0+2mμ0max⁡(μ0+μ,10−6),(7)F(\mu_0,\mu,m) = (1-m)\mu_0 + \frac{2m\mu_0}{\max(\mu_0+\mu,10^{-6})}, \tag*{(7)}
D=F(μ0,μ,mˉ)F(cos⁡30∘,1,mˉ).(8)D = \frac{F(\mu_0,\mu,\bar{m})}{F(\cos30^\circ,1,\bar{m})}. \tag*{(8)}

This interpolates between Lambert and Lommel–Seeliger responses [28].

Geometric shadows. Starting from the weighted camera-ray depth, a surface intersection is refined and traced toward the Sun until the ray leaves the scene bounds. Solar interval opacities αk⊙\alpha_k^\odot are converted to visibility using

V=Vatt∏k(1−αk⊙),Vη=(1−η)Vatt+ηV.(9)V = V_{\mathrm{att}}\prod_k(1-\alpha_k^\odot), \qquad V_\eta= (1-\eta)V_{\mathrm{att}}+\eta V. \tag*{(9)}

Here VattV_{\mathrm{att}} smoothly suppresses light on back-facing slopes and is zero when the Sun is below the local horizon. The startup factor η\eta ramps cast shadows in over 500 updates. Terrain outside the bounds cannot cast shadows in this model.

OrTeN uses a semi-empirical image formation model inspired by EO-NeRF’s shadow-fill and affine brightness approach [27]. The predicted image for acquisition ii is

y^i=gi[VηD+(1−Vη)aamb(s)]aˉ+bi.(10)\hat{y}_i = g_i \left[V_\eta D + (1 - V_\eta)a_{\mathrm{amb}}(s)\right]\bar{a} + b_i. \tag*{(10)}

The Sun-dependent ambient factor aamb∈(0,1)a_{\mathrm{amb}} \in(0, 1) fills shadows without inheriting the direct-light angular response. Scalar gains gi=eui−uˉg_i = e^{u_i-\bar{u}} and offsets absorb acquisition-wide brightness differences, with uˉ=M−1∑iui\bar{u} = M^{-1}\sum_i u_i fixing the geometric mean gain to one across MM acquisitions. The same image model is used throughout training.

Histogram-valley shadow detection [1] is adopted to provide an initial image-space shadow estimate at startup. Only sufficiently dark, distinct pixels are accepted, and mask erosion is applied to remove boundary transients. For an ordinary rendering batch B\mathcal{B}, a regularizer is applied:

Lshadow=∣B∣−1∑p∈Bzpvp2,(11)\mathcal{L}_{\mathrm{shadow}} = |\mathcal{B}|^{-1}\sum_{p\in\mathcal{B}} z_p v_p^2, \tag*{(11)}

where zp=1z_p = 1 marks a detected core and zp=0z_p = 0 means unknown. This loss is one-sided, so unlabeled pixels are not treated as either shadow or sunlit.

Linescan bundle adjustment

A linescan pixel (ℓ,s)(\ell, s) has nominal origin oi0(ℓ)\mathbf{o}_i^0(\ell) and direction di0(ℓ,s)\mathbf{d}_i^0(\ell,s). Intrinsics and distortion are retained, and corrections are optimized in the local scene frame:

oi=oi0+Δpi(u),(12)\mathbf{o}_i = \mathbf{o}_i^0 + \Delta\mathbf{p}_i(u), \tag*{(12)}
di=exp⁡([Δωi(u)]×)di0,(13)\mathbf{d}_i = \exp\left([\Delta\boldsymbol{\omega}_i(u)]_\times\right)\mathbf{d}_i^0, \tag*{(13)}
Δpi(u)=sp∑k=03Bk(u)Pik,(14)\Delta\mathbf{p}_i(u) = s_p\sum_{k=0}^{3} B_k(u)\mathbf{P}_{ik}, \tag*{(14)}
Δωi(u)=sω[∑k=03Bk(u)Wik+rJi(u)].(15)\Delta\boldsymbol{\omega}_i(u) = s_\omega\left[\sum_{k=0}^{3} B_k(u)\mathbf{W}_{ik} + r\mathbf{J}_i(u)\right]. \tag*{(15)}

Here u∈[0,1]u \in[0,1] is normalized acquisition time and BkB_k are cubic Bernstein functions. The matrix [ω]×[\boldsymbol{\omega}]_\times implements the cross product with ω\boldsymbol{\omega}, and its exponential gives the corresponding rotation. The learned controls Pik,Wik\mathbf{P}_{ik},\mathbf{W}_{ik} use physical scales sp,sωs_p,s_\omega, which are constrained by soft priors based on expected trajectory error [11]. During the refinement phase (§ 3.5), the ramp rr introduces angular jitter Ji\mathbf{J}_i, represented by a cubic B-spline with 64 intervals per acquisition. Its global cubic component is removed so it cannot duplicate the smooth trajectory correction.

Surface-guided patch consistency

In addition to uniformly sampled rays, at each training iteration 5×55 \times5 patches are sampled in source images and projected onto the current height field through the refined cameras. Each patch point is then projected from the surface into other acquisitions, where bilinear interpolation is used to calculate observed intensities. For accepted source–neighbor pairs, the normalized cross-correlation loss is

LNCC=EP,i,j[1−⟨Pi−Pˉi,Pj−Pˉj⟩∥Pi−Pˉi∥2∥Pj−Pˉj∥2].(16)\mathcal{L}_{\mathrm{NCC}} = \mathbb{E}_{P,i,j}\left[1-\frac{\left\langle P_i-\bar{P}_i,P_j-\bar{P}_j\right\rangle}{\left\lVert P_i-\bar{P}_i\right\rVert_2\left\lVert P_j-\bar{P}_j\right\rVert_2}\right]. \tag*{(16)}

Here PiP_i, PjP_j contain the observed patch intensities and Pˉi,Pˉj\bar{P}_i,\bar{P}_j are their means. ⟨⋅,⋅⟩\langle\cdot,\cdot\rangle is the inner product. Invalid, occluded, low-contrast, and nearly parallel view pairs are rejected, and up to three of the best eligible neighbors are retained per source patch. NCC is active in both roughing and refinement (§ 3.5).

Architecture and losses

Figure 3 shows the overall network architecture. Training progresses from images downsampled by 4×4\times to 2×2\times, then to native resolution. Active encoding levels are adjusted to approximately match the current image resolution. There are two training stages: roughing and refinement. Roughing ends when terrain and camera motion stabilize or the roughing update limit is reached. Refinement lowers the geometry learning rate, increases NCC sampling, and introduces jitter and local regularization.

OrTeN’s architecture

Figure 3. OrTeN’s architecture. Learned networks are blue, optimized parameters are red, and analytic operations are gray. Layer boxes give output dimensions, and arrow colors indicate activations. One spatial encoding [30] feeds the height, albedo, and Lunar–Lambert material outputs. The ambient network uses a fourth-order spherical harmonic (SH) encoding of the Sun direction. Height also supplies normals and solar visibility VηV_\eta. Proposal sampling is omitted for clarity.

Let sy=Etrain[y2]s_y=\sqrt{\mathbb{E}_{\mathrm{train}}[y^2]} be the RMS brightness of all training pixels. The rendering loss is

LI/F=Ep∈B[(y^p−yp)2/sy2].(17)\mathcal{L}_{I/F}=\mathbb{E}_{p\in B}\left[(\hat{y}_p-y_p)^2/s_y^2\right]. \tag*{(17)}

Albedo starts exactly at sys_y, ambient at 0.2, and gains and offsets at one and zero. The remaining surface and appearance penalties are

Lbend=Ex[sh2∥Hshhθ(x)∥F2],(18)\mathcal{L}_{\mathrm{bend}}=\mathbb{E}_x\left[s_h^2\left\lVert\mathbf{H}_{\mathrm{sh}}h_\theta(x)\right\rVert_F^2\right], \tag*{(18)}
Lwidth=log⁡(1+(β/g0)2),(19)\mathcal{L}_{\mathrm{width}}=\log\left(1+(\beta/g_0)^2\right), \tag*{(19)}
Lradio=Ei[((ui−uˉ)/0.1)2+(bi/0.01)2],(20)\mathcal{L}_{\mathrm{radio}}=\mathbb{E}_i\left[((u_i-\bar{u})/0.1)^2+(b_i/0.01)^2\right], \tag*{(20)}
Lamb=Es[[logit⁡aamb(s)−log⁡9]+2].(21)\mathcal{L}_{\mathrm{amb}}=\mathbb{E}_s\left[\left[\operatorname{logit}a_{\mathrm{amb}}(s)-\log9\right]_+^2\right]. \tag*{(21)}

Here Hsh\mathbf{H}_{\mathrm{sh}} is the finite-difference Hessian. In order, these losses regularize surface bending, control the rendering width, enforce radiometric calibration, and constrain the ambient illumination. The fixed scale g0g_0 is the initial training-image GSD. The optional ambient penalty acts only when the ambient factor exceeds 0.9.

Camera regularization penalizes the dimensionless trajectory controls and, in refinement, the sampled jitter motion. Let ci(u)c_i(u) concatenate the trajectory position and rotation corrections divided by sps_p and sωs_\omega. Its second derivative penalizes trajectory curvature:

Lcamera=λCEi,κ(∥Pik∥22+∥Wik∥22)+λC1r>0Ei,u∥Ji(u)∥22+λKEi∫01∥ci′′(u)∥22du.(22)\mathcal{L}_{\mathrm{camera}}=\lambda_C\mathbb{E}_{i,\kappa}\left(\left\lVert\mathbf{P}_{ik}\right\rVert_2^2+\left\lVert\mathbf{W}_{ik}\right\rVert_2^2\right) +\lambda_C\mathbf{1}_{r>0}\mathbb{E}_{i,u}\left\lVert\mathbf{J}_i(u)\right\rVert_2^2 +\lambda_K\mathbb{E}_i\int_0^1\left\lVert c_i''(u)\right\rVert_2^2du. \tag*{(22)}

For each camera or solar ray, the proposal loss is the histogram-envelope objective used by Nerfstudio [35]:

Lenv=∑lEj[[wj−Wlj]2wj+10−7].(23)\mathcal{L}_{\mathrm{env}} = \sum_l \mathbb{E}_j \left[\frac{[w_j-W_{lj}]^2}{w_j+10^{-7}}\right]. \tag*{(23)}

Here WljW_{lj} is proposal level ll’s total mass in bins overlapping final interval jj. The camera and solar proposal losses are summed as Lproposal\mathcal{L}_{\mathrm{proposal}}. The complete objective is

L=LI/F+ηλNLNCC+λWLwidth+λRLradio+Lcamera+Lproposal+r(λBLbend+λSLshadow+λALamb).(24)\begin{aligned} \mathcal{L} ={}& \mathcal{L}_{I/F}+\eta\lambda_N\mathcal{L}_{\mathrm{NCC}}+\lambda_W\mathcal{L}_{\mathrm{width}}+\lambda_R\mathcal{L}_{\mathrm{radio}}\\ &+\mathcal{L}_{\mathrm{camera}}+\mathcal{L}_{\mathrm{proposal}}\\ &+r\left(\lambda_B\mathcal{L}_{\mathrm{bend}}+\lambda_S\mathcal{L}_{\mathrm{shadow}}+\lambda_A\mathcal{L}_{\mathrm{amb}}\right). \tag*{(24)} \end{aligned}

The refinement ramp rr is zero during roughing and rises over 500 updates after transition.

Experiments and Results

Datasets

Four lunar and Martian datasets were selected for their varied terrain and illumination and the availability of reference DTMs. Jezero Crater, Gale Crater, and Olympus Mons each contain fifteen 1024 × 1024 CTX crops. McGetchin Crater contains ten LROC NAC crops of the same size. Images are calibrated to I/F using the Integrated Software for Imagers and Spectrometers (ISIS) and paired with initial line-scan camera states. Table 1 gives additional details on the instruments and reference DTM quality for each dataset. The Olympus dataset reference uses a DTM separately reconstructed from the same image collection because a high-quality engineering DTM was unavailable.

DatasetGSD (m/px)Band (nm)Reference DTM (post spacing)Acc. / prec. (m)
Jezeroa^{a} Crater6.41500–700Mars 2020 (1 m) [16]∼10–20 / ∼0.1–0.4 [20]
Galea^{a} Crater6.23500–700Gale mosaic (1 m) [10]∼10–20 / ∼0.1–0.4 [20]
Olympusa^{a} Mons5.56500–700Audited ASP (11.12 m)— / 11.68‡
McGetchinb^{b} Crater0.84400–750NEW200CRAT3 (3 m) [5]1.51 / 1.46 [5]

Table 1. Dataset instrument and DTM reference overview. Instruments are a^{a}CTX [25] and b^{b}LROC NAC [32]. Acc. / prec. denotes vertical accuracy / precision. GSD is estimated by intersecting adjacent pixel rays from initial cameras with the local ENU xyxy plane. ‡For Olympus Mons, precision was estimated from the RMS elevation difference between two stereo DTMs generated from image pairs with no cameras in common during construction of the ASP reference used to evaluate OrTeN.

Evaluation protocol and metrics

Metrics for each scene are calculated over a fixed reference footprint that is covered by at least five initial cameras. To resolve the gauge freedom inherent in the bundle adjustment process, reconstructed DTMs are aligned to the reference DTM using a six-degree-of-freedom rigid transformation. No scale change, local warping, or separate alignment for different metrics is applied. For aligned height errors ep=h^a(p)−href(p)e_p=\hat{h}_a(p)-h_{\mathrm{ref}}(p) on NN valid posts, the metrics are

RMSEa=N−1∑pep2,MAEa=N−1∑p∣ep∣.(25)\mathrm{RMSE}_a=\sqrt{N^{-1}\sum_p e_p^2},\qquad\mathrm{MAE}_a=N^{-1}\sum_p\lvert e_p\rvert. \tag*{(25)}

The subscript aa denotes metrics applied to rigidly aligned DTMs, so these scores measure relative shape rather than absolute geolocation.

Slope is evaluated using a local quadratic surface fit calculated over a 9 × 9 sliding window. This follows established practices in terrain analysis [18] and reduces sensitivity to reference DTM artifacts. Slope RMSEa_a is the RMS difference between slope angles in degrees, under the same rigid alignment. Although OrTeN permits direct network differentiation, the reported comparison differentiates polynomial fits to every method’s raster export to match spatial scale. Height RMSE and MAE use unsmoothed elevations.

One 64 × 64 block per acquisition, at the full image resolution, is held out during training for perceptual metric calculation. This is necessary because of learned image-dependent radiometric calibration. PSNR is 10log⁡10(R2/MSE)10\log_{10}(R^2/\mathrm{MSE}) where RR is the 99th minus 1st percentile of the pooled held-out observations per scene. SSIM uses the same range and 11 × 11 Gaussian windows (σ=1.5\sigma=1.5 pixels), averaged over valid window centers.

Training details

All training runs use a single seed, are limited to 15,000 iterations, and use a batch size of 8,192 rays. The same configuration for OrTeN is used across all four scenes. This includes enabling trajectory BA, jitter adjustment, and the detected-shadow prior with weight 0.01. The roughing stage iteration cap is set to 8,000 updates.

Comparisons with depth-supervised EO-NeRF [27] and ASP [7] are reported. EO-NeRF uses the same camera-derived scene bounds as OrTeN, 8,192 rays per update, 128 depth samples per ray, and fixed cameras. Its depth targets come from the unaligned ASP stereo mosaics, without reference-terrain supervision or registration. ASP uses image-based BA, dense stereo, and jitter refinement anchored to its own stereo terrain estimates. The same ASP pipeline is used for all scenes.

Mean setup-plus-training times are 52.7/41.2 minutes for OrTeN on NVIDIA RTX 6000 Ada Generation/RTX PRO 6000 Blackwell GPUs, respectively. These include periodic evaluation and checkpointing, exclude final evaluation, and average different scene/configuration mixes across GPUs. OrTeN uses Adam with a 500-step warmup to learning rates

McGetchin Crater with and without BAVector (PDF)

caption: McGetchin Crater with and without BA. Renderings and crater errors use shared scales. Boxes use the baseline camera. RMSEa_a: 0.43 / 10.36 m. PSNR: 35.13 / 21.04 dB.

of 3×10−33 \times10^{-3} for geometry and appearance, 10−310^{-3} for cameras, and 10−210^{-2} for proposal networks. Geometry decays during roughing, then starts refinement at most 1.5×10−41.5 \times10^{-4} and decays to 5×10−55 \times10^{-5}. Appearance and camera rates finish at 3×10−43 \times10^{-4} and 10−510^{-5}.

Results and ablations

Tab. 2 gives the results of the baseline OrTeN configuration on each scene. OrTeN attains sub-GSD height RMSEa_a in Jezero, Gale, and McGetchin scenes. On the Olympus Mons scene, the RMSEa_a remains at 2.91×2.91 \times GSD. Its large relief (almost 3 km) and substantial cast shadows make reconstruction challenging. The lack of an independent, high-accuracy reference limits comparison with ASP, which performs similarly on this scene.

Depth-supervised EO-NeRF fails to reconstruct accurate DTMs in any of the scenes despite access to structural supervision. It falls short of OrTeN both visually and in structural accuracy. Noting that the default EO-NeRF configuration trains for much longer (300,000 iterations) and takes 10-20 hours to converge, this comparison demonstrates that OrTeN’s approach achieves meaningful improvement in both efficiency and accuracy.

ASP recovers sound DTMs for all four scenes, but performs worse than OrTeN on all scenes except Olympus Mons. as opposed to ASP, which typically processes images pairwise or in smaller subsets. OrTeN also benefits from its semi-empirical appearance modeling which allows it to

Scene / methodRMSEa_a / MAEa_a / slope RMSEa_a (m / m / °)PSNR / SSIM (dB / —)
Jezero Crater
OrTeN2.35 / 1.61 / 1.8728.20 / 0.771
EO-NeRF + depth62.08 / 47.61 / 39.1525.26 / 0.598
ASP4.90 / 3.56 / 4.81— / —
Gale Crater
OrTeN2.82 / 2.00 / 2.5431.37 / 0.840
EO-NeRF + depth156.37 / 125.21 / 27.0429.64 / 0.719
ASP4.73 / 2.89 / 4.02— / —
Olympus Mons
OrTeN16.18 / 8.57 / 3.4329.71 / 0.904
EO-NeRF + depth25.76 / 20.14 / 9.0116.87 / 0.708
ASP15.34 / 7.38 / 4.90— / —
McGetchin Crater
OrTeN0.43 / 0.28 / 0.9835.13 / 0.940
EO-NeRF + depth226.35 / 187.56 / 71.9923.53 / 0.666
ASP0.56 / 0.38 / 1.14— / —

Table 2. DTM reconstruction comparison. Best values per scene are bold. Height errors are evaluated across each method’s valid portion of the fixed evaluation footprint. Slope uses local quadratic fits. PSNR and SSIM use held-out pixels and a fixed intensity range per scene. ASP produces no visual reconstruction.

better deal with shadows and varying illumination conditions which may confound feature matching in purely photogrammetric approaches.

Four ablation studies were conducted to evaluate the impact of the three key features in this work: bundle adjustment with jitter, normalized cross-correlation (NCC) for photometric consistency, and semi-empirical appearance modeling with shadow modeling. The results of the ablation are summarized in Tab. 3.

ConfigurationJezero RMSEa_a / PSNRGale RMSEa_a / PSNROlympus RMSEa_a / PSNRMcGetchin RMSEa_a / PSNR
Baseline2.35 / 28.202.82 / 31.3716.18 / 29.710.43 / 35.13
No BA4.26 / 26.409.25 / 28.3930.01 / 24.3110.36 / 21.04
No jitter2.40 / 28.212.76 / 29.5616.47 / 30.340.46 / 34.64
No NCC16.80 / 29.8542.22 / 29.0967.40 / 22.060.52 / 34.63
No shadows2.32 / 28.062.90 / 31.3115.81 / 29.340.43 / 35.21

Table 3. Core OrTeN ablations. No BA also removes jitter. No shadows removes attached and cast visibility and the shadow prior.

Removing BA significantly degrades the quality of the reconstructed DTMs across all scenes, highlighting the importance of joint optimization of camera parameters and scene geometry. fig:4 shows baseline and no-BA renderings of the same acquisition for the McGetchin Crater scene. The no-BA ablation shows clearly degraded visual reconstruction quality with a drop of more than 10 dB in reserved-pixel PSNR. The DTM is similarly impacted with RMSEa_a increasing from 0.43 to 10.36 m and the quality of the reconstruction visibly degrading, especially in the fine detail around the central crater region.

Disabling NCC has an even more pronounced effect on the quality of the reconstructed DTMs. Figure 5 shows baseline and no-NCC renderings of the same acquisition for the Jezero Crater scene, and again illustrates the degradation of fine detail around the crater region. In some of the scenes, the PSNR improves in the no-NCC ablation compared to the baseline. Rendering loss alone can therefore be misleading when assessing terrain reconstruction, especially in low-disparity orbital scenes.

Figure 5. Jezero Crater with and without NCC. Green boxes locate DTM error crops using each run’s learned camera extrinsics and rigid alignment to the reference terrain. RMSEa_a: 2.35 / 16.80 m. PSNR: 28.20 / 29.85 dB.

Removal of geometric shadows produces a less pronounced effect on the quality of the reconstructed DTMs compared to the other ablations, but can be observed in reconstruction quality. Figure 6 shows baseline and no-shadow renderings of the same acquisition for the Olympus Mons scene. The baseline is able to reconstruct the large shadow cast by the cliff face while the ablated scene cannot.

Figure 6. Olympus Mons shadow ablation. Cyan marks shadows detected by the first-valley prior. Images share an intensity scale.

Conclusion

OrTeN jointly reconstructs terrain and refines line-scan cameras from calibrated orbital images without depth supervision or stereo initialization. Across four Mars and Moon scenes, it achieves sub-GSD aligned terrain RMSE in three and improves on the tested ASP reconstruction in those same three scenes. The ablations demonstrate the importance of camera refinement and surface-patch NCC for achieving high geometric accuracy. They also show that an accurate image reconstruction can coexist with an incorrect surface. Comparison with EO-NeRF with depth supervision also demonstrates that structural priors are not sufficient to achieve accurate terrain reconstruction efficiently.

However, a number of limitations remain. Without structural priors such as ranging measurements or known control points, absolute geolocation is not possible. Furthermore, it is possible that integrating structural priors into OrTeN could improve reconstruction accuracy and efficiency and should be tested. While the height map representation is an effective structural prior for the datasets considered here, when detail becomes sufficient, this representation becomes more limiting. For instance, height maps cannot represent overhangs or complex 3D structures like caves. Future work should work on mitigating these shortcomings to allow ground level imagery to be integrated into the terrain reconstruction process.

References

  1. [1]K. R. M. Adeline, M. Chen, X. Briottet, S. K. Pang, and N. Paparoditis. Shadow detection in very high spatial resolution aerial images: A comparative study. ISPRS Journal of Photogrammetry and Remote Sensing, 80:21–38, 2013.DOI
  2. [2]Luca Savant Aira, Gabriele Facciolo, and Thibaud Ehret. Gaussian Splatting for Efficient Satellite Image Photogrammetry. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5959–5969, 2025.arxiv.org/abs/2412.13047
  3. [3]Oleg Alexandrov and Ross A. Beyer. Multiview Shape-From-Shading for Planetary Images. Earth and Space Science, 5(10):652–666, 2018.DOI
  4. [4]Ames Stereo Pipeline Developers. Ames Stereo Pipeline: jitter_solve. Software documentation, version 3.7.0, 2026. Accessed September 22, 2026.
  5. [5]Arizona State University and Intuitive Machines. NAC_DTM_NEW200CRAT3: New 222 m crater stereo DTM. LROC Planetary Data System product README, January 15, 2026.
  6. [6]J. Bell III, M. Malin, M. Caplinger, J. Fahle, M. Wolff, B. Cantor, P. James, F. Ghaemi, L. Posiolova, Michael Ravine, K. Supluver, W. Calvin, R. Clancy, Kenneth Edgett, Laurence Edwards, R. Haberle, Amy Hale, S. Lee, Melissa Rice, and Rebecca Williams. Calibration and performance of the mars reconnaissance orbiter context camera (CTX). Mars, 8: 1–14, 2013.DOI
  7. [7]Ross A. Beyer, Oleg Alexandrov, and Scott McMichael. The Ames Stereo Pipeline: NASA’s Open Source Software for Deriving and Processing Terrain Data. Earth and Space Science, 5(9):537–548, 2018.DOI
  8. [8]Josef X. Biberstein, Guilherme Cavalheiro, Juyoep Han, and Sertac Karaman. Construction of digital terrain maps from multi-view satellite imagery using neural volume rendering. arXiv preprint arXiv:2508.01386, 2025.
  9. [9]Pierrick Bournz, Luca Savant Aira, Thibaud Ehret, and Gabriele Facciolo. EOGS++: Earth Observation Gaussian Splatting with Internal Camera Refinement and Direct Panchromatic Rendering, 2025.
  10. [10]F. J. Calef III and T. Parker. MSL Gale Merged Orthophoto Mosaic, 2016.
  11. [11]Yang Cheng, Adnan Ansar, and Andrew Johnson. Making an onboard reference map from MRO/CTX imagery for Mars 2020 lander vision system. Earth and Space Science, 8(8): e2020EA001560, 2021.
  12. [12]Melonie de Almeida, George Brydon, Divya M. Persaud, John H. Williamson, and Paul Henderson. Neural 3d reconstruction of planetary surfaces from descent-phase wide-angle imagery. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 10174–10182, 2026.
  13. [13]C. de Franchis, E. Meinhardt-Llopis, J. Michel, J.-M. Morel, and G. Facciolo. An automatic and modular stereo pipeline for pushbroom images. ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences, II-3: 49–56, 2014.
  14. [14]C. de Franchis, E. Meinhardt-Llopis, J. Michel, J.-M. Morel, and G. Facciolo. On stereo-rectification of pushbroom images. In 2014 IEEE International Conference on Image Processing (ICIP), pages 5447–5451, 2014.DOI
  15. [15]Dawda Derksen and Dario Izzo. Shadow Neural Radiance Fields for Multi-view Satellite Photogrammetry, 2021.
  16. [16]Robin L. Fergason, Donna M. Galuszka, Trent M. Hare, David P. Mayer, and Bonnie L. Redding. Mars 2020 terrain relative navigation HiRISE DTM mosaic. U.S. Geological Survey data release, 2020.
  17. [17]Michael Gableman and Avinash Kak. Incorporating Season and Solar Specificity Into Renderings Made by a NeRF Architecture Using Satellite Images. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(6):4348–4365, 2024.
  18. [18]Jing Gao, James E. Burt, and A-Xing Zhu. Neighborhood size and spatial scale in raster-based slope calculations. International Journal of Geographical Information Science, 2012.
  19. [19]Xun Geng, Qing Xu, Shuai Xing, and Chaozhen Lan. A Generic Pushbroom Sensor Model for Planetary Photogrammetry. Earth and Space Science, 7(5):e2019EA001014, 2020.
  20. [20]Matthew Golombek, John A. Grant, D. Kipp, A. Vasavada, R. Kirk, R. Fergason, et al. Selection of the mars science laboratory landing site. Space Science Reviews, 170:641–737, 2012.
  21. [21]Youeyun Jung, Seongheon Lee, and Hyochoong Bang. Digital Terrain Map Based Safe Landing Site Selection for Planetary Landing. IEEE Transactions on Aerospace and Electronic Systems, 56(1):368–380, 2020.
  22. [22]Kelvin Rodriguez and Astrogeology Science Center. Integrated software for imagers and spectrometers (ISIS) 8.3.0, 2024.
  23. [23]Suwan Lee, Jo Ryeong Yim, Kibaek Park, Dong-Gyu Kim, Eunhyeuk Kim, Minsup Jeong, Chae Kyung Sim, and Seokju Lee. LNEM: Lunar neural elevation model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6508–6517, 2026.
  24. [24]Chen-Hsuan Lin, Wei-Chiu Ma, Antonio Torralba, and Simon Lucey. BARF: Bundle-adjusting neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021.arxiv.org/abs/2104.06405
  25. [25]Michael C. Malin, James F. Bell, Bruce A. Cantor, Michael A. Caplinger, Wendy M. Calvin, R. Todd Clancy, Kenneth S. Edgett, Laurence Edwards, Robert M. Haberle, Philip B. James, Steven W. Lee, Michael A. Ravine, Peter C. Thomas, and Michael J. Wolff. Context camera investigation on board the mars reconnaissance orbiter. Journal of Geophysical Research: Planets, 112:E05S04, 2007.
  26. [26]Roger Mari, Gabriele Facciolo, and Thibaud Ehret. Sat-NeRF: Learning Multi-View Satellite Photogrammetry With Transient Objects and Shadow Modeling Using RPC Cameras. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 1310–1320. IEEE, 2022.
  27. [27]Roger Marí, Gabriele Facciolo, and Thibaud Ehret. Multi-Date Earth Observation NeRF: The Detail Is in the Shadows. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 2035–2045. IEEE, 2023.
  28. [28]Alfred S. McEwen. Photometric functions for photoclinometry and other applications. Icarus, 92(2):298–311, 1991.
  29. [29]Alfred S. McEwen, Eric M. Eliason, James W. Bergstrom, Nathan T. Bridges, Candice J. Hansen, W. Alan Delamere, John A. Grant, Virginia C. Gulick, Kenneth E. Herkenhoff, Laszlo Keszthelyi, Randolph L. Kirk, Michael T. Mel Weitz. Mars Reconnaissance Orbiter’s High Resolution Imaging Science Experiment (HiRISE). Journal of Geophysical Research: Planets, 112:E05S02, 2007.
  30. [30]Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant Neural Graphics Primitives with a Multiresolution Hash Encoding. ACM Transactions on Graphics, 41(4):1–15, 2022.DOI
  31. [31]Yingjie Qu and Fei Deng. Sat-Mesh: Learning Neural Implicit Surfaces for Multi-View Satellite Reconstruction. Remote Sensing, 15(17):4297, 2023.DOI
  32. [32]M. S. Robinson, S. M. Brylow, M. Tschimmel, D. Humm, S. J. Lawrence, P. C. Thomas, B. W. Dennevi, E. Bowman-Cisneros, J. Zerr, M. A. Ravine, M. A. Caplinger, F. T. Ghaemi, J. A. Schaffner, M. C. Malin, P. Mahanti, A. Bartels, J. Anderson, T. N. Tran, E. M. Eliason, A. S. McEwen, E. Turtle, B. L. Jolliff, and H. Hiesinger. Lunar Reconnaissance Orbiter Camera (LROC) instrument overview. Space Science Reviews, 150:81–124, 2010.DOI
  33. [33]Johannes Lutz Schönberger and Jan-Michael Frahm. Structure-from-Motion Revisited. In Conference on Computer Vision and Pattern Recognition (CVPR), 2016.DOI
  34. [34]Johannes Lutz Schönberger, Enliang Zheng, Marc Pollefeys, and Jan-Michael Frahm. Pixelwise View Selection for Unstructured Multi-view Stereo. In European Conference on Computer Vision (ECCV), 2016.DOI
  35. [35]Matthew Tancik, Ethan Weber, Evonne Ng, Ruilong Li, Brent Yi, Justin Kerr, Terrance Wang, Alexander Kristoffersen, Jake Austin, Kamyar Salahi, Abhik Ahuja, David McAllister, and Angjoo Kanazawa. Nerfstudio: A Modular Framework for Neural Radiance Field Development. In Special Interest Group on Computer Graphics and Interactive Techniques Conference Conference Proceedings, pages 1–12, 2023.DOI
  36. [36]Valentin Wagner, Sebastian Bullinger, Michael Arens, and Rainer Stiefelhagen. SatGeo-NeRF: Geometrically regularized Neural Radiance Fields for satellite imagery. ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences, XI-2-2026:367–374, 2026.
  37. [37]Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction. In Advances in Neural Information Processing Systems, pages 27171–27183. Curran Associates, Inc., 2021.arxiv.org/abs/2106.10689
  38. [38]Lulin Zhang and Ewelina Rupnik. SparseSat-NeRF: Dense Depth Supervised Neural Radiance Fields for Sparse Satellite Images, 2023.arxiv.org/abs/2309.00277
  39. [39]Lulin Zhang, Ewelina Rupnik, Tri Dung Nguyen, Stéphane Jacquemoud, and Yann Klinger. BRDF-NeRF: Neural radiance fields with optical satellite images and BRDF modelling. International Journal of Applied Earth Observation and Geoinformation, 143:104747, 2025.arxiv.org/abs/2409.12014

Paper details

vlnc:W_frctpzsz
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 4

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents