betaFeedback

WorksComputer Science

X-Splat: Unified Gaussian Splatting for Joint RGB and X-ray 3D Reconstruction

Abstract

Jointly reconstructing RGB and X-ray provides new insights by connecting external appearance with internal structure, enabling a more complete understanding of real-world objects than single modality. Yet existing methods train separate models per modality and suffer from redundant geometry, alignment artifacts, and loose coupling that limits cross-modal insight discovery. We propose a unified representation in which all Gaussian primitives share a single set of geometric parameters, with only surface-proximal primitives additionally carrying appearance attributes. We perform cross-modal registration through clean multi-view stereo and X-ray geometric proxies instead of aligning two sets of optimized Gaussians, improving alignment stability. We then jointly optimize the unified representation via supervision from both modalities, complemented by a surface-depth consistency loss derived from X-ray geometry to guide RGB reconstruction. Experiments on the InsideOut dataset and our Flower & Insect dataset show that our framework achieves more robust cross-modal alignment, higher RGB rendering quality without compromising X-ray reconstruction, and a more compact representation than maintaining two independent geometries.

CCS Concepts: Computing methodologies → Point-based models; Reconstruction.

Additional Key Words and Phrases: 3D Gaussian splatting, cross-modal 3D reconstruction, X-ray tomography, novel view synthesis

ACM Reference Format:

Wangze Xu, Chu Zhou, Mae Yamaguchi, Linrui Dai, I-Chao Shen, Takashi Ijiri, and Imari Sato. 2026. X-Splat: Unified Gaussian Splatting for Joint RGB and X-ray 3D Reconstruction. In SIGGRAPH Asia 2026 Conference Papers (SA Conference Papers ’26), December 01–04, 2026, Kuala Lumpur, Malaysia. ACM, New York, NY, USA, 19 pages. https://doi.org/10.1145/3829340.3842177

This work is licensed under a Creative Commons Attribution 4.0 International License. SA Conference Papers ’26, Kuala Lumpur, Malaysia

© 2026 Copyright held by the owner/author(s).

ACM ISBN 979-8-4007-2842-6/2026/12

https://doi.org/10.1145/3829340.3842177

Introduction

Real-world objects are both surface and interior: we readily see a flower’s outer petals, but not the structure within. Yet existing 3D reconstruction typically captures only one of the two at a time. RGB cameras record the external appearance through surface reflection, while X-ray imaging penetrates objects to reveal their internal volumetric structure through attenuation, leaving the two as separate digital records of the same physical object. We pursue a unified 3D representation that natively captures both, enabling consistent rendering of an object’s exterior and interior from any viewpoint (Figure 1). Such a unified representation supports applications such as digital cultural heritage [6], industrial inspection [3], and physically-grounded visualization [39], providing a more complete understanding of real objects than either modality alone.

Comparison of unified and single-modality Gaussian representations for RGB and X-ray reconstruction

Figure 1. We propose X-Splat, a unified representation where one set of Gaussian primitives carries both reflective (RGB) and absorptive (X-ray) attributes, simultaneously capturing external appearance and internal structure while enabling joint optimization. Single-modality specialists (3DGS [26], R²-Gaussian [64]) cover only one axis; the prior method InsideOut† [29] covers both modalities, but with lower RGB fidelity in our reproduction (§ 5.1). Ours achieves high fidelity on both from a single model. Each Ours/InsideOut panel shows three vertical strips at the same viewpoint: pure RGB / overlay / pure X-ray.

Existing 3D reconstruction work has developed each modality along a separate track. RGB-based approaches built on neural radiance fields (NeRF) [36] and Gaussian splatting [26] reconstruct external surfaces but treat the object interior as unobserved space. X-ray-based methods such as NAF [65], X-Gaussian [7], and R2R^{2}-Gaussian [64] recover internal density distributions but provide no surface appearance. Training separate models per modality is a natural starting point, yet such a decoupled approach keeps two independent representations of the same object with no cross-modal alignment, increasing parameter count and preventing consistent rendering of external appearance and internal structure.

A more principled direction is to design a single framework targeting RGB and X-ray jointly, and the recent InsideOut [29] is the first such attempt. It trains a separate Gaussian set per modality and bridges them through hierarchical alignment between the two optimized representations. While this design takes a first step toward joint reconstruction, it leaves several aspects underexplored. First, the two Gaussian sets independently model the same underlying geometry, doubling the parameter footprint. Second, the alignment operates on optimized Gaussian primitives that are the outcome of geometric and appearance fitting rather than a faithful description of the object’s surface, and contain floaters and optimization artifacts, leading to surface-to-interior misalignment that compromises applications requiring aligned internal-external display, such as digital cultural heritage [6]. Finally, although the scene geometry is accurately recovered from X-ray, it is still underutilized in RGB optimization, leaving RGB weakly supervised and limiting surface fidelity in applications such as industrial inspection [3].

To address these limitations, we present X-Splat, a unified Gaussian framework for joint RGB-X-ray 3D reconstruction. Our design is based on a simple observation: although RGB images arise from surface reflection and X-ray images from volumetric attenuation, both are ultimately conditioned on the geometry of the same physical object. This shared geometric origin suggests that the cleanest design is not to model each modality with its own independent geometry, but to let the two modalities share a single geometric representation while specializing only the attributes that genuinely differ between them. Concretely, we propose a unified Gaussian representation in which all primitives share a single set of geometric parameters, and only primitives close to the surface carry appearance attributes, reflecting the physical fact that RGB observations arise primarily from surface reflection. This eliminates the redundancy of maintaining two independent geometries and lets gradients from both modalities flow into the same shared geometry. To bring the two modalities into a common coordinate frame, we register them through clean geometric proxies derived directly from the input data, namely a multi-view stereo (MVS) [9, 18, 41, 58, 59] point cloud from the RGB views and a mesh extracted from the X-ray reconstruction, rather than aligning the already-optimized Gaussians. Finally, we exploit the shared geometry to enable joint optimization across both modalities, complemented by a surface-depth consistency loss using X-ray-derived surface geometry to supervise RGB rendering. In summary, our contributions are:

  • A unified Gaussian representation with shared geometric parameters and surface-restricted appearance attributes, enabling compact joint modeling of external appearance and internal volumetric structure.

  • A geometry-based alignment pipeline using clean MVS and X-ray proxies to register two modalities, avoiding the surface-to-interior misalignment that arises when registering two sets of already-optimized Gaussians.

  • A joint optimization scheme where both modalities update shared parameters via their respective objectives, complemented by an X-ray-derived surface-depth consistency loss.

We evaluate X-Splat on the InsideOut dataset [29] and a new Flower & Insect dataset combining existing flower scans [24] with newly captured insect specimens, broadening evaluation to natural objects. Our method achieves more robust cross-modal alignment, better RGB rendering quality without compromising X-ray reconstruction, and is more compact than two independent models.

Related Work

External Appearance Reconstruction. Neural radiance fields (NeRF) [36] have substantially advanced the appearance reconstruction of 3D objects from RGB images, which established a differentiable volume-rendering paradigm that fits a continuous radiance field to multi-view images. Subsequent work improved its rendering quality and efficiency through multi-scale anti-aliasing [2] and hash-grid encodings [37]. A parallel line replaces the volumetric formulation with neural implicit surfaces [52, 60] to recover sharper geometry. More recently, 3D Gaussian Splatting [26] represents the scene with explicit Gaussian primitives and a tile-based rasterizer, achieving real-time rendering at comparable visual quality, with Mip-Splatting [63] addressing aliasing under varying sampling rates. Subsequent work has improved surface fidelity by reformulating the primitives as 2D disks [23] or regularizing them toward the underlying surface [20]. Other lines extend to sparse-view [11, 30, 42, 50, 55, 68], pose-free [12, 15, 47, 51, 61], and dynamic [14, 22, 43, 54, 56, 58, 66] settings. While these methods achieve high-quality external appearance reconstruction, the internal structure of objects remains beyond their scope.

Internal Structure Reconstruction. X-ray and computed tomography (CT) reconstruction [17, 32, 62] recovers the internal density distribution of an object from projection measurements. Classical analytical methods such as FDK [13] remain the standard for cone-beam CT, producing volumetric reconstructions in closed form when projections are dense. To handle sparse-view settings, NAF [65] adapts a neural attenuation field to fit projection measurements with a continuous network representation, and SAX-NeRF [8] employs a line-segment Transformer to capture spatial structure from X-ray projections. More recently, Gaussian splatting has been extended to the X-ray modality. X-Gaussian [7] replaces alpha-compositing with a radiative rasterizer that follows the Beer-Lambert law, enabling efficient X-ray novel-view synthesis. R2-Gaussian [64] further corrects an integration bias in this rasterization formulation and introduces a differentiable voxelizer that enables accurate tomographic reconstruction. Subsequent work also explores direction-disentangled volume rendering for realistic radiographs [Gao et al. 2024]. While these methods achieve high-quality internal density reconstruction, external surface appearance lies beyond the X-ray modality.

Cross-modal 3D Reconstruction. 3D reconstruction has long benefited from auxiliary modalities captured by physical sensors that complement RGB cameras. Depth sensors such as LiDAR and commodity time-of-flight devices provide direct geometric measurements that supplement photometric supervision in driving [45, 48] and indoor scenes [25]. Thermal and infrared imaging extend RGB reconstruction to scenarios invisible to RGB cameras, with recent work introducing implicit neural fusion of far-infrared and RGB imagery [21, 31]. Multi-spectral and hyperspectral cameras likewise enrich RGB reconstruction with material- and wavelength-specific signals [35, 49]. Recent Gaussian-splatting frameworks further model RGB jointly with thermal images [34], or with multiple modalities through shared and modality-specific Gaussian components [19]. These auxiliary modalities all carry surface-domain signals, leaving the reconstructed target on the externally observable surface. Few works have attempted to combine external appearance with internal structure. Joining RGB and X-ray for 3D reconstruction differs fundamentally, since they arise from different physical mechanisms. Prior combinations of the two modalities operate at the image level in medical image fusion [38] or serve geometric calibration between the two sensing systems [1], and learned priors have been used to predict unseen object thickness from RGB-D observations [40]. None of these produces a unified 3D representation of appearance and internal structure. The most relevant prior work is InsideOut [29], which hierarchically fits two independently optimized Gaussian sets and bridges them by a post-hoc visual alignment. In contrast, we align via purely geometric registration and adopt a unified representation with shared geometric parameters, enabling truly joint optimization.

Preliminaries

Imaging Formation: Reflection vs. Transmission. RGB and X-ray imaging differ fundamentally in how light interacts with matter. In RGB imaging, most of the radiance received along a camera ray originates near the first surface it encounters on an opaque object, and the observed pixel intensity is modeled as the integral of emitted color along the ray, weighted by accumulated transmittance,

Irgb(r)=∫tntfT(t) α(r(t)) c(r(t)) dt,T(t)=exp⁡(−∫tntα(r(s)) ds),(1)I_{\mathrm{rgb}}(\mathbf{r}) = \int_{t_n}^{t_f} T(t)\,\alpha(\mathbf{r}(t))\,c(\mathbf{r}(t))\,\mathrm{d}t,\quad T(t) = \exp\left(-\int_{t_n}^{t}\alpha(\mathbf{r}(s))\,\mathrm{d}s\right), \tag*{(1)}

where α(⋅)\alpha(\cdot) and c(⋅)c(\cdot) are the volumetric opacity and color fields, tnt_n and tft_f are the near and far bounds of the ray, and T(t)T(t) is the transmittance from the camera to depth tt. In contrast, an X-ray penetrates the object along its entire path, and the log-domain pixel value is the line integral of the density field σ(⋅)\sigma(\cdot) along the ray, governed by the Beer-Lambert law,

Ixray(r)=∫tntfσ(r(t)) dt.(2)I_{\mathrm{xray}}(\mathbf{r}) = \int_{t_n}^{t_f}\sigma(\mathbf{r}(t))\,\mathrm{d}t. \tag*{(2)}

Despite these distinct optical mechanisms, both modalities observe the same underlying object, and each is a different forward operator acting on a common 3D geometry.

3D Gaussian Representation. 3D Gaussian splatting [26] models a scene as NN anisotropic 3D Gaussians, each with a center μi\mu_i and a covariance Σi=RiSiSiTRiT\Sigma_i = R_iS_iS_i^{\mathsf{T}}R_i^{\mathsf{T}} decomposed into a rotation RiR_i and a diagonal scale SiS_i. RGB and X-ray Gaussian splatting share these geometry parameters {μi,Ri,Si}i=1N\{\mu_i,R_i,S_i\}_{i=1}^{N} and differ in their per-primitive optical attributes and rendering operator.

RGB Gaussian Splatting. Each primitive additionally carries an opacity αi∈[0,1]\alpha_i \in[0,1] and a view-dependent color cic_i encoded by spherical harmonics [26]. Primitives are projected onto the image plane by EWA splatting [69], and the pixel color is obtained by depth-ordered front-to-back compositing of the projected 2D footprints,

Irgb(r)=∑i∈N(r)ciαi′∏j<i(1−αj′),(3)I_{\mathrm{rgb}}(\mathbf{r}) = \sum_{i\in\mathcal{N}(\mathbf{r})} c_i\alpha_i' \prod_{j<i}(1-\alpha_j'), \tag*{(3)}

where N(r)\mathcal{N}(\mathbf{r}) is the depth-ordered set of primitives intersected by ray r\mathbf{r} and αi′\alpha_i' is the EWA-projected effective opacity.

Radiative Gaussian Splatting. Each primitive instead carries a single scalar density ρi∈R+\rho_i \in\mathbb{R}_+, since X-ray attenuation depends only on isotropic material density. Integrating the resulting density field along the ray according to Eq. (2) yields, in closed form, a sum of 2D Gaussians on the detector [7, 64],

Ixray(r)=∑i=1Nρ^i N2D(x;μ^i,Σ^i),(4)I_{\mathrm{xray}}(\mathbf{r}) = \sum_{i=1}^{N}\hat{\rho}_i\,\mathcal{N}_{2\mathrm{D}}(\mathbf{x};\hat{\mu}_i,\hat{\Sigma}_i), \tag*{(4)}

where x^\hat{\mathbf{x}}, μ^i\hat{\mu}_i, Σ^i\hat{\Sigma}_i, and ρ^i\hat{\rho}_i are obtained by projective mapping [64]. Here N2D(⋅;μ^i,Σ^i)\mathcal{N}_{2\mathrm{D}}(\cdot;\hat{\mu}_i,\hat{\Sigma}_i) denotes the 2D Gaussian kernel of the ii-th projected primitive, and the sum runs over all NN primitives.

Method

Given calibrated RGB views {Ikrgb}k=1KC\{I_k^{\mathrm{rgb}}\}_{k=1}^{K_C} and X-ray projections {Ilxray}l=1KX\{I_l^{\mathrm{xray}}\}_{l=1}^{K_X} of the same physical object, our goal is to recover a single unified Gaussian set {Gi}i=1N\{\mathcal{G}_i\}_{i=1}^{N} that jointly supports high-fidelity RGB novel-view rendering and X-ray novel-projection rendering. As illustrated in Figure 2, a coarse X-ray reconstruction first anchors cross-modal alignment (§ 4.2) and initializes the unified Gaussian set, in which appearance attributes are restricted to a surface subset for compactness (§ 4.1); the unified set is then jointly optimized under both modalities, bridged by a surface-aware initialization (§ 4.3–§ 4.4).

Framework overview

Figure 2. Framework overview. A coarse X-ray reconstruction anchors cross-modal alignment (§ 4.2) and initializes the unified Gaussian set, in which each primitive carries shared geometry {μ,R,S}\{\mu, R, S\} and an X-ray attribute ρ\rho, with RGB appearance {α,c}\{\alpha, c\} restricted to a surface subset (§ 4.1). The unified set is jointly optimized under three losses (right): X-ray photometric, RGB photometric, and a CT-guided depth loss (§ 4.3).

Unified Gaussian Representation

We represent the object by a single set of NN Gaussian primitives {Gi}i=1N\{\mathcal{G}_i\}_{i=1}^{N}. Each primitive Gi\mathcal{G}_i carries shared geometry parameters {μi,Ri,Si}\{\mu_i, R_i, S_i\}, a radiative attribute given by a density ρi∈R+\rho_i \in\mathbb{R}_{+}, and, for primitives in a surface subset S⊂{1,…,N}S \subset\{1, \ldots, N\}, additional appearance attributes consisting of an opacity αi\alpha_i and a view-dependent color cic_i. The shared geometry parameters follow the parametrization in § 3 and are used by both renderers, while the radiative and appearance attributes are specific to X-ray and RGB rendering respectively. We write a primitive compactly as

Gi={μi,Ri,Si⏟shared geometry,ρi⏟radiative attribute,{αi,ci} if i∈S⏟appearance attributes}.(5)\mathcal{G}_i = \left\{ \underbrace{\mu_i, R_i, S_i}_{\text{shared geometry}},\quad\underbrace{\rho_i}_{\text{radiative attribute}},\quad\underbrace{\{\alpha_i,c_i\}\ \text{if } i \in S}_{\text{appearance attributes}} \right\}. \tag*{(5)}

This restriction has a physical motivation. Restricting appearance attributes to SS reflects the physics of the two modalities. For the opaque objects considered in this work, surface reflection dominates the RGB signal, and interior primitives contribute negligibly to any RGB pixel. Note that only the geometric parameters are shared across modalities; the optical attributes ρ\rho and {α,c}\{\alpha,c\} remain modality-specific. The construction of SS is detailed in § 4.3.

Sharing the geometry parameters {μi,Ri,Si}\{\mu_i, R_i, S_i\} across both modalities, rather than maintaining a separate Gaussian set per modality, is motivated by three considerations. Physically, RGB reflection and X-ray transmission are different forward operators acting on the same underlying object (§ 3), so a single shared geometry is more faithful than aligning two independently reconstructed geometries post-hoc. Compactness-wise, the shared design allocates {μ,R,S}\{\mu, R, S\} only once and {α,c}\{\alpha,c\} only for i∈Si \in S, yielding a substantially smaller parameter count than two separate single-modality reconstructions. Optimization-wise, gradients from both modalities update the shared geometry, anchoring it under X-ray supervision and compensating for the indirect geometric signal of RGB photometry alone.

Geometry-based Alignment

Joint optimization of the unified representation requires the RGB and X-ray modalities to reside in a common coordinate frame. We therefore seek a similarity transformation T∈Sim⁡(3)T \in\operatorname{Sim}(3) that maps

RGB camera poses into the X-ray coordinate frame. Unlike aligning two sets of optimized Gaussian primitives directly [29], which are corrupted by floaters and optimization artifacts, we align on clean geometric proxies extracted from each modality (Figure 3).

Geometry-based alignment

Figure 3. Geometry-based alignment. We extract surface point clouds from each modality independently (left): an X-ray surface Px\mathcal{P}^{\mathrm{x}} via Marching Cubes on the coarse X-ray 3DGS, and an MVS surface Pmvs\mathcal{P}^{\mathrm{mvs}} from multi-view RGB. PCA+ICP (right) registers the two surfaces into a shared frame.

For the RGB side, we run MVSFormer [9] on the calibrated RGB views and their camera parameters {Ikrgb,πkrgb}k=1Kc\{I_k^{\mathrm{rgb}}, \pi_k^{\mathrm{rgb}}\}_{k=1}^{K_c} to obtain an external surface point cloud Pmvs∈RNmvs×3\mathcal{P}^{\mathrm{mvs}} \in\mathbb{R}^{N_{\mathrm{mvs}}\times3}. For the X-ray side, we first run a coarse X-ray-only optimization (§ 4.3) to obtain a coarse Gaussian set Gcoarse\mathcal{G}^{\mathrm{coarse}}, voxelize it via the differentiable voxelizer of R2^2-Gaussian [64] into a CT volume, isolate the outer shell by binarization and morphological closing, extract the CT mesh via Marching Cubes [33], and uniformly sample its surface to obtain the point cloud Pct∈RNct×3\mathcal{P}^{\mathrm{ct}} \in\mathbb{R}^{N_{\mathrm{ct}}\times3}.

With these two surface point clouds, we estimate the transformation TT by registering Pmvs\mathcal{P}^{\mathrm{mvs}} against Pct\mathcal{P}^{\mathrm{ct}} in two steps. We first perform principal component analysis (PCA) on each point cloud’s positions and align the resulting centroids, principal axes, and variances to obtain an initial similarity transformation TpcaT^{\mathrm{pca}} matching translation, rotation, and scale. Starting from TpcaT^{\mathrm{pca}}, we then apply point-to-point iterative closest point (ICP) [4, 46] with scaling to obtain the final transformation TT. We apply TT to all RGB camera poses in homogeneous form,

Ekrgb→ct=EkrgbT−1,(6)E_k^{\mathrm{rgb}\to\mathrm{ct}} = E_k^{\mathrm{rgb}} T^{-1}, \tag*{(6)}

where EkrgbE_k^{\mathrm{rgb}} is the world-to-camera extrinsic of the kk-th RGB view.

Geometry-Constrained Joint Optimization

The two modalities provide complementary supervision, but X-ray imposes stronger constraints on the shared geometry than RGB. We exploit this asymmetry by establishing the shared geometry from X-ray first, then refining appearance attributes against this anchor.

Staged schedule. A coarse X-ray-only optimization suffices to recover a reliable shared geometry that benefits both the alignment in Sec. 4.2 and the subsequent RGB learning. We therefore split the optimization into two stages. In Stage 1, only the X-ray-related parameters {μi,Ri,Si,ρi}i=1N\{\mu_i, R_i, S_i, \rho_i\}_{i=1}^{N} are optimized under the X-ray photometric loss alone, starting from a standard FDK reconstruction [13, 64]. This produces the coarse X-ray Gaussian set Gcoarse\mathcal{G}^{\mathrm{coarse}} used in Sec. 4.2 for CT mesh extraction. In Stage 2, the appearance attributes {αi,ci}i∈S\{\alpha_i, c_i\}_{i\in S} are allocated and initialized as described below, the X-ray-related parameters {μi,Ri,Si,ρi}\{\mu_i, R_i, S_i, \rho_i\} are inherited from their Stage 1 values, and all parameters are optimized jointly. We retain the X-ray photometric loss throughout Stage 2 rather than freezing the X-ray side, since RGB gradients could otherwise drift the shared geometry to improve appearance at the cost of X-ray fidelity.

Surface-aware initialization. The surface subset SS introduced in Eq. (5) partitions the primitives into surface primitives, which carry both the radiative and appearance attributes, and interior primitives, which carry only the shared geometry parameters and the radiative attribute. This partition selectively allocates appearance attributes on top of a shared backbone, rather than splitting the primitives into RGB-only and X-ray-only sets. We construct SS from the CT mesh extracted during alignment to fully leverage the X-ray surface geometry as a prior for RGB optimization, and also to keep the representation compact by preventing interior primitives from participating in RGB rendering. Let did_i denote the distance from μi\mu_i to the CT mesh. The surface set is defined by thresholding,

S={i:di<τd},(7)S = \{i : d_i < \tau_d\}, \tag*{(7)}

where τd\tau_d is a distance threshold in the CT coordinate frame. For i∈Si \in S, the opacity is initialized as a Gaussian decay,

αi=αsurf⋅exp⁡(−di22σ2),(8)\alpha_i = \alpha_{\mathrm{surf}} \cdot\exp\left(-\frac{d_i^2}{2\sigma^2}\right), \tag*{(8)}

with αsurf\alpha_{\mathrm{surf}} a reference opacity and σ\sigma a decay scale proportional to τd\tau_d, so that primitives lying directly on the surface receive the largest initial opacity. The color cic_i is initialized from the color of the nearest point in Prms\mathcal{P}^{\mathrm{rms}}. The membership of SS is updated throughout training, with details deferred to Sec. 4.4.

X-ray-derived geometric constraints. The unified representation enables us to render an RGB surface depth DrgbD^{\mathrm{rgb}} and a CT reference depth DctD^{\mathrm{ct}} from a common viewpoint. Eq. (3) with the along-ray depth ziz_i in place of cic_i, restricted to i∈Si \in S; DctD^{\mathrm{ct}} is rasterized from the X-ray mesh, which is periodically re-extracted from the evolving X-ray volume during Stage 2. To exploit the X-ray geometric signal beyond the photometric term without forcing pixel-wise depth agreement, we apply an edge-aware smoothness loss in which

the CT depth supplies only edge structure as a stop-gradient target,

Lgeo=Er[1D‾ct∑a∈{x,y}exp⁡(−∣∇aDct∣/τ)∣∇aDrgb∣],(9)\mathcal{L}_{\mathrm{geo}} = \mathbb{E}_{\mathrm{r}}\left[\frac{1}{\overline{D}^{\mathrm{ct}}}\sum_{a\in\{x,y\}}\exp\left(-\lvert\nabla_a D^{\mathrm{ct}}\rvert/\tau\right)\lvert\nabla_a D^{\mathrm{rgb}}\rvert\right], \tag*{(9)}

where ∇a\nabla_a is the spatial depth gradient along image axis aa and Er\mathbb{E}_{\mathrm{r}} averages over valid foreground pixels, with τ\tau set to the median of in-mask CT gradient magnitudes and D‾ct\overline{D}^{\mathrm{ct}} the mean foreground CT depth as a scale normalizer. The loss permits RGB depth discontinuities only where the CT depth has edges, aligning the surface boundary structure of the two depth maps without constraining absolute depth values.

Overall objective. The Stage 1 objective combines the L1 plus D-SSIM X-ray photometric loss Lphotoxray\mathcal{L}^{\mathrm{xray}}_{\mathrm{photo}} with the 3D total-variation regularizer LTV\mathcal{L}_{\mathrm{TV}} of R2^2-Gaussian [64],

Lstage1=Lphotoxray+λTVLTV.(10)\mathcal{L}_{\mathrm{stage1}} = \mathcal{L}^{\mathrm{xray}}_{\mathrm{photo}} + \lambda_{\mathrm{TV}}\mathcal{L}_{\mathrm{TV}}. \tag*{(10)}

The Stage 2 objective adds an L1 plus D-SSIM RGB photometric loss Lphotorgb\mathcal{L}^{\mathrm{rgb}}_{\mathrm{photo}} and the geometric consistency loss Lgeo\mathcal{L}_{\mathrm{geo}},

Lstage2=λrgbLphotorgb+λxrayLphotoxray+λTVLTV+λgeoLgeo,(11)\mathcal{L}_{\mathrm{stage2}} = \lambda_{\mathrm{rgb}}\mathcal{L}^{\mathrm{rgb}}_{\mathrm{photo}} + \lambda_{\mathrm{xray}}\mathcal{L}^{\mathrm{xray}}_{\mathrm{photo}} + \lambda_{\mathrm{TV}}\mathcal{L}_{\mathrm{TV}} + \lambda_{\mathrm{geo}}\mathcal{L}_{\mathrm{geo}}, \tag*{(11)}

where the λ\lambda’s are scalar loss weights.

Adaptive Density Control

We extend the adaptive density control of 3DGS [26] to the unified setting. We maintain two independent position-gradient statistics per primitive, gˉirgb\bar{g}_i^{\mathrm{rgb}} and gˉixray\bar{g}_i^{\mathrm{xray}}, computed separately for each modality since the two modalities differ in backward frequency and loss scale. In Stage 2, the set of primitives selected for cloning or splitting is

D={Gi∣i∈{1,…,N}, gˉixray≥θxray∨(gˉirgb≥θrgb∧i∈S)},(12)\mathcal{D} = \left\{\mathcal{G}_i \mid i \in\{1,\ldots,N\},\ \bar{g}_i^{\mathrm{xray}} \ge\theta^{\mathrm{xray}} \lor\left(\bar{g}_i^{\mathrm{rgb}} \ge\theta^{\mathrm{rgb}} \land i \in S\right)\right\}, \tag*{(12)}

so that under-fitting in either modality drives the addition of new primitives, where θxray\theta^{\mathrm{xray}} and θrgb\theta^{\mathrm{rgb}} are the per-modality gradient densification thresholds. The RGB branch is further restricted to SS since interior primitives carry no appearance attributes; the X-ray branch is unrestricted because X-ray supervision legitimately requires primitives throughout the object interior. New primitives inherit the surface-set membership of their parent. For pruning, removing primitives based on low opacity alone is too aggressive, since a primitive may be invisible to RGB rendering yet carry meaningful X-ray density. We instead require both attributes to be low:

R={Gi∣i∈{1,…,N}, αi<αmin⁡∧ρi<ρmin⁡}.(13)\mathcal{R} = \left\{\mathcal{G}_i \mid i \in\{1,\ldots,N\},\ \alpha_i < \alpha_{\min} \land\rho_i < \rho_{\min}\right\}. \tag*{(13)}

where αmin⁡\alpha_{\min} and ρmin⁡\rho_{\min} are the minimum opacity and density thresholds. At each densification interval, primitives in SS whose opacity has fallen below αmin⁡\alpha_{\min} are demoted out of SS rather than pruned, so they no longer participate in RGB rendering but continue contributing to X-ray rendering. In Stage 1, only the X-ray-related criteria apply since no appearance attributes exist yet. The full training procedure is detailed in the supplementary material.

Experiments

Experimental Setup

Datasets. We evaluate our method on two datasets covering nine scenes. The first is the InsideOut dataset [29], which provides four scenes (Matryoshka, Pharaoh, Skull, and Toy Gun) with paired RGB photographs and ground-truth CT volumes. The second, which we refer to as the Flower & Insect dataset, contains two flower scenes (Tulip, Eustoma) from [24] and three insect scenes (E02, E15, E32) acquired from prepared specimens, all with paired CT volumes. To evaluate all baselines under identical X-ray acquisition geometry, we follow the protocol of R2R^2-Gaussian [64] and re-synthesize cone-beam projections from ground-truth CT volume using TIGRE [5], sampling 50 training and 120 testing projections uniformly over [0∘,360∘)[0^\circ, 360^\circ) at 512×512512 \times512 resolution. For RGB, we use the real photographs released by InsideOut [29], the Blender-rendered views of the real-specimen Flower assets [24], and our captured photographs of the prepared insect specimens, all downsampled to width 800. We hold out every fourth view for testing and train on the remainder. More details are in the supplementary material.

Metrics. For RGB, we report PSNR, SSIM [53], and LPIPS [67] on held-out test views inside per-view foreground masks. For X-ray, we follow R2R^2-Gaussian [64] and report 2D-PSNR/SSIM on rendered projections and 3D-PSNR between the reconstructed and ground-truth CT volumes. For the alignment, we report Chamfer Distance (CD) between the aligned MVS and X-ray surfaces as a direct measure of cross-modal alignment, in normalized scene units where each object’s longest extent equals 2. For compactness, we report the number of Gaussians (#G (k) in thousands) and storage (MB).

Implementation Details. We implement our framework on top of the R2R^2-Gaussian [64] codebase and reuse its X-ray training schedule for Stage 1. Stage 1 runs for 3,000 iterations from an FDK initialization [13], optimizing Lphotoxray\mathcal{L}_{\mathrm{photo}}^{\mathrm{xray}} together with the 3D total-variation regularizer at λtv=0.05\lambda_{\mathrm{tv}} = 0.05. After Stage 1 we extract the CT mesh, recover the geometry-based alignment (§ 4.2), and construct the surface set with surface distance threshold τd=0.02\tau_d = 0.02 and reference opacity αsurf=0.5\alpha_{\mathrm{surf}} = 0.5. Stage 2 then jointly optimizes the unified representation for 12,000 iterations under λrgb=λxray=1.0\lambda_{\mathrm{rgb}} = \lambda_{\mathrm{xray}} = 1.0, λtv=0.05\lambda_{\mathrm{tv}} = 0.05, and λgeo=0.05\lambda_{\mathrm{geo}} = 0.05. Each photometric loss combines L1\mathcal{L}_1 with D-SSIM at λDSSIMrgb=0.25\lambda_{\mathrm{DSSIM}}^{\mathrm{rgb}} = 0.25 and λDSSIMrgb=0.2\lambda_{\mathrm{DSSIM}}^{\mathrm{rgb}} = 0.2. The CT mesh is re-extracted every 500 iterations during Stage 2, and Lgeo\mathcal{L}_{\mathrm{geo}} is computed on 180 virtual cameras sampled by Fibonacci sphere. All other hyperparameters follow 3DGS defaults [26]; complete settings, learning rates, and densification thresholds are listed in the supplementary material. Training takes approximately 14–21 minutes per InsideOut scene on a single NVIDIA RTX A6000 GPU. As the official InsideOut [29] implementation is not publicly available, we reproduce their method by following the descriptions and hyperparameters reported in their paper. Specifically, InsideOut’s final X-ray Gaussians come from its X-ray-only stage 2 (matching R2R^2-Gaussian); we omit its subsequent X-ray re-refinement because the opacity-based rendering there is incompatible with our density-based renderer. We denote this reproduction by InsideOut†^\dagger in all tables and figures. The full reproduction setup and a discussion of discrepancies with its originally reported results are provided in the supplementary material.

Per-Modality Reconstruction Quality

Tables 1 and 2 report per-modality novel-view metrics on both datasets, with sub-tables (a) for RGB rendering and (b) for X-ray reconstruction. On the RGB side, Ours achieves consistently competitive metrics across all scenes on both datasets, improving over either 3DGS variant by around 1 dB PSNR on InsideOut and over 2 dB on Flower & Insect. Even FDK initialization, which already encodes the full volumetric structure, fails to outperform the MVS-init baseline, confirming that better initial geometry alone cannot break the single-modality ceiling without persistent geometric supervision. InsideOut†^\dagger [29] yields substantially lower RGB quality in our experiments (reproduction setup in § 5.1). On the X-ray side, our representation produces reconstruction quality consistent with R2R^2-Gaussian [64] across both datasets, with only minor fluctuations across individual metrics, while the classical FDK reconstruction degrades substantially compared to the Gaussian-based methods. The qualitative results shown in Figure 4 and Figure 5 suggest that our method recovers sharper surface texture and fewer floaters artifacts than all RGB baselines on scenes such as Matryoshka and Tulip, and our reconstructed CT slices and X-ray projections produce details closely matching R2R^2-Gaussian on scenes from both datasets. Joint training over a shared geometry therefore substantially improves RGB reconstruction quality without compromising X-ray reconstruction fidelity, validating the staged optimization of § 4.3.

Qualitative comparison of RGB novel-view synthesis

Figure 4. Qualitative comparison of RGB novel-view synthesis. Held-out test views across both datasets; yellow boxes mark zoom-in regions.

CT volume slices and X-ray projections for Skull, Tulip, and E32

Figure 5. CT volume slice and X-ray projection comparison. Left 4 columns: CT slices; right 4: X-ray projections. InsideOut is not shown separately, as its X-ray stage follows R²-Gaussian and coincides with that column.

Joint RGB-X-ray Reconstruction

Cross-Modal Alignment. Table 3 reports Chamfer Distance to the X-ray-reconstructed surface for our alignment pipeline and three representative alternatives, all applied to the same source point cloud and the same target view so that only the estimated similarity transformation TT varies. As shown by both Table 3 and Figure 7 (e.g., the residual misalignment on Skull), the InsideOut hierarchical alignment is less accurate than the other three methods. It operates on optimized Gaussian primitives whose floaters and overfit details propagate directly into the alignment. In contrast, aligning the clean MVS surface and CT mesh proxies with PCA alone already outperforms the InsideOut hierarchical alignment, and our full PCA+ICP pipeline improves Chamfer Distance by roughly 8×. The geometric cleanliness of the alignment source is therefore more decisive than the alignment algorithm itself. PCA provides global orientation and scale alignment, while ICP refines the result locally. PCA alone is insufficient for fine alignment, and identity-initialized ICP without a good PCA seed can fall into local minima and even collapse to near-degenerate scales on symmetric objects (e.g., E15 in Figure 7). The two stages are therefore complementary.

MethodInput SourceCD ↓
InsideOut† Hierarchical Alignment [29]Optimized Gaussian primitives0.0978
ICP OnlyGeometry-based (MVS + CT mesh)0.1453
PCA OnlyGeometry-based (MVS + CT mesh)0.0250
Ours (PCA + ICP)Geometry-based (MVS + CT mesh)0.0122

Table 3. Cross-modal alignment quality on InsideOut Dataset. The Input Source column indicates whether each method operates on optimized Gaussian primitives or on clean geometric proxies (MVS surface + CT mesh).

Cross-modal alignment comparison across both datasets

Figure 7. Cross-modal alignment comparison across both datasets. Aligned MVS surface (orange) overlaid on X-ray surface (blue); lower CD is better.

Joint Quality and Coherence. Table 4 aggregates RGB, X-ray, and compactness metrics for our unified representation, InsideOut, and an Independent baseline that trains our X-ray-only and RGB-only variants as two separate models with parameter counts summed. Ours outperforms Independent on RGB by a consistent margin and significantly surpasses InsideOut, while matching Independent on X-ray reconstruction quality. Ours also uses around 26% less storage than Independent. These savings come from sharing the geometric backbone and restricting appearance attributes to the surface set, while the RGB gain comes from joint X-ray supervision providing geometric constraints that RGB photometry alone cannot offer. Figure 6 visualizes this by rendering RGB and X-ray from a single

Unified RGB-X-ray rendering at the same viewpoint

Figure 6. Unified RGB-X-ray rendering at the same viewpoint. Each method’s cell shows overlay, RGB, X-ray, and an RGB error map at one test view.

MethodRGB Novel View PSNR↑RGB Novel View SSIM↑RGB Novel View LPIPS↓X-ray 2D-PSNR↑X-ray 3D-PSNR↑Compactness #G (K)Compactness Size (MB)
Independent (X-ray / RGB only)31.200.97160.033649.7638.25168.817.28
InsideOut† [29]21.220.92120.110749.7638.25166.312.75
Ours (unified)32.590.97650.029349.8138.16129.112.79

Table 4. Joint inside-outside reconstruction summary on InsideOut Dataset. The Independent baseline’s RGB-only model uses FDK initialization to match Ours’s init source, and its two model sizes are summed.

model at the same test RGB viewpoint, a setting inherently inaccessible to two independent models. On scenes such as Pharaoh, Tulip, and E15, our method produces sharp surface texture together with clear internal structure in one consistent rendering. The middle column, in which our alignment is replaced by the InsideOut hierarchical alignment (Ours + InsideOut Align), shows visible surface misalignment artifacts consistent with row (d) of Table 5, while the InsideOut column exhibits the RGB degradation already evident in the per-modality comparison. Consistently rendering the internal X-ray volume and the external RGB surface from a single model is the key advantage of our unified Gaussian representation. The supplementary video showcases these dynamic renderings.

ConfigurationCT depthSurf. setOurs align.Shared geom.RGB Novel View PSNR↑RGB Novel View SSIM↑RGB Novel View LPIPS↓X-ray 2D-PSNR↑X-ray 3D-PSNR↑Compactness #G (K)Compactness Size (MB)
(a). Ours (full joint)✓✓✓✓32.590.97650.029349.8138.16129.112.79
(b). w/o CT depth✓✓✓32.450.97520.029949.7538.12129.112.24
(c). w/o surface set✓✓✓32.410.97510.029749.6538.12129.113.19
(d). w/o Ours align.✓✓✓31.500.96670.038249.6538.18129.112.18
(e). Independent31.200.97160.033649.7638.25168.817.28

Table 5. Main ablation study on InsideOut Dataset. Components: CT depth, CT-depth consistency loss (Eq. (9)); Surf. set, surface-set mechanism (Eq. (7)); Ours align., geometry-based alignment (§ 4.2); Shared geom., unified Gaussian representation (§ 4.1).

Ablation Studies

Table 5 reports a cumulative ablation on the four InsideOut scenes. The four Components columns indicate which design elements are enabled in each configuration: rows (a)–(d) progressively disable individual components from left to right, while row (e) trains the single-modality variants of our framework as two separate models. CT-depth supervision and the surface-set mechanism both rely on an accurate cross-modal alignment, so they remain disabled once alignment is replaced in row (d).

CT-depth supervision (a →\to b). Disabling the CT-depth consistency loss reduces average RGB PSNR by around 0.16 dB. The loss anchors RGB surface to the X-ray-derived geometry, preventing surface primitives from drifting toward photometric optima that violate the geometric anchor; the resulting artifacts are visible in Figure 8(a).

Figure 8. (a) CT-depth loss ablation: removing the loss introduces visible depth artifacts. (b) Storage breakdown on Flower & Insect: Ours uses ∼2.2× less storage than Independent.

Surface-set mechanism (b →\to c). Disabling the surface-set mechanism preserves rendering quality but inflates storage by roughly 2.5×2.5 \times. Without the surface set, appearance attributes (spherical harmonics) are allocated to every primitive, including interior primitives that contribute nothing to RGB rendering of opaque objects, so these parameters are wasted.

Geometry-based alignment (c →\to d). We replace our geometry-based alignment with the InsideOut-style hierarchical alignment [29] on optimized Gaussians, then run our joint optimization on top of the resulting transformation. RGB quality drops by roughly 0.9 dB PSNR while X-ray quality remains stable, and the resulting surface misalignment artifacts are visible in the middle column of Figure 6. Our purely geometric alignment is therefore more robust, and benefits downstream joint optimization.

Unified representation (a vs. e). Compared with training fully separate single-modality models (row (e)), our unified representation achieves higher RGB rendering quality without compromising X-ray reconstruction. By sharing geometry, both modalities reinforce the same structure, benefiting RGB in particular, where photometry alone offers only indirect geometric signal. This gain is not recovered by simply freezing the CT reconstruction and using it as a depth prior for a separate RGB set (+0.06 dB over 3DGS; see the supplementary material). Combined with the storage savings noted above (with a larger ∼2.2× gap on Flower & Insect; Figure 8(b)), the unified design outperforms the Independent baseline on every measured metric. Finally, X-ray quality remains stable across rows (a)–(d), confirming that staged optimization decouples X-ray fidelity from RGB supervision.

Robustness. We further stress-test the framework under noisy, sparse, and limited-angle X-ray acquisition, alignment derived from degraded CT geometry, and sparse RGB supervision. The results in Sec. C of the supplementary material show that the framework remains robust across all these regimes. X-ray fidelity stays on par with the X-ray-only baseline, alignment and RGB quality are essentially unaffected, and the RGB gain from the shared CT geometry widens as views become sparser.

Conclusion

We presented a unified Gaussian framework for joint reconstruction of external surface appearance and internal volumetric structure. A single set of Gaussian primitives shares geometric parameters across both modalities, with appearance attributes restricted to surface-proximal primitives; cross-modal registration uses clean geometric proxies; and joint optimization is complemented by an X-ray-derived surface-depth consistency loss. Our design improves RGB rendering quality, yields more robust cross-modal alignment than post-hoc strategies, preserves X-ray fidelity, and is substantially more compact than two independent models.

Limitations and future work. First, our appearance model assumes that surface reflection dominates the RGB signal and therefore restricts appearance attributes to surface-proximal primitives. This assumption does not hold for transparent objects such as glass, or for materials with strong subsurface scattering or transmission (e.g., biological tissue or wax), whose appearance arises from light interacting with the object interior. Supporting such materials would require a volumetric appearance model, and paired RGB–X-ray datasets containing them are not yet available; we leave this to future work. Second, our framework depends on a reasonably accurate X-ray reconstruction to anchor the shared geometry, and may degrade when the X-ray inputs are extremely noisy or blurry. Third, current joint training improves RGB but matches the single-modality baseline on X-ray; extending RGB supervision to also benefit X-ray reconstruction is left as future work.

Acknowledgments

This work was supported by JST CRONOS JPMJCR25K6, JST-Mirai Program JPMJMI23G1, JSPS Grant-in-Aid JP23K16921, UTokyo-Google AI Symbiotic Future Society Program, Japan, Royal Society ISPF International Collaboration Award (ICA\R2\252130), and Advanced AI Talent Development to Lead the Next-Generation AI for Intelligent Society (BOOST NAIS) of The University of Tokyo. We thank Kenji Kohiyama for contributing the insect data.

References

Dataset Details

This appendix provides per-scene specifications and additional acquisition and rendering details that complement the Datasets paragraph in the main paper.

Per-scene Specifications

Table 6 reports per-scene RGB train and test view counts under our 1:3 split, the RGB resolution used at training and evaluation time, and the CT volume sizes used for X-ray synthesis. All scenes share the same X-ray sampling protocol (50 training and 120 testing projections of 512×512512 \times512 pixels, uniformly sampled over [0∘,360∘)[0^\circ, 360^\circ)).

Table 6. Per-scene dataset specifications. RGB Views (Tr/Te) lists train and test counts under our 1:3 split. RGB Resolution is the resolution used at training and evaluation (raw images are downsampled to a width of 800 pixels). CT Volume Size is the voxel grid at which we synthesize X-ray projections; for InsideOut it is the released volume downsampled to a uniform 2563256^3, while for Flower and Insect we keep the native voxel grid.

SourceSceneRGB Views (Tr / Te)RGB ResolutionCT Volume Size
InsideOut [29]Matryoshka39 / 14800×800800 \times800256×256×256256 \times256 \times256
InsideOut [29]Pharaoh49 / 17800×800800 \times800256×256×256256 \times256 \times256
InsideOut [29]Skull36 / 13800×800800 \times800256×256×256256 \times256 \times256
InsideOut [29]Toy Gun75 / 26800×800800 \times800256×256×256256 \times256 \times256
Flower [24]Eustoma60 / 20800×800800 \times800310×211×230310 \times211 \times230
Flower [24]Tulip60 / 20800×800800 \times800487×321×337487 \times321 \times337
InsectE0227 / 9800×600800 \times600427×704×797427 \times704 \times797
InsectE1527 / 9800×600800 \times600455×671×870455 \times671 \times870
InsectE3227 / 9800×600800 \times600453×668×868453 \times668 \times868

Table 6. Per-scene dataset specifications.

X-ray Synthesis Geometry

We use TIGRE [5] with cone-beam geometry to synthesize all X-ray projections, with the geometric parameters listed in Table 7. Each ground-truth CT volume is normalized so that its longest physical extent equals 2 (i.e., lies in [−1,+1][-1, +1]), with the remaining axes proportionally scaled to preserve the original aspect ratio; InsideOut volumes are cubic (2563256^3) and therefore fill the unit cube exactly, while Flower and Insect volumes occupy a proportionally smaller region along their shorter axes. The Insect scenes use a slightly larger detector (3.5×3.53.5 \times3.5 vs. 3.0×3.03.0 \times3.0 scene units) because their elongated bodies require a wider field of view. We use a clean (noise-free) synthesis setting to remain faithful to the released ground-truth volumes. Robustness under noisy, sparser, and limited-angle projections is evaluated in Sec. C.

ParameterInsideOut / FlowerInsect
Source-to-origin (DSO)5.05.0
Source-to-detector (DSD)7.07.0
Detector resolution512×512512 \times512512×512512 \times512
Detector physical extent3.0×3.03.0 \times3.03.5×3.53.5 \times3.5
Angular range[0∘,360∘)[0^\circ, 360^\circ)[0∘,360∘)[0^\circ, 360^\circ)
#Training / Testing views50 / 12050 / 120

Table 7. Cone-beam X-ray geometry used in TIGRE. Linear quantities are given in normalized scene units (the longest extent of each object equals 2).

RGB Capture and Rendering

InsideOut. We use the original RGB photographs released by [29]. Native resolutions vary by scene (from 1830×18301830 \times1830 to 2597×25972597 \times2597); for both training and evaluation, we downsample all views to 800×800800 \times800 pixels.

Flower. We render RGB views with Blender (800×800800 \times800, 512 samples per pixel) from the .BLEND assets distributed with [24]. Camera positions follow a smooth top-to-bottom spiral covering 5 revolutions across an elevation band of [−30∘,85∘][-30^\circ, 85^\circ], yielding 80 views per scene.

Insect. We capture 36 RGB photographs of each prepared insect specimen at native resolution 2048×15362048 \times1536 and downsample to 800×600800 \times600 for training and evaluation, preserving the 4:3 aspect ratio.

Foreground masks. Foreground masks are required by our RGB rendering loss and metric computation. For the Flower scenes, we directly use the alpha channel produced by Blender. For InsideOut and Insect, we annotate masks with X-AnyLabeling [CVHub520 and contributors 2024], an open-source labeling tool that integrates the Segment Anything Model [28, 44]; the SAM-predicted masks are then manually refined to remove spurious holes and over-segmented backgrounds. PSNR / SSIM / LPIPS for RGB are computed only inside the foreground mask.

Initial point cloud. As initialization for our unified Gaussian model, we run an MVS reconstruction (MVSFORMER [9]) on the training RGB views of each scene rather than the COLMAP sparse cloud used by 3DGS / InsideOut. The resulting dense point cloud is filtered against the per-view foreground masks (keeping only points seen inside the mask in at least one view), uniformly downsampled by Furthest Point Sampling (FPS), and used both as the alignment source (paired with the X-ray-derived surface) and as the color initialization for the RGB Gaussian splats.

Figure 9 previews the five Flower & Insect scenes used in our experiments. For each scene, we show two representative RGB photographs and two X-ray projections from different angles. The two Flower scenes (Tulip, Eustoma) are rendered from the .blend assets of [24]; the three Insect scenes (E02, E15, E32) are captured in our lab from prepared specimens (see Sec. A.3).

Flower and insect dataset gallery showing RGB photographs and X-ray projections

Figure 9. Flower & Insect dataset gallery. Each row shows one scene with two RGB photographs (left) and two X-ray projections (right) at different angles. X-ray projections are visualized with the viridis colormap.

Implementation Details

This appendix supplements the Implementation Details paragraph in the main paper with the full optimization procedure, complete optimizer settings, densification and pruning thresholds, and the remaining hyperparameters that the main paper does not list explicitly. We reuse the symbols defined in § 4.

Full Optimization Procedure

For ease of reproduction, Algorithm 1 consolidates the procedural flow described in § 4 into a single self-contained pseudocode listing. The algorithm is organized into the same four logical blocks as the main paper: a coarse X-ray pretraining (Stage 1, § 4.3), the cross-modal geometry-based alignment (§ 4.2), the surface-aware initialization that bridges the two stages (§ 4.3), and the joint optimization loop with periodic CT-mesh refresh (Stage 2, § 4.3–§ 4.4).

We reuse the symbols defined in § 4 and refer to the loss equations there rather than restating them. Concrete iteration budgets are N1=3,000N_1=3{,}000 and N2=12,000N_2=12{,}000, with mesh-refresh interval Tmesh=500T_{\mathrm{mesh}}=500 and virtual-camera count ∣Cgeo∣=180\lvert C^{\mathrm{geo}}\rvert=180; the remaining hyperparameters (τd\tau_d, σ\sigma, the loss weights, learning rates, and densification/pruning thresholds) are reported in the subsections below.

A few aspects of Algorithm 1 merit emphasis. (i) The X-ray photometric loss is retained throughout Stage 2 rather than frozen, so that RGB gradients cannot drift the shared geometry at the cost of X-ray fidelity. (ii) The geometric loss LgeoL_{\mathrm{geo}} is evaluated on a virtual viewpoint cgc_g sampled from a Fibonacci sphere CgeoC^{\mathrm{geo}}, decoupled from the training RGB view crc_r used for the photometric loss; this exposes the surface to a denser angular cover than the training cameras alone. (iii) The CT mesh MctM^{\mathrm{ct}} is re-extracted every TmeshT_{\mathrm{mesh}} iterations from the current GG, so DctD^{\mathrm{ct}} tracks the evolving shared geometry rather than freezing at the Stage 1 estimate. (iv) Densification is mode-aware: X-ray gradients can spawn primitives anywhere in the volume, while RGB gradients can only do so within the surface set SS, preserving the surface/interior separation introduced in Eq. (5).

Optimizer and Learning Rates

We optimize all parameters with Adam [27] (β1=0.9\beta_1=0.9, β2=0.999\beta_2=0.999). The view-dependent color cic_i is parametrized by spherical harmonics up to degree 3 (the 3DGS default). Per-attribute learning rates follow 3DGS / R2^2-Gaussian defaults (Table 8).

ParameterLearning rate (init → final)
Position μ\mu2×10−4→2×10−52\times10^{-4}\rightarrow2\times10^{-5}
Density ρ\rho1×10−2→1×10−31\times10^{-2}\rightarrow1\times10^{-3}
Scaling SS5×10−3→5×10−45\times10^{-3}\rightarrow5\times10^{-4}
Rotation RR1×10−3→1×10−41\times10^{-3}\rightarrow1\times10^{-4}
SH features cc2.5×10−32.5\times10^{-3} (constant)
Opacity α\alpha5×10−25\times10^{-2} (constant)

Table 8. Per-attribute learning rates. Position, density, scaling, and rotation rates are linearly decayed from their initial to final values over 30,000 steps.

Surface-aware Initialization

The decay scale in Eq. (8) is set to σ=0.2⋅τd=0.004\sigma=0.2\cdot\tau_d=0.004, so that for i∈Si\in S the initial opacity falls from αsurf=0.5\alpha_{\mathrm{surf}}=0.5 at di=0d_i=0 to a band-edge value of 1×10−31\times10^{-3} at di=τdd_i=\tau_d. Primitives outside SS receive a fixed interior opacity of 1×10−51\times10^{-5}.

Densification and Pruning

Densification follows the 3DGS schedule (interval = 100 iterations, active from iter 500 to iter 8,000). The position-gradient thresholds in Eq. (12) are θxray=5×10−5\theta^{\mathrm{xray}}=5\times10^{-5} (R2^2-Gaussian default) and θrgb=1×10−6\theta^{\mathrm{rgb}}=1\times10^{-6}. Pruning thresholds in Eq. (13) are αmin⁡=5×10−3\alpha_{\min}=5\times10^{-3} and ρmin⁡=1×10−5\rho_{\min}=1\times10^{-5}. Surface-set demotion uses the same opacity threshold αmin⁡\alpha_{\min}.

Alignment Pipeline

The Marching Cubes step [33] in Algorithm 1 uses an iso-level of 0.3 on the Stage 1 density volume. The subsequent ICP refinement uses a voxel-down-sampled correspondence radius of 0.01 and a maximum of 2,000 iterations.

Algorithm 1 Unified RGB-X-ray Gaussian Optimization

Require: RGB views {Ikrgb,πkrgb}k=1KC\{I_k^{\mathrm{rgb}}, \pi_k^{\mathrm{rgb}}\}_{k=1}^{K_C}; X-ray projections {Ilxray}l=1KX\{I_l^{\mathrm{xray}}\}_{l=1}^{K_X}; FDK initialization Ginit\mathcal{G}^{\mathrm{init}}; iteration budgets N1,N2N_1,N_2; mesh-refresh interval TmeshT_{\mathrm{mesh}}; surface threshold τd\tau_d.

Ensure: Unified Gaussian set G={Gi}i=1N\mathcal{G}=\{\mathcal{G}_i\}_{i=1}^{N} with shared geometry {μi,Ri,Si}\{\mu_i,R_i,S_i\}, density ρi\rho_i, and surface-restricted appearance {αi,ci}i∈S\{\alpha_i,c_i\}_{i\in\mathcal{S}}.

1: // Stage 1: X-ray Pretraining ▷ § 4.3 2: G←Ginit\mathcal{G}\leftarrow\mathcal{G}^{\mathrm{init}}; instantiate only {μi,Ri,Si,ρi}\{\mu_i,R_i,S_i,\rho_i\} 3: for iter = 1 to N1N_1 do 4: Sample X-ray view cxc_x from {Ilxray}\{I_l^{\mathrm{xray}}\} 5: Render I^xray(cx)\hat{I}^{\mathrm{xray}}(c_x) from G\mathcal{G} 6: Compute Lstage1\mathcal{L}_{\mathrm{stage1}} ▷ Eq. (10) 7: Backward; X-ray-driven densify & prune ▷ R2R^2-Gaussian protocol [64] 8: end for 9: Gcoarse←G\mathcal{G}^{\mathrm{coarse}}\leftarrow\mathcal{G} 10: // Geometry-based Alignment ▷ § 4.2 11: V←Voxelize⁡(Gcoarse)V\leftarrow\operatorname{Voxelize}(\mathcal{G}^{\mathrm{coarse}}); binarize, morphologically close, retain largest connected component 12: Mct←MarchingCubes⁡(V)\mathcal{M}^{\mathrm{ct}}\leftarrow\operatorname{MarchingCubes}(V); Pct←Sample⁡(Mct)\mathcal{P}^{\mathrm{ct}}\leftarrow\operatorname{Sample}(\mathcal{M}^{\mathrm{ct}}) ▷ X-ray surface proxy 13: Pmvs←MVSFormer⁡({Ikrgb,πkrgb})\mathcal{P}^{\mathrm{mvs}}\leftarrow\operatorname{MVSFormer}(\{I_k^{\mathrm{rgb}},\pi_k^{\mathrm{rgb}}\}) ▷ RGB surface proxy 14: Tpca←PCAAlign⁡(Pmvs,Pct)T^{\mathrm{pca}}\leftarrow\operatorname{PCAAlign}(\mathcal{P}^{\mathrm{mvs}},\mathcal{P}^{\mathrm{ct}}) 15: T←ICPsim3⁡(Pmvs,Pct;Tpca)T\leftarrow\operatorname{ICPsim3}(\mathcal{P}^{\mathrm{mvs}},\mathcal{P}^{\mathrm{ct}};T^{\mathrm{pca}}) 16: for k=1k=1 to KCK_C do 17: Ekrgb→ct←EkrgbT−1E_k^{\mathrm{rgb}\to\mathrm{ct}}\leftarrow E_k^{\mathrm{rgb}}T^{-1} ▷ Eq. (6) 18: end for 19: // Surface-aware Initialization ▷ § 4.3 20: Compute di←dist⁡(μi,Mct)d_i\leftarrow\operatorname{dist}(\mu_i,\mathcal{M}^{\mathrm{ct}}) for all ii 21: S←{i:di<τd}\mathcal{S}\leftarrow\{i:d_i<\tau_d\} ▷ Eq. (7) 22: for i∈Si\in\mathcal{S} do 23: αi←αsurf⋅exp⁡(−di2/2σ2)\alpha_i\leftarrow\alpha_{\mathrm{surf}}\cdot\exp(-d_i^2/2\sigma^2) ▷ Eq. (8) 24: ci←color⁡(NN⁡Pmvs(μi))c_i\leftarrow\operatorname{color}(\operatorname{NN}_{\mathcal{P}^{\mathrm{mvs}}}(\mu_i)) 25: end for 26: Cgeo←FibonacciSphere⁡(nvir;r=DSO,∣elev∣≤75∘)\mathcal{C}^{\mathrm{geo}}\leftarrow\operatorname{FibonacciSphere}(n_{\mathrm{vir}};r=\mathrm{DSO},|\mathrm{elev}|\le75^\circ) 27: // Stage 2: Joint Optimization ▷ § 4.3–§ 4.4 28: for iter = 1 to N2N_2 do 29: if iter mod Tmesh=1T_{\mathrm{mesh}}=1 then 30: Mct←MarchingCubes⁡(Voxelize⁡(G))\mathcal{M}^{\mathrm{ct}}\leftarrow\operatorname{MarchingCubes}(\operatorname{Voxelize}(\mathcal{G})) ▷ periodic refresh 31: end if 32: Sample X-ray view cxc_x, training RGB view crc_r, virtual view cg∼Cgeoc_g\sim\mathcal{C}^{\mathrm{geo}} 33: Render I^xray(cx)\hat{I}^{\mathrm{xray}}(c_x), I^rgb(cr)\hat{I}^{\mathrm{rgb}}(c_r), depths Drgb(cg)D^{\mathrm{rgb}}(c_g) ▷ swap ci→zic_i\to z_i in Eq. (3), restricted to S\mathcal{S} 34: Rasterize Dct(cg)D^{\mathrm{ct}}(c_g) from Mct\mathcal{M}^{\mathrm{ct}} 35: Compute Lphotoxray(cx)\mathcal{L}_{\mathrm{photo}}^{\mathrm{xray}}(c_x), Lphotorgb(cr)\mathcal{L}_{\mathrm{photo}}^{\mathrm{rgb}}(c_r), LTV\mathcal{L}_{\mathrm{TV}}, Lgeo(cg)\mathcal{L}_{\mathrm{geo}}(c_g) ▷ Eq. (9) 36: Compute Lstage2\mathcal{L}_{\mathrm{stage2}} ▷ Eq. (11) 37: Backward 38: Identify densification set D\mathcal{D} ▷ Eq. (12); RGB branch restricted to S\mathcal{S} 39: Identify pruning set R\mathcal{R} ▷ Eq. (13); both attributes low 40: Densify, prune, and demote S\mathcal{S}-members with αi<αmin⁡\alpha_i<\alpha_{\min} 41: Optimizer step 42: end for 43: return G\mathcal{G}

Geometric Constraint

Stage 2 mesh re-extraction reuses the same Marching Cubes settings as the alignment mesh in Sec. B.5. The virtual-camera sphere radius of Algorithm 1 resolves to 5.0 in scene units (matching the X-ray DSO; see Sec. A.2), and the depth maps DrgbD^{\mathrm{rgb}} and DctD^{\mathrm{ct}} used in Eq. (9) are rendered at half the X-ray detector resolution.

Hardware and Training Time

All experiments are run on a single NVIDIA RTX A6000 GPU (48 GB memory). For the InsideOut dataset, training time (Stage 1 X-ray pre-training + Stage 2 joint optimization, excluding evaluation) ranges from 14 to 21 minutes per scene, averaging 16 minutes (Table 9). The runtime correlates with the final number of Gaussian primitives, with Matryoshka taking the longest because of its layered nested-doll structure (~215k primitives at convergence) compared with ~50–150k for the other three scenes. Flower and Insect scenes use larger native voxel grids than InsideOut’s uniform 2563256^3 volumes and therefore incur a proportionally higher cost in X-ray rasterization; total per-scene runtimes are within the same order of magnitude.

SceneTraining time (min)
Matryoshka21.2
Pharaoh14.3
Skull14.7
Toy Gun14.0
Average16.0

Table 9.

Table 9. Per-scene training time on the InsideOut dataset (single NVIDIA RTX A6000). Total includes Stage 1 (3,000 X-ray pretraining iterations) and Stage 2 (12,000 joint optimization iterations); evaluation time is excluded.

InsideOut Reproduction

The official InsideOut implementation and its data preprocessing are not publicly available, so we reproduce the method from the descriptions and hyperparameters reported in its paper (Sec. 5.1). In addition, only part of its capture data is publicly released (e.g., 251 of 555 views for Matryoshka). The released captures span multiple physical placements of each object but carry no metadata relating the placements, so only the views from a single object placement can be registered into a consistent coordinate frame; the resulting per-scene view counts are listed in Table 6. Note that InsideOut reports RGB metrics only in its supplementary material, based on its full non-public captures. The discrepancy between those results and our reproduction in Table 1 therefore likely stems from the unreleased pipeline and the substantially smaller registerable view subset, and our reproduction may underestimate the method’s original performance.

Subject:MatryoshkaPharaohSkullToy GunAverage
(a) RGB Novel View Synthesis
Metric:PSNR↑SSIM↑LPIPS↓PSNR↑SSIM↑LPIPS↓PSNR↑SSIM↑LPIPS↓PSNR↑SSIM↑LPIPS↓PSNR↑SSIM↑LPIPS↓
3DGS (MVS init) [26]29.560.96430.039329.640.97190.029131.080.96700.047534.540.98710.013331.200.97260.0323
3DGS (FDK init) [26]29.750.96280.040829.510.97070.029830.970.96600.050134.550.98700.013631.200.97160.0336
InsideOut†[29]18.130.88360.152221.780.90390.092719.570.90030.149725.390.96090.048121.220.91220.1107
Ours30.370.96930.035430.890.97560.027232.420.97130.043336.680.99000.011432.590.97650.0293
(b) X-ray Reconstruction
Metric:2D-PSNR↑2D-SSIM↑3D-PSNR↑2D-PSNR↑2D-SSIM↑3D-PSNR↑2D-PSNR↑2D-SSIM↑3D-PSNR↑2D-PSNR↑2D-SSIM↑3D-PSNR↑2D-PSNR↑2D-SSIM↑3D-PSNR↑
FDK [13]31.930.901924.4742.920.982935.6531.840.799725.9128.770.931728.2533.860.904028.57
R²-Gaussian [64]48.200.994033.3552.330.998442.2849.560.996337.0948.960.996640.2849.760.996338.25
InsideOut† [29]48.200.994033.3552.330.998442.2849.560.996337.0948.960.996640.2849.760.996338.25
Ours48.340.994133.3952.510.998342.1349.720.996336.9048.690.996440.2549.810.996338.16

Table 1. Per-modality reconstruction quality on InsideOut Dataset. Best in bold, second in underline.

Robustness Analysis

We stress-test the framework under degraded inputs: noisy, sparse, and limited-angle X-ray acquisition and alignment robustness on the InsideOut dataset, and sparse RGB supervision on Flower & Insect. All settings follow the main-paper protocol unless noted.

Noisy X-ray projections. We add noise to the training X-ray projections following R2R^2-Gaussian [64], keeping the test projections clean (Table 10). X-ray 3D-PSNR degrades by only ~0.5 dB and identically for Ours and R2R^2-Gaussian, while RGB changes by at most 0.04 dB per scene. The joint design therefore adds no fragility relative to the X-ray-only backbone, and X-ray noise does not leak into the RGB channel.

MetricMethodMatryoshkaPharaohSkullToy GunAvg.
X-ray 3D-PSNR↑\uparrowR2R^2-Gaussian (clean)33.3542.2837.0940.2838.25
R2R^2-Gaussian (noisy)32.9241.9236.3939.9037.78
Ours (clean)33.3942.1336.9040.2538.17
Ours (noisy)32.9441.6836.3039.7837.68
RGB PSNR↑\uparrowOurs (clean)30.3730.8932.4236.6832.59
Ours (noisy)30.3530.8532.3836.6832.57

Table 10.

Table 10. Robustness to noisy X-ray projections on the InsideOut dataset.

Sparse X-ray projections.

We halve the training projections from the 50 used in our standard protocol to 25 (Table 11). Ours matches R2R^2-Gaussian across X-ray 2D-PSNR, 3D-PSNR, and surface CD, indicating that joint RGB supervision does not compromise X-ray under sparser acquisition.

MetricMethodMatryoshkaPharaohSkullToy GunAvg.
X-ray 2D-PSNR↑\uparrowR2R^2-Gaussian42.5751.2044.6845.6746.03
Ours42.6651.0544.5945.6545.99
X-ray 3D-PSNR↑\uparrowR2R^2-Gaussian28.6740.2333.0136.5434.61
Ours28.7439.7732.9636.5834.51
Surface CD↓\downarrowR2R^2-Gaussian0.00350.00280.00460.00280.0034
Ours0.00350.00290.00460.00280.0034

Table 11.

Table 11. X-ray reconstruction with 25 training projections (uniform over 360°) on the InsideOut dataset. Surface CD denotes the Chamfer distance to the ground-truth CT surface.

Limited-angle X-ray acquisition.

We confine the training projections to a 120∘120^\circ arc, so a 240∘240^\circ wedge is never observed by X-ray while the RGB views still cover the full 360∘360^\circ (Table 12). Ours remains on par with R2R^2-Gaussian across all metrics, both in-arc and out-of-arc, with only marginal differences in either direction. The full-coverage RGB supervision thus neither completes the missing wedge nor measurably degrades the reconstruction, and the X-ray channel remains self-sufficient under limited-angle acquisition.

MetricMethodMatryoshkaPharaohSkullToy GunAvg.
X-ray 2D-PSNR↑\uparrow (out-of-arc)R2R^2-Gaussian41.6346.3841.1940.8942.52
Ours41.6446.0340.9740.7942.36
X-ray 2D-PSNR↑\uparrow (in-arc)R2R^2-Gaussian49.8853.9651.6051.7151.79
Ours50.5154.7851.2351.5952.03
X-ray 3D-PSNR↑\uparrowR2R^2-Gaussian27.4536.6231.6334.8832.64
Ours27.3536.1831.5234.8832.48
Surface CD↓\downarrowR2R^2-Gaussian0.00370.00300.00500.00350.0038
Ours0.00380.00300.00490.00290.0036

Table 12.

Table 12. Limited-angle X-ray reconstruction on the InsideOut dataset. Training projections are confined to a 120∘120^\circ arc; X-ray 2D-PSNR is reported on test projections inside the arc (in-arc) and on the missing wedge (out-of-arc). The alignment is fixed to the clean full-view transformation.

Alignment under degraded X-ray geometry.

The limited-angle setting above also affects the coarse X-ray reconstruction, and hence the CT mesh used for alignment. We therefore re-derive the full PCA+ICP alignment from this limited-angle mesh instead of the clean full-view one (Table 13). RGB changes by at most 0.07 dB per scene, with X-ray 2D-PSNR and surface CD likewise unaffected, showing that the geometric-proxy alignment is not fragile to degraded X-ray geometry.

MetricAlignmentMatryoshkaPharaohSkullToy GunAvg.
RGB PSNR↑\uparrowfx (clean T)30.3630.8332.4236.6632.57
al (derived T)30.4330.8432.3736.6332.57
X-ray 2D-PSNR↑\uparrow (out-of-arc)fx (clean T)41.6446.0340.9740.7942.36
al (derived T)41.5946.0041.2040.8842.42
Surface CD↓\downarrowfx (clean T)0.00380.00300.00490.00290.0036
al (derived T)0.00380.00300.00500.00350.0038

Table 13.

Table 13. Alignment robustness under limited-angle X-ray on the InsideOut dataset. Both settings use our full model, with the alignment fixed to the clean full-view transformation (fx) or re-derived by PCA+ICP from the CT mesh of the 120∘120^\circ limited-angle reconstruction (al).

Sparse RGB views.

We reduce the RGB training views of each Flower & Insect scene from the original 27–60 (Table 6) to 15 (Table 14). Our gain over single-modality 3DGS widens from +2.72 dB at full views to +4.16 dB, peaking on the thinnest structures (Tulip +5.49, Eustoma +6.65 dB). These results indicate that the benefit of the X-ray modality to RGB appearance reconstruction becomes more pronounced under sparse RGB supervision, with the shared CT geometry helping most exactly where purely visual reconstruction is weakest.

MetricMethodTulipEustomaE02E15E32Avg.
PSNR↑\uparrow3DGS (15 views)28.1026.8129.7329.1330.1828.79
Ours (15 views)33.5933.4632.8632.1632.6932.95
Δ\Delta (Ours − 3DGS)+5.49+6.65+3.13+3.03+2.51+4.16
SSIM↑\uparrow3DGS (15 views)0.95960.96110.96440.96470.96910.9638
Ours (15 views)0.98070.98060.97730.97750.97800.9788
LPIPS↓\downarrow3DGS (15 views)0.06620.07350.03230.02870.02790.0457
Ours (15 views)0.03790.05000.02840.02680.02870.0344

Table 14.

Table 14. RGB reconstruction with 15 training views on the Flower & Insect dataset (X-ray unchanged). The 3DGS baseline uses FDK initialization, the same initialization source as Ours; the full-view gap is +2.72 dB (Table 2).

Subject:TulipEustomaE02E15E32Average
(a) RGB Novel View Synthesis
Metric:PSNR↑ SSIM↑ LPIPS↓PSNR↑ SSIM↑ LPIPS↓PSNR↑ SSIM↑ LPIPS↓PSNR↑ SSIM↑ LPIPS↓PSNR↑ SSIM↑ LPIPS↓PSNR↑ SSIM↑ LPIPS↓
3DGS (MVS init) [26]38.34 0.9888 0.032237.98 0.9883 0.041430.63 0.9743 0.028330.10 0.9749 0.024631.55 0.9793 0.023433.72 0.9811 0.0300
3DGS (FDK init) [26]37.85 0.9874 0.038737.74 0.9875 0.046630.88 0.9741 0.028930.11 0.9742 0.025531.53 0.9788 0.023933.62 0.9804 0.0327
InsideOut† [29]18.89 0.9148 0.126217.79 0.9378 0.118419.82 0.8809 0.143519.90 0.8748 0.134719.09 0.8534 0.161119.10 0.8923 0.1368
Ours39.71 0.9917 0.021939.22 0.9901 0.037034.75 0.9839 0.024633.74 0.9844 0.022334.77 0.9851 0.025336.44 0.9871 0.0262
(b) X-ray Reconstruction
Metric:2D-PSNR↑ 2D-SSIM↑ 3D-PSNR↑2D-PSNR↑ 2D-SSIM↑ 3D-PSNR↑2D-PSNR↑ 2D-SSIM↑ 3D-PSNR↑2D-PSNR↑ 2D-SSIM↑ 3D-PSNR↑2D-PSNR↑ 2D-SSIM↑ 3D-PSNR↑2D-PSNR↑ 2D-SSIM↑ 3D-PSNR↑
FDK [13]30.32 0.8487 21.9430.61 0.8542 27.6235.92 0.8996 28.8938.74 0.9514 31.1336.54 0.9146 28.7034.42 0.8937 27.66
R²-Gaussian [64]48.99 0.9958 31.7553.96 0.9976 40.3047.89 0.9978 35.0947.07 0.9973 35.2846.98 0.9975 34.1348.98 0.9972 35.31
InsideOut† [29]48.99 0.9958 31.7553.96 0.9976 40.3047.89 0.9978 35.0947.07 0.9973 35.2846.98 0.9975 34.1348.98 0.9972 35.31
Ours48.99 0.9957 31.7453.74 0.9974 40.1547.86 0.9978 35.1146.97 0.9973 35.2447.05 0.9975 34.1248.92 0.9971 35.27

Table 2. Per-modality reconstruction quality on Flower & Insect Dataset.

More Results

This appendix presents an additional quantitative comparison and qualitative results on additional scenes that complement the four main-paper qualitative figures: Figure 10 for CT slice and X-ray projection, Figure 11 for RGB novel-view synthesis, Figure 12 for unified RGB-X-ray overlay, and Figure 13 for cross-modal alignment.

Additional CT slice and X-ray projection results

Figure 10. Additional CT slice and X-ray projection results (supplemental to Figure 5). Six additional scenes with the same eight-column layout. Findings match the main figure: our unified model reproduces R2R^2-Gaussian’s fidelity on both modalities, while FDK shows the expected closed-form noise floor.

Additional RGB novel-view synthesis scenes comparing Ground Truth, 3DGS (MVS init), 3DGS (FDK init), InsideOut†, and Ours

Figure 11. Additional RGB novel-view synthesis scenes (supplemental to Fig. 4).

Additional unified RGB-X-ray showcase scenes

Figure 12. Additional unified RGB-X-ray showcase scenes (supplemental to Figure 6). Six additional scenes spanning both datasets, with the same three-panel cell layout. Findings match: Ours produces consistent inside/outside rendering across all scenes; Ours + IO-align retains the unified representation but with mild surface-alignment artifacts; our InsideOut reproduction shows consistently lower RGB quality, consistent with the main-paper results.

Additional alignment-comparison scenes

Figure 13. Additional alignment-comparison scenes (supplemental to Figure 7).

Joint optimization vs. a frozen CT prior.

We evaluate the alternative of reconstructing CT once, freezing it, and using it only as a depth prior for a completely separate RGB Gaussian set, keeping the same frozen Stage-1 CT mesh, alignment, and initialization as Ours. As shown in Table 15, this variant is essentially identical to the single-modality 3DGS baseline (+0.06 dB) and 1.33 dB below our joint model. A frozen CT prior therefore recovers almost none of the joint gain, which stems from jointly refining the shared geometry under both modalities’ supervision.

MethodMatryoshkaPharaohSkullToy GunAvg.
3DGS (MVS init)29.5629.6431.0834.5431.20
Separate RGB set + frozen CT prior29.7729.6331.0634.5931.26
Ours (joint)30.3730.8932.4236.6832.59

Table 15. RGB PSNR of a separate RGB Gaussian set with a frozen CT depth prior on the InsideOut dataset. The 3DGS and Ours rows are from Table 1.

References

  1. [1]Francisco Albiol, Alberto Corbi, and Alberto Albiol. 2016. Geometrical calibration of X-ray imaging with RGB cameras for 3D reconstruction. IEEE transactions on medical imaging 35, 8 (2016), 1952–1961.DOI
  2. [2]Jonathan T Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla, and Pratul P Srinivasan. 2021. Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. In Proceedings of the IEEE/CVF international conference on computer vision. 5855–5864.arxiv.org/abs/2103.13415
  3. [3]Paul Bergmann, Michael Fauser, David Sattlegger, and Carsten Steger. 2019. MVTec AD–A comprehensive real-world dataset for unsupervised anomaly detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 9592–9600.DOI
  4. [4]Paul J Besl and Neil D McKay. 1992. Method for registration of 3-D shapes. In Sensor fusion IV: control paradigms and data structures, Vol. 1611. Spie, 586–606.DOI
  5. [5]Ander Biguri, Manjit Dosanjh, Steven Hancock, and Manuherch Soleimani. 2016. TIGRE: a MATLAB-GPU toolbox for CBCT image reconstruction. Biomedical Physics & Engineering Express 2, 5 (2016), 055010.DOI
  6. [6]Francien G Bossema, Willem Jan Palenstijn, Arlen Heginbotham, Madeline Corona, Tristan van Leeuwen, Robert van Liere, Jan Dorscheid, Daniel O’Flynn, Joanne Dyer, Erma Hermens, et al. 2024. Enabling 3D CT-scanning of cultural heritage objects using only in-house 2D X-ray equipment in museums. Nature Communications 15, 1 (2024), 3939.
  7. [7]Yuanhao Cai, Yixun Liang, Jiahao Wang, Angtian Wang, Yulun Zhang, Xiaokang Yang, Zongwei Zhou, and Alan Yuille. 2024a. Radiative gaussian splatting for efficient x-ray novel view synthesis. In European Conference on Computer Vision. Springer, 283–299.
  8. [8]Yuanhao Cai, Jiahao Wang, Alan Yuille, Zongwei Zhou, and Angtian Wang. 2024b. Structure-aware sparse-view x-ray 3d reconstruction. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 11174–11183.
  9. [9]Chenjie Cao, Xinlin Ren, and Yanwei Fu. 2022. Mvformer: Learning robust image representations via transformers and temperature-based depth for multi-view stereo. arXiv preprint arXiv:2208.02541 5, 6 (2022).
  10. [11]Kangle Deng, Andrew Liu, Jun-Yan Zhu, and Deva Ramanan. 2022. Depth-supervised nerf: Fewer views and faster training for free. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 12882–12891.arxiv.org/abs/2107.02791
  11. [12]Zhiwen Fan, Wenyuan Cong, Kairun Wen, Kevin Wang, Jian Zhang, Xinghao Ding, Danfei Xu, Boris Ivanovic, Marco Pavone, Georgios Pavlakos, et al. 2024. InstantSplat: Sparse-view gaussian splatting in seconds. arXiv preprint arXiv:2403.20309 (2024).arxiv.org/abs/2403.20309
  12. [13]Lee A Feldkamp, Lloyd C Davis, and James Wress. 1984. Practical cone-beam algorithm. Journal of the Optical Society of America A 1, 6 (1984), 612–619.DOI
  13. [14]Sara Fridovich-Keil, Giacomo Meanti, Frederik Hahbæk Warburg, Benjamin Recht, and Angjoo Kanazawa. 2023. K-planes: Explicit radiance fields in space, time, and appearance. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 12479–12488.
  14. [15]Yang Fu, Sifei Liu, Amey Kulkarni, Jan Kautz, Alexei A Efros, and Xiaolong Wang. 2024. Comapr-free 3d gaussian splatting. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 20796–20805.
  15. [17]Richard Gordon, Robert Bender, and Gabor T Herman. 1970. Algebraic reconstruction techniques (ART) for three-dimensional electron microscopy and X-ray photography. Journal of theoretical Biology 29, 3 (1970), 471–481.pubmed.ncbi.nlm.nih.gov/5492997
  16. [18]Xiaodong Gu, Zhiwen Fan, Siyu Zhu, Zuoxhuo Dai, Feitong Tan, and Ping Tan. 2020. Cascade cost volume for high-resolution multi-view stereo and stereo matching. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2495–2504.arxiv.org/abs/1912.06378
  17. [19]Zhifeng Gu and Bing Wang. 2025. Mmone: Representing multiple modalities in one scene. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 1088–1098.arxiv.org/abs/2507.11129
  18. [20]Antoine Guédon and Vincent Lepetit. 2024. Sugar: Surface-aligned gaussian splatting for efficient 3d mesh reconstruction and high-quality mesh rendering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 5354–5363.arxiv.org/abs/2311.12775
  19. [21]Mariam Hassan, Florent Forest, Olga Fink, and Malcolm Mielle. 2024. Thermonerf: Multimodal neural radiance fields for thermal novel view synthesis. arXiv preprint arXiv:2403.12154 2, 7 (2024).arxiv.org/abs/2403.12154
  20. [22]Shoukan Hu, Tao Hu, and Ziwei Liu. 2024. Gaumhan: Articulated gaussian splatting from monocular human videos. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 20418–20431.
  21. [23]Binbin Huang, Zehao Yu, Anpei Chen, Andreas Geiger, and Shenghua Gao. 2024. 2d gaussian splatting for geometrically accurate radiance fields. In ACM SIGGRAPH 2024 conference papers. 1–11.DOI
  22. [24]Takashi Ijiri, Shin Yoshizawa, Hideo Yokota, and Takeo Igarashi. 2014. Flower modeling via X-ray computed tomography. ACM Transactions on Graphics (TOG) 33, 4 (2014), 1–10.DOI
  23. [25]Nikhil Keetha, Jay Karhade, Krishna Murthy Jatavallabhula, Gengshan Yang, Sebastian Scherer, Deva Ramanan, and Jonathan Liuten. 2024. Splatam: Splat track & map 3d gaussians for dense rgb-d slam. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 21357–21366.arxiv.org/abs/2312.02126
  24. [26]Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, George Drettakis, et al. 2023. 3d gaussian splatting for real-time radiance field rendering. ACM Trans. Graph. 42, 4 (2023), 139–1.arxiv.org/abs/2308.04079
  25. [27]Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014).arxiv.org/abs/1412.6980
  26. [28]Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tate Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. 2023. Segment anything. In Proceedings of the IEEE/CVF international conference on computer vision. 4015–4026.arxiv.org/abs/2304.02643
  27. [29]Jungmin Lee, Seonghyuk Hong, Juyoung Lee, Jaeyoon Lee, and Jongwon Choi. 2025. Indoor3dgs: 3d gaussian splatting for comprehensive 3d object representation. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 25820–25830.
  28. [30]Jiahe Li, Jiawei Zhang, Xiao Bai, Jin Zheng, Xin Ning, Jun Zhou, and Lin Gu. 2024b. Dngaussian: Optimizing sparse-view 3d gaussian radiance fields with global-local depth normalization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 20775–20785.arxiv.org/abs/2403.06912
  29. [31]Xiangjie Li, Shuxiang Xie, Ken Sakurada, Ryusuke Sagawa, and Takeshi Oishi. 2024a. Implicit neural fusion of rgb and far-infrared 3d imagery for invisible scenes. In 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 12501–12508.DOI
  30. [32]Yingtai Li, Xueming Fu, Shang Zhao, Ruyijang Jin, and S Kevin Zhou. 2023. Sparse-view ct reconstruction with 3d gaussian volumetric representation. arXiv e-prints (2023), arXiv–2312.DOI
  31. [33]William E Lorensen and Harvey E Cline. 1998. Marching cubes: A high resolution 3D surface construction algorithm. In Seminal graphics: pioneering efforts that shaped the field. 347–353.
  32. [34]Rongfeng Lu, Hangyu Chen, Zunjie Zhu, Yuhang Qin, Ming Lu, Le Zhang, Chenggang Yan, and Anke Xue. 2025. ThermalGaussian: Thermal 3D Gaussian Splatting. In International Conference on Learning Representations.arxiv.org/abs/2409.07200
  33. [35]Lukas Meyer, Josef Grün, Maximilian Weiherer, Bernhard Egger, Marc Stamnninger, and Linus Franke. 2025. Multi-Spectral Gaussian Splatting with Neural Color Representation. In Computer Graphics Forum. Wiley Online Library, e70337.arxiv.org/abs/2506.03407
  34. [36]Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. 2021. NeRF: Representing scenes as neural radiance fields for view synthesis. Commun. ACM 65, 1 (2021), 99–106.arxiv.org/abs/2003.08934
  35. [37]Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. 2022. Instant neural graphics primitives with a multiresolution hash encoding. ACM transactions on graphics (TOG) 41, 4 (2022), 1–15.DOI
  36. [38]Rekha R Nair and Tripty Singh. 2018. Multi-sensor, multi-modal medical image fusion for color images: A multi-resolution approach. In 2018 Tenth international conference on advanced computing (ICoAC). IEEE, 249–254.DOI
  37. [39]Junfeng Ni, Yixin Chen, Bohan Jing, Nan Jiang, Bin Wang, Bo Dai, Puhao Li, Yixin Zhu, Song-Chun Zhu, and Siyuan Huang. 2024. Physically plausible neural scene reconstruction. Advances in Neural Information Processing Systems 37 (2024), 25747–25780.
  38. [40]Andrea Nicastro, Ronald Clark, and Stefan Leutenegger. 2019. X-section: Cross-section prediction for enhanced RGB-D fusion. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 1517–1526.
  39. [41]Rui Peng, Rongjie Wang, Zhenyu Wang, Yawen Lai, and Ronggang Wang. 2022. Rethinking depth estimation for multi-view stereo: A unified representation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 8645–8654.arxiv.org/abs/2201.01501
  40. [42]Rui Peng, Wangze Xu, Luyang Tang, Liew Liao, Jianbo Jiao, and Ronggang Wang. 2024. Structure consistent gaussian splatting with matching prior for few-shot novel view synthesis. Advances in neural information processing systems 37 (2024), 97328–97352.arxiv.org/abs/2411.03637
  41. [43]Zhiyin Qian, Shaofei Wang, Marko Mihajlovic, Andreas Geiger, and Siyu Tang. 2024. 3dgs-avatar: Animatable avatars via deformable 3d gaussian splatting. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 5020–5030.arxiv.org/abs/2312.09228
  42. [44]Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, et al. 2024. Sam 2: Segment anything in images and videos. arXiv preprint arXiv:2408.00714 (2024).
  43. [45]Konstantinos Rematas, Andrew Liu, Pratul P Srinivasan, Jonathan T Barron, Andrea Tagliasacchi, Thomas Funkhouser, and Vittorio Ferrari. 2022. Urban radiance fields. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 12932–12942.arxiv.org/abs/2111.14643
  44. [46]Szymon Rusinkiewicz and Marc Levoy. 2001. Efficient variants of the ICP algorithm. In Proceedings third international conference on 3-D digital imaging and modeling. IEEE, 145–152. Wangze Xu, Chu Zhou, Mae Yamaguchi, Linrui Dai, I-Chao Shen, Takashi Ijiri, and Imari SatoDOI
  45. [47]Shihe Shen, Huachen Gao, Wangze Xu, Rui Peng, Luyang Tang, Kaiqiang Xiong, Jianbo Jiao, and Ronggang Wang. 2024. Disentangled generation and aggregation for robust radiance fields. In European Conference on Computer Vision. Springer, 218–236.arxiv.org/abs/2409.15715
  46. [48]Shanlin Sun, Bingbing Zhuang, Ziyu Jiang, Buyu Liu, Xiaohui Xie, and Manmohan Chandraker. 2024. LidarF: Delving into lidar for neural radiance field on street scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 19563–19572.arxiv.org/abs/2405.00900
  47. [49]Christopher Thirgood, Oscar Mendez, Erin Ling, Jon Storey, and Simon Hadfield. 2025. Hypers: Hypergraph 3d gaussian splatting. In Proceedings of the Computer Vision and Pattern Recognition Conference. 5970–5979.
  48. [50]Guangcong Wang, Zhaoxi Chen, Chen Change Loy, and Ziwei Liu. 2023. Sparserf: Distilling depth ranking for few-shot novel view synthesis. In Proceedings of the IEEE/CVF international conference on computer vision. 9065–9076.arxiv.org/abs/2303.16196
  49. [51]Jianyuan Wang, Minghao Chen, Nikita Karaev, Andrea Vedaldi, Christian Rupprecht, and David Novotny. 2025. Vggt: Visual geometry grounded transformer. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 5294–5306.arxiv.org/abs/2503.11651
  50. [52]Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. 2021. NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction. Advances in Neural Information Processing Systems 34 (2021), 27171–27183.arxiv.org/abs/2106.10689
  51. [53]Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. 2004. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing 13, 4 (2004), 600–612.pubmed.ncbi.nlm.nih.gov/15376593
  52. [54]Guanjun Wu, Taoran Yi, Jiemin Fang, Lingxi Xie, Xiaopeng Zhang, Wei Wei, Wenyu Liu, Qi Tian, and Xingang Wang. 2024. 4d gaussian splatting for real-time dynamic scene rendering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 20310–20320.arxiv.org/abs/2310.08528
  53. [55]Wangze Xu, Huachen Gao, Shihe Shen, Rui Peng, Jianbo Jiao, and Ronggang Wang. 2024. Mvs: Excavating multi-view priors for gaussian splatting from sparse input views. In European Conference on Computer Vision. Springer, 203–220.arxiv.org/abs/2409.14316
  54. [56]Wangze Xu, Yifan Zhan, Zhibang Zhong, and Ziwei Zuo. 2025. Sequential gaussian avatars with hierarchical motion context. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 13592–13603.arxiv.org/abs/2411.16768
  55. [58]Ziyi Yang, Xinyu Gao, Wen Zhou, Shaohui Jiao, Yuqing Zhang, and Xiaogang Jin. 2024. Deformable 3d gaussians for high-fidelity monocular dynamic scene reconstruction. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 20331–20341.arxiv.org/abs/2309.13101
  56. [59]Yao Yao, Zixin Luo, Shiwei Li, Tian Fang, and Long Quan. 2018. Mvsnet: Depth inference for unstructured multi-view stereo. In Proceedings of the European conference on computer vision (ECCV). 767–783.arxiv.org/abs/1804.02505
  57. [60]Lior Yariv, Jiatao Guo, Yoni Kasten, and Yaron Lipman. 2021. Volume rendering of neural implicit surfaces. Advances in neural information processing systems 34 (2021), 4805–4815.arxiv.org/abs/2106.12052
  58. [61]Botao Ye, Sifei Liu, Haofei Xu, Xueting Li, Marc Pollefeys, Ming-Hsuan Yang, and Songyou Peng. 2025. No pose, no problem: Surprisingly simple 3d gaussian splats from sparse unposed images. In International Conference on Learning Representations.
  59. [62]Weihai Yu, Yuanhao Cai, Ruvi Zha, Zhiwen Fan, Chenxin Li, and Yixuan Yuan. 2025. X2-gaussian: 4d radiative gaussian splatting for continuous-time tomographic reconstruction. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 24728–24738.
  60. [63]Zehao Yu, Anpei Chen, Binbin Huang, Torsten Sattler, and Andreas Geiger. 2024. Mip-splatting: Alias-free 3d gaussian splatting. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 19447–19456.arxiv.org/abs/2311.16493
  61. [64]Ruyi Zha, Tao Jun Lin, Yuanhao Cai, Jiwen Cao, Yanhao Zhang, and Hongdong Li. 2024. R²-Gaussian: Rectifying Radiative Gaussian Splatting for Tomographic Reconstruction. Advances in Neural Information Processing Systems 37 (2024), 44907–44934.arxiv.org/abs/2405.20693
  62. [65]Ruyi Zha, Yanhao Zhang, and Hongdong Li. 2022. NAF: neural attenuation fields for sparse-view CBCT reconstruction. In International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, 442–452.DOI
  63. [66]Yifan Zhan, Wangze Xu, Qingtian Zhu, Muyang Niu, Mingze Ma, Yifei Liu, Zhihang Zhong, Xiao Sun, and Yinqiang Zheng. 2025. R3-avatar: Record and retrieve temporal codebook for reconstructing photorealistic human avatars. arXiv preprint arXiv:2503.12751 (2025).arxiv.org/abs/2503.12751
  64. [67]Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. 2018. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition. 586–595.arxiv.org/abs/1801.03924
  65. [68]Zehao Zhu, Zhiwen Fan, Yifan Jiang, and Zhangyang Wang. 2024. Fsgs: Real-time few-shot view synthesis using gaussian splatting. In European conference on computer vision. Springer, 145–163.arxiv.org/abs/2312.00451
  66. [69]Matthias Zwicker, Hanspeter Pfister, Jeroen Van Baar, and Markus Gross. 2001. Surface splatting. In Proceedings of the 28th annual conference on Computer graphics and interactive techniques. 371–378.

Paper details

vlnc:W_rbxhzw36
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 10

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents