WorksPhysics and Astronomy
Spectro-Spatial Survey Fusion
Abstract
Imaging and spectroscopy constrain complementary information about the same sky but are usually processed separately. We introduce Spectro-Spatial Survey Fusion, a framework that unifies these measurements by fitting a continuous spectral surface-brightness field jointly through each instrument's response. Adapting from Gaussian splatting in computer vision, we represent the field as a sum of adaptive spatial Gaussians, each carrying a spectrum built from a shared, learned dictionary of templates. We demonstrate this on 400 Sloan Digital Sky Survey (SDSS) exposures and 800 fiber spectra across an region of COSMOS. The joint fit closely reproduces the spatial structure and broad spectral diversity of galaxies, stars, and quasars. For fitted SDSS observations, the summed predicted flux in source-selected pixels agrees with the data to within 2 percent in every band, and 96 percent of fitted spectra agree to within 10 percent in integrated flux. Sparse fiber spectra constrain a dictionary shared by every Gaussian component, so a model spectrum can be queried anywhere in the imaged area, including where no spectrum was fitted. Without refitting, the field predicts withheld SDSS images and spectra, and selected Hyper Suprime-Cam (HSC) images and Dark Energy Spectroscopic Instrument (DESI) spectra, which never enter the fit. The field predicts broad spiral-arm patterns in HSC images of two selected galaxies and DESI continuum shapes for galaxies without a fitted fiber. This framework provides a route to combining surveys so that spatial and spectral information from different observations and instruments can constrain a common representation of the sky.
Introduction
Astronomical surveys provide complementary views of the same sky. Wide-field imaging programs, including the Dark Energy Spectroscopic Instrument (DESI) Legacy Imaging Surveys, Hyper Suprime-Cam (HSC), Rubin, Euclid, and Roman, differ in depth, angular resolution, and wavelength coverage ([12] (Dey et al. 2019; Aihara et al. 2022; Ivezić et al. 2019; Spergel et al. 2015)). Multiplexed fiber spectrographs such as DESI, 4MOST, WEAVE, and PFS add detailed spectra for selected sources ([7, 47] (Jin et al. 2024; Takada et al. 2014)). Combining these data requires reconciling their instrument responses as well as their source catalogs. Coaddition combines exposures on a common pixel grid within a band (Fruchter & Hook 2002; Zackay & Ofek 2017a,b), but the result retains a particular bandpass and effective point-spread function (PSF), and cannot directly incorporate a fiber spectrum. Matching catalogs has a different limitation: neighboring objects often overlap in deep images, so catalogs constructed independently at different resolutions need not agree on which sources are present (Bosch et al. 2018; Melchior et al. 2018). Joint processing at the pixel level allows information from one survey to help constrain measurements in another (Chary et al. 2020). Fiber spectroscopy samples only a fraction of the imaged sources, and each fiber collects a seeing-dependent portion of its target’s light, potentially mixed with emission from neighbors. Fiber spectra are therefore already tied to imaging through models: target selection uses fiber fluxes predicted from imaging (Hahn et al. 2023), and aperture corrections use imaging to extend star-formation rates measured within a fiber to the whole galaxy (Brinchmann et al. 2004). Slitless spectroscopy, as in the Euclid and Roman surveys, designs, records dispersed images whose overlaps must be modeled in both position and wavelength. Combining separately derived survey products does not, by itself, ensure that a single sky model can account for all the underlying measurements.
We instead model the measurements themselves. Under a static-sky approximation, calibrated image pixels and spectral bins can both be modeled as projections of the same underlying quantity: the spectral surface brightness of the sky , a function of sky position and wavelength . An image pixel integrates against a bandpass throughput, a PSF, and a pixel footprint; a fiber spectral bin integrates the same field against a narrow wavelength window, a line-spread function (LSF), and the fiber aperture. Rather than combining data products after the fact, we fit a continuous model of directly to image pixels and spectral bins simultaneously, pushing the model through each instrument’s own forward operator. Each additional instrument then contributes further constraints on the same latent field, which can be queried through different bandpasses, PSFs, apertures, and wavelength grids within the represented domain.
Fitting a common model through different instrument responses is already established in astronomical forward modeling. The Tractor and The Farmer connect photometric measurements across surveys through source models (Lang et al. 2016; Weaver et al. 2023), while scarlet combines morphology and color across bands to separate overlapping sources (Melchior et al. 2018). Joint image reconstruction can account for different PSFs and pixel scales, allowing observations at different resolutions to constrain the same scene (Joseph et al. 2021; Donath et al. 2024). Hyperspectral fusion reconstructs a latent spatio-spectral cube from imaging and integral-field unit (IFU) spectroscopy covering the same region (Guilloteau et al. 2020). Marquis et al. (2026) demonstrate this with James Webb Space Telescope (JWST) observations of a protoplanetary disk and Titan, over fields of and , respectively. In these fusion studies, the integral-field cube samples spectra across the whole reconstructed region. Survey spectroscopy instead places sparse fibers across far larger imaged areas, so a joint model must carry spectral information from the fibers to positions where only images exist.
In computer vision, novel-view synthesis methods fit a scene representation through differentiable rendering, then predict images from new viewpoints. Neural radiance fields represent a scene with a multi-layer perceptron over continuous coordinates (Mildenhall et al. 2020), while Gaussian splatting uses explicit Gaussian components whose positions, shapes, and number are adapted during optimization by adaptive density control (Kerbl et al. 2023). Radiance fields with joint image registration (Lin et al. 2021) have also been used to coadd simulated astronomical images (Hutton et al. 2022). In astronomy, Gaussian sums have long been used to model galaxy light distributions (Cappellari 2002) and to speed up the rendering of standard galaxy profiles (Hogg & Lang 2013). Explicit Gaussians also suit astronomical forward modeling: when the PSF is represented by a Gaussian sum, convolution reduces to combining Gaussian parameters, while affine transformations preserve the Gaussian form when mapping between sky and detector coordinates.
In this work we introduce Spectro-Spatial Survey Fusion, a practical framework that combines heterogeneous astronomical observations by fitting a single field that is continuous in position and wavelength. We represent the field as a sum of anisotropic spatial Gaussians, each carrying a spectrum assembled from a shared dictionary. The component parameters and spectral dictionary are learned jointly, with growth and pruning adapting the Gaussian population to describe source morphology freely without imposing a fixed profile for each object. Each image pixel or spectral bin is predicted through its instrument response, requiring the same spatial and spectral field to account for all fitted measurements. Once fitted, this field can be queried at different positions and wavelengths or through new instrument responses without refitting.
We demonstrate the framework on an region of the COSMOS field using 400 Sloan Digital Sky Survey (SDSS) exposures and 800 fiber spectra. We refer to this joint reconstruction as the ‘canonical fit’. We first assess how closely the field reproduces the fitted observations, then predict 89 withheld fiber spectra and ten withheld image exposures, without refitting. External imaging from HSC and spectroscopy from DESI provide a generalization test of the field’s predictions through instruments that supplied no data to the fit. This framework points toward a common model of the sky that can be refined as observations from additional surveys are incorporated. Jointly fitting observations from several surveys would allow spatial and spectral information resolved by one instrument to constrain what remains uncertain in another, and allow a representation of the sky that contains more information than any single survey could provide.
The paper is organized as follows: § 2 defines the underlying representation and observation operators, and § 3 describes the data and fitting procedure. § 4 and § 5 present the fitted field and its predictions for new measurements, respectively. § 6 discusses their implications and limitations, and § 7 concludes.
Representation and Operators
A common measurement model
We represent the sky’s latent emission by a single, time-independent field. Because the fitted region spans only , we describe sky positions by flat-sky coordinates about the field center given in § 3.1, . Let be the fitted region in these coordinates, and let be the modeled range of observed wavelength. The field
gives the spectral surface brightness at position and observed wavelength before atmospheric and instrumental blurring, as a flux density per unit wavelength and solid angle. Across the field, the east–west scale of these coordinates differs from the true angular scale by at most 0.06 percent.
Each calibrated image pixel or spectral bin is a datum. We index data by , where contains all image pixels and spectral bins, and write for the image exposure or fiber spectrum that contains datum . Images and fiber spectra constrain the same field through their respective observation operators. Let be the calibrated measurement after subtracting the survey-supplied background estimate . We write the general-purpose measurement model as
where is the observation operator that maps the field into the instrument-defined units and sampling of datum , is the predicted source flux, and is measurement noise. The survey background is subtracted from the measurements before fitting and held fixed; it also enters the image noise model (§ 3.3). Errors in this survey-supplied estimate can bias the inferred field or remain in the residuals if not addressed during data cleaning before fitting (§ 6). Appendix § 10 summarizes the notation used throughout.
Figure 1 conceptually illustrates these two kinds of measurements from the same galaxy. Both measurements draw from the same underlying field that is continuous in position and wavelength . An image pixel collects light from a small spatial region across a bandpass that is broad in wavelength, represented by the blue column extending across the wavelength axis. In the same latent field, a spectral bin instead collects light across the fiber aperture with fine spectral resolution but coarse spatial resolution, represented by the pink disk. Together, these measurements constrain the spatial and spectral structure of the same field. The observation operators include the bandpass weighting, spatial and spectral blurring, and flux calibration needed to predict these measurements.
Figure 1. A schematic diagram of an SDSS image pixel and a fiber spectral bin for the same galaxy. The blue rectangular column spans the band at the position of one pixel. The pink disk spans the fiber aperture in a single 1.4 Å bin. Their sizes are not to scale, but emphasize the small spatial footprint and broad-in-wavelength bandpass of an image pixel, compared with the larger spatial aperture and narrow wavelength interval of a spectral bin. The right panels show the two measurements in the observed image and spectrum.
Gaussian components and a shared spectral dictionary
We represent as a sum of spatial Gaussians, each carrying a spectrum formed from a shared spectral dictionary. Gaussians retain their form under the local affine transformations that are used to map the field’s continuous sky coordinates onto each exposure’s pixel grid. Convolving two Gaussians adds their covariance matrices, so a Gaussian sum remains a sum of Gaussians after convolution with a Gaussian-mixture PSF, and we evaluate the PSF-convolved components analytically in the image operator (§ 2.4). Fiber-aperture integration, however, still uses numerical quadrature (§ 2.5). Here, the components’ positions and shapes are fitted across the field, allowing several Gaussians to combine into a source without imposing explicit shape priors or fixed morphological templates.
We express the sky’s spectral surface brightness as a sum of Gaussian components:
where is the component amplitude and is its spectrum. Each is a two-dimensional Gaussian normalized to unit integral, with center and symmetric positive-definite covariance . Two positive standard deviations along its principal axes and a position angle determine its size, elongation, and orientation. The components provide a basis for the light distribution and need not individually correspond to physical objects. The left panels of Figure 2 show a conceptual illustration of how an increasing number of Gaussians allows a Gaussian sum to describe more spatial structure. Our field is fitted jointly to images and spectra, with an evolving Gaussian component lifecycle governed by a modified version of the density-control prescription from Gaussian splatting (Kerbl et al. 2023), as described in § 3.4.
Figure 2. Schematic of the spatial and spectral components of the field’s underlying representation. On the left, sums of 1, 4, 16, and 64 Gaussians capture progressively finer structure in an observed galaxy. The ellipses below mark the contours of the individual components. The right panel shows the 32 spectral templates learned from the full field, ordered by peak wavelength and each normalized to its own peak. Each component learns its own mixture of these 32 templates.
To allow for shared information across the field, we represent the component spectra with a shared dictionary of non-negative templates,
where the coefficients and shift are specific to component . Each component has one learned shift in log wavelength, applied to all template contributions, so the same template shapes can be used at different wavelengths. These shifts are free per component and are intended to make the spectral representation more flexible.
The templates share a grid of 96 knots spanning 2980–11230 Å. The knots are uniformly spaced in log wavelength, and template values are interpolated linearly in wavelength between them. Template values outside the stored interval are set to zero. The amplitudes , mixture coefficients , and template knot values are kept non-negative through softplus parameterizations. The value of alone does not give the component’s flux, as there is a normalization degeneracy among , the mixture coefficients, and the templates. We therefore interpret predicted observables and the combined light of components rather than individual normalizations. The right panel of Figure 2 shows the dictionary learned by the canonical fit (§ 4.3), with templates normalized to their peaks and ordered by peak wavelength for display. Different component spectra allow the model to represent color variations within a source. Each Gaussian has one position and shape shared across wavelengths, while its brightness varies with wavelength.
The dictionary is learned jointly with the component parameters from the fitted observations and their instrument responses, rather than supplied as a fixed library of physical spectra. Individual templates need not correspond to any physical source type. There is no external pretraining corpus or source catalog that informs the model about where to allocate spectral or spatial capacity. Spectra within fitted fibers can therefore constrain template shapes also used by components elsewhere in the field, while those components retain their own positions, shapes, mixtures, and spectral shifts. This shared basis allows information from spectroscopic measurements to affect predictions away from their original fitted apertures. § 5.1.1 tests how including spectra in the fit changes predictions for withheld spectra.
Rendering the field through an instrument operator
We predict the source light in each pixel or spectral bin by summing the contributions from the field’s Gaussian components. We evaluate these contributions at the observed pixel positions and spectral sampling, separating their responses by modality. We also approximate the spatial PSF as independent of wavelength within an image band or an individual fiber spectrum. The predictions below give
in Equation (2). Within the modeled sky region and wavelength range, the same fitted field can predict measurements from other instruments without refitting, provided their response and flux convention are adequately modeled. We demonstrate this with zero-shot HSC imaging and DESI spectroscopy in § 5.2.
Imaging
For an image datum in exposure and band , we first compute the band average of each component’s spectrum using the system throughput . We then combine this with the PSF-convolved spatial profile to predict the source surface brightness at the pixel center :
where is the position of the pixel center and is the normalized, pixel-convolved PSF of exposure at that detector position, expressed in the field’s coordinates. Here denotes convolution in position. The factor of gives the photon-counting weight. The denominator defines a band-averaged flux density, matching the conversion of the calibrated images to surface-brightness units. Because the PSF is taken to be achromatic within the band, the same spatial response acts on the whole band amplitude. We average each shifted template over the full bandpass, treating its contribution as zero at wavelengths outside the template’s range.
Each exposure’s survey-supplied world-coordinate system maps its pixel centers to sky coordinates, which we then convert to the flat-sky coordinates of § 2.1. Local affine transformations carry the component covariances into detector coordinates, with the Jacobian determinant providing the conversion between pixel area and sky area needed to preserve each Gaussian’s total flux. We represent the spatially varying SDSS PSF by fitting signed Gaussian expansions of its supplied eigenimages, combined with the survey’s position-dependent coefficients. This gives 40 Gaussian terms at each pixel. The SDSS PSF eigenimages already account for integration over the detector pixels, so we fit this sampled response directly and evaluate the convolved components at pixel centers without integrating over the pixel footprint again.1 Convolution is analytic for these Gaussian expansions, while the PSF representation and local astrometric linearization remain approximations.
Fiber spectroscopy
A fiber spectrum combines light across a circular aperture before resolving it in wavelength. For spectrum , let be the disk centered on the recorded fiber position, with radius . The SDSS and Baryon Oscillation Spectroscopic Survey (BOSS) spectra fibers (Smee et al. 2013) have diameters of 3″ and 2″, respectively. We convolve each spatial component with a circular Gaussian PSF , using the spectroscopic seeing rather than the imaging seeing. Integrating this profile over by numerical quadrature gives , the fraction of component ’s light collected by the fiber. Convolving its spectrum with the LSF and averaging over the wavelength interval of spectral bin gives :
where is the bin width. Here is the component spectrum’s convolution in wavelength with the LSF. We model spectral blurring with a Gaussian LSF , normalized so that its integral over wavelength is one. We determine the LSF width from the wavelength-dependent instrumental resolution supplied with each spectrum, treating it as constant within each bin. We approximate the recorded samples as finite-bin averages. The convolution and bin integral can be evaluated analytically for the piecewise-linear component spectra. The LSF and bin widths are of order a few angstroms, much smaller than the template knot spacing, which grows in proportion to wavelength (e.g. 84 Å at 6000 Å).
The aperture integral alone is not the flux reported in the SDSS spectra. Their calibration uses standard stars and is normalized to point-source PSF magnitudes.2 We approximate this convention by dividing the collected source light by the fraction of a centered point source admitted under the same Gaussian seeing. We denote this point-source fraction by and use it to normalize the predicted spectral flux density of bin in spectrum :
where is the standard deviation of the Gaussian seeing profile, and is its full width at half maximum (FWHM). For a centered point source, dividing by recovers its total flux under the assumed PSF model. For an extended source, the same correction does not generally recover all the light missed by the fiber. The predicted spectrum therefore remains dependent on the aperture’s size and position through the collected fractions . The approximation uses one wavelength-independent seeing value and a fixed aperture center for each spectrum. It does not reproduce non-Gaussian PSF wings, wavelength-dependent seeing, atmospheric differential refraction, or the full coaddition and calibration procedure.
We retain the native SDSS imaging calibration, which approximates the AB system, and convert the spectra to the same flux scale before fitting. We construct a fixed, wavelength-dependent conversion from the spectroscopic pipeline’s five per-band calibration offsets and the filter throughput curves, and apply it consistently to the spectral fluxes, sky estimates, and uncertainties.
Data and Fitting Procedure
Observations
We apply the framework to an 84′ × 84′ region of the COSMOS field centered at in ICRS, using SDSS imaging and fiber spectroscopy (York et al. 2000). The imaging comprises 82 SDSS fields observed in bandpasses, giving 410 exposures obtained between May 2000 and February 2001. We use cutouts from the individual exposures at their native sampling of 0.396″ per pixel, retaining each exposure’s astrometry and PSF in its observation operator. We use the SDSS-provided frame masks (fpM) to exclude interpolated pixels and those flagged for saturation, cosmic rays, or ghosts from the fit and image statistics.
The spectroscopic sample comprises 889 spectra of galaxies, quasars, and stars observed between December 2000 and March 2011: 557 taken with the SDSS spectrograph (York et al. 2000; Rockosi et al. 2022) and 332 with the BOSS spectrograph (Dawson et al. 2013). Each SDSS spectrum combines several exposures of the same target. We retain bins with finite fluxes and wavelengths, and positive variance. We exclude a bin if the pipeline flagged it in any of those exposures, even when the other exposures were unflagged. Each retained spectrum must also have imaging coverage.
Figure 3 shows the spatial distribution of the fitted and withheld measurements (Section 3.2). In the left panel, shading counts how many of the 400 fitted imaging exposures cover each position, summed over all five bands; darker regions have more overlapping coverage. The red dashed outlines mark the footprints of the ten withheld exposures, with labels identifying the withheld band or bands. Fitted images in other bands, and same-band overlaps from other runs or camera columns, still constrain the sky inside these outlines. In the right panel, filled gray symbols locate the 800 fitted spectra and open red symbols the 89 withheld spectra. Circles, squares, and triangles distinguish galaxies, quasars, and stars, respectively.
Figure 3. SDSS imaging and spectroscopy coverage across the COSMOS field. The shading on the left shows the total number of fitted exposures covering each position. Dashed red outlines mark the ten withheld exposures, labeled by band. On the right, filled gray symbols mark the 800 fitted spectra and open red symbols the 89 withheld spectra. Circles mark galaxies, squares mark quasars, and triangles mark stars.
We fit the field directly to the measured image and spectral fluxes, without using a pre-existing object catalog to fix the source models. Components are seeded from peaks detected in the fitted images (Section 3.4). § 5.2 presents zero-shot experiments with HSC PDR3 imaging (Aihara et al. 2022) and DESI DR1 spectra DESI Collaboration et al. [8] of the same region, and neither dataset contributes to the initialization or fitting of the field and its shared spectral dictionary. Part of the SDSS data is withheld from the fitting objective, as described in § 3.2.
Fitted and withheld measurements
We assess predictions of withheld SDSS measurements using the fixed split shown in red in Figure 3. Ten complete image exposures, two per band, and 89 fiber spectra are omitted from the fitting objective. The remaining 400 exposures and 800 spectra constrain the field. The fitted spectra comprise 433 galaxies, 109 quasars, and 258 stars, while the withheld spectra comprise 41 galaxies, 23 quasars, and 25 stars. Spectral observations were assigned to the proximity groups, linking target positions separated by less than 10″, before withholding whole groups.
The information available within a withheld image depends on the coverage of the fitted exposures. Consecutive SDSS fields share 128 rows of the same detector readout. When both copies of a shared detector sample would enter the fit, we retain only one. If either copy is withheld, through either a whole-exposure omission or the random pixel selection described below, both copies are excluded from the fitting objective. Thus no withheld detector sample enters the objective through a neighboring field. Overlapping observations from different runs or camera columns are retained as separate measurements. For nine of the ten withheld exposures, 86–87 percent of the valid pixels have no fitted same-band coverage after the overlap exclusions and frame masking. For the tenth, the fraction is 12 percent, because most of its area within the field is also imaged by fitted exposures from another run. We evaluate the different coverage regions separately (Section 5.1.2).
In addition to the whole-observation exclusions, two percent of individual image pixels and spectral bins are omitted from the objective, serving as development diagnostics. The image-peak initialization uses the full source images of the fitted exposures, including pixels later excluded from the objective at random, by overlap deduplication, or by the SDSS frame masks, so these randomly omitted image pixels are not an independent test. The withheld spectra and exposures are rendered through their respective operators and presented in § 5.1 as a generalization test.
We assess both fitted and withheld observations by measuring the residuals relative to the uncertainties derived from the observed data. This puts discrepancies in image pixels and spectral bins on a common noise scale. For each datum , we compare the background-subtracted measurement with its prediction (Equation 2), and let be its observational uncertainty. For image pixels, we calculate from the observed source-plus-sky counts and the survey’s gain, read-noise, and dark-current parameters. For spectral bins, we derive from the pipeline inverse variances and rescale it with the fluxes as described in § 2.5. These same uncertainties define the source-pixel selection below. We define the pull of each datum and the mean squared pull (MSP) of a set of data as
where is, for example, the source-selected pixels of one exposure or the bins of one spectrum. The MSP is the chi-square per datum of that set.
In the images, the large number of pixels with little source light can dilute the residuals around bright sources in an average over the whole exposure. We therefore report statistics separately for source-selected pixels, with , and for the remaining pixels, which we call sky pixels even though they may contain faint source light. We calculate these statistics on the original pixels of each exposure, before resampling the images for display. As in the fit, valid pixels have finite fluxes and uncertainties with and pass the SDSS frame-mask selection in § 3.1. For fitted images, we pool pixels within the stated band or region using the same exclusions as the objective, including duplicate samples and the randomly excluded pixels. For withheld images, we use valid pixels within each stated coverage region.
For the spectra, we compute statistics for each observation using all bins retained by the selection in § 3.1, including the randomly excluded bins within fitted spectra. Medians over these per-spectrum statistics summarize the population with each spectrum given equal weight. We also report a pooled MSP over all retained bins, in which spectra with more bins contribute more to the result.
A residual that is small compared with the source flux can still be large compared with the measurement uncertainty. We therefore also report predicted-to-observed flux ratios, summing over the stated image pixels or integrating over retained spectral bins with their individual wavelength widths. Masked wavelength gaps contribute no flux to these integrals. Agreement in summed flux can hide an excess in one pixel that cancels a deficit in another. For source-selected image pixels, we also measure the fractional absolute residual, , to quantify the discrepancy without this cancellation.
The objective
We combine the fitted image pixels and spectral bins in a common objective, weighting their squared residuals inversely by their noise variances. Let denote the component parameters and shared spectral dictionary, and let and contain the image pixels and spectral bins that enter the fitting objective after the exclusions in § 3.2. Using and the predictions , the parameter-dependent part of the objective at update is
where is the image variance used for that update and is the spectral variance derived from the pipeline inverse variance and converted with the flux. The spectral weight is a fixed hyperparameter controlling the relative contribution of spectroscopy to the joint fit. We adopt for the canonical fit. Each fitted pixel or bin contributes one term. The fixed normalization constant , equal to the total number of fitted image pixels and spectral bins, sets the overall scale without changing the relative weights. With the variances held fixed, Equation (9) is the weighted sum of independent Gaussian negative log-likelihood terms, apart from normalization terms that contribute no parameter gradient. We add no explicit prior or regularization term, either on the field or on the shared dictionary. The Gaussian parameterization, non-negativity, finite dictionary, initialization, and density control nevertheless act as implicit structural assumptions.
For image pixels, we use the standard CCD variance model, combining source and sky shot noise with read noise and dark-current noise. Calculating the variance directly from the measured counts gives greater weight to measurements that fluctuate below their expected values, which can bias the fitted flux downward. We instead calculate the shot-noise variance from the current predicted source-plus-sky signal. At the current parameters , the variance in calibrated flux units is
where is the fixed background estimate. The effective gain converts predicted flux to counts, combining the survey gain with the flux calibration, and is the detector noise from read noise and dark current, expressed in flux units. Both quantities follow the detector-column calibration of the exposure. We use unless , in which case we retain the data-derived variance from § 3.2. This guard prevents an underestimated model variance from giving excessive weight to a poorly fitted pixel. The image variances and the guard decision are held fixed when taking parameter gradients and recomputed at the next update. For pixels within the guard and with positive source-plus-sky predictions, holding the variance fixed gives the same gradient as a shifted Poisson negative log-likelihood, with the read-noise and dark-current variance represented by equivalent counts. The image pulls and MSPs reported in the paper continue to use the data-derived uncertainties defined in § 3.2, so that different predictions are evaluated against the same noise estimates.
Optimization and adaptive density control
We initialize Gaussian components from peaks in the fitted source images, then alternate parameter updates with changes to the component population, as shown in the schematic diagram of Figure 4. We search the full source images of all fitted exposures (§ 3.2) to distribute the initial components near observed emission throughout the field. We merge repeated detections across bands and overlapping exposures so that the same brightness peak does not seed multiple sets of components. To represent both compact and extended emission from the outset, we initialize Gaussians of several widths around each retained peak. Their initial amplitudes are set from the measured peak brightness. The shared templates begin as distinct smooth slopes spanning blue to red, giving the initial dictionary a range of broad spectral shapes.
Figure 4. Overview of the field’s fitting procedure. At each step, the field is rendered through the observation operator to predict image pixels and spectral bins. These predictions are compared with the data using the loss defined in § 3.3, and Adam uses gradients of the fitting error to update the model parameters. Solid arrows show this cycle occurring every step, while dashed arrows show adaptive density control, which can split, duplicate, or prune components every 100 steps. The bottom row shows an observed -band galaxy’s evolving representation at four stages of the fit, ending after 11 epochs.
Each update uses a batch of image pixels and 64 spectral bins to form an unbiased estimate of the full-data objective. We draw pixels across the images used in the fit, with each image contributing to the batches in proportion to its number of included pixels. Within each image, we favor pixels with larger observed source signals relative to their noise, since most pixels contain little source light. A five-percent uniform contribution keeps every pixel included in the fit eligible, and the sampling probabilities remain fixed throughout the fit. We weight each sampled image term in inverse proportion to its selection probability, compensating for the extra visits to pixels containing source light so that their expected contribution to the objective is unchanged. Spectral bins are drawn from shuffled passes through the fitted-bin list. We multiply the corrected image average and the spectral average by their respective fitted populations, then apply the spectral weight and normalization in Equation (9).
We minimize this sampled objective using Adam (Kingma & Ba 2015). A sparse variant of Adam updates only the components that contribute to the sampled pixels and bins, while the shared spectral templates are updated at every step. We run the canonical fit for 11 image epochs, which takes 8 h on a single NVIDIA B200 GPU, with a peak allocated GPU memory of 11 GB. One image epoch corresponds to drawing approximately as many pixels as are included in the fit, although an individual pixel may be sampled more than once or not at all. The optimizer settings and implementation details are given in Appendix § 11.
To add spatial freedom during fitting, we adapt the gradient-guided cloning and splitting scheme of Kerbl et al. (2023), using a local port of gsplat’s densification strategy (Ye et al. [43]). The positional gradient measures how the objective responds to moving a component. Components whose mean positional-gradient magnitude exceeds a threshold are considered for duplication or splitting. Smaller candidates are duplicated at the same position and size, so the two components can then move and change shape separately. Larger candidates are split into two narrower Gaussians at positions drawn from the parent, so they can fit smaller features. Both operations preserve the parent’s spectral shape and divide its amplitude equally between the resulting pair. This conserves total flux at each wavelength in our additive field. Because gradient magnitudes depend on the fitting objective, we calibrate the growth threshold from the run’s own gradients. We use the median PSF width of the fitted images to set the size threshold between duplication and splitting. The quantitative thresholds and refinement schedule are given in § 11.1.
Pruning removes components whose estimated PSF-convolved peak is negligible relative to the image noise in every band, together with components above a maximum width. For pruning, we estimate each component’s peak brightness after blurring it with a circular Gaussian PSF, using the median PSF width in each band. Image predictions retain the full observation operator described in § 2.4. After growth stops, we continue updating and pruning the components until the fitting budget is reached.
The Fitted Field
Fitted images across the field
The joint fit reconstructs an patch of the COSMOS sky as a field continuous in position and wavelength, and closely reproduces the SDSS images and fiber spectra that entered the fit. Figure 5 compares the observed SDSS -band mosaic with the corresponding mosaic rendered from the fitted field. The full mosaic includes the withheld exposures, with dashed outlines marking the regions covered only by withheld images. We examine the withheld exposures separately in § 5.1.2.

Figure 5. Observed SDSS -band imaging and the field’s predictions across the COSMOS field. The field is represented by 2.86 million Gaussian components fitted to 873 million image pixels and 800 fiber spectra. The top row compares mosaics of the observations (left) and predictions (right). Dashed outlines mark regions covered only by withheld -band exposures. The bottom row enlarges the region marked by the solid square, showing galaxies and stars at the native SDSS pixel scale of .
The 7″ inset window, marked by the solid white square, shows how the same field represents various astrophysical objects such as the elongated disk of an edge-on galaxy, faint neighboring galaxies, and stars. These morphologies emerge from the joint fit without a fixed parametric profile or explicit morphological prior for any source. The enlargement in the figure uses the native SDSS pixel scale of 0.396″, with the same stretch used for the data and predictions.
We quantify the fit across all 400 fitted exposures using the pixel selections and data-derived uncertainties defined in § 3.2. Across the five bands, the sky-pixel residuals are comparable in magnitude to the observational noise, with MSP values of 0.98–1.06. Their mean signed pull measures the average offset: its values of −0.06 to −0.02 indicate that the predictions lie slightly above the data in every band. For the source-selected pixels, the summed predicted flux is only 0.4–1.7 percent below the observed flux across the five bands. The fractional absolute residual is 5.4–6.8 percent, indicating larger differences in the distribution of light that partially cancel when the flux is summed.
Images and spectra of the same objects
The agreement across the images extends to the spectra of the sources within them. Because the same components carry both the spatial profile and the spectrum, the field captures the extended profiles of galaxies and the compact images of stars and quasars, together with their distinct spectral continua and broad emission features, all built from the same spectral dictionary. Figure 6 shows the spatial and spectral views together for six selected objects: three galaxies, two stars, and a quasar. Each row shows a cutout from a fitted SDSS -band exposure, its rendered image counterpart, the observed spectrum at the marked fiber position, and the field’s spectral prediction. Across these examples, the field closely reproduces both the distribution of light in the images and the broad structure of the spectra. This agreement spans objects with very different morphologies and spectral shapes, demonstrating the flexibility of the joint representation across galaxies, stars, and quasars.
Figure 6. Joint fit of the field to images and spectra of three galaxies, two stars, and a quasar. The three image columns show the observed SDSS -band image, the prediction, and their residual. Dashed circles mark the fiber apertures used for the spectra shown on the right, where gray curves show the observations and red curves show the field’s predictions through the same fibers. A single fitted field produces all the image and spectral predictions shown here, through their respective observation operators.
The close agreement seen in the selected examples extends to the wider fitted sample, with 96 percent of the 800 spectra having integrated fluxes within ten percent of the observations. Table 1 shows that integrated-flux agreement persists across the three brightness terciles, with median flux ratios of 0.993, 0.993, and 0.977 for the brightest, middle, and faintest terciles, respectively. The fit can match
| Tercile | S/N | Flux ratio median | Within 10% | MSP | |
| Brightest | 266 | 32.5 | 0.993 [0.984, 1.002] | 97.4% | 4.37 |
| Middle | 266 | 9.2 | 0.993 [0.974, 1.007] | 96.2% | 1.44 |
| Faintest | 268 | 1.9 | 0.977 [0.957, 1.003] | 94.8% | 0.99 |
| All | 800 | 9.3 | 0.989 [0.973, 1.004] | 96.1% | 1.45 |
Table 1. Integrated-flux ratios and residuals for the 800 fitted spectra, by brightness tercile. Spectra are ranked by their observed source flux integrated over the bins that pass quality cuts. is the number of spectra, and the signal-to-noise ratio (S/N) is the median over spectra of each spectrum’s median flux-to-uncertainty ratio per bin. The flux ratio is the predicted-to-observed integrated flux (Section 3.2), given as its median, its interquartile range (IQR), and the percentage of spectra within ten percent of unity. MSP is reported as the median of the tercile.
the total flux of a spectrum without reproducing every feature accurately, and small mismatches stand out more clearly in the brighter spectra because their relative uncertainties are smaller. The median MSP is 0.99 in the faintest tercile and 4.37 in the brightest.
In Figure 6, the field follows the F9 star’s broad continuum peak and decline toward longer wavelengths, as well as the K7 star’s rising continuum. Their integrated fluxes agree with the observations to within 2.1 percent, even though both lie among the ten percent of fitted stars with the highest MSPs. Several narrow absorption troughs remain too shallow. The galaxy introduced in Figure 1 shows a similar pattern, with a closely matched continuum and narrow emission peaks that remain underpredicted. § 6 considers the template knot spacing relative to the widths of these features.
The fitted components and spectral dictionary
Across the fitted field, 2.86 million Gaussian components draw on the same 32 spectral templates. The bottom row of Figure 4 follows a single galaxy through the field’s growth from 100,000 to 5.64 million components and subsequent pruning to this final population, showing how the representation is refined across updates. Most Gaussian components are narrower than the seeing and combine with broader components to describe the compact and extended emission across the field. The observations constrain their combined light through the instrument responses, so their individual shapes need not trace resolved physical structure, as discussed in § 6.
From the smooth slopes used at initialization, the spectral dictionary developed overlapping shapes with peaks at different wavelengths, shown in the right panel of Figure 2. Their wavelength extents are broad, with a median of approximately 1000 Å enclosing the central 68 percent of template strength on the log-wavelength grid, while variation from knot to knot remains visible within them. These learned shapes provide a common set of contributions that the fit combines in different proportions across the field, but should be interpreted as shared spectral building blocks rather than spectra of individual objects. An object’s spectrum can combine light from many Gaussian components, each carrying its own mixture of these templates.
The fitted component spectra combine several of these shapes, with a median effective count equivalent to about six equally contributing templates. This effective count is unchanged by the normalization rescalings in § 2.2. The contrasting galaxy, stellar, and quasar spectra in Figure 6 are assembled from these mixtures and shifts, summed over the multiple Gaussians contributing to each fiber. Their close continuum agreement is therefore achieved with a common dictionary learned jointly from the 800 fitted spectra and the imaging. The joint fit closely reproduces the spatial structure and broad spectral diversity of the fitted sources within one continuous sky field, without prescribing a separate model for each object. § 5 tests what this fitted representation predicts for measurements excluded from the fit and, zero-shot, for observations from other instruments. The control uses the same imaging, random seed, recipe, and update budget, with all spectra excluded and its objective normalized by the fitted pixel count alone.
Novel Observation Synthesis
Withheld observations
We first predict the 89 SDSS/BOSS fiber spectra and ten SDSS image exposures omitted from the fitting objective (§ 3.2). The field and its shared dictionary remain fixed, so these comparisons test how the fitted measurements constrain predictions for the withheld spectra and images.
Withheld spectra and what spectroscopy contributes
Broadband images constrain integrated fluxes but leave finer spectral structure uncertain. The spectra fitted elsewhere in the field can inform that structure through the shared dictionary. We test their contribution by comparing the joint fit with an images-only control at the 89 withheld fiber positions, with both predictions made without refitting.
Figure 7 illustrates the improvement for selected star, galaxy, and quasar spectra: the joint predictions follow the observed continua much more closely, while the images-only predictions show large excursions with wavelength. Across all 89 withheld spectra, the joint fit has the lower MSP for 88, with a median MSP of 6.01 compared with 98.5 for the control. Repeating both fits with two further random seeds gives the same result, with the joint fit having the lower MSP for 88 of the 89 spectra in each pair. Including spectroscopy therefore substantially improves spectral predictions away from the fitted fiber positions across all three seeds of this field.
Figure 7. Predicting spectra of withheld objects. Predictions of four withheld SDSS fiber spectra, showing a star, a galaxy, and two quasars. Gray curves show the observed spectra, which never entered the fit during training. Blue curves show predictions from the field when fitted to images alone, while red curves show predictions from the canonical fit of the field, which uses the same images and 800 spectra elsewhere in the field. The joint fit follows the observed continua more closely, generating spectra for objects constrained locally by broadband imaging. In these predictions, spectroscopy elsewhere constrains the spectral dictionary, improving predictions of withheld spectra beyond the fitted fiber positions.
For a typical withheld spectrum, the joint prediction still has residuals larger than the observational uncertainties. The median MSP of 6.01 for the withheld sample compares with 1.45 for the 800 fitted spectra in Table 1. The median predicted-to-observed integrated-flux ratio is 0.99 for the 55 SDSS-spectrograph spectra observed through 3″ fibers and 0.90 for the 34 BOSS spectra observed through 2″ fibers. Thus the joint fit can follow the broad spectral shapes at withheld positions while retaining appreciable errors in both spectral structure and flux. The F9 star’s narrow absorption features in Figure 7 are not fully reproduced. § 6 considers the spectral parameterization and its limitations. The fit can be substantially biased where no fitted same-band imaging is available.
Withheld exposures
We withhold ten complete image exposures from the fit, two in each of the five SDSS bands. Each withheld exposure removes all measurements from one image in a single band, while the remaining fitted images and spectra continue to constrain the same sky. Figure 8 shows an -band example for which the corresponding , , , and exposures were all fitted. After the overlap exclusions and masking, 86 percent of the scored pixels have no fitted -band coverage. The two edge strips retain fitted -band measurements from another SDSS run, and we assess them separately. Copies of withheld detector samples in neighboring fields are also excluded from the objective, as described in § 3.2.

Figure 8. Prediction of an SDSS -band exposure withheld from the fit. The top row compares the observation (left) with the field’s prediction (right). Dashed lines enclose the region with no fitted -band coverage, while the edge strips retain -band measurements from another run. The bottom row enlarges the region marked by the solid square, showing the fitted , , , and images alongside the withheld -band image, its prediction, and their residual. Each Gaussian carries a continuous spectrum built from the shared dictionary and constrained by the other fitted bands, allowing the field to render the -band image even where no -band imaging entered the fit.
The predicted -band image reproduces the broad spatial structure of the withheld observation, including the extended emission around the galaxy in the zoom. To measure agreement around sources, we apply the pixel selection from § 3.2 separately within each coverage region. The summed predicted flux over these pixels is 9.9 percent below the observation in the region without fitted -band coverage and 9.3 percent above in the covered edge strips. The corresponding sky-pixel MSPs are 1.06 and 1.08. Across all ten withheld exposures, source-selected pixels in regions without fitted same-band coverage instead have MSPs of 28–603 and summed predicted-to-observed flux ratios of 0.45–1.16. The largest flux deficits occur in the two exposures, with ratios of 0.64 and 0.45. For both, the corresponding or exposure was also withheld. These withheld exposures show that the field can reproduce broad spatial structure in an excluded band, but the summed flux in source-selected pixels can be substantially biased where no fitted same-band imaging is available.
New instruments and operators
The fitted spectral surface-brightness field can be queried through different bandpasses, angular resolutions, and apertures to predict observations from other instruments. We test this by virtually reconstructing the instrument operators of HSC PDR3 imaging (Aihara et al. 2022) and DESI DR1 spectra (DESI Collaboration et al. [8]), then using them to synthesize the corresponding images and spectra. These predictions are zero-shot: the field and its shared dictionary are learned from SDSS observations alone, with no HSC or DESI measurements used in the fit.
A new instrument: HSC imaging
We first test operator transfer by querying the SDSS-fitted field to predict HSC -band images. Figure 9 compares the observed and predicted HSC images of two selected regions, an extended barred spiral galaxy in the top row and a more compact barred spiral galaxy in the bottom row, alongside the SDSS cutouts from those locations. For these two selected galaxies, the predictions through the virtual HSC instrument response reproduce the extended light distribution in the top row and the broad pattern of the more compact spiral in the bottom row. To make these predictions, we render the field through the HSC bandpass and local PSF on each coadd’s own pixel grid, using the HSC-provided astrometry and ABC calibration.3 Predicted-to-observed HSC flux ratios are measured within fixed -radius apertures centered on the DESI DR1 catalog positions of the selected galaxies (§ 5.3). SDSS adapts the surface-brightness quantity of Petrosian (1976) and sets the flux aperture radius to twice the -band Petrosian radius to include most of a galaxy’s light while limiting sky noise (Strauss et al. 2002). Of the 80 galaxies without a fitted SDSS spectrum within , 79 have a unique match in the SDSS DR17 catalog (Abdurro’uf et al. 2022). Their median Petrosian aperture radius is , so we use a fixed radius for the HSC flux ratios. The observed and predicted aperture fluxes are measured at HSC’s native resolution using the same unmasked pixels. The field remains fixed, and the predictions are not rescaled to match the observed HSC fluxes. The HSC mask, coverage, and PSF-quality cuts retain 49 of the 80 galaxies without a fitted SDSS spectrum. These galaxies have a median flux ratio of 1.01 and an IQR of 0.95–1.07. We applied the same HSC quality cuts to all 80 DESI galaxies with a fitted SDSS spectrum, each within of…

Figure 9. Zero-shot HSC imaging. We render the field fitted to SDSS alone through the HSC -band operator, using the HSC bandpass, PSF, and pixel grid. Each row shows a spiral galaxy. The first column shows the fitted SDSS -band image and the second shows the independent HSC -band observation that was not in training. The third column shows the field’s prediction through the reconstructed HSC imaging operator, and the fourth shows the HSC-minus-field residual. Neither galaxy has a fitted SDSS fiber within of its center.
or BOSS fiber (Table 2). The 55 that pass have a median flux ratio of 0.973 and an IQR of 0.943–0.994. At the 3″ diameter of SDSS spectrograph fibers, the same 49 galaxies without fitted SDSS spectra have a median flux ratio of 0.967 and an IQR of 0.939–1.004. Of the 55 galaxies with fitted SDSS spectra, 54 pass the mask-coverage cut at this aperture. Their median flux ratio is 0.967 with an IQR of 0.934–0.999. For the galaxies without fitted SDSS spectra, the change in median flux ratio between apertures shows that the near-unity agreement in the larger aperture can depend on the spatial scale being measured.
| Class | Spectrum | Median ratio | IQR | Shape (%) | |
| Galaxies | Absent | 80 | 0.986 | [0.924, 1.019] | 9.9 |
| Galaxies | Present | 80 | 0.932 | [0.850, 1.017] | 5.6 |
| Quasars | Absent | 79 | 0.974 | [0.852, 1.213] | 22.8 |
| Quasars | Present | 80 | 0.993 | [0.790, 1.191] | 14.7 |
| Stars | Absent | 80 | 0.814 | [0.678, 0.915] | 16.6 |
| Stars | Present | 80 | 0.988 | [0.924, 1.027] | 4.6 |
Table 2. Table 2. Predictions of selected DESI DR1 spectra from the fixed SDSS-fitted field. is the number of spectra. Flux ratios compare predicted and observed flux summed over valid pixels from 4000 to 9000 Å. The median ratio and IQR are reported. The shape error is the RMS residual in 100 Å bins with the overall flux level removed, divided by the median absolute observed bin flux. Its median over spectra is given in percent. In the Spectrum column, ‘Present’ denotes an SDSS or BOSS spectrum included in the fit within 1″ of the target, while ‘Absent’ denotes no fitted SDSS or BOSS fiber within 10″.
Figure 9 shows that the agreement extends beyond aperture flux to the broad spiral-arm patterns of both galaxies. In the bottom row, the arms are difficult to distinguish in the blurred SDSS image, yet the prediction reproduces the pattern seen independently by HSC. The same prediction contains a straight streak. This artifact arises from elongated Gaussian components in the fitted field and is more pronounced in the example shown in Figure 11(a). HSC’s sharper PSF and finer pixel sampling also reveal knots and core structure that are not fully reproduced, on scales less well constrained by the lower-resolution SDSS observations used to fit the field. § 6 examines these artifacts and what the fitted observations constrain below their seeing scale.

Figure 11. Examples of known failure modes. Artifacts in the fitted field. Each row compares the observed SDSS -band image (left) with its prediction (right). In (a), elongated Gaussian components produce thin, straight streaks around the sources. Similar elongated artifacts are a known limitation of the original 3D Gaussian-splatting method that we adapt here (Kerbl et al. 2023). In (b), SDSS exposures flagged as non-photometric were retained in the fit. With the survey background held fixed, their rectangular background excess is absorbed into the field as extended, filamentary structure.
A new spectrograph: DESI
We further test operator transfer by querying the field through a DESI spectrograph, which has a smaller fiber and a different spectroscopic response. Of the 479 DESI spectra in our sample, 250 (52 percent) combine multiple exposures. For these coadded spectra, we predict each exposure separately using its recorded fiber position and angular aperture before combining the predictions. The seeing is represented by a Moffat profile with (Guy et al. 2023). We use the seeing recorded for each exposure when available and otherwise adopt the 1.1″ default of the DESI point-source flux correction routine. DESI’s calibration accounts for point-source light falling outside the fiber (Guy et al. 2023), so we apply the corresponding correction to our predictions. The exposure predictions are then combined, the coadd resolution matrices applied on DESI’s wavelength grid, and the fluxes converted to AB.
Targets are selected from DESI DR1 using its galaxy, stellar, and quasar classifications, with for galaxies. We rank eligible targets by observed SDSS -band aperture flux, so the samples below are deliberately bright and test operator transfer rather than represent the full DESI populations. For the samples without fitted SDSS spectra, we select the brightest 80 targets in each class more than 10″ from any fitted SDSS or BOSS fiber. Stars and quasars in these samples also have no fitted fiber within their observed image footprints. We exclude one quasar without a fitted SDSS spectrum whose redshift DESI flags as uncertain, leaving 79. For the samples with fitted SDSS spectra, we select the brightest 80 eligible targets in each class with a fitted SDSS or BOSS fiber within 1″. Three stellar candidates failed the DESI redshift or wavelength-coverage cuts, so we used the next three eligible stars in brightness order.
Table 2 summarizes the flux ratio and shape error for our sample. The flux ratio is the summed predicted flux divided by the summed observed flux over valid DESI pixels from 4000 to 9000 Å, which lies within the wavelength coverage shared by the SDSS and DESI spectrographs. For the shape comparison, we average the observed and predicted spectra in 100 Å bins using DESI-provided inverse-variance weights, giving greater weight to wavelengths with smaller observational uncertainties. The shape error measures the remaining root-mean-square (RMS) difference after removing the overall flux
level, which the flux ratio already measures, as a percentage of the median absolute observed bin flux. Figure C1 in Appendix § 12 shows the six spectra in each sample without fitted SDSS spectra whose predictions agree most closely with DESI.
Figure C1. Zero-shot DESI DR1 spectra for the six closest matches in each sample without fitted SDSS spectra. No DESI data were used to fit the field or its shared spectral dictionary. The targets have only SDSS imaging constraints locally, while SDSS spectra fitted elsewhere inform their predictions through the shared dictionary. The columns show galaxies, quasars, and stars from left to right. Gray curves show the DESI observations, and red curves show the field rendered through the DESI operator at its fitted flux scale. Each panel gives the predicted-to-observed flux ratio and the RMS difference in 100 Å bins over 4000–9000 Å, expressed as a percentage of the median absolute observed bin flux. Table 2 gives the medians for the corresponding samples without fitted SDSS spectra.
The samples with fitted SDSS spectra have smaller median shape errors than the separately selected samples without them in all three classes (Table 2). For galaxies, the median shape error is 5.6 percent for those with fitted SDSS spectra and 9.9 percent for those without them, while the median integrated-flux ratio is closer to unity for galaxies without fitted SDSS spectra (0.986 versus 0.932). Quasar flux ratios span a wide range in both samples, with median shape errors of 14.7 percent for quasars with fitted SDSS spectra and 22.8 percent for those without them. The stellar predictions for sources without fitted SDSS spectra are systematically faint, with a median flux ratio of 0.814 compared with 0.988 for stars with fitted SDSS spectra. The DESI observations were taken about two decades after the SDSS imaging (DESI Collaboration et al. [8]), and we apply no variability or proper-motion correction. § 6 examines the brightness dependence of the flux deficit among stars without fitted SDSS spectra.
Figure 10 compares three apertures in the region around the galaxy in the top row of Figure 9. At the core, the prediction follows the broad DESI continuum with an integrated-flux ratio of 0.999 and a shape error of 12.8 percent, although it remains redder and does not reproduce the narrow emission peaks. The nearby source labeled “Red companion” is a background galaxy at with a fitted BOSS spectrum. The BOSS prediction agrees closely with the data, as does the spectrum was fitted. On the blue wing, a virtual SDSS fiber of 3″ diameter is rendered to showcase a spectrum where no observed spectrum is available. It is placed on the galaxy’s extended spiral arm, about 18″ from the core. Its broad color is constrained by the fitted images, while its finer spectral detail comes only from the shared dictionary.

Figure 10. Virtual fiber spectra around the spiral galaxy in the top row of Figure 9. The color image and all three spectral predictions are rendered from the field. Circles mark the apertures, with matching colors for the predicted spectra on the right. Gray curves show the observed spectra. The top panel compares the zero-shot prediction through a virtual DESI operator with the independent DESI spectrum. The middle panel shows the fitted BOSS spectrum of the background galaxy. The bottom panel predicts a -diameter SDSS fiber spectrum on the blue spiral arm, where no observed spectrum is available. Spectra are normalized for display. Because the field is continuous in position and wavelength, a virtual aperture can be placed anywhere in the fitted field to predict a spectrum, as illustrated by the blue wing.
Discussion
Jointly fitting images and spectra to the same underlying model allows spectroscopic information to affect predictions beyond the fitted fibers. The comparison in Section 5.1.1 demonstrates this for the 89 withheld spectra, where including spectroscopy substantially improves the predictions compared with images-only fits. The shared spectral dictionary allows spectra observed at a subset of positions to constrain the templates shared by every Gaussian component, regardless of whether a spectrum was observed at that point. This provides a way to connect sparse fiber measurements to the wider imaging coverage, creating an IFU-like field of predicted spectra across the entire imaged area. The blue-wing aperture in Figure 10 demonstrates an application of this inherent multimodality by predicting a spectrum within an extended source where no observed spectrum is available. Its detailed shape therefore remains unvalidated, but the spectra fitted elsewhere in the field still constrain it through the shared dictionary.
Because the field is a spectral surface brightness on the SDSS flux scale, constructing virtual instrument operators allows the same fitted field to predict measurements through instruments that supplied no data to the fit. § 5.2 builds HSC and DESI operators and applies them to the field fitted to SDSS data alone. A common source model rendered through different PSFs is already used in forced photometry. The Farmer (Weaver et al. 2023) fits each source with a model from The Tractor (Lang et al. 2016) and can hold its shape and position fixed while fitting only its flux in each band. In our framework, the joint fit connects image pixels and fiber spectral bins, allowing both kinds of measurements to be predicted from the same field. Galaxies with no fitted SDSS spectrum within have median flux ratios of 1.01 in -radius HSC apertures (49 galaxies; 0.967 at diameter) and 0.986 for DESI spectra (80 galaxies), showcasing constraints that generalize beyond what the fitted data supply. Galaxies with a fitted SDSS or BOSS spectrum within have a median DESI shape error of 5.6 percent, compared with 9.9 percent for galaxies without fitted SDSS spectra. Quasars and stars with fitted SDSS spectra also have lower median shape errors than those without them (Table 2).
Joint reconstruction has been demonstrated for images with different PSFs and pixel scales (Joseph et al. 2021; Melchior et al. 2026) and for multiband imaging combined with integral-field spectroscopy (Guilloteau et al. 2020; Marquis et al. 2026). In the JWST fusion of Marquis et al. 2026, the spectral basis is derived from the observed NIRSpec cube. Peng et al. (2026) predict spatially resolved galaxy spectra with calibrated uncertainties by learning from a large sample of images and single-fiber DESI spectra. Our framework instead learns its spectral dictionary and spatial components jointly from the 400 exposures and 800 fiber spectra of a single sky region, requiring neither an IFU-derived spectral basis nor an external pretraining corpus.
Among the stars without fitted SDSS spectra in Table 2, the median flux ratio is 0.814, and the shortfall grows with brightness. For the brightest third of the stars (Gaia ), total-flux estimates from a -radius aperture with a point-source correction give a median ratio to DESI of 0.73, compared with 1.01 in the faintest third. These bright stars also have broader profiles in the field than point sources, further reducing the fraction of their light collected by the DESI fiber. Gaia DR2 photometry provides a separate check on this brightness dependence ([14, 15]). We fit separate empirical color conversions for the field totals and DESI fluxes using the faintest third, then hold these conversions fixed for the brighter stars. From the faintest to the brightest third, the median DESI-to-Gaia ratio changes only from 1.005 to 1.039, whereas the field-to-Gaia ratio falls from 1.018 to 0.745, meaning the brightness-dependent deficit lies in the field’s predicted total flux. Of the 27 stars in the brightest third, 25 have cores saturated in at least one fitted , , or exposure, and those pixels were masked from the fit (§ 3.1). Brighter stars also tend to have more extensive saturation, so we cannot determine from this sample whether saturation is responsible for their flux deficit. Zero-shot fluxes for the brightest third of stars are therefore biased low in this field, because the fitted field underestimates their total light and spreads it more widely than a point source.
Our representation has spatial degrees of freedom below the SDSS seeing, but recovering that structure from the blurred observations is an ill-posed inverse problem. Of the fitted components, 84.2 percent, carrying 74.6 percent of the -band light, have intrinsic major-axis FWHM below 1.4″, close to the 1.408″ median PSF FWHM of the fitted images. The independent HSC images provide a qualitative check for two selected galaxies (Figure 9), showing that the field reproduces broad spiral-arm patterns while knots and core structure are not fully reproduced. Figure 11(a) shows thin, straight streaks in the SDSS rendering that are absent from the data, resembling the elongated artifacts reported for Gaussian splatting (Kerbl et al. 2023). We tested an axis-ratio cut to remove these components, but an unconditional cut could also delete components carrying well-fitted source light, so we leave this problem for future work. Different intrinsic component distributions can produce similar predictions after PSF blurring, so the data constrain the final light profile through the instrument operator more directly than the underlying components that make it up. In this sub-PSF regime, the Gaussian basis, initialization, and density-control rules influence aspects of the component structure that the data constrain only weakly. We apply no explicit spatial or spectral prior to preserve the representation’s flexibility to represent diverse morphologies and spectra within a common model. Structure below the SDSS seeing therefore requires independent validation before it can be interpreted as resolved morphology (Lanusse et al. 2021).
This intended flexibility allows errors in the survey inputs, such as the background estimates, to become structure in the learned field. The survey background is held fixed, so broad components can absorb excess light left by an underestimated background. Figure 11(b) shows a block of SDSS -band data that sits above its surroundings and is reproduced as extended, filamentary structure in the field. The affected SDSS exposures carry non-photometric calibration flags but were retained in the fit, and the fit consequently treats the residual background excess as genuine flux and adds it to the field. Our reconstruction therefore relies on the survey calibration and background estimates being sufficiently accurate for the adopted noise model. Scaling to larger survey areas will require careful quality control and data cleaning to identify and correct or exclude contaminated measurements before they enter the fit.
The fitted field rendered through fiber spectrograph operators can reproduce continuum shapes and integrated fluxes while also missing narrow features. Absorption troughs remain too shallow in the fitted and withheld stellar examples, and narrow emission peaks are broadly smoothed over. The coarse template grid may contribute: its knots are evenly spaced in log wavelength, giving a separation of about 84 Å at 6000 Å, for example, compared to an instrumental LSF width of only a few angstroms (§ 2.5). Other grid spacings and definitions have not been tested. The continuum and integrated-flux agreement in § 4 and § 5 therefore does not establish accurate line strengths or widths.
Our comparisons measure prediction errors for selected observed samples, but we do not yet provide calibrated uncertainties for an individual flux or spectrum queried from the field. Estimating those uncertainties requires accounting for the different combinations of spatial components and shared spectral templates consistent with the data. Component degeneracies also complicate error estimation in scarlet (Melchior et al. 2018). Covariance approximations or posterior sampling, discussed for Jolideco by Donath et al. (2024), are possible approaches to providing uncertainties on measurements derived from the fitted field.
Conclusions
We have presented Spectro-Spatial Survey Fusion, a framework for jointly fitting a continuous spectral surface-brightness field to images and fiber spectra through their instrument responses. An adaptive Gaussian basis and a shared spectral dictionary are learned together from the observations, allowing the same model to describe galaxies, stars, and quasars without prescribing a separate profile for each source. We applied this representation to 400 SDSS exposures and 800 fiber spectra over an 84′ × 84′ region of COSMOS. The field reproduces the spatial structure and broad spectral continua of the sources, with 96 percent of the fitted spectra agreeing in integrated flux to within ten percent, although narrow spectral features are reproduced less accurately.
Our framework represents the underlying sky as a continuous spectral surface-brightness field, allowing the same fitted model to predict measurements through different instrument operators. We use this field to predict withheld fiber spectra, HSC images, and DESI spectra without refitting. Including spectroscopy lowers the mean squared pull relative to an images-only control for 88 of the 89 withheld spectra at each of three random seeds. In the selected galaxy samples with no fitted SDSS fiber within 10″, the median predictedto-observed flux ratios are 1.01 in -radius HSC -band apertures (0.967 at diameter) and 0.986 for integrated DESI spectra. The predictions through new instrument responses motivate joint fits of complementary surveys. Including higher-resolution imaging such as HSC would add direct constraints on fine spatial structure, alongside the wavelength information supplied by spectroscopy. Extending the fit to larger sky regions would test the representation across a wider range of sources and observing conditions. This framework offers a route toward a continuous model of the sky built from the combined spatial and spectral information of many surveys, where the detail resolved by one instrument can help constrain what remains unresolved by another.
Acknowledgements
This research is supported by a grant from Alex Gerko, Founder of XTX Markets.
ChatGPT and Codex (OpenAI) and Claude (Anthropic) were used to assist with code development, analysis, and manuscript outlining and editing. The authors take full responsibility for the content and conclusions of this work.
Data Availability
The SDSS, HSC PDR3, DESI DR1, and Gaia DR2 data are available through their respective public archives. A frozen snapshot of the research code and supporting metadata is available at https://github.com/GeorgeVassilakis/survey-fusion-paper-code. This unmaintained release excludes fitted checkpoints and the complete HSC/DESI analysis workflows.
Notation
Table A1 lists the symbols used in this paper and the section that defines each.
| Symbol | Meaning | Section |
| , | fitted region and position in flat-sky coordinates | 2.1 |
| , | modeled wavelength range and observed wavelength | 2.1 |
| spectral surface-brightness field | 2.1 | |
| , | datum index and set of all image pixels and spectral bins | 2.1 |
| , | image pixels and spectral bins | 2.1 |
| exposure or spectrum containing datum | 2.1 | |
| , | background-subtracted measurement and prediction | 2.1 |
| , , | survey background, observation operator, and noise | 2.1 |
| , | component index and number of components | 2.2 |
| , | component amplitude and unit-normalized spatial Gaussian | 2.2 |
| , | center and covariance of | 2.2 |
| component spectrum | 2.2 | |
| , , | template index, count, and shape | 2.2 |
| , | template coefficients and log-wavelength shift | 2.2 |
| , | band of image datum and system throughput of band | 2.3.1 |
| , | pixel-center position and pixel-convolved PSF | 2.3.1 |
| band average of | 2.3.1 | |
| , | fiber aperture and radius | 2.3.2 |
| , | spectroscopic seeing PSF and standard deviation | 2.3.2 |
| fraction of component ’s light collected by fiber | 2.3.2 | |
| , | wavelength interval and LSF of bin | 2.3.2 |
| LSF-convolved bin average of | 2.3.2 | |
| point-source fraction admitted by fiber | 2.3.2 | |
| , | observational uncertainty and pull | 3.2 |
| mean squared pull over | 3.2 | |
| model parameters | 3.3 | |
| , | fitted image pixels and spectral bins | 3.3 |
| , | image variance at update and spectral weight | 3.3 |
| , | effective gain and detector noise | 3.3 |
| fixed normalization of the objective | 3.3 |
Table A1. Summary of notation.
Optimization settings
The shared-template and sparse component optimizers both use momentum coefficients and . The initial learning rates in Table B1 follow the gsplat defaults mapped onto our parameter groups, with the position rate reduced tenfold. The position rate decreases exponentially to one percent of its initial value over the first 30,000 updates and is then held fixed. The other rates remain constant. We clip the combined gradient norm of the shared template parameters at unity, leaving per-component gradients unclipped. The implementation uses PyTorch (Paszke et al. 2019) with custom Triton kernels (Tillet et al. 2019) for rendering and sparse parameter updates.
| Parameter | Initial learning rate |
| Position | |
| Log width | |
| Position angle | |
| Amplitude | |
| Spectral mixture coefficients | |
| Template values | |
| Spectral shift |
Table B1. Initial learning rates for the canonical fit. Rates act on the stored parameters before exponentiation or softplus. Spatial coordinates and the template log-wavelength interval are each normalized to . Only the position rate decays.
Component growth and pruning
Growth is considered every 100 updates from update 600 through 14,900. At the first refinement, we set the growth threshold to the 70th percentile of the mean positional-gradient magnitudes among visited components and hold it fixed thereafter. At subsequent refinements, each mean is measured over the component’s sampled visits since the previous refinement. Components above the threshold that are not already marked for removal are duplicated if their largest intrinsic standard deviation is at most three times the median PSF FWHM of the fitted images. Duplication retains the parent’s position and widths. Larger candidates are replaced by two children with positions drawn from the parent Gaussian and both principal-axis widths divided by 1.6. Both operations copy the parent’s spectral mixture coefficients and shift, with the amplitude division described in § 3.4. There is no opacity-reset operation.
Pruning is evaluated on the same 100-update cadence and continues after growth ends. A component is removed if its estimated PSF-convolved peak is below of the representative image noise in every band, or if its largest intrinsic standard deviation exceeds about 252″. The peak estimate uses a circular Gaussian PSF with the median fitted-image FWHM in each band and includes the pixel-response variance. The noise reference is the median data-derived uncertainty of sampled unflagged pixels in those exposures. The prediction operator itself is unchanged by this approximation.
For the canonical fit, one image epoch comprises updates, giving 960,201 updates over 11 epochs. It reaches 5,639,839 components at update 14,900 and ends with 2,855,026 components. The last pruning event, at update 960,200, still removed 18 components.
Zero-shot DESI spectra
Figure C1 shows the six DESI DR1 spectra in each sample without fitted SDSS spectra of Table 2 whose zero-shot predictions agree most closely with the observations. We rank each sample by the RMS difference of the 100 Å bins of § 5.3 without removing the overall flux level, so that a spectrum must match in both flux and shape. The panels show the closest agreement in each sample, and Table 2 gives the population medians.
This paper has been typeset from a TeX/LaTeX file prepared by the author.
References
- [7]DESI Collaboration Aghamousa A., Aguilar J., et al., 2016, arXiv e-prints, p. arXiv:1611.00036
- [8]DESI Collaboration et al., 2026, AJ, 171, 285
- [12]Euclid Collaboration Mellier Y., et al., 2025, A&A, 697, A1
- [14]Gaia Collaboration et al., 2016, A&A, 595, A1
- [15]Gaia Collaboration et al., 2018, A&A, 616, A1
- [43]Ye V., et al., 2025, Journal of Machine Learning Research, 26, 1
- [47]de Jong R. S., Agertz O., Berbel A. A., et al., 2019, The Messenger, 175, 3






