WorksPhysics and Astronomy
Mitigating variance shrinkage in multi-task galaxy property inference from imaging data via statistical manifold alignment
Abstract
Inferring photometric redshifts () and key stellar population properties (, , and ) directly from multi-band imaging is increasingly critical for spectroscopically limited large surveys such as LSST and CSST. However, conventional point-wise loss functions (e.g., mean squared error or Huber loss) employed in deep neural networks often induce ``variance shrinkage''—a regression-to-the-mean effect that contracts predicted variances, distorts non-linear physical manifolds (such as bimodal distributions), and truncates heavy-tail populations. Here, we present ViTMultiTask, a vision transformer framework that incorporates a multi-scale maximum mean discrepancy (MMD) loss to enforce higher-order joint distribution alignment in reproducing kernel Hilbert space (RKHS). Trained on 83,846 galaxies from Subaru HSC five-band () imaging cross-matched with DESI Data Release 1 Bright Galaxy Survey spectra, the model achieves high point-estimate precision on an independent test set of 5,000 galaxies: for , for , for , and for . Ablation experiments demonstrate that incorporating the MMD loss effectively mitigates variance shrinkage and more accurately reproduces the observed spectroscopic parameter distributions (including bimodal structures and heavy tails), while maintaining point-estimate accuracy statistically consistent with the baseline (RMSE changes ) and catastrophic outlier rates fully comparable to the baseline. This kernel-based manifold alignment offers a statistically grounded and practical paradigm for comprehensive galaxy property inference in the deep imaging era.
Introduction
Accurately determining the physical properties of galaxies for large statistical samples is fundamental to understanding cosmic structure formation and galaxy evolution [16, 52]. Next-generation wide-field imaging surveys—including the Vera C. Rubin Observatory’s Legacy Survey of Space and Time (LSST; [37]), the Euclid space telescope ([42] (Euclid Collaboration et al. 2025)), and the Chinese Space Station Telescope (CSST; [67, 12, 18])—will acquire imaging data for billions of galaxies. This vastly exceeds the spectroscopic follow-up capacity of contemporary spectroscopic surveys such as the Dark Energy Spectroscopic Instrument [20, 22 b]. Consequently, directly inferring photometric redshift () and key stellar population evolutionary parameters—total stellar mass (), break index, and H equivalent width—from pixel-level multiband images has become an important objective bridging large-scale photometric surveys and galaxy evolution theory.
Traditional methods fall broadly into two categories: spectral energy distribution (SED) template fitting (e.g., LePhare, EAZY; [5, 9]) and classical machine learning algorithms (e.g., random forests, self-organizing maps; [53, 38] (Lu et al. 2024)). Template fitting is limited by template library incompleteness and parameter degeneracies in non-typical galaxies [46, 55]; classical machine learning methods rely on tabular inputs such as integrated photometric magnitudes or colors, discarding rich spatial morphological and dynamical information—such as concentration, bulge-to-disk ratio, and tidal structures—that are physically correlated with galaxy evolutionary stages [40, 17]. Deep learning methods represented by convolutional neural networks (CNNs) have achieved end-to-end inference directly from image cutouts ([24, 56]). However, CNNs are restricted by local receptive fields, posing inherent limitations in capturing long-range cross-band dependencies and cross-scale spatial correlations. Vision transformers (ViT; [26, 66, 48]), leveraging global self-attention mechanisms, can efficiently compute global relationships between image patches, providing a superior architectural choice to simultaneously probe subtle spatial structures and multi-color spatial gradients in galaxies.
Although deep neural networks have demonstrated good precision in mapping multiband pixel fluxes to scalar galaxy properties, most existing approaches still treat individual physical parameters as independent single-task regression problems and optimize them using point-wise loss functions, such as mean squared error (MSE) or Huber loss. This paradigm has two potential limitations. First, processing tasks independently makes it difficult to fully share underlying representations across parameters, which may lead to internally inconsistent predictions and thus affect the recovery of physical scaling relations, such as the star-forming main sequence [61] or the mass–metallicity relation [65]. Second, and more critically, point-wise loss functions tend to induce variance shrinkage (i.e., regression to the mean): to minimize mean squared residuals under observational noise, the network tends to predict the conditional expectation . By Jensen’s inequality, the variance of the conditional expectation is upper-bounded by the total variance, . This can artificially narrow the predicted distribution, potentially suppressing heavy tails (e.g., low-mass or high-redshift galaxies) and distorting astrophysical bimodal structures, such as the transition between the star-forming blue cloud and the quiescent red sequence [63, 6]. For statistical cosmology applications—constructing the galaxy stellar mass function (GSMF; [54]), tracing star-formation quenching histories [57], or selecting rare extreme objects—preserving overall distribution fidelity and physical boundaries is often more important than minimizing residuals for individual objects. Generative models, such as normalizing flows [39, 59], can model full probability density functions, but they generally incur substantial computational overhead and training complexity when scaled to high-dimensional multi-task parameter spaces.
To overcome these limitations without the heavy computational burden of generative models, non-parametric statistical distances provide a useful alternative for encouraging distribution alignment. Distribution alignment methods, especially those developed in the domain adaptation (DA) literature, have gained increasing traction in astrophysics for mitigating dataset shifts and cross-survey domain gaps [13, 14, 15]. Such methods have been applied to tasks including galaxy merger classification [13], semi-supervised universal domain adaptation for cross-survey morphology classification and anomaly detection (DeepAstroUDA; [15]), and strong gravitational lensing modeling [1]. Whereas much of this work focuses on inter-domain transfer—aligning feature representations across distinct datasets—the same distribution-alignment principle can be adapted to improve distributional fidelity within a single survey sample. In our setting, aligning the predicted joint parameter distribution with the distribution of observed spectroscopic labels can help mitigate the variance shrinkage and tail under-representation associated with pointwise regression losses.
Among the statistical distances and divergences used in the DA literature—such as optimal transport (OT; [58]), especially the Wasserstein distance, and Kullback–Leibler (KL) divergence—maximum mean discrepancy (MMD; [32, 60]) offers practical advantages for multivariate parameter regression. Unlike KL objectives that rely on explicit density estimation, MMD avoids explicit mini-batch density estimation, which is often unstable for small batches or in higher dimensions. Compared with OT, which typically requires solving a linear program or Sinkhorn iterations at every training step, MMD has a closed-form kernel-based estimator that can be integrated directly into standard GPU mini-batch gradient descent, and it incurs lower computational overhead. Moreover, a mixture of Gaussian kernels at multiple scales allows MMD to penalize both local manifold discrepancies and broader distributional mismatches. This makes MMD a computationally efficient, stable, and smooth regularization term for better reproducing the breadth and bimodal or multimodal structure of the observed target distribution during continuous multi-task training.
In this paper, we propose ViTMultiTask—a statistically grounded multi-task learning framework that jointly infers , , , and directly from multiband photometric images. By introducing task-specific tokens into a vision transformer backbone and incorporating a multi-scale MMD penalty, our framework encourages the predicted joint distribution to align with the empirical joint distribution of the spectroscopic labels in a reproducing kernel Hilbert space (RKHS; [62]). We train ViTMultiTask on 83,846 five-band () Subaru Hyper Suprime-Cam (HSC) image cutouts cross-matched with high-quality spectroscopic labels from the Dark Energy Spectroscopic Instrument [33] Data Release 1 Bright Galaxy Survey (BGS), and show that MMD regularization effectively mitigates variance shrinkage, more closely reproducing the observed distributions of the spectroscopic labels (including bimodal structures and heavy-tailed populations) and maintaining catastrophic outlier rates comparable to those of the baseline, with only a marginal cost in point-estimate root mean square error (RMSE).
This paper is structured as follows: Section 2 describes the observational data and preprocessing pipeline; Section 3 details the ViTMultiTask architecture and the MMD-constrained loss function; Section 4 presents inference performance, distribution recovery, and ablation study results; Section 5 summarizes the main conclusions and outlines future applications to CSST, Euclid, and LSST.
Data and sample construction
To assemble a benchmark dataset containing both multi-band astronomical imaging and high-fidelity physical property labels, we cross-match and curate samples from two recent data products derived from the DESI Data Release 1 [18]. The imaging components are sourced from the GalaxiesML-Spectra dataset3 (Do et al. 2024; Himes et al. 2025), which spatially cross-matches spectroscopic targets from DESI Data Release 1 with five-band () imaging from the public data release 2 (PDR2; [2]) of Hyper Suprime-Cam Subaru Strategic Program (HSC-SSP; [3]). HSC-SSP offers exceptionally deep optical imaging with superb spatial resolution (median seeing ) and high signal-to-noise ratio across all bands, enabling the extraction of fine morphological structures and color gradients essential for stellar population inferences. The initial base catalog was constructed by selecting targets with clean spectroscopic redshifts (ZWARN ; Hahn et al. 2023), filtering out inconsistent redshifts and duplicate observations, aligning within a matching radius, and excluding stars classified by DESI, yielding a foundational sample of 134,533 galaxy images. Although the original GalaxiesML-Spectra release includes one-dimensional spectra and metadata, we strictly utilize its pixel five-band HSC image cutouts and unique target identifiers (TARGETID) for catalog cross-matching.
Ground-truth astrophysical labels are derived from the DESI Data Release 1 BGS [33] Value-Added Catalog 2 (VAC; Li et al. 2026), which covers approximately 6.7 million galaxy spectra. This VAC executes systematic stellar population modeling via a full-spectral fitting pipeline, deriving comprehensive physical parameters including spectroscopic redshift (), total stellar mass (), light-weighted stellar age (), stellar metallicity (), stellar velocity dispersion (), dust attenuation , spectral indices (e.g., break and H Balmer absorption line equivalent width), as well as flux and equivalent widths of major optical emission lines.
We cross-match the HSC imaging dataset with the DESI Data Release 1 BGS VAC using the unique TARGETID, securing 84,284 initial matches. To ensure physical reliability and eliminate non-physical outliers, we apply range constraints on the derived parameters: , (i.e., ), , and Å. This quality cut discards only 438 unphysical anomalies ( rejection rate), resulting in a clean dataset of 83,846 galaxies. Each object contains high-resolution imaging alongside robust spectroscopic labels (, , , and ).
We prioritize , , , and as our primary target variables, which together form a multi-dimensional diagnostic system characterizing galaxy state and evolutionary pathway [40, 51, 45]. establishes the cosmological distance scale and rest-frame spectral coverage; governs the depth of the gravitational potential well and long-term evolutionary trajectory; is sensitive to old stellar populations and traces mean light-weighted stellar age, though it is also degenerate with metallicity; and equivalent width spikes violently within 0.1–1 Gyr following a starburst, acting as a crucial tracer of recent star formation activity and rapid quenching episodes.
Direct inference of these high-dimensional parameters from multiband imaging is supported by a well-established physical basis. Broadband colors (e.g., , ) offer constraints on low-resolution SEDs, which are generally informative for estimating , , and . At the same time, image cutouts resolve fine spatial features—such as angular size, concentration, radial color gradients, and structural perturbations [17]—that are often physically linked to star formation history and violent evolutionary events (e.g., core structural compaction frequently accompanies enhanced absorption; [19]). Deep neural networks are capable of mapping these nonlinear pixel-level spatial-color patterns to small-scale spectroscopic diagnostics [11, 27], thereby facilitating cross-modal parameter inference.
As illustrated in Figure Fig. 1, the final 83,846 galaxies span a broad and representative parameter space: redshifts are concentrated within , consistent with the selection function of the low-redshift DESI BGS; stellar mass follows a slightly left-skewed, near-Gaussian distribution with a median of . Furthermore, exhibits a prominent bimodal distribution, whereas displays a right-skewed heavy tail, jointly reflecting a diverse population comprising star-forming blue galaxies, green-valley transition systems, and quenched red sequence galaxies. For neural network training and evaluation, we randomly partition the dataset into training (73,846 galaxies), validation (5,000 galaxies), and test (5,000 galaxies) subsets, corresponding to an approximate ratio of 88 : 6 : 6. This partitioning strategy maximizes the training sample size required to effectively optimize the high-capacity vision transformer backbone, while reserving a test set large enough for overall performance estimates. By adopting uniform random sampling, the test set is, by construction, an unbiased random subset of the parent population. A comparison of the parent and test distributions shows no significant bias beyond expected sampling fluctuations across the main parameter ranges and the targeted tail regions, although the absolute counts in the extreme tails remain limited by the parent sample size.
Methodology and training pipeline
Data Preprocessing and Label Standardization
Before model training, we applied a rigorous data preprocessing and standardization pipeline to the multiband Subaru HSC images and their corresponding spectroscopic physical labels. Notably, no additional per-channel scaling or cross-band normalization was applied to the images, so as to directly preserve the intrinsic physical flux ratios and color gradient information across bands. This choice was made to retain the inherent color diagnostic power, while the input dynamic range is handled by the network’s adaptive weight mechanisms during training.
For the physical labels, given that stellar mass spans several orders of magnitude, we first applied a logarithmic transformation, , to compress its dynamic range. In contrast, redshift (typically in our sample), the index, and the equivalent width are intrinsically linear-scale quantities with relatively compact empirical distributions, thus requiring no additional non-linear compression.
To strictly prevent data leakage, we first split the full set of 83,846 samples into training, validation, and test sets, containing 73,846, 5,000, and 5,000 samples, respectively (approximately 88%, 6.0%, and 6.0% of the total). The validation set was used solely for hyperparameter tuning and selection of the best model weights, while the test set was reserved exclusively for final inference evaluation; neither participated in gradient updates during training. Subsequently, we computed the mean and standard deviation of the four physical labels exclusively from the training set samples, and applied Z-score standardization:
where and are the mean and standard deviation of the -th label in the training set. This standardization removes the interference caused by different numerical scales among the physical targets on multi-task gradient balancing, and significantly improves the convergence stability of multi-task learning.
For the validation and test sets, we strictly applied the same training-set statistics for normalization and inverse transformation () during inference, ensuring strict self-consistency across cross-dataset evaluation.
ViTMultiTask architecture and task tokens
We construct ViTMultiTask, a deep multi-task network integrating a vision transformer (ViT; Dosovitskiy et al. [26]) backbone with a task-specific token mechanism (Figure 2). The overall architecture proceeds from input embedding through transformer encoding to task-specific prediction.

Figure 2. Architecture of the multi-task galaxy parameter prediction network (ViTMultiTask). Left: Overall multi-task prediction workflow. After mapping the multiband image cutout into patch embeddings, four learnable task tokens – are prepended to the sequence and element-wise added with positional embeddings before being fed into the transformer encoder backbone for feature extraction and representation decoupling. Finally, the four decoupled task tokens are routed into their respective heterogeneous MLP prediction heads to concurrently infer , , , and . Right: Internal structure of a single transformer encoder block, comprising layer normalization, multi-head attention, and MLP feed-forward layers with residual connections, stacked across layers.
Taking image cutouts as input, the backbone first maps image patches into sequence embeddings via linear projection of flattened patches. In practice, this is implemented using convolutional kernels with a stride of 4 and zero-padding, which downsamples the spatial dimensions to and yields 256 spatial patch vectors with an embedding dimension of .
To effectively decouple features across the four target variables and eliminate the need for global average pooling, we introduce four learnable task-specific tokens , a strategy inspired by recent advances in multi-task vision transformers [64]. These tokens are prepended to the spatial patch sequence, forming an input sequence of length . A learnable 1D position embedding is added across the entire sequence, enabling the task tokens to attend to spatial semantics from the first layer onward.
Figure 1. Parameter distributions of the four target physical properties across the galaxy sample: (a) redshift ; (b) stellar mass ; (c) index; and (d) H absorption line equivalent width. Vertical dashed black lines in each panel indicate the median value of the corresponding parameter.
The backbone comprises transformer blocks. Each block incorporates a 4-head self-attention module with a head dimension of , an MLP feed-forward network with an expansion factor of 4 (hidden dimension 320), layer normalization, and dropout (). Through multi-layer self-attention interactions, the four task tokens at the sequence head are dynamically updated by aggregating information from all spatial patches, effectively decoupling representations for each target without spatial compression.
After the final transformer layer, these task tokens are extracted and routed to task-specific prediction heads. The heads adopt heterogeneous architectures tailored to the distinct physical characteristics of each target variable. For global, scale-dependent quantitiesredshift and stellar masswhich are primarily governed by overall SED shapes and galaxy sizes, we employ two-layer fully connected networks with dimensions 256 and 192, using ReLU activations [31] and layer normalization (Lei Ba et al. 2016).
For , which probes the 4000 Å break and is sensitive to stellar population age, we use a [256, 128] architecture with GELU activations [34] and batch normalization [36], leveraging mini-batch statistics to enhance sensitivity to break strength. Finally, to extract subtle absorption-line signatures from broadband imaging, the H head adopts a deeper three-layer structure (512, 256, 128) with GELU activations and dual batch normalization, providing the high non-linear capacity needed to resolve signatures of recent starbursts and rapid quenching episodes.
Joint loss function with MMD manifold alignment
In galaxy astrophysics, the four target parameters are not independent scalar variables; rather, they are intrinsically coupled through stellar population synthesis (SPS; Bruzual & Charlot [10]) and star formation histories (SFHs). Specifically, and trace a tight, non-linear evolutionary manifold (Kauffmann et al. [40]): rapidly peaks in post-starburst galaxies before decaying as the population ages, while generally rises with mean stellar age (though it remains degenerate with metallicity). Concurrently, dictates the gravitational potential depth and gas retention, while sets the cosmic epoch and redshifted spectral window, jointly modulating the evolutionary timescale and flux normalization of the SED. Jointly modeling these parameters ensures that ViTMultiTask respects this intrinsic astrophysical manifold, preventing the non-physical decoupling that typically arises from independent single-target regressions.
Standard point-wise loss functions (e.g., MSE or Huber loss alone) treat target variables independently and ignore cross-variable covariances, inducing variance shrinkage and oversmoothing of rare evolutionary phases. To improve fidelity to the observed spectroscopic-label distribution, we construct a joint loss function combining a weighted point-wise Huber loss with a maximum mean discrepancy (MMD; Gretton et al. [32]) distribution alignment penalty. All losses are computed in the standardized target space (see § 3.1) to ensure that the MMD kernel operates on unit-variance dimensions:
where and denote the standardized spectroscopic labels and predicted values (), respectively, and (with threshold ) ensures point-estimate robustness against labeling noise and outliers. The task weight vector is set to for , , , and , respectively. The higher weights assigned to and reflect their relatively smaller dynamic ranges and lower per-instance signal-to-noise ratios compared to redshift and mass, ensuring that the gradient contribution from these spectroscopically sensitive indices is not dominated by the more readily predicted global parameters.
The MMD term penalizes discrepancies between the predicted and observed joint distributions in a reproducing kernel Hilbert space (RKHS; Sriperumbudur et al. [62]), acting on the standardized four-dimensional target space . Here, the MMD term is applied to the empirical distribution of the spectroscopic labels, , rather than to an unobservable intrinsic physical distribution . Because these labels carry measurement uncertainties, spectral-fitting systematics, and selection effects, agreement with should be interpreted as improved fidelity to the observed spectroscopic-label distribution, not as exact recovery of the underlying physical distribution. Specifically, we minimize the squared MMD:
where and denote independent and identically distributed (i.i.d.) draws from the respective target distributions. We adopt a multi-scale Gaussian radial basis function (RBF) mixture kernel with bandwidths . Because all physical targets are standardized to unit variance prior to loss computation, the values in are expressed in units of the standard deviation of each target dimension. The chosen range (0.1 to 5.0) spans scales from fine-grained feature structures to broad population-level variations, providing comprehensive multi-scale coverage and reducing the need for dataset-specific kernel tuning. By constraining higher-order statistical moments, MMD preserves physical covariances—such as the non-linear – evolutionary sequence—that would otherwise be smoothed out by point-wise losses alone.
To determine the MMD loss weight, we performed a sensitivity analysis over . The results indicate that is insufficient to mitigate variance shrinkage, whereas may over-emphasize global distribution alignment at the cost of increased point-wise RMSE penalty (). We therefore set , which provides a favorable balance between point-estimate precision and distribution alignment.
Training Protocol and Data Augmentation
The network is implemented in TensorFlow and trained on a single NVIDIA L40S GPU. To improve generalization and enforce robustness against the arbitrary orientation of galaxies on the sky, we apply spatial symmetry augmentations during training, including random horizontal and vertical flips alongside rotations. These operations correspond to the dihedral symmetry group and encourage the model to learn orientation-invariant representations, preventing overfitting to particular morphological alignments in the sky plane.
We train the model using the Adam optimizer (Kingma & Ba 2014) with a batch size of 256 for a total of 300 epochs, which completes in approximately 2.5 hours on our hardware setup. With a training set size of 73,846 samples, each epoch comprises approximately 289 iterations. The initial learning rate is set to and decays following a cosine schedule over the first 50,000 steps (corresponding to roughly 173 epochs) down to a terminal value of (1% of the initial rate). For the remaining 127 epochs, the learning rate is held constant at to refine convergence near the loss minimum.
The validation loss is monitored after each epoch, and the model checkpoint achieving the lowest validation loss is retained for final evaluation. During inference, the normalized predictions are transformed back to physical units using the training-set mean and standard deviation , strictly maintaining consistency with the label standardization protocol described in Section 3.1.
Results and Astrophysical Discussion
Evaluation Metrics
To evaluate the predictive accuracy and distributional fidelity of our multi-task pipeline, we assess the model on an independent test set of galaxies. For physical parameters , we define the prediction residual for the -th galaxy as , where and denote the spectroscopic truth and predicted value, respectively. For redshift (), following standard cosmological practice, we adopt the normalized residual (Brammer et al. 2008), where represents the redshift truth . Unless otherwise specified, all residual-based metrics (RMSE, MAE, , and outlier rates) for redshift are computed using , while for all other parameters they are computed using the absolute residual .
Point-wise statistical performance is quantified using the root mean square error (RMSE) and mean absolute error (MAE):
To suppress sensitivity to catastrophic outliers, we employ the normalized median absolute deviation (), a robust scatter estimator widely used in observational cosmology (Brammer et al. 2008; Luo et al. 2026b):
where the normalization factor 1.4826 ensures equivalence to the standard deviation for a pure Gaussian distribution. While RMSE reflects global prediction errors, specifically quantifies the intrinsic scatter of the core sample population without contamination from extreme tails.
The catastrophic outlier fraction (, expressed as a percentage) quantifies predictions deviating beyond strict tolerance thresholds:
where is a binary outlier indicator. For redshift, a prediction is classified as catastrophic if the normalized residual satisfies , a threshold commonly adopted in photometric redshift studies (e.g., Brammer et al. 2008; Euclid Collaboration et al. 2020; Euclid Collaboration: et al. 2025). For the remaining physical parameters, we adopt a unified threshold for each parameter, defined as . Here is a fixed reference scatter determined from the ViTMultiTask with MMD model on the test set. The indicator is formally defined as:
In addition to point-wise residual metrics, global distribution fidelity and multi-parameter physical covariances are evaluated using one- and two-dimensional kernel density estimation (1D/2D KDE). For 1D KDE, we compare the predicted and observed spectroscopic-label distributions across all four physical targets to assess overall profile consistency and to examine whether the predicted parameters suffer from distributional distortion, variance shrinkage, or tail truncation. For multi-parameter physical coupling, we construct 2D KDE contour overlays across multiple enclosed probability quantiles, focusing in particular on key astrophysical scaling relations—including the non-linear – evolutionary manifold and the underlying – distribution—to quantify how accurately the predicted joint probability density reproduces the observed spectroscopic labels.
To quantify the statistical uncertainties of the reported metrics, we estimate uncertainties for all continuous metrics (RMSE, MAE, and ) from 1,000 bootstrap resamples of the test set. For the catastrophic outlier fractions (), uncertainties are estimated using Poisson/binomial statistics.
Multi-task parameter estimation performance
Figure 3 presents the scatter comparisons between predicted physical parameters and spectroscopic labels across the 5,000 test-set galaxies. Quantitative metrics for all parameters are summarized in Table 1. To establish a consistent evaluation benchmark throughout this study, the unified outlier thresholds for non-redshift physical parameters, , are defined using the test-set scatter of the full ViTMultiTask model (with MMD) as the reference, as described in § 4.1.
Figure 3. Predicted physical parameters versus spectroscopic labels for the ViTMultiTask model on an independent test set of 5,000 galaxies. The four panels display: (a) redshift , (b) stellar mass , (c) break index, and (d) H equivalent width [Å]. The black dashed line represents the 1 : 1 identity line, blue circles denote well-fitted targets, and red crosses indicate catastrophic outliers. Corresponding evaluation metrics (RMSE, MAE, , and outlier fraction ) are annotated in each panel.
| Parameter | MMD | RMSE | MAE | (%) | |
| w/o | 0.0225 ± 0.0007 | 0.0165 ± 0.0002 | 0.0194 ± 0.0003 | 0.08 ± 0.04 | |
| w/ | 0.0223 ± 0.0004 | 0.0166 ± 0.0002 | 0.0195 ± 0.0004 | 0.06 ± 0.03 | |
| (dex) | w/o | 0.211 ± 0.004 | 0.152 ± 0.002 | 0.163 ± 0.003 | 2.94 ± 0.24 |
| w/ | 0.218 ± 0.004 | 0.157 ± 0.002 | 0.169 ± 0.003 | 3.16 ± 0.25 | |
| w/o | 0.093 ± 0.002 | 0.067 ± 0.001 | 0.072 ± 0.002 | 2.98 ± 0.24 | |
| w/ | 0.094 ± 0.002 | 0.067 ± 0.001 | 0.073 ± 0.001 | 3.18 ± 0.25 | |
| H (Å) | w/o | 1.350 ± 0.019 | 1.006 ± 0.013 | 1.118 ± 0.021 | 2.08 ± 0.20 |
| w/ | 1.373 ± 0.018 | 1.036 ± 0.012 | 1.185 ± 0.022 | 1.96 ± 0.20 |
Table 1. Ablation comparison of model performance with and without MMD loss constraint.
For redshift , the model achieves the highest point-wise precision among the four tasks, with RMSE = 0.0223, MAE = 0.0166, = 0.0195, and a virtually zero catastrophic outlier fraction of = 0.06%. Predictions exhibit no systematic bias across the core range , with only a marginal overestimation trend at where the training sample becomes sparse. This high performance reflects the strong constraining power of five-band photometric colors in breaking color–redshift degeneracies.
Stellar mass is arguably the most dynamically demanding target, spanning nearly five orders of magnitude (). Despite this broad range, the model delivers RMSE = 0.218 dex, MAE = 0.157 dex, = 0.169 dex, and = 3.16%. This compact scatter indicates that ViTMultiTask effectively extracts non-linear mappings linking broadband photometry, spatial morphology, and color gradients to global stellar mass, consistent with well-established scaling relations for low-redshift galaxies ([7, 8, 40]).
For , a classic tracer of mean stellar population age, the model yields RMSE = 0.093, MAE = 0.067, = 0.073, and = 3.18% over the domain . A slight systematic underestimation emerges in the old-galaxy tail (), likely reflecting the training set’s under-representation of such objects, which biases the network toward the more densely sampled intermediate-age regime. Notably, this deviation is confined to the extreme tail and does not affect the model’s overall reliability across the main population ().
The Balmer absorption index presents the most challenging task, given its extreme sensitivity to recent starburst quenching episodes, and potential emission-line infill. Nevertheless, the model attains RMSE = 1.373 Å, MAE = 1.036 Å, = 1.185 Å, and = 1.96% over Å. This fidelity suggests that the task-specific token mechanism effectively preserves localized spatial information, allowing the network to capture subtle absorption signatures from broadband imaging without the feature-averaging losses inherent to global pooling.
Overall, outliers across all four parameter spaces remain sparse and randomly scattered, with no obvious un-physical systematic trends. These results demonstrate that the proposed architecture robustly handles multi-task regression across physically distinct observables, providing a viable pathway for inferring high-dimensional astrophysical parameters—traditionally requiring high-signal-to-noise spectroscopy—directly from imaging alone.
Ablation study: Impact of MMD manifold alignment
To isolate the role of MMD manifold alignment, we compare two model configurations that are identical in all architectural and training aspects except for the loss function: a baseline trained with pure Huber loss (), and the full model with MMD regularization (). Throughout this section, agreement with the observed spectroscopic labels () indicates improved fidelity to the empirical target distribution, rather than exact recovery of the underlying intrinsic physical distribution (). Quantitative results are summarized in Table 1, and kernel density estimates (KDE) of the predicted distributions against spectroscopic labels are illustrated in Figure 4 and Figure 5.
Figure 4. One-dimensional kernel density estimates (1D KDE) of predicted physical parameters, comparing predictions without MMD (blue dashed line) and with MMD (red solid line) against the spectroscopic labels (gray filled area). Panels display: (a) redshift , (b) stellar mass , (c) , and (d) H [Å]. The baseline model without MMD exhibits severe variance shrinkage, resulting in unphysical over-concentrated peaks and truncated distribution tails (notably in panels c and d). In contrast, incorporating MMD better reproduces the variance, multimodal structure, and heavy tails present in the observed spectroscopic-label distributions.
Figure 5. Two-dimensional kernel density estimates (2D KDE) evaluating joint physical parameter distribution fidelity. Top panel: H [Å] versus D evolutionary manifold; bottom panel: D versus stellar mass bimodal distribution. Spectroscopic labels are indicated by gray shaded regions with four discrete intensity gradient levels. Predictions from models trained without MMD (blue dashed contours) and with MMD (red solid contours) are overlaid at the 20%, 40%, 60%, and 80% enclosed probability quantiles, demonstrating how MMD manifold alignment reproduces the observed parameter couplings in the spectroscopic label and mitigates variance shrinkage.
Unconstrained point-wise losses (e.g., Huber loss) encourage network predictions toward conditional expectations , inducing "variance shrinkage" and regression to the mean. As summarized in Table 1, the baseline model without MMD achieves marginally lower point-wise residual metrics for stellar mass and spectral indices (e.g., RMSE = 0.211 dex for and 0.093 for ). However, as demonstrated in Figure 4, this marginal numeric gain comes at the cost of severe distribution distortion. Without MMD, the predicted distributions artificially narrow, producing unphysical sharp peaks around modal nodes (e.g., , Å) and truncating distribution tails in extreme parameter regimes (e.g., failing to capture the extended negative tail of Å and over-concentrating stellar mass near ).
By contrast, the multi-scale MMD loss encourages the predicted joint distribution to align with the target joint distribution within the RKHS framework. This effectively mitigates variance shrinkage, better preserving the multimodal shapes, predictive dispersion, and heavy-tail structures observed in the spectroscopic labels, which are critical for astrophysical interpretation. Under this standardized benchmark, the MMD-regularized model introduces no significant degradation in point-estimate accuracy (RMSE varies by across all physical quantities and remains statistically consistent with the baseline) and yields comparable outlier rates. Specifically, for the stellar-population tracer , the MMD model achieves a slightly lower outlier fraction than the baseline (1.96% vs. 2.08%), although the difference is not statistically significant. For global parameters such as stellar mass and , the outlier fractions remain consistent within Poisson/binomial uncertainties (3.16% vs. 2.94% for ; 3.18% vs. 2.98% for ; MMD vs. baseline). Overall, MMD delivers clear gains in global distribution fidelity—a property of particular value for cosmological applications such as galaxy stellar mass functions and galaxy evolution modeling.
To further evaluate the model’s fidelity in preserving multi-dimensional stellar population correlations, Figure 5 presents the predicted 2D joint probability density contours for the H versus evolutionary manifold (top panel) and the versus sequence (bottom panel). The spectroscopic labels (gray shaded regions) clearly delineates the characteristic bimodal distribution of galaxy populations, spanning from young, star-forming galaxies (low , high H, lower mass) in the upper-left region of the joint parameter spaces to old, quenched systems (high , negative H, higher mass) in the quiescent sequence.
While both model configurations capture the primary bimodal structure, the benefits of MMD regularization become clear when examining the two joint spaces separately. For the H– evolutionary manifold (top panel), the spectroscopic labels traces a characteristic curved sequence from post-starburst galaxies (high H, low ) down to old, quiescent systems (negative H, high ). The baseline model trained without MMD (blue dashed contours) captures the central ridge but compresses the outer low-density envelopes, struggling to reproduce the extended tails at both ends of the evolutionary track. Incorporating MMD regularization (red solid contours) expands the predicted contours outward across the 20%–80% enclosed probability quantiles, achieving substantially better alignment with the observed distribution across the full dynamic range—particularly in recovering the high-H tail that traces recent starburst activity.
For the – joint distribution (bottom panel), the observed labels exhibit a well-known bimodal structure: a low-mass star-forming cloud (, low ) and a high-mass red sequence (, high ), connected by a relatively sparsely populated green valley. The baseline model again displays variance shrinkage, over-concentrating predictions around the modal peaks and under-representing the low-density bridge and the low-mass end of the star-forming sequence. Incorporating MMD expands the predicted contours toward the observed distribution boundaries, recovering the full bimodal geometry and providing more faithful coverage of the transitional green-valley population. Taken together across both joint spaces, the MMD-constrained model Notes. Reported values include continuous metrics alongside their statistical uncertainties, estimated via 1,000 bootstrap resamplings for RMSE, MAE, and , and Poisson/binomial uncertainties for catastrophic outlier fractions (). For a fair comparison, outlier rates () for both models are evaluated using a common threshold () adopted from the ViTMultiTask (w/ MMD) model as the reference.
Notes. Lower values indicate better agreement with the observed spectroscopic label distribution. Bold entries highlight the better-performing model configuration for each metric.
can effectively capture the non-linear evolutionary manifold between Balmer absorption, 4000 Å break strength, and stellar mass, while preserving the coupling between these parameters observed in the spectroscopic labels that point-wise losses alone tend to smooth over.
Regarding density estimation, all 1D and 2D KDE distributions in Figure 4 and Figure 5 are generated using Gaussian kernels. For 1D KDEs, bandwidths are selected using Scott’s rule of thumb. For 2D KDEs, a minor adjustment (bw_adjust = 0.8) is applied relative to Scott’s rule to resolve fine bivariate manifold structures. Importantly, the same smoothing parameters and density thresholds are applied consistently to both the spectroscopic labels and model predictions, ensuring that the visual comparison is not dominated by bandwidth selection.
To quantitatively supplement the kernel density visualizations, we evaluate distribution alignment for all individual parameters and selected joint spaces using three metrics: the Wasserstein-1 distance (; [50] (Arjovsky et al. 2017)), the Kolmogorov–Smirnov statistic (), and the multivariate MMD distance () in Table 2. We denote MMD values for 1D marginals and 2D joint distributions as and , respectively. Compared with the baseline model without MMD, incorporating MMD regularization yields substantial reductions in distribution discrepancies, particularly for spectroscopically sensitive features. For , decreases by 50.6% (0.0251 → 0.0124) and decreases by 48.7%. For H, decreases by 44.0% (0.3790 → 0.2123), decreases by 68.9% (0.1468 → 0.0456), and decreases by 60.2%. For the overall shape of stellar mass, and also show consistent improvements, while their distances remain essentially unchanged (e.g., 0.0141 → 0.0130 for and 0.0610 → 0.0633 for ). In the 2D joint spaces, MMD regularization reduces by 33.6% for the [H, ] evolutionary manifold and by 22.0% for the [, ] bimodal sequence, indicating improved consistency with the observed distribution contours.
| Parameter | Wasserstein-1 () w/o MMD | Wasserstein-1 () w/ MMD | Kolmogorov–Smirnov () w/o MMD | Kolmogorov–Smirnov () w/ MMD | 1D MMD () w/o MMD | 1D MMD () w/ MMD |
| 0.0141 | 0.0130 | 0.0578 | 0.0560 | 0.1093 | 0.0973 | |
| (dex) | 0.0610 | 0.0633 | 0.0742 | 0.0624 | 0.1123 | 0.1023 |
| 4000 | 0.0251 | 0.0124 | 0.0666 | 0.0334 | 0.1283 | 0.0658 |
| H (Å) | 0.3790 | 0.2123 | 0.1468 | 0.0456 | 0.2430 | 0.0966 |
| 2D Joint Distribution MMD Distance () | w/o MMD | w/ MMD | ||||
| [, 4000] Evolutionary Manifold | 0.2321 | 0.1542 | ||||
| [ 4000, ] Bimodal Sequence | 0.1206 | 0.0941 |
Table 2. Quantitative evaluation of 1D marginal and 2D joint distribution agreement between predicted parameter distributions and the observed spectroscopic label distribution on the independent test set ().
A slight residual gap persists between the outermost MMD contours and the observed distribution boundary. This minor offset is primarily attributed to two factors. First, intrinsic observational noise and measurement uncertainties in the spectroscopic estimates introduce additional broadening into the empirical label distribution, making its outermost boundary inherently difficult to reproduce exactly from imaging alone. Second, the joint optimization of point-wise Huber regression and global MMD regularization entails a fundamental trade-off: prioritizing distribution alignment can slightly compromise per-sample point accuracy, and vice versa. One promising direction for future work would be to directly incorporate measurement uncertainties into the distribution alignment loss, which may help further narrow this residual gap.
Summary and Conclusions
We have presented ViTMultiTask, a multi-task deep learning framework that integrates a vision transformer (ViT) backbone with task-specific tokens for feature decoupling, optimized via a multi-scale maximum mean discrepancy (MMD) loss for joint distribution alignment. The framework is designed to directly infer photometric redshifts () and stellar population parameters (, , and H) from Subaru HSC five-band (, , , , ) imaging cutouts. By encouraging alignment between predicted and observed spectroscopic-label distributions in a reproducing kernel Hilbert space (RKHS), the MMD loss effectively mitigates the variance shrinkage and distribution distortion inherent to conventional point-wise regression losses.
Evaluated on 83,846 galaxies cross-matched between HSC PDR2 and DESI Data Release 1 BGS, the network demonstrates robust cross-modal mapping from broadband imaging to high-resolution spectroscopic diagnostics. On an independent test set of 5,000 galaxies, the pipeline achieves (RMSE , ) for redshift and dex (RMSE dex) for stellar mass. For the spectroscopic indices, yields RMSE , (), while H achieves RMSE Å, Å (). These results indicate that the ViT backbone effectively extracts spatially resolved morphological and color gradient information directly from multi-band imaging.
Ablation experiments highlight the central role of MMD-based distribution alignment. Although the baseline model trained with Huber loss alone achieves marginally lower point-wise residuals for certain parameters, it exhibits pronounced variance shrinkage and fails to reproduce the observed bimodal structures and heavy tails in the empirical distributions of the spectroscopic labels. In 1D kernel density estimates, the baseline artificially narrows the predicted distributions, producing over-concentrated peaks around modal nodes and truncating extreme tails (e.g., the extended negative tail of H and the highmass end of . In 2D joint spaces, it compresses the H–D evolutionary manifold and under-represents the low-mass blue cloud and green-valley transition in the D– plane. Incorporating MMD regularization () incurs no significant degradation in pointwise precision (changes in RMSE of , statistically consistent with the baseline) while clearly improving global distribution fidelity. For large-scale cosmological surveys, improving agreement with observed global parameter distributions (while accounting for target sample uncertainties) is more crucial for reducing bias in statistical inference than solely minimizing per-sample residuals.
Minor systematic underestimations persist at high D () and at the highest redshifts (), reflecting the intrinsic sparsity of massive, quenched systems in the training sample.
Future work will explore self-supervised pre-training strategies (such as Masked Autoencoders, MAE) to enhance feature extraction for rare populations and further improve robustness in data-sparse regions. More broadly, applying this manifold alignment paradigm to upcoming wide-field surveys—such as CSST, Euclid, and LSST—offers a promising pathway toward reliable, large-scale galaxy property estimation for the billions of photometrically observed targets that will lack spectroscopic coverage, fully addressing the spectroscopically limited nature of the deep imaging era.
Acknowledgements. Z.L. acknowledges the support from the National Natural Science Foundation of China (Grant No. 12573009) and the scientific research grants from the China Manned Space Project with Grant Nos. CMS-CSST-2025-A07 and CMS-CSST-2025-A05. This work is also supported by the National Natural Science Foundation of China under Grant No. 12141302.
Data availability
The code for ViTMultiTask supporting the findings of this study is publicly available on GitHub at https://github.com/zjluo-code/ViMultitask. The HSC imaging data and spectroscopic redshift labels were derived from Himes et al. [35] (https://zenodo.org/records/16989593). The target physical parameters, including stellar mass (), D, and H, were obtained from ScienceDB (Li et al. 2026) (http://www.doi.org/10.57760/sciencedb.35360). The curated training, validation, and test datasets compiled for this work are archived on Zenodo at https://doi.org/10.5281/zenodo.24154596 and will be made publicly accessible upon publication.
References
- [1]Agarwal, S., Ćiprijanović, A., & Nord, B. D. 2024, arXiv e-prints, arXiv:2411.03334arxiv.org/abs/2411.03334
- [2]Aihara, H., Aihara, Y., Ando, M., et al. 2019, PASJ, 71, 114
- [3]Aihara, H., Arimoto, N., Armstrong, R., et al. 2018, PASJ, 70, S4
- [5]Arnouts, S., Cristiani, S., Moscardini, L., et al. 1999, MNRAS, 310, 540
- [6]Baldry, I. K., Glazebrook, K., Brinkmann, J., et al. 2004, ApJ, 600, 681
- [7]Bell, E. F. & de Jong, R. S. 2001, ApJ, 550, 212
- [8]Bell, E. F., McIntosh, D. H., Katz, N., & Weinberg, M. D. 2003, ApJS, 149, 289
- [9]Brammer, G. B., van Dokkum, P. G., & Coppi, P. 2008, ApJ, 686, 1503
- [10]Bruzual, G. & Charlot, S. 2003, MNRAS, 344, 1000
- [11]Buck, T. & Wolf, S. 2021, arXiv e-prints, arXiv:2111.01154arxiv.org/abs/2111.01154
- [12]Cao, Y., Gong, Y., Meng, X.-M., et al. 2018, MNRAS, 480, 2178
- [13]Ćiprijanović, A., Kafkes, D., Downey, K., et al. 2021, MNRAS, 506, 677
- [14]Ćiprijanović, A., Kafkes, D., Snyder, G., et al. 2022, Machine Learning: Science and Technology, 3, 035007
- [15]Ćiprijanović, A., Lewis, A., Pedro, K., et al. 2023, Machine Learning: Science and Technology, 4, 025013
- [16]Conroy, C. 2013, ARA&A, 51, 393
- [17]Conselice, C. J. 2014, ARA&A, 52, 291
- [18]CSST Collaboration, Gong, Y., Miao, H., et al. 2026, Science China Physics, Mechanics, and Astronomy, 69, 239501
- [19]Dekel, A. & Burkert, A. 2014, MNRAS, 438, 1870
- [20]DESI Collaboration, Abareshi, B., Aguilar, J., et al. 2022, AJ, 164, 207
- [22]DESI Collaboration, Adame, A. G., Aguilar, J., et al. 2024a, AJ, 167, 62
- [24]D’Isanto, A. & Polsterer, K. L. 2018, A&A, 609, A111
- [26]Dosovitskiy, A., Beyer, L., Kolesnikov, A., et al. 2020, arXiv e-prints, arXiv:2010.11929arxiv.org/abs/2010.11929
- [27]Euclid Collaboration, Bisigello, L., Conselice, C. J., et al. 2023, MNRAS, 520, 3529
- [31]Glorot, X., Bordes, A., & Bengio, Y. 2011, in Proceedings of the fourteenth international conference on artificial intelligence and statistics, JMLR Workshop and Conference Proceedings, 315–323
- [32]Gretton, A., Sejdinovic, D., Strathmann, H., et al. 2012, Advances in neural information processing systems, 25
- [33]Hahn, C., Wilson, M. J., Ruiz-Macias, O., et al. 2023, AJ, 165, 253
- [34]Hendrycks, D. & Gimpel, K. 2016, arXiv e-prints, arXiv:1606.08415
- [35]Himes, M., Krishnamurthy, S., Lizarraga, A., et al. 2025, arXiv e-prints, arXiv:2510.22527arxiv.org/abs/2510.22527
- [36]Ioffe, S. & Szegedy, C. 2015, arXiv e-prints, arXiv:1502.03167arxiv.org/abs/1502.03167
- [37]Ivezić, Ž., Kahn, S. M., Tyson, J. A., et al. 2019, ApJ, 873, 111
- [38]Jafariyazani, M., Masters, D., Faisst, A. L., Teplitz, H. I., & Ilbert, O. 2024, ApJ, 967, 60
- [39]Jimenez Rezende, D. & Mohamed, S. 2015, arXiv e-prints, arXiv:1505.05770
- [40]Kauffmann, G., Heckman, T. M., White, S. D. M., et al. 2003, MNRAS, 341, 33
- [42]Laureijs, R., Amiaux, J., Arduini, S., et al. 2011, arXiv e-prints, arXiv:1110.3193arxiv.org/abs/1110.3193
- [45]Li, T.-X. & Wu, P.-F. 2025, arXiv e-prints, arXiv:2510.25216arxiv.org/abs/2510.25216
- [46]Lower, S., Narayanan, D., Leja, J., et al. 2020, ApJ, 904, 33
- [48]Luo, Z., Li, Y., Chen, J., et al. 2026a, ApJS, 82, 62
- [50]Luo, Z., Zhang, S., Chen, J., et al. 2025, ApJS, 277, 22
- [51]Luo, Z.-J., Shu, C.-G., & Huang, J.-S. 2007, PASJ, 59, 541
- [52]Madau, P. & Dickinson, M. 2014, ARA&A, 52, 415
- [53]Mucesh, S., Hartley, W. G., Palmese, A., et al. 2021, MNRAS, 502, 2770
- [54]Muzzin, A., Marchesini, D., Stefanon, M., et al. 2013, ApJ, 777, 18
- [55]Pacifici, C., Iyer, K. G., Mobasher, B., et al. 2023, ApJ, 944, 141
- [56]Pasquet, J., Bertin, E., Treyer, M., Arnouts, S., & Fouchez, D. 2019, A&A, 621, A26
- [57]Peng, Y.-J., Lilly, S. J., Kovač, K., et al. 2010, ApJ, 721, 193
- [58]Peyré, G. & Cuturi, M. 2020, Computational Optimal Transport
- [59]Ren, Y., Chan, K. C., Zhang, L., et al. 2026, ApJ, 997, 65
- [60]Schrab, A., Kim, I., Albert, M., et al. 2021, arXiv e-prints, arXiv:2110.15073arxiv.org/abs/2110.15073
- [61]Speagle, J. S., Steinhardt, C. L., Capak, P. L., & Silverman, J. D. 2014, ApJS, 214, 15
- [62]Sriperumbudur, B. K., Fukumizu, K., & Lanckriet, G. R. G. 2010, arXiv e-prints, arXiv:1003.0887arxiv.org/abs/1003.0887
- [63]Strateva, I., Ivezić, Ž., Knapp, G. R., et al. 2001, AJ, 122, 1861
- [64]Tian, Y. & Bai, K. 2024, IEEE Transactions on Neural Networks and Learning Systems, 35, 9579
- [65]Tremonti, C. A., Heckman, T. M., Kauffmann, G., et al. 2004, ApJ, 613, 898
- [66]Yao-Yu Lin, J., Liao, S.-M., Huang, H.-J., Kuo, W.-T., & Hsuan-Min Ou, O. 2021, arXiv e-prints, arXiv:2110.01024
- [67]Zhan, H. 2011, Scientia Sinica Physica, Mechanica & Astronomica, 41, 1441



