Connecting galaxy structure, quenching, and halo environments during cosmic noon

Introduction

Observations of the nearby Universe have revealed a clear bimodality in galaxy colors and star formation activity, with galaxies occupying two distinct populations of actively star-forming and quenched systems [72, 11]. The transition from star-forming to quenched galaxies is particularly prominent among massive galaxies, which exhibit higher quenched fractions and provide important insights into the physical processes responsible for shutting down star formation. Deep observations have revealed that massive quenched galaxies were already in place at $z \sim2-3$ [41], during cosmic noon ($1 \le z < 3$) when the cosmic star formation rate density reached its peak. Understanding the physical processes responsible for quenching massive galaxies remains a central challenge in galaxy evolution. Galaxy quenching is generally considered to result from a combination of internal and external processes. Internal mechanisms regulate star formation by altering the availability and physical state of cold gas within galaxies. Stellar and Active Galactic Nuclei (AGN) feedback can expel gas from galaxies or heat the surrounding gas, reducing the fuel available for star formation (Dekel & Silk 1986; Fabian 2012; Zhang et al. 2021; Chen et al. 2025a,b; Li et al. 2025; Lyu et al. 2026; Yu et al. 2026), while morphological transformation can stabilize gas against gravitational collapse and suppress the formation of new stars [57]. External mechanisms are associated with the surrounding environment, including ram-pressure stripping, merger and tidal interactions, which can remove or redistribute gas reservoirs and thereby suppress star formation [35, 60]. Halo heating prevents efficient gas cooling and suppresses the replenishment of cold gas through accretion [25] (Mo et al. 2024). The relative importance of these mechanisms depends on galaxy properties, environment, and evolutionary stage (Li et al. 2025). For massive galaxies, increasing evidence suggests that internal processes play an important role in regulating quenching, as their deep gravitational potentials make environmental processes less efficient at regulating gas [65, 8], while their structural evolution is closely linked to the shutdown of star formation [32, 4].

The connection between galaxy structure and quenching has been extensively investigated in the local Universe. Using SDSS observations, Kauffmann et al. [44] showed that stellar surface density is more strongly associated with stellar population age than stellar mass, with more compact galaxies preferentially hosting older stellar populations. Subsequent studies linked the internal distribution of stellar mass to star formation activity, showing that galaxies with prominent bulges and higher Sérsic indices are more likely to be quenched [30, 5]. Fang et al. [32] further quantified this relation using the central stellar mass surface density within 1 kpc ($\Sigma_1$). They showed that quenched galaxies preferentially occupy the high-$\Sigma_1$ regime, with the quenched fraction increasing rapidly above a characteristic density threshold. They further demonstrated that $\Sigma_1$ is tightly correlated with central velocity dispersion. Consistent with this picture, Hong et al. [39] showed that central velocity dispersion is closely associated with galaxy quenching, highlighting the role of galaxy dynamical state in regulating star formation. The distribution of galaxies in the $\Sigma_1$–sSFR plane exhibits a L-shaped structure, where quenched galaxies occupy a relatively narrow high-$\Sigma_1$ sequence, whereas star-forming galaxies span a broader range of central densities, including both extended low-density and compact high-density galaxies [74, 4, 61]. At higher redshifts, this structural connection is further supported by the existence of compact quenched galaxies at $z \sim2$, known as “red nuggets” [78, 12], while compact star-forming galaxies at similar epochs exhibit similarly dense and spheroid-like structures [2, 3].

These observational trends have motivated evolutionary scenarios linking different structural states along the quenching pathway. A widely discussed pathway proposes that extended star-forming galaxies undergo structural transformation through dissipative processes, including gas compaction and violent disk instabilities, which drive rapid central stellar mass growth [27, 94, 74, 47, 21]. These processes can lead to the formation of compact star-forming galaxies, which subsequently quench and evolve into compact quenched galaxies [4, 80] (Wang et al. 2026). However, this interpretation is complicated by progenitor bias, which arises because newly quenched galaxies continuously join the quenched population over cosmic time. In this scenario, the observed evolution of quenched galaxy properties, such as the increase in size and decrease in compactness toward lower redshifts [89], may partly reflect changes in the population composition rather than the intrinsic evolution of individual galaxies [50] (Chen et al. 2024). Moreover, the existence of a minority population of extended quenched galaxies further suggests that multiple quenching pathways may operate. Dynamical heating associated with processes such as violent accretion and mergers can modify galaxy structures after quenching (Chen et al. 2024; Ma et al. 2026), further complicating the interpretation of observations. Thus, whether compact star-forming galaxies represent the dominant progenitors of compact quenched galaxies still remains an open question.

To quantify the connection between structural properties and quenching, previous studies have adopted various measures of galaxy structure, including effective stellar mass surface density ($\Sigma_{\rm e}$; e.g., [22, 4, 61, 85, 89]) and $\Sigma_1$ [22, 32, 4, 61, 73, 80, 36]. While global density measures capture overall compactness, $\Sigma_1$ directly quantifies the central stellar mass concentration, which is closely linked to bulge growth and central velocity dispersion [32, 87, 4]. Compared with global density measures such as $\Sigma_{\rm e}$, $\Sigma_1$ shows a tighter connection with star formation activity [22, 61]. Therefore, we adopt $\Sigma_1$ as a physically motivated tracer of galaxy compactness. Following recent machine-learning approaches that quantify the relative predictive power of different galaxy properties (Bluck et al. 2020a,b; Chen et al. 2020, 2021a,b; Bluck et al. 2022), we employ a Random Forest framework to assess the relative importance of $\Sigma_1$ and other galaxy properties in predicting quenching.

Galaxies form and evolve within dark matter halos, and environmental effects may provide additional influences on their evolutionary pathways [86] (Mo et al. 2010). The structure of galaxies may partially reflect the properties of their host halos, as dark matter halos provide the gravitational potential that drives baryons to accumulate and reach the conditions required for star formation (Chen et al. 2024), and both dark matter halos and baryonic gas acquire angular momentum through tidal torques and merger processes. Furthermore, halo assembly history may imprint correlations between galaxy properties and large-scale structure through assembly bias (Gao et al. 2005; Zentner et al. 2014; Chen et al. 2020; Lyu et al. 2023; Wang et al. 2023a,b; Lyu et al. 2024; Wang & Peng 2025; Zhang et al. 2025). Although massive galaxies are generally considered to be primarily regulated by internal processes, their structural diversity may also reflect differences in their halo environments and assembly histories (see e.g. Mo et al. 2024). Therefore, simultaneously characterizing galaxy structure and halo environment is essential for obtaining a complete picture of massive galaxy evolution.

Recent advances with the James Webb Space Telescope (JWST) provide unprecedented opportunities to study the interplay between galaxy structure, star formation, and environment at cosmic noon. JWST near-infrared imaging enables detailed measurements of rest-frame optical structures at $1 \le z < 3$. The COSMOS-Web Casey et al. [15] is the largest contiguous JWST NIRCam survey field to date, combines deep imaging with extensive ancillary data, enabling statistical studies of galaxy evolution and large-scale environments with reduced cosmic variance.

The main goal of this paper is to investigate the connection between galaxy structural properties, quenching, and large-scale environments at cosmic noon using JWST/COSMOS-Web observations. In § 2, we describe the data and galaxy sample selection used in this work. In § 3, we present the methodology, including measurements of galaxy structural properties, structural classification, clustering analysis, and the Random Forest approach. In § 4, we present our main results on the relative importance of galaxy properties in predicting quenching, the properties of compact and extended galaxy populations, and differences in their clustering properties. In § 5, we discuss the implications of our results. Finally, we summarize our conclusions in § 6. Throughout this paper, we assume a flat $\Lambda$CDM cosmology with $\Omega_{\mathrm{m}} = 0.307$, $\Omega_{\mathrm{b}} = 0.048$, $\Omega_{\Lambda} = 0.693$ and $h = 0.678$. Here, $h = H_{0}/(100\ \mathrm{km\ s}^{-1}\ \mathrm{Mpc}^{-1})$ and $H_{0}$ is the Hubble constant [66].

Observational data

The COSMOS-Web catalog

The COSMOS-Web survey Casey et al. [15] is a JWST Cycle 1 program covering $0.54\ \mathrm{deg}^{2}$ in the COSMOS field with four NIRCam filters (F115W, F150W, F277W, F444W), reaching a $5\sigma$ depth of $\sim27.5$–$28.2$ mag, and supplemented by MIRI F770W imaging over a non-contiguous $\sim0.19\ \mathrm{deg}^{2}$ area with depths of $\sim25$–$26$ mag. The observations were obtained across multiple epochs between 2023 and 2024, and the data reduction follows standard JWST pipelines with detailed processing described in Franco et al. [33] and Harish et al. [37].

The survey is combined with extensive ancillary multi-wavelength data in COSMOS, enabling the construction of the COSMOS-Web photometric catalog¹ [70], which contains $\sim780,000$ sources. Photometry and structural parameters are derived using SourceXtractor++ with PSF-convolved Sérsic profile fitting, which provides robust structural parameters, including the Sérsic effective radius ($R_{\mathrm{e}}$), defined as the radius enclosing half of the galaxy’s total light, and the Sérsic index ($n$) used in this work. By taking advantage of JWST’s high spatial resolution, this approach enables reliable source deblending and improved photometric measurements, making the dataset a powerful resource for studying galaxy evolution across cosmic time.

Photometric redshifts are derived using LePHARE [1, 40] through spectral energy distribution (SED) fitting based on Bruzual & Charlot [13] stellar population models and a range of star formation histories [42]. The fitting includes dust attenuation (with multiple extinction laws and varying E(B−V)), emission lines, and intergalactic medium absorption. Redshifts ($z$) are taken as the median of the posterior probability distributions, with uncertainties from the 16th–84th percentiles. The photo-$z$ accuracy is evaluated against $\sim12,000$ spectroscopic redshifts in COSMOS, showing good agreement.

We adopt the non-parametric star formation histories (SFHs) and the corresponding star-formation rate (SFR), stellar mass and stellar ages derived from SED fitting with CIGALE [10], provided in Column 4 of the catalog (“CIGALE Physical Parameters and Non-Parametric SFH”). Here, the SFR is calculated over the look-back time of the last 100 Myr, the stellar age is defined as the look-back time between the epoch of observation and the epoch at which a galaxy formed 50% of its total stellar mass. Using the observed redshift $z_{\mathrm{obs}}$ together with the stellar age, we further calculate the formation redshift ($z_{\mathrm{form}}$) for each galaxy by converting the stellar age into the corresponding cosmic epoch:

where $t(z)$ is the cosmic age at redshift $z$. Equivalently, $z_{\mathrm{form}}$ corresponds to the redshift at which the galaxy had formed 50% of its total stellar mass.

Galaxy sample selection

Following Paquereau et al. [63], to ensure a representative galaxy sample for clustering measurements, we applied a series of cleaning steps. Stars were removed based on their better fits to stellar or brown dwarf templates in LePHARE, combined with a compactness criterion. We also excluded sources with sizes smaller than twice the FWHM in the $F115W$ band, as well as objects flagged hot pixels or other image artifacts. Regions around bright stars in both NIRCam and optical (HSC) images were masked, removing a small fraction of the sample ($\sim17\%$ in total).

We then imposed a magnitude limit of $m_{444W}=27.75$, following Paquereau et al. [63], defined from completeness tests against the deeper PRIMER-COSMOS survey and adopting an 80% completeness threshold. Stellar mass completeness was estimated following Pozzetti et al. [67], by computing, for each galaxy, the limiting mass it would have at the survey magnitude limit in the $F444W$ band. The final stellar mass completeness in each redshift bin is defined as the mass above which 90% of galaxies are expected to be complete, increasing from $\log(M_{\ast}/M_{\odot}) \sim7.5$ at $z \sim1$ to $\log(M_{\ast}/M_{\odot}) \sim8$ at $z \sim3$. We define the parent sample as galaxies with stellar masses above the corresponding completeness limits. Over the cosmic noon period, we adopt a conservative mass cut of $\log(M_{\ast}/M_{\odot}) > 9$.

We further divide the parent sample into two redshift bins, $1.0 \le z < 1.7$ and $1.7 \le z < 3.0$, to examine whether the observed galaxy properties and their halo connections show systematic variations across different epochs of cosmic noon. The two redshift intervals correspond to time spans of approximately 2 and 1.6 Gyr, respectively. We excluded galaxies for which less than 30% of the photometric-redshift probability distribution lies within the redshift bin of interest, i.e., $P(z_{\min} \le z \le z_{\max}) < 0.3$, where $z_{\min}$ and $z_{\max}$ are the lower and upper boundaries of the redshift bin, respectively.

In this work, we investigate the evolution of massive galaxies with $\log(M_{\ast}/M_{\odot}) > 10.5$. These galaxies are selected from the parent sample described above and are referred to as the full samples throughout this work. They are predominantly central galaxies in their host dark matter halos, with satellite fractions estimated from the halo occupation distribution (HOD) method to be $\lesssim7\%$ [63].

To separate star-forming and quenched galaxies from the full samples, we adopt the NUV$-$r$-$J$ color-color diagram, which has been widely used in studies of galaxies at intermediate to high redshifts (e.g., [41, 89]):

where $M_{\mathrm{NUV}}$, $M_r$ and $M_J$ are the absolute magnitudes in the NUV, r, and J bands, respectively, estimated from SED fitting.

The specific star-formation rate (sSFR) was also commonly applied to separate star-forming and quenched galaxies, with quenched galaxies defined as having $\log\mathrm{sSFR} < \log0.2/t(z_{\mathrm{obs}})$, where $t(z_{\mathrm{obs}})$ is the age of the Universe at the corresponding redshift (e.g., [62, 14, 89]). We further apply an sSFR criterion together with the color-color selection to construct conservative star-forming and quenched samples. Specifically, the color-selected star-forming galaxies are required to have $\log\mathrm{sSFR} > \log0.2/t(z_{\mathrm{obs}})$, while the color-selected quenched galaxies are required to have $\log\mathrm{sSFR} < \log0.2/t(z_{\mathrm{obs}})$. The separation between star-forming and quenched galaxies in our sample is shown in Figure 1. This combined selection leaves a small fraction of galaxies with inconsistent classifications, including blue galaxies with low sSFRs and red galaxies with high sSFRs. We exclude these ambiguous populations and the resulting star-forming and quenched samples as our fiducial samples. The impact of these ambiguous populations is assessed by including them in the galaxy samples and adopting alternative star-forming/quenched classification criteria. The corresponding tests are presented in Appendix § 8.

To ensure reliable measurements of the central stellar density, we only select galaxies with $0.5 < n < 6$ and $R_e > 0.3$ kpc (see § 3.5). We further divide the fiducial star-forming and quenched samples into compact and extended populations based on their structural properties (see § 4.2). The detailed properties of each galaxy subsample selected using the fiducial criteria, including the sample size, galaxy bias, and inferred halo mass in each redshift bin, are summarized in Table 1.

Subsample$^{(a)}$	$1 \le z < 1.7$			$1.7 \le z < 3$
	$N_{\mathrm{gal}}^{(b)}$	$b_{\mathrm{g}}^{(c)}$	$\log M_{\mathrm{h}}/\mathrm{M}_{\odot}^{(d)}$	$N_{\mathrm{gal}}$	$b_{\mathrm{g}}$	$\log M_{\mathrm{h}}/\mathrm{M}_{\odot}$
Full	2,841	$1.81^{+0.25}_{-0.29}$	$12.4^{+0.23}_{-0.35}$	2,304	$3.11^{+0.54}_{-0.66}$	$12.48^{+0.24}_{-0.39}$
Star-forming (SF)	1,737	$1.2^{+0.25}_{-0.32}$	$11.47^{+0.48}_{-1.12}$	1,435	$2.84^{+0.47}_{-0.57}$	$12.33^{+0.24}_{-0.38}$
Quenched (Q)	858	$2.12^{+0.5}_{-0.66}$	$12.68^{+0.33}_{-0.72}$	512	$5.06^{+1.46}_{-2.11}$	$13.15^{+0.3}_{-0.75}$
Extended star-forming (eSF)	1,274	$1.02^{+0.21}_{-0.27}$	$10.81^{+0.62}_{-1.64}$	989	$2.2^{+0.67}_{-0.99}$	$11.83^{+0.46}_{-1.49}$
Compact star-forming (cSF)	463	$0.36^{+0.59}_{-0.16}$	$9.17^{+1.4}_{-0.0}$	446	$2.0^{+0.76}_{-1.38}$	$11.63^{+0.59}_{-2.47}$
Extended quenched (eQ)	49	–	–	31	–	–
Compact quenched (cQ)	809	$1.93^{+0.47}_{-0.62}$	$12.44^{+0.36}_{-0.84}$	481	$4.78^{+1.91}_{-3.79}$	$13.04^{+0.4}_{-3.47}$

Environmental density

We adopt the environmental density map reconstructed by Hatamnia et al. [38], who applied the weighted kernel density estimation (wKDE) method to reconstruct the galaxy number density field using a

COSMOS-Web galaxy sample with a stellar mass limit of $\log(M_{\star}/\mathrm{M}_{\odot}) \ge8$. The dimensionless galaxy overdensity is defined as

where $\mathbf{X}$ denotes the angular coordinates of the target galaxy, $\sigma^{\mathrm{s}}(\mathbf{X})$ represents the weighted density at the galaxy position, and $\bar{\sigma}^{\mathrm{s}}$ is the mean weighted density within the corresponding redshift slice. The density field is reconstructed in COSMOS-Web photometric redshift slices by accounting for galaxy redshift uncertainties and correcting for survey geometry and masking effects ([38]). Previous comparisons based on simulations ([24]) have demonstrated that both wKDE and Voronoi tessellation can effectively recover the underlying density field, with wKDE exhibiting greater robustness against shot noise and sparse sampling.

Methods of analysis

Stellar mass and redshift matching

To minimize the impact of different stellar mass and redshift distributions among galaxy populations, we construct stellar mass- and redshift-matched samples using an iterative reweighting procedure.

For the target galaxy sample, we assign an individual weight to each galaxy and iteratively adjust these weights to match the stellar mass and redshift distributions of the base sample ([92]). Starting from equal weights, the weights are updated in each iteration according to

where $w^{(i)}$ represents the weight of each galaxy after the $i$-th iteration. The terms $\mathrm{PDF}_{\mathrm{base}}(M_{\star})$ and $\mathrm{PDF}_{\mathrm{base}}(z)$ denote the stellar mass and redshift distributions of the base sample, respectively, while $\mathrm{PDF}^{(i)}_{w}(M_{\star})$ and $\mathrm{PDF}^{(i)}_{w}(z)$ represent the corresponding weighted distributions of the target sample at the $i$-th iteration. The PDF ratios are evaluated in the corresponding stellar mass and redshift bins, and each target galaxy’s weight is updated according to the ratios in the bins containing that galaxy.

The stellar mass and redshift reweighting steps are alternately repeated until the weighted target sample converges to the base stellar mass and redshift distributions. In our analysis, we perform 10 iterations for the target galaxy sample, which is sufficient to achieve convergence. The final weights will be used in measuring the clustering signals (see § 3.2).

When comparing the clustering properties of the full, star-forming, and quenched samples, we adopt the full sample within each redshift bin as the base sample for the matching procedure. For comparisons among extended star-forming, compact star-forming, and compact quenched galaxies, we use the compact star-forming sample in each redshift bin as the base sample. For illustration, the weighted and unweighted distributions of the full, star-forming, and quenched samples are shown in Figure 12 in the Appendix § 9.

Galaxy clustering measurements

We measure the galaxy clustering using the two-point angular auto-correlation function (ACF) and adopt the estimator of Landy & Szalay Landy & Szalay [45], defined as follows:

where $GG$, $mathit{GR}$, and $RR$ represent the normalized numbers of galaxy-galaxy, galaxy-random, and random-random pairs, respectively. For the galaxy-galaxy and galaxy-random pair counts ($GG$ and $GR$), we incorporate the weights of the galaxy samples when computing the pair counts. The random catalog is assigned unit weight. In this work, we use the Corrfunc package⁶ Sinha & Garrison [71] to efficiently compute the pair counts.

We construct a random catalog that matches the survey geometry and angular selection function of the parent sample (§ 2.2). To reduce Poisson noise, the random catalog contains objects that is 15 times of the parent sample. Random sources are distributed uniformly across the effective survey area of the parent sample, while applying the same survey masks as those used for the data. This procedure ensures that the random catalog accurately reproduces the observational footprint and selection effects of the survey.

We estimate the values and uncertainties of the ACF using the jackknife resampling method. The observed field is divided into $6 \times5$ spatial subregions. For each jackknife realization, one subregion is removed and the ACF is recalculated, yielding $\omega_{\mathrm{obs},k}(\theta)$, where $k$ denotes the $k$-th jackknife realization. The final observed ACF signal in each angular bin is obtained by averaging over all jackknife realizations,

where $N$ is the number of jackknife realizations. The uncertainty of $\omega_{\mathrm{obs}}(\theta)$ is estimated from the jackknife variance:

The corresponding uncertainty is given by $\sigma(\theta) = \sqrt{\operatorname{Var}(\theta)}$.

The COSMOS-Web survey covers an area of only about $0.54\ \mathrm{deg}^{2}$, which can introduce the integral constraint bias and lead to an underestimation of the observed correlation function. To correct for this effect, following Ito et al. [43], the integral constraint $C$ is estimated as

where $\theta_i$ denotes the $i$-th angular separation bin, and $\omega_{\mathrm{obs}}(\theta)$ is corrected according to

In our analysis, the multiplicative integral constraint correction factor becomes non-physical in the largest angular bin, we therefore do not apply the multiplicative correction to this bin and use the measured correlation amplitude directly.

Galaxy bias and halo mass

The ACF is commonly described by a power-law form,

where $A_{\omega}$ is the clustering amplitude and $\gamma$ is the slope of the correlation function.

To determine $\gamma$, we first measure the observed ACF of the parent sample, which provides the highest signal-to-noise ratio compared with individual subsamples. The parent sample is assigned unit weight. Since the integral constraint correction depends on the assumed $\gamma$, we adopt an iterative procedure to obtain a self-consistent value of $\gamma$. Starting from an initial value of $\gamma= 1.8$ Coupon et al. [23]; Ito et al. [43], we calculate the integral constraint correction and fit the corrected correlation function with a power-law (Equation (10)), allowing both $A_{\omega}$ and $\gamma$ to vary. The updated value of $\gamma$ is then used for the next iteration. This procedure is repeated for 10 times, after which $\gamma$ converges to a stable value. The final $\gamma$ is adopted for the clustering analysis of the other galaxy subsamples at the same redshift bin (see Table 4). The fitting to the power-law is performed over angular scales of $\theta> 0.01^{\circ}$ for the corrected ACF.

$z$-bin	Sample	$\gamma$	$C$	$A_{\omega}(\times10^{-3}\gamma^{\gamma-1})$	$r_0$ (cMpc/$h$)
$1 \le z < 1.7$	Parent	1.77	4.03	$2.27^{+1.31}_{-1.31}$	–
$1 \le z < 1.7$	Full	1.77	4.03	$4.34^{+1.28}_{-1.28}$	$4.27^{+0.67}_{-0.76}$
$1 \le z < 1.7$	SF	1.77	4.03	$2.01^{+0.91}_{-0.91}$	$2.70^{+0.64}_{-0.78}$
$1 \le z < 1.7$	Q	1.77	4.03	$5.21^{+2.73}_{-2.73}$	$5.11^{+1.38}_{-1.75}$
$1 \le z < 1.7$	eSF	1.77	4.03	$2.27^{+1.05}_{-1.05}$	$2.15^{+0.51}_{-0.64}$
$1 \le z < 1.7$	cSF	1.77	4.03	$0.32^{+1.93}_{-0.22}$	$0.67^{+1.34}_{-0.33}$
$1 \le z < 1.7$	cQ	1.77	4.03	$7.04^{+3.81}_{-3.81}$	$4.45^{+1.23}_{-1.59}$
$1.7 \le z < 3$	Parent	1.83	4.57	$1.74^{+1.11}_{-1.11}$	–
$1.7 \le z < 3$	Full	1.83	4.57	$3.51^{+1.33}_{-1.33}$	$5.59^{+1.07}_{-1.27}$
$1.7 \le z < 3$	SF	1.83	4.57	$3.05^{+1.09}_{-1.09}$	$5.05^{+0.92}_{-1.09}$
$1.7 \le z < 3$	Q	1.83	4.57	$7.70^{+5.09}_{-5.09}$	$9.52^{+3.04}_{-4.25}$
$1.7 \le z < 3$	eSF	1.83	4.57	$2.11^{+1.47}_{-1.47}$	$3.75^{+1.26}_{-1.80}$
$1.7 \le z < 3$	cSF	1.83	4.57	$1.94^{+1.75}_{-1.75}$	$3.36^{+1.42}_{-2.43}$
$1.7 \le z < 3$	cQ	1.83	4.57	$7.79^{+7.46}_{-7.46}$	$8.78^{+3.88}_{-7.20}$

The three-dimensional spatial correlation function is also assumed to follow a power-law form,

where $r_0$ is the correlation length. We then employ the Limber inversion Peebles [64] to convert the projected two-dimensional ACF into the spatial correlation function:

where

and

Here, $c$ is the light speed, $D_\theta(z)$ is the angular diameter distance, $N(z)$ is the redshift distribution of the target galaxy sample, and $\Gamma$ denotes the Gamma function. The function

describes the redshift evolution of the spatial correlation function, where we adopt $\epsilon=-1.2$ following Roche & Eales [69], and $\bar{z}$ is the mean redshift of the target galaxy sample.

Assuming a linear galaxy bias model, the galaxy bias of the target sample is estimated by comparing its spatial correlation function with the dark matter correlation function on scales dominated by the two-halo term. Specifically, we compute

where the bias is averaged over the comoving scale range of $3-5$ cMpc/$h$. The dark matter correlation function, $\xi_{\mathrm{DM}}(r,\bar{z})$, is computed at the mean redshift $\bar{z}$ using the COLOSSUS package⁷ (Diemer [29]).

The halo mass of the target galaxy sample is inferred by interpolating the galaxy bias derived from Equation (16) onto the halo mass–bias relation of Tinker et al. [76], computed using the COLOSSUS package. Here, the halo mass is defined as $M_{200\mathrm{m}}$, i.e., the mass enclosed within a radius $R_{200\mathrm{m}}$ where the mean mass density is 200 times the mean matter density of the Universe.

Random Forest framework for quantifying the predictive power of galaxy properties

To investigate the galaxy properties associated with quenching, we employ a Random Forest (RF) classifier to distinguish star-forming and quenched galaxies. The input feature set consists of five galaxy properties: $\Sigma_1$, $z_{\mathrm{form}}$, $M_\ast$, $1+\delta$, and $R_{\mathrm{e}}$. Galaxies with unavailable measurements are excluded from the analysis. For each redshift bin, the galaxy sample is randomly divided into training and test sets with a ratio of 4:1. The RF classifier consists of 300 decision trees, with a maximum tree depth of 6 and a minimum of 20 galaxies in each terminal leaf. Class-balanced weights are adopted to compensate for the imbalance between star-forming and quenched galaxies.

To quantify the relative importance of different galaxy properties, we train an RF classifier for each redshift bin using all five input features and interpret the trained model using SHAP (SHapley Additive exPlanations) scores computed with the TreeExplainer algorithm. The SHAP score of each feature quantifies its contribution to the predicted probability for an individual galaxy. Because several galaxy properties are mutually correlated (e.g., $M_\ast$, $\Sigma_1$, and $R_{\mathrm{e}}$), we evaluate the importance of both individual parameters and their pairwise combinations. For each feature group, the SHAP scores of the constituent features are summed for every galaxy, and the group importance is defined as the mean absolute SHAP score over the test sample. The feature groups are then ranked according to this quantity, allowing a direct comparison of the relative contributions of different galaxy properties to quenching. The Random Forest classification is performed using the scikit-learn⁸ and SHAP⁹ Python packages.

Measurement of central stellar mass density

We compute $\Sigma_1$ by assuming that the stellar mass follows a Sérsic profile. The enclosed stellar mass fraction within a projected radius $r$ is calculated using the incomplete gamma function:

where $R_{\mathrm{e}}$ is the effective radius, $n$ is the Sérsic index, and $b_n$ is determined by the condition that $R_{\mathrm{e}}$ encloses half of the total stellar mass. We adopt the approximation

The stellar mass enclosed within 1 kpc is then calculated as

and the central stellar mass density is defined as

We restrict our analysis to galaxies with $0.5 < n < 6$ and $R_e > 0.3\ \mathrm{kpc}$ to minimize uncertainties in the inferred $\Sigma_1$. Extreme Sérsic indices can lead to unstable estimates of the enclosed mass fraction, while galaxies with very small effective radii are more susceptible to resolution effects and uncertainties in structural parameter measurements.

Classification of compact and extended galaxies

We define compact and extended populations based on the structural sequence of quenched galaxies. Following previous studies [4, 79], we characterize the structural sequence of quenched galaxies by fitting a ridgeline in the $M_\star$–$\Sigma_1$ plane within each redshift bin:

where $a$ and $b$ are determined through a least-squares fit. Unlike a fixed $\Sigma_1$ threshold, the ridgeline method accounts for the mass dependence of galaxy structure and allows us to quantify compactness relative to the quenched population at a given stellar mass. We define the offset from this relation as

which quantifies the relative compactness of each galaxy at fixed stellar mass and redshift bin.

The scatter of the quenched structural sequence is quantified using the median absolute deviation (mad) of $\Delta\Sigma_1$:

where the mad provides a robust estimate of the scatter around the quenched ridgeline, and the factor 1.4826 scales it to the equivalent standard deviation for a Gaussian distribution. We then classify star-forming and quenched galaxies as compact or extended according to their offsets from the quenched ridgeline:

Results

Relative importance of galaxy properties in predicting galaxy quenching

The quenching of massive galaxies is closely associated with both their internal properties and surrounding environments. Previous studies have suggested that stellar mass, central stellar density, galaxy age, and local environment may all correlate with the transition from star-forming to quenched populations [65, 32, 88]. We therefore first investigate the two-dimensional distributions of the quenched fraction ($f_q$) as a function of $M_\star$, $\Sigma_1$, $z_{\mathrm{form}}$, and $(1+\delta)$ in both redshift bins. The quenched fraction is defined as $f_q = N_Q/N_{\mathrm{all}}$, where $N_Q$ and $N_{\mathrm{all}}$ denote the numbers of quenched galaxies and all galaxies within a given bin, respectively. The results are shown in Figure 2.

The quenched fraction exhibits a strong dependence on $\Sigma_1$ in both redshift bins. In contrast, $f_q$ shows only a weak dependence on $M_\star$ within our massive galaxy sample. Galaxies with low $\Sigma_1$ predominantly have very low $f_q$, while regions with high $\Sigma_1$ span a broader range of quenched fractions, including parameter bins with substantially elevated $f_q$. A threshold-like behavior is observed around $\log\Sigma_1/\mathrm{M_\odot\,pc^{-2}} \simeq3.6$–$3.7$, marked by the grey vertical line in Figure 2: above this characteristic $\Sigma_1$, bins with high $f_q$ become progressively more prevalent. This behavior is broadly consistent with previous findings that quenched galaxies preferentially occupy regions of high $\Sigma_1$ [32, 87, 4]. Fang et al. [32] showed that galaxies with $\log\Sigma_1/\mathrm{M_\odot\,pc^{-2}} \gtrsim3.5$ host the majority of red galaxies, while Barro et al. [4] found that a mass-dependent compactness criterion, $\log\Sigma_1/\mathrm{M_\odot\,pc^{-2}} - 0.65(\log M_\star/\mathrm{M_\odot} - 10.5) \gtrsim3.5$, efficiently selects compact star-forming and quenched galaxies at $z \gtrsim0.5$.

The dependence of $f_q$ on $z_{\mathrm{form}}$ is mainly observed among galaxies with high $\Sigma_1$ and varies between the two redshift bins. In the lower-redshift bin ($1 \leq z < 1.7$), $f_q$ increases toward higher $z_{\mathrm{form}}$ for galaxies with high $\Sigma_1$, with the highest quenched fractions found in bins combining high $\Sigma_1$ and high $z_{\mathrm{form}}$. In the higher-redshift bin ($1.7 \leq z < 3$), this trend becomes weaker, although a mild preference for higher $f_q$ at larger $z_{\mathrm{form}}$ remains visible among high-$\Sigma_1$ galaxies. In both redshift bins, $f_q$ does not exhibit a clear dependence on $(1+\delta)$.

Applying the Random Forest analysis described in § 3.4, we evaluate the relative importance of different galaxy properties in predicting galaxy quenching using SHAP scores. Figure 3 presents the SHAP importance of individual parameters and feature combinations in the two redshift bins. In both redshift bins, among the five input features considered here, $\Sigma_1$ emerges as the most important predictor in distinguishing quenched and star-forming galaxies. Feature combinations containing $\Sigma_1$ consistently exhibit higher SHAP scores than those without $\Sigma_1$. $R_{\mathrm{e}}$ is the second most important parameter, likely reflecting its connection with $\Sigma_1$, followed by $z_{\mathrm{form}}$, $M_\ast$, and $(1+\delta)$.

These results are consistent with previous studies demonstrating the close connection between $\Sigma_1$ and quenching across cosmic time ([32, 4, 85]. Other galaxy properties provide additional but relatively limited predictive information compared with $\Sigma_1$. This motivates us to investigate whether galaxies with different structural states represent distinct stages in galaxy evolution.

Structural diversity of star-forming and quenched galaxies

While quenching is often discussed in the framework of the star-forming and quenched dichotomy, galaxies with similar star formation states can exhibit diverse structural properties ([32, 4, 79]. Such structural diversity may provide additional clues about the evolutionary pathways of massive galaxies.

We adopt the method described in § 3.6 to fit the quenched ridgeline and separate the galaxy sample into extended and compact subsamples. The quenched ridgeline and the corresponding scatter are listed in Table 2 and also illustrated in Figure 4. The quenched galaxies in our COSMOS-Web sample follow a tight $\Sigma_1$–$M_\ast$ relation, with a scatter of $\sim0.1$ dex. The fitted ridgelines are in good agreement with those reported by Barro et al. [4], with comparable slopes ($\sim0.58$ vs. $\sim0.65$) and a similarly tight distribution around the relation ($\sim0.1$ dex scatter).

	$a$	$b$	$\sigma_{\mathrm{ridge}}$
$1 \le z < 1.7$	4.05	0.58	0.1
$1.7 \le z < 3$	4.15	0.58	0.09

We then compare the distributions of the extended star-forming (eSF), compact star-forming (cSF), extended quenched (eQ), and compact quenched (cQ) populations in different parameter spaces. The results are presented in Figure 5.

Across both redshift bins, galaxies exhibit a clear L-shaped distribution in the sSFR–$\Sigma_1$ plane, consistent with previous studies (e.g., [32, 4, 79]) and highlighting the strong association between $\Sigma_1$ and star formation activity. Star-forming galaxies primarily occupy the high-sSFR regime and span a broad range of $\Sigma_1$, whereas quenched galaxies are concentrated at low sSFR and high $\Sigma_1$. Only a small fraction of quenched galaxies are found in the low-$\Sigma_1$ regime; in our sample, extended quenched galaxies account for only $\sim6\%$ of the quenched population (see Table 1), with the majority of quenched galaxies classified as compact.

The separation between star-forming and quenched populations is also evident in their $M_\ast$–$R_{\mathrm{e}}$ and $M_\ast$–$z_{\mathrm{form}}$ relations. All populations exhibit positive correlations between $M_\ast$ and $R_{\mathrm{e}}$. At fixed $M_\ast$, quenched galaxies are generally more compact than star-forming galaxies (e.g., [77, 4, 89], with smaller median $R_{\mathrm{e}}$ values across the stellar mass range. Extended galaxies have larger median $R_{\mathrm{e}}$ values than compact galaxies in both star-forming and quenched populations. Within each structural class, star-forming galaxies show slightly larger $R_{\mathrm{e}}$ values than quenched galaxies.

The right panels show $z_{\mathrm{form}}$ as a function of $M_\ast$. At fixed $M_\ast$, compact quenched galaxies exhibit higher median $z_{\mathrm{form}}$ values than star-forming galaxies in both redshift bins. The difference between the two populations is larger in the lower-redshift bin ($1 \le z < 1.7$) than in the higher-redshift bin ($1.7 \le z < 3$), suggesting a stronger separation in inferred stellar formation histories at later cosmic times. Among star-forming galaxies, compact systems show slightly higher median $z_{\mathrm{form}}$ values than extended systems, although the difference is relatively small. The trend for extended quenched galaxies is less constrained due to their limited sample size, resulting in larger fluctuations. These variations in $z_{\mathrm{form}}$ provide additional constraints on the evolution of structurally distinct populations, which are further discussed in Section 5.1.

The comparison of different structural populations further highlights the diversity of star-forming galaxies. A non-negligible fraction of star-forming galaxies reach high $\Sigma_1$ values comparable to the compact quenched galaxies (e.g., [4, 79], demonstrating that high $\Sigma_1$ alone does not uniquely determine the quenching state. This motivates further investigation into whether structurally distinct populations are associated with different large-scale environments (see § 4.3).

The environmental dependence of structurally distinct galaxy populations

Galaxy clustering provides a direct probe of the connection between galaxy properties and their dark mat-

ter halos. Before investigating the environmental dependence of structurally distinct galaxy populations, we first examine whether the conventional star-forming and quenched classification is associated with different large-scale environments. Figure 6 and Table 1 present the measured clustering signals, galaxy biases, and inferred halo masses for the full, star-forming, and quenched samples in the two redshift bins, respectively. The methods used to estimate the galaxy biases and halo masses are described in § 3.3. The best-fit parameters of the power-law models used to characterize the angular correlation functions are summarized in Table C in the Appendix § 10. All clustering measurements are performed using stellar mass- and redshift-matched samples (see § 3.1). At $1 \le z < 1.7$, the full, star-forming, and quenched populations show broadly similar large-scale clustering signals ($\theta\gtrsim0.01$). However, the inferred bias of quenched galaxies is higher than that of both the full and star-forming samples, with $b_{\rm Q} = 2.12^{+0.50}_{-0.66}$, compared with $b_{\rm full} = 1.81^{+0.25}_{-0.29}$ and $b_{\rm SF} = 1.20^{+0.25}_{-0.32}$. At $1.7 \le z < 3$, quenched galaxies show a similar tendency toward a higher inferred bias, with $b_{\rm Q} = 5.06^{+1.46}_{-2.11}$, $b_{\rm full} = 3.11^{+0.54}_{-0.66}$, and $b_{\rm SF} = 2.84^{+0.47}_{-0.57}$. However, the larger uncertainties in the inferred biases, mainly caused by the smaller number of quenched galaxies, prevent a statistically significant distinction among these populations. Overall, the conventional star-forming/quenched classification reveals a marginal difference in large-scale clustering for massive galaxies at $1 < z < 3$. Although not statistically significant given the current uncertainties, the systematically higher bias of quenched galaxies suggests a possible tendency for quenched galaxies to reside in more massive halos (Li et al. 2006; Zhang et al. [93]).

We further examine whether structural properties provide additional information on the galaxy–halo connection. Figure 7 presents clustering measurements of eSF, cSF, and cQ galaxies, with the eQ sample omitted due to limited sample size and large clustering uncertainties. Table 1 lists their galaxy biases and inferred halo masses, and Table C in Appendix § 10 summarizes the best-fit power-law model parameters.

At $1 \le z < 1.7$, the structural populations show a tentative difference in their clustering amplitudes, with cQ galaxies exhibiting the highest inferred bias among the three populations. The inferred bias of cQ galaxies is $b_{\rm cQ} = 1.93^{+0.47}_{-0.62}$, compared with $b_{\rm eSF} = 1.02^{+0.21}_{-0.27}$ and $b_{\rm cSF} = 0.36^{+0.59}_{-0.16}$. However, the uncertainties, particularly for the cSF population, prevent a statistically significant separation among these populations. The comparison between eSF and cSF galaxies shows no significant difference in their large-scale clustering properties within the current uncertainties.

At $1.7 \le z < 3$, the clustering differences among structural populations become less significant. The clustering signals of eSF, cSF, and cQ galaxies overlap substantially within the uncertainties, with inferred biases of $b_{\rm cQ} = 4.78^{+1.91}_{-3.79}$, $b_{\rm cSF} = 2.00^{+0.76}_{-1.38}$, and $b_{\rm eSF} = 2.20^{+0.67}_{-0.99}$. The higher central value of the cQ bias suggests a possible enhancement of clustering for cQ galaxies, whereas eSF and cSF galaxies show similar clustering properties.

Taken together, structural classification provides additional information on the galaxy–halo connection beyond the conventional star-forming/quenched division. We find no significant difference in large-scale clustering between structurally distinct star-forming galaxies, while cQ galaxies show a tentative tendency toward stronger clustering and higher inferred halo masses. Previous clustering studies of cSF and cQ galaxies at $z \sim1.2-3$ have reported comparable clustering strengths for these two populations (e.g., Lin et al. [51]), consistent with our finding that the difference between cSF and cQ galaxies is not statistically significant within the current uncertainties, although cQ galaxies exhibit higher central values of the inferred bias and halo mass. In contrast, eSF and cSF galaxies show similar large-scale clustering signals despite their different structural properties, consistent with previous findings that central density-related properties are more closely associassociated with quenching than with environment [79].

Discussion

$\Sigma_1$ as a key predictor of quenching at cosmic noon

Our analysis shows that central stellar density, $\Sigma_1$, is the strongest predictor of galaxy quenching during cosmic noon. Among the galaxy properties considered in the Random Forest analysis, $\Sigma_1$ exhibits the highest predictive importance, substantially exceeding that of stellar mass. This indicates that the internal distribution of stellar mass provides a closer observational connection to quenching than global stellar mass for massive galaxies at $1 \le z < 3$.

Previous studies across different redshift ranges have established a strong association between central stellar density and quenching, often identifying characteristic $\Sigma_1$ thresholds separating star-forming and quenched populations [22, 32, 4, 61, 85]. These studies have demonstrated the importance of central density as a quenching-related structural parameter, but mainly focused on correlations or structural differences between quenched and star-forming galaxies. The relative importance of $\Sigma_1$ compared with other quenching-related properties has been less explored. Recent machine-learning studies have enabled quantitative comparisons among different galaxy properties, showing that central galaxy properties, including central velocity dispersion and bulge-related quantities, provide stronger predictions of quenching than global stellar mass or environmental indicators [7, 6]. Extending this approach to cosmic noon, our analysis simultaneously evaluates structural, stellar population, and environmental properties, revealing the dominant predictive role of $\Sigma_1$ for quenching at $1 \le z < 3$.

The strong association between $\Sigma_{1}$ and quenching is further supported by the quenched fraction distribution. In our sample, the quenched fraction increases rapidly above $\log\Sigma_{1} \simeq3.6 - 3.7$, while showing a much weaker dependence on stellar mass. This rapid increase suggests that galaxies above this central density regime are preferentially associated with the quenched population. At fixed stellar mass, galaxies with higher $\Sigma_{1}$ exhibit more centrally concentrated stellar distributions and generally more prominent bulge components [32, 46].

Several mechanisms may contribute to the buildup of central stellar components. Gas-rich processes, including cold gas accretion, gravitational instabilities, and gas-rich interactions, can drive baryons toward galaxy centers, enhancing central gas concentration and subsequent star formation, thereby increasing stellar concentration [26, 94, 47, 55] (Ma et al. 2026). Such dissipative compaction can produce cSF systems, or “blue nuggets”, which have been proposed as an intermediate evolutionary stage between eSF and cQ galaxies [94, 4, 79].

Our results are broadly consistent with this structural transformation scenario. We find that cSF galaxies have slightly higher $z_{\mathrm{form}}$ than eSF galaxies, but lower $z_{\mathrm{form}}$ than cQ galaxies. This ordering suggests that cSF galaxies may represent an intermediate population between eSF and cQ systems. However, this interpretation should be treated with caution. Differences in structure and formation history among eSF, cSF, and cQ galaxies do not necessarily imply a direct evolutionary sequence, as progenitor bias may contribute to the observed population differences. Galaxies that quenched at different epochs may enter the quenched population with different structural properties and formation histories [50, 73, 89]. With the current data, we cannot distinguish whether these differences primarily reflect genuine evolutionary transformations, progenitor bias, or a combination of both.

In addition to internal galaxy properties, we also examined the role of environment. The relatively weak contribution of local overdensity $(1+\delta)$ suggests that environment provides less predictive information than internal structural properties for our massive galaxies [79]. This does not imply that halo environments are irrelevant, as halo-related processes may regulate gas accretion, thereby influencing quenching [26, 87]. We next discuss how halo environments are connected to different structural populations, from eSF to cQ galaxies.

The evolutionary connection between structural transformation, quenching, and halo environments

The role of halo environments in structural transformation and quenching has been explored in previous studies, including comparisons of the clustering signals between cSF and cQ galaxies [51]. However, whether the full structural diversity of eSF, cSF, and cQ galaxies is accompanied by systematic differences in their halo environments remains less well constrained. Using a homogeneous JWST COSMOS-Web sample, our clustering measurements allow us to examine the galaxy–halo connection across these structural populations.

A key result from our clustering analysis is that eSF and cSF galaxies exhibit comparable galaxy biases within the uncertainties. Despite their substantial differences in $\Sigma_{1}$ and size, these two populations appear to reside in halos with similar masses. If structural differences between eSF and cSF galaxies were primarily driven by different halo environments, we would expect a measurable difference in their large-scale bias. The absence of such a difference suggests that halo mass variations alone cannot account for the observed structural transformation, pointing to an important role of internal baryonic processes in shaping galaxy structures [26, 94, 47]. The lack of significant halo mass differences may also suggests that structural transformation can occur on timescales shorter than major evolution in halo mass.

In contrast, cQ galaxies show a tendency toward higher inferred halo masses than cSF galaxies. This difference may suggests that, after quenching, stellar masses evolve relatively slowly while host halos continue to grow, allowing older quenched populations to reside in progressively more massive halos [93]. This interpretation is consistent with previous studies showing that quenching is associated with the combination of dense central structures and sufficiently massive halos [87, 36]. In this picture, the buildup of $\Sigma_{1}$ provides an important structural condition for quenching, while subsequent halo growth shapes the evolution of the quenched population.

Taken together, our results favor a picture in which internal structural evolution and halo growth need not proceed synchronously. The substantial structural differences between eSF and cSF can develop within broadly similar halo environments, structural evolution can occur on a shorter timescale than substantial halo growth. Meanwhile, the more massive halos associated with cQ may reflect continued halo growth after or around the onset of quenching.

Caveats and future prospects

Our analysis provides observational constraints on the galaxy–halo connection during cosmic noon, but several limitations remain. First, the limited area and sample size of the COSMOS-Web field introduce significant uncertainties in clustering measurements. Upcoming wide-field surveys such as Euclid, Roman, CSST, and LSST will be essential to establish these trends with improved statistical significance.

Second, our structural measurements are based on Sérsic profile fitting, with $\Sigma_1$ inferred from the stellar mass enclosed within 1 kpc. This model-dependent approach may introduce uncertainties due to assumptions about the underlying light profile, PSF correction, and spatial variations in stellar populations (Ma et al. 2026). Future analyses based on spatially resolved stellar mass maps will enable more direct and robust measurements of $\Sigma_1$.

Finally, although $\Sigma_1$ shows the strongest association with quenching in our analysis, this correlation alone does not establish a causal connection [50]. Future observations of molecular gas reservoirs, gas kinematics, and black hole activity will be required to identify the physical mechanisms linking structural transformation to the suppression of star formation.

Summary

In this work, we investigate the connection between galaxy structural properties, star formation activity, and large-scale environments for massive galaxies at $1 < z < 3$ using JWST/COSMOS-Web observations. By combining structural measurements, Random Forest analysis, and galaxy clustering measurements, we quantify the role of different galaxy properties in predicting quenching and explore the halo environments of structurally distinct galaxy populations. Our main conclusions are summarized as follows:

1. We quantify the relative importance of different galaxy properties in predicting galaxy quenching, including $M_*$, $\Sigma_1$, $R_{\mathrm{e}}$, $z_{\mathrm{form}}$, and $(1 + \delta)$. Among these parameters, $\Sigma_1$ provides the strongest predictive power, while the additional parameters contribute only marginal improvements. The strong dependence of quenched fraction on $\Sigma_1$ demonstrates that the buildup of central stellar mass surface density is closely associated with quenching in massive galaxies during cosmic noon.

2. We systematically investigate the large-scale environments of structurally classified galaxy populations, including extended star-forming (eSF), compact star-forming (cSF), and compact quenched (cQ) galaxies. We find no significant difference in the clustering properties of eSF and cSF galaxies despite their distinct structures, suggesting similar halo environments for these two star-forming populations. cQ galaxies exhibit higher central values of clustering amplitude and inferred halo mass, although the uncertainties prevent a statistically significant distinction from the other populations. The clustering of extended quenched galaxies remains unconstrained due to the limited sample size.

These results emphasize the role of central stellar mass surface density in galaxy quenching and provide new constraints on the halo connection of structurally distinct populations during cosmic noon. Future wide-field surveys will extend this analysis with larger samples and improved statistical precision.

Acknowledgements

This work is supported by the China Postdoctoral Science Foundation-Anhui Joint Support Program under Grant Number 2025T004AH and Anhui Postdoctoral Scientific Research Program Foundation (No. BH2030000803). This work is supported by the National Natural Science Foundation of China (NSFC, Nos. 12595312, 12192224). HYW thanks the support of CAS Project for Young Scientists in Basic Research, Grant No. YSBR-062, the New Cornerstone Science Foundation through the XPLORER PRIZE, and the China Manned Space Program with grant no. CMS-CSST-2025-A04. The work is also supported by the Supercomputer Center of University of Science and Technology of China.

Robustness tests with alternative sample selection

To examine the robustness of our results against the sample selection criteria, we construct alternative star-forming and quenched samples by including the galaxies with inconsistent color and sSFR classifications that were excluded from the fiducial samples. The corresponding sample properties, including the number of galaxies, galaxy biases, and inferred halo masses, are summarized in Table 3.

Subsample	$1 \le z < 1.7$			$1.7 \le z < 3$
	$N_{\mathrm{gal}}$	$b_{\mathrm{g}}$	$\log(M_{\mathrm{h}}/M_{\odot})$	$N_{\mathrm{gal}}$	$b_{\mathrm{g}}$	$\log(M_{\mathrm{h}}/M_{\odot})$
Star-forming (SF)	1752	$1.25^{+0.24}_{-0.30}$	$11.58^{+0.43}_{-0.92}$	1450	$2.90^{+0.48}_{-0.58}$	$12.36^{+0.24}_{-0.38}$
Quenched (Q)	1089	$2.15^{+0.43}_{-0.55}$	$12.71^{+0.28}_{-0.54}$	854	$2.53^{+1.05}_{-1.98}$	$12.14^{+0.54}_{-2.98}$
Extended star-forming (eSF)	1160	$1.04^{+0.26}_{-0.35}$	$10.90^{+0.68}_{-1.74}$	897	$2.38^{+0.75}_{-1.14}$	$11.98^{+0.46}_{-1.54}$
Compact star-forming (cSF)	592	$0.20^{+0.61}_{-0.00}$	$9.17^{+0.54}_{-0.00}$	553	$1.68^{+0.74}_{-1.22}$	$11.25^{+0.75}_{-2.08}$
Extended quenched (eQ)	87	–	–	62	–	–
Compact quenched (cQ)	1002	$1.99^{+0.36}_{-0.44}$	$12.50^{+0.27}_{-0.48}$	792	$4.42^{+1.18}_{-1.64}$	$12.93^{+0.30}_{-0.68}$

The SHAP importance of different galaxy properties for the alternative sample is presented in Figure 8. The relative importance of galaxy properties in predicting quenching remains unchanged: $\Sigma_1$ is the most important predictor of galaxy quenching, followed by $R_e$, $z_{\rm form}$, $M_*$, and $(1+\delta)$.

The galaxy distributions obtained with the alternative selection are shown in Figure 9. The distributions of galaxy properties, including structural parameters, sSFR and $z_{\rm form}$, among extended and compact star-forming and quenched populations remain broadly consistent with those of the fiducial sample (Figure 5).

The clustering measurements of the full, star-forming, and quenched populations are shown in Figure 10. At $1 \le z < 1.7$, quenched galaxies exhibit a higher inferred bias than star-forming galaxies. At $1.7 \le z < 3$, however, the inferred biases of the three populations overlap within the uncertainties, mainly due to the large uncertainty of the quenched sample.

Figure 11 presents the clustering measurements of the structurally selected populations. The alternative selection yields the same qualitative trend as the fiducial analysis: $c$Q galaxies show a tendency toward stronger clustering, while $c$SF and $e$SF galaxies exhibit comparable clustering amplitudes within the uncertainties.

Overall, the alternative sample selection does not change the main conclusions of this work. The strong association between $\Sigma_1$ and quenching, as well as the slightly higher galaxy bias of quenched galaxies compared with other populations, remain robust against reasonable variations in sample selection.

Stellar mass and redshift matching

Figure 12 shows the stellar mass and redshift distributions before and after the iterative reweighting procedure.

Power-law fitting of angular correlation functions

The best-fit parameters of the power-law models used to describe the angular correlation functions are summarized in Table 4.