betaFeedback

WorksPhysics and Astronomy

Less is More: Unbiased Galaxy Clustering from Fewer Merger Trees

Abstract

Galaxy clustering could strongly constrain galaxy formation models but is rarely used, because the two-point correlation function (2PCF) requires a complete galaxy catalogue at every point in parameter space. Semi-analytic models (SAMs) process each halo merger tree independently, so running a SAM on mm of kk disjoint partitions of the trees, each spanning the full box, cuts the cost by k/mk/m. Because a halo's galaxies are kept or discarded together, one-halo pairs are diluted less than two-halo pairs, and the naive estimate overstates small-scale clustering by a factor k/mk/m. We introduce the Sparse and Unbiased Clustering Estimator (SaUCE), which removes this bias with weights set by mm and kk alone. It has no free parameters, gives exactly unbiased pair counts, and applies to any catalogue built from a random subset of whole haloes and to any pair-count statistic. Running a SAM on 7 per cent of the P-Millennium merger trees, 68 per cent of random selections recover the full-box ξ(r)ξ(r) at z=0z=0 to within 0.05 dex in each bin from 0.03 to 30 h−1 Mpc30\,h^{-1}\,\mathrm{Mpc}, at a density of 10−2 h3 Mpc−310^{-2}\,h^{3}\,\mathrm{Mpc}^{-3}: a fourteen-fold computational saving. The required fraction rises with redshift and as the number density falls, is the same for two model variants, and transfers between boxes through the galaxy and pair counts. The scatter, derived in closed form, is strongly correlated between bins, so a likelihood must use the full covariance. A recipe for choosing the fraction makes the 2PCF affordable for model calibration and simulation-based inference.

Introduction

Galaxy clustering depends on how galaxies populate dark matter haloes. The physical processes of galaxy formation – supernova feedback, feedback from active galactic nuclei (AGN), gas cooling, star formation and galaxy mergers – shape this occupation, each changing the number and distribution of galaxies across haloes of different mass (Baugh 2006; Benson 2010; Wechsler & Tinker 2018). Clustering can constrain galaxy formation physics, and complements the one-point statistics, such as the luminosity function, on which model calibration has traditionally relied.

The advent of precision clustering measurements from surveys such as the Dark Energy Spectroscopic Instrument (DESI Collaboration 2016) makes it even more timely to incorporate the 2PCF into model calibration.

Despite this, galaxy clustering is seldom used to calibrate physically motivated galaxy formation models. The principal obstacle is not evaluating the two-point correlation function (2PCF; [39]) but the cost of producing the galaxy catalogue from which the clustering is measured.

Hydrodynamical simulations of cosmological volumes (e.g. Schaye et al. 2023) are too expensive to run at every step of a parameter search, and are instead calibrated on smaller boxes (Kugel et al. 2023), which can only probe the clustering over a narrow range of scales.

Semi-analytic models (SAMs) are cheaper by orders of magnitude, but still costly: a full galaxy catalogue for the P-Millennium simulation (Baugh et al. 2019) requires processing the merger trees of more than 77 million haloes with masses above 2.12×109h−1 M⊙2.12 \times10^{9} h^{-1}\,\mathrm{M}_{\odot} at z=0z=0, and using the 2PCF in the calibration requires one such catalogue at every point in parameter space – thousands of them for a Markov chain Monte Carlo (MCMC) search, and comparable numbers for simulation-based inference (Crammer et al. 2020).

Henriques et al. [25] made this affordable for one-point statistics by running the model on a small, representative sample of haloes, and van Daalen et al. [49] extended the approach to clustering. Calibrating the model of Guo et al. (2013), they sampled haloes at a mass-dependent rate, using fewer than 0.04 per cent of all haloes in the Millennium I simulation, measured the halo occupation and satellite profiles of the sampled galaxies, and used these in a halo model ([3]) to predict the projected correlation function. Adding clustering to the MCMC narrowed degeneracies left open by the stellar mass function, tightening the supernova feedback parameters and constraining the timescale on which orphan satellites merge. The halo model, however, is itself an approximation: even applied to the full galaxy catalogue, their prediction departed from the directly measured clustering by up to ~15 per cent outside their most massive bin, and the model’s two-halo term took its halo clustering from linear theory and a fitted bias rather than from the sampled haloes.

Kaiser [2] showed that a sparsely sampled catalogue gives an unbiased estimate of the 2PCF, because random subsampling raises the shot noise without biasing the clustering signal. We apply this idea to merger trees. Because most SAMs process each tree independently, we can divide the trees of a cosmological volume into partitions, run the model on only some of them, and correct the correlation function for the fraction run, the partition fraction. Unlike Kaiser [2], we sample haloes rather than galaxies: a halo’s galaxies are kept or discarded together, so one-halo pairs, which survive whenever their halo is kept, are diluted less than two-halo pairs, which survive only if both haloes are kept. Without correction the small-scale clustering is overestimated, increasingly so as the partition fraction falls. We remove this bias exactly, with weights set only by the numbers of partitions run and available, so that, unlike the approach of van Daalen et al. [49], nothing is tuned to a particular galaxy catalogue.

We introduce the Sparse and Unbiased Clustering Estimator (SaUCE) and test it with the GALFORM SAM (Cole et al. 2000; Bower et al. 2006; Lacey et al. 2016). § 2 describes the simulations, models and galaxy samples. § 3 derives the estimator, a correction with no free parameters, exact for the pair counts, that holds for any catalogue built from a random subset of whole haloes, and § 4 defines how we validate it. § 5 measures the partition fraction needed as a function of scale, redshift and number density; shows that it is the same for two GALFORM variants with different feedback but, at fixed number density, similar halo occupation distributions (HODs); shows how it transfers between simulations of different size; characterises the scatter and the bin-to-bin covariance a likelihood needs; and tests the correction in redshift space. § 6 collects these into a practical recipe.

Simulated galaxy catalogues

We test SaUCE with two variants of GALFORM (Appendix B) run on three NN-body simulations. GALFORM follows the baryonic components of galaxies within dark matter halo merger trees, each of which records the mergers and accretion by which a halo grew; this history sets the galaxy content of the halo and hence its contribution to the clustering. The two variants populate the same trees differently, which tests whether the scatter of SaUCE depends on galaxy formation physics.

The trees are extracted from the NN-body simulations. SUBFIND subhaloes (Springel et al. 2001) are grouped into DHalo haloes (Jiang et al. 2014; see also Merson et al. 2013) whose histories are strictly hierarchical, so no halo is shared between trees, a property SaUCE requires (§ 3.5). Tree-building choices leave artefacts that change the galaxy output (Gómez et al. 2022; Chandro-Gómez et al. 2025), but the reference and sparse estimates use the same trees, so the comparison tests SaUCE on whatever population those trees produce.

N-body simulations

Table 1 lists the properties of the three simulations and the number of partitions, kk, into which their merger trees are divided (§ 2.2). The P-Millennium has by far the largest number of partitions, so its partition fraction can be set most finely, in steps of 1/10241/1024.

SimulationLL (h−1 Mpch^{-1}\,\mathrm{Mpc})kkNpN_{\mathrm{p}}MpM_{\mathrm{p}} (h−1 M⊙h^{-1}\,\mathrm{M}_{\odot})Cosmology
P-Millennium542.2102450403^{3}1.06×1081.06 \times10^{8}Planck 2013
Millennium I5006421603^{3}8.61×1088.61 \times10^{8}WMAP1
Millennium II1006421603^{3}6.88×1066.88 \times10^{6}WMAP1

Table 1. Summary of NN-body simulations used. LL is the box size, kk the number of merger-tree partitions, NpN_{\mathrm{p}} the particle number and MpM_{\mathrm{p}} the particle mass; the cosmology column lists the underlying parameters: Planck 2013 (Planck Collaboration [40]) for the P-Millennium and approximately first-year WMAP (Spergel et al. 2003) for the Millennium simulations. References: P-Millennium, Baugh et al. (2019); Millennium I, Springel et al. (2005); Millennium II, Boylan-Kolchin et al. (2009).

Millennium I and Millennium II share k=64k=64 but differ in volume by a factor of 125, so comparing them varies the box size at a fixed partition count (§ 5.4).

We run two variants of GALFORM, chosen because they populate the same merger trees differently: L16 (Lacey et al. 2016) and GP14 (Gonzalez-Perez et al. 2014), which were calibrated for different purposes and differ in the parameters listed in Table B1. Both models were originally calibrated on an NN-body simulation of resolution similar to Millennium I but with a different cosmology, and were recalibrated on the P-Millennium by Baugh et al. (2019). We run them on the Millennium I and Millennium III trees without further recalibration, so on those boxes they do not reproduce the calibration data in detail. This does not affect the tests here: SaUCE requires a galaxy population, not a realistic one, and the reference and sparse estimates are always drawn from the same population.

ParameterDescriptionL16GP14
Supernova feedback and gas return
VhotV_{\mathrm{hot}}Mass-loading velocity scale (km s−1^{-1})320380
αhot\alpha_{\mathrm{hot}}Mass-loading power-law slope3.43.2
αreheat\alpha_{\mathrm{reheat}}Return rate of ejected gas1.001.26
AGN feedback
ϵAGN\epsilon_{\mathrm{AGN}}Fraction of Eddington luminosity for heating0.010.04
αcool\alpha_{\mathrm{cool}}Hot-halo threshold for AGN feedback0.800.72
Star formation
νSF\nu_{\mathrm{SF}}H2_2 depletion rate (Gyr−1^{-1})0.740.50
fstabf_{\mathrm{stab}}Disc stability threshold0.900.80
IMF in burstsTop-heavyKennicutt

Table B1. Parameters that differ between the two GALFORM variants; all others are identical. Values are those recalibrated for the P-Millennium by Baugh et al. (2019) where recalibration changed them, and as published otherwise. The recalibration changed αcool\alpha_{\mathrm{cool}} (from 0.60) and VhotV_{\mathrm{hot}} (from 425 km s−1^{-1}) in GP14, and αhot\alpha_{\mathrm{hot}} (from 3.2) and αreheat\alpha_{\mathrm{reheat}} (from 0.64) in L16, to the values listed. VhotV_{\mathrm{hot}} applies to both quiescent and burst star formation. Both variants use a Kennicutt (1983) IMF for quiescent star formation; in bursts L16 uses a top-heavy IMF with slope x=1x=1.

Merger tree partitioning

The merger trees of each simulation are divided between kk files, the partitions. Each tree is rooted on a halo at z=0z=0, contains that halo’s full progenitor history and lies in exactly one partition, so the partitions hold disjoint sets of haloes. We run GALFORM on each partition separately.

In cosmological simulations, how the haloes are distributed between the partitions is a free choice, illustrated in Figure 1. The simulation volume is typically divided into spatial sub-volumes, so that the merger histories in one file all come from a single contiguous region many times smaller than the box. For the simulations used here the tree-building code instead assigns each z=0z=0 halo, with its progenitors, to one of the kk files independently and with equal probability, so the haloes in a partition can come from anywhere in the volume and each partition spans the full box. Each partition therefore samples every scale the box resolves. This does not affect the bias of SaUCE (§ 3.5), but it keeps the scatter small, and the partition fractions of § 5 depend on it. Because the assignment is random, the partitions are equal in size only on average; this spread adds to the variance of the estimate (Appendix § 10.1).

Partition architecture showing spatial sub-volume division and merger-tree sparse sampling

Figure 1. Partition architecture. (a) Spatial sub-volume division, with haloes grouped by location; each coloured region is one contiguous sub-volume. (b) The merger-tree sparse-sampling or partitioning used here, with partitions (blue, red, green) assigned at random to z=0z = 0 merger trees rather than by spatial position; grey points are unselected haloes. Partition members are spread throughout the full simulation volume, preserving the spatial distribution.

The random assignment applies to the z=0z=0 haloes. At z>0z>0 each halo is still selected with probability m/km/k, since it belongs to exactly one tree, but progenitors that share a z=0z=0 root are always selected together. This adds to the variance at z>0z>0 but not to the bias (§ 3.5).

Galaxy sample definitions

We define galaxy samples by number density rather than by stellar mass. A fixed stellar-mass threshold selects a different number density in every box, model and redshift, because each combination has its own stellar mass function; a number-density cut removes that confound, so that a comparison between boxes or between model variants isolates the quantity of interest. We rank galaxies by stellar mass and set the threshold that gives the target nˉ\bar{n} in each full catalogue.

Three number-density samples underpin the convergence results. The galaxy counts below follow from nˉ\bar{n} and the P-Millennium volume, so they are the same for either model and at every redshift; the stellar-mass thresholds are those of the L16 model at z=0z=0, and vary with model and redshift:

  • (i) nˉ=10−2 h3 Mpc−3\bar{n} = 10^{-2}\ h^{3}\ \mathrm{Mpc}^{-3}: 1.59×1061.59 \times10^{6} galaxies (M∗>109.71 h−1 M⊙M_{*} > 10^{9.71}\ h^{-1}\ M_{\odot}).

  • (ii) nˉ=10−3 h3 Mpc−3\bar{n} = 10^{-3}\ h^{3}\ \mathrm{Mpc}^{-3}: 1.59×1051.59 \times10^{5} galaxies (M∗>1010.64 h−1 M⊙M_{*} > 10^{10.64}\ h^{-1}\ M_{\odot}).

  • (iii) nˉ=10−4 h3 Mpc−3\bar{n} = 10^{-4}\ h^{3}\ \mathrm{Mpc}^{-3}: 1.59×1041.59 \times10^{4} galaxies (M∗>1010.96 h−1 M⊙M_{*} > 10^{10.96}\ h^{-1}\ M_{\odot}). Shot-noise limited at small scales.

Across all simulations, model variants and three redshifts, these densities correspond to stellar-mass thresholds spanning M∗≈109.4M_{*} \approx10^{9.4}–1011.5 h−1 M⊙10^{11.5}\ h^{-1}\ M_{\odot}. The comparisons between models and boxes (Sections § 5.3 and § 5.4) use a fourth, denser sample, nˉ=0.0408 h3 Mpc−3\bar{n} = 0.0408\ h^{3}\ \mathrm{Mpc}^{-3}, the density of M∗>109 h−1 M⊙M_{*} > 10^{9}\ h^{-1}\ M_{\odot} in the P-Millennium L16 catalogue at z=0z = 0 (6.50×1066.50 \times10^{6} galaxies). We use a denser sample there because Millennium II has 1/125 of the volume of Millennium I: at nˉ=10−2 h3 Mpc−3\bar{n} = 10^{-2}\ h^{3}\ \mathrm{Mpc}^{-3} it would hold only 10410^{4} galaxies.

We evaluate the convergence of the SaUCE estimator at three redshifts, z=0z = 0, z≈0.5z \approx0.5 and z≈1.5z \approx1.5, spanning the primary redshift coverage of surveys such as DESI (DESI Collaboration 2016) and Euclid ([7]).

Redshift-space catalogues

We also consider the clustering in redshift space. GALFORM assigns each galaxy the peculiar velocity of its host halo or subhalo. We place a galaxy in redshift space by displacing it along one box axis, the line of sight, by v∥/(aH)v_{\parallel}/(aH), where v∥v_{\parallel} is its line-of-sight peculiar velocity, aa the scale factor and HH the Hubble parameter at the snapshot redshift; displacements wrap at the periodic boundary.

SaUCE estimator

We now derive the main result of the paper: an estimator that recovers the full-box 2PCF from the galaxy catalogues of m≤km \leq k partitions.

Reference estimator

We measure ξ(r)\xi(r) from its definition as the excess probability, relative to a random distribution of points of the same mean density, of finding a galaxy pair at a given separation. For a separation bin centred on rr,

ξ^(r)=DD(r)RR(r)−1,RR(r)=12ND(ND−1)ΔV(r)V,(1)\hat{\xi}(r) = \frac{DD(r)}{RR(r)} - 1,\qquad RR(r) = \frac{1}{2}N_D(N_D - 1)\frac{\Delta V(r)}{V}, \tag*{(1)}

where DD(r)DD(r) is the number of galaxy pairs with separations in the bin, NDN_D is the number of galaxies in the catalogue being counted, VV is the volume of the simulation box and ΔV(r)\Delta V(r) is the volume of the radial shell. RR(r)RR(r) is the number of pairs expected from NDN_D points placed independently and uniformly in the box, and for a periodic box it is exact: with periodic boundaries every shell is complete for r<L/2r < L/2, so there are no boundary or mask effects for a random catalogue to measure, as there are in observational surveys ([1]).

Why a partition subset biases the estimate

Random subsampling of galaxies scales DDDD and RRRR by the same factor, and so adds shot noise but no bias ([2]).

Selecting mm of the kk partitions is different, because it samples haloes rather than galaxies: every galaxy in a halo is retained or discarded with its host. Whether a pair survives is then decided by how many partitions it occupies, so what we need are the probabilities that one, and that two, specified partitions are among those drawn. The mm are drawn uniformly from the kk without replacement, and we call one such draw a selection. Any one partition is drawn with probability m/km/k, and once it has been, a second specified partition must be one of the remaining m−1m - 1 taken from the remaining k−1k - 1:

p1=mk,p2=m(m−1)k(k−1).(2)p_{1} = \frac{m}{k},\qquad p_{2} = \frac{m(m - 1)}{k(k - 1)}. \tag*{(2)}

A pair confined to one partition survives with probability p1p_{1}, and a pair spread across two survives only if both are drawn, with probability p2p_{2}.

Galaxy pairs divide into one-halo pairs, whose two galaxies share a halo, and two-halo pairs, whose galaxies occupy different haloes (e.g. Cooray & Sheth [3]). How a sample divides between the two is set by HOD (Berlind & Weinberg [4]; Zheng et al. [5]), which differs between galaxy formation models and between samples. No halo is split across partitions (§ 2.2), so both galaxies of a one-halo pair lie in one partition and the pair survives with probability p1p_{1}. Two-halo pairs generally have their haloes in different partitions, and survive with probability p2p_{2}: at z=0z = 0 two distinct haloes share a partition with probability only 1/k1/k, although at z>0z > 0 progenitors of a common root always do (§ 3.5).

The naive estimate normalises the combined catalogue by its own RRRR, which scales on average as the square of the galaxy count and hence as (m/k)2(m/k)^2. Two-halo pairs in different partitions are retained at p2=(m/k)2(1−1/m)/(1−1/k)p_{2} = (m/k)^2(1 - 1/m)/(1 - 1/k), so they are understated by a factor of about (m−1)/m(m - 1)/m; the fraction 1/k1/k that share a partition are retained at p1p_{1} and overstated by k/mk/m, so at z=0z = 0 two-halo pairs as a whole are retained, on average, at exactly (m/k)2(m/k)^2. One-halo pairs, all retained at p1=m/kp_{1} = m/k, are overstated by a factor k/mk/m. For k=1024k = 1024 and m=8m = 8 this is a factor of 128.

Derivation of the weights

What sets a pair’s survival probability is not whether it is a one-halo or a two-halo pair, but whether its galaxies lie in one partition or two. We therefore weight pairs by partition, with the corrected pair count:

DD(r)=wautoDDauto(r)+wcrossDDcross(r),(3)DD(r) = w_{\mathrm{auto}}DD_{\mathrm{auto}}(r) + w_{\mathrm{cross}}DD_{\mathrm{cross}}(r), \tag*{(3)}

where DDautoDD_{\mathrm{auto}} is the number of pairs whose galaxies lie in the same selected partition and DDcrossDD_{\mathrm{cross}} the number whose galaxies lie in two different selected partitions. Because the classification is by partition rather than by halo, it can be read directly off the catalogue, and no approximation enters. This weighted count, not the raw sum DDauto+DDcrossDD_{\mathrm{auto}} + DD_{\mathrm{cross}} of the naive estimate, is the DD(r)DD(r) that enters equation (1).

Write DDauto(k)DD_{\mathrm{auto}}^{(k)} and DDcross(k)DD_{\mathrm{cross}}^{(k)} for the same two counts in the full kk-partition catalogue; together they give the full-box pair count. Every pair in DDauto(k)DD_{\mathrm{auto}}^{(k)} survives with probability p1p_1 and every pair in DDcross(k)DD_{\mathrm{cross}}^{(k)} with probability p2p_2. Writing E\mathbb{E} for the average over the random choice of partitions, at fixed full-box catalogue, summing over pairs gives, exactly,

E[DDauto]=p1DDauto(k),(4)\mathbb{E}[DD_{\mathrm{auto}}] = p_1 DD_{\mathrm{auto}}^{(k)}, \tag*{(4)}
E[DDcross]=p2DDcross(k).(5)\mathbb{E}[DD_{\mathrm{cross}}] = p_2 DD_{\mathrm{cross}}^{(k)}. \tag*{(5)}

On average, the selected catalogue’s RRRR is smaller than the full-box RRRR by (m/k)2(m/k)^2, so the corrected count must on average be smaller than the full-box count by the same factor:

wautop1DDauto(k)+wcrossp2DDcross(k)DDauto(k)+DDcross(k)=m2k2.(6)\frac{w_{\mathrm{auto}}p_1DD_{\mathrm{auto}}^{(k)}+w_{\mathrm{cross}}p_2DD_{\mathrm{cross}}^{(k)}}{DD_{\mathrm{auto}}^{(k)}+DD_{\mathrm{cross}}^{(k)}}=\frac{m^2}{k^2}. \tag*{(6)}

One pair of weights must serve every separation bin, and DDauto(k)DD_{\mathrm{auto}}^{(k)} and DDcross(k)DD_{\mathrm{cross}}^{(k)} stand in a different ratio in each bin, so the two classes must be matched separately. Each class is divided by its survival probability and multiplied by (m/k)2(m/k)^2:

wauto=(m/k)2p1=mk,wcross=(m/k)2p2=m(k−1)k(m−1).(7)w_{\mathrm{auto}}=\frac{(m/k)^2}{p_1}=\frac{m}{k},\qquad w_{\mathrm{cross}}=\frac{(m/k)^2}{p_2}=\frac{m(k-1)}{k(m-1)}. \tag*{(7)}

Up to the common factor (m/k)2(m/k)^2, which matches the normalisation of the selected catalogue, these are Horvitz–Thompson weights (Horvitz & Thompson 1952): each pair is weighted by the inverse of its probability of being selected, so the weighted count is an unbiased estimate of the full-box count. The partitions play the role of clusters in single-stage cluster sampling (Cochran 1977), and a pair’s selection probability is the probability that the one or two partitions it occupies are all drawn. The same idea underlies the pairwise inverse-probability weights that correct for galaxies left unobserved by fibre assignment in redshift surveys (Bianchi & Percival 2017); here the probabilities are known exactly, rather than estimated from many realisations of the assignment.

Equations (4) and (5) involve no approximation, so the corrected pair count is exactly unbiased (Appendix A). ξ^\hat{\xi} itself is a ratio: RRRR is built from the selected galaxy count, which varies from one selection to another, so ξ^\hat{\xi} is unbiased only to leading order; Appendix C2 finds its mean offset from the reference consistent with zero.

Setting m=km=k gives wauto=wcross=1w_{\mathrm{auto}}=w_{\mathrm{cross}}=1 and recovers the full catalogue. The smallest usable configuration is m=2m=2, the fewest partitions for which cross pairs exist.

Substituting (7) into equations (3) and (1) gives the SaUCE estimator,

ξ^(r)=1RR(r)[mkDDauto+m(k−1)k(m−1)DDcross]−1,(8)\hat{\xi}(r)=\frac{1}{RR(r)}\left[\frac{m}{k}DD_{\mathrm{auto}}+\frac{m(k-1)}{k(m-1)}DD_{\mathrm{cross}}\right]-1, \tag*{(8)}

with RR(r)RR(r) built from the galaxy count NDN_D of the combined mm-partition catalogue.

Redshift space

Converting redshift to radial distance under the assumption of pure Hubble flow (§ 2.4) displaces galaxies along the line of sight by their peculiar velocities, making the clustering anisotropic (Kaiser 1987). We bin pairs in the redshift-space separation ss, measured from the displaced positions, and in μ\mu, the cosine of the angle between the pair separation and the line of sight. The periodic-box estimator generalises directly,

ξ^(s,μ)=DD(s,μ)RR(s,μ)−1,(9)\hat{\xi}(s,\mu)=\frac{DD(s,\mu)}{RR(s,\mu)}-1, \tag*{(9)}

where RR(s,μ)RR(s,\mu) is given by equation (1) with the volume of the (s,μ)(s,\mu) bin in place of ΔV(r)\Delta V(r). The anisotropy is summarised by the multipole moments (e.g. Sánchez et al. 2017)

ξ^ℓ(s)=2ℓ+12∫−11ξ^(s,μ)Lℓ(μ) dμ,(10)\hat{\xi}_{\ell}(s)=\frac{2\ell+1}{2}\int_{-1}^{1}\hat{\xi}(s,\mu)L_{\ell}(\mu)\,d\mu, \tag*{(10)}

where LℓL_{\ell} is the Legendre polynomial of order ℓ\ell; the monopole (ℓ=0\ell=0) and quadrupole (ℓ=2\ell=2) carry most of the signal.

The correction is unchanged. The derivation of § 3.3 is made bin by bin and uses only each galaxy’s partition label, which a line-of-sight displacement does not change, so the same weights apply to DD(s,μ)DD(s,\mu), and indeed to any binning of the pairs, including the projected correlation function wp(rp)w_p(r_p). Each multipole is a linear functional of ξ^(s,μ)\hat{\xi}(s,\mu), so the multipoles inherit the same leading-order unbiasedness.

Assumptions and validity

The weights require only that whole partitions are selected, mm of the kk drawn uniformly without replacement, and that whether a galaxy enters the sample does not depend on which partitions are drawn. In our tests the stellar-mass threshold is fixed from the full catalogue (§ 4.1); in a calibration run it must instead be set from the sparse catalogue itself, and then varies slightly between selections. Galaxies near the threshold then enter some selections and not others, which in our tests shifts the mean ξ^\hat{\xi} by at most 1 per cent and barely changes the scatter.

Two further conditions ensure that the sparse catalogue contains the same galaxies as the full one. The galaxy formation model must process each merger tree independently, so that running it on a subset of trees reproduces those trees’ galaxies; this holds for standard semi-analytic models but fails for processes that couple trees, such as a non-local treatment of reionisation. And no tree may be split between partitions, since the model needs a halo’s full history to populate it; the weights themselves use only each galaxy’s partition label.

The weights place no condition on how trees are assigned to partitions, since equations (4) and (5) hold however the full-box pairs divide between DDauto(k)DD_{\mathrm{auto}}^{(k)} and DDcross(k)DD_{\mathrm{cross}}^{(k)}. Two consequences follow. Archives that store trees in files by spatial region can use those files directly as partitions: the estimate remains unbiased, but its variance is larger, because compact partitions sample the box unevenly (Figure 1), so the partition fractions of § 5 should not be assumed to carry over. And because our trees are partitioned at z=0z=0, progenitors that share a root always lie in the same partition, so at z>0z>0 partition membership is correlated with pair separation. This changes how pairs divide between the two classes, and hence the variance, but not the bias.

The correction acts only on DDDD. We normalise analytically because the box is periodic, which leaves the choice of partitions as the only source of scatter (Appendix A), but the correction works equally with RRRR counted from a random catalogue, as in the estimator of Landy & Szalay [1] for a survey geometry.

Validation

Every result in Section 5 compares a SaUCE estimate built from mm partitions with a reference built from all kk; we call f=m/kf=m/k the partition fraction. Each selection (Section 3.2) corresponds to the first mm partitions of a random permutation of the kk, so selections at fixed mm are independent, while a larger ff contains every partition used at a smaller one, and results at neighbouring ff are correlated. Unless stated otherwise, each configuration is measured over 105 permutations; the convergence thresholds use 10,500 (Section 4.4).

Reference

The reference ξref(r)\xi_{\rm ref}(r) is the correlation function measured from all kk partitions, from equation (1). All pair counts, for the reference and for every selection, are tabulated with Corrfunc (Sinha & Garrison (2020)). Each result uses a subset of the test cases of Sections 2.1 and 2.3 and states which; within a case, the same stellar-mass threshold is applied to the reference and to every selection.

SaUCE estimates the full-box correlation function of a given NN-body realisation. The reference is the estimate made using all of the partitions, so cosmic variance of the realisation does not enter the comparison; an application must add it separately (Section 5.6).

Separation bins

We count pairs in logarithmic bins of width 0.1 dex. The two ends of the range are not free, being set by the resolution of the simulation and by the size of the box; the width is a resolution choice.

The bin range is set by the simulation. The P-Millennium resolves haloes down to Mh=2.12×109 h−1 M⊙M_{\rm h}=2.12 \times10^{9}\,h^{-1}\,M_{\odot}, the mass of 20 particles (Baugh et al. (2019)), whose comoving r200r_{200} (at 200 times the mean density) is 0.031 h−1h^{-1} Mpc and is independent of redshift in comoving units. Pairs closer than this increasingly involve satellites whose subhaloes the simulation does not resolve, so the separation is set by GALFORM’s orbital assignment rather than by the NN-body calculation. We therefore report r≥0.03 h−1r \ge0.03\,h^{-1} Mpc, the nearest bin centre on the grid to that scale. This floor comes from resolution: at nˉ=10−2 h3 Mpc−3\bar{n}=10^{-2}\,h^{3}\,{\rm Mpc}^{-3} every bin down to r=0.003 h−1r=0.003\,h^{-1} Mpc holds more than 2000 reference pairs, ten times closer in than we report.

The outer edge is set by the box. Clustering predictions are conventionally restricted to a tenth of the box side (e.g. Orsi et al. (2008)), which is 54 h−1h^{-1} Mpc for the P-Millennium. We report r≤30 h−1r \le30\,h^{-1} Mpc, comfortably inside this limit and the largest separation at which ξref\xi_{\rm ref} is positive in every sample and redshift tested.

The scatter fits of Section 5.5, the covariance of Section 5.6 and the tests of Appendix C use a coarser grid: 30 bins of width 0.148 dex with limits of 0.01 and 271 h−1h^{-1} Mpc; in Section 5.5, Millennium II and II use 30 bins of 0.12 dex, from 0.01 to 50 and from 0.002 to 10 h−1h^{-1} Mpc respectively. The fits and Appendix C keep the 25 bins with centres below 50 h−1h^{-1} Mpc, and the covariance keeps the first 27, spanning 0.01–98 h−1h^{-1} Mpc. These ranges are wider than the one above because they characterise the scatter of the estimator, not its accuracy.

Accuracy metrics

We measure accuracy on the ratio of a selection’s estimate to the reference,

R(r)=ξ^(r)ξref(r),(11)R(r)=\frac{\hat{\xi}(r)}{\xi_{\rm ref}(r)}, \tag*{(11)}

over the ensemble of independent selections at fixed mm. We report two quantities:

(i) the bias, ⟨R⟩−1\langle R\rangle-1, the mean over selections. The mean is the statistic in which SaUCE is unbiased; a median of RR, or a mean of log⁡10R\log_{10}R, is offset from zero even for an exactly unbiased estimator (Appendix C);

(ii) the 1σ\sigma band, the 16th–84th percentile range of log⁡10R\log_{10}R, in dex. This is the scatter a user running a single selection faces. Its half-extent is the larger of the distances of the 16th and 84th percentiles from zero, so an offset or asymmetric band is judged by its worse side. The distribution of RR is close to lognormal (Appendix C), so the width of the band follows from the variance, which Appendix A predicts in closed form and Section 5.5 fits.

Results are sometimes broken down by clustering regime rather than bin by bin, as diagnostic columns in Table 2. Those breakdowns use three ranges: one-halo, r∈[0.3,1]r \in[0.3,1]; transition, r∈[1,5]r \in[1,5]; and two-halo, r∈[5,30] h−1r \in[5,30]\,h^{-1} Mpc, holding six, seven and eight bins respectively on this grid.

Acceptance criterion

The convergence threshold f∗f^{*} is the smallest partition fraction at which the 1σ1\sigma band of log⁡10R\log_{10}R lies inside ±0.05\pm0.05 dex in every bin of the range considered, and continues to do so at every larger fraction measured. At f∗f^{*}, therefore, at least 68 per cent of selections lie within 0.05 dex in each bin. The bins are correlated but not perfectly (Section 5.6), so the fraction of selections within tolerance in all bins at once is lower than in the worst bin and can fall below 68 per cent. A selection returning ξ^≤0\hat{\xi}\leq0 in a bin lies outside the band and counts as a failure.

The tolerance, 0.05 dex, is a factor of 100.05=1.1210^{0.05}=1.12, or 12 per cent. We adopt it to define convergence; calibration against data may need a tighter tolerance, as discussed in item (ii) of Section 6.

A selection’s corrected pair count is a sum of the auto-pair counts of its partitions and the cross-pair counts of its partition pairs, so we count pairs once for every pair of the kk partitions and obtain any selection by summation. This gives f∗f^{*} from 10,500 selections at every integer mm from 2 to k−1k-1; the first 105 are those used elsewhere in the paper. Depth matters: requiring acceptance at every larger fraction lets one noisy failure at large mm push f∗f^{*} up, so 105 selections give f∗f^{*} typically 15, and up to 40, per cent higher. At 10,500 the bootstrap 68 per cent interval on f∗f^{*} is within ±3\pm3 per cent, and halves and quarters of the ensemble agree to within 3.4 and 7 per cent. We quote f∗f^{*} to two significant figures.

The criterion cannot be met where selections often contain no pair in a bin, since the weights cannot reweight an empty count. This happens at the smallest separations and smallest mm in the sparser samples (Appendix C3), and is why the sparsest sample, nˉ=10−4 h3 Mpc−3\bar{n}=10^{-4}\,h^{3}\,{\rm Mpc}^{-3}, meets the criterion over the full range only above f=96f=96 per cent, where SaUCE saves almost nothing.

Redshift-space multipoles

In redshift space the reference is the monopole ξ0,ref(s)\xi_{0,\mathrm{ref}}(s) and quadrupole ξ2,ref(s)\xi_{2,\mathrm{ref}}(s) of the merged kk-partition catalogue, from equation (10). We bin pairs in ss on the grid of § 4.2 and in μ\mu in 24 linear bins over 0≤μ≤10\leq\mu\leq1; the periodic box with a fixed line of sight gives ξ(s,μ)=ξ(s,−μ)\xi(s,\mu)=\xi(s,-\mu), so negative μ\mu adds nothing.

The quadrupole changes sign near the one-halo to two-halo transition, so its log-ratio is undefined there. Both multipoles are therefore assessed on the residual normalised by the monopole,

eℓ(s)=ξℓ(s)−ξℓ,ref(s)ξ0,ref(s),ℓ=0, 2,(12)e_{\ell}(s)=\frac{\xi_{\ell}(s)-\xi_{\ell,\mathrm{ref}}(s)}{\xi_{0,\mathrm{ref}}(s)},\qquad\ell=0,\,2, \tag*{(12)}

which is defined where ξ2,ref\xi_{2,\mathrm{ref}} crosses zero. We apply the criterion of § 4.4 to the 16th–84th percentile band of log⁡10(1+eℓ)\log_{10}(1+e_{\ell}), in dex. For the monopole, 1+e0=ξ0/ξ0,ref1+e_{0}=\xi_{0}/\xi_{0,\mathrm{ref}}, so this is the criterion used for ξ(r)\xi(r), with ξ0\xi_{0} in place of ξ\xi.

Convergence and accuracy of SaUCE

Section 5.1 shows how the estimated 2PCF depends on the partition fraction. Section 5.2 extends these results to different galaxy samples and redshifts. Section 5.3 shows that the scatter is the same for two GALFORM variants, and Section 5.4 whether a partition fraction transfers between boxes. Section 5.5 gives a practical recipe for predicting the estimator’s variance, and Section 5.6 characterises its bin-to-bin covariance. Section 5.7 tests the correction in redshift space, and asks what each multipole costs. Partition fractions quoted here are specific to the P-Millennium unless stated otherwise.

Convergence with partition fraction

Figure 2 shows the effect of the correction for nˉ=10−2 h3 Mpc−3\bar{n}=10^{-2}\,h^3\,\mathrm{Mpc}^{-3} at z=0z=0. The naive estimate overstates ξ(r)\xi(r) on small scales by the factor k/mk/m predicted in § 3.2, while on large scales, where almost all pairs are two-halo pairs, the overstatement vanishes.

Figure 2. The naive and SaUCE estimates of ξ(r)\xi(r) against the reference, for nˉ=10−2 h3 Mpc−3\bar{n}=10^{-2}\,h^3\,\mathrm{Mpc}^{-3} in the L16 model on the P-Millennium at z=0z=0. (a) ξ(r)\xi(r) from the naive and SaUCE estimators at f=1.56f=1.56 per cent, against the reference (all k=1024k=1024 partitions; solid black), from which the SaUCE estimate is indistinguishable on this axis. (b) Fractional residual of the naive estimator, (ξnaive−ξref)/∣ξref∣(\xi_{\mathrm{naive}}-\xi_{\mathrm{ref}})/|\xi_{\mathrm{ref}}|. On small scales, where almost all pairs are one-halo pairs, it reaches k/m−1k/m-1 (§ 3.2); it falls to zero on large scales, where almost all pairs are two-halo pairs. (c) Fractional residual of the SaUCE estimator. Lines show the mean over the selections available at each ff, between 192 and 415 of them; shaded bands span the 16th–84th percentiles, the scatter of a single selection. The panels extend below the reported range, r≥0.03 h−1 Mpcr\geq0.03\,h^{-1}\,\mathrm{Mpc}, to show the naive bias reaching its plateau. Note the different yy-axis ranges of panels (b) and (c).

The estimator is unbiased, and what improves with ff is the scatter. The mean ratio ⟨R⟩\langle R\rangle of equation (11) stays within a few per cent of unity at every ff and separation, with no trend in ff, consistent with the noise on the mean (Appendix C2). The scatter falls steeply and smoothly with ff (Figure 3), and for nˉ=10−2 h3 Mpc−3\bar{n}=10^{-2}\,h^3\,\mathrm{Mpc}^{-3} is smaller at small separations than at large ones.

Figure 3. The scatter of SaUCE against the partition fraction ff, one radial bin per panel (P-Millennium, L16). The panels are the bins centred on r=0.03r = 0.03, 0.3, 3 and 30 h−1h^{-1} Mpc, and the nine curves in each are three number-density samples at three redshifts (legend); the r=0.03r = 0.03 h−1h^{-1} Mpc panel omits the sparsest sample, whose ξ^\hat{\xi} is too often ≤0\le0 there. The ordinate is the half-extent of the 1σ1\sigma band of log⁡10R\log_{10} R, so a curve crosses the dashed 0.05 dex line where that bin meets the criterion of § 4.4; the dotted line marks 0.01 dex. A selection returning ξ^≤0\hat{\xi} \le0 counts as a failure (§ 4.4). m=1m = 1 is excluded, since wcrossw_{\rm cross} is undefined for a single partition.

Which scales set the threshold depends on the sample (Table 2). Usually it is the two-halo range. For nˉ=10−3 h3 Mpc−3\bar{n}=10^{-3}\,h^3\,\mathrm{Mpc}^{-3} it is the smallest scales, r<0.3 h−1 Mpcr<0.3\,h^{-1}\,\mathrm{Mpc}, where that sample has few pairs per bin, though at every redshift only narrowly ahead of the two-halo range: 15, 21 and 29 per cent against 14, 20 and 28. The one-halo range is never limiting.

zznˉ\bar{n}mmf∗f^{*} [per cent]one-halotransitiontwo-halo
010−210^{-2}757.33.96.17.3
010−310^{-3}157159.61314
0.510−210^{-2}118122.84.312
0.510−310^{-3}212218.19.320
1.510−210^{-2}209201.43.120
1.510−310^{-3}299297.37.728

Table 2. The convergence threshold f∗f^{*} for the P-Millennium, from 10,500 selections at every mm: the smallest partition fraction at which the 1σ1\sigma band of log⁡10R\log_{10}R lies inside ±0.05\pm0.05 dex in every radial bin of the stated range, and at every larger fraction measured (§ 4.4). The full range is 0.03≤r≤30 h−1 Mpc0.03\leq r\leq30\,h^{-1}\,\mathrm{Mpc}. The one-halo (0.3–1), transition (1–5) and two-halo (5–30 h−1 Mpch^{-1}\,\mathrm{Mpc}) columns show which scales set it; the full range also contains the smallest scales, r<0.3 h−1 Mpcr<0.3\,h^{-1}\,\mathrm{Mpc}, which is why it exceeds all three columns. mm is the partition count, f∗=m/kf^{*}=m/k with k=1024k=1024, and nˉ\bar{n} is in h3 Mpc−3h^{3}\,\mathrm{Mpc}^{-3}. The sparsest sample, nˉ=10−4 h3 Mpc−3\bar{n}=10^{-4}\,h^{3}\,\mathrm{Mpc}^{-3}, meets the criterion over the full range only above f=96f=96 per cent.

Generalisation across samples and redshifts

Figure 3 shows how the scatter depends on the sample and redshift: within a panel, curves of one line style differ only in sample and curves of one colour only in redshift, and comparing panels isolates the scale.

The required fraction rises as the number density falls, because a sparser sample has fewer pairs in each bin. Over the full range and on two-halo scales it also rises with redshift at fixed nˉ\bar{n}. The number of galaxies is then the same, but the two-halo clustering amplitude ends_mid=1

is lower (near 10 h−1h^{-1} Mpc, ξref\xi_{\rm ref} is about a third lower at z≈1.5z \approx1.5 than at z=0z = 0), so the same scatter in the pair count is a larger fraction of ξ\xi. On one-halo and transition scales the threshold instead falls with redshift, for both samples in Table 2: for nˉ=10−2 h3 Mpc−3\bar{n} = 10^{-2}\,h^{3}\,\mathrm{Mpc}^{-3} it falls from 3.9 to 1.4 per cent on one-halo scales and from 6.1 to 3.1 per cent on transition scales between z=0z = 0 and z≈1.5z \approx1.5. We return to the cause in § 5.4.

Table 2 gives the fraction needed over the full range. A user who needs only smaller separations can run much less: Figure 4 gives, against rr, the fraction needed for every bin from 0.03 h−1h^{-1} Mpc out to rr to meet the criterion.

Figure 4. The partition fraction needed as a function of the largest separation used, for nˉ=10−2 h3 Mpc−3\bar{n} = 10^{-2}\,h^{3}\,\mathrm{Mpc}^{-3} on the P-Millennium. The ordinate is the largest per-bin threshold over all bins from 0.03 h−1h^{-1} Mpc out to rr, not the threshold of the bin at rr. nˉ=10−3 h3 Mpc−3\bar{n} = 10^{-3}\,h^{3}\,\mathrm{Mpc}^{-3} is not shown, since its smallest scales set the full-range requirement and its curves would be flat (Table 2).

Galaxy formation models

The bias of SaUCE does not depend on the galaxy formation model (§ 3.3), but its scatter could. We compare the L16 and GP14 models, which have different feedback prescriptions (Appendix B), in all three simulations, with galaxies selected to nˉ=0.0408 h3 Mpc−3\bar{n} = 0.0408\,h^{3}\,\mathrm{Mpc}^{-3} (Figure 5). The GP14 circles lie on the L16 lines throughout: the bands of the two models differ by no more than the uncertainty of a band measured from 105 selections (bin by bin, the L16/GP14 band ratio has median 0.94–1.00 in each box and redshift), and their thresholds agree to within one step of the power-of-two grid in mm, a factor of two in ff. The scatter is therefore the same for these two GALFORM variants, although their full-box correlation functions differ by up to 8 per cent in the P-Millennium at z=0z = 0, and by less than 3 per cent at $z \approx 1.5 ({xref}Appendix B`). Matched in number density, however, their HODs are similar: for this sample in Millennium I, at both redshifts, the halo masses at which ⟨Ncen⟩=1/2\langle N_{\rm cen}\rangle= 1/2 and ⟨Nsat⟩=1\langle N_{\rm sat}\rangle= 1 agree to within 0.02 dex, and ⟨Nsat⟩\langle N_{\rm sat}\rangle above the ⟨Nsat⟩=1\langle N_{\rm sat}\rangle= 1 mass is two within about 10 per cent.

Figure 5. The scatter of SaUCE for two galaxy formation models in three boxes, for nˉ=0.0408 h3 Mpc−3\bar{n}=0.0408\ h^3\ \mathrm{Mpc}^{-3}. Every panel has the ordinate of Figure 3, the half-extent of the 1σ1\sigma band at a single radial bin, with the 0.05 dex tolerance dashed and 0.01 dex dotted. The abscissa is the partition fraction ff in the left pair of panels and the galaxy count in the selected partitions, N‾gal\overline{N}_{\mathrm{gal}}, in the right pair. Within each pair the panels are r=0.3r=0.3 and 3 h−1 Mpc3\ h^{-1}\ \mathrm{Mpc}. Colour is the simulation (P-Millennium L=542.2L=542.2, Millennium I L=500L=500, Millennium II L=100 h−1 MpcL=100\ h^{-1}\ \mathrm{Mpc}), line style the redshift (solid z=0z=0, dashed z∼1.5z\sim1.5), and the model is L16 for the lines and GP14 for the circles on them. Against N‾gal\overline{N}_{\mathrm{gal}} the boxes fall on one relation at r=0.3 h−1 Mpcr=0.3\ h^{-1}\ \mathrm{Mpc} but not at 3 h−1 Mpc3\ h^{-1}\ \mathrm{Mpc} (§ 5.4).

The test therefore shows that the scatter is insensitive to differences in galaxy formation physics at fixed occupation; models whose HODs differ more, for example in satellite fraction, could have a different scatter.

Box size

We now ask whether a partition fraction measured in one box transfers to another. Millennium I and Millennium II both have k=64k = 64 partitions, and we test powers of two in mm over 0.03–7.5 h−1h^{-1} Mpc, the range all three boxes share, on the 0.1 dex grid of § 4.2. Millennium I meets the criterion at a similar fraction to the P-Millennium, within one step in mm, a factor of two in ff. Millennium II, whose volume is 125 times smaller, never meets it, even at f=50f = 50 per cent.

The cause is the small number of galaxies per partition. Each partition holds a fraction 1/k1/k of the galaxies while spanning the whole volume (§ 2.2), and all boxes are selected to the same nˉ\bar{n}, so each Millennium II partition holds 125 times fewer galaxies than a Millennium I partition. Millennium II’s scatter keeps falling as ff increases, as expected for sampling noise; the corrected pair counts are unbiased in any box (§ 3.3).

The right-hand panels of Figure 5 plot the scatter against the mean number of galaxies in the selected partitions, Ngal=nˉ f Lbox3N_{\rm gal} = \bar{n}\,f\,L_{\rm box}^{3}, instead of ff. The three boxes then fall on one relation at r=0.3 h−1r = 0.3\,h^{-1} Mpc: there the scatter is set by whether a halo enters the selection with its satellites, which depends only on how many galaxies are drawn. At r=3 h−1 Mpcr=3\ h^{-1}\ \mathrm{Mpc} they do not, because the pairs come from distinct haloes and their number depends on the density of the selection as well as its size. The number of pairs in a shell of volume VsV_{\mathrm{s}} is N‾galVs/(2Lbox3)\overline{N}_{\mathrm{gal}}V_{\mathrm{s}}/(2L_{\mathrm{box}}^{3}), so N‾gal/Lbox3\overline{N}_{\mathrm{gal}}/L_{\mathrm{box}}^{3} measures the pair count per unit shell volume; plotted against it (not shown), the boxes fall on one relation at r=3 h−1 Mpcr=3\ h^{-1}\ \mathrm{Mpc}, their median offsets from a single power law spanning a factor of 1.3, compared with 2.3 for N‾gal\overline{N}_{\mathrm{gal}}. A larger box therefore needs a smaller partition fraction for the same scatter: ff transfers between boxes through the galaxy count on small scales and through the pair count per unit shell volume on large scales.

In every panel of Figure 5 the z≈1.5z\approx1.5 band lies below the z=0z=0 band, and neither variable removes this: at these separations one-halo pairs, which enter a selection in bursts, are a smaller share of the total at higher redshift, as in the one-halo and transition columns of Table 2.

A recipe for the estimator's scatter

The closed-form variance of Appendix A confirms that the measured scatter is what the random choice of partitions predicts, but a user cannot evaluate it: it needs the full-box pair counts that SaUCE exists to avoid. We therefore fit the scatter empirically. Pair separation alone does not predict it: at fixed rr, the fractional scatter σ/∣ξref∣\sigma/|\xi_{\mathrm{ref}}| differs between number densities (Figure A1a, at z=0z=0) and between redshifts (Figure 3, whose band width follows from this variance; § 4.3), because rr says nothing about how many pairs contribute to a bin. Here σ\sigma is the standard deviation of ξ\xi over selections at fixed ff, measured bin by bin and reduced to one number per fitted row by the median over the bins of the range in question.

Figure A1. Monte Carlo versus closed-form partition-selection variance against pair separation, at z=0z=0 on the P-Millennium; the other two redshifts agree equally well (not shown). (a): empirical σ/∣ξref∣(r)\sigma/|\xi_{\mathrm{ref}}|(r) at m=32m=32 from 105 and 105 independent selections for the denser and sparser threshold respectively (solid, coloured by number-density threshold), with the closed-form ratio variance of equation (20) overplotted (dashed). (b): the residual σMC/σan−1\sigma_{\mathrm{MC}}/\sigma_{\mathrm{an}}-1 in each bin, where σMC\sigma_{\mathrm{MC}} is the empirical scatter of panel (a) and σan\sigma_{\mathrm{an}} one of two closed-form predictions. Solid: the ratio variance of equation (20). Dotted: the same calculation with the normalisation held fixed at its expectation, Var⁡[DD]/E[RR]\sqrt{\operatorname{Var}[DD]/\mathbb{E}[RR]}; the gap between the two curves is the variance that cancels between numerator and denominator. The sparsest threshold, nˉ=10−4.3 h3Mpc−3\bar{n}=10^{-4.3}\,h^3\mathrm{Mpc}^{-3}, is shot-noise limited and is excluded here as it is from the P-Millennium-only recipe (see § 5.5).

We fit two forms. The first is a power law in the galaxy count N‾gal\overline{N}_{\mathrm{gal}} of § 5.4,

σ∣ξref∣=ANN‾galpN(13)\frac{\sigma}{|\xi_{\mathrm{ref}}|}=A_N\overline{N}_{\mathrm{gal}}^{p_N} \tag*{(13)}

and the second separates ff and nˉ\bar{n},

σ∣ξref∣=AfBnˉC.(14)\frac{\sigma}{|\xi_{\mathrm{ref}}|}=A f^B\bar{n}^C. \tag*{(14)}

We fit ANA_N, pNp_N, AA, BB and CC. Within one box N‾gal∝nˉf\overline{N}_{\mathrm{gal}}\propto\bar{n}f, so equation (13) is equation (14) with B=CB=C; what it adds is LboxL_{\mathrm{box}}, which lets it cross box sizes. Equation (14) tests whether B=CB=C, and gives the dependence on ff at fixed nˉ\bar{n}. We fit both by least squares in log⁡10\log_{10}, over all separations and separately on one-halo (r<1r<1), transition (1≤r<51\leq r<5) and two-halo (5≤r<50 h−1 Mpc5\leq r<50\ h^{-1}\ \mathrm{Mpc}) scales, ranges wider than those of § 4.3; Table 3 gives the all-scale coefficients and selection cuts. The fits use powers of two in mm on the coarser grids of § 4.2, each over at least 105 selections.

Equation (13)
CalibrationANA_NpNp_Nrms [dex]
P-Millennium45.6−0.584-0.5840.176
P-Millennium + Millennium I38.0−0.570-0.5700.159
Equation (14), P-Millennium + Millennium I
AABBCCrms [dex]
8.32×10−48.32\times10^{-4}−0.677-0.677−0.514-0.5140.136

Table 3. Coefficients for the all-scale recipes, fitted to the median fractional scatter over 0.01≤r<50 h−1 Mpc0.01\leq r<50\ h^{-1}\ \mathrm{Mpc}. The rms is the residual scatter in log⁡10(σ/∣ξref∣)\log_{10}(\sigma/|\xi_{\mathrm{ref}}|). In equation (14), use f=m/kf=m/k as a fraction and nˉ\bar{n} in h3 Mpc−3h^3\ \mathrm{Mpc}^{-3}. The P-Millennium-only fits use the nˉ=10−2\bar{n}=10^{-2} and 10−3 h3 Mpc−310^{-3}\ h^3\ \mathrm{Mpc}^{-3} samples and m=2,4,…,128m=2,4,\ldots,128; the two-simulation fits use the nˉ=10−2\bar{n}=10^{-2}–10−410^{-4} samples with power-of-two m≥2m\geq2, f≤0.125f\leq0.125 and selected galaxy counts above 1000.

Intermediate partition fractions were run only inside brackets around the 0.05 dex crossings of a preliminary, 105-selection estimate of the thresholds; they sample mm unevenly, so they are not used in fitting. The recipe predicts the typical scatter, the median over bins, whereas the criterion of § 4.4 is set by the worst bin, and equation (14) has no redshift term. A single bin meets a tolerance of tt dex when σ/∣ξref∣≲tln⁡10\sigma/|\xi_{\mathrm{ref}}|\lesssim t\ln10, or 0.115 for t=0.05t=0.05 dex, but at the thresholds of Table 2 equation (14) gives (extrapolating above f=0.125f=0.125) a median σ/∣ξref∣\sigma/|\xi_{\mathrm{ref}}| of only 0.026–0.103, so inverting it at 0.115 would underestimate f∗f^\ast by a factor of up to about nine. No single target fixes this, because the factor ranges from 1.2 to 2 across Table 2. The partition fraction is therefore taken from Table 2 and Figure 4 where they apply, and otherwise from repeated selections.

The left panel of Figure 6 shows that within one box N‾gal\overline{N}_{\mathrm{gal}} predicts the scatter: pooling every density, redshift and ff of the P-Millennium, the nˉ=10−2\bar{n}=10^{-2} and 10−3 h3 Mpc−310^{-3}\ h^3\ \mathrm{Mpc}^{-3} samples fall on one power law with pN=−0.58p_N=-0.58 and an rms scatter about the fit of 0.18 dex, which is the accuracy to which the recipe predicts σ\sigma. At fixed nˉ\bar n and box size σ∝fpN\sigma\propto f^{p_N}, so this is an rms of 0.18/0.58≈0.30.18/0.58 \approx0.3 dex, a factor of two, in the ff that gives a given typical scatter. Fitted range by range, the exponent is close to the value −1/2-1/2 expected for independent counts only on one-halo scales, where the scatter is set by how many galaxies are drawn; on larger scales the scatter falls faster with NgalN_{\mathrm{gal}}. The exponent depends on the range of ff fitted. At larger ff the measurements lie below the fit, since the scatter must reach zero at f=1f=1, so the recipe is conservative at large ff.

Figure 6. The fractional scatter σ/∣ξref∣\sigma/|\xi_{\mathrm{ref}}| of the SaUCE estimator and the recipe for predicting it (Section 5.5). Left: the median σ/∣ξref∣\sigma/|\xi_{\mathrm{ref}}| over the radial bins, 0.01≤r<50 h−1 Mpc0.01 \le r < 50\,h^{-1}\,\mathrm{Mpc} in 0.148 dex bins, against the galaxy count in the selected partitions, NgalN_{\mathrm{gal}}, for every density, redshift and ff of the P-Millennium. The solid line is equation (13) fitted to the nˉ=10−2\bar n=10^{-2} and 10−3 h3 Mpc−310^{-3}\,h^3\,\mathrm{Mpc}^{-3} samples (filled) at f=12.5f=12.5 per cent; the dashed line is the nˉ=10−4 h3 Mpc−3\bar n=10^{-4}\,h^3\,\mathrm{Mpc}^{-3} (open), is not fitted. Right: measured against predicted scatter for equation (14), fitted on the P-Millennium and Millennium I. Each filled circle is predicted by the fit to the other simulation; open circles are measurements outside the fitted range, shown against the fit to both; Millennium II (diamonds) is not fitted but predicted by the fit to both; the four full diamonds (two models, two redshifts; the models nearly coincide) pass the cuts of Table 3 and give the comparison of Section 5.5. The dotted line is 1:1. Both axes are capped at σ/∣ξref∣=10\sigma/|\xi_{\mathrm{ref}}|=10, which excludes one of the 649 points. Colour is number density on the left and simulation on the right.

For the P-Millennium-only fit, the sparsest sample is not fitted. Its points with σ<∣ξref∣\sigma<|\xi_{\mathrm{ref}}| lie on the extrapolated relation; points with σ>∣ξref∣\sigma>|\xi_{\mathrm{ref}}|, in any sample, lie above it. The recipe therefore applies where the scatter is smaller than the signal. Where the predicted σ\sigma approaches ∣ξ∣|\xi|, σ\sigma must come from repeated selections.

Pooled over all scales, and on one-halo scales, BB and CC differ by less than 0.2. On two-halo scales B<CB<C in each simulation separately (B−C=−0.63B-C=-0.63 and −0.68-0.68): spreading a fixed number of galaxies over more partitions lowers the two-halo scatter, since two-halo pairs lie mostly between partitions, so there ff and nˉ\bar n do not act only through NgalN_{\mathrm{gal}}.

The right panel of Figure 6 tests whether the recipe transfers between simulations. Fitted over all scales on the P-Millennium and Millennium I together, equation (14) gives B=−0.68B=-0.68 and C=−0.51C=-0.51. Predicting each simulation from the fit to the other raises the rms from 0.14 to 0.16 dex, across a change of cosmology and partition count; divided by ∣B∣|B|, these are factors of 1.6 and 1.7 in ff.

The data cannot constrain a box-size term: the P-Millennium and Millennium I have similar box sizes, and only four Millennium II measurements pass the cuts of Table 3, all at f=0.125f=0.125 and nˉ=10−2 h3 Mpc−3\bar n=10^{-2}\,h^3\,\mathrm{Mpc}^{-3} (two models at two redshifts), a single location in (f,nˉ)(f,\bar n). The two-simulation fit underestimates Millennium II’s σ\sigma (filled diamonds in Figure 6; its medians are taken over 0.010.01–10 h−1 Mpc10\,h^{-1}\,\mathrm{Mpc}, the edge of its grid) by about a factor of eight, overstating the precision. Equation (13), refitted on the same two simulations, contains the box size through NgalN_{\mathrm{gal}}, and overestimates it by about a factor of 2.3. We therefore recommend equation (14) for boxes of 500500–542 h−1 Mpc542\,h^{-1}\,\mathrm{Mpc}, and equation (13) for other box sizes.

Bin-to-bin covariance

The scatter characterised above is not independent between separation bins, and Figure 7 shows the bin-to-bin correlation matrix ρij\rho_{ij} of the SaUCE estimator, measured across selections at z≈1.5z\approx1.5 for the nˉ=10−2 h3 Mpc−3\bar n=10^{-2}\,h^3\,\mathrm{Mpc}^{-3} sample. A block of strongly correlated bins, with ρij≳0.5\rho_{ij}\gtrsim0.5, extends across the one-halo to two-halo scales (r∼0.1r\sim0.1–10 h−1 Mpc10\,h^{-1}\,\mathrm{Mpc}) at every partition fraction shown. This structure is a consequence of the partition sampling: adding or removing a partition changes the galaxy content coherently across a wide range of separations, so a selection that scatters high in one bin tends to scatter high in its neighbours. The correlation is therefore a property of which haloes are selected rather than of the pair counting within a fixed catalogue. Each ρij\rho_{ij} carries noise of ≈0.1\approx0.1 from 100 selections, so the isolated high and low cells in the largest-separation corners, where pair counts per partition are low, are noise. The final panel quantifies how that coherence decays, plotting the median off-diagonal correlation against bin separation ∣log⁡10ri−log⁡10rj∣|\log_{10}r_i-\log_{10}r_j|. The three sparser fractions track one another closely; f=1.25f=1.25 per cent decays more slowly, staying some 0.1 higher at every separation out to 2 dex.

Bin-to-bin correlation matrices and median off-diagonal correlation

Figure 7. Bin-to-bin correlation matrix ρij\rho_{ij} of the SaUCE estimator across selections (z≈1.5z \approx1.5, nˉ=10−2 h3 Mpc−3\bar{n} = 10^{-2}\,h^3\,\mathrm{Mpc}^{-3}, 0.148 dex bins), for f=1.56f = 1.56, 3.12, 6.25 and 12.5 per cent, from 100 selections each, the same number at every ff so that the panels are directly comparable. The final panel shows the median off-diagonal correlation against bin separation ∣log⁡10ri−log⁡10rj∣\lvert\log_{10}r_i-\log_{10}r_j\rvert. Error bars are the 16th–84th percentiles of 800 bootstrap resamplings of the bin pairs contributing to each point, the uncertainty on the median; bin pairs sharing a radial bin are not independent, so the bars are a lower bound. Separation bins holding fewer than four bin pairs are omitted.

How many independent quantities the bins carry is measured by the effective number of bins, Neff=(∑aλa)2/∑aλa2N_{\mathrm{eff}}=(\sum_a\lambda_a)^2/\sum_a\lambda_a^2, where λa\lambda_a is the aa-th eigenvalue of ρij\rho_{ij} and the sums run over all of them. NeffN_{\mathrm{eff}} equals the number of bins when the bins are uncorrelated and falls

to one when they are perfectly correlated. We compute NeffN_{\mathrm{eff}} from the correlation matrices of Figure 7, once for all 27 bins and once for each clustering regime of § 4.3, restricting ρij\rho_{ij} to that regime’s bins; the ranges quoted span the four fractions shown. Across all 27 bins Neff=6.0N_{\mathrm{eff}} = 6.0–7.8. The one-halo and transition regimes, four bins each on this grid, give 1.3–1.5 and 1.6–2.0, and the two-halo regime, six bins, gives 2.6–3.4.

Subdividing the radial range more finely therefore adds little independent information about the partition-selection scatter: the additional bins are almost perfectly correlated with their neighbours, raising the nominal bin count without raising NeffN_{\mathrm{eff}}.

A likelihood must therefore use the full covariance: a diagonal approximation would count correlated fluctuations as independent evidence and give artificially tight constraints. The correlation structure changes only slowly with ff and may be reused across the range used for calibration, but should be estimated at the largest ff in use, since denser selections retain correlations to larger separations.

The matrix of Figure 7 is the covariance of the estimator across selections: it captures the uncertainty introduced by choosing mm of kk partitions in a fixed simulation, and is measured cheaply by repeating the selection. It does not contain the cosmic variance of the parent NN-body realisation, which is common to every selection, nor the covariance of the data; a full error budget must add both.

Redshift-space distortions

Figure 8 shows the ratio of the redshift-space monopole to the real-space correlation function for nˉ=10−2 h3 Mpc−3\bar{n}=10^{-2}\,h^{3}\,\mathrm{Mpc}^{-3}. SaUCE tracks the reference at all three redshifts, including the finger-of-God suppression at small ss, where the velocity dispersion within haloes stretches pairs along the line of sight, and the Kaiser boost on large scales (Kaiser 1987).

Figure 8. Ratio of the redshift-space monopole to the real-space correlation function, ξ0(s)/ξ(r)\xi_0(s)/\xi(r), for nˉ=10−2 h3 Mpc−3\bar{n} = 10^{-2}\,h^3\,\mathrm{Mpc}^{-3} at the three redshifts annotated. Numerator and denominator come from the same selection. The reference is thick black, beneath the SaUCE estimates at f=3.12f = 3.12 per cent (red dashed) and 9.77 per cent (blue dash-dotted). The horizontal dashed line, annotated in each panel, is the linear-theory Kaiser boost 1+2βKaiser/3+βKaiser2/51 + 2\beta_{\mathrm{Kaiser}}/3 + \beta_{\mathrm{Kaiser}}^2/5, with βKaiser=(dln⁡D/dln⁡a)/b\beta_{\mathrm{Kaiser}} = (\mathrm{d}\ln D/\mathrm{d}\ln a)/b, DD the linear growth factor and the galaxy bias b=(ξgal/ξDM)1/2b = (\xi_{\mathrm{gal}}/\xi_{\mathrm{DM}})^{1/2} measured over 10–30 h−1 Mpch^{-1}\,\mathrm{Mpc} with ξDM\xi_{\mathrm{DM}} from linear theory.

At fixed ff the departure from the reference grows with redshift, for the reason given in § 5.2.

Figure 9 shows the scatter of both multipoles, from the same selections as in real space. The multipoles were measured at powers of two in mm up to 512 and at every integer mm in some of the intervals between them; where a threshold falls in an interval that was not filled, we quote the bracket. At a depth of 105 selections, the monopole’s full-range threshold is 9.5 per cent at z=0z=0 and 18 per cent at z≈0.5z\approx0.5, compared with 8.2 and 12 per cent for ξ(r)\xi(r). At z≈1.5z\approx1.5 the bound is 12.5<f∗≤2512.5<f^{*}\leq25 per cent, which contains the real-space threshold of 24 per cent, so their ordering is unresolved. At z≤0.5z\leq0.5 its requirement is set by the smallest separations, and at z≈1.5z\approx1.5 by the two-halo range, as in real space. The quadrupole is far more expensive: over the full range it meets the criterion at f∗≃50f^{*}\simeq50 per cent at z=0z=0, the top of the measured range, but not at the measured fractions up to 50 per cent at the higher redshifts.

Figure 9. The scatter of the redshift-space multipoles against the partition fraction ff (P-Millennium, L16, nˉ=10−2 h3 Mpc−3\bar{n}=10^{-2}\,h^{3}\,\mathrm{Mpc}^{-3}), at z=0z=0, z≈0.5z\approx0.5 and z≈1.5z\approx1.5 (colour). Left: the monopole ξ0(s)\xi_0(s). Right: the quadrupole ξ2(s)\xi_2(s), from the same selections. The ordinate is the half-extent of the 1σ1\sigma band of log⁡10(1+eℓ)\log_{10}(1+e_\ell) (Section 4.5) in the worst bin over 0.03≤s≤30 h−1 Mpc0.03\leq s\leq30\,h^{-1}\,\mathrm{Mpc}, with eℓe_\ell from equation (12). A curve meets the criterion of § 4.4 where it falls below the dashed 0.05 dex line; the dotted line marks 0.01 dex. At the smallest ff the band extends below eℓ=−1e_\ell=-1, where log⁡10(1+eℓ)\log_{10}(1+e_\ell) is undefined, so those points are not drawn.

At every redshift the quadrupole’s worst bins are the three smallest, s≤0.05 h−1 Mpcs\leq0.05\,h^{-1}\,\mathrm{Mpc}. Without them the threshold falls to 31 per cent at z=0z=0, while 25<f∗≤5025<f^{*}\leq50 per cent at the higher redshifts; intermediate partition counts within this bracket were not run. Range by range, the quadrupole needs more than the monopole everywhere, two to four times as much on one-halo scales.

The smallest scales are hardest because the quadrupole is a difference between the clustering along and across the line of sight. There that difference comes from the velocity dispersion within individual haloes, and SaUCE keeps or discards each halo whole, so the massive, high-dispersion haloes that contribute most of it are either all in a selection or all out of it. We attribute to the same effect the cost of the monopole’s smallest separations: at the same depth of 105 selections, its bins at s≤0.05 h−1 Mpcs\leq0.05\,h^{-1}\,\mathrm{Mpc} need 5.4–18 per cent, against at most 1 per cent for the same separations in real space (Figure 3). A quadrupole analysis should therefore budget for the range of separations it uses, not for the monopole’s fraction.

Conclusions

SaUCE allows galaxy clustering to play a role in the calibration of galaxy formation models. The weights wautow_{\mathrm{auto}} and wcrossw_{\mathrm{cross}} (equation (7)) depend only on mm and kk, so the correction has no free parameters, and the corrected pair counts are exactly unbiased whatever model populates the haloes. We tested it with two GALFORM variants on three NN-body simulations, at number densities from 10−410^{-4} to 4×10−2 h3 Mpc−34\times10^{-2}\,h^{3}\,\mathrm{Mpc}^{-3} and redshifts z≤1.5z\leq1.5. The results reduce to the following recipe.

  1. Partition the trees and select the sample. Assign whole merger trees to kk partitions, draw mm of them uniformly at random, and weight within- and between-partition pairs by wautow_{\mathrm{auto}} and wcrossw_{\mathrm{cross}} (§ 3.3); the weights apply unchanged to any binning of the correlation function, including (s,μ)(s,\mu) and wp(rp)w_p(r_p). The galaxy formation model must process each tree independently, and whether a galaxy enters the sample should not depend on which partitions are drawn, except through a threshold set from the sparse catalogue (§ 3.5).

  1. Choose the tolerance. We require at least 68 per cent of selections to lie within 0.05 dex (12 per cent) of the full-box result in each bin. This is larger than the difference of up to 8 per cent in clustering between the two models used here (Appendix B), so distinguishing models this close needs a tighter tolerance and a larger ff.

  1. Read off the partition fraction where it has been measured. In the P-Millennium, f=7.3f=7.3 per cent meets the 0.05 dex tolerance over 0.030.03–30 h−1 Mpc30\,h^{-1}\,\mathrm{Mpc} at z=0z=0 and nˉ=10−2 h3 Mpc−3\bar{n}=10^{-2}\,h^{3}\,\mathrm{Mpc}^{-3}: fourteen times fewer trees to process. The fraction rises with redshift and as the galaxy number density falls, to 29 per cent at z≈1.5z\approx1.5 and nˉ=10−3 h3 Mpc−3\bar{n}=10^{-3}\,h^{3}\,\mathrm{Mpc}^{-3} (Table 2); smaller separations alone need less (Figure 4). For the sparsest sample, 10−4 h3 Mpc−310^{-4}\,h^{3}\,\mathrm{Mpc}^{-3}, the full range needs 96 per cent, so SaUCE saves almost nothing unless the smallest separations are dropped.

  1. Otherwise, measure it. Determine ff with repeated selections. Equations (13) and (14) (§ 5.5) show how the typical scatter changes with ff, nˉ\bar{n} and box size, and so which way to move from the nearest case in Table 2; between boxes, ff transfers through galaxy count on small scales and through the pair count per unit shell volume on large scales (§ 5.4).

  1. Use the full covariance. The scatter is strongly correlated between bins (§ 5.6), so a likelihood must use the full covariance, measured at the largest ff in use, and must add to it the cosmic variance of the simulation and the covariance of the data.

  1. Redshift space. The monopole costs up to 1.5 times as much as ξ(r)\xi(r), the quadrupole far more, because its smallest separations are dominated by a few high-dispersion haloes; an analysis should budget for the separations it fits (§ 5.7).

We next plan to use SaUCE as the forward model in simulation-based inference, propagating the covariance measured here, and to apply it to other semi-analytic models, to halo occupation models, where the same weights hold, and to abundance matching, where matching on the sparse catalogue sets the threshold tested in § 3.5. SaUCE is complementary to methods that reduce cost at other stages, such as the cosmological rescaling of merger trees used by Stiskalek et al. (2026): rescaling reduces the cost of building trees for each cosmology, while SaUCE reduces how many are populated.

Acknowledgements

OJH was supported by a UKRI PhD Studentship (grant number UKRI1738) and Durham University. CMB, BL, SB and KW acknowledge support from STFC (ST/X001075/1). SB is supported by a UKRI Future Leaders Fellowship [MR/V023381/1 and UKRI2044]. BL is funded by the ERC Advanced Grant UNCA (UKRI Frontiers Research Guarantee EP/Z533877/1).

This work used the DiRAC@Durham facility managed by the Institute for Computational Cosmology on behalf of the STFC DiRAC HPC Facility (www.dirac.ac.uk). The equipment was funded by BEIS capital funding via STFC capital grants ST/K00042X/1, ST/P002293/1, ST/R002371/1 and ST/S002502/1, Durham University and STFC operations grant ST/R000832/1. DiRAC is part of the National e-Infrastructure.

OJH acknowledges the use of Large Language Models as a writing and research aid in the preparation of this manuscript, including literature searches, drafting assistance, and language editing; all scientific content, analysis and conclusions are the work of the authors.

Software

The SaUCE package is written in Rust (Matsakis & Klock 2014), with a Python interface ([50]). This work also made use of Corrfunc (Sinha & Garrison 2019, 2020) for pair counting, NumPy (Harris et al. 2020), SciPy (Virtanen et al. 2020) and Matplotlib (Hunter 2007) for analysis and figure production, and NASA’s Astrophysics Data System for bibliographic information.

The Rust/Python package implementing SaUCE is released on PyPI as pySaUCE (https://pypi.org/project/pySaUCE/), with source hosted on GitHub at https://github.com/OscarHickman/SaUCE under the MIT licence.

Data availability

The GALFORM galaxy catalogues may be shared on reasonable request to the corresponding author.

Closed-form partition-selection variance

The scatter of the estimator from one selection to another can be computed without repeating the selection, and this confirms (Figure A1) that the empirical variance fitted in Section 5.5 is what the random choice of partitions predicts, rather than a coincidence of the samples tested. The calculation uses one property of the selection: a pair survives only if every partition it involves is selected.

Write SS for the set of mm selected partitions, DDaaDD_{aa} for the pair count within partition aa, and DDabDD_{ab} for the count between partitions aa and bb, all within a fixed separation bin. The two terms of equation (3) are then sums over SS,

DDauto=∑a∈SDDaa,DDcross=∑a<ba,b∈SDDab,(15)DD_{\mathrm{auto}}=\sum_{a\in S}DD_{aa},\qquad DD_{\mathrm{cross}}=\sum_{\substack{a<b\\a,b\in S}}DD_{ab}, \tag*{(15)}

and DDauto(k)DD_{\mathrm{auto}}^{(k)}, DDcross(k)DD_{\mathrm{cross}}^{(k)} are the same sums taken over all kk partitions. The corrected count is DD=wautoDDauto+wcrossDDcrossDD=w_{\mathrm{auto}}DD_{\mathrm{auto}}+w_{\mathrm{cross}}DD_{\mathrm{cross}}, equation (3). The pair counts are fixed by the catalogue and the weights by mm and kk; all of the randomness is in SS, so every E[⋅]\mathbb{E}[\cdot] below is an average over the choice of SS, as in Section 3.3.

Extending equation (2), the probability that jj specified partitions are all selected is

pj=m(m−1)⋯(m−j+1)k(k−1)⋯(k−j+1).(16)p_j = \frac{m(m-1)\cdots(m-j+1)}{k(k-1)\cdots(k-j+1)}. \tag*{(16)}

A term of DDautoDD_{\mathrm{auto}} names one partition and survives with probability p1p_1; a term of DDcrossDD_{\mathrm{cross}} names two and survives with probability p2p_2. Taking expectations term by term returns equations (4) and (5).

The variance needs the same rule applied to products. Squaring DDDD multiplies every term by every other, and such a product survives only if all the partitions the two terms name between them are selected, so its expectation is pjp_j with jj the number of distinct partitions in that union. Four sums over the full catalogue, with DDba≡DDabDD_{ba}\equiv DD_{ab}, are enough to write the moment down, one for each way two terms can overlap, by coinciding or by sharing an index,

T1=∑aDDaa2,T2=∑aDDaa∑b≠aDDab,T3=∑a<bDDab2,T4=∑a(∑b≠aDDab)2.(17)\begin{aligned} T_1 &= \sum_a DD_{aa}^2, & T_2 &= \sum_a DD_{aa}\sum_{b\ne a}DD_{ab},\\ T_3 &= \sum_{a<b}DD_{ab}^2, & T_4 &= \sum_a\left(\sum_{b\ne a}DD_{ab}\right)^2. \tag*{(17)} \end{aligned}

Expanding the square, two auto terms name one partition if they coincide, contributing T1T_1 at p1p_1, and two if they do not, contributing the remainder (DDauto(k))2−T1\left(DD_{\mathrm{auto}}^{(k)}\right)^2-T_1 at p2p_2. An auto and a cross term name two if the auto index lies in the cross pair, contributing T2T_2, and three if it does not, contributing DDauto(k)DDcross(k)−T2DD_{\mathrm{auto}}^{(k)}DD_{\mathrm{cross}}^{(k)}-T_2. Two cross terms name two, three or four: their pairs coincide, giving T3T_3; share one index, giving T4−2T3T_4-2T_3, the subtraction removing the coincident products T4T_4 also collects, each counted from both ends; or are disjoint, giving the remainder (DDcross(k))2−T4+T3\left(DD_{\mathrm{cross}}^{(k)}\right)^2-T_4+T_3. Hence

E[DD2]=wauto2[p1T1+p2((DDauto(k))2−T1)]+2wautowcross[p2T2+p3(DDauto(k)DDcross(k)−T2)]+wcross2[p2T3+p3(T4−2T3)+p4((DDcross(k))2−T4+T3)].(18)\begin{aligned} \mathbb{E}[DD^2] ={}& w_{\mathrm{auto}}^2\left[p_1T_1+p_2\left(\left(DD_{\mathrm{auto}}^{(k)}\right)^2-T_1\right)\right]\\ &+2w_{\mathrm{auto}}w_{\mathrm{cross}}\left[p_2T_2+p_3\left(DD_{\mathrm{auto}}^{(k)}DD_{\mathrm{cross}}^{(k)}-T_2\right)\right]\\ &+w_{\mathrm{cross}}^2\left[p_2T_3+p_3\left(T_4-2T_3\right)\right.\\ &\left.\qquad\qquad+p_4\left(\left(DD_{\mathrm{cross}}^{(k)}\right)^2-T_4+T_3\right)\right]. \tag*{(18)} \end{aligned}

and Var⁡[DD]=E[DD2]−E[DD]2\operatorname{Var}[DD]=\mathbb{E}[DD^2]-\mathbb{E}[DD]^2. Every sum on the right is a property of the full kk-partition catalogue; the selection enters only through wautow_{\mathrm{auto}}, wcrossw_{\mathrm{cross}} and pjp_j, each a function of mm and kk alone.

We validated equation (18) against brute-force evaluation on all (8m)\binom{8}{m} possible selections at k=8k=8, recovering the enumerated variance to machine precision at every mm from 2 to 7.

Brute-force enumeration is only feasible for small kk, so for the production partitioning we evaluate equation (18) directly on the measured pair counts, with no averaging over random selections.

The random normalisation

Var⁡[DD]\operatorname{Var}[DD] is not by itself the variance of the estimator. The quantity measured is ξ^=DD/RR−1\hat{\xi}=DD/RR-1, and RRRR is built from the galaxy count actually selected, Nsel=∑a∈SnaN_{\mathrm{sel}}=\sum_{a\in S}n_a with nan_a the number of galaxies in partition aa: RR∝W≡Nsel(Nsel−1)RR\propto W\equiv N_{\mathrm{sel}}(N_{\mathrm{sel}}-1), the number of ordered pairs of selected galaxies, with a constant that does not depend on the selection and cancels below. RRRR is therefore random too, and correlated with DDDD, since a selection that happens to contain denser-than-average partitions raises both.

Because WW has the same form as DDDD, equation (18) applies to it directly. Multiplying out Nsel(Nsel−1)N_{\mathrm{sel}}(N_{\mathrm{sel}}-1),

W=∑a∈S(na2−na)+∑a<ba,b∈S2nanb.(19)W=\sum_{a\in S}(n_a^2-n_a)+\sum_{\substack{a<b\\a,b\in S}}2n_an_b. \tag*{(19)}

again a sum over selected partitions plus a sum over selected pairs of them.

Replacing wautoDDaaw_{\mathrm{auto}}DD_{aa} by na2−nan_a^2-n_a and wcrossDDabw_{\mathrm{cross}}DD_{ab} by 2nanb2n_an_b in equation (18) therefore gives E[W2]\mathbb{E}[W^2]. The bilinear form of the same expansion, in which each square becomes a product of a DDDD term and a WW term and the auto–cross term is split into its two orderings, gives E[DD W]\mathbb{E}[DD\,W]. With these, the delta method gives the variance of the estimator,

Var⁡[ξ^]≃(E[DD]E[RR])2[Var⁡[DD]E[DD]2+Var⁡[W]E[W]2−2Cov⁡[DD,W]E[DD]E[W]].(20)\operatorname{Var}[\hat{\xi}]\simeq\left(\frac{\mathbb{E}[DD]}{\mathbb{E}[RR]}\right)^2\left[\frac{\operatorname{Var}[DD]}{\mathbb{E}[DD]^2}+\frac{\operatorname{Var}[W]}{\mathbb{E}[W]^2}-\frac{2\operatorname{Cov}[DD,W]}{\mathbb{E}[DD]\mathbb{E}[W]}\right]. \tag*{(20)}

The three moments are exact, and agree with enumeration to the same precision as above; only truncating the ratio expansion is approximate. That truncation error is set by the partition-to-partition scatter in galaxy count, measured by its coefficient of variation cnc_n: the standard deviation of nan_a over the kk partitions divided by their mean.

Enumeration at k=8k=8, with partitions given the cnc_n of our samples, puts it at about one per cent, never more than 1.4 per cent, always in the sense of underestimating σ\sigma.

The bracket shows whether dividing by RRRR raises or lowers the

directly. Multiplying out Nsel(Nsel−1)N_{\mathrm{sel}}(N_{\mathrm{sel}}-1), fractional variance. Write cDDc_{DD} and cWc_W for the coefficients of variation of DDDD and WW across selections, and ρDW\rho_{DW} for their correlation. The bracket is then cDD2+cW2−2ρDWcDDcWc_{DD}^{2} + c_W^{2} - 2\rho_{DW}c_{DD}c_W, which is smaller than the numerator’s own cDD2c_{DD}^{2} when cW<2ρDWcDDc_W < 2\rho_{DW}c_{DD}. Here cW≃2cn(1−f)/mc_W \simeq2c_n\sqrt{(1-f)/m}, which reproduces the enumerated cWc_W to 2–4 per cent; Var⁡[W]/E⁡[W]2\operatorname{Var}[W]/\operatorname{E}[W]^2 therefore falls as (1−f)/m(1-f)/m. Our samples satisfy cW<2ρDWcDDc_W < 2\rho_{DW}c_{DD} at every separation and for both densities.

Beyond r≃5 h−1 Mpcr \simeq5\,h^{-1}\,\mathrm{Mpc} almost every pair spans two partitions, so DDDD and WW both scale as the square of the selected galaxy count and are nearly perfectly correlated, ρDW=0.96\rho_{DW}=0.96 in the largest bin of Figure A1: a selection that catches denser partitions raises numerator and denominator together, and the fluctuation cancels from ξ^\hat{\xi}. The bracket falls from ≃0.9 cDD2\simeq0.9\,c_{DD}^{2} at r∼5 h−1 Mpcr \sim5\,h^{-1}\,\mathrm{Mpc} to 0.08 cDD20.08\,c_{DD}^{2} in the largest bin. Holding the normalisation fixed at its expectation ignores this cancellation, and Figure A1 shows the cost: it overstates the scatter by up to a factor of about three in the largest bins, whereas equation (A6) matches it to within a few per cent in the median.

Galaxy formation models

The two GALFORM variants were calibrated for different purposes: L16 (Lacey et al. 2016) to reproduce galaxy populations from the ultraviolet to the submillimetre from z=0z=0 to z=6z=6, and GP14 (Gonzalez-Perez et al. 2014) to reproduce the local bJb_J- and KK-band luminosity functions with a universal solar-neighbourhood stellar initial mass function (IMF). They differ in the parameters of Table B1, which control supernova feedback and the return of ejected gas, AGN feedback, star formation and the IMF. L16 forms stars in bursts with a top-heavy IMF, which helps it reproduce the number counts of submillimetre galaxies at high redshift; GP14 uses the Kennicutt (Kennicutt 1983) IMF throughout. These differences change the stellar masses of the galaxies in each halo, how many galaxies populate each halo and hence which galaxies enter a sample of given number density. For nˉ=0.0408 h3 Mpc−3\bar{n}=0.0408\,h^{3}\,\mathrm{Mpc}^{-3} at z=0z=0 in the P-Millennium, satellites (non-central galaxies) make up 39.6 per cent of the L16 sample and 39.9 per cent of the GP14 sample. The full-box correlation functions of the two samples differ by up to 8 per cent, L16 being lower at all but the smallest separations and by about 2 per cent on the largest scales measured for these samples, r≃9 h−1 Mpcr \simeq9\,h^{-1}\,\mathrm{Mpc}; at the same density at z≃1.5z \simeq1.5 they differ by less than 3 per cent.

Shape of the partition-selection distribution

Appendix A gives the width of the scatter between selections; this appendix gives its shape, and justifies three choices made in Section 4. The scatter is close to lognormal (Appendix C1), so the 1σ1\sigma band is stated in log⁡10R\log_{10}R. Because the distribution is skewed, a typical selection falls below the reference even though the mean does not (Appendix C2), so the bias is reported as a mean. And discarding selections that return ξ^≤0\hat{\xi}\leq0 biases the result upwards (Appendix C3), so the acceptance criterion counts them as failures.

Unless stated otherwise we use the P-Millennium L16 catalogue at z=0z=0 and the samples nˉ=10−2\bar{n}=10^{-2} and 10−3 h3 Mpc−310^{-3}\,h^{3}\,\mathrm{Mpc}^{-3}. We use the 25 bins of the coarse grid of Section 4.2 with centres below 50 h−1 Mpc50\,h^{-1}\,\mathrm{Mpc}. We call each (r,m,nˉ)(r,m,\bar{n}) combination a cell and study the ratio RR of equation (11) over its selections: 105 in 95 per cent of uncensored cells, and between 192 and 415 in the rest. A cell in which any selection returns ξ^≤0\hat{\xi}\leq0 is censored: ln⁡R\ln R is undefined for that selection, and the missing low tail would distort any test of shape. Of the 3425 cells with 2≤m≤1002\leq m\leq100 selections, 131 are censored; Appendices C1 and C2 use only the 3294 uncensored cells, and Appendix C3 treats the censored ones. We measure the width of a cell by the standard deviation of ln⁡R\ln R over its selections, which spans 0.008 to 1.07.

Normality of the log-ratio

Figure C1(a) shows one of the widest cells, σln⁡=0.49\sigma_{\ln}=0.49, with a lognormal and a normal overplotted, each given the moments of the data rather than fitted to the histogram. The lognormal better matches the floor at ξ^=0\hat{\xi}=0 and the long upper tail, though it still understates the skewness. Figure C1(b) extends the comparison to every cell, applying the Anderson–Darling test for normality to ln⁡R\ln R and to RR.

Figure C1. The distribution of R=ξ^/ξrefR=\hat{\xi}/\xi_{\rm ref} over selections. (a): one cell (nˉ=10−2 h3 Mpc−3\bar{n}=10^{-2}\,h^{3}\,\mathrm{Mpc}^{-3}, m=8m=8, r=7.6 h−1 Mpcr=7.6\,h^{-1}\,\mathrm{Mpc}, 315 selections), with a lognormal (solid) and a normal (dashed) given the sample mean and standard deviation of ln⁡R\ln R and of RR respectively; the rightmost bin collects every selection with R>3R>3. (b): fraction of the 3294 uncensored cells in which the Anderson–Darling test rejects normality at five per cent, for ln⁡R\ln R (solid) and RR (dashed), against σln⁡\sigma_{\ln}, the standard deviation of ln⁡R\ln R over a cell’s selections, with binomial errors. (c): the measured displacement of the median below the reference, 1−med1-\mathrm{med}, divided by the lognormal prediction 1−e−σln⁡2/21-e^{-\sigma_{\ln}^{2}/2} of equation (21), against σln⁡\sigma_{\ln}; the dotted line marks unity. Each point is the ratio of the binned medians of the two quantities, in eight equal-count bins of σln⁡\sigma_{\ln} per sample; bars are the 16th–84th percentiles under bootstrap resampling of the cells in the bin, and are a lower bound. Bins below σln⁡≈0.07\sigma_{\ln}\approx0.07 are not drawn (Appendix C2).

The log-ratio ln⁡R\ln R is closer to normal than RR: its Anderson–Darling statistic is smaller in 73 per cent of cells, and normality is rejected at the five per cent level for ln⁡R\ln R in 18 per cent of cells against 34 per cent for RR, with the lower rate for ln⁡Ratevery\ln R at every\sigma_{\ln}$. For the 986 cells with σln⁡<0.10\sigma_{\ln}<0.10 the rejection rate for ln⁡R\ln R is 4.9 per cent, the rate expected if every cell were exactly lognormal. In the widest sixth of cells, σln⁡≥0.22\sigma_{\ln}\geq0.22, it rises to 47 per cent: the denser sample has a heavier tail than lognormal, and the sparser one tends back towards a Gaussian as shot noise dominates. The contrast holds at z≃1.5z\simeq1.5, where normality is rejected in 14 per cent of the 3836 uncensored cells for ln⁡R\ln R and 22 per cent for RR. Neighbouring cells share partitions and, so these data describe the ensemble rather than forming independent tests.

The acceptance criterion is met at σln⁡≲0.12\sigma_{\ln}\lesssim0.12, at the edge of the range σln⁡<0.10\sigma_{\ln}<0.10 in which the test finds no departure from lognormality, so the 1σ1\sigma band can be stated in log⁡10R\log_{10}R and its width predicted from the variance.

Skew and the mean–median offset

Equations (4) and (5) make E⁡[DD]\operatorname{E}[DD] equal to (m/k)2(m/k)^2 times the full-box count at every mm, matching the normalisation of RR on average,

For RR, cc is the fractional scatter σ∣ξ∣/∣ξref∣\sigma_{|\xi|}/|\xi_{\rm ref}| of equation (20), so for a lognormal distribution the median offset is set by the variance. Figure C1(c) tests this, comparing binned medians of the measured and predicted 1−med1-\mathrm{med}, since a per-cell ratio is too noisy at small σln⁡\sigma_{\ln}. With no free parameter, the prediction follows the measurement over more than a decade to within a factor of about two; it understates the offset at nˉ=10−2 h3 Mpc−3\bar{n}=10^{-2}\,h^{3}\,\mathrm{Mpc}^{-3} and overstates it in the widest cells at 10−3 h3 Mpc−310^{-3}\,h^{3}\,\mathrm{Mpc}^{-3}. For a single cell the predicted offset, about σln⁡2/2\sigma_{\ln}^{2}/2, is smaller than the sampling error of a median over 105 selections, about 0.12 σln⁡0.12\,\sigma_{\ln}, up to σln⁡≈0.25\sigma_{\ln}\approx0.25; binning the cells reduces that error, and below σln⁡≈0.07\sigma_{\ln}\approx0.07 even the binned medians cannot measure the offset, so panel (c) starts there.

A median of RR, or a mean of log⁡10R\log_{10}R, is therefore offset by σln⁡2/(2ln⁡10)≈0.217 σln⁡2\sigma_{\ln}^{2}/(2\ln10)\approx0.217\,\sigma_{\ln}^{2} dex even for an exactly unbiased estimator: a tenth of the 0.05 dex tolerance at σln⁡=0.15\sigma_{\ln}=0.15, and all of it at σln⁡=0.5.\sigma_{\ln}=0.5. A small RR estimate will doubtless mostly measure the skew of the distribution, so we report the bias as a mean of RR.

so ξ^\hat{\xi} is unbiased to leading order (§ 3.3). We test this in every uncensored cell by dividing the offset of the mean of RR from unity by its standard error, giving tt, which for an unbiased estimator follows a unit normal. The median ∣t∣|t| is 0.71 against the 0.67 expected, the 68th percentile is 1.07 against 1.00, and 6 per cent of cells lie beyond ∣t∣=2|t|=2, against 4.6 per cent. The typical mean offset, the median over cells, is −0.0005-0.0005. The cells are not independent, so this is a consistency check rather than a formal test, but it excludes a bias at the per-cent level.

The median is biased low, as Figure C2 shows at r=7.6 h−1 Mpcr=7.6\,h^{-1}\,\mathrm{Mpc}: the mean stays at zero at every uncensored ff, while the median lies below it and approaches zero only as ff grows. Across all uncensored cells the median offset, med(R)−1\mathrm{med}(R)-1, is typically −0.010-0.010, more than an order of magnitude larger than the mean offset, consistently negative, and larger at smaller ff.

Figure C2. Mean (dotted, circles, with standard errors) and median (solid, squares) of R−1R-1 against partition fraction ff at r=7.6 h−1 Mpcr=7.6\,h^{-1}\,\mathrm{Mpc}, for both samples, at m=2,4,8,…m=2,4,8,\ldots with at least 100 selections returning ξ^>0\hat{\xi}>0. Figure C1(a) shows the distribution at one point of the green curve. The open circle marks a cell in which more than five per cent of selections return ξ^≤0\hat{\xi}\leq0, biasing both statistics (Appendix C3). The yy-axis is linear within ±0.05\pm0.05 and logarithmic outside it.

The offset follows from the lognormal shape. For a lognormal variable of unit mean whose logarithm has standard deviation σln⁡\sigma_{\ln}, the median, and the relation between σln⁡\sigma_{\ln} and the coefficient of variation cc, are

med=e−σln⁡2/2,σln⁡2=ln⁡(1+c2).(21)\mathrm{med}=e^{-\sigma_{\ln}^{2}/2},\qquad\sigma_{\ln}^{2}=\ln\left(1+c^{2}\right). \tag*{(21)}

Positivity censoring in the sparse limit

At m=2m=2, 11 and 32 per cent of selections and bins over 0.01≤r<50 h−1 Mpc0.01\leq r<50\,h^{-1}\,\mathrm{Mpc} return ξ^≤0\hat{\xi}\leq0 for the denser and sparser samples respectively. At small separations, where ξ^≫1\hat{\xi}\gg1, this happens only when the selected partitions contain no pair in the bin. Discarding these selections keeps only those that contained a pair, removing the low tail and biasing both the mean and the median upwards.

The bias is confined to a few cells but is large in them. In the 17 cells with more than five per cent of selections returning ξ^≤0\hat{\xi}\leq0 and at least 100 returning ξ^>0\hat{\xi}>0, the median over cells of the mean offset of the remaining cells, ⟨R⟩−1\langle R\rangle-1, is +0.351, against −0.0005-0.0005 for the uncensored cells, and it reaches +3.5; almost all are positive. The bias tracks the censored fraction: for nˉ=10−3 h3 Mpc−3\bar{n}=10^{-3}\,h^{3}\,\mathrm{Mpc}^{-3} at r=0.023 h−1 Mpcr=0.023\,h^{-1}\,\mathrm{Mpc}, the fraction is 34.3, 5.7 and 0.0 per cent at m=2,4m=2,4 and 8, and the median offset, med(R)−1\mathrm{med}(R)-1, falls from +0.32 to a value consistent with zero.

This paper has been typeset from a \TeX/\LaTeX file prepared by the author.

References

  1. [1]Landy, S. D. & Szalay, A. S. 1993, ApJ, 412, 64DOI
  2. [2]Kaiser, N. 1986, MNRAS, 227, 1DOI
  3. [3]Cooray, A. & Sheth, R. 2002, Phys. Rep., 372, 1DOI
  4. [4]Berlind, A. A., & Weinberg, D. H. (2002). The HOD and clustering of galaxies. The Astrophysical Journal, 575(2), 587–616.
  5. [5]Zheng, Z., & al., . et . (2005). Theoretical models of the halo occupation distribution: separating central and satellite galaxies. The Astrophysical Journal, 633(2), 791–809.
  6. [7]Laureijs, R., & al., . et . (2011). Euclid Definition Study Report. ESA/SRE.
  7. [25]Henriques B. M. B., White S. D. M., Thomas P. A., Angulo R. E., Guo Q., Lemson G., Springel V., 2013, MNRAS, 431, 3373
  8. [39]Peebles, P. J. E. (1980). The Large-Scale Structure of the Universe. In Princeton University Press.
  9. [40]Planck Collaboration 2014, A&A, 571, A16
  10. [49]van Daalen M. P., Henriques B. M. B., Angulo R. E., White S. D. M., 2016, MNRAS, 458, 934
  11. [50]van Rossum, G., & Drake, F. L. (2009). Python 3 Reference Manual. In CreateSpace.

Paper details

vlnc:W_nsvpphdv
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 1

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Physics and Astronomy

Large Language Models

View statement
Looking for related papers…
Contents