In this paper
AbstractIntroductionSimulated galaxy cataloguesN-body simulationsMerger tree partitioningGalaxy sample definitionsRedshift-space cataloguesSaUCE estimatorReference estimatorWhy a partition subset biases the estimateDerivation of the weightsRedshift spaceAssumptions and validityValidationReferenceSeparation binsAccuracy metricsAcceptance criterionRedshift-space multipolesConvergence and accuracy of SaUCEConvergence with partition fractionGeneralisation across samples and redshiftsGalaxy formation modelsBox sizeA recipe for the estimator's scatterBin-to-bin covarianceRedshift-space distortionsConclusionsAcknowledgementsSoftwareData availabilityClosed-form partition-selection varianceThe random normalisationGalaxy formation modelsShape of the partition-selection distributionNormality of the log-ratioSkew and the mean–median offsetPositivity censoring in the sparse limitReferencesWorksPhysics and Astronomy
Less is More: Unbiased Galaxy Clustering from Fewer Merger Trees
Abstract
Galaxy clustering could strongly constrain galaxy formation models but is rarely used, because the two-point correlation function (2PCF) requires a complete galaxy catalogue at every point in parameter space. Semi-analytic models (SAMs) process each halo merger tree independently, so running a SAM on of disjoint partitions of the trees, each spanning the full box, cuts the cost by . Because a halo's galaxies are kept or discarded together, one-halo pairs are diluted less than two-halo pairs, and the naive estimate overstates small-scale clustering by a factor . We introduce the Sparse and Unbiased Clustering Estimator (SaUCE), which removes this bias with weights set by and alone. It has no free parameters, gives exactly unbiased pair counts, and applies to any catalogue built from a random subset of whole haloes and to any pair-count statistic. Running a SAM on 7 per cent of the P-Millennium merger trees, 68 per cent of random selections recover the full-box at to within 0.05 dex in each bin from 0.03 to , at a density of : a fourteen-fold computational saving. The required fraction rises with redshift and as the number density falls, is the same for two model variants, and transfers between boxes through the galaxy and pair counts. The scatter, derived in closed form, is strongly correlated between bins, so a likelihood must use the full covariance. A recipe for choosing the fraction makes the 2PCF affordable for model calibration and simulation-based inference.
Introduction
Galaxy clustering depends on how galaxies populate dark matter haloes. The physical processes of galaxy formation – supernova feedback, feedback from active galactic nuclei (AGN), gas cooling, star formation and galaxy mergers – shape this occupation, each changing the number and distribution of galaxies across haloes of different mass (Baugh 2006; Benson 2010; Wechsler & Tinker 2018). Clustering can constrain galaxy formation physics, and complements the one-point statistics, such as the luminosity function, on which model calibration has traditionally relied.
The advent of precision clustering measurements from surveys such as the Dark Energy Spectroscopic Instrument (DESI Collaboration 2016) makes it even more timely to incorporate the 2PCF into model calibration.
Despite this, galaxy clustering is seldom used to calibrate physically motivated galaxy formation models. The principal obstacle is not evaluating the two-point correlation function (2PCF; [39]) but the cost of producing the galaxy catalogue from which the clustering is measured.
Hydrodynamical simulations of cosmological volumes (e.g. Schaye et al. 2023) are too expensive to run at every step of a parameter search, and are instead calibrated on smaller boxes (Kugel et al. 2023), which can only probe the clustering over a narrow range of scales.
Semi-analytic models (SAMs) are cheaper by orders of magnitude, but still costly: a full galaxy catalogue for the P-Millennium simulation (Baugh et al. 2019) requires processing the merger trees of more than 77 million haloes with masses above at , and using the 2PCF in the calibration requires one such catalogue at every point in parameter space – thousands of them for a Markov chain Monte Carlo (MCMC) search, and comparable numbers for simulation-based inference (Crammer et al. 2020).
Henriques et al. [25] made this affordable for one-point statistics by running the model on a small, representative sample of haloes, and van Daalen et al. [49] extended the approach to clustering. Calibrating the model of Guo et al. (2013), they sampled haloes at a mass-dependent rate, using fewer than 0.04 per cent of all haloes in the Millennium I simulation, measured the halo occupation and satellite profiles of the sampled galaxies, and used these in a halo model ([3]) to predict the projected correlation function. Adding clustering to the MCMC narrowed degeneracies left open by the stellar mass function, tightening the supernova feedback parameters and constraining the timescale on which orphan satellites merge. The halo model, however, is itself an approximation: even applied to the full galaxy catalogue, their prediction departed from the directly measured clustering by up to ~15 per cent outside their most massive bin, and the model’s two-halo term took its halo clustering from linear theory and a fitted bias rather than from the sampled haloes.
Kaiser [2] showed that a sparsely sampled catalogue gives an unbiased estimate of the 2PCF, because random subsampling raises the shot noise without biasing the clustering signal. We apply this idea to merger trees. Because most SAMs process each tree independently, we can divide the trees of a cosmological volume into partitions, run the model on only some of them, and correct the correlation function for the fraction run, the partition fraction. Unlike Kaiser [2], we sample haloes rather than galaxies: a halo’s galaxies are kept or discarded together, so one-halo pairs, which survive whenever their halo is kept, are diluted less than two-halo pairs, which survive only if both haloes are kept. Without correction the small-scale clustering is overestimated, increasingly so as the partition fraction falls. We remove this bias exactly, with weights set only by the numbers of partitions run and available, so that, unlike the approach of van Daalen et al. [49], nothing is tuned to a particular galaxy catalogue.
We introduce the Sparse and Unbiased Clustering Estimator (SaUCE) and test it with the GALFORM SAM (Cole et al. 2000; Bower et al. 2006; Lacey et al. 2016). § 2 describes the simulations, models and galaxy samples. § 3 derives the estimator, a correction with no free parameters, exact for the pair counts, that holds for any catalogue built from a random subset of whole haloes, and § 4 defines how we validate it. § 5 measures the partition fraction needed as a function of scale, redshift and number density; shows that it is the same for two GALFORM variants with different feedback but, at fixed number density, similar halo occupation distributions (HODs); shows how it transfers between simulations of different size; characterises the scatter and the bin-to-bin covariance a likelihood needs; and tests the correction in redshift space. § 6 collects these into a practical recipe.
Simulated galaxy catalogues
We test SaUCE with two variants of GALFORM (Appendix B) run on three -body simulations. GALFORM follows the baryonic components of galaxies within dark matter halo merger trees, each of which records the mergers and accretion by which a halo grew; this history sets the galaxy content of the halo and hence its contribution to the clustering. The two variants populate the same trees differently, which tests whether the scatter of SaUCE depends on galaxy formation physics.
The trees are extracted from the -body simulations. SUBFIND subhaloes (Springel et al. 2001) are grouped into DHalo haloes (Jiang et al. 2014; see also Merson et al. 2013) whose histories are strictly hierarchical, so no halo is shared between trees, a property SaUCE requires (§ 3.5). Tree-building choices leave artefacts that change the galaxy output (Gómez et al. 2022; Chandro-Gómez et al. 2025), but the reference and sparse estimates use the same trees, so the comparison tests SaUCE on whatever population those trees produce.
N-body simulations
Table 1 lists the properties of the three simulations and the number of partitions, , into which their merger trees are divided (§ 2.2). The P-Millennium has by far the largest number of partitions, so its partition fraction can be set most finely, in steps of .
| Simulation | () | () | Cosmology | ||
| P-Millennium | 542.2 | 1024 | 5040 | Planck 2013 | |
| Millennium I | 500 | 64 | 2160 | WMAP1 | |
| Millennium II | 100 | 64 | 2160 | WMAP1 |
Table 1. Summary of -body simulations used. is the box size, the number of merger-tree partitions, the particle number and the particle mass; the cosmology column lists the underlying parameters: Planck 2013 (Planck Collaboration [40]) for the P-Millennium and approximately first-year WMAP (Spergel et al. 2003) for the Millennium simulations. References: P-Millennium, Baugh et al. (2019); Millennium I, Springel et al. (2005); Millennium II, Boylan-Kolchin et al. (2009).
Millennium I and Millennium II share but differ in volume by a factor of 125, so comparing them varies the box size at a fixed partition count (§ 5.4).
We run two variants of GALFORM, chosen because they populate the same merger trees differently: L16 (Lacey et al. 2016) and GP14 (Gonzalez-Perez et al. 2014), which were calibrated for different purposes and differ in the parameters listed in Table B1. Both models were originally calibrated on an -body simulation of resolution similar to Millennium I but with a different cosmology, and were recalibrated on the P-Millennium by Baugh et al. (2019). We run them on the Millennium I and Millennium III trees without further recalibration, so on those boxes they do not reproduce the calibration data in detail. This does not affect the tests here: SaUCE requires a galaxy population, not a realistic one, and the reference and sparse estimates are always drawn from the same population.
| Parameter | Description | L16 | GP14 |
| Supernova feedback and gas return | |||
| Mass-loading velocity scale (km s) | 320 | 380 | |
| Mass-loading power-law slope | 3.4 | 3.2 | |
| Return rate of ejected gas | 1.00 | 1.26 | |
| AGN feedback | |||
| Fraction of Eddington luminosity for heating | 0.01 | 0.04 | |
| Hot-halo threshold for AGN feedback | 0.80 | 0.72 | |
| Star formation | |||
| H depletion rate (Gyr) | 0.74 | 0.50 | |
| Disc stability threshold | 0.90 | 0.80 | |
| IMF in bursts | Top-heavy | Kennicutt |
Table B1. Parameters that differ between the two GALFORM variants; all others are identical. Values are those recalibrated for the P-Millennium by Baugh et al. (2019) where recalibration changed them, and as published otherwise. The recalibration changed (from 0.60) and (from 425 km s) in GP14, and (from 3.2) and (from 0.64) in L16, to the values listed. applies to both quiescent and burst star formation. Both variants use a Kennicutt (1983) IMF for quiescent star formation; in bursts L16 uses a top-heavy IMF with slope .
Merger tree partitioning
The merger trees of each simulation are divided between files, the partitions. Each tree is rooted on a halo at , contains that halo’s full progenitor history and lies in exactly one partition, so the partitions hold disjoint sets of haloes. We run GALFORM on each partition separately.
In cosmological simulations, how the haloes are distributed between the partitions is a free choice, illustrated in Figure 1. The simulation volume is typically divided into spatial sub-volumes, so that the merger histories in one file all come from a single contiguous region many times smaller than the box. For the simulations used here the tree-building code instead assigns each halo, with its progenitors, to one of the files independently and with equal probability, so the haloes in a partition can come from anywhere in the volume and each partition spans the full box. Each partition therefore samples every scale the box resolves. This does not affect the bias of SaUCE (§ 3.5), but it keeps the scatter small, and the partition fractions of § 5 depend on it. Because the assignment is random, the partitions are equal in size only on average; this spread adds to the variance of the estimate (Appendix § 10.1).

Figure 1. Partition architecture. (a) Spatial sub-volume division, with haloes grouped by location; each coloured region is one contiguous sub-volume. (b) The merger-tree sparse-sampling or partitioning used here, with partitions (blue, red, green) assigned at random to merger trees rather than by spatial position; grey points are unselected haloes. Partition members are spread throughout the full simulation volume, preserving the spatial distribution.
The random assignment applies to the haloes. At each halo is still selected with probability , since it belongs to exactly one tree, but progenitors that share a root are always selected together. This adds to the variance at but not to the bias (§ 3.5).
Galaxy sample definitions
We define galaxy samples by number density rather than by stellar mass. A fixed stellar-mass threshold selects a different number density in every box, model and redshift, because each combination has its own stellar mass function; a number-density cut removes that confound, so that a comparison between boxes or between model variants isolates the quantity of interest. We rank galaxies by stellar mass and set the threshold that gives the target in each full catalogue.
Three number-density samples underpin the convergence results. The galaxy counts below follow from and the P-Millennium volume, so they are the same for either model and at every redshift; the stellar-mass thresholds are those of the L16 model at , and vary with model and redshift:
(i) : galaxies ().
(ii) : galaxies ().
(iii) : galaxies (). Shot-noise limited at small scales.
Across all simulations, model variants and three redshifts, these densities correspond to stellar-mass thresholds spanning –. The comparisons between models and boxes (Sections § 5.3 and § 5.4) use a fourth, denser sample, , the density of in the P-Millennium L16 catalogue at ( galaxies). We use a denser sample there because Millennium II has 1/125 of the volume of Millennium I: at it would hold only galaxies.
We evaluate the convergence of the SaUCE estimator at three redshifts, , and , spanning the primary redshift coverage of surveys such as DESI (DESI Collaboration 2016) and Euclid ([7]).
Redshift-space catalogues
We also consider the clustering in redshift space. GALFORM assigns each galaxy the peculiar velocity of its host halo or subhalo. We place a galaxy in redshift space by displacing it along one box axis, the line of sight, by , where is its line-of-sight peculiar velocity, the scale factor and the Hubble parameter at the snapshot redshift; displacements wrap at the periodic boundary.
SaUCE estimator
We now derive the main result of the paper: an estimator that recovers the full-box 2PCF from the galaxy catalogues of partitions.
Reference estimator
We measure from its definition as the excess probability, relative to a random distribution of points of the same mean density, of finding a galaxy pair at a given separation. For a separation bin centred on ,
where is the number of galaxy pairs with separations in the bin, is the number of galaxies in the catalogue being counted, is the volume of the simulation box and is the volume of the radial shell. is the number of pairs expected from points placed independently and uniformly in the box, and for a periodic box it is exact: with periodic boundaries every shell is complete for , so there are no boundary or mask effects for a random catalogue to measure, as there are in observational surveys ([1]).
Why a partition subset biases the estimate
Random subsampling of galaxies scales and by the same factor, and so adds shot noise but no bias ([2]).
Selecting of the partitions is different, because it samples haloes rather than galaxies: every galaxy in a halo is retained or discarded with its host. Whether a pair survives is then decided by how many partitions it occupies, so what we need are the probabilities that one, and that two, specified partitions are among those drawn. The are drawn uniformly from the without replacement, and we call one such draw a selection. Any one partition is drawn with probability , and once it has been, a second specified partition must be one of the remaining taken from the remaining :
A pair confined to one partition survives with probability , and a pair spread across two survives only if both are drawn, with probability .
Galaxy pairs divide into one-halo pairs, whose two galaxies share a halo, and two-halo pairs, whose galaxies occupy different haloes (e.g. Cooray & Sheth [3]). How a sample divides between the two is set by HOD (Berlind & Weinberg [4]; Zheng et al. [5]), which differs between galaxy formation models and between samples. No halo is split across partitions (§ 2.2), so both galaxies of a one-halo pair lie in one partition and the pair survives with probability . Two-halo pairs generally have their haloes in different partitions, and survive with probability : at two distinct haloes share a partition with probability only , although at progenitors of a common root always do (§ 3.5).
The naive estimate normalises the combined catalogue by its own , which scales on average as the square of the galaxy count and hence as . Two-halo pairs in different partitions are retained at , so they are understated by a factor of about ; the fraction that share a partition are retained at and overstated by , so at two-halo pairs as a whole are retained, on average, at exactly . One-halo pairs, all retained at , are overstated by a factor . For and this is a factor of 128.
Derivation of the weights
What sets a pair’s survival probability is not whether it is a one-halo or a two-halo pair, but whether its galaxies lie in one partition or two. We therefore weight pairs by partition, with the corrected pair count:
where is the number of pairs whose galaxies lie in the same selected partition and the number whose galaxies lie in two different selected partitions. Because the classification is by partition rather than by halo, it can be read directly off the catalogue, and no approximation enters. This weighted count, not the raw sum of the naive estimate, is the that enters equation (1).
Write and for the same two counts in the full -partition catalogue; together they give the full-box pair count. Every pair in survives with probability and every pair in with probability . Writing for the average over the random choice of partitions, at fixed full-box catalogue, summing over pairs gives, exactly,
On average, the selected catalogue’s is smaller than the full-box by , so the corrected count must on average be smaller than the full-box count by the same factor:
One pair of weights must serve every separation bin, and and stand in a different ratio in each bin, so the two classes must be matched separately. Each class is divided by its survival probability and multiplied by :
Up to the common factor , which matches the normalisation of the selected catalogue, these are Horvitz–Thompson weights (Horvitz & Thompson 1952): each pair is weighted by the inverse of its probability of being selected, so the weighted count is an unbiased estimate of the full-box count. The partitions play the role of clusters in single-stage cluster sampling (Cochran 1977), and a pair’s selection probability is the probability that the one or two partitions it occupies are all drawn. The same idea underlies the pairwise inverse-probability weights that correct for galaxies left unobserved by fibre assignment in redshift surveys (Bianchi & Percival 2017); here the probabilities are known exactly, rather than estimated from many realisations of the assignment.
Equations (4) and (5) involve no approximation, so the corrected pair count is exactly unbiased (Appendix A). itself is a ratio: is built from the selected galaxy count, which varies from one selection to another, so is unbiased only to leading order; Appendix C2 finds its mean offset from the reference consistent with zero.
Setting gives and recovers the full catalogue. The smallest usable configuration is , the fewest partitions for which cross pairs exist.
Substituting (7) into equations (3) and (1) gives the SaUCE estimator,
with built from the galaxy count of the combined -partition catalogue.
Redshift space
Converting redshift to radial distance under the assumption of pure Hubble flow (§ 2.4) displaces galaxies along the line of sight by their peculiar velocities, making the clustering anisotropic (Kaiser 1987). We bin pairs in the redshift-space separation , measured from the displaced positions, and in , the cosine of the angle between the pair separation and the line of sight. The periodic-box estimator generalises directly,
where is given by equation (1) with the volume of the bin in place of . The anisotropy is summarised by the multipole moments (e.g. Sánchez et al. 2017)
where is the Legendre polynomial of order ; the monopole () and quadrupole () carry most of the signal.
The correction is unchanged. The derivation of § 3.3 is made bin by bin and uses only each galaxy’s partition label, which a line-of-sight displacement does not change, so the same weights apply to , and indeed to any binning of the pairs, including the projected correlation function . Each multipole is a linear functional of , so the multipoles inherit the same leading-order unbiasedness.
Assumptions and validity
The weights require only that whole partitions are selected, of the drawn uniformly without replacement, and that whether a galaxy enters the sample does not depend on which partitions are drawn. In our tests the stellar-mass threshold is fixed from the full catalogue (§ 4.1); in a calibration run it must instead be set from the sparse catalogue itself, and then varies slightly between selections. Galaxies near the threshold then enter some selections and not others, which in our tests shifts the mean by at most 1 per cent and barely changes the scatter.
Two further conditions ensure that the sparse catalogue contains the same galaxies as the full one. The galaxy formation model must process each merger tree independently, so that running it on a subset of trees reproduces those trees’ galaxies; this holds for standard semi-analytic models but fails for processes that couple trees, such as a non-local treatment of reionisation. And no tree may be split between partitions, since the model needs a halo’s full history to populate it; the weights themselves use only each galaxy’s partition label.
The weights place no condition on how trees are assigned to partitions, since equations (4) and (5) hold however the full-box pairs divide between and . Two consequences follow. Archives that store trees in files by spatial region can use those files directly as partitions: the estimate remains unbiased, but its variance is larger, because compact partitions sample the box unevenly (Figure 1), so the partition fractions of § 5 should not be assumed to carry over. And because our trees are partitioned at , progenitors that share a root always lie in the same partition, so at partition membership is correlated with pair separation. This changes how pairs divide between the two classes, and hence the variance, but not the bias.
The correction acts only on . We normalise analytically because the box is periodic, which leaves the choice of partitions as the only source of scatter (Appendix A), but the correction works equally with counted from a random catalogue, as in the estimator of Landy & Szalay [1] for a survey geometry.
Validation
Every result in Section 5 compares a SaUCE estimate built from partitions with a reference built from all ; we call the partition fraction. Each selection (Section 3.2) corresponds to the first partitions of a random permutation of the , so selections at fixed are independent, while a larger contains every partition used at a smaller one, and results at neighbouring are correlated. Unless stated otherwise, each configuration is measured over 105 permutations; the convergence thresholds use 10,500 (Section 4.4).
Reference
The reference is the correlation function measured from all partitions, from equation (1). All pair counts, for the reference and for every selection, are tabulated with Corrfunc (Sinha & Garrison (2020)). Each result uses a subset of the test cases of Sections 2.1 and 2.3 and states which; within a case, the same stellar-mass threshold is applied to the reference and to every selection.
SaUCE estimates the full-box correlation function of a given -body realisation. The reference is the estimate made using all of the partitions, so cosmic variance of the realisation does not enter the comparison; an application must add it separately (Section 5.6).
Separation bins
We count pairs in logarithmic bins of width 0.1 dex. The two ends of the range are not free, being set by the resolution of the simulation and by the size of the box; the width is a resolution choice.
The bin range is set by the simulation. The P-Millennium resolves haloes down to , the mass of 20 particles (Baugh et al. (2019)), whose comoving (at 200 times the mean density) is 0.031 Mpc and is independent of redshift in comoving units. Pairs closer than this increasingly involve satellites whose subhaloes the simulation does not resolve, so the separation is set by GALFORM’s orbital assignment rather than by the -body calculation. We therefore report Mpc, the nearest bin centre on the grid to that scale. This floor comes from resolution: at every bin down to Mpc holds more than 2000 reference pairs, ten times closer in than we report.
The outer edge is set by the box. Clustering predictions are conventionally restricted to a tenth of the box side (e.g. Orsi et al. (2008)), which is 54 Mpc for the P-Millennium. We report Mpc, comfortably inside this limit and the largest separation at which is positive in every sample and redshift tested.
The scatter fits of Section 5.5, the covariance of Section 5.6 and the tests of Appendix C use a coarser grid: 30 bins of width 0.148 dex with limits of 0.01 and 271 Mpc; in Section 5.5, Millennium II and II use 30 bins of 0.12 dex, from 0.01 to 50 and from 0.002 to 10 Mpc respectively. The fits and Appendix C keep the 25 bins with centres below 50 Mpc, and the covariance keeps the first 27, spanning 0.01–98 Mpc. These ranges are wider than the one above because they characterise the scatter of the estimator, not its accuracy.
Accuracy metrics
We measure accuracy on the ratio of a selection’s estimate to the reference,
over the ensemble of independent selections at fixed . We report two quantities:
(i) the bias, , the mean over selections. The mean is the statistic in which SaUCE is unbiased; a median of , or a mean of , is offset from zero even for an exactly unbiased estimator (Appendix C);
(ii) the 1 band, the 16th–84th percentile range of , in dex. This is the scatter a user running a single selection faces. Its half-extent is the larger of the distances of the 16th and 84th percentiles from zero, so an offset or asymmetric band is judged by its worse side. The distribution of is close to lognormal (Appendix C), so the width of the band follows from the variance, which Appendix A predicts in closed form and Section 5.5 fits.
Results are sometimes broken down by clustering regime rather than bin by bin, as diagnostic columns in Table 2. Those breakdowns use three ranges: one-halo, ; transition, ; and two-halo, Mpc, holding six, seven and eight bins respectively on this grid.
Acceptance criterion
The convergence threshold is the smallest partition fraction at which the band of lies inside dex in every bin of the range considered, and continues to do so at every larger fraction measured. At , therefore, at least 68 per cent of selections lie within 0.05 dex in each bin. The bins are correlated but not perfectly (Section 5.6), so the fraction of selections within tolerance in all bins at once is lower than in the worst bin and can fall below 68 per cent. A selection returning in a bin lies outside the band and counts as a failure.
The tolerance, 0.05 dex, is a factor of , or 12 per cent. We adopt it to define convergence; calibration against data may need a tighter tolerance, as discussed in item (ii) of Section 6.
A selection’s corrected pair count is a sum of the auto-pair counts of its partitions and the cross-pair counts of its partition pairs, so we count pairs once for every pair of the partitions and obtain any selection by summation. This gives from 10,500 selections at every integer from 2 to ; the first 105 are those used elsewhere in the paper. Depth matters: requiring acceptance at every larger fraction lets one noisy failure at large push up, so 105 selections give typically 15, and up to 40, per cent higher. At 10,500 the bootstrap 68 per cent interval on is within per cent, and halves and quarters of the ensemble agree to within 3.4 and 7 per cent. We quote to two significant figures.
The criterion cannot be met where selections often contain no pair in a bin, since the weights cannot reweight an empty count. This happens at the smallest separations and smallest in the sparser samples (Appendix C3), and is why the sparsest sample, , meets the criterion over the full range only above per cent, where SaUCE saves almost nothing.
Redshift-space multipoles
In redshift space the reference is the monopole and quadrupole of the merged -partition catalogue, from equation (10). We bin pairs in on the grid of § 4.2 and in in 24 linear bins over ; the periodic box with a fixed line of sight gives , so negative adds nothing.
The quadrupole changes sign near the one-halo to two-halo transition, so its log-ratio is undefined there. Both multipoles are therefore assessed on the residual normalised by the monopole,
which is defined where crosses zero. We apply the criterion of § 4.4 to the 16th–84th percentile band of , in dex. For the monopole, , so this is the criterion used for , with in place of .
Convergence and accuracy of SaUCE
Section 5.1 shows how the estimated 2PCF depends on the partition fraction. Section 5.2 extends these results to different galaxy samples and redshifts. Section 5.3 shows that the scatter is the same for two GALFORM variants, and Section 5.4 whether a partition fraction transfers between boxes. Section 5.5 gives a practical recipe for predicting the estimator’s variance, and Section 5.6 characterises its bin-to-bin covariance. Section 5.7 tests the correction in redshift space, and asks what each multipole costs. Partition fractions quoted here are specific to the P-Millennium unless stated otherwise.
Convergence with partition fraction
Figure 2 shows the effect of the correction for at . The naive estimate overstates on small scales by the factor predicted in § 3.2, while on large scales, where almost all pairs are two-halo pairs, the overstatement vanishes.
Figure 2. The naive and SaUCE estimates of against the reference, for in the L16 model on the P-Millennium at . (a) from the naive and SaUCE estimators at per cent, against the reference (all partitions; solid black), from which the SaUCE estimate is indistinguishable on this axis. (b) Fractional residual of the naive estimator, . On small scales, where almost all pairs are one-halo pairs, it reaches (§ 3.2); it falls to zero on large scales, where almost all pairs are two-halo pairs. (c) Fractional residual of the SaUCE estimator. Lines show the mean over the selections available at each , between 192 and 415 of them; shaded bands span the 16th–84th percentiles, the scatter of a single selection. The panels extend below the reported range, , to show the naive bias reaching its plateau. Note the different -axis ranges of panels (b) and (c).
The estimator is unbiased, and what improves with is the scatter. The mean ratio of equation (11) stays within a few per cent of unity at every and separation, with no trend in , consistent with the noise on the mean (Appendix C2). The scatter falls steeply and smoothly with (Figure 3), and for is smaller at small separations than at large ones.
Figure 3. The scatter of SaUCE against the partition fraction , one radial bin per panel (P-Millennium, L16). The panels are the bins centred on , 0.3, 3 and 30 Mpc, and the nine curves in each are three number-density samples at three redshifts (legend); the Mpc panel omits the sparsest sample, whose is too often there. The ordinate is the half-extent of the band of , so a curve crosses the dashed 0.05 dex line where that bin meets the criterion of § 4.4; the dotted line marks 0.01 dex. A selection returning counts as a failure (§ 4.4). is excluded, since is undefined for a single partition.
Which scales set the threshold depends on the sample (Table 2). Usually it is the two-halo range. For it is the smallest scales, , where that sample has few pairs per bin, though at every redshift only narrowly ahead of the two-halo range: 15, 21 and 29 per cent against 14, 20 and 28. The one-halo range is never limiting.
| [per cent] | one-halo | transition | two-halo | |||
| 0 | 75 | 7.3 | 3.9 | 6.1 | 7.3 | |
| 0 | 157 | 15 | 9.6 | 13 | 14 | |
| 0.5 | 118 | 12 | 2.8 | 4.3 | 12 | |
| 0.5 | 212 | 21 | 8.1 | 9.3 | 20 | |
| 1.5 | 209 | 20 | 1.4 | 3.1 | 20 | |
| 1.5 | 299 | 29 | 7.3 | 7.7 | 28 |
Table 2. The convergence threshold for the P-Millennium, from 10,500 selections at every : the smallest partition fraction at which the band of lies inside dex in every radial bin of the stated range, and at every larger fraction measured (§ 4.4). The full range is . The one-halo (0.3–1), transition (1–5) and two-halo (5–30 ) columns show which scales set it; the full range also contains the smallest scales, , which is why it exceeds all three columns. is the partition count, with , and is in . The sparsest sample, , meets the criterion over the full range only above per cent.
Generalisation across samples and redshifts
Figure 3 shows how the scatter depends on the sample and redshift: within a panel, curves of one line style differ only in sample and curves of one colour only in redshift, and comparing panels isolates the scale.
The required fraction rises as the number density falls, because a sparser sample has fewer pairs in each bin. Over the full range and on two-halo scales it also rises with redshift at fixed . The number of galaxies is then the same, but the two-halo clustering amplitude ends_mid=1
is lower (near 10 Mpc, is about a third lower at than at ), so the same scatter in the pair count is a larger fraction of . On one-halo and transition scales the threshold instead falls with redshift, for both samples in Table 2: for it falls from 3.9 to 1.4 per cent on one-halo scales and from 6.1 to 3.1 per cent on transition scales between and . We return to the cause in § 5.4.
Table 2 gives the fraction needed over the full range. A user who needs only smaller separations can run much less: Figure 4 gives, against , the fraction needed for every bin from 0.03 Mpc out to to meet the criterion.
Figure 4. The partition fraction needed as a function of the largest separation used, for on the P-Millennium. The ordinate is the largest per-bin threshold over all bins from 0.03 Mpc out to , not the threshold of the bin at . is not shown, since its smallest scales set the full-range requirement and its curves would be flat (Table 2).
Galaxy formation models
The bias of SaUCE does not depend on the galaxy formation model (§ 3.3), but its scatter could. We compare the L16 and GP14 models, which have different feedback prescriptions (Appendix B), in all three simulations, with galaxies selected to (Figure 5). The GP14 circles lie on the L16 lines throughout: the bands of the two models differ by no more than the uncertainty of a band measured from 105 selections (bin by bin, the L16/GP14 band ratio has median 0.94–1.00 in each box and redshift), and their thresholds agree to within one step of the power-of-two grid in , a factor of two in . The scatter is therefore the same for these two GALFORM variants, although their full-box correlation functions differ by up to 8 per cent in the P-Millennium at , and by less than 3 per cent at $z \approx 1.5 ({xref}Appendix B`). Matched in number density, however, their HODs are similar: for this sample in Millennium I, at both redshifts, the halo masses at which and agree to within 0.02 dex, and above the mass is two within about 10 per cent.
Figure 5. The scatter of SaUCE for two galaxy formation models in three boxes, for . Every panel has the ordinate of Figure 3, the half-extent of the band at a single radial bin, with the 0.05 dex tolerance dashed and 0.01 dex dotted. The abscissa is the partition fraction in the left pair of panels and the galaxy count in the selected partitions, , in the right pair. Within each pair the panels are and . Colour is the simulation (P-Millennium , Millennium I , Millennium II ), line style the redshift (solid , dashed ), and the model is L16 for the lines and GP14 for the circles on them. Against the boxes fall on one relation at but not at (§ 5.4).
The test therefore shows that the scatter is insensitive to differences in galaxy formation physics at fixed occupation; models whose HODs differ more, for example in satellite fraction, could have a different scatter.
Box size
We now ask whether a partition fraction measured in one box transfers to another. Millennium I and Millennium II both have partitions, and we test powers of two in over 0.03–7.5 Mpc, the range all three boxes share, on the 0.1 dex grid of § 4.2. Millennium I meets the criterion at a similar fraction to the P-Millennium, within one step in , a factor of two in . Millennium II, whose volume is 125 times smaller, never meets it, even at per cent.
The cause is the small number of galaxies per partition. Each partition holds a fraction of the galaxies while spanning the whole volume (§ 2.2), and all boxes are selected to the same , so each Millennium II partition holds 125 times fewer galaxies than a Millennium I partition. Millennium II’s scatter keeps falling as increases, as expected for sampling noise; the corrected pair counts are unbiased in any box (§ 3.3).
The right-hand panels of Figure 5 plot the scatter against the mean number of galaxies in the selected partitions, , instead of . The three boxes then fall on one relation at Mpc: there the scatter is set by whether a halo enters the selection with its satellites, which depends only on how many galaxies are drawn. At they do not, because the pairs come from distinct haloes and their number depends on the density of the selection as well as its size. The number of pairs in a shell of volume is , so measures the pair count per unit shell volume; plotted against it (not shown), the boxes fall on one relation at , their median offsets from a single power law spanning a factor of 1.3, compared with 2.3 for . A larger box therefore needs a smaller partition fraction for the same scatter: transfers between boxes through the galaxy count on small scales and through the pair count per unit shell volume on large scales.
In every panel of Figure 5 the band lies below the band, and neither variable removes this: at these separations one-halo pairs, which enter a selection in bursts, are a smaller share of the total at higher redshift, as in the one-halo and transition columns of Table 2.
A recipe for the estimator's scatter
The closed-form variance of Appendix A confirms that the measured scatter is what the random choice of partitions predicts, but a user cannot evaluate it: it needs the full-box pair counts that SaUCE exists to avoid. We therefore fit the scatter empirically. Pair separation alone does not predict it: at fixed , the fractional scatter differs between number densities (Figure A1a, at ) and between redshifts (Figure 3, whose band width follows from this variance; § 4.3), because says nothing about how many pairs contribute to a bin. Here is the standard deviation of over selections at fixed , measured bin by bin and reduced to one number per fitted row by the median over the bins of the range in question.
Figure A1. Monte Carlo versus closed-form partition-selection variance against pair separation, at on the P-Millennium; the other two redshifts agree equally well (not shown). (a): empirical at from 105 and 105 independent selections for the denser and sparser threshold respectively (solid, coloured by number-density threshold), with the closed-form ratio variance of equation (20) overplotted (dashed). (b): the residual in each bin, where is the empirical scatter of panel (a) and one of two closed-form predictions. Solid: the ratio variance of equation (20). Dotted: the same calculation with the normalisation held fixed at its expectation, ; the gap between the two curves is the variance that cancels between numerator and denominator. The sparsest threshold, , is shot-noise limited and is excluded here as it is from the P-Millennium-only recipe (see § 5.5).
We fit two forms. The first is a power law in the galaxy count of § 5.4,
and the second separates and ,
We fit , , , and . Within one box , so equation (13) is equation (14) with ; what it adds is , which lets it cross box sizes. Equation (14) tests whether , and gives the dependence on at fixed . We fit both by least squares in , over all separations and separately on one-halo (), transition () and two-halo () scales, ranges wider than those of § 4.3; Table 3 gives the all-scale coefficients and selection cuts. The fits use powers of two in on the coarser grids of § 4.2, each over at least 105 selections.
| Equation (13) | ||||
| Calibration | rms [dex] | |||
| P-Millennium | 45.6 | 0.176 | ||
| P-Millennium + Millennium I | 38.0 | 0.159 | ||
| Equation (14), P-Millennium + Millennium I | ||||
| rms [dex] | ||||
| 0.136 |
Table 3. Coefficients for the all-scale recipes, fitted to the median fractional scatter over . The rms is the residual scatter in . In equation (14), use as a fraction and in . The P-Millennium-only fits use the and samples and ; the two-simulation fits use the – samples with power-of-two , and selected galaxy counts above 1000.
Intermediate partition fractions were run only inside brackets around the 0.05 dex crossings of a preliminary, 105-selection estimate of the thresholds; they sample unevenly, so they are not used in fitting. The recipe predicts the typical scatter, the median over bins, whereas the criterion of § 4.4 is set by the worst bin, and equation (14) has no redshift term. A single bin meets a tolerance of dex when , or 0.115 for dex, but at the thresholds of Table 2 equation (14) gives (extrapolating above ) a median of only 0.026–0.103, so inverting it at 0.115 would underestimate by a factor of up to about nine. No single target fixes this, because the factor ranges from 1.2 to 2 across Table 2. The partition fraction is therefore taken from Table 2 and Figure 4 where they apply, and otherwise from repeated selections.
The left panel of Figure 6 shows that within one box predicts the scatter: pooling every density, redshift and of the P-Millennium, the and samples fall on one power law with and an rms scatter about the fit of 0.18 dex, which is the accuracy to which the recipe predicts . At fixed and box size , so this is an rms of dex, a factor of two, in the that gives a given typical scatter. Fitted range by range, the exponent is close to the value expected for independent counts only on one-halo scales, where the scatter is set by how many galaxies are drawn; on larger scales the scatter falls faster with . The exponent depends on the range of fitted. At larger the measurements lie below the fit, since the scatter must reach zero at , so the recipe is conservative at large .
Figure 6. The fractional scatter of the SaUCE estimator and the recipe for predicting it (Section 5.5). Left: the median over the radial bins, in 0.148 dex bins, against the galaxy count in the selected partitions, , for every density, redshift and of the P-Millennium. The solid line is equation (13) fitted to the and samples (filled) at per cent; the dashed line is the (open), is not fitted. Right: measured against predicted scatter for equation (14), fitted on the P-Millennium and Millennium I. Each filled circle is predicted by the fit to the other simulation; open circles are measurements outside the fitted range, shown against the fit to both; Millennium II (diamonds) is not fitted but predicted by the fit to both; the four full diamonds (two models, two redshifts; the models nearly coincide) pass the cuts of Table 3 and give the comparison of Section 5.5. The dotted line is 1:1. Both axes are capped at , which excludes one of the 649 points. Colour is number density on the left and simulation on the right.
For the P-Millennium-only fit, the sparsest sample is not fitted. Its points with lie on the extrapolated relation; points with , in any sample, lie above it. The recipe therefore applies where the scatter is smaller than the signal. Where the predicted approaches , must come from repeated selections.
Pooled over all scales, and on one-halo scales, and differ by less than 0.2. On two-halo scales in each simulation separately ( and ): spreading a fixed number of galaxies over more partitions lowers the two-halo scatter, since two-halo pairs lie mostly between partitions, so there and do not act only through .
The right panel of Figure 6 tests whether the recipe transfers between simulations. Fitted over all scales on the P-Millennium and Millennium I together, equation (14) gives and . Predicting each simulation from the fit to the other raises the rms from 0.14 to 0.16 dex, across a change of cosmology and partition count; divided by , these are factors of 1.6 and 1.7 in .
The data cannot constrain a box-size term: the P-Millennium and Millennium I have similar box sizes, and only four Millennium II measurements pass the cuts of Table 3, all at and (two models at two redshifts), a single location in . The two-simulation fit underestimates Millennium II’s (filled diamonds in Figure 6; its medians are taken over –, the edge of its grid) by about a factor of eight, overstating the precision. Equation (13), refitted on the same two simulations, contains the box size through , and overestimates it by about a factor of 2.3. We therefore recommend equation (14) for boxes of –, and equation (13) for other box sizes.
Bin-to-bin covariance
The scatter characterised above is not independent between separation bins, and Figure 7 shows the bin-to-bin correlation matrix of the SaUCE estimator, measured across selections at for the sample. A block of strongly correlated bins, with , extends across the one-halo to two-halo scales (–) at every partition fraction shown. This structure is a consequence of the partition sampling: adding or removing a partition changes the galaxy content coherently across a wide range of separations, so a selection that scatters high in one bin tends to scatter high in its neighbours. The correlation is therefore a property of which haloes are selected rather than of the pair counting within a fixed catalogue. Each carries noise of from 100 selections, so the isolated high and low cells in the largest-separation corners, where pair counts per partition are low, are noise. The final panel quantifies how that coherence decays, plotting the median off-diagonal correlation against bin separation . The three sparser fractions track one another closely; per cent decays more slowly, staying some 0.1 higher at every separation out to 2 dex.

Figure 7. Bin-to-bin correlation matrix of the SaUCE estimator across selections (, , 0.148 dex bins), for , 3.12, 6.25 and 12.5 per cent, from 100 selections each, the same number at every so that the panels are directly comparable. The final panel shows the median off-diagonal correlation against bin separation . Error bars are the 16th–84th percentiles of 800 bootstrap resamplings of the bin pairs contributing to each point, the uncertainty on the median; bin pairs sharing a radial bin are not independent, so the bars are a lower bound. Separation bins holding fewer than four bin pairs are omitted.
How many independent quantities the bins carry is measured by the effective number of bins, , where is the -th eigenvalue of and the sums run over all of them. equals the number of bins when the bins are uncorrelated and falls
to one when they are perfectly correlated. We compute from the correlation matrices of Figure 7, once for all 27 bins and once for each clustering regime of § 4.3, restricting to that regime’s bins; the ranges quoted span the four fractions shown. Across all 27 bins –7.8. The one-halo and transition regimes, four bins each on this grid, give 1.3–1.5 and 1.6–2.0, and the two-halo regime, six bins, gives 2.6–3.4.
Subdividing the radial range more finely therefore adds little independent information about the partition-selection scatter: the additional bins are almost perfectly correlated with their neighbours, raising the nominal bin count without raising .
A likelihood must therefore use the full covariance: a diagonal approximation would count correlated fluctuations as independent evidence and give artificially tight constraints. The correlation structure changes only slowly with and may be reused across the range used for calibration, but should be estimated at the largest in use, since denser selections retain correlations to larger separations.
The matrix of Figure 7 is the covariance of the estimator across selections: it captures the uncertainty introduced by choosing of partitions in a fixed simulation, and is measured cheaply by repeating the selection. It does not contain the cosmic variance of the parent -body realisation, which is common to every selection, nor the covariance of the data; a full error budget must add both.
Redshift-space distortions
Figure 8 shows the ratio of the redshift-space monopole to the real-space correlation function for . SaUCE tracks the reference at all three redshifts, including the finger-of-God suppression at small , where the velocity dispersion within haloes stretches pairs along the line of sight, and the Kaiser boost on large scales (Kaiser 1987).
Figure 8. Ratio of the redshift-space monopole to the real-space correlation function, , for at the three redshifts annotated. Numerator and denominator come from the same selection. The reference is thick black, beneath the SaUCE estimates at per cent (red dashed) and 9.77 per cent (blue dash-dotted). The horizontal dashed line, annotated in each panel, is the linear-theory Kaiser boost , with , the linear growth factor and the galaxy bias measured over 10–30 with from linear theory.
At fixed the departure from the reference grows with redshift, for the reason given in § 5.2.
Figure 9 shows the scatter of both multipoles, from the same selections as in real space. The multipoles were measured at powers of two in up to 512 and at every integer in some of the intervals between them; where a threshold falls in an interval that was not filled, we quote the bracket. At a depth of 105 selections, the monopole’s full-range threshold is 9.5 per cent at and 18 per cent at , compared with 8.2 and 12 per cent for . At the bound is per cent, which contains the real-space threshold of 24 per cent, so their ordering is unresolved. At its requirement is set by the smallest separations, and at by the two-halo range, as in real space. The quadrupole is far more expensive: over the full range it meets the criterion at per cent at , the top of the measured range, but not at the measured fractions up to 50 per cent at the higher redshifts.
Figure 9. The scatter of the redshift-space multipoles against the partition fraction (P-Millennium, L16, ), at , and (colour). Left: the monopole . Right: the quadrupole , from the same selections. The ordinate is the half-extent of the band of (Section 4.5) in the worst bin over , with from equation (12). A curve meets the criterion of § 4.4 where it falls below the dashed 0.05 dex line; the dotted line marks 0.01 dex. At the smallest the band extends below , where is undefined, so those points are not drawn.
At every redshift the quadrupole’s worst bins are the three smallest, . Without them the threshold falls to 31 per cent at , while per cent at the higher redshifts; intermediate partition counts within this bracket were not run. Range by range, the quadrupole needs more than the monopole everywhere, two to four times as much on one-halo scales.
The smallest scales are hardest because the quadrupole is a difference between the clustering along and across the line of sight. There that difference comes from the velocity dispersion within individual haloes, and SaUCE keeps or discards each halo whole, so the massive, high-dispersion haloes that contribute most of it are either all in a selection or all out of it. We attribute to the same effect the cost of the monopole’s smallest separations: at the same depth of 105 selections, its bins at need 5.4–18 per cent, against at most 1 per cent for the same separations in real space (Figure 3). A quadrupole analysis should therefore budget for the range of separations it uses, not for the monopole’s fraction.
Conclusions
SaUCE allows galaxy clustering to play a role in the calibration of galaxy formation models. The weights and (equation (7)) depend only on and , so the correction has no free parameters, and the corrected pair counts are exactly unbiased whatever model populates the haloes. We tested it with two GALFORM variants on three -body simulations, at number densities from to and redshifts . The results reduce to the following recipe.
Partition the trees and select the sample. Assign whole merger trees to partitions, draw of them uniformly at random, and weight within- and between-partition pairs by and (§ 3.3); the weights apply unchanged to any binning of the correlation function, including and . The galaxy formation model must process each tree independently, and whether a galaxy enters the sample should not depend on which partitions are drawn, except through a threshold set from the sparse catalogue (§ 3.5).
Choose the tolerance. We require at least 68 per cent of selections to lie within 0.05 dex (12 per cent) of the full-box result in each bin. This is larger than the difference of up to 8 per cent in clustering between the two models used here (Appendix B), so distinguishing models this close needs a tighter tolerance and a larger .
Read off the partition fraction where it has been measured. In the P-Millennium, per cent meets the 0.05 dex tolerance over – at and : fourteen times fewer trees to process. The fraction rises with redshift and as the galaxy number density falls, to 29 per cent at and (Table 2); smaller separations alone need less (Figure 4). For the sparsest sample, , the full range needs 96 per cent, so SaUCE saves almost nothing unless the smallest separations are dropped.
Otherwise, measure it. Determine with repeated selections. Equations (13) and (14) (§ 5.5) show how the typical scatter changes with , and box size, and so which way to move from the nearest case in Table 2; between boxes, transfers through galaxy count on small scales and through the pair count per unit shell volume on large scales (§ 5.4).
Use the full covariance. The scatter is strongly correlated between bins (§ 5.6), so a likelihood must use the full covariance, measured at the largest in use, and must add to it the cosmic variance of the simulation and the covariance of the data.
Redshift space. The monopole costs up to 1.5 times as much as , the quadrupole far more, because its smallest separations are dominated by a few high-dispersion haloes; an analysis should budget for the separations it fits (§ 5.7).
We next plan to use SaUCE as the forward model in simulation-based inference, propagating the covariance measured here, and to apply it to other semi-analytic models, to halo occupation models, where the same weights hold, and to abundance matching, where matching on the sparse catalogue sets the threshold tested in § 3.5. SaUCE is complementary to methods that reduce cost at other stages, such as the cosmological rescaling of merger trees used by Stiskalek et al. (2026): rescaling reduces the cost of building trees for each cosmology, while SaUCE reduces how many are populated.
Acknowledgements
OJH was supported by a UKRI PhD Studentship (grant number UKRI1738) and Durham University. CMB, BL, SB and KW acknowledge support from STFC (ST/X001075/1). SB is supported by a UKRI Future Leaders Fellowship [MR/V023381/1 and UKRI2044]. BL is funded by the ERC Advanced Grant UNCA (UKRI Frontiers Research Guarantee EP/Z533877/1).
This work used the DiRAC@Durham facility managed by the Institute for Computational Cosmology on behalf of the STFC DiRAC HPC Facility (www.dirac.ac.uk). The equipment was funded by BEIS capital funding via STFC capital grants ST/K00042X/1, ST/P002293/1, ST/R002371/1 and ST/S002502/1, Durham University and STFC operations grant ST/R000832/1. DiRAC is part of the National e-Infrastructure.
OJH acknowledges the use of Large Language Models as a writing and research aid in the preparation of this manuscript, including literature searches, drafting assistance, and language editing; all scientific content, analysis and conclusions are the work of the authors.
Software
The SaUCE package is written in Rust (Matsakis & Klock 2014), with a Python interface ([50]). This work also made use of Corrfunc (Sinha & Garrison 2019, 2020) for pair counting, NumPy (Harris et al. 2020), SciPy (Virtanen et al. 2020) and Matplotlib (Hunter 2007) for analysis and figure production, and NASA’s Astrophysics Data System for bibliographic information.
The Rust/Python package implementing SaUCE is released on PyPI as pySaUCE (https://pypi.org/project/pySaUCE/), with source hosted on GitHub at https://github.com/OscarHickman/SaUCE under the MIT licence.
Data availability
The GALFORM galaxy catalogues may be shared on reasonable request to the corresponding author.
Closed-form partition-selection variance
The scatter of the estimator from one selection to another can be computed without repeating the selection, and this confirms (Figure A1) that the empirical variance fitted in Section 5.5 is what the random choice of partitions predicts, rather than a coincidence of the samples tested. The calculation uses one property of the selection: a pair survives only if every partition it involves is selected.
Write for the set of selected partitions, for the pair count within partition , and for the count between partitions and , all within a fixed separation bin. The two terms of equation (3) are then sums over ,
and , are the same sums taken over all partitions. The corrected count is , equation (3). The pair counts are fixed by the catalogue and the weights by and ; all of the randomness is in , so every below is an average over the choice of , as in Section 3.3.
Extending equation (2), the probability that specified partitions are all selected is
A term of names one partition and survives with probability ; a term of names two and survives with probability . Taking expectations term by term returns equations (4) and (5).
The variance needs the same rule applied to products. Squaring multiplies every term by every other, and such a product survives only if all the partitions the two terms name between them are selected, so its expectation is with the number of distinct partitions in that union. Four sums over the full catalogue, with , are enough to write the moment down, one for each way two terms can overlap, by coinciding or by sharing an index,
Expanding the square, two auto terms name one partition if they coincide, contributing at , and two if they do not, contributing the remainder at . An auto and a cross term name two if the auto index lies in the cross pair, contributing , and three if it does not, contributing . Two cross terms name two, three or four: their pairs coincide, giving ; share one index, giving , the subtraction removing the coincident products also collects, each counted from both ends; or are disjoint, giving the remainder . Hence
and . Every sum on the right is a property of the full -partition catalogue; the selection enters only through , and , each a function of and alone.
We validated equation (18) against brute-force evaluation on all possible selections at , recovering the enumerated variance to machine precision at every from 2 to 7.
Brute-force enumeration is only feasible for small , so for the production partitioning we evaluate equation (18) directly on the measured pair counts, with no averaging over random selections.
The random normalisation
is not by itself the variance of the estimator. The quantity measured is , and is built from the galaxy count actually selected, with the number of galaxies in partition : , the number of ordered pairs of selected galaxies, with a constant that does not depend on the selection and cancels below. is therefore random too, and correlated with , since a selection that happens to contain denser-than-average partitions raises both.
Because has the same form as , equation (18) applies to it directly. Multiplying out ,
again a sum over selected partitions plus a sum over selected pairs of them.
Replacing by and by in equation (18) therefore gives . The bilinear form of the same expansion, in which each square becomes a product of a term and a term and the auto–cross term is split into its two orderings, gives . With these, the delta method gives the variance of the estimator,
The three moments are exact, and agree with enumeration to the same precision as above; only truncating the ratio expansion is approximate. That truncation error is set by the partition-to-partition scatter in galaxy count, measured by its coefficient of variation : the standard deviation of over the partitions divided by their mean.
Enumeration at , with partitions given the of our samples, puts it at about one per cent, never more than 1.4 per cent, always in the sense of underestimating .
The bracket shows whether dividing by raises or lowers the
directly. Multiplying out , fractional variance. Write and for the coefficients of variation of and across selections, and for their correlation. The bracket is then , which is smaller than the numerator’s own when . Here , which reproduces the enumerated to 2–4 per cent; therefore falls as . Our samples satisfy at every separation and for both densities.
Beyond almost every pair spans two partitions, so and both scale as the square of the selected galaxy count and are nearly perfectly correlated, in the largest bin of Figure A1: a selection that catches denser partitions raises numerator and denominator together, and the fluctuation cancels from . The bracket falls from at to in the largest bin. Holding the normalisation fixed at its expectation ignores this cancellation, and Figure A1 shows the cost: it overstates the scatter by up to a factor of about three in the largest bins, whereas equation (A6) matches it to within a few per cent in the median.
Galaxy formation models
The two GALFORM variants were calibrated for different purposes: L16 (Lacey et al. 2016) to reproduce galaxy populations from the ultraviolet to the submillimetre from to , and GP14 (Gonzalez-Perez et al. 2014) to reproduce the local - and -band luminosity functions with a universal solar-neighbourhood stellar initial mass function (IMF). They differ in the parameters of Table B1, which control supernova feedback and the return of ejected gas, AGN feedback, star formation and the IMF. L16 forms stars in bursts with a top-heavy IMF, which helps it reproduce the number counts of submillimetre galaxies at high redshift; GP14 uses the Kennicutt (Kennicutt 1983) IMF throughout. These differences change the stellar masses of the galaxies in each halo, how many galaxies populate each halo and hence which galaxies enter a sample of given number density. For at in the P-Millennium, satellites (non-central galaxies) make up 39.6 per cent of the L16 sample and 39.9 per cent of the GP14 sample. The full-box correlation functions of the two samples differ by up to 8 per cent, L16 being lower at all but the smallest separations and by about 2 per cent on the largest scales measured for these samples, ; at the same density at they differ by less than 3 per cent.
Shape of the partition-selection distribution
Appendix A gives the width of the scatter between selections; this appendix gives its shape, and justifies three choices made in Section 4. The scatter is close to lognormal (Appendix C1), so the band is stated in . Because the distribution is skewed, a typical selection falls below the reference even though the mean does not (Appendix C2), so the bias is reported as a mean. And discarding selections that return biases the result upwards (Appendix C3), so the acceptance criterion counts them as failures.
Unless stated otherwise we use the P-Millennium L16 catalogue at and the samples and . We use the 25 bins of the coarse grid of Section 4.2 with centres below . We call each combination a cell and study the ratio of equation (11) over its selections: 105 in 95 per cent of uncensored cells, and between 192 and 415 in the rest. A cell in which any selection returns is censored: is undefined for that selection, and the missing low tail would distort any test of shape. Of the 3425 cells with selections, 131 are censored; Appendices C1 and C2 use only the 3294 uncensored cells, and Appendix C3 treats the censored ones. We measure the width of a cell by the standard deviation of over its selections, which spans 0.008 to 1.07.
Normality of the log-ratio
Figure C1(a) shows one of the widest cells, , with a lognormal and a normal overplotted, each given the moments of the data rather than fitted to the histogram. The lognormal better matches the floor at and the long upper tail, though it still understates the skewness. Figure C1(b) extends the comparison to every cell, applying the Anderson–Darling test for normality to and to .
Figure C1. The distribution of over selections. (a): one cell (, , , 315 selections), with a lognormal (solid) and a normal (dashed) given the sample mean and standard deviation of and of respectively; the rightmost bin collects every selection with . (b): fraction of the 3294 uncensored cells in which the Anderson–Darling test rejects normality at five per cent, for (solid) and (dashed), against , the standard deviation of over a cell’s selections, with binomial errors. (c): the measured displacement of the median below the reference, , divided by the lognormal prediction of equation (21), against ; the dotted line marks unity. Each point is the ratio of the binned medians of the two quantities, in eight equal-count bins of per sample; bars are the 16th–84th percentiles under bootstrap resampling of the cells in the bin, and are a lower bound. Bins below are not drawn (Appendix C2).
The log-ratio is closer to normal than : its Anderson–Darling statistic is smaller in 73 per cent of cells, and normality is rejected at the five per cent level for in 18 per cent of cells against 34 per cent for , with the lower rate for \sigma_{\ln}$. For the 986 cells with the rejection rate for is 4.9 per cent, the rate expected if every cell were exactly lognormal. In the widest sixth of cells, , it rises to 47 per cent: the denser sample has a heavier tail than lognormal, and the sparser one tends back towards a Gaussian as shot noise dominates. The contrast holds at , where normality is rejected in 14 per cent of the 3836 uncensored cells for and 22 per cent for . Neighbouring cells share partitions and, so these data describe the ensemble rather than forming independent tests.
The acceptance criterion is met at , at the edge of the range in which the test finds no departure from lognormality, so the band can be stated in and its width predicted from the variance.
Skew and the mean–median offset
Equations (4) and (5) make equal to times the full-box count at every , matching the normalisation of on average,
For , is the fractional scatter of equation (20), so for a lognormal distribution the median offset is set by the variance. Figure C1(c) tests this, comparing binned medians of the measured and predicted , since a per-cell ratio is too noisy at small . With no free parameter, the prediction follows the measurement over more than a decade to within a factor of about two; it understates the offset at and overstates it in the widest cells at . For a single cell the predicted offset, about , is smaller than the sampling error of a median over 105 selections, about , up to ; binning the cells reduces that error, and below even the binned medians cannot measure the offset, so panel (c) starts there.
A median of , or a mean of , is therefore offset by dex even for an exactly unbiased estimator: a tenth of the 0.05 dex tolerance at , and all of it at A small estimate will doubtless mostly measure the skew of the distribution, so we report the bias as a mean of .
so is unbiased to leading order (§ 3.3). We test this in every uncensored cell by dividing the offset of the mean of from unity by its standard error, giving , which for an unbiased estimator follows a unit normal. The median is 0.71 against the 0.67 expected, the 68th percentile is 1.07 against 1.00, and 6 per cent of cells lie beyond , against 4.6 per cent. The typical mean offset, the median over cells, is . The cells are not independent, so this is a consistency check rather than a formal test, but it excludes a bias at the per-cent level.
The median is biased low, as Figure C2 shows at : the mean stays at zero at every uncensored , while the median lies below it and approaches zero only as grows. Across all uncensored cells the median offset, , is typically , more than an order of magnitude larger than the mean offset, consistently negative, and larger at smaller .
Figure C2. Mean (dotted, circles, with standard errors) and median (solid, squares) of against partition fraction at , for both samples, at with at least 100 selections returning . Figure C1(a) shows the distribution at one point of the green curve. The open circle marks a cell in which more than five per cent of selections return , biasing both statistics (Appendix C3). The -axis is linear within and logarithmic outside it.
The offset follows from the lognormal shape. For a lognormal variable of unit mean whose logarithm has standard deviation , the median, and the relation between and the coefficient of variation , are
Positivity censoring in the sparse limit
At , 11 and 32 per cent of selections and bins over return for the denser and sparser samples respectively. At small separations, where , this happens only when the selected partitions contain no pair in the bin. Discarding these selections keeps only those that contained a pair, removing the low tail and biasing both the mean and the median upwards.
The bias is confined to a few cells but is large in them. In the 17 cells with more than five per cent of selections returning and at least 100 returning , the median over cells of the mean offset of the remaining cells, , is +0.351, against for the uncensored cells, and it reaches +3.5; almost all are positive. The bias tracks the censored fraction: for at , the fraction is 34.3, 5.7 and 0.0 per cent at and 8, and the median offset, , falls from +0.32 to a value consistent with zero.
This paper has been typeset from a \TeX/\LaTeX file prepared by the author.
References
- [1]Landy, S. D. & Szalay, A. S. 1993, ApJ, 412, 64DOI
- [2]Kaiser, N. 1986, MNRAS, 227, 1DOI
- [3]Cooray, A. & Sheth, R. 2002, Phys. Rep., 372, 1DOI
- [4]Berlind, A. A., & Weinberg, D. H. (2002). The HOD and clustering of galaxies. The Astrophysical Journal, 575(2), 587–616.
- [5]Zheng, Z., & al., . et . (2005). Theoretical models of the halo occupation distribution: separating central and satellite galaxies. The Astrophysical Journal, 633(2), 791–809.
- [7]Laureijs, R., & al., . et . (2011). Euclid Definition Study Report. ESA/SRE.
- [25]Henriques B. M. B., White S. D. M., Thomas P. A., Angulo R. E., Guo Q., Lemson G., Springel V., 2013, MNRAS, 431, 3373
- [39]Peebles, P. J. E. (1980). The Large-Scale Structure of the Universe. In Princeton University Press.
- [40]Planck Collaboration 2014, A&A, 571, A16
- [49]van Daalen M. P., Henriques B. M. B., Angulo R. E., White S. D. M., 2016, MNRAS, 458, 934
- [50]van Rossum, G., & Drake, F. L. (2009). Python 3 Reference Manual. In CreateSpace.









