In this paper
AbstractHighlightsWhat is already knownWhat is newPotential impact for readers outside the authors’ fieldIntroductionWhy Forecast Replication Results from Meta-Analyses?First Source of Discrepancy: Publication Bias in Meta-AnalysesSecond Source of Discrepancy: Differences in Study CharacteristicsPrediction Intervals and Evaluating Replication SuccessContributions and Preview of FindingsOrganization of the PaperMotivating Study: ManyBabiesManyBabies 1: Infant-Directed Speech PreferenceManyBabies 4: Social EvaluationMethodsNotationCalibrated EstimatePublication Bias CorrectionPrediction of Replication Studies Using Meta-Analysis ResultsAssumptionsPrediction IntervalForecasting modelsEvaluation MetricsSimulation StudySimulation DesignSimulation scenariosData generating processEstimation ProcedureSimulation ResultsBetween-study heterogeneity ()Moderator strength ()Number of studies in the meta-analysis ()Application to ManyBabiesData PreprocessingManyBabies 1ManyBabies 4Estimation ProcedureResultsDiscussionDiscussionPrincipal findingsAccuracy-calibration tradeoffUnexplained heterogeneity and unmeasured moderatorsUnexplained heterogeneity and unmeasured moderatorsFuture directionsAppendix for ``Forecasting Replication Study Results from Meta-Analyses''Setup and assumptionsProof of Proposition 1. Calibrated estimatesProof of Proposition 2. Asymptotic coverageStandard parametric prediction interval versus KDE-based prediction intervalSimulation results under normal distribution of ReferencesWorksMathematics
Forecasting Replication Study Results from Meta-Analyses
Abstract
Multisite replication projects often rely on published meta-analyses to plan their sample sizes, but meta-analyses tend to overestimate population effects because of publication bias and differences in study characteristics. We developed methods to forecast site-level estimates of a not-yet-conducted replication from a meta-analysis. Extending calibrated estimates from meta-regression, we constructed nonparametric prediction intervals that condition on each site's moderator levels and correct for publication bias through inverse-probability-of-publication weighting under an assumed severity. These intervals achieve nominal coverage asymptotically when population effects are normally distributed, with a bounded coverage error otherwise. We compared four forecasting models (intercept-only meta-analysis or meta-regression, each with and without publication bias correction) in simulation study (81 scenarios and blinded retrospective forecasts of two multisite replications in infant cognition: ManyBabies 1 (104 sites) and ManyBabies 4 (33 sites). Correcting for publication bias improved interval calibration, with gains that increased with bias severity. Meta-regression improved forecasts only when measured moderators explained effect variation and replication sites differed meaningfully on them. In practice, publication bias correction should be the default when forecasting replications from meta-analyses, and replication projects that deliberately vary measured moderators across sites would enable more informative forecasts.
Research Articles
Highlights
What is already known
In psychology, meta-analytic estimates are on average nearly three times larger than the matching multisite replication estimates. This gap is largely attributed to publication bias.
There is no consensus on how to judge replication success. Proposed criteria include significance agreement, combined -value tests, and checking whether the replication estimate falls within a prediction interval. Most of these compare one original study with one replication and ignore heterogeneity across multiple replication sites.
Existing prediction intervals for meta-analysis cannot handle study-level moderators and publication bias at the same time, hence cannot produce site-specific forecasts for a planned replication.
What is new
We propose nonparametric prediction intervals for each replication site with an asymptotic coverage guarantee. They are built from meta-regression calibrated estimates that condition on the replication site’s moderators and correct for publication bias in the meta-analysis study with inverse-probability-of-publication weighting.
In a simulation study and in two ManyBabies replications, publication bias correction drove most of the improvement in forecasts. Meta-regression helped only when the measured moderators explained effect variation and the replication sites varied on them.
No single model was both the most accurate and the best calibrated, revealing a tradeoff between point accuracy and interval calibration.
Potential impact for readers outside the authors’ field
Before a multisite replication begins, the method turns an existing meta-analysis into a point forecast and a 95% prediction interval for each site. Researchers can use these to plan sample sizes, choose site conditions, and judge whether an observed result is surprising. The method applies in any field where a published meta-analysis exists for the effect a multisite replication will test.
When forecasting from a meta-analysis, correct for publication bias by default. Use meta-regression only when the moderators explain effect variation in the meta-analysis and the replication sites differ on them.
Replication projects can make forecasts more informative by varying prespecified moderators across sites. A fully standardized protocol leaves no moderation variation for meta-regression, which limits forecasting and assessment of whether the effect generalizes across study characteristics.
Introduction
Multisite replication projects, in which many independent laboratories follow a shared protocol to reverify a published finding, have become a central tool for assessing the credibility of published research. However, these projects are costly and time consuming. When one is being planned, the best available evidence about the effect size usually comes from a meta-analysis of the published literature. The meta-analysis literature, however, suffers from publication bias and spans heterogeneous study designs: a study found that across 15 psychology topics, meta-analytic estimates were on average nearly three times larger than the corresponding multisite replication estimates [1]. This then raises a practical question for researchers planning a replication: given an existing meta-analysis, what effect should they expect at each replication site, and how much should that effect plausibly vary across sites?
In this paper, we develop and evaluate methods for forecasting the results of not-yet-conducted replication studies from published meta-analyses. For each replication site, our approach yields a point prediction and a 95% prediction interval that adjust for a hypothesized degree of publication bias (the estimated selection ratio depending on the discipline[2]) and condition on the replication study site’s own study-level characteristics through meta-regression. The selection ratio quantifies the severity of publication bias as the number of times more likely affirmative results are to be published than nonaffirmative results. We compare four forecasting models, crossing intercept-only meta-analysis versus meta-regression with and without publication bias correction, in a simulation study and in two multisite replications of infant cognition findings, ManyBabies 1[3] and ManyBabies 4[4]. Finally, we discuss the limitations of the forecasting models and the conditions under which each is likely to be informative. In particular, we consider how the different purposes of meta-analyses and multisite replications bear on forecasting: meta-analyses synthesize studies whose populations and procedures vary, whereas multisite replications typically standardize their protocols to estimate a single effect precisely. We also examine the role of measured and unmeasured moderators in explaining differences between meta-analytic and replication results, and what these considerations imply for the design of future replication projects in psychology and beyond.
Why Forecast Replication Results from Meta-Analyses?
Large-scale efforts to measure the reliability of published research have shown that a single study is rarely decisive. The Systematizing Confidence in Open Research and Evidence (SCORE) program assessed the credibility of 3,900 claims from the social and behavioral sciences and empirically tested a subset of them for reproducibility, robustness, and replicability [5, 6, 7]. Most claims could be reproduced when the original data and code were available [6], but only about a third proved robust across independent reanalyses [5], and half of well-powered replication attempts reproduced the original study’s statistically significant result, typically with substantially smaller effect sizes [7]. These findings caution against treating a single published analysis, or even a single replication of it, as dispositive evidence for or against a claim.
One solution has been the multisite, or “many-to-one,” replication design, in which a single original study is replicated across many independent laboratories following a standardized, preregistered protocol, often developed together with the original authors [8]. “Many Labs” projects extend this design by replicating several original studies simultaneously across the same set of sites [9, 10]. Because their results are reported regardless of outcome, multisite replications provide large-sample estimates that are largely free of publication bias, and they have become a benchmark against which the published literature is judged.
The discrepancy between estimates of meta-analysis studies and large preregistered replications is consistent in direction rather than random [1]. This suggests that a meta-analysis, once corrected for selective publication and conditioned on the characteristics of each replication site, may still carry useful predictive information about what a planned replication study will find. A forecast made before data collection would serve several purposes: it provides a prespecified benchmark for judging whether a replication result is surprising, informs sample size calculations and the choice of study conditions, and clarifies whether publication bias or differences in study design could drive any discrepancy with the replication.
Developmental psychology offers a natural test case. Research with infants faces discipline-specific replicability challenges, notably small samples and noisy measurement [11]. The ManyBabies consortium conducts multisite, preregistered replications of influential findings in infant cognition and selects effects for replication in part on the basis of existing meta-analyses [11]. As a result, both projects we study in this paper were paired with a meta-analysis of the same effect. The two projects also differed in a way that matters for forecasting: ManyBabies 1 sites varied on several study characteristics that were also measured and recorded in the corresponding meta-analysis [3], whereas ManyBabies 4 standardized its protocol and stimuli across all study sites [4]. We next consider two mechanisms that may account for discrepancies between meta-analytic and multisite replication estimates.
First Source of Discrepancy: Publication Bias in Meta-Analyses
Meta-analyses are widely used to synthesize evidence across studies, but their validity can be compromised at two levels: within individual studies, through confounding and measurement error, and across studies, through the selective publication of results. In this paper, we focus on across-study level, the most direct explanation for the systematic gap between meta-analytic and replication estimates.
Kvarven et al.[1] compared 15 psychology topics for which both a meta-analysis and a preregistered multi-laboratory replication were available. In 12 of the 15 pairs, the meta-analytic effect size was significantly larger than the replication effect size, and on average the meta-analytic estimates were nearly three times as large. This inflation was consistent with publication bias and selective reporting in the primary literature. However, applying three established publication bias correction methods (trim-and-fill, PET-PEESE, and the three-parameter selection model) to the meta-analyses did not meaningfully improve their agreement with the replication estimates.
One possible reason these corrections failed is that each must infer the severity of selection from the same, often small, set of published studies it is trying to correct, and such estimates can be unstable. An alternative approach treats the severity of publication bias as a sensitivity parameter rather than estimating it from the meta-analysis itself, and corrects the estimate under an assumed degree of selection favoring statistically significant results in the expected direction [12]. This assumed degree of selection can be informed by evidence pooled across many meta-analyses. In a meta-meta-analysis spanning medicine and psychology, Mathur and VanderWeele[2] found that publication bias was generally mild, with one notable exception: selective publication was considerably more severe among meta-analyses in developmental psychology. Although this finding was based on only five meta-analyses, the authors suggested that the small samples typical of infant research may contribute to more severe selection in this field.
Second Source of Discrepancy: Differences in Study Characteristics
Even under a shared protocol, replication results vary across sites: eight of sixteen effects in the first Many Labs project showed statistically significant heterogeneity [10]. Published studies in a meta-analysis typically vary even more, in participant populations, stimuli, and procedures. A single pooled estimate can mask this variation, and a forecast that ignores it implicitly assumes that every replication site resembles an “average” published study.
Meta-regression [13] addresses this by modeling the mean effect as a function of study-level characteristics, or moderators, such as participant age or experimental method. Mathur and VanderWeele[14, 15] extended this approach with calibrated estimates, which shrink each study’s observed estimate toward its model-predicted mean in proportion to the study’s imprecision. The resulting set of calibrated estimates approximates the distribution of true effects at a given moderator level, not just its mean. They used this distribution to quantify, for example, the percentage of true effects that are meaningfully strong at a given moderator level. These metrics describe heterogeneity in existing evidence but were not designed to project the results of future studies.
Prediction Intervals and Evaluating Replication Success
A confidence interval for a meta-analytic mean describes uncertainty about the average effect. A prediction interval instead gives the range within which the estimate from a new study is expected to fall, accounting for both between-study heterogeneity and the new study’s own sampling error. For forecasting a replication site’s result, the prediction interval is the relevant quantity. Wang and Lee[16] proposed a nonparametric prediction interval for standard meta-analysis based on calibrated estimates which makes the variance of the empirical distribution equal the true variance asymptotically. However, their approach, like most existing prediction interval methods[17, 18, 19, 20, 21, 22, 23, 24], does not accommodate study-level moderators and publication bias simultaneously.
Forecasting differs from a related body of literature that evaluates replication success after both the original and replication results are available. Common practice relies on “significance agreement,” a criterion whose expected value depends on the statistical power of both studies. More principled criteria compare a single original study with a single replication. Held et al.[25] proposed a sum-of--values test that avoids requiring independent significance in both studies[26], a test of whether the replication estimate falls within the original study’s prediction interval[27], and the skeptical -value, which penalizes shrinkage from the original to the replication estimate[28]. Inverting Edgington’s combined test[29, 30] additionally yields a combined point estimate and confidence interval with favorable coverage properties[31]. These pairwise criteria, however, do not account for heterogeneity among multiple replications.
Mathur and VanderWeele[32] addressed this evaluation metric for multisite designs by assessing whether a single estimate is statistically consistent with a heterogeneous distribution of effects estimated from many others, rather than with a single pooled estimate. Because this framework compares one estimate against a distribution, it can be extended naturally to comparing a replication result with a meta-analysis.
Contributions and Preview of Findings
This paper makes three contributions. First, we construct nonparametric prediction intervals for replication studies by extending the meta-regression calibrated estimates of Mathur and VanderWeele[15] and the prediction intervals of Wang and Lee[16]. The intervals condition on each replication site’s moderator levels and incorporate a weighting-based correction for publication bias. We show that these intervals achieve nominal coverage asymptotically when the population effects are normally distributed, and approximately nominal coverage otherwise, with a bound on the coverage error. Second, we evaluate four forecasting models, together with kernel density estimate (KDE)-based and standard parametric prediction intervals, in a simulation study that varies the severity of publication bias, between-study heterogeneity, moderator strength, and the number of meta-analyzed studies. Third, we apply the models to two ManyBabies projects whose sites differ sharply in how much moderators varied.
We highlight three main findings. First, correcting for publication bias consistently reduced the tendency of meta-analyses to overpredict replication results. Second, meta-regression improved forecasts only when the measured moderators explained variation in the population effects: in simulation, the corrected meta-regression gave the most accurate forecasts whenever moderation was present, but the corrected meta-analysis was more accurate without moderation, and in the ManyBabies applications meta-regression offered little improvement. Third, correcting for publication bias improved the calibration of the prediction intervals as well as the location of the forecasts, provided the inverse probability of publication weight (IPPW) weights were also applied to the KDE, but it did not fully restore nominal coverage. Under strong publication bias in the simulation, the corrected models’ KDE intervals covered substantially more replication estimates than the uncorrected models’, and in ManyBabies 4, where the shared protocol fixed most moderators across sites, no model reached nominal coverage, suggesting that unmeasured factors drove much of the variation across sites. This pattern has a design implication: replication projects that deliberately vary prespecified moderators across sites would allow more informative forecasts and tests of generalizability.
Organization of the Paper
Section 2 describes the ManyBabies 1 and ManyBabies 4 replication projects and their corresponding meta-analyses. Section 3 introduces notation, calibrated estimates, publication bias correction, and the proposed methods for prediction intervals. Section 4 presents a simulation study, and Section 5 applies the four forecasting models to the ManyBabies data. Section 6 discusses the trade-off between accuracy and calibration, the role of unmeasured moderators, and implications for the design of future replication projects.
Motivating Study: ManyBabies
Developmental psychology research with infant participants faces discipline-specific replicability challenges, notably small samples and a reliance on indirect behavioral measures, including both looking- and action-based measures (e.g. looking time, eye tracking, manual choice, head-turn procedure, imitation). ManyBabies is a large-scale, multi-laboratory collaborative initiative that conducts multisite, preregistered replications of landmark findings in developmental psychology[11]. Studies are selected for replication based in part on evidence from existing meta-analyses.
ManyBabies 1: Infant-Directed Speech Preference
Infant-directed speech (IDS) is the characteristic speech caregivers use with infants across cultures. Compared with adult-directed speech (ADS), IDS is typically higher pitched, has greater pitch excursions, and is characterized by shorter utterances, among other differences [33]. A substantial literature has demonstrated that infants prefer IDS over ADS [34, 35, 36, 37, 38, 39, 40, 41, 42, 43], with looking time as a typical dependent measure.
A meta-analysis of 34 traditionally published studies including 840 infants estimated an uncorrected point estimate of 0.67 on Cohen’s scale (95% CI: [0.57, 0.76]) for the IDS preference in looking time [44]. In the ManyBabies 1 replication project [3, 10, 11, 45], the pooled estimate from 67 labs including 2,329 infants was approximately half as large (0.35; 95% CI: [0.29, 0.42]) [46]. The moderated model found that infant’s age and experimental method (single screen central fixation, head-turn preference procedure, or eye tracking) was predictive of the IDS preference in ManyBabies 1 only, but not in the meta-analysis[46].
A recent synthesis [46] re-analyzed the original meta-analysis [44] and the ManyBabies-Consortium[3] data, concluding that the IDS preference generalizes across a broad range of participants, ages, experimental methods, and stimuli, albeit with substantial remaining uncertainty about how moderators influence the effect magnitude. This work also highlights the key distinctions between meta-analyses and multi-lab replications: meta-analyses estimate effects across heterogeneous design choices and populations in the extant literature, while multi-lab replications offer large-scale, high-precision estimation of key effects within similar implementations, free from publication bias.
ManyBabies 4: Social Evaluation
Hamlin, Wynn, and Bloom [47] conducted a study among preverbal infants to determine if they are able to evaluate people around them as friends or foes. The study used two paradigms: (1) a choice paradigm[48], where infants indicated preferences through reaching behavior, and (2) a violation-of-expectation paradigm [49], in which infants’ looking times are measured on the premise that infants look longer at events that violate their expectations. In the latter paradigm, infants who had watched a climber being helped by one character and hindered by another looked longer when the climber subsequently approached the hinderer than when it approached the helper, suggesting that infants expected the climber to prefer its helper [47, 50].
This initial paper has motivated numerous studies to explore infants’ sociomoral preferences and their expectations regarding scenarios involving morally good and bad agents [51, 52, 53, 54, 55, 56, 57, 58, 59, 60]. Although many studies have reported significant preferences for helpful or fair agents over hindering or unfair agents, several subsequent published studies[60, 61, 62, 63, 64, 65, 66] and unpublished raw data and manuscripts [67, 68, 69] failed to replicate Hamlin et al.’s findings[47].
These failed attempts motivated Margoni and Surian’s meta-analysis of 26 studies[70]. Margoni and Surian[70] included only the studies with the manual task outcome, i.e. the choice paradigm, in Hamlin et al.[47], and their results suggested that about two thirds of infants preferred prosocial agents to antisocial agents. The estimated effect was not moderated by age, the type of dependent variable (selective reaching or helping), or the modality of stimulus presentation (cartoons or real events). Furthermore, there was evidence of publication bias, with the point estimate in the published studies greater than the unpublished ones.
Tan’s[71] overview of the recently published studies in this field including 65 papers and 213 point estimates[70] meta-analyzed studies that examined infants’ social moral preferences with manual tasks and infants’ looking time (e.g., violation of expectation regarding socially relevant events, preferential looking between the two agents, anticipatory looking for expectation of event completion). For the manual task outcome, Margoni and Surian found that scenario type (help/hinder, fair/unfair, give/take, other) and presentation modality (movie, live show) were two significant moderators of the point estimate, but the lab of origin, age, dependent variable (reaching, helping), modality of stimuli presentation (real events, cartoons), and choice object (shapes, puppets, people, experimenters) were not, and there was evidence of publication bias, with the point estimate in the published studies greater than the unpublished ones[70]. For the looking time outcome, the only moderator was the method of measurement (i.e., the type of paradigm used in the task: violation of expectation, preferential looking, anticipatory looking).
The recently published ManyBabies 4[4] attempted to obtain a more precise estimate of the true point estimate for infants’ preference for helpers over hinderers by standardizing methodology across 37 labs worldwide: each lab followed the same protocol and used the same video stimuli. Consequently, most of the moderators recorded in the meta-analytic literature took a single, constant value across the 37 replication studies. The study found no evidence that infants prefer helpers over hinderers (49.34% of infants preferred helpers over hinderers). The exploratory analyses examined several moderators, including attention to video events, clarity of action choices, and experiment-, child-, and lab-level factors. None of the interaction effects were significant.
ManyBabies 1 and ManyBabies 4 differ sharply in how much their replication sites varied on study-level covariates: ManyBabies 1 sites varied meaningfully in infant age, test language, and experimental method, whereas ManyBabies 4 standardized protocol and stimuli across all 37 sites. As we show in § 6, this contrast affects how informative ManyBabies 1 and ManyBabies 4 are for comparing the performance of the four forecasting models.
Methods
The ManyBabies studies described in § 3 share a common structure: each is a multisite replication of a key finding in infant cognition, motivated by a meta-analysis of the published literature on that finding. Each replication site contributes its own estimate of the effect. Our goal is to use the meta-analysis dataset to forecast each site’s estimate before the replication is conducted. Both ManyBabies examples also illustrate the two sources of discrepancy introduced in § 2: the published literature may be affected by publication bias, and the replication sites may differ from the meta-analyzed studies in their moderator distribution. In this section, we introduce and develop forecasting models that address both. § 4.1 introduces notation and the intercept-only and the meta-regression models. We then describe calibrated estimates, which adjust each meta-analyzed study’s estimate to a given moderator level (§ 4.1.1), and a weighting approach that corrects for publication bias (§ 4.1.2). § 4.2 combines these components to construct prediction intervals for the estimate at each replication site, states the assumptions under which the intervals are valid, and establishes their asymptotic coverage. § 4.3 introduces four forecasting models, and § 4.4 introduces the evaluation metrics.
Notation
Let denote the total number of studies in the meta-analysis dataset. For the th meta-analysis study (), let , , and denote the true population effect, point estimate, and estimated within-study variance, respectively.
Denote as the row vector of study-level covariates for study , with the first entry fixed at ( for all ) to accommodate the population-level intercept. Denote as the vector of realized values of covariates observed for study , with scalar entries .
Under a linear meta-regression model,
the full moderator matrix is
with coefficient vector , so that , where . We assume the residual variance of the population effects conditional on the covariates, , is a constant ; this is the residual heterogeneity conditional on the meta-regression covariates, as distinct from the marginal of a standard intercept-only meta-analysis model.
When the model contains only an intercept, and reduce to the mean and between-study variance of the population effects, denoted and , respectively, with estimates and . More generally, let , , and denote the meta-regression estimates of , , and . In practice, we estimate these parameters using robust variance estimation[72, 73], a semiparametric approach analogous to generalized estimating equations that yields consistent coefficient estimates without distributional assumptions on the true population effect of study [15].
For replication studies, we use the index , and denote the point estimate and estimated variance of the th replication study as and , respectively, and the point forecast as , with estimated variance . The realized value of the th replication study’s covariate vector is . For the meta-analysis of studies, denotes the pooled point estimate from an intercept-only model; and denote the estimated coefficient vector of a meta-regression and its robust covariance matrix, respectively; and denotes the estimated between-study variance, which is the residual between-study variance for meta-regression models.
Calibrated Estimate
When the model contains moderators, following Mathur and VanderWeele [15], define the calibrated estimate of the th meta-analysis study, shifted to a specific covariate level , as
The shrinkage factor pulls each study’s residual toward zero in proportion to the study’s imprecision: relatively imprecise estimates (large ) are shrunk strongly toward the meta-regression mean , while precise estimates remain close to their reported values [15, 16]. When there are no moderators, equation (1) reduces to the standard calibrated estimate of Wang and Lee[16]:
When studies share a common dataset or population (e.g., multiple publications from the same study population), we group them into clusters, stacking by cluster:
where the th cluster contributes studies with . When referring to the clustered structure, we index study within cluster by the double subscript , e.g., , , ; when clustering is not relevant we retain the single index used above.
Publication Bias Correction
Let indicate whether study in cluster is affirmative, defined as “having a positive point estimate and a two-sided p-value below 0.05”. Following Mathur and VanderWeele[12], we refer to this as “asymmetric selection process”, i.e. “favoring studies that are both statistically significant and positive in direction”, as “one-tailed selection”, in contrast to a “two-tailed selection” model in which publication favors significant results regardless of the direction. Note that this terminology describes the direction of the selection process itself, not that individual studies conducted one-tailed hypothesis tests; the p-value used to define remains two-sided throughout.
Let denote the ratio of publication probabilities for affirmative versus non-affirmative studies, used to parameterize publication bias. Following Mathur and VanderWeele[12], for publication-bias-corrected analyses, let
denote the inverse probability of publication weight (IPPW) for study in cluster , so that non-affirmative studies receive weight times that of affirmative studies. The per-cluster weight matrix is , and the full block-diagonal weight matrix is .
Remark. We recommend obtaining as a naive estimate from a standard meta-regression without correction for publication bias rather than jointly re-estimating under IPPW [12]. Although this naive estimate may itself be biased by publication bias, this does not compromise the consistency of or the validity of its robust sandwich variance estimator (later in Equation (6)), as the robust specification of Hedges et al. [72] provides unbiased point estimation and valid inference for any fixed choice of weight matrix , so an inaccurate can only reduce estimation efficiency but not compromise validity.
Prediction of Replication Studies Using Meta-Analysis Results
Wang and Lee [16] introduces nonparametric prediction intervals using calibrated estimates in standard meta-analysis, but their approach does not accommodate study-level moderators. Mathur and VanderWeele [15] extends calibrated estimates to meta-regression by conditioning on study-level covariates, but does not construct prediction intervals. To fill both gaps simultaneously, we derive prediction intervals from the meta-regression calibrated estimates of Mathur and VanderWeele [15], conditioning on each replication study’s own covariate vector. We additionally incorporate IPPW to correct for publication bias.
Assumptions
The proposed method rests on four assumptions:
(A1) Random-effects meta-regression: In the population of all conducted studies, , where are independent and identically distributed across studies, the moderators have bounded support, and the residuals are independent of , with mean zero, positive variance , and an absolutely continuous distribution. No parametric form is assumed for the distribution of .
(A2) Within-study normality: , with the known within-study variances, bounded between zero and infinity.
(A3) Selection model (publication-bias-corrected models only): Affirmative studies are times more likely to be published than nonaffirmative studies, that is, , where is known and . Publication depends on a study’s point estimate only through whether a study is affirmative.
(A4) Transportability and independence: The population effect at replication site , with moderator level , follows the same conditional distribution as the population effects in the meta-analyzed literature, with having the same distribution as ; the replication estimate satisfies and is independent of the meta-analysis data.
The following Proposition 1 shows that the meta-regression calibrated estimates (Equation (1)) match the population effects at in mean and variance, extending the key property of Wang and Lee [16], and match their full distribution when the population effects are normal. This justifies their use as the nonparametric basis for prediction intervals.
Proposition 1 (Calibrated estimates). Under (A1) and (A2), in the population of all conducted studies, the oracle calibrated estimates , defined as in Equation (1) but with the true parameters , are independent and identically distributed with mean and variance for every . If is consistent, the feasible calibrated estimates , which use the estimated parameters as in Equation (1), satisfy in probability as . If, in addition, the population effects are normally distributed, then has the same distribution as , and converges in distribution to .
Proposition 1 formalizes the moment-matching argument of Mathur and VanderWeele [15] for the shifted calibrated estimates and extends it to convergence in distribution when the population effects are normal. The proof is given in Appendix § 9.§ 3.
Remarks. Denote , , and , so that . Proposition 1 guarantees that the calibrated estimates match the population effects in mean and variance, but not in shape. When the population effects are not normal, the oracle estimate mixes a shrunken residual with normal sampling noise, so its distribution generally differs from that of ; the two are close when the population effects are approximately normal or when within-study variances are small relative to . Proposition 1 also describes conducted studies; among published studies, the weights need to be used to correct for publication bias under (A3).
We now construct prediction intervals for the point estimate of a future replication study by using the empirical distribution of calibrated estimates from the meta-analysis studies, shifted to the replication study’s covariate level .
Prediction Interval
Uncorrected for publication bias.
The goal is to estimate the marginal predictive density
where indexes the replication study. We assume , i.e., the reported point estimate is normally distributed around the true population effect. Because the distribution of true population effects is unknown, we approximate it nonparametrically using the empirical distribution of calibrated estimates from the meta-analysis studies shifted to .
Following Wang and Lee [16], the predictive density via KDE is
where denotes the Gaussian density with mean and variance , and is the bandwidth, selected using the Sheather–Jones method [74], as recommended by Wang and Lee [16]. The prediction interval consists of the and quantiles of ; for example, the 95% interval uses the 2.5th and 97.5th percentiles.
Corrected for publication bias.
Following Mathur and VanderWeele[12], we assume that affirmative studies are times more likely to be published than non-affirmative studies. We use IPPW, with weights and weight matrices as defined in § 4.1.2, to upweight non-affirmative studies.
The IPPW meta-regression coefficient estimate is
with sandwich variance estimator [12]:
where is the residual vector for cluster .
We then obtain IPPW-calibrated estimates using Equation (1) with in place of the unweighted , and construct prediction intervals as described above. The next proposition characterizes the asymptotic coverage of these intervals.
To state it, we need the limit of the heterogeneity estimate. For the corrected models, is taken from the corresponding uncorrected model, so it need not be consistent for under publication bias. Let denote its probability limit, i.e., as , with when is consistent. Replacing with in the oracle calibrated estimate of Proposition 1 gives the limiting calibrated estimate
Let denote the density of in the population of conducted studies, and the density of . When , is the oracle calibrated estimate .
Proposition 2 below shows that the asymptotic coverage of the KDE-based prediction interval deviates from the nominal level by at most the distance between and , and attains the nominal level when is consistent and the population effects are normal.
Proposition 2 (Asymptotic coverage). Assume (A1), (A2), and (A4). For the corrected models, assume (A3); for the uncorrected models, assume that (A3) holds with ,
Suppose that as ,
(a) , with in place of for the corrected models;
(b) ;
(c) the bandwidth satisfies and .
Then the KDE-based prediction interval satisfies
In particular, if and the population effects are normally distributed, then , and the interval achieves nominal coverage asymptotically.
Condition (a) above states that the meta-regression coefficients are root- consistent, which holds for the robust variance estimators used here [72]. In the special case above, is the density of the oracle calibrated estimates, which equals by Proposition 1. The proof is given in Appendix § 9.§ 4.
Remarks. Four conditions determine how closely the prediction interval approaches nominal coverage in practice.
(i) Beyond the exact case in Proposition 2, is small when the population effects are approximately normal or when the meta-analyzed studies are precise relative to the heterogeneity, because the calibrated estimates then closely resemble the population effects themselves.
(ii) The heterogeneity estimate determines the shrinkage applied to each calibrated estimate. For the uncorrected models, is consistent under standard meta-regression theory when publication does not depend on the study results. For the corrected models, we use the estimate from the corresponding uncorrected model as a plug-in (§ 4.1.2), because the IPPW estimator does not directly yield a heterogeneity estimate. Under publication bias, this plug-in generally converges to some , so the shrinkage factors are misspecified and the coverage error need not vanish even under normality. This error shrinks as , and also when , because then regardless of .
(iii) The IPPW weights in the KDE address a separate problem: they ensure that the estimated distribution represents all conducted studies rather than only the published ones. Without these weights, the uncorrected intervals target the distribution of published studies, and Proposition 2 provides no guarantee for them under publication bias.
(iv) If, conditional on a given moderator level , the population effects at the replication sites differ systematically from those in the meta-analyzed literature, for example, because of unmeasured moderators, Assumption (A4) fails and nominal coverage is not guaranteed even asymptotically.
For comparison with the KDE-based intervals, one may also construct standard parametric prediction intervals, which assume that the replication estimate follows a -distribution centered at the point forecast :
where is the robust standard error of the point forecast and is the quantile of the -distribution with Satterthwaite degrees of freedom [20, 73]. Unlike the KDE-based intervals, these intervals assume that the population effects are normally distributed around the forecast. Appendix § 9.§ 5 describes both types of intervals in detail.
Forecasting models
Throughout the paper, we compare four forecasting models, formed by crossing two settings: whether the model conditions on study-level moderators, and whether it corrects for publication bias.
MA-U: intercept-only meta-analysis, uncorrected for publication bias;
MR-U: meta-regression, uncorrected for publication bias;
MA-C: intercept-only meta-analysis, corrected for publication bias;
MR-C: meta-regression, corrected for publication bias.
The intercept-only meta-analysis models produce the same point forecast for every replication study, whereas the meta-regression models produce study-specific forecasts at each replication study’s moderator levels. For each model, we construct both KDE-based prediction intervals and standard parametric prediction intervals for comparison.
All models are fitted in R using the robumeta package[75], which provides robust variance estimation with Satterthwaite small-sample degrees of freedom[73]. The uncorrected models used hierarchical weights where studies contribute multiple estimates, or correlated-effects weights where each study contributed a single estimate, and the corrected models used the IPPW weights, supplied via the userweights argument. Kernel density bandwidths were selected using the Sheather–Jones method[74].
Evaluation Metrics
We evaluate the four forecasting models using seven metrics that compare each model’s prediction to the reported estimate in each replication study. Three are accuracy metrics: mean absolute error (MAE), root mean square error (RMSE), and mean prediction interval width. Four are calibration metrics [76]: raw difference, which quantifies the directional difference between the meta-analysis studies and replications; empirical coverage of the 95% prediction interval, which tests calibration at the tails of the predictive distribution; percentage of overestimation, which tests calibration at the center by measuring the proportion of replication point estimates below the point prediction; and the mean , which measures how consistent each replication study’s estimate is with the distribution of effects predicted by the forecasting model.
(1) Raw difference: the signed difference between the pooled meta-analytic effect estimate from the replication studies and the pooled estimate from the meta-analysis studies. Because this comparison is made at the pooled level, positive and negative study-level deviations can cancel, so raw difference captures systematic directional bias rather than the per-study error magnitude captured by MAE and RMSE below.
(2) [32]: the probability that replication study would yield a point estimate at least as extreme as the one observed, if its true effect were drawn from the same distribution of true effects as the meta-analyzed studies (conditional on moderators for the meta-regression models). Mathur and VanderWeele [32] proposed to assess whether a single original study is consistent with a heterogeneous distribution of effects estimated from replications. We reverse the roles: each replication study is the single estimate under assessment, and the meta-analyzed studies supply the reference distribution. Both setups share the same many-to-one structure, in which one estimate is compared with a heterogeneous distribution estimated from many others, so the statistic applies without modification. Specifically, for replication study ,
where and are the point estimate and standard error of replication study , and is the model-based prediction for that study. For the intercept-only models (MA-U and MA-C), , the uncorrected or bias-corrected pooled estimate. For the meta-regression models (MR-U and MR-C), , where is the moderator vector of replication study . is the robust standard error of this prediction, that is, or , with the robust covariance matrix of . is the between-study variance, estimated from the uncorrected model (MA-U or MR-U); and MR-C use the from MA-U and MR-U, respectively. A small indicates that replication study lies in the tails of the distribution predicted by the meta-analysis; a large value indicates relative consistency. For each model, we report the mean across the replication studies to characterize average consistency between the replication study estimate and its predicted value.
(3) Mean absolute error (MAE): the mean of the absolute differences between the replication study estimate and the forecasting model’s predicted value .
(4) Root mean square error (RMSE):
(5) Percentage of overestimation: the percentage of replication studies for which the model’s predicted estimate exceeded the reported estimate (). The target is 50%.
(6) Coverage of prediction intervals: the percentage of replication studies whose reported point estimate fell within its corresponding prediction interval. Nominal target is 95%.
(7) Mean width of prediction intervals: the mean difference between the upper and lower bounds of the prediction intervals across all replication studies.
Simulation Study
Proposition 2 characterizes the asymptotic coverage of the KDE-based prediction intervals, but several practically important questions can be addressed only through simulation. First, the proposition provides no guarantee about finite-sample performance when the meta-analysis includes few studies. Second, the IPPW-corrected models use a plug-in heterogeneity estimate from the corresponding uncorrected model, which under publication bias converges to some ; Proposition 2 implies that the resulting coverage error need not vanish asymptotically, but its practical magnitude is unknown without empirical evaluation. Third, Proposition 2 relies on transportability conditional on the measured moderators (Assumption (A4)), which is unlikely to hold exactly in practice: the meta-analyzed studies may differ on moderators that are not recorded, and moderator effects may differ at the replication sites. Fourth, no single model is expected to dominate across all metrics, and the tradeoffs among point prediction accuracy, interval calibration, and robustness to publication bias depend on the combination of sample size, heterogeneity, moderator strength, and selection severity in ways that are difficult to characterize analytically. The simulation study in this section addresses these questions by comparing the four forecasting models across scenarios that vary the number of studies in the meta-analysis, the between-study variance, the severity of selection, and the strength of the moderators. To assess robustness to departures from (A4), the data generating process includes unmeasured moderators that are omitted from all fitted models, and moderator effects at the replication sites differ from those in the meta-analyzed literature.
Simulation Design
In each iteration of the simulation study, we generated a meta-analysis dataset subject to publication bias and a separate set of replication studies not subject to selection. In both, the population effects were drawn from a right-skewed distribution around their conditional means (§ 5.1.2), reflecting literatures in which a few large effects coexist with many small ones; results under normally distributed population effects are reported in the Supplementary Material. We fitted the four forecasting models in § 4.3 to the meta-analysis dataset and used each fitted model to forecast the estimate of each replication study, yielding a point forecast and a 95% prediction interval; for the meta-regression models, forecasts were evaluated at each replication study’s measured moderator levels. We then compared these forecasts with the simulated replication estimates using the metrics in § 4.4. § 5.1.1 describes the four design factors varied across scenarios, and § 5.1.2 describes how the moderators, meta-analysis studies, and replication studies were generated.
Simulation scenarios
We generated 81 scenarios from the full factorial combination of four design factors: (a) the number of studies in the meta-analysis, ; (b) the residual variance of the population effects, , corresponding to standard deviations of 0.2, 0.3, and 0.5 on the Cohen’s scale; (c) the severity of publication bias, , where denotes no selection; and (d) the overall moderator strength, (Table 1), where denotes no moderation. All other parameters, including the skewed distribution of the population effects, were held fixed across scenarios. We ran iterations per scenario, which kept the Monte Carlo standard error of coverage below 0.5 percentage points in every scenario [77].
| Moderator | Distribution | Measured? | ||||
| Bernoulli(0.5) | Yes | 1.00 | 0 | 0.20 | 0.40 | |
| Yes | 0.50 | 0 | 0.10 | 0.20 | ||
| Yes | 0.75 | 0 | 0.15 | 0.30 | ||
| Bernoulli(0.5) | No | 0.50 | 0 | 0.10 | 0.20 | |
| No | 0.25 | 0 | 0.05 | 0.10 |
Table 1. Moderators in the simulation and their mean slopes in the meta-analysis studies under each level of the overall moderator strength .
Data generating process
Moderators.
Each study, whether in the meta-analysis or the replication, had five true moderators, generated independently from the distributions in Table 1. All five were predictors of the population effects, but only , , and were measured and included in the meta-regression models; and were unmeasured and omitted from every fitted model, representing moderators that were not recorded in the literature. The mean slope of each moderator was for , where is the overall moderator strength and are fixed scaling constants that allow the moderators to differ in their influence on the population effects. Varying therefore scaled all five slopes together while holding their relative influence fixed, so that differences in performance across levels of reflected overall moderation strength. The scaling constants were chosen for illustration.
Meta-analysis studies.
To reflect variation in moderator effects across studies, each meta-analysis study drew its own slopes,
We then generated the true population effect and the reported point estimate as
with intercept and fixed within-study variance . The residual has mean zero, variance , and skewness . Because the slopes varied across studies and two moderators were omitted from the model fitting procedure, the residual variance of the population effects conditional on the measured moderators exceeded and depended on the moderator levels, with the excess due to the omitted moderators increasing with . The simulation therefore evaluated the forecasting models under a departure from the constant residual variance assumed in (A1), as is likely in practice.
Publication bias.
A study was defined as affirmative if and its two-sided . Affirmative studies were published with probability 1 and nonaffirmative studies with probability . We applied this selection process to a large pool of simulated studies and retained the first published studies as the meta-analysis dataset.
Replication studies.
Replication studies were generated in the same way as the meta-analysis studies, including the same skewed residual distribution, with two differences. First, because replication populations may differ systematically from those in the published literature, each mean slope was shifted by , with , so that replication study drew its slopes from
This shift applied at every level of , including . Second, because replication results are reported regardless of outcome, replication studies were not subject to publication bias. We generated replication studies per iteration.
Estimation Procedure
In each iteration, we fitted the four forecasting models in § 4.3 to the simulated meta-analysis dataset only, using the measured moderators , , and in the meta-regression models, and used each fitted model to forecast the estimate of every replication study. Because the replication studies were simulated with a known within-study variance, , we treated each replication study’s sampling variance as known, as in the application (§ 6.2). The selection ratio used for publication bias correction was set to its true value in each scenario.
MA-U and MR-U.
We fitted an intercept-only meta-analysis and a meta-regression on , , and using robumeta with correlated-effects weights and small-sample corrections[73, 75]. The point forecast for every replication study was the pooled mean under MA-U, and under MR-U, evaluated at replication study ’s moderator levels.
MA-C and MR-C.
We refitted the same two models with the IPPW weights of Equation (3), supplied to robumeta via the userweights argument. Following § 4.2, the heterogeneity estimate in the weights and in the prediction intervals was taken from the corresponding uncorrected model (MA-U or MR-U). The point forecasts are the corrected pooled mean under MA-C and under MR-C.
Prediction intervals.
We constructed prediction intervals using two approaches: KDE-based and standard parametric intervals. For the KDE-based intervals, we computed calibrated estimates of the meta-analysis studies from each fitted model (Equation (1)) and selected the kernel bandwidth from the unweighted calibrated estimates by the Sheather–Jones method [74]. Convolving the kernel density with the replication study’s sampling distribution yields the Gaussian mixture (Equation (4)), with for the uncorrected models and for the corrected models; we evaluated its cumulative distribution on a grid from to in steps of and took the 2.5th and 97.5th percentiles. For the meta-regression models, the interval was computed at a reference moderator level and shifted to each replication study’s moderator level, which is equivalent to calibrating at because the bandwidth and sampling variance do not depend on the moderators.
For comparison, the parametric intervals are , with obtained from the robust variance estimate of the model coefficients and based on Satterthwaite degrees of freedom (for the meta-regression models, the smallest across coefficients) (Appendix § 9.§ 5).
Both interval methods were applied to identical simulated datasets, so that differences between them reflect the interval method alone. For each model, the seven evaluation metrics (§ 4.4) were computed across the replication studies within each iteration and then averaged across the iterations of each scenario. To summarize performance across scenarios, we calculated win rates: in each of the 81 scenarios, we identified the best-performing model on each of the seven metrics, defined as the smallest absolute raw difference; the lowest MAE, RMSE, and mean prediction interval width; the overestimation rate closest to 50%; the coverage closest to 95%; and the highest mean . We then counted the number of scenarios in which each model performed best, separately for the KDE-based and parametric intervals.
Simulation Results
Table 7 summarizes, for each metric, the number of the 81 simulation scenarios in which each model performed best. MR-C led on MAE, RMSE, and mean , whereas MA-C led on raw difference and overestimation rate. For coverage, MA-C led with the KDE-based intervals and MR-U with the parametric intervals.
| Model | Raw Diff | MAE | RMSE | Mean | Overestimation | Coverage KDE | Coverage Param. | PI Width KDE | PI Width Param. |
| MA-U | 14 | 9 | 9 | 4 | 17 | 21 | 14 | 0 | 49 |
| MR-U | 13 | 24 | 25 | 22 | 10 | 6 | 21 | 80 | 31 |
| MA-C | 39 | 22 | 21 | 4 | 22 | 45 | 26 | 0 | 1 |
| MR-C | 15 | 26 | 26 | 51 | 32 | 9 | 20 | 1 | 0 |
Table 7. Win rates across 81 simulation scenarios (under normal distribution of ). Each cell reports the number of scenarios in which a model achieved the best value for the given metric. Best is defined as: lowest MAE, RMSE, and PI width; closest to 50% overestimation and 95% coverage; highest mean .
Publication bias ().
Without publication bias (), the corrected and uncorrected models coincide because IPPW weights equal 1, and the two meta-regression models were more accurate than the two intercept-only models. Overestimation rates were close to the 50% benchmark, and mean was close to 50%. As increased, the uncorrected models increasingly overpredicted the replication results: overestimation rates increased well above 50% to over 70%, raw difference became increasingly negative, mean declined substantially, MAE and RMSE increased. Correcting for publication bias largely removed this directional bias. The overestimation rates of the corrected models stayed below 55%, with MA-C remaining near the benchmark and MR-C rising moderately above it under strong selection; the raw difference of MA-C remained close to zero, whereas that of MR-C drifted moderately below zero. The corrected models’ MAE and RMSE increased only slightly with selection, and their mean declined much less than that of the uncorrected models. As a result, MR-C remained the most accurate model at every level of publication bias.
Correcting for publication bias also improved the coverage of the KDE-based intervals. KDE coverage was slightly below the nominal 95% when and declined for all four models as increased, but much less for the corrected models: under strong publication bias, the uncorrected models fell more than 10 percentage points below the nominal level, whereas MA-C and MR-C remained closer to it. MA-C’s KDE intervals also narrowed less than those of MA-U as increased. In contrast, MR-C and MR-U had nearly identical widths, so MR-C’s coverage advantage came from shifting its intervals toward the replication estimates, correcting the upward bias induced by publication bias.
The parametric intervals behaved differently. Coverage of the uncorrected models declined with , while increased with . MA-C maintained coverage close to the nominal 95% level and MR-C increased above it. The corrected models achieved this by widening their parametric intervals as increased, whereas the parametric intervals of the uncorrected models narrowed.
Between-study heterogeneity ()
Point forecasts from all four models became less accurate as heterogeneity increased: MAE and RMSE increased with , and both KDE and parametric intervals widened accordingly. MR-C remained the most accurate model at every level of , but the accuracy advantage of the meta-regression models over the intercept-only models declined as increased. This pattern was consistent with the measured moderators explaining a smaller share of the variation in population effects as the residual heterogeneity increased. Heterogeneity also amplified the bias of the uncorrected models: their raw difference became more negative and their overestimation rates increased slightly as increased, whereas the raw difference of the corrected models changed little and their overestimation rates remained much closer to 50%.
In contrast, coverage was largely stable across levels. KDE interval coverage remained below the nominal 95% for all four models, but the corrected models stayed consistently above the uncorrected models at every level of . Mean was also stable, with the corrected models consistently above the uncorrected models.
Moderator strength ()
Moderator strength showed a crossover in the accuracy of the point forecasts between the intercept-only and meta-regression models. Without moderation (), MA-C was the most accurate model, because the meta-regression models lost some precision by estimating slopes that were in fact zero. As increased, the MAE and RMSE of the intercept-only models increased substantially, because they ignored variation in the population effects due to moderators, whereas those of the meta-regression models remained nearly flat; at nonzero levels of moderation, MR-C was the most accurate model and had the highest mean . The overestimation rates of the corrected models remained close to 50% at all levels of moderator strength, whereas those of the uncorrected models declined somewhat as moderation increased but remained well above 50%. However, the raw difference of MR-C became increasingly negative with , while that of MA-C remained close to zero.
Moderation also affected interval performance. As increased, the KDE intervals of the intercept-only models widened, yet their KDE coverage declined, whereas the KDE intervals and coverage of the meta-regression models changed little; MA-C had the KDE coverage closest to the nominal level at every level of moderation, followed by MR-C when moderation was present, and all four models remained below the nominal level. Under the parametric method, MR-C stayed above the nominal level at all levels of moderation, whereas MA-C moved slightly below it as moderation increased.
Number of studies in the meta-analysis ()
The number of meta-analyzed studies had a smaller impact than , , or on most metrics, and its impact depended on the model. The accuracy of the point forecasts from the intercept-only models changed little with , whereas the meta-regression models, which estimate more parameters, became more accurate as increased, MR-C most of all. The bias of MR-C also decreased with : its raw difference moved toward zero, and its overestimation rate decreased toward 50%. KDE coverage increased with for all four models, most for the meta-regression models, while KDE interval widths changed little; even at the largest , however, KDE coverage remained below the nominal 95%. In contrast, the parametric intervals of the corrected models and of MR-U narrowed substantially as increased, and their coverage decreased accordingly: MA-C moved from the nominal level to slightly below it, and MR-C moved closer to, but remained above, the nominal level. At small , the parametric intervals of MR-C were therefore conservative but wide.
Summary. Table 7 summarizes, for each metric, the number of the 81 scenarios in which each model performed best. MR-C was the most accurate model, with the lowest MAE and RMSE in 32 scenarios each, and most often had the highest mean (37 scenarios). MA-C most often had a raw difference closest to zero (39 scenarios), an overestimation rate closest to 50% (43 scenarios), and KDE coverage closest to the nominal level (36 scenarios), whereas MR-U most often had parametric coverage closest to the nominal level (35 scenarios). MR-U also produced the narrowest KDE intervals in 55 scenarios but rarely achieved the best KDE coverage (only 7 scenarios), indicating that its narrower intervals reflected undercoverage rather than greater precision. When publication bias is present (), the corrected models performed best in all or nearly all of the 54 scenarios on every point-forecast metric and on KDE coverage, and in 34 of 54 scenarios on parametric coverage. Overall, the results supported correcting for publication bias in the meta-analysis dataset when forecasting replication results.
| Model | Raw Diff | MAE | RMSE | Mean | Overestimation | Coverage KDE | Coverage Param. | PI Width KDE | PI Width Param. |
| MA-U | 16 | 9 | 9 | 9 | 16 | 20 | 12 | 0 | 52 |
| MR-U | 11 | 18 | 19 | 18 | 11 | 7 | 35 | 55 | 29 |
| MA-C | 39 | 22 | 21 | 17 | 43 | 36 | 26 | 0 | 0 |
| MR-C | 15 | 32 | 32 | 37 | 11 | 18 | 8 | 26 | 0 |
Table 2. Win rates across 81 simulation scenarios. Each cell reports the number of scenarios in which a model achieved the best value for the given metric. Best is defined as: lowest MAE, RMSE, and PI width; closest to 50% overestimation and 95% coverage; highest mean .
MA-U: MA, uncorrected; MR-U: MR, uncorrected;
MA-C: MA, IPPW-corrected; MR-C: MR, IPPW-corrected.
PI: prediction interval.
Param.: standard parametric prediction interval assuming a -distribution.

Figure 1. (a) Shared metrics across KDE and parametric methods. (b) Coverage and prediction interval width by PI method (KDE vs. parametric). Marginal performance of four forecasting models across simulation parameters. Each panel shows one metric (row) as a function of one design parameter (column), averaging over the remaining parameters.
Normally distributed population effects.
As a benchmark, we repeated the simulation with normally distributed population effects, the setting in which Proposition 2 guarantees nominal asymptotic coverage of the KDE-based intervals; all other settings were unchanged (Appendix § 9.§ 6). The main conclusions were the same as under skewed population effects. MR-C was again the most accurate model on MAE and RMSE and most often had the highest mean , and MA-C again most often had a raw difference closest to zero and KDE coverage closest to the nominal level. Two leads changed with the distribution of the population effects. Under normality, MR-C most often had an overestimation rate closest to 50% and MA-C most often had parametric coverage closest to the nominal level, whereas under skewness these leads shifted to MA-C and MR-U, respectively. MR-C’s lead on mean was also larger under normality, consistent with being derived under a normal model for the population effects.
Application to ManyBabies
Now we re-analyze the ManyBabies 1 and ManyBabies 4 meta-analysis and replication datasets using the four models and compared their performances.
Data Preprocessing
ManyBabies 1
We excluded studies with missing Cohen’s or its variance from the corrected Dunst et al. [44] dataset in [46]. For the replication data, we used the same dataset as ManyBabies-Consortium [3] and Zettersten et al.[46]. This resulted in 104 point estimates in ManyBabies 1 and 51 point estimates in the corresponding meta-analyses. The effect size was Cohen’s of looking time. The measured covariates were mean age (days), test language and experimental method.
Across the 51 point estimates from the meta-analyses, the mean infant age was 144.3 days. Most studies (86.3%) used the infants’ native language, with 7.8% using a non-native language and 5.9% an artificial language. Central fixation was the most common experimental method (66.7%), followed by head-turn preference procedure (19.6%). Among the 104 point estimates from the replication dataset, infants were older on average (285.7 days) and more likely to be tested in a non-native language (55.8% vs. 44.2% native). Central fixation remained the predominant method (66.3%), with a larger share using head-turn preference procedure (33.7%) compared to the meta-analytic samples.
The naive meta-analytic estimate of the 51 point estimates in the meta-analyses was 0.69. The funnel plot is shown in Figure 2(a). The selection model provided no evidence of selection favoring statistically significant results (selection ratio 2.33, 95% CI 0.18 to 4.49; LRT P-value = 0.07). The selection-adjusted pooled estimate (0.92, 95% CI 0.60 to 1.23) was larger than the unadjusted estimate (0.69, 95% CI 0.50 to 0.88), suggesting that the main finding is robust to potential publication bias.

Figure 2. Significance funnel plots of the meta-analysis studies for ManyBabies 1 and ManyBabies 4. Gray line: . Publication bias was assumed to favor significant positive estimates. Gray diamond: the pooled estimate using only the nonaffirmative studies; black diamond: the naive pooled estimate using all studies, not corrected for publication bias. The gap between the two pooled-estimate diamonds was noticeably wider in panel (a) versus a much smaller gap in panel (b), indicating that the meta-analysis studies for ManyBabies 1 had more severe publication bias than those for ManyBabies 4.
ManyBabies 4
We used the dataset from [71] (available here), retaining only studies with the dependent measure of “target selection” and excluding one study [78] that used a between-condition comparison (prosocial vs. antisocial) rather than the standard design. For the replication dataset, we excluded studies with missing values in the number of exclusions due to infants “misperforming” during the choice task. This results in 33 point estimates in ManyBabies 4 and 138 point estimates in the corresponding meta-analyses.
The raw effect size was the odds ratio of infants who selected the more prosocial agent. The measured covariates are mean age (days), scenarios, stimuli, stimuli type, choice object, whether the study was from Hamlin lab (Hamlin lab tended to have larger point estimate than other labs [71]), intent valence, and outcome valence. Odds ratios were converted to Cohen’s scale for analysis [79].
Across the 138 point estimates from the meta-analyses, the mean infant age was 401.3 days, with an average of 3.5 excluded infants per study. The majority of studies used a reach method (85.5%), the help/hinder scenarios (62.3%), live show stimuli (61.6%) with real events (82.6%), and puppets as the choice object (50.0%). Most studies measured opposing intent valence (84.1%) and opposing outcome valence (75.4%), and 34.1% originated from the Hamlin lab. Among the 33 point estimates from ManyBabies 4, infants were younger on average (253.1 vs. 401.3 days) but had a higher number of excluded infants (6.2 vs. 3.5). ManyBabies 4 exclusively used reach methods, help/hinder scenarios, movie stimuli with real events, shapes as choice objects, opposing intent valence, and opposing outcome valence, reflecting a more homogeneous design across most covariates. Only one study was from the Hamlin lab.
Distributions of covariates are presented in Table 3 and Table 4.
| Meta-analysis dataset | Replication dataset | |
| Mean age, mean (sd) | 144.3 (79.4) | 285.7 (91.6) |
| Test language (%) | ||
| Native | 86.3 | 44.2 |
| Non-native | 7.8 | 55.8 |
| Artificial | 5.9 | 0 |
| Experimental method (%) | ||
| Central fixation | 66.7 | 66.3 |
| Head-turn preference procedure | 19.6 | 33.7 |
| Other | 13.7 | 0 |
Table 3. Covariate distributions in ManyBabies 1.
| Meta-analysis dataset | Replication dataset | |
| Mean age, mean (sd) | 401.3 (199.0) | 253.1 (24.8) |
| Number of excluded infants, mean (sd) | 3.5 (4.0) | 6.2 (4.8) |
| Method (%) | ||
| Help | 14.5 | 0 |
| Reach | 85.5 | 100 |
| Scenarios (%) | ||
| Fair/unfair | 19.6 | 0 |
| Give/take | 10.1 | 0 |
| Help/hinder | 62.3 | 100 |
| Other | 8.0 | 0 |
| Stimuli (%) | ||
| Live show | 61.6 | 0 |
| Movies | 38.4 | 100 |
| Stimuli type (%) | ||
| Cartoon | 17.4 | 0 |
| Real events | 82.6 | 100 |
| Choice object (%) | ||
| Experimenters | 21.0 | 0 |
| People | 1.4 | 0 |
| Puppets | 50.0 | 0 |
| Shapes | 27.5 | 100 |
| Hamlin lab (%) | 34.1 | 3 |
| Intent valence (%) | ||
| Opposite (positive/negative) | 84.1 | 100 |
| Positive | 9.4 | 0 |
| Negative | 6.5 | 0 |
| Outcome valence (%) | ||
| Same (positive/positive, neutral/neutral, negative/negative) | 13.0 | 0 |
| Opposite (positive/negative) | 75.4 | 100 |
| Reversed (negative/positive) | 0.7 | 0 |
| Neutral | 10.9 | 0 |
Table 4. Covariate distributions in ManyBabies 4.
The naive meta-analytic estimate of the 138 point estimates in the meta-analysis was 0.33. In Figure 2(b), the funnel plot of the meta-analysis was less asymmetric than that in Figure 2(a). The selection model provided no evidence of selection favoring statistically significant results (selection ratio 1.29, 95% CI 0.02 to 2.56; LRT P-value = 0.062). The selection-adjusted pooled estimate (0.36, 95% CI 0.21 to 0.51) was similar to the unadjusted estimate (0.33, 95% CI 0.26 to 0.39), suggesting that the main finding is robust to potential publication bias.
Estimation Procedure
Because the ManyBabies 1 and ManyBabies 4 replication datasets were already available when we conducted this project, we forecasted the replication estimates retrospectively. The first author was blinded to the replication point estimates but had access to all other variables, including each replication study’s moderator levels and sampling variance. For a not-yet-conducted replication, the sampling variance would instead be projected from the planned sample size; here, we treated it as known.
We followed the estimation procedure of the simulation study (§ 5.2), fitting the four forecasting models to the meta-analysis dataset, constructing KDE-based and parametric prediction intervals, and evaluating the forecasts using the seven metrics in § 4.4, with two differences. First, because the true severity of publication bias was unknown, the corrected models used a selection ratio of , the estimated selection ratio for meta-analyses in Metalab, a database of developmental psychology meta-analyses, where affirmative results were an estimated 4.70 times more likely to be published than non-affirmative results[2]. Second, prediction intervals used each replication study’s own sampling variance, , in place of the common variance used in the simulation. The meta-regression models conditioned on the moderators described in § 6.1.
Results
Prediction intervals for ManyBabies 1 and ManyBabies 4 are shown in Figure 3 and Figure 4. The reported point estimates (black dots in both figures) in ManyBabies 1’s replication dataset were more heterogeneous than those of ManyBabies 4. Table 5 and Table 6 show the seven evaluation metrics for the four forecasting models applied to the ManyBabies 1 and ManyBabies 4 replication datasets.
| Model | Raw diff. | Mean | MAE | RMSE | Over-estimation | Coverage KDE | Coverage Param. | PI Width KDE | PI Width Param. |
| MA–U | −0.36 | 0.53 | 0.42 | 0.51 | 89.4% | 100% | 100% | 2.71 | 2.56 |
| MR–U | −0.41 | 0.44 | 0.61 | 0.71 | 87.5% | 88.5% | 100% | 2.65 | 5.27 |
| MA–C | −0.06 | 0.65 | 0.28 | 0.37 | 52.9% | 99.0% | 100% | 2.32 | 2.77 |
| MR–C | −0.17 | 0.50 | 0.49 | 0.58 | 76.9% | 95.2% | 100% | 2.15 | 4.47 |
Table 5. Performance of four forecasting models on the ManyBabies 1 replication dataset. Point-forecast metrics are identical under the KDE and parametric methods because both use the same point forecasts; only coverage and interval width differ.
MA-U: MA, uncorrected; MR-U: MR, uncorrected; MA-C: MA, IPPW-corrected; MR-C: MR, IPPW-corrected; PI: prediction interval; Param.: standard parametric prediction interval assuming a -distribution for the predictive density.
Table 6: Performance of four forecasting models on the ManyBabies 4 replication dataset. Point-forecast metrics are identical under the KDE and parametric methods because both use the same point forecasts; only coverage and interval width differ.
| Model | Raw diff. | Mean | MAE | RMSE | Over-estimation | Coverage KDE | Coverage Param. | PI Width KDE | PI Width Param. |
| MA–U | −0.30 | 0.36 | 0.82 | 1.07 | 60.6% | 84.8% | 84.8% | 2.68 | 2.79 |
| MR–U | −0.32 | 0.42 | 0.79 | 1.05 | 57.6% | 84.8% | 100% | 2.62 | 6.30 |
| MA–C | −0.24 | 0.36 | 0.81 | 1.05 | 54.5% | 84.8% | 87.9% | 2.67 | 2.83 |
| MR–C | −0.26 | 0.41 | 0.79 | 1.03 | 51.5% | 87.9% | 100% | 2.62 | 9.63 |
Table 6. MA-U: MA, uncorrected; MR-U: MR, uncorrected; MA-C: MA, IPPW-corrected; MR-C: MR, IPPW-corrected; PI: prediction interval; Param.: standard parametric prediction interval assuming a -distribution for the predictive density.

Figure 3. (b) MB1: Prediction intervals under MR-U and MR-C models. KDE-based 95% prediction intervals for ManyBabies 1 replication studies. MA-U: uncorrected intercept-only meta-analysis; MA-C: publication-bias-corrected intercept-only meta-analysis; MR-U: uncorrected meta-regression; MR-C: publication-bias-corrected meta-regression; Black dot: reported point estimate; Vertical orange line: uncorrected meta-analyzed estimate; Vertical blue line: corrected meta-analyzed estimate; Yellow triangle: uncorrected point forecast by meta-regression; Green square: corrected point forecast by meta-regression.

Figure 4. (b) MB4: Prediction intervals under MR-U and MR-C models
Figure 4: KDE-based 95% prediction intervals for ManyBabies 4 replication studies. MA-U: uncorrected intercept-only meta-analysis; MA-C: publication-bias-corrected intercept-only meta-analysis; MR-U: uncorrected meta-regression; MR-C: publication-bias-corrected meta-regression; Black dots=: reported point estimate; Vertical orange line: uncorrected meta-analyzed estimate; Vertical blue line: corrected meta-analyzed estimate; Yellow triangle: uncorrected point forecast by meta-regression; Green square: corrected point forecast by meta-regression.
MB1.
For the intercept-only models, moving from MA-U to MA-C reduced the overestimation rate from 89.4% to 52.9%, close to the 50% benchmark, shrank the raw difference from −0.36 to −0.06 on the Cohen’s scale, reduced the MAE from 0.42 to 0.28 and the RMSE from 0.51 to 0.37, and increased the mean from 0.53 to 0.65. Correction improved the meta-regression models in the same direction: moving from MR-U to MR-C reduced the overestimation rate by 10.6 percentage points (from 87.5% to 76.9%), shrank the raw difference from −0.41 to −0.17, reduced the MAE from 0.61 to 0.49 and the RMSE from 0.71 to 0.58, increased the mean from 0.44 to 0.50, and increased KDE coverage from 88.5% to 95.2%, the nominal level, while narrowing the KDE interval width.
In contrast, moving from meta-analysis to meta-regression did not improve point forecasts in ManyBabies 1. Among the uncorrected models, MR-U slightly reduced the overestimation rate relative to MA-U (87.5% versus 89.4%) but had a higher MAE (0.61 versus 0.42); among the corrected models, MR-C had a higher MAE (0.49 versus 0.28), a higher overestimation rate (76.9% versus 52.9%), and a lower mean (0.50 versus 0.65) than MA-C. The main benefit of meta-regression was in interval calibration: the KDE intervals of the intercept-only models covered all or nearly all replication estimates (100% for MA-U and 99.0% for MA-C), exceeding the nominal level, whereas MR-C attained the nominal coverage (95.2%) with the narrowest KDE interval of the four models (2.15). The parametric intervals of the meta-regression models also attained 100% coverage, but only because they were considerably wider than their KDE intervals.
MB4.
In ManyBabies 4, publication bias correction and meta-regression each improved the forecasts, but both improvements were modest, and the four models differed much less than in ManyBabies 1. Correcting for publication bias reduced the overestimation rate by 6.1 percentage points for both model types, from 60.6% (MA-U) to 54.5% (MA-C) and from 57.6% (MR-U) to 51.5% (MR-C), and shrank the raw difference (−0.30 to −0.24 for the intercept-only models; −0.32 to −0.26 for the meta-regression models). Its effect on point accuracy was small (MAE lower by at most 0.01; RMSE lower by 0.02).
Moving from meta-analysis to meta-regression produced comparably small gains. The meta-regression models had slightly lower MAE (0.79 versus 0.81 to 0.82) and RMSE (1.03 to 1.05 versus 1.05 to 1.07), overestimation rates about 3 percentage points lower, and higher mean (0.41 to 0.42 versus 0.36) than the corresponding intercept-only models. Combining both adjustments, MR-C performed best on most point-forecast metrics, with the lowest MAE and RMSE and an overestimation rate closest to the 50% benchmark (51.5%). However, mean remained well below 0.5 for all four models (0.36 to 0.42), indicating that the replication estimates were more dispersed around the forecasts than the meta-analyses predicted. Because the shared protocol held most moderators constant across replication sites, the meta-regression models could shift their forecasts toward the ManyBabies 4 design but could not capture variation among sites, which likely aincreased from unmeasured factors.
Neither adjustment substantially improved interval calibration. KDE coverage was 84.8% for MA-U, MR-U, and MA-C and 87.9% for MR-C, so correction raised the coverage of the meta-regression model by about 3 percentage points but left all four models 7 to 10 percentage points below the nominal 95%, with similar KDE interval widths. Under the parametric method, correction raised the coverage of the intercept-only model from 84.8% to 87.9%, and the meta-regression models attained 100% coverage, but only because their intervals were much wider than the corresponding KDE intervals (6.30 for MR-U and 9.63 for MR-C, compared with 2.62), making them largely uninformative. The especially wide parametric intervals of MR-C may reflect the interaction of IPPW reweighting with the robust variance estimator, which inflates the estimated uncertainty of the coefficients.
The minimal improvement from publication bias correction in ManyBabies 4 was consistent with the less severe asymmetry as we see in Figure 2(b). The minimal improvement from conditioning on moderators likely reflects the fact that, although the main moderators in ManyBabies 4 were homogenous across replication sites due to the shared protocol [4], the replication point estimates were nevertheless heterogeneous as shown in Figure 4. Unmeasured moderators likely drove the between-study variability in replication outcomes in this specific dataset.

Figure a. MB1: Prediction intervals under MA-U and MA-C models
Discussion

Figure a (PDF p. 29). MB4: Prediction intervals under MA-U and MA-C models
Discussion
Principal findings
We developed and evaluated methods for forecasting replication study results from published meta-analyses, combining meta-regression calibrated estimates, IPPW correction for publication bias, and KDE-based prediction intervals. Taken together, the simulation and the two ManyBabies applications clarified when each component of the approach improves forecasts.
Publication bias correction improved forecasts roughly in proportion to the severity of selection. When selection was strong, as in the simulation with large (severe publication bias), the uncorrected models systematically overpredicted the replication results, and correction largely removed this bias and substantially improved accuracy; when selection was mild, correction still moved the forecasts in the right direction, although the gain was small. Correction also improved the calibration of the prediction intervals when the IPPW weights were applied to the KDE as well as to the model fit. In the simulation, the corrected models’ KDE coverage declined much less with increasing selection than that of the uncorrected models. Under strong selection, however, coverage remained below the nominal level, likely because the plug-in heterogeneity estimate from the uncorrected model was biased downward under selection, and because the meta-analysis and replication studies may differ in their underlying populations.
Meta-regression, in turn, improved forecasts only when two conditions held: the measured moderators explained variation in the population effects, and the replication sites differed on those moderators. The simulation met both conditions whenever moderation was present, and the corrected meta-regression was then the most accurate model. ManyBabies 1 met only the second condition, and meta-regression did not improve its forecasts; ManyBabies 4 met neither, and the gains were slight. In ManyBabies 4, the replication estimates varied considerably more across sites than any model predicted, driven by sources of heterogeneity that none of the measured moderators captured. These results suggest a practical guideline: publication bias correction, applied to both the model and the prediction interval, should be the default when forecasting replication results from meta-analyses, whereas meta-regression is worthwhile only when the meta-analysis provides evidence that the moderators matter and the replication sites vary on them.
Accuracy-calibration tradeoff
No single model performed best on both point accuracy and interval calibration. In the simulation, MR-C most often had the best point accuracy, lowest overestimation rate, and mean , whereas MA-C most often had the raw difference closest to zero and the coverage closest to the nominal level, clearly so for the KDE-based intervals and narrowly for the parametric intervals. ManyBabies 1 showed a similar tradeoff: MA-C gave the most accurate forecasts, but had the full coverage, exceeding the nominal level, whereas MR-C attained the nominal coverage with the narrowest KDE-based interval of the four models, at the cost of a higher MAE and overestimation rate.
The tradeoff also appeared between the two interval methods. With IPPW weights applied to the KDE, the corrected models’ KDE-based intervals were better calibrated than those of the uncorrected models and remained informative, but under strong selection they still fell short of the nominal level. The parametric intervals of the corrected models, especially MR-C, reached or exceeded nominal coverage, but only by becoming considerably wider, to the point of being largely uninformative in ManyBabies 4. Neither method produced intervals that were both fully calibrated and informative under strong selection. MR-U illustrated the opposite failure in the simulation: it produced the narrowest KDE intervals in nearly all scenarios but rarely achieved the best coverage, but intervals are of little value when the point forecasts are biased.
Unexplained heterogeneity and unmeasured moderators
Unexplained heterogeneity and unmeasured moderators
In ManyBabies 4, the improvement from meta-regression and IPPW correction was minimal. The replication dataset exhibited substantial unexplained heterogeneity in point estimates despite a shared protocol that held key moderator levels constant across sites. Because the meta-regression and IPPW-corrected models improve predictions by conditioning on moderator levels and correcting for publication bias, their advantage diminishes when replication-study moderators are homogeneous and residual heterogeneity is driven by unmeasured factors. This pattern highlights a broader tension in applying replication forecasting methods: meta-analysis datasets are typically heterogeneous across study populations and designs, which is precisely the variation that meta-regression exploits, whereas large-scale replication projects deliberately standardize their protocols to isolate the effect of interest. When this standardization compresses moderator variability in the replication dataset, the forecasting models lose their primary source of predictive capability. In this respect, ManyBabies 1, where replication sites varied meaningfully on study-level covariates, provided a more informative demonstration of the four models’ comparative strengths than ManyBabies 4.
This limitation is not unique to our application; it mirrors a broader pattern documented in meta-analyses and replication studies across developmental psychology. Using data from Metalab, Cao et al.[80] unexpectedly found no evidence of age-related growth in infants’ linguistic and cognitive skills, linear or nonlinear, across all 25 datasets examined, contrary to the field’s longstanding assumption that such skills refine with age. They further found that an age-related meta-analytic effect was present in only a subset of studies, identified either by a specific feature of the experimental paradigm (e.g., paradigms depicting “giving and taking” actions to infants) [70] or by particular participant characteristics (e.g., bilingual infants)[81]. This subset-specific pattern indicates that moderation effects are not uniform across studies, challenging the common assumption in developmental psychology that such effects are generalizable.
Cao et al.[80] offered two explanations for this pattern, both relevant to replication forecasting. First, the psychometric properties of many infant paradigms, such as their reliability and validity across ages, are rarely examined[82], so measurement artifacts may distort effect size estimates. Second, studies pooled in a meta-analysis differ in participants and procedures, such as infant age, stimuli, and testing paradigm, largely by the choice of individual laboratories rather than by design, so differences across studies may reflect true heterogeneity, measurement artifacts, or both. Because primary studies are not designed to be pooled, many potentially relevant moderators are neither measured nor reported, which we view as an inherent limitation of meta-analysis. Cao et al. therefore called for more standardized methods and more detailed reporting, which would reduce avoidable heterogeneity and allow moderators to be studied more precisely in future meta-analyses.
Our ManyBabies applications illustrated this same pattern. The meta-analysis datasets underlying both ManyBabies 1 and ManyBabies 4 showed markedly different distributions of covariates across studies (mean age, test language, and experimental method in ManyBabies 1; method, scenario, stimuli, choice object, intent valence, and outcome valence in ManyBabies 4), consistent with the paradigm-driven moderation described by Cao et al.[80]. In the ManyBabies 1 meta-analysis dataset, most studies tested infants in their native language, whereas most replication studies tested infants in a non-native language; this difference could plausibly alter infants’ responses, though the mechanism cannot be confirmed from the available data. More generally, because most experiment-level moderators were fixed at a single level across ManyBabies replication sites, distinguishing true moderators from unmeasured sources of heterogeneity remains difficult.
Future directions
Given the findings and ongoing replication projects, we do not advocate for homogenizing moderators across all replication studies. Doing so is unnecessary for drawing valid conclusions about replicability, and it forecloses the opportunity to characterize real-world heterogeneity, in which study participants exhibit both genuinely heterogeneous responses to the outcome of interest and study-level variation attributable to differences in experimental setting and methodology. Building on this perspective, we outline directions for strengthening the use of meta-analytic results in forecasting replication studies, particularly in settings where replication protocols allow moderator levels to vary in a controlled manner rather than being fully standardized.
Replication projects could adopt designs that deliberately assign different moderator levels across sites. In ManyBabies 4, the shared protocol held key moderators constant, which compressed the moderator variation that meta-regression relies on for conditional prediction. By contrast, a design in which sites vary a prespecified set of moderators (e.g., participant age groups, stimulus types, or experimental methods) under careful data collection would preserve the meta-regression’s predictive leverage while maintaining the measurement quality and internal validity characteristic of large-scale replications. The central methodological question is which moderators to vary and at what levels, a problem that connects replication planning to optimal experimental design.
Given a fitted meta-regression from the existing literature, researchers could allocate moderator configurations across replication sites to maximize the informativeness of the resulting predictions. Sites could be assigned to fill gaps in the moderator space covered by the meta-analysis, targeting the covariate combinations where conditional predictions are most uncertain or where the corrected and uncorrected models diverge most sharply. This would improve the precision of forecasts by meta-regression while avoiding extrapolation beyond the range of the meta-analytic evidence.
Historically, large-scale replication projects have standardized moderator levels across sites so that differences in observed point estimates can be attributed to replication failure rather than methodological variation, simplifying the assessment of whether a finding replicates. However, this simplification is not strictly necessary when appropriate statistical methods are available. The meta-regression and calibrated estimation framework developed in this paper, for example, can generate moderator-conditional predictions that account for design differences across sites, allowing researchers to evaluate replicability even when study protocols vary on key moderators. As such methods continue to develop, replication projects can move toward designs that deliberately vary moderators while still drawing valid conclusions about whether the underlying effect replicates.
Original Publication. The manuscript contains original unpublished work and is not being submitted for publication elsewhere at the same time.
Funding Statement. This research was supported by the National Institutes of Health grants R01 LM013866, UL1TR003142, P30CA124435, and P30DK116074.
Competing Interests. None.
Data Availability Statement. All code and data required to reproduce the simulation study and applied examples are publicly available and documented at https://github.com/einsley1993/rfmb14.
Ethical Standards. The research meets all ethical guidelines, including adherence to the legal requirements of the study country.
Author Contributions. Conceptualization: Y.L; M.B.M. Methodology: Y.L; M.B.M. Data curation: M.C.F; A.W.T; F.M. Data visualisation: Y.L. Writing original draft: Y.L; M.B.M. All authors approved the final submitted draft.
Appendix for ``Forecasting Replication Study Results from Meta-Analyses''
Setup and assumptions
For notational simplicity, we index the published meta-analysis studies by and treat their estimates as independent; the arguments extend to independent clusters of bounded size, as in the main text, in which case the number of clusters and are of the same order. Let denote the conditional mean of the population effects at moderator level , and let indicate an affirmative study. We use the following assumptions.
(A1) Random-effects meta-regression: In the population of all conducted studies, , where are independent and identically distributed across studies, the moderators have bounded support, and the residuals are independent of , with mean zero, positive variance , and an absolutely continuous distribution. No parametric form is assumed for the distribution of .
(A2) Within-study normality: , with the known within-study variances, bounded between zero and infinity.
(A3) Selection model (publication-bias-corrected models only): Affirmative studies are times more likely to be published than nonaffirmative studies, that is, , where is known and . Publication depends on a study’s point estimate only through whether a study is affirmative.
(A4) Transportability and independence: The population effect at replication site , with moderator level , follows the same conditional distribution as the population effects in the meta-analyzed literature, with having the same distribution as ; the replication estimate satisfies and is independent of the meta-analysis data.
For the uncorrected models, we assume instead that (A3) holds with , so that publication does not depend on the study data and the published studies are a random sample of the conducted studies.
Let denote the study weight used in the KDE, so that for the uncorrected models. Under (A3), these weights invert the selection probabilities. Write , so that . If and denote the densities of among published and conducted studies, then , and the published studies are independent and identically distributed draws from . Hence, for any integrable ,
The constant does not depend on the unknown , and when . Because every estimator below normalizes by , the constant cancels. That is, the -weighted empirical distribution of the published studies estimates the distribution of all conducted studies.
Proof of Proposition 1. Calibrated estimates
For fixed and study , denote , , and
(i) Moments of the oracle estimates. By (A1), is independent of with mean zero and variance . By (A2), . Hence
Because is a function of only,
Neither depends on , so by the laws of total expectation and total variance, has mean and variance .
Finally, because are independent and identically distributed across studies by (A1)–(A2), so are the , which are a fixed function of them.
(ii) Feasible versus oracle estimates. Let and , so that the feasible estimate is . Since ,
(a): error in
(b): error in
For term (a), for any . Because and lie in a bounded set and ,
for some constant .
For term (b), is a continuous function of , so implies by the continuous mapping theorem. Because has finite variance, , and hence (b).
(iii) Normal population effects. Suppose the population effects are normal. Because has the distribution of under (A1), this means .
Given , by (A1), and by (A2), , which does not depend on ; hence is independent of . Their sum is therefore normal:
Because this conditional distribution does not depend on , is independent of , and its marginal distribution is also . Adding the constant gives
which is exactly the distribution of .
Finally, by (ii), . Since has the distribution of for every , Slutsky’s theorem implies that converges in distribution to . □
Remarks. Let denote the density of among conducted studies, and the density of . (i) The shrinkage factor removes the extra dispersion that sampling error adds to each estimate [16]. (ii) Without normality, is a shrunken residual plus scaled normal noise. Its distribution matches that of in mean and variance but generally differs in shape, so . The two are close when is approximately normal, or when ; in the latter case, , is negligible relative to , and . (iii) Proposition 1 focuses on describing conducted studies. Among published studies, the distribution of is distorted by publication bias; under (A3), Equation (A.1) implies that the -weighted distribution among published studies recovers .
Proof of Proposition 2. Asymptotic coverage
Notation. For the replication study , denote , , and , and recall from Appendix § 9.§ 3. Let be the Gaussian kernel. The weighted KDE of the population effects and the predictive density of the replication estimate are
which reduces to Equation (4) when . The interval is , where is the -quantile of with CDF . The corresponding targets are
where is the density of under Assumption (A4), with CDF , and has CDF and -quantile .
From published to all conducted studies (IPPW). Under Assumption (A3), for any integrable function of , Equation (A.1) gives
where is the expectation over published studies, E is the expectation over all conducted studies, and . For the uncorrected models, and , so . The value of is not needed, because it either multiplies a zero expectation (Step 1) or cancels in a ratio with in Steps 2–3.
The proof proceeds in 5 steps. Step 1 verifies that condition (a) in Proposition 2 is attainable. Steps 2–3 show in , Step 4 carries this to , and Step 5 converts density convergence into coverage.
Step 1: Root- consistency of .
With including the intercept, solves
so that is the weight in Equation (3), with for the uncorrected models. By (A1)–(A2),
Because depends on , which is estimated from all studies, we work with its limit , a function of study ’s data only; replacing with does not affect the limit of , because .
Applying Equation (A.3) with , and then iterated expectations,
The estimating equation is therefore unbiased at the true , and is root- consistent by standard estimating-equation arguments [12, 72]. The same argument applies to any weights that depend only on the within-study variances and , including the hierarchical and correlated-effects weights of Hedges et al. [72]. In particular, it does not require .
Step 2: Consistency of the KDE based on .
Let denote computed from . The published studies are independent and identically distributed, with . By Equation (A.3) with and ,
Split the error into a stochastic and a bias term:
by the consistency of KDE, which holds for any density when and and extends to bounded weights [83].
Step 3: Replacing with the feasible .
For the first two terms, for any . Because and lie in a bounded set and , both terms are at most .
For the third term, write the shrinkage factor as a function of the heterogeneity, i.e., , so that and . Its derivative is bounded:
where the first inequality follows from and the second from . By the mean value theorem,
Because , as . When , and , so , and hence
Combining the bounds on all three terms, with probability tending to one, for every ,
We now carry this bound over to the KDEs. Shifting a Gaussian kernel changes it by at most a multiple of the shift: by the mean value theorem,
Since both KDEs use the same weights,
so, by the triangle inequality and the bound above,
By Equation (A.3) and the law of large numbers, and , so the weighted average of converges to and is .
By conditions (a) and (b) in Proposition 2, the right-hand side is therefore
by condition (c) in Proposition 2.
Combining with Step 2,
Step 4: Convergence of the predictive density.
By Young’s inequality, for integrable and . Since convolution is linear and ,
where the limit follows from Step 3. In words, adding the replication study’s sampling noise to both densities does not increase the distance between them.
Step 5: Quantiles and coverage.
Let and be any two densities with CDFs and . Because , we have , so
Taking the supremum over ,
By Equation (A.4) and Step 4, . Because is a convolution with a normal density, it is strictly positive, so is continuous and strictly increasing. Hence, for each and any ,
By the uniform convergence of , the same strict inequalities hold for with probability tending to one, and then . Thus ([84], Lemma 21.2).
By Assumption (A4), has CDF , which is continuous because is a density, and is independent of the meta-analysis data, hence of . Conditioning on the quantiles,
By the continuous mapping theorem, ; since is bounded by 1, the bounded convergence theorem gives
Since ,
Each bracket is at most in absolute value. By Equation (A.4), and by Young’s inequality as in Step 4 (since ),
If , then is the oracle estimate, and if the population effects are normal, its density equals by Proposition 1, so . □
Remarks. (i) Misspecified shrinkage. If , then even if the population effects are normal,
so and in general. For the uncorrected models with , is consistent by standard method-of-moments theory [72], so .
(ii) Uncorrected models under publication bias. If but , Equation (A.3) no longer holds: the estimating equation in Step 1 is biased for , and the KDE in Step 2 targets the distribution of calibrated estimates among published studies rather than all conducted studies. Proposition 2 then provides no guarantee.
(iii) Bandwidth. Condition (c) is slightly stronger than the usual . It ensures that the error from estimating the calibration parameters, which Step 3 magnifies by , still vanishes. It holds for bandwidths of the form , such as the Sheather–Jones bandwidth [74], because then . In practice, is selected from the unweighted calibrated estimates and is therefore dependent on data. Our proof treats as a fixed sequence. We expect the result to hold also for a data-dependent bandwidth that behaves like such a sequence in large samples, as the Sheather–Jones bandwidth does.
Standard parametric prediction interval versus KDE-based prediction interval
Standard parametric interval. The point forecast is for the meta-regression models and the pooled estimate for the intercept-only models, with the IPPW coefficients for the corrected models. The total prediction variance combines three components: (1) the estimated between-study variance (residual heterogeneity for the meta-regression models), taken from the corresponding uncorrected model (MA-U for the intercept-only models and MR-U for the meta-regression models); (2) the squared robust standard error of the point forecast, ; and (3) the within-study sampling variance of the replication study. The prediction interval is
where is the quantile of the -distribution with Satterthwaite degrees of freedom, which account for the small-sample properties of the robust variance estimator [73]. For the meta-regression models, we used the smallest degrees of freedom among the coefficients as a conservative approximation. This interval assumes that the population effects are normally distributed around the forecast.
KDE-based interval. We first computed the calibrated estimates of the meta-analysis studies by adding each study’s shrunken residual to the forecast at the replication study’s moderator level,
where is the meta-analysis study’s own sampling variance. We then estimated the distribution of the population effects by a Gaussian KDE of these calibrated estimates, weighted by (Appendix § 9.§ 2; for the uncorrected models), with bandwidth selected by the Sheather–Jones method from the unweighted calibrated estimates. Convolving this density with the replication study’s sampling distribution yields the predictive density in Equation (A.2), a weighted mixture of normal densities. We evaluated its CDF on a grid with step size 0.01 (from to 3 in the simulation and from to 5 in the ManyBabies applications) and took the and quantiles as the prediction interval. Unlike the parametric interval, the KDE-based interval does not impose a normal shape on the estimated distribution of population effects.
Simulation results under normal distribution of
MA-U: MA, uncorrected; MR-U: MR, uncorrected;
MA-C: MA, IPPW-corrected; MR-C: MR, IPPW-corrected.
PI: prediction interval.
Param.: standard parametric prediction interval assuming a -distribution.

(a) Shared metrics across KDE and parametric methods.

(b) Coverage and prediction interval width by PI method (KDE vs. parametric).
Figure 5. Marginal performance of four forecasting models across simulation parameters (under normal distribution of ). Each panel shows one metric (row) as a function of one design parameter (column), averaging over the remaining parameters.
References
- [1]Kvarven A, Strømland E, and Johannesson M. Comparing meta-analyses and preregistered multiple-laboratory replication projects. Nature Human Behaviour 2020;4:423–34.DOI
- [2]Mathur MB and VanderWeele TJ. Estimating publication bias in meta-analyses of peer-reviewed studies: A meta-meta-analysis across disciplines and journal tiers. Research Synthesis Methods 2021;12:176–91.
- [3]ManyBabies-Consortium. Quantifying sources of variability in infancy research using the infant-directed-speech preference. Advances in Methods and Practices in Psychological Science 2020;3:24–52.
- [4]Lucca K, Yuen F, Wang Y, et al. Infants’ Social Evaluation of Helpers and Hinderers: A Large-Scale, Multi-Lab, Coordinated Replication Study. Developmental Science 2025;28:e13581.
- [5]Aczel B, Szaszi B, Clelland HT, et al. Investigating the analytical robustness of the social and behavioural sciences. Nature 2026;652:135–42.
- [6]Miske O, Abatayo AL, Daley M, et al. Investigating the reproducibility of the social and behavioural sciences. Nature 2026;652:126–34.
- [7]Tyner AH, Abatayo AL, Daley M, et al. Investigating the replicability of the social and behavioural sciences. Nature 2026;652:143–50.
- [8]Simons DJ, Holcombe AO, and Spellman BA. An introduction to registered replication reports at perspectives on psychological science. Perspectives on Psychological Science 2014;9:552–5.DOI
- [9]Ebersole CR, Atherton OE, Belanger AL, et al. Many Labs 3: Evaluating participant pool quality across the academic semester via replication. Journal of Experimental Social Psychology 2016;67:68–82.
- [10]Klein R, Ratliff K, Vianello M, et al. Data from investigating variation in replicability: A "many labs" replication project. Journal of Open Psychology Data 2014;2.DOI
- [11]Frank MC, Bergelson E, Bergmann C, et al. A collaborative approach to infant research: Promoting reproducibility, best practices, and theory-building. Infancy 2017;22:421–35.
- [12]Mathur MB and VanderWeele TJ. Sensitivity analysis for publication bias in meta-analyses. Journal of the Royal Statistical Society Series C: Applied Statistics 2020;69:1091–119.
- [13]Sutton AJ, Abrams KR, Jones DR, Sheldon TA, and Song F. Methods for meta-analysis in medical research. Vol. 348. Wiley Chichester, 2000.
- [14]Mathur MB and VanderWeele TJ. New metrics for meta-analyses of heterogeneous effects. Statistics in Medicine 2019;38:1336–42.
- [15]Mathur MB and VanderWeele TJ. Meta-regression methods to characterize evidence strength using meaningful-effect percentages conditional on study characteristics. Research synthesis methods 2021;12:731–49.
- [16]Wang CC and Lee WC. A simple method to estimate prediction intervals and predictive distributions: summarizing meta-analyses beyond means and confidence intervals. Research Synthesis Methods 2019;10:255–66.DOI
- [17]Burr D and Doss H. A Bayesian semiparametric model for random-effects meta-analysis. Journal of the American Statistical Association 2005;100:242–51.DOI
- [18]Ohlssen DI, Sharples LD, and Spiegelhalter DJ. Flexible random-effects models using Bayesian semi-parametric models: applications to institutional comparisons. Statistics in medicine 2007;26:2088–112.pubmed.ncbi.nlm.nih.gov/16906554
- [19]Lee KJ and Thompson SG. Flexible parametric models for random-effects distributions. Statistics in medicine 2008;27:418–34.pubmed.ncbi.nlm.nih.gov/17477434
- [20]JPT H. A re-evaluation of random-effects meta-analysis. JR Stat. Soc. Ser. A Stat. Soc. 2009;172:137–59.
- [21]Graham PL and Moran JL. Robust meta-analytic conclusions mandate the provision of prediction intervals in meta-analysis summaries. Journal of clinical epidemiology 2012;65:503–10.DOI
- [22]Kontopantelis E and Reeves D. Performance of statistical methods for meta-analysis when true study effects are non-normally distributed: a simulation study. Statistical methods in medical research 2012;21:409–26.
- [23]IntHout J, Ioannidis JP, Rovers MM, and Goeman JJ. Plea for routinely presenting prediction intervals in meta-analysis. BMJ open 2016;6:e010247.DOI
- [24]Yamaguchi Y, Maruo K, Bartlett C, and Riley RD. A random effects meta-analysis model with Box-Cox transformation. BMC medical research methodology 2017;17:109.DOI
- [25]Held L, Micheloud C, Pawel S, Hofmann F, and Gerber F. Design and analysis of replication studies with ReplicationSuccess. URL https://crsuzh.github.io/ReplicationSuccess/index.html 2022.DOI
- [26]Held L, Pawel S, and Micheloud C. The assessment of replicability using the sum of p-values. Royal Society Open Science 2024;11.arxiv.org/abs/2401.13615
- [27]Cochran WG. The combination of estimates from different experiments. Biometrics 1954;10:101–29.DOI
- [28]Held L. A new standard for the analysis and design of replication studies. Journal of the Royal Statistical Society Series A: Statistics in Society 2020;183:431–48.DOI
- [29]Edgington ES. An additive method for combining probability values from independent experiments. The Journal of Psychology 1972;80:351–63.DOI
- [30]Edgington ES. A normal curve method for combining probability values from independent experiments. The Journal of psychology 1972;82:85–9.DOI
- [31]Held L, Hofmann F, and Pawel S. A comparison of combined p-value functions for meta-analysis. Research Synthesis Methods 2025;16:758–85.DOI
- [32]Mathur MB and VanderWeele TJ. New statistical metrics for multisite replication projects. Journal of the Royal Statistical Society Series A: Statistics in Society 2020;183:1145–66.
- [33]Fernald A. Intonation and communicative intent in mothers’ speech to infants: Is the melody the message? Child development 1989:1497–510.pubmed.ncbi.nlm.nih.gov/2612255
- [34]Cooper RP, Abraham J, Berman S, and Staska M. The development of infants’ preference for motherese. Infant behavior and development 1997;20:477–88.DOI
- [35]Cooper RP and Aslin RN. Developmental differences in infant attention to the spectral properties of infant-directed speech. Child development 1994;65:1663–77.
- [36]Fernald A. Four-month-old infants prefer to listen to motherese. Infant behavior and development 1985;8:181–95.DOI
- [37]Hayashi A, Tamekawa Y, and Kiritani S. Developmental change in auditory preferences for speech stimuli in Japanese infants. Journal of Speech, Language, and Hearing Research 2001;44:1189–200.pubmed.ncbi.nlm.nih.gov/11776357
- [38]Kitamura C and Lam C. Age-specific preferences for infant-directed affective intent. Infancy 2009;14:77–100.DOI
- [39]Newman RS and Hussain I. Changes in preference for infant-directed speech in low and moderate noise by 4.5-to 13-month-olds. Infancy 2006;10:61–76.DOI
- [40]Pegg JE, Werker JF, and McLeod PJ. Preference for infant-directed over adult-directed speech: Evidence from 7-week-old infants. Infant behavior and development 1992;15:325–45.DOI
- [41]Santesso DL, Schmidt LA, and Trainor LJ. Frontal brain electrical activity (EEG) and heart rate in response to affective infant-directed (ID) speech in 9-month-old infants. Brain and cognition 2007;65:14–21.pubmed.ncbi.nlm.nih.gov/17659820
- [42]Singh L, Morgan JL, and Best CT. Infants’ listening preferences: Baby talk or happy talk? Infancy 2002;3:365–94.DOI
- [43]Werker JF and McLeod PJ. Infant preference for both male and female infant-directed talk: a developmental study of attentional and affective responsiveness. Canadian Journal of Psychology/Revue canadienne de psychologie 1989;43:230.pubmed.ncbi.nlm.nih.gov/2486497
- [44]Dunst C, Gorman E, and Hamby D. Preference for infant-directed speech in preverbal young children. Center for Early Literacy Learning 2012;5:1–13.
- [45]Byers-Heinlein K, Bergmann C, Davies C, et al. Building a collaborative psychological science: Lessons learned from ManyBabies 1. Canadian Psychology/Psychologie Canadienne 2020;61:349.DOI
- [46]Zettersten M, Cox C, Bergmann C, et al. Evidence for infant-directed speech preference is consistent across large-scale, multi-site replication and meta-analysis. Open Mind 2024;8:439–61.
- [47]Hamlin K, Wynn K, and Bloom P. Social evaluation by preverbal infants. Nature 2007;450:557–9.
- [48]Choi Y and Luo Y. Understanding preferences in infancy. Wiley Interdisciplinary Reviews: Cognitive Science 2023;14:e1643.DOI
- [49]Margoni F, Surian L, and Baillargeon R. The violation-of-expectation paradigm: A conceptual overview. Psychological Review 2024;131:716.DOI
- [50]Lee Y, Yun JE, Kim EY, and Song HJ. The development of infants’ sensitivity to behavioral intentions when inferring others’ social preferences. PLoS One 2015;10:e0135588.DOI
- [51]Bloom P and Wynn K. What develops in moral development. Core knowledge and conceptual change 2016:347–64.DOI
- [52]Buon M, Jacob P, Margules S, et al. Friend or foe? Early social evaluation of human interactions. PloS one 2014;9:e88612.DOI
- [53]Chae JJK and Song Hj. Negativity bias in infants’ expectations about agents’ dispositions. British Journal of Developmental Psychology 2018;36:620–33.DOI
- [54]Hamlin K, Wynn K, and Bloom P. Three-month-olds show a negativity bias in their social evaluations. Developmental science 2010;13:923–9.DOI
- [55]Hamlin K and Wynn K. Young infants prefer prosocial to antisocial others. Cognitive development 2011;26:30–9.pubmed.ncbi.nlm.nih.gov/21499550
- [56]Scola C, Holvoet C, Arciszewski T, and Picard D. Further evidence for infants’ preference for prosocial over antisocial behaviors. Infancy 2015;20:684–92.DOI
- [57]Van de Vondervoort JW and Hamlin K. Evidence for intuitive morality: Preverbal infants make sociomoral evaluations. Child Development Perspectives 2016;10:143–8.
- [58]Holvoet C, Scola C, Arciszewski T, and Picard D. Infants’ preference for prosocial behaviors: A literature review. Infant Behavior and Development 2016;45:125–39.DOI
- [59]Loheide-Niesmann L, Lijster J de, Hall R, Bakel H van, and Cima M. Toddlers’ preference for prosocial versus antisocial agents: No associations with empathy or attachment security. Social Development 2021;30:410–27.
- [60]Shimizu Y, Senzaki S, and Cowell JM. Cultural similarities and differences in the development of sociomoral judgments: An eye-tracking study. Cognitive development 2021;57:100974.DOI
- [61]Scarf D, Imuta K, Colombo M, and Hayne H. Social evaluation or simple association? Simple associations may explain moral reasoning in infants. 2012.DOI
- [62]Schlingloff L, Csibra G, and Tatone D. Do 15-month-old infants prefer helpers? A replication of Hamlin et al.(2007). Royal Society open science 2020;7:191795.
- [63]Salvadori E, Blazsekova T, Volein A, et al. Probing the strength of infants’ preference for helpers over hinderers: Two replication attempts of Hamlin and Wynn (2011). PloS one 2015;10:e0140570.
- [64]Cowell JM and Decety J. Precursors to morality in development as a complex interplay between neural, socioenvironmental, and behavioral facets. Proceedings of the National Academy of Sciences 2015;112:12657–62.DOI
- [65]Nighbor T, Kohn C, Normand M, and Schlinger H. Stability of infants’ preference for prosocial others: Implications for research based on single-choice paradigms. PloS one 2017;12:e0178818.DOI
- [66]Vaporova E and Zmyj N. Social evaluation and imitation of prosocial and antisocial agents in infants, children, and adults. Plos one 2020;15:e0235595.DOI
- [67]Abramson L, Dar M, Te’eni A, and Knafo-Noam A. Preferences for helpers and hinders in 9-and 18-month-old infants. Unpublished raw data 2016.
- [68]Hamlin K. A box show replication attempt. Unpublished raw data 2016.
- [69]Woo B and Hamlin K. Older, but not younger, babies prefer helpers in a cartoon version of the hill paradigm. Manuscript in preparation 2016.
- [70]Margoni F and Surian L. Infants’ evaluation of prosocial and antisocial agents: A meta-analysis. Developmental psychology 2018;54:1445.DOI
- [71]Tan AW. Prosocial evaluation: A new meta-analysis. 2024.
- [72]Hedges LV, Tipton E, and Johnson MC. Robust variance estimation in meta-regression with dependent effect size estimates. Research synthesis methods 2010;1:39–65.DOI
- [73]Tipton E. Small sample adjustments for robust variance estimation with meta-regression. Psychological methods 2015;20:375.DOI
- [74]Sheather SJ and Jones MC. A reliable data-based bandwidth selection method for kernel density estimation. Journal of the Royal Statistical Society Series B: Statistical Methodology 1991;53:683–90.DOI
- [75]Fisher Z and Tipton E. robmeta: An R-package for robust variance estimation in meta-analysis. arXiv preprint arXiv:1503.02220 2015.arxiv.org/abs/1503.02220
- [76]Gneiting T and Raftery AE. Strictly proper scoring rules, prediction, and estimation. Journal of the American statistical Association 2007;102:359–78.
- [77]Morris TP, White IR, and Crowther MJ. Using simulation studies to evaluate statistical methods. Statistics in medicine 2019;38:2074–102.DOI
- [78]Van de Vondervoort JW and Hamlin JK. The early emergence of sociomoral evaluation: Infants prefer prosocial others. Current Opinion in Psychology 2018;20:77–81.
- [79]Borenstein M, Cooper H, Hedges L, and Valentine J. Effect sizes for continuous data. The handbook of research synthesis and meta-analysis 2009;2:221–35.
- [80]Cao A, Lewis M, Tsuji S, Bergmann C, Cristia A, and Frank MC. Estimating Age-Related Change in Infants’ Linguistic and Cognitive Development Using (Meta-) Meta-Analysis. Developmental Science 2025;28:e70028.DOI
- [81]Tsui ASM, Byers-Heinlein K, and Fennell CT. Associative word learning in infancy: A meta-analysis of the switch task. Developmental psychology 2019;55:934.DOI
- [82]Byers-Heinlein K, Bergmann C, and Savalei V. Six solutions for more reliable infant research. Infant and Child Development 2022;31:e2296.
- [83]Devroye L. Nonparametric density estimation. The L_1 View 1985.
- [84]Van der Vaart AW. Asymptotic statistics. Vol. 3. Cambridge university press, 2000.