WorksPhysics and Astronomy
Testing the statistical compatibility of data sets with a common prior
Abstract
The Parameter Goodness-of-fit (PG) test [hep-ph/0304176] offers a simple test statistic to study the compatibility of different data sets within a given theoretical model. In its original formulation it assumes that the data sets are statistically independent. We propose an extension of the PG test for the situation when some parameters of the model are constrained by an auxiliary data set called prior here. As the prior is imposed for all data sets, they are no longer statistically independent. We show that under the Gaussian approximation, the prior-constrained PG test statistic follows a generalised distribution, which can be used to obtain -values in a straight-forward way. In the approximation of a strong prior constraint, the test converges towards the standard PG test for the unconstrained model parameters.
Introduction
A common statistics problem in fundamental physics is to test, whether different data sets are compatible within a given theoretical model. In Ref. [1] a frequentist test has been discussed, called the Parameter Goodness-of-fit (PG) test, originally introduced in the context of sterile neutrino searches [2]. Various compatibility tests or tension metrics are applied in the cosmological context (mostly in a Bayesian framework), see Refs. [3, 4] for reviews or [5, 6, 7, 8, 9] for other examples for cosmological applications. In this work we focus on the PG test in a frequentist framework. This test is based on simple variables and under rather general conditions – similar to the ones required for Wilks’ theorem [10] to apply – the corresponding test statistic follows a distribution with the number of degrees of freedom determined by the number of common parameters of the model used to describe the data sets to be tested for compatibility [1].
A requirement of the applicability of this analytic result for the PG test is the statistical independence of the tested data sets. However in many situations of interest the model used to describe the data may depend on nuisance parameters, which are constrained by a different set of observations or external knowledge called “prior” in the following. The important observation is that in the presence of a common prior the data sets are no longer statistically independent, as in general the common prior leads to correlations. Therefore the standard PG test cannot be applied in this case. Two typical examples are: (i) The data sets may depend on common systematic uncertainties, which are parametrised by nuisance parameters subject to external measurements. For example, the data sets could correspond to different neutrino oscillation experiments of the same type (e.g., reactor experiments, or accelerator neutrino experiments), suffering from the same neutrino flux or detection cross section uncertainties. () In some applications one may want to impose an external constraint to one or more of the model parameters, and ask the question, whether the data sets of interest are compatible under the external constraint.
The aim of this work is to generalise the PG test and allow for the presence of a common prior. We will introduce two versions of a modified test statistic, which allow to test the compatibility of data sets linked by a common prior. In general, the prior-constrained PG statistics will not follow a simple distribution. However, we will show that under the usual Gaussian approximation the modified statistics will be distributed as a certain linear combination of variables, which is a particular case of a so-called generalised distribution [11]. While no closed form exists for the generalised distribution, it has a simple analytic characteristic function, which by numerical integration allows to construct the probability distribution function and the cumulative distribution function of the test statistic. Along with this paper we provide a Python script to calculate the relevant distributions [12]. Furthermore, we can use our analytical results to define the conditions under which we expect the standard PG test to apply approximately (namely in the limit of “strong” prior, to be defined in detail below).
In the context of cosmology, a test similar to the PG is often called DMAP (Difference of log-likelihoods at their Maximum a Posteriori point) [6]. Ref. [13] discusses tension metrics in the case of correlated data sets, which has some overlap with our considerations. In particular, they discuss the DMAP test and find that under certain approximations its distribution is a linear combination of variables, in agreement with our findings. Our work differs from Ref. [13] in that we provide two variants of possible test statistics (one of them being strictly positive), formulate all correlations explicitly in terms of the prior data set, we work in the pure frequentist framework, and we provide simple-to-use recipes for the -value calculation in the Gaussian limit, based on the characteristic function of the test statistic distribution.
The outline of the rest of the paper is as follows. In § 2 we set the notation and define two versions of a test statistics for the prior-constrained parameter goodness-of-fit, denoted by PG and PG. Their analytic distributions under the Gaussian approximation are derived in § 3 and § 4, respectively. In § 5 we consider a simple toy experiment simulation inspired by neutrino oscillation disappearance measurements, which we use to study the PG and PG test statistics by Monte Carlo simulations. We verify that the Monte Carlo distribution agrees with the analytic results in the Gaussian approximation. Furthermore, we provide a comparison of the two test statistics and further discussion of their properties, as well as an illustration of the power of the test to reject a model with tension between data sets (including a prior). We summarize our results in § 6. Throughout the main text we limit the discussion to the compatibility of two data sets plus a prior, while we comment on the multi-data set case in appendix § 8.
Setup and definition of the test statistics
Let us consider two statistically independent data sets and , consisting of , data points respectively. We want to describe these data with a model depending on parameters common for both data sets. We can construct functions as
where for . The parameter goodness-of-fit test [1] provides a test-statistic for testing the compatibility of the two data sets under the hypothesis that the model is true:
where denotes the best fit parameters from the combination of both data sets. Under general conditions, similar to the ones required for Wilks’ theorem to hold, will be distributed, i.e., a distribution with degrees-of-freedom [1]. We will call this limit in the following “Gaussian approximation”.
Now we want to consider the situation where, in addition to the parameters , the model depends also on nuisance parameters , which are constrained by another independent data set called “prior”. We describe the prior by
We want to ask now the question, whether and are consistent under the additional constrain imposed by the prior data set on the nuisance parameters. The nuisance parameters can describe common systematic uncertainties, or an external constraint on some of the model parameters (in which case we will replace so that the total number of parameters remains the same, and that consistently counts the parameters not affected by the prior).
We define individual functions for the two data sets including the prior as
where we ignore and add only , as we are only interested in the constraint on the nuisance parameters imposed by the prior. Note, that now and are no longer statistically independent, because they include the same statistical fluctuations from the prior data set. Therefore, a straight-forward application of the original PG test is not possible, as it applies only for statistically independent data sets.
We are proposing in the following two possible test statistics including the prior constraint, which both are natural generalizations of the PG test statistic. First we define the PG* test statistic by
generalizing the PG version displayed in eq. (3). Unlike the original , this quantity is not strictly positive and can also acquire negative values. Hence, it cannot follow a distribution. Indeed, as we are going to show below, in the Gaussian approximation will follow a generalised distribution. We use the tilde to indicate that this quantity in general is not expected to be distributed.
As an alternative version, we define the PG+ test statistic, inspired by eq. (4):
where and denotes the best fit parameters from the combination of , and the prior . Each bracket of eq. (10) is non-negative and therefore by construction, therefore the ‘+’ symbol in the acronym. As we will show, its distribution will also be a generalised distribution in the Gaussian limit.
Useful identities are
Here, denotes the standard PG test statistic for the consistency of the three independent data sets , , defined by
which follows a distribution with degrees of freedom [1], and , are the PG test statistics for the consistency of the data sets and with the prior, respectively. Hence, the three terms in eq. (11) are individually distributed as , , , respectively. However, it is important to note that they are not statistically independent, as they are constructed from different combinations of the same random data.
Equation (12) shows that and differ just by a prior term. This term appears only once in but twice in , ensuring positivity. As a consequence, PG+ gives somewhat more weight to a tension appearing in the prior, whereas PG* has more weight on the “data” part of the combination. In the following we will discuss the PG* and PG+ test statistics in turn.
The PG* test statistic
Derivation of the distribution for ,
We first provide a derivation of the distribution of for the simplest case of , i.e., one single model parameter and one single nuisance parameter . The derivation makes use of standard methods from the theory of quadratic forms, see e.g., [14] for a text book.
In the Gaussian approximation, the functions can be written in the following way:
Here, the hat denotes the best-fit parameters, i.e., the parameter values at the minimum of the corresponding data set, while and are the covariance matrices of from the data sets and , respectively. They are either predicted by the model, or estimated by
with and . Similarly, is the variance of from the prior. The expressions in eqs. (14) to (16) hold exactly for a model with linear parameter dependence; in general, they are valid sufficiently close to the minima, as long as these are located far enough from the boundaries of the parameter space.
Let us consider the determination of the nuisance parameter from the data sets and without the prior. The marginal variances for from data set , are given by the element of the corresponding covariance matrix:
We can now evaluate the consistency of the data sets and with the prior by considering the standardised discrepancies
Under the null-hypothesis of the model being true, they will be standard normal distributed. Furthermore, one can show that (), and therefore eq. (11) can be written as
The three terms are individually distributed as , , but, as emphasized above, they are not statistically independent.
In order to write in terms of statistically independent quantities, we use the correlation of the discrepancies and . The estimators from , , and are independent, hence we have
and the correlation between and is given by
Since , we have . The limiting values 0 and 1 are obtained for an infinitely strong () or infinitely weak () constraint from the prior , respectively.
We are now in the position to derive the distribution of . Considering the form in eq. (20), we note that the first term, , being distributed as , is the squared norm of a three-dimensional vector of standard-normal random variables:
Now we can take the as basis to express the discrepancies and providing the remaining terms in eq. (20) by expressing and as rank-one projection quadratic forms. Taking into account the correlation eq. (22), we write eq. (20) as
where
and and are unit vectors satisfying
The matrix has eigenvalues , which implies that has eigenvalues . Therefore, upon diagonalizing eq. (24) we obtain for the distribution of the test statistic:
where are three independent standard-normal random variables. Or equivalently,
where are mutually independent variables with 1 degree of freedom. Thus, has a generalized distribution [11], see also [15]. Its distribution depends on the experimental setup only through the single dimensionless parameter , eq. (22). For , the negative eigenvalue implies that can take negative values. Hence it is not, in general, an ordinary random variable. From eq. (28) we see that when taking the expectation value of , the two terms proportional to cancel and we obtain .
Using the independent representation in eq. (28), the characteristic function of is [11, 15]
This characteristic function completely specifies the null distribution of the test statistic; the probability distribution function (PDF) and the cumulative distribution function (CDF) can be calculated in a straight-forward way:
where for the latter we have used the Gil-Pelaez theorem. This expression allows directly to calculate -values for the PG* test.
Limiting cases: weak vs strong priors. From eq. (28) it is clear that, for small , will be approximately distributed as (which holds exactly for ). Note that for it is enough that either or is much larger than , see eq. (22). Hence, if only one of the data sets provides a much weaker constraint on than the prior we expect a distribution close to . This will be referred to as the “strong prior” limit.
Conversely, in the “weak prior” limit , and . In this case, eq. (28) implies that , which does not converge towards a normal but still remains a generalized distribution. Hence, the limit is not equivalent to removing the prior completely (in which case we would recover the normal PG, distributed as ). The reason is that although the limit removes effectively any constraint on from the prior, the best fit value will obtain fluctuations of order , tending to infinity, with the effect that the last term in eq. (16) will still contribute as , irrespective of the value of .
Generalization to an arbitrary number of parameters
The above reasoning can be generalised to any number of model parameters and nuisance parameters . Without providing the full derivation we sketch the main idea below. The expressions for and from eqs. (14) and (15) generalize with being a -vector. In generalisation of eq. (18) we obtain the covariance matrices , for the nuisance parameters from data sets , as the sub-block of the covariance matrices and , whereas denotes now the prior covariance matrix for . We define the two discrepancy vectors
Under the common-parameter null hypothesis, their covariance matrices are
Therefore, we can define standardised discrepancies as
which are vectors of standard normal variables with the correlation matrix
Let be the singular values of . These are the canonical correlations between the two standardized discrepancy vectors and . Equivalently, the squared canonical correlations are the eigenvalues of
and they satisfy . For , eq. (38) reduces to eq. (22). Now we can write eq. (11) as
with the three terms being , , . Similar as above, we consider a standard Gaussian vector of dimension . Within the space of , the and terms in the above sum are rank- orthogonal projection quadratic forms. Consequently, we can write
where and are rank- orthogonal projectors corresponding to the and discrepancy subspaces. Geometrically, the cosines of the principal angles between these two subspaces are the canonical correlations , in generalization of eq. (26). The eigenvalues of are
Therefore, under the common-parameter null hypothesis,
where , , are mutually independent standard-normal variables. Or equivalently,
with and all variables mutually independent. Thus we again obtain that is a generalized random variable. It is an ordinary variable only when all canonical correlations vanish.
We see from eq. (44) that
independent of the prior. Hence, a value of signals a low -value, irrespective of the prior.
From Eq. (44), the characteristic function is [11]
Hence, the task in the general case is to find the correlation matrices , , , calculate the matrix and the canonical correlations . Once they are known, we can obtain the PDF or CDF via eqs. (30) and (31) using the above expression for the characteristic function. Similar as for the one-parameter case, we can define the limit of “strong prior” by (), in which case the PG* becomes the standard PG, distributed as a normal variable with degrees of freedom.
The PG+ test statistic
Let us move now to the PG+ test and derive its distribution under the Gaussian approximation. We work this out here only for the case (with and being a trivial extension); we comment on the general case at the end of this section. According to eq. (12) we have
We proceed in analogy to section 3.1. First we need the distribution of the last term in eq. (49). We start by calculating , the variance of . Define and
Then we have
Next we find
It follows that , and therefore
We see that the last term in eq. (49) is a re-scaled :
with , and we can write
where we have used the discrepancies defined in eq. (19). With eq. (52) we have defined as in eq. (22) and, additionally,
In analogy to eq. (24) we write
and
with are unit vectors satisfying
Diagonalising leads to
where are the eigenvalues of and are mutually independent standard normal variables. Hence, also follows a generalised distribution with the characteristic function
The eigenvalues can be obtained from diagonalising the matrix numerically by choosing an explicit basis for the vectors or as a solution of the cubic equation.1
It can be shown that all , and therefore is always non-negative, as per construction.
Limiting cases: weak vs strong priors. Let us consider again the strong and weak prior limits for this test statistic. In the strong prior limit () we have and all . The cubic equation becomes , which has the solutions , and therefore , as expected for the PG once is fixed. Conversely, in the weak prior limit () we have and all . The cubic equation in this case becomes , which has the solutions , . This implies , which corresponds to the PG of versus without a prior. Hence, the PG+ has the property that both the strong and weak prior limits converge towards a standard distribution.2
From eq. (61) we also see that
where we have used that each projector in eq. (59) has trace 1. This is consistent with the above discussed limiting distributions for strong and weak prior, with interpolating between . For , this generalises to .
Generalization to an arbitrary number of parameters. Finally, let us briefly comment on the case and . In principle, the distribution under the Gaussian approximation can be calculated using similar methods as in section 3.2, which we do not discuss explicitly here (the corresponding distribution is provided numerically in the accompanying code [12]). Let us just remark that also in the general case, in the strong and weak prior limit the PG+ statistics converges towards the standard PG. This follows from the definition in eq. (10) together with eqs. (7) and (8). For a prior much stronger than the data sets, we have , such that
and the terms from the prior cancel. Hence, reduces to the standard PG with fixed to best fit from the prior alone. Therefore, it will follow a distribution. In contrast, for a prior much weaker than the determination from the data sets, we have and , such that
In this case the prior terms cancel as distribution, because the prior terms will be dominated by the fluctuations of , and therefore and fully correlated. Hence, in this limit, reduces to the PG for and without prior, which will be distributed.
Numerical results
Toy Monte-Carlo details
The theoretical distributions derived in the previous sections can be validated against Monte Carlo (MC) data. As an example, let us consider a toy experimental setup inspired by a neutrino oscillation experiment. In particular, we will work in a two-family approximation, where the probability is fully determined by two parameters: ( being the mass of each neutrino eigenstate) and a single mixing angle. In the flavor basis, the survival probability for a neutrino after propagating over a distance from the source, can be simply written as:
where stands for the neutrino energy, and parametrizes the oscillation amplitude (which is a function of the mixing angle). For the purposes of this work it is useful to rewrite the probability in terms of the dimensionless parameter for some fixed reference value , which yields
where . Choosing , and such that implies that in the region of optimal sensitivity to the oscillation probability, which is convenient.
The distribution of signal events as a function of reconstructed neutrino energy can be computed in terms of the total neutrino flux , the detection cross section (, which depends on the neutrino energy) and the oscillation probability, as
where is an overall normalization constant (which depends, for example, on the number of targets for the neutrino interaction inside the detector and its geometrical acceptance) and for simplicity we have neglected energy-dependent detection efficiencies or smearing effects. The physics observable is the binned number of events as a function of neutrino energy, , where refers to the bin index, which is obtained by integrating eq. (69) within the limits of each energy bin. In our simulations, we specifically consider a reactor neutrino flux (taken from Refs. [16, 17]) and an inverse beta-decay detection cross section (taken from Ref. [18]). The value of is chosen such that we obtain hundreds of events (even after oscillations) for all bins, so the Gaussian approximation applies. The main parameters used for the configuration of our toy experiments and , as well as the total event rates within the energy window used in the analysis, are summarized in Table 1 for completeness.
| Exp. | (km) | Number of bins | Unoscillated events | ||
| 5.5 | 3 | 4.1 MeV | 0.2 MeV | 4,153 | |
| 3.5 | 3 | 4.1 MeV | 0.2 MeV | 15,384 |
Table 1. Main parameters used for the configuration of the toy experiments in our simulations. The energy bins are taken with equal width , starting at . The total number of events reported has been obtained without oscillations (that is, for ). No backgrounds or systematic uncertainties are considered in our analysis.
Given the values of and in Table 1, oscillations are maximized around . Thus, is a natural choice, and unless otherwise noted we will take as our true value. Furthermore, in order to ensure that the value of always remains away from the physical boundaries at we set its true value at .
Once we have defined our physical observables, we may consider two toy experiments , each of aiming to determine the oscillation parameters from their observed data . Within the Gaussian approximation, we may define a binned function for each of them as
where is the statistical error in each bin. The covariance matrices for experiments and (defined in Table 1), computed at the true input values for and , read:
The prior constraint is implemented in our simulation through the addition of quadratic pull terms, as in eq. (16):
In order to derive the expected distribution for our toy experiments, we need to simulate a large sample of pseudo-data in presence of statistical fluctuations: these will be used as in eq. (70). In doing so, we first pick a set of “true” oscillation parameters and nuisance parameters and compute the predicted event rates . Then, random fluctuations are generated around the predicted number of events in each bin assuming a Gaussian distribution with width . In full analogy, a similar procedure should be followed for the priors since these are also obtained from data (and are therefore subject to statistical fluctuations). After picking a set of true values for each prior , for each pseudo-experiment a random number is generated around using a Gaussian distribution with a width according to the prior experimental uncertainty. The resulting value, , will be used as central value for the pull term in eq. (72). As outlined in previous sections, the resulting distributions will depend on the number of nuisance parameters included in the fit. Therefore, for our simulations we have considered the following two scenarios:
We first assume that there is a prior measurement only on , which now acts as a nuisance parameter in our fit. Hence, , in this case, with and , we only impose from eq. (72) and have .
Second, we consider a case with two prior measurements imposing both terms from eq. (72), that is, , , with . In this case is a diagonal matrix with entries and , i.e., we assume the prior measurements to be uncorrelated.
Results for the null hypothesis
We present now the numerical results of our simulations, comparing them to the theoretical distributions derived for the test statistics in § 3 and § 4. Figure 1 shows our results for the test statistic, where we compare the complementary CDF calculated from simulation (thick solid lines) to the theoretical distributions in eq. (31) (thin dashed lines). The left panel shows our results for , (that is, when only a prior on is imposed) while the right panel shows the corresponding result for , (when priors are imposed on both and ). In both panels, the different lines illustrate how the results change as a function of the priors, between the strong prior limit (, light blue lines) and weak prior limit (, dark blue lines).
Figure 1. Complementary cumulative distribution function of the test statistic. Thick solid lines show the numerical results obtained for pseudo-experiments, while thin dashed lines show the theoretical distributions obtained in § 3. Dash-dotted gray lines show the distributions obtained for a with one and two degrees of freedom. The left panel corresponds to the case , , and we show curves for different values of the correlation parameter defined in eq. (22). The right panel corresponds to the case , , and we give the values for the prior variances and in the legend, see eq. (72).
In all cases the agreement between the numerical results and the theoretical distributions is excellent, with mild differences that can be attributed to the limited statistics of the simulated sample of pseudo-data. We also see that, for all lines, the random variable corresponding to the admits negative values (as expected). As can be seen from the figure, for the distribution remains reasonably close to a , whereas for $\rho_{AB} \gtrsim 0.2 relevant deviations appear. Figure 1 also shows how in the weak prior limit the obtained distributions do not converge towards a normal ; instead, they remain a generalized distribution, as anticipated in § 3.
Our results for the PG test statistic are shown in Figure 2, where the complementary CDF calculated from our simulation (thick solid lines) is compared to the theoretical distributions obtained in § 4 (thin dashed lines), for , (left panel) and , (right panel). Again in this case, the agreement between our numerical results and the theoretical distributions is excellent. Darker/lighter lines show how the results change as we approach the weak/strong prior limits, respectively. As discussed in § 4, for , in the strong prior limit () the distribution matches a with one degree of freedom. However, the obtained distributions start to show large deviations from the limit even for small . This contrasts with the behaviour of the PG test, where a remains a relatively good approximation even for as large as 0.2. Figure 2 also shows that for intermediate values of the tail of the distribution can go above that of a ; however, as the strength of the prior approaches the weak limit, the distribution converges smoothly to a , as anticipated in § 4.
Figure 2. Complementary cumulative distribution function of the PG test statistic. Thick solid lines show the numerical results obtained for pseudo-experiments, while thin dashed lines show the theoretical distributions obtained in § 4. Dash-dotted gray lines show the distributions obtained for a with one and two degrees of freedom. The left panel corresponds to the case , , and we show curves for different values of the correlation parameter defined in eq. (22). The right panel corresponds to the case , , and we give the values for the prior variances and in the legend, see eq. (72).
Note that for the case shown in the right panels of Figure 1 and Figure 2, the limit of infinitely strong prior would formally lead to a distribution with zero degrees-of-freedom, meaning no randomness. In this case, the prior-constrained PG tests would measure just the consistency of the prior with itself, which by construction is perfect.
Let us mention that our toy MC is on purpose constructed such that we expect the Gaussian approximation to hold (high-statistics data, true parameter values far away from boundaries, and only one minimum in the ). A detailed study of a real-life setup using actual neutrino oscillation data, where some of these properties do not hold, will be presented in Ref. [19].
Comparison between the two test statistics. Since the two test statistics considered here differ on the way the prior is introduced (with a larger relative weight for PG than for PG), a relevant question to ask is whether one of the two test statistics is more sensitive to tension in the fit than the other. In fig. 3 we show the distribution of -values for the two tests from our MC simulation, where we use the shorthand notation () for the -value of PG (PG). Again, the left panel corresponds to , while the right panel shows the results for , as indicated by the labels.
The upper panels in fig. 3 show the -values obtained for the two tests statistics considered, for toy MC realizations. As shown in the upper panels in fig. 3, the points tend to cluster in the region above the diagonal, indicating that under the null-hypothesis it is likely to obtain a larger -value for than for . Analytically, this behaviour can be understood since statistical fluctuations can only make by construction, see eq. (12), which translates into a strict upper bound on . Specifically, using the shorthand notation and , it is easy to see that
where refers to the complementary CDF for the test statistic, and we have by definition. Therefore, the upper bound on can be obtained by the curve
parametrized by . The curve from eq. (74) is shown by the dashed black lines in the upper two panels in Figure 3. As can be seen, the agreement with our numerical results is again excellent. We have verified that in the strong prior limit (for ), the distribution of and values concentrate to points close to the diagonal , as in that limit both the and tests become identical and converge towards the standard PG.
Figure 3. Comparison between the -values obtained for the PG () and PG () test statistics, for two examples with , (left panels) and , (right panels). Upper panels: scatter plot of the -values obtained for the two tests statistics considered, for experimental realizations (red points). The diagonal black line corresponds to , while the dashed black line shows the curve corresponding to eq. (74), see text for details. Lower panels: fraction of points for which as a function of for the same simulations.
In the same figure, the lower panels show the fraction of points where as a function of , for the same data sample. As can be seen, this fraction approaches a constant value around in the region , while for larger values of it drops to zero. We note that the fraction remains at for small values of regardless of the prior strength; however the exact value where the fraction drops depends on the number of priors considered and the values assumed, as illustrated by the two examples shown in Figure 3. The value of where the fraction starts to drop is approximately set by the -value obtained for , where the dashed curve from eq. (74) reaches . In fact, as decreases towards negative values, the upper bound on does not change (since it had already saturated at 1) but keeps increasing, and therefore the fraction of points where drops.
Hence, for not too large -values, we find that under the null-hypothesis it is likely to find a larger -value for than for . As the prior has more weight in the test statistic, this indicates that it is somewhat more likely that fluctuations contribute to the -terms involving the data sets and rather than to the prior term . Comparing the values of and for a given realisation of the data may be used as an additional diagnostic tool to identify the origin of a tension.
Results for alternative hypotheses
Until now we have focused on the computation of the expected distributions under the null hypothesis . It assumes that the model is true, i.e., that the two experiments as well as the prior measurement are random realisations of the data for the same true values of the underlying model parameters. However, it is also worth considering the distribution of the test statistics under an alternative hypothesis .
The theoretical distribution for an alternative hypothesis can be derived following the same procedure as in sections 3 and 4 bearing in mind that, in this case, the variables will no longer be normally distributed around zero but around some non-vanishing value instead (which eventually depends on the alternative hypothesis chosen). For example, if we consider a hypothesis where an experiment and the prior observe data that are generated with different true values of the nuisance parameter, and , then will follow a non-central chi-squared distribution with one d.o.f. and non-centrality parameter . Likewise, it can be shown that will follow a non-central distribution with 3 d.o.f., where the statistically independent variables can be obtained through the same change of basis as in § 3 and § 4. Therefore, both and can be written as a sum of non-central chi-squared distributions which, by definition, follows a generalized distribution [11, 15]. Our accompanying code [12] provides auxiliary functions to compute the non-centrality parameters (as well as the resulting distribution) for any choice of true values for the underlying model parameters.
As outlined above, the choice of alternative hypothesis is not unique. While a systematic study of this is beyond the scope of this work, we have identified two general cases that are physically motivated:
Case I: both data sets and are based on the same set of true model parameters while the prior measurement is generated from different true values. In this case the tension in the combined fit arises from the mismatch between the prior and the data sets.
Case II: the true value of the prior is in agreement with the one of used to simulate the data, but different true values of are used for experiments and . In this case the prior is consistent with both experiments separately; however, when analysed in combination a tension emerges in the fit.
We are mostly interested in a scenario where there is significant tension between the different data sets and/or the prior. This implies that the best-fit points for the joint fit will be significantly displaced from the minimum obtained for the Asimov data set for each experiment, which could eventually make our results sensitive to non-linearities entering the . Thus, in order to guarantee that our simulations still fall within the linear regime also for the alternative hypotheses, the results obtained in the remainder of this section have been obtained multiplying by 30 the statistics of the experiments with respect to our default setups defined in § 5.1.
Figure 4 shows the interplay of the different data sets in the combined fit for two specific examples satisfying the criteria outlined above. In both panels, the best-fit points for each experiment separately (blue and red dots, for and , respectively) do not show any tension: they are found to be in good agreement with the sensitivity obtained using the Asimov data as well as with the prior. Conversely, a tension is observed for the joint fit (purple dots) in both panels. The distributions of pseudo-experiments for the null and alternative hypotheses are shown in Figure 5 taking case I as an example, for the two test statistics under consideration. We find that the theoretical distributions obtained for the alternative hypothesis (solid blue lines) are in excellent agreement with our numerical results. Note that the results for case II are qualitatively very similar and are therefore not shown here.
Figure 4. Example showing the sensitivity regions at CL (for 2 d.o.f.) for the two experiments under consideration. The star indicates the true values of the parameters used to simulate each experiment, and the solid contours show the sensitivity regions obtained with the Asimov data set for each experiment separately (blue/red lines) and in combination (purple line). The dots indicate the location of the global minima found when pseudo-data for the two experiments are fitted separately (blue and red dots for experiments A and B, respectively) or jointly (purple dots), in all cases imposing the same prior on . Left panel (case I): the best-fit of the prior measurement is set to a different value of , as indicated by the gray band. Right panel (case II): the best-fit of the prior measurement is in agreement with the value of used to simulate the data, but different true values of are used for experiments and .

Figure 5. Example showing the distributions of pseudo-experiments for the alternative hypothesis (, purple histograms), compared to the null hypothesis (, teal histograms). The solid blue curve on top of the purple histograms shows the theoretical distributions, in excellent agreement with our simulation. The vertical purple lines indicate the sensitivity of an average experiment, see text for details. This example corresponds to case I; the results for case II are qualitatively very similar.
Median sensitivity. The tension observed in Figure 4 can be further quantified by the -value corresponding to the median of the distribution of pseudo-experiments for the alternative hypothesis. This defines the sensitivity of an average experiment or median sensitivity. In Figure 5, in each panel, the vertical purple lines indicates the -value corresponding to the median sensitivity. For our alternative hypotheses we find:
where the reported number of standard deviations corresponds to a one-sided Gaussian. We see that for case I we obtain a somewhat larger tension for than for . This is expected, since in this case the tension emerges from the prior itself, which has a larger relative weight for than for . Conversely, for case II the difference in -values is reduced, with a slightly lower tension for than for . This is again expected since the tension comes from the data instead of the prior and, moreover, the relative weight of the data is lower for .
Overall, the example provided above illustrates how a comparison between the -values obtained for and may aid in determining the source of a possible tension in the data, as previously mentioned in § 5.2.
Power of the test. Until now we have focused on the computation of the type-I error rate associated to a given test statistic, that is, the probability to reject the null hypothesis even though it is true (i.e., the -value). However, the type-II error rate is also relevant: it is defined as the probability of not rejecting although it is false. In order to compute it, we first need to pick a (true) alternative hypothesis and obtain its corresponding distribution for the test statistic. Next, we choose a -value and determine the associated value of the test statistic under . The corresponding type-II error can be obtained as the fraction of experiments within which would yield a lower value. Then, is denoted as the power of the test.
Figure 6 shows the type-II error rate of the test statistics as a function of the -value for the two cases considered here. By definition, the -value corresponding to (indicated by a horizontal dotted line) yields the median sensitivity in eqs. (75) and (76). However, the relative difference in the tension reported by the two test statistics is also present for other values of : while for case I (left panel) the two curves are more separated with PG having slightly more power (i.e., lower ), for case II (right panel) the two test statistics behave in a very similar manner.3 In the same figure we also show how the curves change as a function of . We find that the relative comparison between the two sets of curves generally holds as is varied, except when the weak prior limit is approached. This is expected since, in the weak prior limit, the tension from the prior disappears and therefore regardless of the -value (that is, the discrimination between the two hypothesis is no longer possible for the particular examples chosen here). We also see that in the strong prior limit the two lines become very similar even for case I, since in this limit both tests converge to the standard PG.
Figure 6. Type-II error rate of the test as a function of the -value, for the two test statistics under consideration. Left and right panels show the results for alternative hypotheses matching cases I and II, respectively. In each panel, each pair of lines has been obtained for a different prior, between (lightest) to (darkest).
Summary
In this work we have proposed an extension of the Parameter Goodness-of-fit (PG) test, which can be used when some of the parameters of the model are constrained by a prior common to the data sets. In such a case, the data and the prior are no longer statistically independent, so the original PG test cannot be applied. We have proposed two modified versions of the PG test statistic, denoted by and . Both can be used to test the compatibility of data sets subject to a common prior, so we call the corresponding test prior-constrained parameter goodness-of-fit.
The two test statistics differ by a term involving the prior . The is constructed to be strictly non-negative thanks to the additional prior term, whereas can assume also negative values and gives somewhat less weight to the prior. Both test statistics converge towards the standard PG test in the case of “strong prior”, i.e., when the constraint on the parameters from the prior is much stronger than the one from the data sets, with a number of degrees of freedom that is determined by the number of parameters unconstrained by the prior.
In the Gaussian approximation the and test statistics are distributed according to a so-called generalised distribution, which can be easily calculated via its characteristic function. The distribution is determined just by the parameter covariance matrices of the data sets and the prior. Once the distribution is known, -values can be calculated in a straightforward manner. While the distribution of the test statistics depend on the prior strength, the expectation value of the test statistic is given by the numbers of parameters unconstrained by the prior, , independently of the prior strength, see eq. (45). Hence a value of the test statistic much larger than indicates a low -value. For the PG test statistic the expectation value depends on the prior strength and varies between for a strong prior till for a weak prior, where () is the number of parameters unconstrained (constrained) by the prior. We provide an accompanying Python code [12] which allows to calculate the corresponding distribution functions and -values for any number of model and nuisance parameters, given a set of covariance matrices.
Our results have been validated through Monte Carlo simulation of two toy experiments inspired by neutrino oscillation facilities. The simulated setup is close to the Gaussian limit (large statistics, no multiple minima, and away from physical boundaries). The simulations show excellent agreement with the analytic expressions for the distributions of the PG and PG test statistics. We find that, under the null hypothesis and for not too large -values, it is more likely to obtain a larger -value for PG than for PG. This indicates that it is more likely that statistical fluctuations contribute to the of individual experiments than to the prior term. A comparison between both -values may be used as an additional tool to diagnose the origin of a tension. Finally, we have also computed the power of the test for an alternative hypothesis, for two physically motivated examples, illustrating that our test statistics are able to detect tensions between data sets in the presence of a common prior. We find that both test statistics have comparable power, with PG performing somewhat better, especially in cases where the tension is mostly driven by the prior.
In summary, we believe that the test statistics proposed here offer a useful generalisation of the PG test for analysing the statistical compatibility of data sets, and extends its range of applicability.
Acknowledgements
We thank Yufeng Li for asking a relevant question about the PG test, which triggered this work. We acknowledge the help of ChatGPT and Claude for developing code as well as for some of the mathematical derivations. This project is funded by the European Union’s Horizon Europe research and innovation programme (Marie Skłodowska-Curie Staff Exchange grant agreement 101086085-ASYMMETRY) and by ERDF “A way of making Europe”. It also receives support from the R&D&I project CEX2025-001574-S “Centro de Excelencia Severo Ochoa” (the research presented in this publication falls within the research line “Particle Physics in the Standard Model and Beyond (BSM)” ) and from grants PID2022-142545NB-C21 and PID2025-171019NB-C21, all funded by MICIU/AEI/10.13039/501100011033.
Prior-constrained PG for more than two data sets
Similar as the original PG test, also the prior-constrained PG can be extended in a straightforward way to the case of more than two data sets with a common prior. We consider data sets, with for , and define
where denotes the best fit parameters from the combination of all data sets and the prior . Notice that, as already seen in eq. (12), and differ just by a prior term.
We outline here the main idea for PG* for the simplest case of , and leave the generalization to the multi-parameter case as well as for PG+ to the reader. In the Gaussian approximation we can derive the distribution function of this statistic in analogy to section 3.1. Following eq. (20), we write
where is the standard PG test statistic for all data sets plus the prior , and
Under the null-hypothesis of the model being true, is distributed as with and the are standard-normal variables with the correlation matrix
for .
Hence, we write eq. (A.3) as
where is a vector of independent standard normal variables and the matrix is given by
with are unit vectors satisfying and the matrix . We note that the non-zero eigenvalues of are the same as the eigenvalues of . Let us denote these eigenvalues by , . Then the eigenvalues of are
for . Therefore, upon diagonalizing eq. (A.6) we obtain for the distribution of the test statistic:
where () are mutually independent standard-normal () variables. The characteristic function is obtained as
Hence, in this case the distribution of the test statistic is obtained by calculating the eigenvalues of the correlation matrix eq. (A.5).
Strong prior limit. For , eq. (A.5) gives and (). In this limit, eq. (A.8) for the eigenvalues gives times the eigenvalue and all remaining eigenvalues are zero. Therefore, eq. (A.9) gives a normal distribution with degrees of freedom, which is the result of the normal PG for data sets with one common parameter .
References
- [1]M. Maltoni and T. Schwetz, Testing the Statistical Compatibility of Independent Data Sets, Phys. Rev. D 68 (2003) 033020, [hep-ph/0304176].DOI
- [2]M. Maltoni, T. Schwetz, M. A. Tortola, and J. W. F. Valle, Ruling Out Four Neutrino Oscillation Interpretations of the LSND Anomaly?, Nucl. Phys. B 643 (2002) 321–338, [hep-ph/0207157].DOI
- [3]DES, P. Lemos et al., Assessing tension metrics with dark energy survey and Planck data, Mon. Not. Roy. Astron. Soc. 505 (2021), no. 4 6179–6194, [2012.09554].
- [4]M. Leizerovich, S. J. Landau, and C. G. Scóccola, Tensions in cosmology: A discussion of statistical tools to determine inconsistencies, Phys. Lett. B 855 (2024) 138841, [2312.08542].
- [5]L. Amendola, V. Marra, and M. Quartin, Internal Robustness: Systematic Search for Systematic Bias in Sn Ia Data, Mon. Not. Roy. Astron. Soc. 430 (2013) 1867–1879, [1209.1897].DOI
- [6]M. Raveri and W. Hu, Concordance and Discordance in Cosmology, Phys. Rev. D 99 (2019), no. 4 043506, [1806.04649].DOI
- [7]W. Lin and M. Ishak, A Bayesian Interpretation of Inconsistency Measures in Cosmology, JCAP 05 (2021) 009, [1909.10991].DOI
- [8]S. Gariazzo, O. Mena, and T. Schwetz, Quantifying the Tension Between Cosmological and Terrestrial Constraints on Neutrino Masses, Phys. Dark Univ. 40 (2023) 101226, [2302.14159].
- [9]P. R. Mello, M. Quartin, B. M. Schaefer, and B. Schosser, On the full non-Gaussian Surprise statistic and the cosmological concordance between DESI, SDSS and Pantheon+, Open J. Astrophys. 8 (8, 2024) 2025, [2408.08385].DOI
- [10]S. Wilks, The Large-Sample Distribution of the Likelihood Ratio for Testing Composite Hypotheses, Annals Math. Statist. 9 (1938), no. 1 60–62.DOI
- [11]“Generalized chi-squared distribution.” Wikipedia.
- [12]P. Coloma, M. Maltoni, and T. Schwetz. Code provided in arXiv supplementary material.
- [13]M. Raveri, G. Zacharegkas, and W. Hu, Quantifying concordance of correlated cosmological data sets, Phys. Rev. D 101 (2020), no. 10 103527, [1912.04880].DOI
- [14]A. M. Mathai and S. B. Provost, Quadratic Forms in Random Variables: Theory and Applications. Marcel Dekker, 1992.DOI
- [15]J. P. Imhof, Computing the distribution of quadratic forms in normal variables, Biometrika 48 (1961), no. 3/4 419–426.
- [16]H. Murayama and A. Pierce, Energy spectra of reactor neutrinos at KamLAND, Phys. Rev. D 65 (2002) 013012, [hep-ph/0012075].DOI
- [17]KamLAND, K. Eguchi et al., First results from KamLAND: Evidence for reactor anti-neutrino disappearance, Phys. Rev. Lett. 90 (2003) 021802, [hep-ex/0212021].DOI
- [18]P. Vogel and J. F. Beacom, Angular distribution of neutron inverse beta decay, anti-neutrino(e) + p —> e+ + n, Phys. Rev. D 60 (1999) 053003, [hep-ph/9903554].DOI
- [19]P. Blanco-Mas, P. Coloma, M. Maltoni, and T. Schwetz. In preparation.




