betaFeedback

WorksPhysics and Astronomy

Testing the statistical compatibility of data sets with a common prior

Abstract

The Parameter Goodness-of-fit (PG) test [hep-ph/0304176] offers a simple test statistic to study the compatibility of different data sets within a given theoretical model. In its original formulation it assumes that the data sets are statistically independent. We propose an extension of the PG test for the situation when some parameters of the model are constrained by an auxiliary data set called prior here. As the prior is imposed for all data sets, they are no longer statistically independent. We show that under the Gaussian approximation, the prior-constrained PG test statistic follows a generalised χ2χ^2 distribution, which can be used to obtain pp-values in a straight-forward way. In the approximation of a strong prior constraint, the test converges towards the standard PG test for the unconstrained model parameters.

Introduction

A common statistics problem in fundamental physics is to test, whether different data sets are compatible within a given theoretical model. In Ref. [1] a frequentist test has been discussed, called the Parameter Goodness-of-fit (PG) test, originally introduced in the context of sterile neutrino searches [2]. Various compatibility tests or tension metrics are applied in the cosmological context (mostly in a Bayesian framework), see Refs. [3, 4] for reviews or [5, 6, 7, 8, 9] for other examples for cosmological applications. In this work we focus on the PG test in a frequentist framework. This test is based on simple χ2\chi^2 variables and under rather general conditions – similar to the ones required for Wilks’ theorem [10] to apply – the corresponding test statistic follows a χ2\chi^2 distribution with the number of degrees of freedom determined by the number of common parameters of the model used to describe the data sets to be tested for compatibility [1].

A requirement of the applicability of this analytic result for the PG test is the statistical independence of the tested data sets. However in many situations of interest the model used to describe the data may depend on nuisance parameters, which are constrained by a different set of observations or external knowledge called “prior” in the following. The important observation is that in the presence of a common prior the data sets are no longer statistically independent, as in general the common prior leads to correlations. Therefore the standard PG test cannot be applied in this case. Two typical examples are: (i) The data sets may depend on common systematic uncertainties, which are parametrised by nuisance parameters subject to external measurements. For example, the data sets could correspond to different neutrino oscillation experiments of the same type (e.g., reactor experiments, or accelerator neutrino experiments), suffering from the same neutrino flux or detection cross section uncertainties. (iiii) In some applications one may want to impose an external constraint to one or more of the model parameters, and ask the question, whether the data sets of interest are compatible under the external constraint.

The aim of this work is to generalise the PG test and allow for the presence of a common prior. We will introduce two versions of a modified test statistic, which allow to test the compatibility of data sets linked by a common prior. In general, the prior-constrained PG statistics will not follow a simple χ2\chi^2 distribution. However, we will show that under the usual Gaussian approximation the modified statistics will be distributed as a certain linear combination of χ2\chi^2 variables, which is a particular case of a so-called generalised χ2\chi^2 distribution [11]. While no closed form exists for the generalised χ2\chi^2 distribution, it has a simple analytic characteristic function, which by numerical integration allows to construct the probability distribution function and the cumulative distribution function of the test statistic. Along with this paper we provide a Python script to calculate the relevant distributions [12]. Furthermore, we can use our analytical results to define the conditions under which we expect the standard PG test to apply approximately (namely in the limit of “strong” prior, to be defined in detail below).

In the context of cosmology, a test similar to the PG is often called DMAP (Difference of log-likelihoods at their Maximum a Posteriori point) [6]. Ref. [13] discusses tension metrics in the case of correlated data sets, which has some overlap with our considerations. In particular, they discuss the DMAP test and find that under certain approximations its distribution is a linear combination of χ2\chi^2 variables, in agreement with our findings. Our work differs from Ref. [13] in that we provide two variants of possible test statistics (one of them being strictly positive), formulate all correlations explicitly in terms of the prior data set, we work in the pure frequentist χ2\chi^2 framework, and we provide simple-to-use recipes for the pp-value calculation in the Gaussian limit, based on the characteristic function of the test statistic distribution.

The outline of the rest of the paper is as follows. In § 2 we set the notation and define two versions of a test statistics for the prior-constrained parameter goodness-of-fit, denoted by PG∗^* and PG+^+. Their analytic distributions under the Gaussian approximation are derived in § 3 and § 4, respectively. In § 5 we consider a simple toy experiment simulation inspired by neutrino oscillation disappearance measurements, which we use to study the PG∗^* and PG+^+ test statistics by Monte Carlo simulations. We verify that the Monte Carlo distribution agrees with the analytic results in the Gaussian approximation. Furthermore, we provide a comparison of the two test statistics and further discussion of their properties, as well as an illustration of the power of the test to reject a model with tension between data sets (including a prior). We summarize our results in § 6. Throughout the main text we limit the discussion to the compatibility of two data sets plus a prior, while we comment on the multi-data set case in appendix § 8.

Setup and definition of the test statistics

Let us consider two statistically independent data sets AA and BB, consisting of NAN_A, NBN_B data points respectively. We want to describe these data with a model depending on pp parameters θ\boldsymbol{\theta} common for both data sets. We can construct χ2\chi^2 functions as

χA2(θ)=χA,min⁡2+ΔχA2(θ),(1)\chi_A^2(\boldsymbol{\theta}) = \chi_{A,\min}^2 + \Delta\chi_A^2(\boldsymbol{\theta}), \tag*{(1)}
χB2(θ)=χB,min⁡2+ΔχB2(θ),(2)\chi_B^2(\boldsymbol{\theta}) = \chi_{B,\min}^2 + \Delta\chi_B^2(\boldsymbol{\theta}), \tag*{(2)}

where χS,min⁡2=min⁡θχS2(θ)\chi_{S,\min}^2 = \min_{\boldsymbol{\theta}} \chi_S^2(\boldsymbol{\theta}) for S=A,BS = A, B. The parameter goodness-of-fit test [1] provides a test-statistic for testing the compatibility of the two data sets under the hypothesis that the model is true:

χPG2=min⁡θ[χA2(θ)+χB2(θ)]−χA,min⁡2−χB,min⁡2(3)\chi_{\mathrm{PG}}^2 = \min_{\boldsymbol{\theta}} \left[\chi_A^2(\boldsymbol{\theta}) + \chi_B^2(\boldsymbol{\theta})\right] - \chi_{A,\min}^2 - \chi_{B,\min}^2 \tag*{(3)}
=[χA2(θ^AB)−χA,min⁡2]+[χB2(θ^AB)−χB,min⁡2](4)= \left[\chi_A^2(\widehat{\boldsymbol{\theta}}_{AB}) - \chi_{A,\min}^2\right] + \left[\chi_B^2(\widehat{\boldsymbol{\theta}}_{AB}) - \chi_{B,\min}^2\right] \tag*{(4)}
=ΔχA2(θ^AB)+ΔχB2(θ^AB),(5)= \Delta\chi_A^2(\widehat{\boldsymbol{\theta}}_{AB}) + \Delta\chi_B^2(\widehat{\boldsymbol{\theta}}_{AB}), \tag*{(5)}

where θ^AB\widehat{\boldsymbol{\theta}}_{AB} denotes the best fit parameters from the combination of both data sets. Under general conditions, similar to the ones required for Wilks’ theorem to hold, χPG2\chi_{\mathrm{PG}}^2 will be χp2\chi_p^2 distributed, i.e., a χ2\chi^2 distribution with pp degrees-of-freedom [1]. We will call this limit in the following “Gaussian approximation”.

Now we want to consider the situation where, in addition to the pp parameters θ\boldsymbol{\theta}, the model depends also on qq nuisance parameters ξ\boldsymbol{\xi}, which are constrained by another independent data set called “prior”. We describe the prior by

χP2(ξ)=χP,min⁡2+ΔχP2(ξ).(6)\chi_P^2(\boldsymbol{\xi}) = \chi_{P,\min}^2 + \Delta\chi_P^2(\boldsymbol{\xi}). \tag*{(6)}

We want to ask now the question, whether AA and BB are consistent under the additional constrain imposed by the prior data set on the nuisance parameters. The nuisance parameters can describe common systematic uncertainties, or an external constraint on some of the model parameters (in which case we will replace p→p−qp \to p - q so that the total number of parameters remains the same, and that pp consistently counts the parameters not affected by the prior).

We define individual χ2\chi^2 functions for the two data sets including the prior as

χAP2(θ,ξ)=χA2(θ,ξ)+ΔχP2(ξ),(7)\chi_{AP}^2(\boldsymbol{\theta}, \boldsymbol{\xi}) = \chi_A^2(\boldsymbol{\theta}, \boldsymbol{\xi}) + \Delta\chi_P^2(\boldsymbol{\xi}), \tag*{(7)}
χBP2(θ,ξ)=χB2(θ,ξ)+ΔχP2(ξ),(8)\chi_{BP}^2(\boldsymbol{\theta}, \boldsymbol{\xi}) = \chi_B^2(\boldsymbol{\theta}, \boldsymbol{\xi}) + \Delta\chi_P^2(\boldsymbol{\xi}), \tag*{(8)}

where we ignore χP,min⁡2\chi_{P,\min}^2 and add only ΔχP2(ξ)\Delta\chi_P^2(\boldsymbol{\xi}), as we are only interested in the constraint on the nuisance parameters imposed by the prior. Note, that now χAP2\chi_{AP}^2 and χBP2\chi_{BP}^2 are no longer statistically independent, because they include the same statistical fluctuations from the prior data set. Therefore, a straight-forward application of the original PG test is not possible, as it applies only for statistically independent data sets.

We are proposing in the following two possible test statistics including the prior constraint, which both are natural generalizations of the PG test statistic. First we define the PG* test statistic by

χ~PG∗2=min⁡θ,ξ[χA2(θ,ξ)+χB2(θ,ξ)+ΔχP2(ξ)]−χAP,min⁡2−χBP,min⁡2,(9)\widetilde{\chi}^{2}_{\mathrm{PG}*}=\min_{\theta,\boldsymbol{\xi}}\left[\chi^{2}_{A}(\theta,\boldsymbol{\xi})+\chi^{2}_{B}(\theta,\boldsymbol{\xi})+\Delta\chi^{2}_{P}(\boldsymbol{\xi})\right]-\chi^{2}_{AP,\min}-\chi^{2}_{BP,\min}, \tag*{(9)}

generalizing the PG version displayed in eq. (3). Unlike the original χPG2\chi^{2}_{\mathrm{PG}}, this quantity is not strictly positive and can also acquire negative values. Hence, it cannot follow a χν2\chi^{2}_{\nu} distribution. Indeed, as we are going to show below, in the Gaussian approximation χ~PG∗2\widetilde{\chi}^{2}_{\mathrm{PG}*} will follow a generalised χ2\chi^{2} distribution. We use the tilde to indicate that this quantity in general is not expected to be χ2\chi^{2} distributed.

As an alternative version, we define the PG+ test statistic, inspired by eq. (4):

χ~PG+2=[χAP2(x^ABP)−χAP,min⁡2]+[χBP2(x^ABP)−χBP,min⁡2],(10)\widetilde{\chi}^{2}_{\mathrm{PG}+}=\left[\chi^{2}_{AP}(\widehat{\boldsymbol{x}}_{ABP})-\chi^{2}_{AP,\min}\right]+\left[\chi^{2}_{BP}(\widehat{\boldsymbol{x}}_{ABP})-\chi^{2}_{BP,\min}\right], \tag*{(10)}

where x=(θ,ξ)\boldsymbol{x}=(\theta,\boldsymbol{\xi}) and x^ABP\widehat{\boldsymbol{x}}_{ABP} denotes the best fit parameters from the combination of AA, BB and the prior PP. Each bracket of eq. (10) is non-negative and therefore χ~PG+2≥0\widetilde{\chi}^{2}_{\mathrm{PG}+}\geq0 by construction, therefore the ‘+’ symbol in the acronym. As we will show, its distribution will also be a generalised χ2\chi^{2} distribution in the Gaussian limit.

Useful identities are

χ~PG∗2=χPG,ABP2−χPG,AP2−χPG,BP2,(11)\widetilde{\chi}^{2}_{\mathrm{PG}*}=\chi^{2}_{\mathrm{PG},ABP}-\chi^{2}_{\mathrm{PG},AP}-\chi^{2}_{\mathrm{PG},BP}, \tag*{(11)}
χ~PG+2=χ~PG∗2+ΔχP2(ξ^ABP).(12)\widetilde{\chi}^{2}_{\mathrm{PG}+}=\widetilde{\chi}^{2}_{\mathrm{PG}*}+\Delta\chi^{2}_{P}(\widehat{\boldsymbol{\xi}}_{ABP}). \tag*{(12)}

Here, χPG,ABP2\chi^{2}_{\mathrm{PG},ABP} denotes the standard PG test statistic for the consistency of the three independent data sets AA, BB, PP defined by

χPG,ABP2=min⁡θ,ξ[χA2(θ,ξ)+χB2(θ,ξ)+ΔχP2(ξ)]−χA,min⁡2−χB,min⁡2,(13)\chi^{2}_{\mathrm{PG},ABP}=\min_{\theta,\boldsymbol{\xi}}\left[\chi^{2}_{A}(\theta,\boldsymbol{\xi})+\chi^{2}_{B}(\theta,\boldsymbol{\xi})+\Delta\chi^{2}_{P}(\boldsymbol{\xi})\right]-\chi^{2}_{A,\min}-\chi^{2}_{B,\min}, \tag*{(13)}

which follows a χν2\chi^{2}_{\nu} distribution with ν=2(p+q)+q−(p+q)=p+2q\nu=2(p+q)+q-(p+q)=p+2q degrees of freedom [1], and χPG,AP2\chi^{2}_{\mathrm{PG},AP}, χPG,BP2\chi^{2}_{\mathrm{PG},BP} are the PG test statistics for the consistency of the data sets AA and BB with the prior, respectively. Hence, the three terms in eq. (11) are individually distributed as χp+2q2\chi^{2}_{p+2q}, χq2\chi^{2}_{q}, χq2\chi^{2}_{q}, respectively. However, it is important to note that they are not statistically independent, as they are constructed from different combinations of the same random data.

Equation (12) shows that χ~PG∗2\widetilde{\chi}^{2}_{\mathrm{PG}*} and χ~PG+2\widetilde{\chi}^{2}_{\mathrm{PG}+} differ just by a prior term. This term appears only once in χ~PG∗2\widetilde{\chi}^{2}_{\mathrm{PG}*} but twice in χ~PG+2\widetilde{\chi}^{2}_{\mathrm{PG}+}, ensuring positivity. As a consequence, PG+ gives somewhat more weight to a tension appearing in the prior, whereas PG* has more weight on the “data” part of the combination. In the following we will discuss the PG* and PG+ test statistics in turn.

The PG* test statistic

Derivation of the distribution for p=1p=1, q=1q=1

We first provide a derivation of the distribution of χ~PG∗2\widetilde{\chi}^{2}_{\mathrm{PG}*} for the simplest case of p=q=1p=q=1, i.e., one single model parameter θ\theta and one single nuisance parameter ξ\xi. The derivation makes use of standard methods from the theory of quadratic forms, see e.g., [14] for a text book.

In the Gaussian approximation, the χ2\chi^2 functions can be written in the following way:

χA2(x)=χA,min⁡2+(x−x^A)TVA−1(x−x^A),(14)\chi_A^2(x) = \chi_{A,\min}^2 + (x - \widehat{x}_A)^T V_A^{-1}(x - \widehat{x}_A), \tag*{(14)}
χB2(x)=χB,min⁡2+(x−x^B)TVB−1(x−x^B),(15)\chi_B^2(x) = \chi_{B,\min}^2 + (x - \widehat{x}_B)^T V_B^{-1}(x - \widehat{x}_B), \tag*{(15)}
χP2(ξ)=χP,min⁡2+(ξ−ξ^P)2sP.(16)\chi_P^2(\xi) = \chi_{P,\min}^2 + \frac{(\xi- \widehat{\xi}_P)^2}{s_P}. \tag*{(16)}

Here, the hat denotes the best-fit parameters, i.e., the parameter values at the χ2\chi^2 minimum of the corresponding data set, while VAV_A and VBV_B are the covariance matrices of xx from the data sets AA and BB, respectively. They are either predicted by the model, or estimated by

(VS−1)xy=12∂2χS2∂x∂y∣x^(17)(V_S^{-1})_{xy} = \left.\frac{1}{2}\frac{\partial^2\chi_S^2}{\partial x\partial y}\right|_{\widehat{x}} \tag*{(17)}

with x,y∈{θ,ξ}x,y \in\{\theta,\xi\} and S=A,BS = A,B. Similarly, sPs_P is the variance of ξ\xi from the prior. The expressions in eqs. (14) to (16) hold exactly for a model with linear parameter dependence; in general, they are valid sufficiently close to the χ2\chi^2 minima, as long as these are located far enough from the boundaries of the parameter space.

Let us consider the determination of the nuisance parameter ξ\xi from the data sets AA and BB without the prior. The marginal variances for ξ\xi from data set AA, BB are given by the ξξ\xi\xi element of the corresponding covariance matrix:

sA=(VA)ξξ,sB=(VB)ξξ.(18)s_A = (V_A)_{\xi\xi}, \qquad s_B = (V_B)_{\xi\xi}. \tag*{(18)}

We can now evaluate the consistency of the data sets AA and BB with the prior by considering the standardised discrepancies

dAP=ξ^A−ξ^PsA+sP,dBP=ξ^B−ξ^PsB+sP.(19)d_{AP} = \frac{\widehat{\xi}_A-\widehat{\xi}_P}{\sqrt{s_A+s_P}}, \qquad d_{BP} = \frac{\widehat{\xi}_B-\widehat{\xi}_P}{\sqrt{s_B+s_P}}. \tag*{(19)}

Under the null-hypothesis of the model being true, they will be standard normal distributed. Furthermore, one can show that χPG,SP2=dSP2\chi^2_{\mathrm{PG},SP}=d_{SP}^2 (S=A,BS=A,B), and therefore eq. (11) can be written as

χ~PG∗2=χPG,ABP2−dAP2−dBP2.(20)\widetilde{\chi}^2_{\mathrm{PG}^*} = \chi^2_{\mathrm{PG},ABP} - d_{AP}^2 - d_{BP}^2. \tag*{(20)}

The three terms are individually distributed as χ32\chi^2_3, χ12\chi^2_1, χ12\chi^2_1 but, as emphasized above, they are not statistically independent.

In order to write χ~PG∗2\widetilde{\chi}^2_{\mathrm{PG}^*} in terms of statistically independent quantities, we use the correlation of the discrepancies dAPd_{AP} and dBPd_{BP}. The estimators from AA, BB, and PP are independent, hence we have

Cov⁡(ξ^A−ξ^P,ξ^B−ξ^P)=Var⁡(ξ^P)=sP,(21)\operatorname{Cov}(\widehat{\xi}_A-\widehat{\xi}_P,\widehat{\xi}_B-\widehat{\xi}_P)=\operatorname{Var}(\widehat{\xi}_P)=s_P, \tag*{(21)}

and the correlation between dAPd_{AP} and dBPd_{BP} is given by

ρAB≡Corr⁡(dAP,dBP)=sP(sA+sP)(sB+sP).(22)\rho_{AB} \equiv\operatorname{Corr}(d_{AP},d_{BP}) = \frac{s_P}{\sqrt{(s_A+s_P)(s_B+s_P)}}. \tag*{(22)}

Since sA,sB,sP>0s_A, s_B, s_P > 0, we have 0<ρAB<10 < \rho_{AB} < 1. The limiting values 0 and 1 are obtained for an infinitely strong (sP→0s_P \to0) or infinitely weak (sP→∞s_P \to\infty) constraint from the prior PP, respectively.

We are now in the position to derive the distribution of χ~PG∗2\widetilde{\chi}^{2}_{\mathrm{PG}*}. Considering the form in eq. (20), we note that the first term, χPG,ABC2\chi^{2}_{\mathrm{PG},ABC}, being distributed as χ32\chi^{2}_{3}, is the squared norm of a three-dimensional vector of standard-normal random variables:

χPG,ABC2=g2,g∼N(0,I3).(23)\chi^{2}_{\mathrm{PG},ABC} = \boldsymbol{g}^{2}, \qquad\boldsymbol{g} \sim N(0,I_{3}). \tag*{(23)}

Now we can take the g\boldsymbol{g} as basis to express the discrepancies dAPd_{AP} and dBPd_{BP} providing the remaining terms in eq. (20) by expressing dAP2d_{AP}^{2} and dBP2d_{BP}^{2} as rank-one projection quadratic forms. Taking into account the correlation eq. (22), we write eq. (20) as

χ~PG∗2=gTKg,(24)\widetilde{\chi}^{2}_{\mathrm{PG}*} = \boldsymbol{g}^{T}K\boldsymbol{g}, \tag*{(24)}

where

K=I3−aaT−bbT,(25)K = I_{3} - \boldsymbol{a}\boldsymbol{a}^{T} - \boldsymbol{b}\boldsymbol{b}^{T}, \tag*{(25)}

and a\boldsymbol{a} and b\boldsymbol{b} are unit vectors satisfying

aTb=ρAB.(26)\boldsymbol{a}^{T}\boldsymbol{b} = \rho_{AB}. \tag*{(26)}

The matrix aaT+bbT\boldsymbol{a}\boldsymbol{a}^{T}+\boldsymbol{b}\boldsymbol{b}^{T} has eigenvalues (1+ρAB,1−ρAB,0)(1+\rho_{AB},1-\rho_{AB},0), which implies that KK has eigenvalues (1,ρAB,−ρAB)(1,\rho_{AB},-\rho_{AB}). Therefore, upon diagonalizing eq. (24) we obtain for the distribution of the test statistic:

χ~PG∗2=dG02+ρAB(G12−G22),(27)\widetilde{\chi}^{2}_{\mathrm{PG}*} \mathrel{\mathop{=}^{d}} G_{0}^{2}+\rho_{AB}(G_{1}^{2}-G_{2}^{2}), \tag*{(27)}

where G0,G1,G2G_0,G_1,G_2 are three independent standard-normal random variables. Or equivalently,

χ~PG∗2=dX0+ρAB(X1−X2),(28)\widetilde{\chi}^{2}_{\mathrm{PG}*} \mathrel{\mathop{=}^{d}} X_0+\rho_{AB}(X_1-X_2), \tag*{(28)}

where Xi∼χ12X_i \sim\chi^{2}_{1} are mutually independent χ2\chi^{2} variables with 1 degree of freedom. Thus, χ~PG∗2\widetilde{\chi}^{2}_{\mathrm{PG}*} has a generalized χ2\chi^{2} distribution [11], see also [15]. Its distribution depends on the experimental setup only through the single dimensionless parameter ρAB\rho_{AB}, eq. (22). For 0<ρAB<10 < \rho_{AB} < 1, the negative eigenvalue −ρAB-\rho_{AB} implies that χ~PG∗2\widetilde{\chi}^{2}_{\mathrm{PG}*} can take negative values. Hence it is not, in general, an ordinary χ2\chi^{2} random variable. From eq. (28) we see that when taking the expectation value of χ~PG∗2\widetilde{\chi}^{2}_{\mathrm{PG}*}, the two terms proportional to ρAB\rho_{AB} cancel and we obtain E[χ~PG∗2]=E[χ12]=1\mathbb{E}[\widetilde{\chi}^{2}_{\mathrm{PG}*}] = \mathbb{E}[\chi^{2}_{1}] = 1.

Using the independent representation in eq. (28), the characteristic function ϕz(t)≡E[eitz]\phi_z(t) \equiv\mathbb{E}[e^{itz}] of z=χ~PG∗2z=\widetilde{\chi}^{2}_{\mathrm{PG}*} is [11, 15]

ϕz(t)=(1−2it)−1/2(1−2itρAB)−1/2(1+2itρAB)−1/2=1(1−2it)(1+4ρAB2t2).(29)\phi_z(t) = (1-2it)^{-1/2}(1-2it\rho_{AB})^{-1/2}(1+2it\rho_{AB})^{-1/2} = \frac{1}{\sqrt{(1-2it)(1+4\rho_{AB}^{2}t^{2})}}. \tag*{(29)}

This characteristic function completely specifies the null distribution of the test statistic; the probability distribution function (PDF) f(z)f(z) and the cumulative distribution function (CDF) F(z)=∫zdz′f(z′)F(z)=\int^z dz' f(z') can be calculated in a straight-forward way:

f(z)=12π∫dt e−iztϕz(t),(30)f(z)=\frac{1}{2\pi}\int dt\,e^{-izt}\phi_z(t), \tag*{(30)}
F(z)=12−1π∫0∞dt 1tIm⁡[e−iztϕz(t)](31)F(z)=\frac{1}{2}-\frac{1}{\pi}\int_0^\infty dt\,\frac{1}{t}\operatorname{Im}\left[e^{-izt}\phi_z(t)\right] \tag*{(31)}

where for the latter we have used the Gil-Pelaez theorem. This expression allows directly to calculate pp-values for the PG* test.

Limiting cases: weak vs strong priors. From eq. (28) it is clear that, for small ρAB\rho_{AB}, χ~PG∗2\widetilde{\chi}^2_{\mathrm{PG}^*} will be approximately distributed as χ12\chi^2_1 (which holds exactly for ρAB=0\rho_{AB}=0). Note that for ρAB≪1\rho_{AB}\ll1 it is enough that either sAs_A or sBs_B is much larger than sPs_P, see eq. (22). Hence, if only one of the data sets provides a much weaker constraint on ξ\xi than the prior we expect a distribution close to χ12\chi^2_1. This will be referred to as the “strong prior” limit.

Conversely, in the “weak prior” limit sP≫sA,sBs_P\gg s_A,s_B, and ρAB≈1\rho_{AB}\approx1. In this case, eq. (28) implies that χ~PG∗2∼χ22−χ12\widetilde{\chi}^2_{\mathrm{PG}^*}\sim\chi^2_2-\chi^2_1, which does not converge towards a normal χ2\chi^2 but still remains a generalized χ2\chi^2 distribution. Hence, the limit sP→∞s_P\to\infty is not equivalent to removing the prior completely (in which case we would recover the normal PG, distributed as χ22\chi^2_2). The reason is that although the limit sP→∞s_P\to\infty removes effectively any constraint on ξ\xi from the prior, the best fit value ξ^P\hat{\xi}_P will obtain fluctuations of order sP\sqrt{s_P}, tending to infinity, with the effect that the last term in eq. (16) will still contribute as χ12\chi^2_1, irrespective of the value of sPs_P.

Generalization to an arbitrary number of parameters

The above reasoning can be generalised to any number of model parameters p>1p>1 and nuisance parameters q>1q>1. Without providing the full derivation we sketch the main idea below. The expressions for χA2\chi^2_A and χB2\chi^2_B from eqs. (14) and (15) generalize with x=(θ,ξ)\boldsymbol{x}=(\boldsymbol{\theta},\boldsymbol{\xi}) being a (p+q)(p+q)-vector. In generalisation of eq. (18) we obtain the q×qq\times q covariance matrices SAS_A, SBS_B for the nuisance parameters from data sets AA, BB as the ξ\xi sub-block of the covariance matrices VAV_A and VBV_B, whereas SPS_P denotes now the prior covariance matrix for ξ\xi. We define the two discrepancy vectors

DAP=ξ^A−ξ^P,DBP=ξ^B−ξ^P.(32)\boldsymbol{D}_{AP}=\hat{\boldsymbol{\xi}}_A-\hat{\boldsymbol{\xi}}_P,\qquad\boldsymbol{D}_{BP}=\hat{\boldsymbol{\xi}}_B-\hat{\boldsymbol{\xi}}_P. \tag*{(32)}

Under the common-parameter null hypothesis, their covariance matrices are

Cov⁡(DAP)=SA+SP,(33)\operatorname{Cov}(\boldsymbol{D}_{AP})=S_A+S_P, \tag*{(33)}
Cov⁡(DBP)=SB+SP,(34)\operatorname{Cov}(\boldsymbol{D}_{BP})=S_B+S_P, \tag*{(34)}
Cov⁡(DAP,DBP)=SP.(35)\operatorname{Cov}(\boldsymbol{D}_{AP},\boldsymbol{D}_{BP})=S_P. \tag*{(35)}

Therefore, we can define standardised discrepancies as

dAP=(SA+SP)−1/2DAP,(36)\boldsymbol{d}_{AP}=(S_A+S_P)^{-1/2}\boldsymbol{D}_{AP}, \tag*{(36)}
dBP=(SB+SP)−1/2DBP.(37)\boldsymbol{d}_{BP}=(S_B+S_P)^{-1/2}\boldsymbol{D}_{BP}. \tag*{(37)}

which are vectors of standard normal variables with the correlation matrix

R≡Corr⁡(dAP,dBP)=(SA+SP)−1/2SP(SB+SP)−1/2.(38)R \equiv\operatorname{Corr}(\boldsymbol{d}_{AP},\boldsymbol{d}_{BP}) = (S_A + S_P)^{-1/2}S_P(S_B + S_P)^{-1/2}. \tag*{(38)}

Let ρ1,…,ρq\rho_1,\ldots,\rho_q be the singular values of RR. These are the canonical correlations between the two standardized discrepancy vectors dAP\boldsymbol{d}_{AP} and dBP\boldsymbol{d}_{BP}. Equivalently, the squared canonical correlations ρi2\rho_i^2 are the eigenvalues of

RRT=(SA+SP)−1/2SP(SB+SP)−1SP(SA+SP)−1/2(39)RR^T = (S_A + S_P)^{-1/2}S_P(S_B + S_P)^{-1}S_P(S_A + S_P)^{-1/2} \tag*{(39)}

and they satisfy 0≤ρi≤10 \leq\rho_i \leq1. For q=1q = 1, eq. (38) reduces to eq. (22). Now we can write eq. (11) as

χ~PG∗2=χPG,ABP2−dAPTdAP−dBPTdBP,(40)\widetilde{\chi}^{2}_{\mathrm{PG}^*} = \chi^{2}_{\mathrm{PG},ABP} - \boldsymbol{d}_{AP}^{T}\boldsymbol{d}_{AP} - \boldsymbol{d}_{BP}^{T}\boldsymbol{d}_{BP}, \tag*{(40)}

with the three terms being χp+2q2\chi^{2}_{p+2q}, χq2\chi^{2}_{q}, χq2\chi^{2}_{q}. Similar as above, we consider a standard Gaussian vector g\boldsymbol{g} of dimension p+2qp + 2q. Within the space of g\boldsymbol{g}, the APAP and BPBP terms in the above sum are rank-qq orthogonal projection quadratic forms. Consequently, we can write

χ~PG∗2=gTKg,K=I−PA−PB,(41)\widetilde{\chi}^{2}_{\mathrm{PG}^*} = \boldsymbol{g}^{T}K\boldsymbol{g}, \qquad K = I - P_A - P_B, \tag*{(41)}

where PAP_A and PBP_B are rank-qq orthogonal projectors corresponding to the A−CA-C and B−CB-C discrepancy subspaces. Geometrically, the cosines of the principal angles between these two subspaces are the canonical correlations ρ1,…,ρq\rho_1,\ldots,\rho_q, in generalization of eq. (26). The eigenvalues of KK are

1,…,1⏟p times,ρ1,−ρ1,…,ρq,−ρq.(42)\underbrace{1,\ldots,1}_{p\ \text{times}},\quad\rho_1,-\rho_1,\quad\ldots,\quad\rho_q,-\rho_q. \tag*{(42)}

Therefore, under the common-parameter null hypothesis,

χ~PG∗2=d∑j=1pGj2+∑i=1qρi(Ui2−Vi2),(43)\widetilde{\chi}^{2}_{\mathrm{PG}^*} \mathrel{\overset{d}{=}} \sum_{j=1}^{p}G_j^2 + \sum_{i=1}^{q}\rho_i(U_i^2 - V_i^2), \tag*{(43)}

where GjG_j, UiU_i, ViV_i are mutually independent standard-normal variables. Or equivalently,

χ~PG∗2=dχp2+∑i=1qρi(Xi−Yi),(44)\widetilde{\chi}^{2}_{\mathrm{PG}^*} \mathrel{\overset{d}{=}} \chi_p^2 + \sum_{i=1}^{q}\rho_i(X_i-Y_i), \tag*{(44)}

with Xi,Yi∼χ12X_i,Y_i \sim\chi_1^2 and all χ2\chi^2 variables mutually independent. Thus we again obtain that χ~PG∗2\widetilde{\chi}^{2}_{\mathrm{PG}^*} is a generalized χ2\chi^2 random variable. It is an ordinary χ2\chi^2 variable only when all canonical correlations vanish.

We see from eq. (44) that

E[χ~PG∗2]=E[χp2]=p,(45)\mathbb{E}\left[\widetilde{\chi}^{2}_{\mathrm{PG}^*}\right] = \mathbb{E}\left[\chi_p^2\right] = p, \tag*{(45)}

independent of the prior. Hence, a value of χ~PG∗2≫p\widetilde{\chi}^{2}_{\mathrm{PG}^*} \gg p signals a low pp-value, irrespective of the prior.

From Eq. (44), the characteristic function is [11]

ϕz(t)=(1−2it)−p/2∏i=1q(1−2itρi)−1/2(1+2itρi)−1/2(46)\phi_z(t) = (1 - 2it)^{-p/2} \prod_{i=1}^{q}(1 - 2it\rho_i)^{-1/2}(1 + 2it\rho_i)^{-1/2} \tag*{(46)}
=(1−2it)−p/2∏i=1q(1+4ρi2t2)−1/2(47)= (1 - 2it)^{-p/2} \prod_{i=1}^{q}(1 + 4\rho_i^2t^2)^{-1/2} \tag*{(47)}
=(1−2it)−p/2det⁡(Iq+4t2RRT)−1/2.(48)= (1 - 2it)^{-p/2} \det\left(I_q + 4t^2RR^{T}\right)^{-1/2}. \tag*{(48)}

Hence, the task in the general case is to find the ξ\xi correlation matrices SAS_A, SBS_B, SPS_P, calculate the matrix RR and the canonical correlations ρi\rho_i. Once they are known, we can obtain the PDF or CDF via eqs. (30) and (31) using the above expression for the characteristic function. Similar as for the one-parameter case, we can define the limit of “strong prior” by ρi≪1\rho_i \ll1 (i=1,…,qi = 1, \ldots, q), in which case the PG* becomes the standard PG, distributed as a normal χ2\chi^2 variable with pp degrees of freedom.

The PG+ test statistic

Let us move now to the PG+ test and derive its distribution under the Gaussian approximation. We work this out here only for the case p=q=1p = q = 1 (with p>1p > 1 and q=1q = 1 being a trivial extension); we comment on the general case at the end of this section. According to eq. (12) we have

χ~PG+2=χ~PG∗2+(ξ^ABP−ξ^P)2sP.(49)\widetilde{\chi}^2_{\mathrm{PG+}} = \widetilde{\chi}^2_{\mathrm{PG^*}} + \frac{(\widehat{\xi}_{ABP} - \widehat{\xi}_P)^2}{s_P}. \tag*{(49)}

We proceed in analogy to section 3.1. First we need the distribution of the last term in eq. (49). We start by calculating sABPs_{ABP}, the variance of ξ^ABP\widehat{\xi}_{ABP}. Define F=VA−1+VB−1F = V_A^{-1} + V_B^{-1} and

FP=F+(0001/sP).(50)F_P = F + \begin{pmatrix} 0 & 0 \\ 0 & 1/s_P \end{pmatrix}. \tag*{(50)}

Then we have

sABP=(FP−1)ξξ=11sP+Fξξ−Fξθ2Fθθ.(51)s_{ABP} = (F_P^{-1})_{\xi\xi} = \frac{1}{\frac{1}{s_P} + F_{\xi\xi} - \frac{F_{\xi\theta}^2}{F_{\theta\theta}}}. \tag*{(51)}

Next we find

Cov⁡(ξ^ABP,ξ^P)=Cov⁡(ξ^ABP,ξ^A)=Cov⁡(ξ^ABP,ξ^B)=sABP.(52)\operatorname{Cov}(\widehat{\xi}_{ABP}, \widehat{\xi}_P) = \operatorname{Cov}(\widehat{\xi}_{ABP}, \widehat{\xi}_A) = \operatorname{Cov}(\widehat{\xi}_{ABP}, \widehat{\xi}_B) = s_{ABP}. \tag*{(52)}

It follows that Var⁡(ξ^ABP−ξ^P)=sP−sABP\operatorname{Var}(\widehat{\xi}_{ABP} - \widehat{\xi}_P) = s_P - s_{ABP}, and therefore

dP=ξ^ABP−ξ^PsP−sABP∼N(0,1).(53)d_P = \frac{\widehat{\xi}_{ABP} - \widehat{\xi}_P}{\sqrt{s_P - s_{ABP}}} \sim N(0,1). \tag*{(53)}

We see that the last term in eq. (49) is a re-scaled χ12\chi^2_1:

(ξ^ABP−ξ^P)2sP=κdP2∼κχ12,κ=sP−sABPsP,(54)\frac{(\widehat{\xi}_{ABP} - \widehat{\xi}_P)^2}{s_P} = \kappa d_P^2 \sim\kappa\chi^2_1,\qquad\kappa= \frac{s_P - s_{ABP}}{s_P}, \tag*{(54)}

with 0≤κ≤10 \le\kappa\le1, and we can write

χ~PG+2=χPG,ABP2−dAP2−dBP2+κdP2,(55)\widetilde{\chi}^{2}_{\mathrm{PG+}} = \chi^{2}_{\mathrm{PG},ABP} - d^{2}_{AP} - d^{2}_{BP} + \kappa d^{2}_{P}, \tag*{(55)}

where we have used the discrepancies dSPd_{SP} defined in eq. (19). With eq. (52) we have ρAB\rho_{AB} defined as in eq. (22) and, additionally,

ρAP≡Corr⁡(dAP,dP)=sP(sA+sP)(sP−sABP),(56)\rho_{AP} \equiv\operatorname{Corr}(d_{AP},d_{P}) = \frac{s_{P}}{\sqrt{(s_{A}+s_{P})(s_{P}-s_{ABP})}}, \tag*{(56)}
ρBP≡Corr⁡(dBP,dP)=sP(sB+sP)(sP−sABP).(57)\rho_{BP} \equiv\operatorname{Corr}(d_{BP},d_{P}) = \frac{s_{P}}{\sqrt{(s_{B}+s_{P})(s_{P}-s_{ABP})}}. \tag*{(57)}

In analogy to eq. (24) we write

χ~PG+2=gTKg,g∼N(0,I3)(58)\widetilde{\chi}^{2}_{\mathrm{PG+}} = \boldsymbol{g}^{T}K\boldsymbol{g}, \qquad\boldsymbol{g} \sim N(0,I_{3}) \tag*{(58)}

and

K=I3−aaT−bbT+κccT,(59)K = I_{3} - \boldsymbol{a}\boldsymbol{a}^{T} - \boldsymbol{b}\boldsymbol{b}^{T} + \kappa\boldsymbol{c}\boldsymbol{c}^{T}, \tag*{(59)}

with a,b,c\boldsymbol{a},\boldsymbol{b},\boldsymbol{c} are unit vectors satisfying

aTb=ρAB,aTc=ρAP,bTc=ρBP.(60)\boldsymbol{a}^{T}\boldsymbol{b} = \rho_{AB}, \qquad\boldsymbol{a}^{T}\boldsymbol{c} = \rho_{AP}, \qquad\boldsymbol{b}^{T}\boldsymbol{c} = \rho_{BP}. \tag*{(60)}

Diagonalising KK leads to

χ~PG+2=∑i=13λiXi2,(61)\widetilde{\chi}^{2}_{\mathrm{PG+}} = \sum_{i=1}^{3} \lambda_{i}X_{i}^{2}, \tag*{(61)}

where λi\lambda_{i} are the eigenvalues of KK and XiX_{i} are mutually independent standard normal variables. Hence, χ~PG+2\widetilde{\chi}^{2}_{\mathrm{PG+}} also follows a generalised χ2\chi^{2} distribution with the characteristic function

ϕz(t)=∏i=13(1−2itλi)−1/2.(62)\phi_{z}(t) = \prod_{i=1}^{3}(1-2it\lambda_{i})^{-1/2}. \tag*{(62)}

The eigenvalues can be obtained from diagonalising the matrix KK numerically by choosing an explicit basis for the vectors a,b,c\boldsymbol{a},\boldsymbol{b},\boldsymbol{c} or as a solution of the cubic equation.1

λ3−(1+κ)λ2+[κ(ρAP2+ρBP2)−ρAB2]λ+ρAB[(1+κ)ρAB−2κρAPρBP]=0.(63)\lambda^{3} - (1+\kappa)\lambda^{2} + \left[\kappa(\rho_{AP}^{2}+\rho_{BP}^{2})-\rho_{AB}^{2}\right]\lambda+ \rho_{AB}\left[(1+\kappa)\rho_{AB}-2\kappa\rho_{AP}\rho_{BP}\right] = 0. \tag*{(63)}

It can be shown that all λi≥0\lambda_{i} \ge0, and therefore χ~PG+2\widetilde{\chi}^{2}_{\mathrm{PG+}} is always non-negative, as per construction.

Limiting cases: weak vs strong priors. Let us consider again the strong and weak prior limits for this test statistic. In the strong prior limit (sP→0s_P \to0) we have κ→0\kappa\to0 and all ρXY→0\rho_{XY} \to0. The cubic equation becomes λ3−λ2=0\lambda^3 - \lambda^2 = 0, which has the solutions λ1=λ2=0\lambda_1 = \lambda_2 = 0, λ3=1\lambda_3 = 1 and therefore χ~PG+2∼χ12\widetilde{\chi}^{2}_{\mathrm{PG+}} \sim\chi^2_1, as expected for the PG once ξ\xi is fixed. Conversely, in the weak prior limit (sP→∞s_P \to\infty) we have κ→1\kappa\to1 and all ρXY→1\rho_{XY} \to1. The cubic equation in this case becomes λ3−2λ2+λ=0\lambda^3 - 2\lambda^2 + \lambda= 0, which has the solutions λ1=0\lambda_1 = 0, λ2=λ3=1\lambda_2 = \lambda_3 = 1. This implies χ~PG+2∼χ22\widetilde{\chi}^{2}_{\mathrm{PG+}} \sim\chi^2_2, which corresponds to the PG of AA versus BB without a prior. Hence, the PG+ has the property that both the strong and weak prior limits converge towards a standard χ2\chi^2 distribution.2

From eq. (61) we also see that

E[χ~PG+2]=∑i=13λi=Tr⁡[K]=1+κ,(64)\mathbb{E}\left[\widetilde{\chi}^{2}_{\mathrm{PG+}}\right] = \sum_{i=1}^{3} \lambda_i = \operatorname{Tr}[K] = 1 + \kappa, \tag*{(64)}

where we have used that each projector in eq. (59) has trace 1. This is consistent with the above discussed limiting distributions for strong and weak prior, with κ\kappa interpolating between 0→10 \to1. For p>1p > 1, q=1q = 1 this generalises to E[χ~PG+2]=p+κ\mathbb{E}\left[\widetilde{\chi}^{2}_{\mathrm{PG+}}\right] = p + \kappa.

Generalization to an arbitrary number of parameters. Finally, let us briefly comment on the case p>1p > 1 and q>1q > 1. In principle, the distribution under the Gaussian approximation can be calculated using similar methods as in section 3.2, which we do not discuss explicitly here (the corresponding distribution is provided numerically in the accompanying code [12]). Let us just remark that also in the general case, in the strong and weak prior limit the PG+ statistics converges towards the standard PG. This follows from the definition in eq. (10) together with eqs. (7) and (8). For a prior much stronger than the data sets, we have ξ^ABP≈ξ^SP≈ξ^P\widehat{\xi}_{ABP} \approx\widehat{\xi}_{SP} \approx\widehat{\xi}_{P}, such that

χSP2(x^ABP)−χSP,min⁡2 → χS2(θ^ABP,ξ^P)−χS2(θ^SP,ξ^P)(strong prior)(65)\chi^2_{SP}(\widehat{\mathbf{x}}_{ABP}) - \chi^2_{SP,\min} \ \to\ \chi^2_{S}(\widehat{\theta}_{ABP},\widehat{\xi}_{P}) - \chi^2_{S}(\widehat{\theta}_{SP},\widehat{\xi}_{P}) \qquad\text{(strong prior)} \tag*{(65)}

and the terms from the prior cancel. Hence, χ~PG+2\widetilde{\chi}^{2}_{\mathrm{PG+}} reduces to the standard PG with ξ\xi fixed to best fit from the prior alone. Therefore, it will follow a χp2\chi^2_p distribution. In contrast, for a prior much weaker than the ξ\xi determination from the data sets, we have ξ^ABP≈ξ^AB\widehat{\xi}_{ABP} \approx\widehat{\xi}_{AB} and ξ^SP≈ξ^S\widehat{\xi}_{SP} \approx\widehat{\xi}_{S}, such that

χSP2(x^ABP)−χSP,min⁡2 → χS2(x^AB)−χS2(x^S)(weak prior).(66)\chi^2_{SP}(\widehat{\mathbf{x}}_{ABP}) - \chi^2_{SP,\min} \ \to\ \chi^2_{S}(\widehat{\mathbf{x}}_{AB}) - \chi^2_{S}(\widehat{\mathbf{x}}_{S}) \qquad\text{(weak prior)} . \tag*{(66)}

In this case the prior terms cancel as distribution, because the prior terms will be dominated by the fluctuations of ξ^P\widehat{\xi}_{P}, and therefore ΔχP2(x^AB)∼ΔχP2(x^S)\Delta\chi^2_{P}(\widehat{\mathbf{x}}_{AB}) \sim\Delta\chi^2_{P}(\widehat{\mathbf{x}}_{S}) and fully correlated. Hence, in this limit, χ~PG+2\widetilde{\chi}^{2}_{\mathrm{PG+}} reduces to the PG for AA and BB without prior, which will be χp+q2\chi^2_{p+q} distributed.

Numerical results

Toy Monte-Carlo details

The theoretical distributions derived in the previous sections can be validated against Monte Carlo (MC) data. As an example, let us consider a toy experimental setup inspired by a neutrino oscillation experiment. In particular, we will work in a two-family approximation, where the probability is fully determined by two parameters: Δm2=m22−m12\Delta m^{2} = m_{2}^{2} - m_{1}^{2} (mim_i being the mass of each neutrino eigenstate) and a single mixing angle. In the flavor basis, the survival probability for a neutrino after propagating over a distance LL from the source, can be simply written as:

P(U,Δm2)=1−Usin⁡2(Δm2L4Eν),(67)P(U, \Delta m^{2}) = 1 - U \sin^{2}\left(\frac{\Delta m^{2}L}{4E_{\nu}}\right), \tag*{(67)}

where EνE_{\nu} stands for the neutrino energy, and U∈[0,1]U \in[0, 1] parametrizes the oscillation amplitude (which is a function of the mixing angle). For the purposes of this work it is useful to rewrite the probability in terms of the dimensionless parameter r≡Δm2/Δm02r \equiv\Delta m^{2}/\Delta m_{0}^{2} for some fixed reference value Δm02\Delta m_{0}^{2}, which yields

P(U,r)=1−Usin⁡2(rΔ0),(68)P(U, r) = 1 - U \sin^{2}(r\Delta_{0}), \tag*{(68)}

where Δ0≡Δm02L/(4Eν)\Delta_{0} \equiv\Delta m_{0}^{2}L/(4E_{\nu}). Choosing LL, EνE_{\nu} and Δm02\Delta m_{0}^{2} such that Δ0∼π/2\Delta_{0} \sim\pi/2 implies that r∼O(1)r \sim O(1) in the region of optimal sensitivity to the oscillation probability, which is convenient.

The distribution of signal events as a function of reconstructed neutrino energy can be computed in terms of the total neutrino flux ϕ\phi, the detection cross section (σdet\sigma_{\mathrm{det}}, which depends on the neutrino energy) and the oscillation probability, as

dN(U,r)dEν=NdϕdEνσdetP(U,r),(69)\frac{dN(U,r)}{dE_{\nu}} = \mathcal{N}\frac{d\phi}{dE_{\nu}}\sigma_{\mathrm{det}}P(U,r), \tag*{(69)}

where N\mathcal{N} is an overall normalization constant (which depends, for example, on the number of targets for the neutrino interaction inside the detector and its geometrical acceptance) and for simplicity we have neglected energy-dependent detection efficiencies or smearing effects. The physics observable is the binned number of events as a function of neutrino energy, NiN_i, where ii refers to the bin index, which is obtained by integrating eq. (69) within the limits of each energy bin. In our simulations, we specifically consider a reactor neutrino flux (taken from Refs. [16, 17]) and an inverse beta-decay detection cross section (taken from Ref. [18]). The value of N\mathcal{N} is chosen such that we obtain hundreds of events (even after oscillations) for all bins, so the Gaussian approximation applies. The main parameters used for the configuration of our toy experiments AA and BB, as well as the total event rates within the energy window used in the analysis, are summarized in Table 1 for completeness.

Exp.LL (km)Number of binsEν,min⁡E_{\nu,\min}ΔEν\Delta E_{\nu}Unoscillated events
AA5.534.1 MeV0.2 MeV4,153
BB3.534.1 MeV0.2 MeV15,384

Table 1. Main parameters used for the configuration of the toy experiments in our simulations. The energy bins are taken with equal width ΔEν\Delta E_{\nu}, starting at Eν,min⁡E_{\nu,\min}. The total number of events reported has been obtained without oscillations (that is, for U=0U = 0). No backgrounds or systematic uncertainties are considered in our analysis.

Given the values of LL and EνE_{\nu} in Table 1, oscillations are maximized around Δm2∼2.5×10−3 eV2\Delta m^{2} \sim2.5 \times10^{-3}\ {\rm eV}^{2}. Thus, Δm02=10−3 eV2\Delta m_{0}^{2} = 10^{-3}\ {\rm eV}^{2} is a natural choice, and unless otherwise noted we will take rˉ=2.5\bar{r} = 2.5 as our true value. Furthermore, in order to ensure that the value of UU always remains away from the physical boundaries at U=0,1U = 0, 1 we set its true value at Uˉ=0.587\bar{U} = 0.587.

Once we have defined our physical observables, we may consider two toy experiments S≡A,BS \equiv A, B, each of aiming to determine the oscillation parameters θ={r,U}\theta= \{r, U\} from their observed data OiSO_i^S. Within the Gaussian approximation, we may define a binned χ2\chi^2 function for each of them as

χS2(θ,ξ)=∑i(OiS−NiS(θ,ξ)σiS)2(70)\chi_S^2(\theta,\xi) = \sum_i \left(\frac{O_i^S-N_i^S(\theta,\xi)}{\sigma_i^S}\right)^2 \tag*{(70)}

where σiS=OiS\sigma_i^S = \sqrt{O_i^S} is the statistical error in each bin. The covariance matrices for experiments AA and BB (defined in Table 1), computed at the true input values for rr and UU, read:

VA≃(3.5−5.1−5.17.8)×10−3;VB≃(3.75.75.79.2)×10−3.(71)V_A \simeq\begin{pmatrix} 3.5 & -5.1 \\ -5.1 & 7.8 \end{pmatrix} \times10^{-3}; \qquad V_B \simeq\begin{pmatrix} 3.7 & 5.7 \\ 5.7 & 9.2 \end{pmatrix} \times10^{-3}. \tag*{(71)}

The prior constraint is implemented in our simulation through the addition of quadratic pull terms, as in eq. (16):

ΔχU2=(U−U^)2sU,Δχr2=(r−r^)2sr.(72)\Delta\chi_U^2 = \frac{(U-\widehat{U})^2}{s_U}, \qquad\Delta\chi_r^2 = \frac{(r-\widehat{r})^2}{s_r}. \tag*{(72)}

In order to derive the expected distribution for our toy experiments, we need to simulate a large sample of pseudo-data in presence of statistical fluctuations: these will be used as OiO_i in eq. (70). In doing so, we first pick a set of “true” oscillation parameters and nuisance parameters {θˉ,ξˉ}\{\bar{\theta},\bar{\xi}\} and compute the predicted event rates Ni(θˉ,ξˉ)N_i(\bar{\theta},\bar{\xi}). Then, random fluctuations are generated around the predicted number of events in each bin assuming a Gaussian distribution with width σi=Ni\sigma_i = \sqrt{N_i}. In full analogy, a similar procedure should be followed for the priors since these are also obtained from data (and are therefore subject to statistical fluctuations). After picking a set of true values for each prior (ξˉ)(\bar{\xi}), for each pseudo-experiment a random number is generated around ξˉ\bar{\xi} using a Gaussian distribution with a width according to the prior experimental uncertainty. The resulting value, ξ^P\widehat{\xi}_P, will be used as central value for the pull term in eq. (72). As outlined in previous sections, the resulting distributions will depend on the number of nuisance parameters included in the fit. Therefore, for our simulations we have considered the following two scenarios:

  • We first assume that there is a prior measurement only on UU, which now acts as a nuisance parameter in our fit. Hence, p=1p = 1, q=1q = 1 in this case, with θ=r\theta= r and ξ=U\xi= U, we only impose ΔχU2\Delta\chi_U^2 from eq. (72) and have sP=sUs_P = s_U.

  • Second, we consider a case with two prior measurements imposing both terms from eq. (72), that is, p=0p = 0, q=2q = 2, with ξ=(r,U)\boldsymbol{\xi} = (r, U). In this case SPS_P is a 2×22 \times2 diagonal matrix with entries srs_r and sUs_U, i.e., we assume the prior measurements to be uncorrelated.

Results for the null hypothesis

We present now the numerical results of our simulations, comparing them to the theoretical distributions derived for the test statistics in § 3 and § 4. Figure 1 shows our results for the PG∗PG^{*} test statistic, where we compare the complementary CDF calculated from simulation (thick solid lines) to the theoretical distributions in eq. (31) (thin dashed lines). The left panel shows our results for p=1p = 1, q=1q = 1 (that is, when only a prior on UU is imposed) while the right panel shows the corresponding result for p=0p = 0, q=2q = 2 (when priors are imposed on both UU and rr). In both panels, the different lines illustrate how the results change as a function of the priors, between the strong prior limit (sP→0s_P \to0, light blue lines) and weak prior limit (sP→∞s_P \to\infty, dark blue lines).

Figure 1. Complementary cumulative distribution function of the PG∗PG^{*} test statistic. Thick solid lines show the numerical results obtained for 10410^{4} pseudo-experiments, while thin dashed lines show the theoretical distributions obtained in § 3. Dash-dotted gray lines show the distributions obtained for a χ2\chi^{2} with one and two degrees of freedom. The left panel corresponds to the case p=1p = 1, q=1q = 1, and we show curves for different values of the correlation parameter ρAB\rho_{AB} defined in eq. (22). The right panel corresponds to the case p=0p = 0, q=2q = 2, and we give the values for the prior variances sUs_U and srs_r in the legend, see eq. (72).

In all cases the agreement between the numerical results and the theoretical distributions is excellent, with mild differences that can be attributed to the limited statistics of the simulated sample of pseudo-data. We also see that, for all lines, the random variable corresponding to the PG∗PG^{*} admits negative values (as expected). As can be seen from the figure, for ρAB≲0.2\rho_{AB} \lesssim0.2 the distribution remains reasonably close to a χ12\chi^{2}_{1}, whereas for $\rho_{AB} \gtrsim 0.2 relevant deviations appear. Figure 1 also shows how in the weak prior limit the obtained distributions do not converge towards a normal χ2\chi^{2}; instead, they remain a generalized χ2\chi^{2} distribution, as anticipated in § 3.

Our results for the PG+^{+} test statistic are shown in Figure 2, where the complementary CDF calculated from our simulation (thick solid lines) is compared to the theoretical distributions obtained in § 4 (thin dashed lines), for p=1p=1, q=1q=1 (left panel) and p=0p=0, q=2q=2 (right panel). Again in this case, the agreement between our numerical results and the theoretical distributions is excellent. Darker/lighter lines show how the results change as we approach the weak/strong prior limits, respectively. As discussed in § 4, for p=1p=1, q=1q=1 in the strong prior limit (sP→0s_P \to0) the distribution matches a χ2\chi^{2} with one degree of freedom. However, the obtained distributions start to show large deviations from the χ12\chi^{2}_{1} limit even for small ρAB∼O(0.01)\rho_{AB} \sim\mathcal{O}(0.01). This contrasts with the behaviour of the PG∗^{*} test, where a χ12\chi^{2}_{1} remains a relatively good approximation even for ρAB\rho_{AB} as large as 0.2. Figure 2 also shows that for intermediate values of sPs_P the tail of the distribution can go above that of a χ22\chi^{2}_{2}; however, as the strength of the prior approaches the weak limit, the distribution converges smoothly to a χ22\chi^{2}_{2}, as anticipated in § 4.

Figure 2. Complementary cumulative distribution function of the PG+^{+} test statistic. Thick solid lines show the numerical results obtained for 10410^{4} pseudo-experiments, while thin dashed lines show the theoretical distributions obtained in § 4. Dash-dotted gray lines show the distributions obtained for a χ2\chi^{2} with one and two degrees of freedom. The left panel corresponds to the case p=1p=1, q=1q=1, and we show curves for different values of the correlation parameter ρAB\rho_{AB} defined in eq. (22). The right panel corresponds to the case p=0p=0, q=2q=2, and we give the values for the prior variances srs_r and sUs_U in the legend, see eq. (72).

Note that for the case p=0p=0 shown in the right panels of Figure 1 and Figure 2, the limit of infinitely strong prior would formally lead to a χ2\chi^{2} distribution with zero degrees-of-freedom, meaning no randomness. In this case, the prior-constrained PG tests would measure just the consistency of the prior with itself, which by construction is perfect.

Let us mention that our toy MC is on purpose constructed such that we expect the Gaussian approximation to hold (high-statistics data, true parameter values far away from boundaries, and only one minimum in the χ2\chi^{2}). A detailed study of a real-life setup using actual neutrino oscillation data, where some of these properties do not hold, will be presented in Ref. [19].

Comparison between the two test statistics. Since the two test statistics considered here differ on the way the prior is introduced (with a larger relative weight for PG+{}^{+} than for PG∗{}^{*}), a relevant question to ask is whether one of the two test statistics is more sensitive to tension in the fit than the other. In fig. 3 we show the distribution of pp-values for the two tests from our MC simulation, where we use the shorthand notation p∗p^{*} (p+p^{+}) for the pp-value of PG∗{}^{*} (PG+{}^{+}). Again, the left panel corresponds to p=1p=1, q=1q=1 while the right panel shows the results for p=0p=0, q=2q=2 as indicated by the labels.

The upper panels in fig. 3 show the pp-values obtained for the two tests statistics considered, for 10410^{4} toy MC realizations. As shown in the upper panels in fig. 3, the points tend to cluster in the region above the diagonal, indicating that under the null-hypothesis it is likely to obtain a larger pp-value for PG+\mathrm{PG}^{+} than for PG∗\mathrm{PG}^{*}. Analytically, this behaviour can be understood since statistical fluctuations can only make χ~PG+2≥χ~PG∗2\widetilde{\chi}^{2}_{\mathrm{PG}^{+}} \ge\widetilde{\chi}^{2}_{\mathrm{PG}^{*}} by construction, see eq. (12), which translates into a strict upper bound on p+p^{+}. Specifically, using the shorthand notation z∗=χ~PG∗2z^{*} = \widetilde{\chi}^{2}_{\mathrm{PG}^{*}} and z+=χ~PG+2z^{+} = \widetilde{\chi}^{2}_{\mathrm{PG}^{+}}, it is easy to see that

z+≥z∗⇒p+=F‾+(z+)≤F‾+(z∗) ,(73)z^{+} \ge z^{*} \quad\Rightarrow\quad p^{+} = \overline{F}^{+}(z^{+}) \le\overline{F}^{+}(z^{*}) \, , \tag*{(73)}

where F‾+/∗(z)=1−F+/∗(z)\overline{F}^{+/*}(z) = 1 - F^{+/*}(z) refers to the complementary CDF for the PG+/PG∗\mathrm{PG}^{+}/\mathrm{PG}^{*} test statistic, and we have p+/∗=F‾+/∗(z)p^{+/*} = \overline{F}^{+/*}(z) by definition. Therefore, the upper bound on p+p^{+} can be obtained by the curve

γ(z)=[F‾∗(z),F‾+(z)] ,(74)\gamma(z) = \left[\overline{F}^{*}(z), \overline{F}^{+}(z)\right] \, , \tag*{(74)}

parametrized by zz. The curve from eq. (74) is shown by the dashed black lines in the upper two panels in Figure 3. As can be seen, the agreement with our numerical results is again excellent. We have verified that in the strong prior limit (for p>0p > 0), the distribution of p∗p^{*} and p+p^{+} values concentrate to points close to the diagonal p∗=p+p^{*} = p^{+}, as in that limit both the PG∗\mathrm{PG}^{*} and PG+\mathrm{PG}^{+} tests become identical and converge towards the standard PG.

Figure 3. Comparison between the pp-values obtained for the PG∗{}^{*} (p∗p^{*}) and PG+{}^{+} (p+p^{+}) test statistics, for two examples with p=1p=1, q=1q=1 (left panels) and p=0p=0, q=2q=2 (right panels). Upper panels: scatter plot of the pp-values obtained for the two tests statistics considered, for 10410^{4} experimental realizations (red points). The diagonal black line corresponds to p+=p∗p^{+}=p^{*}, while the dashed black line shows the curve corresponding to eq. (74), see text for details. Lower panels: fraction of points for which p+>p∗p^{+}>p^{*} as a function of p∗p^{*} for the same simulations.

In the same figure, the lower panels show the fraction of points where p+>p∗p^{+} > p^{*} as a function of p∗p^{*}, for the same data sample. As can be seen, this fraction approaches a constant value around ∼0.7−0.8\sim0.7 - 0.8 in the region p∗≲O(0.75)p^{*} \lesssim O(0.75), while for larger values of p∗p^{*} it drops to zero. We note that the fraction remains at ∼0.7−0.8\sim0.7 - 0.8 for small values of p∗p^{*} regardless of the prior strength; however the exact value where the fraction drops depends on the number of priors considered and the values assumed, as illustrated by the two examples shown in Figure 3. The value of p∗p^{*} where the fraction starts to drop is approximately set by the pp-value obtained for χ~PG∗2=0\widetilde{\chi}^{2}_{\mathrm{PG}^{*}} = 0, where the dashed curve from eq. (74) reaches p+=1p^{+} = 1. In fact, as χ~PG∗2\widetilde{\chi}^{2}_{\mathrm{PG}^{*}} decreases towards negative values, the upper bound on p+p^{+} does not change (since it had already saturated at 1) but p∗p^{*} keeps increasing, and therefore the fraction of points where p+>p∗p^{+} > p^{*} drops.

Hence, for not too large pp-values, we find that under the null-hypothesis it is likely to find a larger pp-value for PG+\mathrm{PG}^{+} than for PG∗\mathrm{PG}^{*}. As the prior has more weight in the PG+\mathrm{PG}^{+} test statistic, this indicates that it is somewhat more likely that fluctuations contribute to the χ2\chi^{2}-terms involving the data sets AA and BB rather than to the prior term ΔχP2\Delta\chi^{2}_{\mathcal{P}}. Comparing the values of p∗p^{*} and p+p^{+} for a given realisation of the data may be used as an additional diagnostic tool to identify the origin of a tension.

Results for alternative hypotheses

Until now we have focused on the computation of the expected distributions under the null hypothesis H0H_{0}. It assumes that the model is true, i.e., that the two experiments as well as the prior measurement are random realisations of the data for the same true values of the underlying model parameters. However, it is also worth considering the distribution of the test statistics under an alternative hypothesis H1H_{1}.

The theoretical distribution for an alternative hypothesis can be derived following the same procedure as in sections 3 and 4 bearing in mind that, in this case, the variables will no longer be normally distributed around zero but around some non-vanishing value instead (which eventually depends on the alternative hypothesis chosen). For example, if we consider a hypothesis where an experiment SS and the prior PP observe data that are generated with different true values of the nuisance parameter, ξˉS\bar{\xi}_S and ξˉP\bar{\xi}_P, then χPG,SP2=dSP2\chi^2_{\mathrm{PG},SP} = d_{SP}^2 will follow a non-central chi-squared distribution with one d.o.f. and non-centrality parameter (ξˉS−ξˉP)2/(sS+sP)(\bar{\xi}_S-\bar{\xi}_P)^2/(s_S+s_P). Likewise, it can be shown that χPG,ABP2\chi^2_{\mathrm{PG},ABP} will follow a non-central distribution with 3 d.o.f., where the statistically independent variables can be obtained through the same change of basis as in § 3 and § 4. Therefore, both PG∗PG^* and PG+PG^+ can be written as a sum of non-central chi-squared distributions which, by definition, follows a generalized χ2\chi^2 distribution [11, 15]. Our accompanying code [12] provides auxiliary functions to compute the non-centrality parameters (as well as the resulting distribution) for any choice of true values for the underlying model parameters.

As outlined above, the choice of alternative hypothesis is not unique. While a systematic study of this is beyond the scope of this work, we have identified two general cases that are physically motivated:

Case I: both data sets AA and BB are based on the same set of true model parameters while the prior measurement is generated from different true values. In this case the tension in the combined fit arises from the mismatch between the prior and the data sets.

Case II: the true value of the prior is in agreement with the one of UU used to simulate the data, but different true values of rr are used for experiments AA and BB. In this case the prior is consistent with both experiments separately; however, when analysed in combination a tension emerges in the fit.

We are mostly interested in a scenario where there is significant tension between the different data sets and/or the prior. This implies that the best-fit points for the joint fit will be significantly displaced from the minimum obtained for the Asimov data set for each experiment, which could eventually make our results sensitive to non-linearities entering the χ2\chi^2. Thus, in order to guarantee that our simulations still fall within the linear regime also for the alternative hypotheses, the results obtained in the remainder of this section have been obtained multiplying by 30 the statistics of the experiments with respect to our default setups defined in § 5.1.

Figure 4 shows the interplay of the different data sets in the combined fit for two specific examples satisfying the criteria outlined above. In both panels, the best-fit points for each experiment separately (blue and red dots, for AA and BB, respectively) do not show any tension: they are found to be in good agreement with the sensitivity obtained using the Asimov data as well as with the prior. Conversely, a tension is observed for the joint fit (purple dots) in both panels. The distributions of pseudo-experiments for the null and alternative hypotheses are shown in Figure 5 taking case I as an example, for the two test statistics under consideration. We find that the theoretical distributions obtained for the alternative hypothesis (solid blue lines) are in excellent agreement with our numerical results. Note that the results for case II are qualitatively very similar and are therefore not shown here.

Figure 4. Example showing the sensitivity regions at 2σ2\sigma CL (for 2 d.o.f.) for the two experiments under consideration. The star indicates the true values of the parameters used to simulate each experiment, and the solid contours show the sensitivity regions obtained with the Asimov data set for each experiment separately (blue/red lines) and in combination (purple line). The dots indicate the location of the global minima found when pseudo-data for the two experiments are fitted separately (blue and red dots for experiments A and B, respectively) or jointly (purple dots), in all cases imposing the same prior on UU. Left panel (case I): the best-fit of the prior measurement is set to a different value of UU, as indicated by the gray band. Right panel (case II): the best-fit of the prior measurement is in agreement with the value of UU used to simulate the data, but different true values of rr are used for experiments AA and BB.

Distributions of pseudo-experiments for the alternative and null hypotheses

Figure 5. Example showing the distributions of pseudo-experiments for the alternative hypothesis (H1H_1, purple histograms), compared to the null hypothesis (H0H_0, teal histograms). The solid blue curve on top of the purple histograms shows the theoretical distributions, in excellent agreement with our simulation. The vertical purple lines indicate the sensitivity of an average experiment, see text for details. This example corresponds to case I; the results for case II are qualitatively very similar.

Median sensitivity. The tension observed in Figure 4 can be further quantified by the pp-value corresponding to the median of the distribution of pseudo-experiments for the alternative hypothesis. This defines the sensitivity of an average experiment or median sensitivity. In Figure 5, in each panel, the vertical purple lines indicates the pp-value corresponding to the median sensitivity. For our alternative hypotheses we find:

\begin{aligned} p^{*}\vert_{\mathrm{median}} &= 1.6 \cdot10^{-4}\ (3.6\sigma) \\ p^{+}\vert_{\mathrm{median}} &= 2.9 \cdot10^{-5}\ (4.0\sigma) \end{aligned} \qquad\text{for case I} \tag*{(75)}\end{aligned}
\begin{aligned} p^{*}\vert_{\mathrm{median}} &= 2.6 \cdot10^{-5}\ (4.1\sigma) \\ p^{+}\vert_{\mathrm{median}} &= 3.8 \cdot10^{-5}\ (4.0\sigma) \end{aligned} \qquad\text{for case II.} \tag*{(76)}\end{aligned}

where the reported number of standard deviations corresponds to a one-sided Gaussian. We see that for case I we obtain a somewhat larger tension for PG+\mathrm{PG}^{+} than for PG∗\mathrm{PG}^{*}. This is expected, since in this case the tension emerges from the prior itself, which has a larger relative weight for PG+\mathrm{PG}^{+} than for PG∗\mathrm{PG}^{*}. Conversely, for case II the difference in pp-values is reduced, with a slightly lower tension for PG+\mathrm{PG}^{+} than for PG∗\mathrm{PG}^{*}. This is again expected since the tension comes from the data instead of the prior and, moreover, the relative weight of the data is lower for PG+\mathrm{PG}^{+}.

Overall, the example provided above illustrates how a comparison between the pp-values obtained for PG∗\mathrm{PG}^{*} and PG+\mathrm{PG}^{+} may aid in determining the source of a possible tension in the data, as previously mentioned in § 5.2.

Power of the test. Until now we have focused on the computation of the type-I error rate associated to a given test statistic, that is, the probability to reject the null hypothesis H0H_0 even though it is true (i.e., the pp-value). However, the type-II error rate β\beta is also relevant: it is defined as the probability of not rejecting H0H_0 although it is false. In order to compute it, we first need to pick a (true) alternative hypothesis and obtain its corresponding distribution for the test statistic. Next, we choose a pp-value and determine the associated value of the test statistic under H0H_0. The corresponding type-II error can be obtained as the fraction of experiments within H1H_1 which would yield a lower value. Then, 1−β1-\beta is denoted as the power of the test.

Figure 6 shows the type-II error rate β\beta of the test statistics as a function of the pp-value for the two cases considered here. By definition, the pp-value corresponding to β=0.5\beta= 0.5 (indicated by a horizontal dotted line) yields the median sensitivity in eqs. (75) and (76). However, the relative difference in the tension reported by the two test statistics is also present for other values of β\beta: while for case I (left panel) the two curves are more separated with PG+^+ having slightly more power (i.e., lower β\beta), for case II (right panel) the two test statistics behave in a very similar manner.3 In the same figure we also show how the curves change as a function of ρAB\rho_{AB}. We find that the relative comparison between the two sets of curves generally holds as ρAB\rho_{AB} is varied, except when the weak prior limit is approached. This is expected since, in the weak prior limit, the tension from the prior disappears and therefore β→1\beta\to1 regardless of the pp-value (that is, the discrimination between the two hypothesis is no longer possible for the particular examples chosen here). We also see that in the strong prior limit the two lines become very similar even for case I, since in this limit both tests converge to the standard PG.

Figure 6. Type-II error rate of the test as a function of the pp-value, for the two test statistics under consideration. Left and right panels show the results for alternative hypotheses matching cases I and II, respectively. In each panel, each pair of lines has been obtained for a different prior, between ρAB=3.5⋅10−3\rho_{AB}=3.5 \cdot10^{-3} (lightest) to ρAB=0.6\rho_{AB}=0.6 (darkest).

Summary

In this work we have proposed an extension of the Parameter Goodness-of-fit (PG) test, which can be used when some of the parameters of the model are constrained by a prior common to the data sets. In such a case, the data and the prior are no longer statistically independent, so the original PG test cannot be applied. We have proposed two modified versions of the PG test statistic, denoted by PG∗PG^{*} and PG+PG^{+}. Both can be used to test the compatibility of data sets subject to a common prior, so we call the corresponding test prior-constrained parameter goodness-of-fit.

The two test statistics differ by a term involving the prior χ2\chi^{2}. The PG+PG^{+} is constructed to be strictly non-negative thanks to the additional prior term, whereas PG∗PG^{*} can assume also negative values and gives somewhat less weight to the prior. Both test statistics converge towards the standard PG test in the case of “strong prior”, i.e., when the constraint on the parameters from the prior is much stronger than the one from the data sets, with a number of degrees of freedom that is determined by the number of parameters unconstrained by the prior.

In the Gaussian approximation the PG∗PG^{*} and PG+PG^{+} test statistics are distributed according to a so-called generalised χ2\chi^{2} distribution, which can be easily calculated via its characteristic function. The distribution is determined just by the parameter covariance matrices of the data sets and the prior. Once the distribution is known, pp-values can be calculated in a straightforward manner. While the distribution of the test statistics depend on the prior strength, the expectation value of the PG∗PG^{*} test statistic is given by the numbers of parameters unconstrained by the prior, pp, independently of the prior strength, see eq. (45). Hence a value of the test statistic much larger than pp indicates a low pp-value. For the PG+^{+} test statistic the expectation value depends on the prior strength and varies between pp for a strong prior till p+qp+q for a weak prior, where pp (qq) is the number of parameters unconstrained (constrained) by the prior. We provide an accompanying Python code [12] which allows to calculate the corresponding distribution functions and pp-values for any number of model and nuisance parameters, given a set of covariance matrices.

Our results have been validated through Monte Carlo simulation of two toy experiments inspired by neutrino oscillation facilities. The simulated setup is close to the Gaussian limit (large statistics, no multiple minima, and away from physical boundaries). The simulations show excellent agreement with the analytic expressions for the distributions of the PG∗^{*} and PG+^{+} test statistics. We find that, under the null hypothesis and for not too large pp-values, it is more likely to obtain a larger pp-value for PG+^{+} than for PG∗^{*}. This indicates that it is more likely that statistical fluctuations contribute to the χ2\chi^{2} of individual experiments than to the prior term. A comparison between both pp-values may be used as an additional tool to diagnose the origin of a tension. Finally, we have also computed the power of the test for an alternative hypothesis, for two physically motivated examples, illustrating that our test statistics are able to detect tensions between data sets in the presence of a common prior. We find that both test statistics have comparable power, with PG+^{+} performing somewhat better, especially in cases where the tension is mostly driven by the prior.

In summary, we believe that the test statistics proposed here offer a useful generalisation of the PG test for analysing the statistical compatibility of data sets, and extends its range of applicability.

Acknowledgements

We thank Yufeng Li for asking a relevant question about the PG test, which triggered this work. We acknowledge the help of ChatGPT and Claude for developing code as well as for some of the mathematical derivations. This project is funded by the European Union’s Horizon Europe research and innovation programme (Marie Skłodowska-Curie Staff Exchange grant agreement 101086085-ASYMMETRY) and by ERDF “A way of making Europe”. It also receives support from the R&D&I project CEX2025-001574-S “Centro de Excelencia Severo Ochoa” (the research presented in this publication falls within the research line “Particle Physics in the Standard Model and Beyond (BSM)” ) and from grants PID2022-142545NB-C21 and PID2025-171019NB-C21, all funded by MICIU/AEI/10.13039/501100011033.

Prior-constrained PG for more than two data sets

Similar as the original PG test, also the prior-constrained PG can be extended in a straightforward way to the case of more than two data sets with a common prior. We consider NN data sets, with χSP2(θ,ξ)=χS2(θ,ξ)+ΔχP2(ξ)\chi^{2}_{SP}(\theta,\xi)=\chi^{2}_{S}(\theta,\xi)+\Delta\chi^{2}_{P}(\xi) for S=1,…,NS=1,\ldots,N, and define

χ~PG∗2=min⁡θ,ξ[∑S=1NχS2(θ,ξ)+ΔχP2(ξ)]−∑S=1NχSP,min⁡2,(77)\widetilde{\chi}^{2}_{PG*}=\min_{\theta,\xi}\left[\sum_{S=1}^{N}\chi^{2}_{S}(\theta,\xi)+\Delta\chi^{2}_{P}(\xi)\right]-\sum_{S=1}^{N}\chi^{2}_{SP,\min}, \tag*{(77)}
χ~PG+2=∑S=1N[χSP2(x^all)−χSP,min⁡2]=χ~PG∗2+(N−1)ΔχP2(ξ^all)(78)\widetilde{\chi}^{2}_{PG+}=\sum_{S=1}^{N}\left[\chi^{2}_{SP}(\widehat{x}_{\mathrm{all}})-\chi^{2}_{SP,\min}\right]=\widetilde{\chi}^{2}_{PG*}+(N-1)\Delta\chi^{2}_{P}(\widehat{\xi}_{\mathrm{all}}) \tag*{(78)}

where x^all=(θ^all,ξ^all)\widehat{x}_{\mathrm{all}}=(\widehat{\theta}_{\mathrm{all}},\widehat{\xi}_{\mathrm{all}}) denotes the best fit parameters from the combination of all NN data sets and the prior PP. Notice that, as already seen in eq. (12), χ~PG∗2\widetilde{\chi}^{2}_{PG*} and χ~PG+2\widetilde{\chi}^{2}_{PG+} differ just by a prior term.

We outline here the main idea for PG* for the simplest case of p=q=1p=q=1, and leave the generalization to the multi-parameter case as well as for PG+ to the reader. In the Gaussian approximation we can derive the distribution function of this statistic in analogy to section 3.1. Following eq. (20), we write

χ~PG∗2=χPG,all2−∑S=1NdSP2,(79)\widetilde{\chi}^{2}_{PG*}=\chi^{2}_{PG,\mathrm{all}}-\sum_{S=1}^{N}d_{SP}^{2}, \tag*{(79)}

where χPG,all2\chi^{2}_{PG,\mathrm{all}} is the standard PG test statistic for all NN data sets plus the prior PP, and

dSP=ξ^S−ξ^PsS+sP,S=1,…,N.(80)d_{SP}=\frac{\widehat{\xi}_{S}-\widehat{\xi}_{P}}{\sqrt{s_{S}+s_{P}}},\qquad S=1,\ldots,N. \tag*{(80)}

Under the null-hypothesis of the model being true, χPG,all2\chi^{2}_{PG,\mathrm{all}} is distributed as χν2\chi^{2}_{\nu} with ν=2N+1−2=2N−1\nu=2N+1-2=2N-1 and the dSPd_{SP} are standard-normal variables with the correlation matrix

ρXY={1X=Y,sP(sX+sP)(sY+sP)X≠Y,(81)\rho_{XY}=\begin{cases} 1 & X=Y,\\ \frac{s_{P}}{\sqrt{(s_{X}+s_{P})(s_{Y}+s_{P})}} & X\ne Y, \end{cases} \tag*{(81)}

for X,Y=1,…,NX,Y=1,\ldots,N.

Hence, we write eq. (A.3) as

χ~PG∗2=gTKg,(82)\widetilde{\chi}^{2}_{PG*}=\boldsymbol{g}^{T}K\boldsymbol{g}, \tag*{(82)}

where g\boldsymbol{g} is a vector of ν=2N−1\nu=2N-1 independent standard normal variables and the ν×ν\nu\times\nu matrix KK is given by

K=Iν−∑S=1NaSaST=Iν−AAT,(83)K=I_{\nu}-\sum_{S=1}^{N}\boldsymbol{a}_{S}\boldsymbol{a}_{S}^{T}=I_{\nu}-AA^{T}, \tag*{(83)}

with aS\boldsymbol{a}_{S} are unit vectors satisfying aXTaY=ρXY\boldsymbol{a}_{X}^{T}\boldsymbol{a}_{Y}=\rho_{XY} and the ν×N\nu\times N matrix A=(a1,…,aN)A=(\boldsymbol{a}_{1},\ldots,\boldsymbol{a}_{N}). We note that the non-zero eigenvalues of AATAA^{T} are the same as the eigenvalues of ATA=ρA^{T}A=\rho. Let us denote these eigenvalues by λS(ρ)\lambda^{(\rho)}_{S}, S=1,…,NS=1,\ldots,N. Then the eigenvalues of KK are

(λi(K))=(1,…,1⏟N−1 times, 1−λ1(ρ),…,1−λN(ρ))(84)\left(\lambda^{(K)}_{i}\right)=\left(\underbrace{1,\ldots,1}_{N-1\ \mathrm{times}},\ 1-\lambda^{(\rho)}_{1},\ldots,1-\lambda^{(\rho)}_{N}\right) \tag*{(84)}

for i=1,…,νi = 1, \ldots, \nu. Therefore, upon diagonalizing eq. (A.6) we obtain for the distribution of the test statistic:

χ~PG∗2=d∑i=1νλi(K)Gi(K)2=d∑i=1νλi(K)Xi(85)\widetilde{\chi}_{\mathrm{PG}*}^{2} \overset{d}{=} \sum_{i=1}^{\nu} \lambda_i^{(K)} G_i^{(K)2} \overset{d}{=} \sum_{i=1}^{\nu} \lambda_i^{(K)} X_i \tag*{(85)}

where GiG_i (XiX_i) are ν\nu mutually independent standard-normal (χ12\chi_1^2) variables. The characteristic function is obtained as

ϕz(t)=∏i=1ν(1−2itλi(K))−1/2.(86)\phi_z(t) = \prod_{i=1}^{\nu} \left(1 - 2it\lambda_i^{(K)}\right)^{-1/2}. \tag*{(86)}

Hence, in this case the distribution of the test statistic is obtained by calculating the eigenvalues of the correlation matrix eq. (A.5).

Strong prior limit. For sP→0s_P \to0, eq. (A.5) gives ρ→IN\rho\to I_N and λS(ρ)=1\lambda_S^{(\rho)} = 1 (S=1,…,NS = 1, \ldots, N). In this limit, eq. (A.8) for the eigenvalues λi(K)\lambda_i^{(K)} gives (N−1)(N - 1) times the eigenvalue 11 and all remaining eigenvalues are zero. Therefore, eq. (A.9) gives a normal χ2\chi^2 distribution with N−1N - 1 degrees of freedom, which is the result of the normal PG for NN data sets with one common parameter θ\theta.

References

  1. [1]M. Maltoni and T. Schwetz, Testing the Statistical Compatibility of Independent Data Sets, Phys. Rev. D 68 (2003) 033020, [hep-ph/0304176].DOI
  2. [2]M. Maltoni, T. Schwetz, M. A. Tortola, and J. W. F. Valle, Ruling Out Four Neutrino Oscillation Interpretations of the LSND Anomaly?, Nucl. Phys. B 643 (2002) 321–338, [hep-ph/0207157].DOI
  3. [3]DES, P. Lemos et al., Assessing tension metrics with dark energy survey and Planck data, Mon. Not. Roy. Astron. Soc. 505 (2021), no. 4 6179–6194, [2012.09554].
  4. [4]M. Leizerovich, S. J. Landau, and C. G. Scóccola, Tensions in cosmology: A discussion of statistical tools to determine inconsistencies, Phys. Lett. B 855 (2024) 138841, [2312.08542].
  5. [5]L. Amendola, V. Marra, and M. Quartin, Internal Robustness: Systematic Search for Systematic Bias in Sn Ia Data, Mon. Not. Roy. Astron. Soc. 430 (2013) 1867–1879, [1209.1897].DOI
  6. [6]M. Raveri and W. Hu, Concordance and Discordance in Cosmology, Phys. Rev. D 99 (2019), no. 4 043506, [1806.04649].DOI
  7. [7]W. Lin and M. Ishak, A Bayesian Interpretation of Inconsistency Measures in Cosmology, JCAP 05 (2021) 009, [1909.10991].DOI
  8. [8]S. Gariazzo, O. Mena, and T. Schwetz, Quantifying the Tension Between Cosmological and Terrestrial Constraints on Neutrino Masses, Phys. Dark Univ. 40 (2023) 101226, [2302.14159].
  9. [9]P. R. Mello, M. Quartin, B. M. Schaefer, and B. Schosser, On the full non-Gaussian Surprise statistic and the cosmological concordance between DESI, SDSS and Pantheon+, Open J. Astrophys. 8 (8, 2024) 2025, [2408.08385].DOI
  10. [10]S. Wilks, The Large-Sample Distribution of the Likelihood Ratio for Testing Composite Hypotheses, Annals Math. Statist. 9 (1938), no. 1 60–62.DOI
  11. [11]“Generalized chi-squared distribution.” Wikipedia.
  12. [12]P. Coloma, M. Maltoni, and T. Schwetz. Code provided in arXiv supplementary material.
  13. [13]M. Raveri, G. Zacharegkas, and W. Hu, Quantifying concordance of correlated cosmological data sets, Phys. Rev. D 101 (2020), no. 10 103527, [1912.04880].DOI
  14. [14]A. M. Mathai and S. B. Provost, Quadratic Forms in Random Variables: Theory and Applications. Marcel Dekker, 1992.DOI
  15. [15]J. P. Imhof, Computing the distribution of quadratic forms in normal variables, Biometrika 48 (1961), no. 3/4 419–426.
  16. [16]H. Murayama and A. Pierce, Energy spectra of reactor neutrinos at KamLAND, Phys. Rev. D 65 (2002) 013012, [hep-ph/0012075].DOI
  17. [17]KamLAND, K. Eguchi et al., First results from KamLAND: Evidence for reactor anti-neutrino disappearance, Phys. Rev. Lett. 90 (2003) 021802, [hep-ex/0212021].DOI
  18. [18]P. Vogel and J. F. Beacom, Angular distribution of neutron inverse beta decay, anti-neutrino(e) + p —> e+ + n, Phys. Rev. D 60 (1999) 053003, [hep-ph/9903554].DOI
  19. [19]P. Blanco-Mas, P. Coloma, M. Maltoni, and T. Schwetz. In preparation.

Paper details

vlnc:W_r5dpaf8b
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Physics and Astronomy

ChatGPT, Claude

View statement
Looking for related papers…
Contents