betaFeedback

WorksPhysics and Astronomy

A regularization-independent Information Criterion for correlated data

Abstract

We generalize the Akaike Information Criterion (AIC) to allow model and data subset selection when regularized fits are performed on correlated data. While the standard generalization to regularized fits is inherently dependent on the type of regularization applied to the data covariance matrix, our definition, the RIAIC, is, on average, regularization-independent. The regularization-independence is achieved by defining the new Information Criterion as an estimator of the model-dependent contribution to the correlated Kullback–Leibler divergence. The Bayesian extension is derived as well. We test the new criterion on lattice QCD data by performing regularized fits and show that it reproduces the selection properties of the standard correlated AIC, with fluctuations which are compatible with the expected statistical uncertainties. These results enable the application of subset and model selection techniques to cases where regularizations are applied to the covariance matrix of the fitted dataset.

Introduction

The goal of identifying the optimal description of a dataset plays a pivotal role in both experimental and theoretical data analysis. By optimal we refer to a parameterization that simultaneously maximizes the information extracted from the data while minimizing the complexity of the model itself. In addition to the optimal parameterization for a fixed dataset, comparisons of different data subsets are also necessary. More generally, model and data subset selection provides a valuable tool to assess and quantify systematic uncertainties arising from arbitrary choices on the model and on the fitted dataset.

Information-Theoretic Criteria (ICs) are a widely used approach to model selection problems. They are theoretically sound, since their definition is rooted in information theory through the Kullback–Leibler divergence, and, by construction, they represent a trade-off between quality of the fit and complexity of the description.

One commonly used IC is the Akaike Information Criterion (AIC) [1, 2]. The standard definition of the AIC explicitly assumes that correlations in the dataset under study are properly taken into account.

However, it is often the case that covariance matrices are estimated on a limited, and noisy, sample of data. This may lead to close-to-singular covariance matrices, for which minimization procedures, such as least squares methods, become impractical. In this context, a possible solution to extract additional information from the data is to perform fits by employing regularized versions of the covariance matrix. Typical regularizations include the application of SVD cuts or the limiting case of a fully uncorrelated fit with a diagonal covariance matrix.

While a proper treatment of the covariance matrix and the impact of regularizations can be addressed at the level of the quality of the fit, by redefining the expectation values of chi-square variables, see Ref. [3], accounting for these effects at the level of the AIC still represents a gap which prevents a reliable comparison of models and subsets in the case of regularized fits. In general, ICs are rooted in information theory and regularizations of the covariance matrix effectively modify the amount of information contained in the selected dataset, by removing poorly determined information. Similarly, in a Bayesian framework, the use of priors modifies the information used as input to the fit, by adding knowledge on the system under study. On the basis of these considerations, the application of any regularization to the data covariance matrix introduces a dependence on the regularization itself in the AIC. As such, a systematic comparison of models and datasets when fits are performed with regularized covariance matrices is still not possible. This is particularly relevant when the size of the fitted dataset is varied, since the regularization required to obtain a stable fit typically changes with it. A typical example in the analysis of lattice field theory data is represented by the search for the optimal fit window of a correlation function, where different windows may require different regularizations, mostly depending on the size of the window itself. Using the approach proposed here, and detailed in the following sections, the regularization of the covariance matrix, which is necessary to stabilize the fit, is incorporated into the AIC.

This work is organized as follows. In § 2 we briefly review the idea behind Information Criteria, by focusing on the definition of the Akaike Information Criterion and its extension to dataset selection. In the same section, we report the standard generalization of the AIC to regularized fits and we show that its extension to subset selection problems induces an explicit dependence on the regularization which makes it unreliable. In § 3 we propose a new Information Criterion which is explicitly built to be, on average, regularization-independent. Finally, in § 4 we explicitly carry out numerical tests on actual lattice data to validate the formulae derived in this work.

The main results of this work are represented by the expressions in eqs. (22) and (25), together with their Bayesian extension in eq. (29).

Throughout this work we often refer to examples or typical situations encountered in lattice field theory studies, e.g. the identification of the optimal fit window in the extraction of the energy spectrum from Euclidean lattice correlators. Nevertheless the formulae derived here apply to any situation in which the data follow a multivariate Gaussian distribution.

Information-Theoretic Criteria

Given a dataset y={y1,…,ynd}y = \{y_1,\ldots,y_{n_d}\}, and a model function η\eta which depends on the parameter vector θ={θ1,…,θnη}\boldsymbol{\theta} = \{\theta_1,\ldots,\theta_{n_\eta}\} and, optionally, on some other variables, let us consider a set of candidate model probabilities, i.e. a set of probability density functions [4]

MΣ={L(y,Σ∣θ,η)∣θ∈Θη}(1)\mathcal{M}_{\Sigma} = \left\{\mathcal{L}(y,\Sigma\mid\boldsymbol{\theta},\eta) \mid\boldsymbol{\theta} \in\Theta_{\eta}\right\} \tag*{(1)}

where Θη\Theta_{\eta} is the parameter space for a given model function η\eta and dim⁡[Θη]=nη\dim[\Theta_{\eta}] = n_{\eta}. The true covariance matrix Σ\Sigma is in most cases unknown and it is usually replaced by the sample covariance matrix [5, 6] which provides a consistent and asymptotically unbiased estimator of Σ\Sigma.

Given a candidate probability density function L∈MΣ\mathcal{L} \in\mathcal{M}_{\Sigma} and the data-generating unknown distribution of the dataset P(y)P(y), we define the corresponding Kullback–Leibler (KL) divergence [7] as

DKL(P∥L)=∫dyP(y)log⁡[P(y)L(y,Σ∣θ,η)]=E[log⁡P(y)]+E[−log⁡L(y,Σ∣θ,η)],(2)\begin{aligned} D_{\mathrm{KL}}(P \mathbin{\|} \mathcal{L}) = \int\mathrm{d}y P(y) \log\left[\frac{P(y)}{\mathcal{L}(y,\Sigma\mid\boldsymbol{\theta},\eta)}\right] \\ &= \mathbb{E}\left[\log P(y)\right] + \mathbb{E}\left[-\log\mathcal{L}(y,\Sigma\mid\boldsymbol{\theta},\eta)\right], \tag*{(2)} \end{aligned}

where in the last line E[⋅]\mathbb{E}[\cdot] denotes the expectation value over the true probability distribution P(y)P(y). Note that DKL(P∥L)=0D_{\mathrm{KL}}(P \mathbin{\|} \mathcal{L}) = 0 if and only if L=P\mathcal{L} = P. Therefore the empirical description that best approximates the true probability distribution is found by minimizing the KL divergence, and eq. (2) can be interpreted as a measure of how large the deviations between the candidate and the true distribution are.

The first term appearing in eq. (2), is, by definition, independent of the candidate model. Therefore, minimizing the Kullback–Leibler divergence with respect to the parameter vector θ\boldsymbol{\theta} is equivalent to minimizing the model-dependent quantity

Δ(Σ∣θ,η)=E[−2log⁡L(y,Σ∣θ,η)],(3)\Delta(\Sigma\mid\boldsymbol{\theta},\eta) = \mathbb{E}\left[-2\log\mathcal{L}(y,\Sigma\mid\boldsymbol{\theta},\eta)\right], \tag*{(3)}

where the additional factor of 2 is conventional. Let us stress that the expression above depends on the data-generating distribution P(y)P(y) and it is, therefore, not directly accessible. However, given an estimator θ∗\boldsymbol{\theta}^{*} for the parameter vector θ\boldsymbol{\theta}, Information Criteria are defined as unbiased plug-in [8, 9] estimators of eq. (3) as

ICη=−2log⁡L(y,Σ∣θ∗,η)+B(y,Σ,η),(4)\mathrm{IC}_{\eta} = -2\log\mathcal{L}(y,\Sigma\mid\boldsymbol{\theta}^{*},\eta) + B(y,\Sigma,\eta), \tag*{(4)}

where B(y,Σ,η)B(y,\Sigma,\eta) is a bias term that depends on the dataset and on the parameter space associated with the candidate model L(y,Σ∣θ,η)\mathcal{L}(y,\Sigma\mid\boldsymbol{\theta},\eta) and its expression depends on the choice of the plug-in estimator θ∗\boldsymbol{\theta}^{*} used in eq. (4). In eq. (4) the subscript indicates the model function. As usual, the bias is computed as the expectation value of the difference between the biased estimator and the target quantity, i.e.

B(y,Σ,η)≡E[2log⁡L(y,Σ∣θ⋆,η)+Δ(Σ∣θ⋆,η)].(5)B(\boldsymbol{y},\Sigma,\eta) \equiv\mathbb{E}\left[2\log\mathcal{L}(\boldsymbol{y},\Sigma\mid\boldsymbol{\theta}^{\star},\eta)+\Delta(\Sigma\mid\boldsymbol{\theta}^{\star},\eta)\right]. \tag*{(5)}

As we will see in the following, the bias in eq. (5) usually plays the role of a penalty term which penalizes overly complex models and favors simpler parameterizations for a given dataset; in this sense, the bias naturally discourages overfitting.

Akaike Information Criterion

In this section we briefly review the standard definition of the Akaike Information Criterion (AIC) [1, 2]. For a detailed introduction to the topic we refer to Ref. [4]. In the following we work under the assumption that the model is correctly or over-specified. In practice, a model can be considered correctly specified if the corresponding fit is accepted based on its quality, e.g. if the corresponding pp-value is above a given threshold. For regularized fits the quality of the fit must be computed as in Ref. [3]. In this case the data-generating distribution belongs to the set MΣ\mathcal{M}_{\Sigma} and can be written as P(y)=L(y,Σ∣θ0,η)P(\boldsymbol{y})=\mathcal{L}(\boldsymbol{y},\Sigma\mid\boldsymbol{\theta}_{0},\eta), where θ0\boldsymbol{\theta}_{0} is the true value of the parameter vector.

The standard definition of the AIC assumes as plug-in estimator for the parameter vector θ\boldsymbol{\theta} the maximum likelihood estimator (MLE) θMLEΣ≡argmax⁡θ[L(y,Σ∣θ,η)]\boldsymbol{\theta}_{\mathrm{MLE}}^{\Sigma}\equiv\operatorname*{argmax}_{\boldsymbol{\theta}}\left[\mathcal{L}(\boldsymbol{y},\Sigma\mid\boldsymbol{\theta},\eta)\right] which is asymptotically unbiased. Following the definitions given in the previous section, the negative log-likelihood evaluated at the MLE parameter defines an Information Criterion according to eq. (4) with bias computed as in eq. (5). In Refs. [1, 2] it was shown that the bias term has a particularly simple expression and depends only on the overall dimension of the parameter space Θη\Theta_{\eta}, leading to

B(y,Σ,η)≡E[2log⁡L(y,Σ∣θMLEΣ,η)+Δ(Σ∣θMLEΣ,η)]=2nη.(6)B(\boldsymbol{y},\Sigma,\eta) \equiv\mathbb{E}\left[2\log\mathcal{L}(\boldsymbol{y},\Sigma\mid\boldsymbol{\theta}_{\mathrm{MLE}}^{\Sigma},\eta)+\Delta(\Sigma\mid\boldsymbol{\theta}_{\mathrm{MLE}}^{\Sigma},\eta)\right]=2n_{\eta}. \tag*{(6)}

For a detailed derivation of the bias see Ref. [4]. Therefore an unbiased estimator for eq. (3) is given by the so-called Akaike Information Criterion [1, 2], which is defined as

AICη=−2log⁡L(y,Σ∣θMLEΣ,η)+2nη.(7)\mathrm{AIC}_{\eta}=-2\log\mathcal{L}(\boldsymbol{y},\Sigma\mid\boldsymbol{\theta}_{\mathrm{MLE}}^{\Sigma},\eta)+2n_{\eta}. \tag*{(7)}

Under the assumption that both the data-generating and the candidate model probabilities are multivariate Gaussian distributions, the AIC can be written, up to terms which are constant across all models, in the standard way as

AICη=χΣ2+2nη,(8)\mathrm{AIC}_{\eta}=\chi_{\Sigma}^{2}+2n_{\eta}, \tag*{(8)}

where χΣ2\chi_{\Sigma}^{2} is the chi-square evaluated at the MLE parameters and the subscript makes explicit the covariance matrix used in its definition. Therefore, smaller values of the AIC correspond to smaller values of eq. (3) and, as a consequence, a better agreement between the candidate model and the true probability distribution. Note that, in general, it holds E[AICη]=nd+nη\mathbb{E}[\mathrm{AIC}_{\eta}] = n_d+n_{\eta}. Therefore parameterizations with a large number of parameters will be penalized, if the quality of the fit is similar. In this sense, the AIC is a trade-off between the quality of the fit and the complexity of the model.

Model and dataset selection

The problem of data subset selection, i.e. the problem of finding the most informative data subset which can be described by the simplest model, can be rephrased in terms of model selection problems as detailed in Refs. [8, 9].

The core idea is that a given dataset, which hereafter we refer to as the full dataset, can be split into two different subsets: one which is included in the actual fit and one which is excluded. Under the assumption that the fitted and non-fitted datasets are statistically independent, the total KL divergence and, as a consequence, eq. (3) factorize into a sum of two independent terms 2. This leads to a factorization of the corresponding AIC as

AICfull=AICη+AICC(9)\mathrm{AIC}_{\mathrm{full}} = \mathrm{AIC}_{\eta} + \mathrm{AIC}_{\mathrm{C}} \tag*{(9)}

where the first term on the r.h.s. corresponds to the AIC of the fitted dataset y\boldsymbol{y} where the model function η\eta has been used to describe the data and AICC\mathrm{AIC}_{\mathrm{C}} is the AIC of the non-fitted dataset3.

In the literature different definitions for AICC\mathrm{AIC}_{\mathrm{C}} have been proposed, see Refs. [10, 11] and Ref. [9] for a detailed comparison. The results presented here are independent of the chosen definition; nevertheless, in the following we will always assume that the perfect model definition is employed [11]. The generalization to other definitions is straightforward.

The perfect model is a model for the non-fitted dataset with as many parameters as data points, e.g. a polynomial interpolant. Therefore, under the Gaussian approximation of the non-fitted dataset, since the perfect model provides a vanishing chi-square, the AIC associated with the non-fitted subset reduces to

AICC=2nC+log⁡det⁡ΣC+nClog⁡2π,(10)\mathrm{AIC}_{\mathrm{C}} = 2n_{\mathrm{C}} + \log\det\Sigma_{\mathrm{C}} + n_{\mathrm{C}}\log2\pi, \tag*{(10)}

where the first term on the r.h.s. is twice the number of parameters of the perfect model, which is, by definition, equal to the number of data points of the non-fitted dataset and, most importantly, it does not depend on the explicit functional form of the perfect model. As a consequence, for normally-distributed data the AIC of the full dataset can be written as

AICfull=χΣ2+2nη+2nC+log⁡det⁡Σfull+nfulllog⁡2π,(11)\mathrm{AIC}_{\mathrm{full}} = \chi_{\Sigma}^{2} + 2n_{\eta} + 2n_{\mathrm{C}} + \log\det\Sigma_{\mathrm{full}} + n_{\mathrm{full}}\log2\pi, \tag*{(11)}

where nfull=nd+nCn_{\mathrm{full}} = n_d + n_C and we employed the fact that the fitted and non-fitted data subsets are statistically uncorrelated, i.e. Σfull=diag⁡(Σ,ΣC)\Sigma_{\mathrm{full}} = \operatorname{diag}(\Sigma, \Sigma_C). Finally, by neglecting terms which are common to all models and to the full dataset, it simplifies to the known result [9, 11]

AICfull=χΣ2+2nη+2nC.(12)\mathrm{AIC}_{\mathrm{full}} = \chi_{\Sigma}^{2} + 2n_{\eta} + 2n_C . \tag*{(12)}

For a specific full dataset and model function η\eta, eq. (12) can be used to extend model selection problems to subset selection ones. Specifically, the use of AICfull\mathrm{AIC}_{\mathrm{full}} will automatically promote models which provide a good description of a data subset which is as large as possible, with a model depending on a number of parameters which is as small as possible.

Akaike Information Criterion and regularized fits

The definition of the AIC given in eq. (7) can be easily extended to the scenario where fits with regularized covariance matrices are performed on correlated, normally distributed, data. This result is reported in Ref. [12] and in this section we briefly review the framework in which it is derived.

As before, we assume that the model η\eta is correctly specified and the true probability distribution is a multivariate normal distribution P(y)=L(y,Σ∣θ0,η)P(y) = \mathcal{L}(y, \Sigma\mid\theta_0, \eta), where θ0\theta_0 is the true value of the parameter vector. If the covariance matrix Σ\Sigma is numerically ill-conditioned, it is common practice to perform the fit using a regularized matrix. Therefore, the plug-in estimator used to estimate eq. (3) is θMLEC≡argmax⁡θL(y,C∣θ,η)\theta_{\mathrm{MLE}}^C \equiv\operatorname*{argmax}_{\theta} \mathcal{L}(y, C \mid\theta, \eta) which is, unlike θMLEΣ\theta_{\mathrm{MLE}}^\Sigma, obtained by maximizing the log-likelihood using a generic positive-definite matrix CC whose elements are O(N−1)O(N^{-1}), where NN is the size of the Monte Carlo sample. Note that, in general, these two estimators might differ and their difference depends on how much the regularized covariance matrix CC differs from Σ\Sigma.

In this context, we are testing model probabilities which are no longer in the set MΣ\mathcal{M}_{\Sigma} but in the much larger set MC\mathcal{M}_C which contains MΣ\mathcal{M}_{\Sigma} as the specific case of models based on correlated fits. By defining the estimator of the regularized model-dependent term of the KL-divergence as in eq. (3), the bias to be computed is

B(y,C,η)≡E[2log⁡L(y,C∣θMLEC,η)+Δ(C∣θMLEC,η)]=2nηeff,(13)\mathcal{B}(y,C,\eta) \equiv\mathbb{E}\left[2\log\mathcal{L}(y,C\mid\theta_{\mathrm{MLE}}^C,\eta)+\Delta(C\mid\theta_{\mathrm{MLE}}^C,\eta)\right] = 2n_{\eta}^{\mathrm{eff}}, \tag*{(13)}

where, in analogy with the standard expression in eq. (7), we introduced the effective number of parameters as

nηeff=tr⁡[C−1/2ΣC−1/2P],(14)n_{\eta}^{\mathrm{eff}} = \operatorname{tr}\left[C^{-1/2}\Sigma C^{-1/2}\mathcal{P}\right], \tag*{(14)}

where the projector P\mathcal{P} is defined in Ref. [3] and satisfies tr⁡[P]=nη\operatorname{tr}[\mathcal{P}] = n_{\eta}, see eq. (A.3). By using this expression for the bias, the regularized version of the AIC, which hereafter we refer to as AIC‾\overline{\mathrm{AIC}} in order to distinguish it from the standard definition, reads

AIC‾η=−2log⁡L(y,C∣θMLEC,η)+2nηeff.(15)\overline{\mathrm{AIC}}_{\eta} = -2\log\mathcal{L}(y,C\mid\theta_{\mathrm{MLE}}^C,\eta) + 2n_{\eta}^{\mathrm{eff}} . \tag*{(15)}

The expression in eq. (15) corresponds, up to constant terms, to the one reported in Ref. [12]. Notice that the definition of the AIC is inherently regularization-dependent and, in particular, under the assumption of normally-distributed data, its expectation value reads

E[AIC‾η]=ndeff+nηeff+log⁡det⁡C+ndlog⁡2π,(16)\mathbb{E}\left[\overline{\mathrm{AIC}}_{\eta}\right] = n_{d}^{\mathrm{eff}} + n_{\eta}^{\mathrm{eff}} + \log\det C + n_d \log2\pi, \tag*{(16)}

where ndeff=tr⁡[C−1Σ]n_{d}^{\mathrm{eff}} = \operatorname{tr}\left[C^{-1}\Sigma\right]. Therefore, by applying different regularizations to the covariance matrix, the AIC will yield different values. This is not surprising, since eq. (15) is an estimate of how much the regularized candidate model and the true model probability differ. As a final remark, note that in the correlated limit, i.e. for C=ΣC = \Sigma, the expression in eq. (15) reproduces exactly the standard result in eq. (7).

Dataset selection and regularized fits

In this section we show that the definition given in eq. (15), which naturally arises when regularized fits are performed on correlated data, is incompatible with the type of dataset selection problems described in § 2.1.1. The typical case is encountered when some sort of regularization is applied to the covariance matrix of a fitted subset Σ\Sigma, while the covariance matrix of the non-fitted dataset is left untouched. In this case, the expression for AICC\mathrm{AIC}_{C} is not altered with respect to eq. (10). This leads to

AIC‾full=χC2+2nηeff+2nC+log⁡det⁡Cfull+nfulllog⁡2π,(17)\overline{\mathrm{AIC}}_{\mathrm{full}} = \chi_C^2 + 2n_{\eta}^{\mathrm{eff}} + 2n_C + \log\det C_{\mathrm{full}} + n_{\mathrm{full}}\log2\pi, \tag*{(17)}

where Cfull=diag⁡(C,ΣC)C_{\mathrm{full}} = \operatorname{diag}(C,\Sigma_C). In general the term log⁡det⁡Cfull\log\det C_{\mathrm{full}} is no longer common to all the subsets of the full dataset, since CC explicitly depends on the regularization employed on the chosen fitted subset. Therefore, in model and data subset selection procedures which involve regularized fits, by removing terms which are common to the full dataset, the estimator to be used is

AIC‾full=χC2+2nηeff+2nC+log⁡det⁡Cfull,(18)\overline{\mathrm{AIC}}_{\mathrm{full}} = \chi_C^2 + 2n_{\eta}^{\mathrm{eff}} + 2n_C + \log\det C_{\mathrm{full}}, \tag*{(18)}

in contrast to the standard case of correlated fits, where the simpler formula in eq. (12) applies.

As mentioned in § 1, regularizing the covariance matrix induces a loss of information. In the expression above part of this loss of information is encoded in the different determinant appearing in the expression of the AIC and can be easily understood if the applied regularization involves an SVD cut on the smallest eigenvalues of Σ\Sigma in order to make it less ill-conditioned. In this case, it follows that det⁡Σfull<det⁡Cfull\det\Sigma_{\mathrm{full}} < \det C_{\mathrm{full}} and the definition in eq. (18) always promotes models and datasets with a regularized covariance matrix CC closer to Σ\Sigma. The crucial point is that if the null-hypothesis is not rejected, i.e. there is no statistical evidence that the model is misspecified, then θMLEC\theta_{\mathrm{MLE}}^{C} is still a good estimator of the true parameter vector θ0\theta_0. This implies that the information lost in the regularization procedure only plays the role of stabilizing the fit and it is not needed to have a good estimator of the parameter vector. The main reason is that the removed information is poorly estimated since the smallest eigenvalues of the covariance matrix are usually affected by large errors.

Therefore, by construction, the AIC in eq. (18) does not allow one to compare subsets fitted with different regularizations and the standard trade-off between complexity and quality of the fit is no longer the main feature of the AIC. The regularization dependence of eqs. (15) and (18) and its effects are illustrated in § 4.2 on a fixed dataset and in § 4.3 and § 4.4 in the context of dataset and model selection, respectively.

Regularization-independent AIC

We tackle the problem outlined in the previous section by deriving an expression of the Akaike Information Criterion which is, on average, independent of the chosen regularization. This is obtained by defining an estimator of eq. (3) which is able to account for the loss of information due to the regularization of the covariance matrix.

The regularization dependence of the AIC arises from the fact that the expression in eq. (15) has been derived as the unbiased estimator of the regularized version of eq. (3), i.e. computed with covariance matrix CC instead of the standard Σ\Sigma. To cope with this issue, in the following we define a regularization-independent Information Criterion as an estimator of the correlated Δ(Σ∣θ,η)\Delta(\Sigma\mid\theta,\eta). This allows us to absorb the loss of information due to the regularization applied to the covariance matrix into the bias and, therefore, to get an expression which is, on average, regularization-independent. We now take the plug-in estimator θMLEC\theta_{\mathrm{MLE}}^{C}; therefore the Information Criterion reads

RIAIC⁡η=−2log⁡L(y,C∣θMLEC,η)+B(y,C,η),(19)\operatorname{RIAIC}_{\eta}=-2\log\mathcal{L}(y,C\mid\theta_{\mathrm{MLE}}^{C},\eta)+\mathcal{B}(y,C,\eta), \tag*{(19)}

where RIAIC stands for regularization-independent AIC. The crucial step to absorb any regularization-dependent term is taking the regularized log-likelihood in the equation above as an estimator of eq. (3). The bias to be computed reads

B(y,C,η)≡E[2log⁡L(y,C∣θMLEC,η)+Δ(Σ∣θMLEΣ,η)].(20)\mathcal{B}(y,C,\eta)\equiv\mathbb{E}\left[2\log\mathcal{L}(y,C\mid\theta_{\mathrm{MLE}}^{C},\eta)+\Delta(\Sigma\mid\theta_{\mathrm{MLE}}^{\Sigma},\eta)\right]. \tag*{(20)}

Note that the second term on the r.h.s. is exactly the term appearing in the bias of the standard AIC in eq. (6) and is evaluated at the correlated MLE estimator, i.e. θMLEΣ\theta_{\mathrm{MLE}}^{\Sigma}. This is in contrast to eq. (13) where it is evaluated at the regularized MLE.

If the model is correctly specified, under the assumption of normally-distributed data, see App. § 7 for the detailed derivation, the bias reads

B(y,C,η)=nd−ndeff+nη+nηeff+log⁡det⁡Σ−log⁡det⁡C,(21)\mathcal{B}(y,C,\eta)=n_d-n_d^{\mathrm{eff}}+n_{\eta}+n_{\eta}^{\mathrm{eff}}+\log\det\Sigma-\log\det C, \tag*{(21)}

where nηeffn_{\eta}^{\mathrm{eff}} and ndeffn_d^{\mathrm{eff}} have been introduced in eqs. (14) and (16). This leads to the expression of the regularization-independent AIC

RIAIC⁡η=χC2+nd−ndeff+nη+nηeff+ndlog⁡2π+log⁡det⁡Σ.(22)\operatorname{RIAIC}_{\eta}=\chi_C^2+n_d-n_d^{\mathrm{eff}}+n_{\eta}+n_{\eta}^{\mathrm{eff}}+n_d\log2\pi+\log\det\Sigma. \tag*{(22)}

Note that the regularization-dependent term log⁡det⁡C\log\det C appearing in the bias in eq. (21) exactly cancels out the contribution arising from the log-likelihood in eq. (19). Similarly to the AIC in eq. (15), the formulation in eq. (22) reproduces, in the correlated limit C=ΣC = \Sigma, the standard AIC in eq. (7). However, unlike the AIC, see eq. (16), the expectation value of eq. (22) is exactly the one of eq. (7), i.e.

E[RIAICη]=nd+nη+log⁡det⁡Σ+ndlog⁡2π,(23)\mathbb{E}[\mathrm{RIAIC}_{\eta}] = n_d + n_{\eta} + \log\det\Sigma+ n_d\log2\pi, \tag*{(23)}

and it only depends on the non-regularized dataset and on the number of parameters of the model. In this sense, eq. (22) is, on average, regularization-independent and deviations from eq. (7) might arise only due to fluctuations in the quality of the fit. The size of the statistical fluctuations can be quantified by computing the variance, for a given fitted model, of ΔRIAIC=RIAIC−AIC\Delta_{\mathrm{RIAIC}} = \mathrm{RIAIC} - \mathrm{AIC}; see eq. (C.5) for the full expression and App. § 9 for the derivation.

The idea behind eq. (22) is that, provided that the null-hypothesis has not been rejected, the maximum likelihood estimator θMLEC\theta_{\mathrm{MLE}}^{C} is still a good estimator of the true parameter vector θ0\theta_0. Therefore, the regularization of the covariance matrix Σ\Sigma is only relevant for the convergence of the fit but the poorly-estimated information which is lost in the regularization step is irrelevant for the purpose of obtaining a reliable estimator of θ0\theta_0. In practice, care must be taken in choosing the regularization of the covariance matrix. Indeed, this argument relies on the fact that the regularization only removes poorly-estimated information; however, if the regularization removes information which is relevant for a correct estimation of the model parameters, the RIAIC may deviate from the corresponding correlated AIC.

The criterion in eq. (22), being on average regularization-independent, allows the comparison between models fitted on data subsets which are regularized in different ways; see § 4 for some detailed examples.

Extension to dataset selection

As shown in § 2.2.1, once we try to rephrase data subset selection problems in terms of model selection ones, the definition of the AIC fails to provide the usual trade-off criterion between quality of the fit and complexity; see § 4.3 for a detailed example. This issue is solved if the definition in eq. (22) is used instead. As before, we assume that the covariance matrix of the non-fitted dataset ΣC\Sigma_C is not regularized. Under this assumption, the bias correction in eq. (21) absorbs the loss of information due to the regularization of the covariance matrix of the fitted subset and the RIAICfull\mathrm{RIAIC}_{\mathrm{full}} depends only on the determinant of the original covariance matrix Σfull=diag⁡(Σ,ΣC)\Sigma_{\mathrm{full}} = \operatorname{diag}(\Sigma,\Sigma_C), i.e.

RIAICfull=χC2+nd−ndeff+nη+nηeff+2nC+log⁡det⁡Σfull+nfulllog⁡2π,(24)\mathrm{RIAIC}_{\mathrm{full}} = \chi_C^2 + n_d - n_d^{\mathrm{eff}} + n_{\eta} + n_{\eta}^{\mathrm{eff}} + 2n_C + \log\det\Sigma_{\mathrm{full}} + n_{\mathrm{full}}\log2\pi, \tag*{(24)}

where, again, we used the perfect-model prescription for the AIC of the non-fitted dataset, but generalizations to other definitions are possible as well. Therefore, by removing terms which are common to the full dataset and all models we obtain the final expression

RIAICfull=χC2+nd−ndeff+nη+nηeff+2nC,(25)\mathrm{RIAIC}_{\mathrm{full}} = \chi_C^2 + n_d - n_d^{\mathrm{eff}} + n_{\eta} + n_{\eta}^{\mathrm{eff}} + 2n_C, \tag*{(25)}

which does not depend on the log-determinant of the regularized covariance matrix, unlike eq. (18). This expression inherits all the properties of the RIAIC: it reproduces, in the correlated limit, the standard result in eq. (12) and its expectation value is regularization-independent.

Extension to the Bayesian framework

Assume that the a priori knowledge about some of the parameters of the model function η\eta is encoded in nπn_\pi Gaussian priors, with nπ≤nηn_\pi\le n_\eta, characterized by mean θ~\tilde{\theta} and covariance matrix CθC_\theta. Then the posterior probability distribution, up to an overall normalization factor, reads

p(θ∣y)=L(y,C∣θ,η)×π(θ;θ~,Cθ).(26)p(\boldsymbol{\theta}\mid\boldsymbol{y})=\mathcal{L}(\boldsymbol{y},C\mid\boldsymbol{\theta},\eta)\times\pi(\boldsymbol{\theta};\tilde{\boldsymbol{\theta}},C_\theta). \tag*{(26)}

In Refs. [8, 11] different possible extensions of the AIC to the Bayesian framework are presented. Given the inherently frequentist nature of the AIC, here we generalize to our framework what the authors of Ref. [8] call BAIC, namely the Bayesian AIC; however, the extension to other proposals is straightforward. We use as plug-in estimator for the Information Criterion the maximum a posteriori (MAP) estimator obtained from a regularized fit, i.e. θMAPC=argmaxθ{p(θ∣y)}\boldsymbol{\theta}_{\mathrm{MAP}}^{C}=\mathop{\mathrm{argmax}}_{\boldsymbol{\theta}}\{p(\boldsymbol{\theta}\mid\boldsymbol{y})\}. In this way the regularization-independent BAIC (RIBAIC) reads

RIBAICη=−2log⁡L(y,C∣θMAPC,η)+B(y,C,η).(27)\mathrm{RIBAIC}_{\eta}=-2\log\mathcal{L}(\boldsymbol{y},C\mid\boldsymbol{\theta}_{\mathrm{MAP}}^{C},\eta)+\mathcal{B}(\boldsymbol{y},C,\eta). \tag*{(27)}

Similarly to the frequentist case in eq. (20), any regularization-dependent term is removed by the bias

B(y,C,η)≡E[2log⁡L(y,C∣θMAPC,η)+Δ(Σ∣θMLEΣ,η)].(28)\mathcal{B}(\boldsymbol{y},C,\eta)\equiv\mathbb{E}\left[2\log\mathcal{L}(\boldsymbol{y},C\mid\boldsymbol{\theta}_{\mathrm{MAP}}^{C},\eta)+\Delta(\Sigma\mid\boldsymbol{\theta}_{\mathrm{MLE}}^{\Sigma},\eta)\right]. \tag*{(28)}

It is worth noticing that, in contrast to the frequentist case, the estimator θMAPC\boldsymbol{\theta}_{\mathrm{MAP}}^{C} does not minimize the chi-square appearing in the log-likelihood in eq. (27), but rather the so-called augmented chi-square [13], which encompasses both the data and the prior contributions. Nevertheless, if the model, including the priors, is correctly specified, the difference between θMAPC\boldsymbol{\theta}_{\mathrm{MAP}}^{C} and θMLEC\boldsymbol{\theta}_{\mathrm{MLE}}^{C} is expected to be small, leading to a bias correction which is numerically close to the one in eq. (21). The derivation of the full bias is carried out in App. § 8 and leads, for normally-distributed data, to the expression

RIBAICη=χC2+nd−ndeff+nη+n^ηeff+ndlog⁡2π+log⁡det⁡Σ−ε(θ~),(29)\mathrm{RIBAIC}_{\eta}=\chi_{C}^{2}+n_{d}-n_{d}^{\mathrm{eff}}+n_{\eta}+\hat{n}_{\eta}^{\mathrm{eff}}+n_{d}\log2\pi+\log\det\Sigma-\varepsilon(\tilde{\theta}), \tag*{(29)}

where we introduced the effective number of parameters

n^ηeff=2tr⁡[C−1/2ΣC−1/2P1]−tr⁡[C−1/2ΣC−1/2P0],(30)\hat{n}_{\eta}^{\mathrm{eff}}=2\operatorname{tr}\left[C^{-1/2}\Sigma C^{-1/2}P_{1}\right]-\operatorname{tr}\left[C^{-1/2}\Sigma C^{-1/2}P_{0}\right], \tag*{(30)}

where P1P_{1} is simply the Bayesian extension of the projector P\mathcal{P} defined in eq. (A.3) and P0P_{0} has a similar definition, see eqs. (B.7) and (B.8) and App. § 8 for the full derivation. Note that the last term appearing on the r.h.s. of eq. (29) encodes the residual bias induced by a misspecification of the priors. It reads

ε(θ~)=(θ~−θMAPc)TQ−1(θ~−θMAPc),(31)\varepsilon(\tilde{\theta}) = \left(\tilde{\theta} - \theta_{\mathrm{MAP}}^{c}\right)^{T} Q^{-1} \left(\tilde{\theta} - \theta_{\mathrm{MAP}}^{c}\right), \tag*{(31)}

where the matrix Q−1Q^{-1} is defined in eq. (B.9). As already remarked, the underlying assumption of the entire derivation is that the model, and the priors in a Bayesian framework, are correctly specified. In this scenario, ε(θ~)\varepsilon(\tilde{\theta}) provides a small correction and the overall bias in the Bayesian framework is expected to be numerically similar to the one in eq. (21).

As a final remark, note that in the limit of non-informative priors eq. (29) correctly reproduces the RIAIC in eq. (22), while it reproduces the standard AIC if the correlated limit is taken as well. Conversely, if only the correlated limit is taken, eq. (29) reproduces the expression of the BAIC derived in Ref. [8] up to small corrections induced by the difference between the MLE and the MAP estimators, since the expression in Ref. [8], by construction, assumes the equivalence of the two estimators. Similarly to eq. (25), the RIBAIC presented in this section can be readily generalized to dataset selection problems by including the standard term for the non-fitted dataset.

Numerical example

In this section we numerically test the Information Criteria discussed in the previous sections on data obtained from lattice QCD simulations. In particular, we compare the RIAIC and the AIC and use the value of the AIC obtained from the corresponding fully-correlated fit as a benchmark.

Setup

The Information Criteria are tested on the zero-momentum single-nucleon correlation function computed on the D251 ensemble generated by the CLS initiative and already presented in Refs. [14, 15, 16]. The correlator has been computed within the distillation framework on 32 timeslices on Ncfg=209N_{\mathrm{cfg}} = 209 independent gauge field configurations; further details on the ensemble and the number of eigenvectors of the covariant three-dimensional Laplacian operator used are given in Ref. [15]. The subsequent statistical analysis employs the jackknife resampling method. In all the fits we keep tmax⁡/a=31t_{\max}/a = 31 fixed and vary the size of the fitted dataset by varying tmin⁡/at_{\min}/a, i.e. the first timeslice included in the fit.

As fit ansatz we use a multi-state model parameterized as

ηn(t)=A0e−E0t(1+∑j=1n−1∣rj∣2e−ΔEjt).(32)\eta_n(t) = A_0 e^{-E_0 t} \left(1 + \sum_{j=1}^{n-1} |r_j|^2 e^{-\Delta E_j t}\right). \tag*{(32)}

In the following, a model is considered to be correctly specified if the pp-value of the corresponding fit is larger than a threshold, which is set a priori to the value 0.05; if the fit is regularized, the pp-value is computed as in Ref. [3]. Fits which do not satisfy this requirement are therefore discarded.

For each regularized fit, the sample covariance matrix Σ\Sigma of the fitted dataset is replaced by its regularized version CC. The regularized covariance matrix is obtained through an SVD cut which acts on the smallest eigenvalues of Σ\Sigma. In particular, eigenvalues which are smaller than λmax⁡/κˉ\lambda_{\max}/\bar{\kappa}, where λmax⁡\lambda_{\max} is the largest eigenvalue of Σ\Sigma and κˉ\bar{\kappa} is a threshold on the condition number of CC which is chosen a priori, are set to be equal to λmax⁡/κˉ\lambda_{\max}/\bar{\kappa}, such that κ(C)=κˉ\kappa(C)=\bar{\kappa}. Conversely, if κ(Σ)≤κˉ\kappa(\Sigma)\leq\bar{\kappa}, no SVD cut is applied to Σ\Sigma. Note that we work under the same assumptions used in the derivations of the previous sections; therefore, no regularization is applied to the covariance matrix of the non-fitted dataset ΣC\Sigma_C. In § 4.2 the threshold κˉ\bar{\kappa} is varied in order to probe the regularization dependence of the criteria, while in the subsequent sections it is fixed a priori to the condition number of the covariance matrix of the dataset in t∈[25a,31a]t\in[25a,31a], i.e. κˉ=κ(Σ[25a,31a])\bar{\kappa}=\kappa\left(\Sigma_{[25a,31a]}\right), which corresponds to the smallest fitted dataset considered in § 4.4.

The comparison among the different Information Criteria is carried out by employing the expressions in eq. (24) for the RIAIC, eq. (17) for the AIC, and eq. (11) for the standard AIC, with block diagonal covariance matrices, i.e. by neglecting correlations between the fitted and non-fitted subsets, as described in the previous sections.

For the chosen datasets, the resulting covariance matrices, although ill-conditioned, are not singular and fully-correlated fits are still feasible. This allows us to use the AIC obtained from the corresponding correlated fits as a benchmark to test whether the RIAIC reproduces its outcome when regularized fits are performed on the same dataset. In realistic scenarios, e.g. on larger datasets, where fully-correlated fits are not viable, the RIAIC then provides a regularization-independent alternative where the standard correlated AIC cannot be computed. Since the RIAIC is expected to be, on average, independent of the chosen regularization, small deviations compared to the AIC are expected if the overall quality of the fit is the same and if the fit parameters can still be determined reliably. Conversely, larger deviations are expected to appear for the AIC due to its inherent regularization dependence. In the subsequent analysis we, therefore, compare different criteria through the differences ΔIC=IC−AIC\Delta_{\mathrm{IC}}=\mathrm{IC}-\mathrm{AIC} and through the model and subset they select.

The tests are organized as follows. In § 4.2 we study the regularization dependence of the criteria for a fixed dataset and model. In § 4.3 and § 4.4 the criteria are applied to dataset selection and to combined dataset and model selection, respectively. Finally, in § 4.5 we consider fits with Bayesian priors.

Regularization dependence

As a first numerical test we study the regularization dependence of the RIAIC and of the AIC for a fixed dataset and model. We fit the single-nucleon correlator with a single-exponential model, η1\eta_1. As previously mentioned, the benchmark for the comparison is provided by the AIC obtained from correlated fits on the dataset. To this aim, we identify the optimal subset by performing fully-correlated fits with tmax⁡=31at_{\max}=31a and different choices of tmin⁡t_{\min}. Fits with pp-value lower than 0.05 are discarded and the smallest AIC identifies t∈[22a,31a]t\in[22a,31a] as the optimal window for the chosen single-exponential model.

On the same data subset we apply several SVD cuts to its covariance matrix Σ\Sigma, such that κ(Σ)/κ(C)\kappa(\Sigma)/\kappa(C) ranges from 1 to ∼4×103\sim4 \times10^{3}, and perform the corresponding regularized fits. The quality of the fit in this case is assessed as in Ref. [3] and we employ the same threshold used for the correlated case.

In the left panel of Figure 1 we show ΔIC=IC−AIC\Delta_{\mathrm{IC}} = \mathrm{IC} - \mathrm{AIC} as a function of κ(Σ)/κ(C)\kappa(\Sigma)/\kappa(C). For κ(Σ)/κ(C)=1\kappa(\Sigma)/\kappa(C) = 1 both the AIC and the RIAIC reproduce the standard AIC, as expected. When stronger regularizations are applied, the AIC‾\overline{\mathrm{AIC}} increasingly deviates from the AIC, up to ΔIC≃42\Delta_{\mathrm{IC}} \simeq42 for the strongest regularization considered, and the deviation is expected to grow further for even stronger regularizations. This behavior is not surprising; indeed, the AIC‾\overline{\mathrm{AIC}} in eq. (15) is an unbiased estimator of the regularization-dependent Δ(C∣θ,η)\Delta(C \mid\theta,\eta), which measures the distance between the data-generating probability distribution, containing Σ\Sigma, and the fitted regularized distribution, containing CC. The SVD cut acts as a regulator by raising the smallest eigenvalues of Σ\Sigma; this implies that log⁡det⁡C>log⁡det⁡Σ\log\det C > \log\det\Sigma, and, therefore, the AIC‾\overline{\mathrm{AIC}} is dominated by this term as the regularization becomes stronger, which is the behavior observed in Figure 1.

Figure 1. Left panel: deviations of the ICs under study compared to the standard AIC as a function of the regularization expressed in terms of κ(Σ)/κ(C)\kappa(\Sigma)/\kappa(C). Orange points correspond to the RIAIC, while blue points correspond to the AIC. The orange band is the error computed as in App. C. Right panel: ground-state determinations for each fit as a function of κ(Σ)/κ(C)\kappa(\Sigma)/\kappa(C). The horizontal gray band corresponds to the result obtained from a correlated fit. The fitted dataset has been kept fixed to the window t∈[22a,31a]t \in[22a,31a] and the model is the single-exponential one.

On the other hand, the RIAIC is an estimator of the correlated Δ(Σ∣θ,η)\Delta(\Sigma\mid\theta,\eta), which is, by definition, regularization-independent, and only contains the determinant of the covariance matrix Σ\Sigma. As a consequence, its value remains stable across the entire range of regularizations explored and exhibits a deviation ΔIC\Delta_{\mathrm{IC}} which is always below one unit and compatible with zero within the error computed as in eq. (C.5), suggesting that such deviation is mainly due to statistical fluctuations of χC2\chi_C^2 around its expectation value.

In the right panel of Figure 1 we show the obtained ground-state energy estimates. Over the whole range of κ(Σ)/κ(C)\kappa(\Sigma)/\kappa(C), the correlated difference between the ground-state energy from a regularized fit and the one obtained from the corresponding correlated fit is compatible with zero within ∼1\sim1 standard deviation, and its statistical error increases by at most ∼30%\sim30\%. In particular, for κ(Σ)/κ(C)≲102\kappa(\Sigma)/\kappa(C) \lesssim10^2, the standard deviation of the estimated ground-state energy obtained from the regularized fit differs by less than 5% from the correlated result. The latter result, together with the fact that deviations between the RIAIC and the AIC differ from zero by less than one standard deviation, confirms that, for moderate regularizations, the information removed by the SVD cut does not affect the overall final estimate of the ground-state energy. Conversely, at the same values of κ(Σ)/κ(C)\kappa(\Sigma)/\kappa(C), ΔIC\Delta_{\mathrm{IC}} is O(10)O(10) for the AIC and may lead to the wrong identification of the optimal subset or model, or, in general, affects the final estimate in model averaging procedures.

Dataset selection

We fit the single-nucleon correlator used in the previous section with a two-exponential model η2\eta_2, keeping tmax⁡=31at_{\max} = 31a fixed and varying tmin⁡t_{\min}. As before, fits with a pp-value lower than 0.05 are discarded from the analysis. In addition, we exclude from the analysis fits where excited-state contamination cannot be resolved, i.e. when the associated parameters are compatible with zero within one standard deviation. For both correlated and regularized fits, this leads to restricting the analysis to fit windows with 8a≤tmin⁡≤15a8a \le t_{\min} \le15a. The benchmark optimal fit range is identified through the AIC of fully-correlated fits and corresponds to the window starting at tmin⁡=10at_{\min} = 10a.

The AIC and the RIAIC are tested in the context of data subset selection, by repeating the same fits with the regularization described in § 4.1, i.e. with threshold on the condition number of CC given by κˉ=κ(Σ[25a,31a])\bar{\kappa} = \kappa\left(\Sigma_{[25a,31a]}\right). The strength of the regularization, κ(Σ)/κ(C)\kappa(\Sigma)/\kappa(C), increases by lowering tmin⁡t_{\min} from ∼102\sim10^2 at tmin⁡=15at_{\min} = 15a to ∼4×103\sim4 \times10^3 at tmin⁡=8at_{\min} = 8a, i.e. within the range explored in § 4.2.

The effective energy of the single-nucleon correlator is shown in the left panel of Figure 2, together with the optimal correlated double-exponential fit starting at tmin⁡=10at_{\min} = 10a and the estimates of the ground-state energy selected by the ICs under study.

Figure 2. Left panel: the effective energy of the single-nucleon correlator. The lines and bands represent the optimal double-exponential model and the estimated ground-state energy selected by the AIC in a correlated fit (red) and by the RIAIC (orange) and the AIC (blue) in a regularized fit. Upper right panel: ground-state energies obtained for several choices of tmin⁡t_{\min} with correlated (red) and regularized (gray) fits. Middle right panel: tmin⁡t_{\min} plot with the values of the AIC (red), RIAIC (orange) and AIC (blue). Stars mark the optimal tmin⁡t_{\min} according to each criterion. The vertical lines represent the optimal tmin⁡t_{\min} identified using the AIC and the RIAIC (dashed line) and the AIC (dotted line). Lower right panel: the deviation ΔIC\Delta_{\mathrm{IC}} of each IC from the AIC; the orange band is the error associated with the difference between the RIAIC and the AIC; see App. § 9.

In the middle right panel we show the values of the three Information Criteria under study for each tmin⁡t_{\min}. Red points correspond to the AIC obtained from correlated fits, used as a benchmark, while orange and blue points represent the RIAIC and the AIC, respectively. The RIAIC exhibits deviations of a few units from the AIC over the whole range of tmin⁡t_{\min}. An interpretation of such deviations is given by the lower panel on the right, where we explicitly show the difference ΔIC\Delta_{\mathrm{IC}} for both the RIAIC and the AIC. The orange band is the error associated with the deviations of the RIAIC relative to the standard AIC, computed from eq. (C.5). In general, the difference between the RIAIC and the AIC deviates from zero by less than 1.8 standard deviations, and by 1.2 standard deviations for tmin⁡=10at_{\min} = 10a, which is identified as the optimal one by the RIAIC. The RIAIC exhibits a systematic shift to smaller values compared to the AIC. This behavior is understood from the fact that the values of ΔRIAIC\Delta_{\mathrm{RIAIC}} obtained on different fit windows are strongly correlated, with correlation coefficients ρ∈[0.79,0.99]\rho\in[0.79, 0.99] computed from eq. (C.3).

Ground-state energies obtained from regularized and correlated fits are shown in the upper right panel of Figure 2. These are compatible for all tmin⁡t_{\min} and, therefore, deviations between the AIC and the RIAIC can be traced back to fluctuations of χC2\chi^2_C around its expectation value, as discussed in § 4.2. Remarkably, since such deviations are approximately constant in tmin⁡t_{\min}, they do not affect the selection and the RIAIC identifies the same optimal window, starting at tmin⁡=10at_{\min}=10a, obtained by applying the AIC to fully-correlated fits.

Conversely, ΔIC\Delta_{\mathrm{IC}} for the AIC increases monotonically from tmin⁡=15at_{\min}=15a to tmin⁡=8at_{\min}=8a. As discussed in § 4.2, this behavior is due to the fact that the AIC is an estimator of the regularization-dependent Δ(C∣θ,η)\Delta(C\mid\theta,\eta). As a consequence, the AIC selects the largest tmin⁡t_{\min} among those which yield a valid fit, i.e. it promotes subsets which are less regularized and, therefore, loses the usual trade-off between quality of the fit and complexity of the description.

As a final remark, the identification of different optimal data subsets affects the estimate of the ground-state energy as shown in the upper right panel of Figure 2. The correlated difference between the determinations based on the RIAIC and the AIC is compatible with zero within less than 0.2 standard deviations and the two ground-state energy estimates have similar statistical errors. In contrast, the determination obtained with the AIC, despite being compatible with the estimate based on the AIC, has a statistical uncertainty which is almost twice as large.

Dataset and model selection

In this section we test one- and two-exponential models η1\eta_{1} and η2\eta_{2} in eq. (32) for different choices of tmin⁡t_{\min} to identify the optimal subset-model description of the data, based on the Information Criteria under study. As before, we use as benchmark the results based on the AIC obtained from correlated fits and we discard or accept models based on the same criteria discussed in § 4.3. In the correlated analysis, the double-exponential model is preferred for 8a≤tmin⁡≤15a8a \le t_{\min} \le15a and the single-exponential one for 18a≤tmin⁡≤25a18a \le t_{\min} \le25a. For tmin⁡=16a,17at_{\min}=16a,17a no model is accepted based on our criteria. The reason is that, for such windows, single-exponential models are discarded based on their poor quality due to the residual excited-state contamination at low tmin⁡t_{\min}, while double-exponential models cannot clearly resolve the contribution from the first excited state. As in the previous section, the correlated analysis based on the AIC identifies the double-exponential model starting at tmin⁡=10at_{\min}=10a as the optimal subset-model description. We then repeat the same analysis by performing regularized fits on each dataset in order to compare the outcome of the AIC and RIAIC. As before, we regularize the covariance matrix of a given dataset if its condition number is larger than a given threshold set to κˉ=κ(Σ[25a,31a])\bar{\kappa}=\kappa\left(\Sigma_{[25a,31a]}\right). For each tmin⁡t_{\min} we only select the model with the lowest IC.

In the middle right panel of Figure 3 we show the values of the three Information Criteria for each tmin⁡t_{\min}. Let us stress that, for each tmin⁡t_{\min}, both the RIAIC and the AIC identify the same optimal model obtained from the benchmark analysis based on the standard AIC. For 8a≤tmin⁡≤15a8a \le t_{\min} \le15a, the double-exponential model is preferred, and the new analysis reproduces the results seen in § 4.3, with the RIAIC deviating from the AIC by a few units and the AIC deviating by up to ∼55\sim55 units from the AIC. For 18a≤tmin⁡≤25a18a \le t_{\min} \le25a all the ICs identify the single-exponential model as the optimal one and deviations from the AIC are smaller both for the RIAIC and the AIC due to the much milder regularization applied to the covariance matrix of the fitted dataset. The limiting case is achieved for tmin⁡=25at_{\min}=25a, where no regularization is applied and the three ICs coincide. As a consequence, the RIAIC identifies the same subset-model description obtained by applying the AIC to correlated fits, i.e. the double-exponential model starting at tmin⁡=10at_{\min}=10a, whereas the AIC selects the single-exponential model starting at tmin⁡=24at_{\min}=24a. As discussed in § 4.2, this result is mainly due to the regularization dependence of the AIC which penalizes regularized subsets with low tmin⁡t_{\min} and promotes fit windows with larger values of tmin⁡t_{\min} where the regularization has smaller effects, i.e. the regularized covariance matrix CC is closer to the one with intact correlations.

Figure 3. As in Figure 2, but for simultaneous dataset and model selection. Squares and circles denote single- (η1\eta_1) and double-exponential (η2\eta_2) fits, respectively. For each tmin⁡t_{\min}, only the model with the lowest value of the corresponding Information Criterion is shown; the vertical lines mark the optimal subset-model description identified by the AIC and the RIAIC (dashed line) and by the AIC (dotted line).

The lower right panel shows that, for 8a≤tmin⁡≤15a8a \le t_{\min} \le15a, the deviation ΔIC\Delta_{\mathrm{IC}} of the RIAIC coincides with the one discussed in § 4.3, while for tmin⁡≥18at_{\min} \ge18a it is compatible with zero within less than 0.7 standard deviations. As expected, the statistical error shrinks as tmin⁡t_{\min} increases and vanishes at tmin⁡=25at_{\min} = 25a, where C=ΣC = \Sigma; see eq. (C.5).

The impact of the different selections on the ground-state energy is shown in the upper right panel of Figure 3. Regularized and correlated fits yield compatible results for all tmin⁡t_{\min} both for single- and double-exponential models. The determination based on the RIAIC agrees with the one based on the AIC: their correlated difference is compatible with zero within less than 0.2 standard deviations and the two estimates have similar statistical errors. In contrast, the determination based on the AIC, obtained from the single-exponential fit starting at tmin⁡=24at_{\min} = 24a, results in a correlated difference which is ∼1.3\sim1.3 standard deviations away from zero and a statistical uncertainty which is more than 1.5 times larger.

These results show that, when regularizations are applied to the covariance matrix, the AIC may lead to a different subset-model selection. On the other hand, the RIAIC, whose bias accounts for the regularization of the covariance matrix, is expected to reproduce, on average, the selection obtained by applying the AIC to correlated fits.

As a final remark, note that the small deviations between the RIAIC and the AIC may, in principle, lead to different results if the ICs are used as weights in model-averaging procedures. However, these differences are expected to be even larger if the AIC is used instead.

Bayesian fits and prior dependence

In this section we test the RIBAIC defined in eq. (29) on Bayesian fits to the same single-nucleon correlator analyzed in the previous numerical tests. As discussed in § 3.2, in the limit of non-informative priors, the RIBAIC is expected to reproduce the RIAIC in eq. (22) and, if the correlated limit is taken as well, i.e. C=ΣC = \Sigma, it is expected to reproduce the standard AIC. To this aim, we keep fixed both the model and the fitted dataset based on the results of § 4.3 and § 4.4, i.e. we fit the dataset starting at tmin⁡=10at_{\min} = 10a with the double-exponential model, and we vary the width of the priors.

We use Gaussian priors for all the model parameters, i.e. nπ=nη=4n_{\pi} = n_{\eta} = 4 with diagonal prior covariance matrix CθC_{\theta} and priors given by

E0∼N(E^,λE^),A0∼N(A^,λA^),ΔE1∼N(2mπ,2λmπ),r1∼N(1,λ),(33)E_0 \sim\mathcal{N}\left(\hat{E}, \lambda\hat{E}\right), \quad A_0 \sim\mathcal{N}\left(\hat{A}, \lambda\hat{A}\right), \quad\Delta E_1 \sim\mathcal{N}\left(2m_{\pi}, 2\lambda m_{\pi}\right), \quad r_1 \sim\mathcal{N}(1,\lambda), \tag*{(33)}

where the second argument in N(⋅,⋅)\mathcal{N}(\cdot,\cdot) here denotes the standard deviation. In the priors above, E^\hat{E} and A^\hat{A} are estimated from the effective energy at t=23at = 23a by assuming a single-state saturation at the selected timeslice and amπ=0.09am_{\pi} = 0.09; for a more precise estimate of the pion mass on this ensemble see Table II of Ref. [15] and Table 4 of Ref. [17]. The parameter λ\lambda is varied between 10−310^{-3} and 10210^2 to probe the effects of constraining and non-informative priors. For each selection of priors, we perform both fully-correlated and regularized fits with covariance matrix Σ\Sigma and CC respectively, where CC is obtained by applying an SVD cut such that its condition number is equal to κˉ=κΣ[25a,31a]\bar{\kappa} = \kappa_{\Sigma[25a,31a]}. The quality of the Bayesian fit is assessed through the pp-value associated with the augmented chi-square which includes the contribution of the priors [13].

In Figure 4 we show the results of the analysis for both the correlated and the regularized fits. For λ≳10−1\lambda\gtrsim10^{-1} all the fits have pp-value larger than the selected threshold and the RIBAIC approaches the expected non-informative limit, i.e. the standard AIC in the case of fully-correlated fits and the RIAIC in the regularized case. Note that, in contrast to ΔIC\Delta_{\mathrm{IC}} in the previous numerical tests, here δRIBAIC\delta_{\mathrm{RIBAIC}} denotes the difference between the RIBAIC and the corresponding expected limit for λ→∞\lambda\to\infty. In particular, for λ≳0.8\lambda\gtrsim0.8 the deviations of the RIBAIC from the expected non-informative limit are already below 10−310^{-3} and the ground-state energy estimates reported in the lower panels are stable and with a statistical error similar to the one extracted from the correlated fit. In this regime, since the MAP and the MLE estimators are numerically close, the correlated RIBAIC (left column) also reproduces the BAIC of Ref. [8] up to small corrections. In the middle panels we show the term ϵ(θ~)\epsilon(\tilde{\theta}) as a function of λ\lambda for both the correlated (left) and the regularized (right) fits. As expected, when the width of the priors increases, ϵ(θ~)\epsilon(\tilde{\theta}) monotonically decreases and, for priors with λ≳1\lambda\gtrsim1, it is already of the order of 10−410^{-4}.

Figure 4. Results from Bayesian fits to the single-nucleon correlator with the double-exponential model η2\eta_2 on the window t∈[10a,31a]t \in[10a,31a], as a function of the relative width λ\lambda of the priors. Left (right) column: correlated (regularized) fits. Upper panels: deviation of the RIBAIC from the AIC (RIAIC) of correlated (regularized) frequentist fits, δRIBAIC\delta_{\mathrm{RIBAIC}}; the dashed lines mark the AIC (left) and the RIAIC (right), i.e. the expected limits for non-informative priors. Middle panels: the term ε(θ~)\varepsilon(\tilde{\theta}) in eq. (31). Lower panels: ground-state energy estimates. The red band corresponds to the result of the correlated frequentist fit. Open symbols in the gray regions denote fits with Bayesian pp-value lower than 0.05.

For λ≲10−1\lambda\lesssim10^{-1}, fits are prior dominated and do not provide stable results for the fit parameters. The ground-state energy estimate approaches the prior means as λ→0\lambda\to0. Similarly, the RIBAIC increases and, in the same limit, is expected to saturate to the value of χC2\chi^2_C evaluated at the prior means. Note that, even though fits in this region are discarded based on their pp-value, the value of ϵ(θ~)\epsilon(\tilde{\theta}) remains relatively small and does not signal the misspecification of the priors. The main reason, as discussed in App. § 8, is that the MAP estimator is biased if priors are misspecified and therefore the substitution θ0≈θMAPC\theta_0 \approx\theta_{\mathrm{MAP}}^C is no longer justified and may lead to an unreliable estimate of the value of ϵ(θ~)\epsilon(\tilde{\theta}).

These results show that, for moderately wide priors, the RIBAIC of a Bayesian fit and the RIAIC of the corresponding frequentist fit on the same dataset are numerically equivalent. This is due to the fact that, under the assumption that both model and priors are correctly specified, these criteria are estimators of the same correlated quantity, independently of any type of regularization. As a consequence, they can be used in a unified framework, in which frequentist and Bayesian fits, possibly with different regularizations, are compared in model and dataset selection or combined in model-averaging procedures.

Summary

In this work, we have derived an extension of the Akaike Information Criterion which allows one to tackle model and data subset selection problems when regularized fits are performed on correlated data. The main result is represented by the RIAIC and is reported in eq. (22) for standard frequentist fits. The RIAIC allows the comparison between models and subsets when regularizations are applied to the covariance matrix of the fitted dataset. The generalization to maximum a posteriori fits, i.e. the RIBAIC, has been derived as well and is reported in eq. (29). In § 4, we tested the numerical and statistical properties of the RIAIC and of the RIBAIC. We found that, while the naive generalization of the AIC to the case of regularized fits, which throughout this work we called the AIC, systematically favors less regularized datasets, the RIAIC deviates from the AIC of the corresponding correlated fit only by a few units, consistently with the statistical fluctuations in χc2\chi_c^2 around its expectation value; see App. § 9. In the numerical tests that we carried out, this led to the identification of the same optimal model and data subset obtained with the AIC, as expected since the two criteria have the same expectation value.

The RIAIC and the RIBAIC were obtained by defining the Information Criterion as an unbiased estimator of the correlated model-dependent contribution to the KL divergence. Equivalently, the expectation values of the RIAIC and of the RIBAIC are equal to the expectation value of the standard, correlated Akaike Information Criterion. By taking the correlated limit, both the RIAIC and the RIBAIC reproduce standard results in the literature, namely the AIC [1, 2] and, if the MAP and the MLE estimators are numerically close, the BAIC [8], respectively.

These criteria apply in the most general setting, where both regularizations of the covariance matrix and priors are included in the fit. We interpret the use of regularizations at the level of the covariance matrix and the inclusion of priors as a modification of the total amount of information encoded in the data we aim to model. In the first case, regularizing the covariance matrix has the role of removing poorly-estimated information. Conversely, the inclusion of priors provides additional information which effectively acts as a regulator of the fit. In both cases the regularization is properly absorbed into the bias of the Information Criteria in eqs. (22) and (29), leading to an overall result which is, on average, independent of the chosen regularization.

The estimators that have been derived are obtained starting from the Akaike Information Criterion. Nonetheless, the generalization to other Information-Theoretic Criteria is possible.

The application of these results encompasses not only data subset and model selection, but also model averaging procedures as detailed in Ref. [11]. In particular, as shown in § 4.5, for moderately wide priors, the RIBAIC reproduces the numerical value of the RIAIC. As a consequence, frequentist and Bayesian fits performed on the same dataset, with or without regularization, can be compared and combined within a unified framework, either in model and dataset selection or in model averaging procedures.

Acknowledgements

I thank my co-authors of Refs. [15, 16] and colleagues of the Baryon Scattering (BaSc) Collaboration for sharing the single-nucleon correlator used in this work, in particular F. Romero-López for computing perambulators on the D251 ensemble and J.R. Green for the Wick contractions. I would like to thank J. Bulava for comments on an early version of the manuscript and A. Baião Raposo, M. Bruno and M. Ginesi for useful discussions on the topic of this work. My work is supported by the European Research Council (ERC) consolidator grant StrangeScatt-101088506. Part of the analysis in § 4 was performed on Elysium, the HPC cluster of the Ruhr-Universität Bochum, subsidized by the DFG(INST213/1055-1).

The analysis of § 4.5 was performed using the gvar [18] and lsqfit [19] libraries.

RIAIC bias correction

Recall the expression of the bias appearing in the regularization-independent definition of the AIC in eq. (20)

B(y,C,η)≡E[2log⁡L(y,C∣θMLEC,η)+Δ(Σ∣θMLEΣ,η)].(34)B(y,C,\eta) \equiv\mathrm{E}\left[2\log\mathcal{L}\left(y,C \mid\theta_{\mathrm{MLE}}^{C},\eta\right)+\Delta\left(\Sigma\mid\theta_{\mathrm{MLE}}^{\Sigma},\eta\right)\right]. \tag*{(34)}

The derivation in this section follows the presentation in Ref. [4] and generalizes it to the case of regularized fits for the first term appearing in the bias. The same steps, applied to the definition in § 2.2, provide the expression of the bias of the AIC in eq. (15). The first term of the equation above is readily computed by inserting the expression of the log-likelihood, which leads to

E[2log⁡L(y,C∣θMLEC,η)]=−E[χC2(θMLEC)+log⁡det⁡C+ndlog⁡(2π)]=−log⁡det⁡C−ndlog⁡(2π)−tr⁡[C−1Σ]+tr⁡[C−1/2ΣC−1/2P],(35)\begin{aligned} \mathrm{E}\left[2\log\mathcal{L}\left(y,C \mid\theta_{\mathrm{MLE}}^{C},\eta\right)\right] &= -\mathrm{E}\left[\chi_C^2\left(\theta_{\mathrm{MLE}}^{C}\right)+\log\det C+n_d\log(2\pi)\right] \\ &= -\log\det C-n_d\log(2\pi)-\operatorname{tr}\left[C^{-1}\Sigma\right]+\operatorname{tr}\left[C^{-1/2}\Sigma C^{-1/2}\mathcal{P}\right], \tag*{(35)} \end{aligned}

where χC2(θMLEC)\chi_C^2\left(\theta_{\mathrm{MLE}}^{C}\right) is the regularized chi-square variable evaluated at θMLEC\theta_{\mathrm{MLE}}^{C}. In the second line we have used the result from Ref. [3] for the expected value of the chi-square. The projector P\mathcal{P} is defined as

P=C−1/2J(θMLEC)HC−1(θMLEC)JT(θMLEC)C−1/2,(36)\mathcal{P}=C^{-1/2}J\left(\theta_{\mathrm{MLE}}^{C}\right)H_C^{-1}\left(\theta_{\mathrm{MLE}}^{C}\right)J^{T}\left(\theta_{\mathrm{MLE}}^{C}\right)C^{-1/2}, \tag*{(36)}

where we have introduced the estimated nη×nηn_{\eta}\times n_{\eta} inverse covariance matrix of the MLE parameters, associated with the regularized matrix CC, as

HC(θMLEC)=JTC−1J∣θMLEC,(37)H_C\left(\theta_{\mathrm{MLE}}^{C}\right)=J^{T}C^{-1}J\big|_{\theta_{\mathrm{MLE}}^{C}}, \tag*{(37)}

and JJ is the Jacobian of the model function with respect to the model parameters θ\theta. As in Ref. [3], the unknown θ0\theta_0 is replaced in practice by the estimator θMLEC\theta_{\mathrm{MLE}}^{C}. Therefore, by introducing the effective number of parameters nηeffn_{\eta}^{\mathrm{eff}} as in eq. (14) and ndeff=tr⁡[C−1Σ]n_d^{\mathrm{eff}}=\operatorname{tr}\left[C^{-1}\Sigma\right], the first term of the bias reads

E[2log⁡L(y,C∣θMLEC,η)]=−log⁡det⁡C−ndlog⁡(2π)−ndeff+nηeff.(38)\mathrm{E}\left[2\log\mathcal{L}\left(y,C \mid\theta_{\mathrm{MLE}}^{C},\eta\right)\right]=-\log\det C-n_d\log(2\pi)-n_d^{\mathrm{eff}}+n_{\eta}^{\mathrm{eff}}. \tag*{(38)}

The second term of the bias is evaluated at θMLEΣ\theta_{\mathrm{MLE}}^{\Sigma}, i.e. the MLE parameter obtained from a correlated fit, which is in principle not accessible if Σ\Sigma is close-to-singular. By inserting the expression in eq. (3), it can be written as

E[Δ(Σ∣θMLEΣ,η)]=E[E[−2log⁡L(y,Σ∣θ,η)]∣θMLEΣ].(39)\mathrm{E}\left[\Delta\left(\Sigma\mid\theta_{\mathrm{MLE}}^{\Sigma},\eta\right)\right]=\mathrm{E}\left[\left.\mathrm{E}\left[-2\log\mathcal{L}(y,\Sigma\mid\theta,\eta)\right]\right|_{\theta_{\mathrm{MLE}}^{\Sigma}}\right]. \tag*{(39)}

By expanding the likelihood, the innermost expectation value reads

E[−2log⁡L(y,Σ∣θ,η)]∣θMLEΣ=E[χΣ2(θ)]∣θMLEΣ+log⁡det⁡Σ+ndlog⁡(2π),(40)\mathbb{E}\left[-2\log\mathcal{L}(y,\Sigma\mid\theta,\eta)\right]\big|_{\theta^\Sigma_{\mathrm{MLE}}}=\mathbb{E}\left[\chi^2_\Sigma(\theta)\right]\big|_{\theta^\Sigma_{\mathrm{MLE}}}+\log\det\Sigma+n_d\log(2\pi), \tag*{(40)}

where the expectation value of the chi-square is taken before evaluating it at the MLE parameter θMLEΣ\theta^\Sigma_{\mathrm{MLE}}. In order to compute the expectation value appearing on the r.h.s. of eq. (A.7) we Taylor-expand the model function η\eta around the true value θ0\theta_0. By using the standard result for the expectation value of quadratic forms and recalling that E[y]=η(θ0)\mathbb{E}[y]=\eta(\theta_0), the expected value on the r.h.s. reads

E[χΣ2(θ)]∣θMLEΣ=nd+δθTJT(θ0)Σ−1J(θ0)δθ,(41)\mathbb{E}\left[\chi^2_\Sigma(\theta)\right]\big|_{\theta^\Sigma_{\mathrm{MLE}}}=n_d+\delta\theta^T J^T(\theta_0)\Sigma^{-1}J(\theta_0)\delta\theta, \tag*{(41)}

where we introduced δθ≡θMLEΣ−θ0\delta\theta\equiv\theta^\Sigma_{\mathrm{MLE}}-\theta_0. By combining the expression above with eqs. (A.6) and (A.7), the second term of the bias reads

E[Δ(Σ∣θMLEΣ,η)]=log⁡det⁡Σ+ndlog⁡(2π)+nd+E[δθTHΣ(θ0)δθ],(42)\mathbb{E}\left[\Delta(\Sigma\mid\theta^\Sigma_{\mathrm{MLE}},\eta)\right]=\log\det\Sigma+n_d\log(2\pi)+n_d+\mathbb{E}\left[\delta\theta^T H_\Sigma(\theta_0)\delta\theta\right], \tag*{(42)}

where we defined HΣH_\Sigma according to eq. (A.4). The remaining expectation value can be readily computed by using the condition at the minimum for χΣ2(θ)\chi^2_\Sigma(\theta), which leads to the relation

θMLEΣ−θ0=HΣ−1(θ0)JT(θ0)Σ−1(y−η(θ0)).(43)\theta^\Sigma_{\mathrm{MLE}}-\theta_0=H_\Sigma^{-1}(\theta_0)J^T(\theta_0)\Sigma^{-1}\left(y-\eta(\theta_0)\right). \tag*{(43)}

By using this relation and the standard results for quadratic forms, the expectation value on the r.h.s. reduces to the total number of parameters of the model function η\eta [4, 12]. Therefore the second term of the bias in eq. (20) can be written in the compact form

E[Δ(Σ∣θMLEΣ,η)]=log⁡det⁡Σ+ndlog⁡(2π)+nd+nη.(44)\mathbb{E}\left[\Delta(\Sigma\mid\theta^\Sigma_{\mathrm{MLE}},\eta)\right]=\log\det\Sigma+n_d\log(2\pi)+n_d+n_\eta. \tag*{(44)}

Finally, by combining eqs. (A.2) and (A.11), the bias takes the form in eq. (21) and the final expression for the regularization-independent criterion is the one reported in eq. (22). To conclude this section, notice that in the correlated limit the trace involving the projector P\mathcal{P} reduces to the total number of parameters and the expression of the bias reproduces the bias of the standard correlated AIC, which reads, up to model-independent terms, 2nη2n_\eta.

RIBAIC bias correction

The bias of the Bayesian extension in eq. (27) can be computed similarly to App. § 7. It reads

B(y,C,η)≡E[2log⁡L(y,C∣θMAPC,η)+Δ(Σ∣θMLEΣ,η)].(45)\mathcal{B}(y,C,\eta)\equiv\mathbb{E}\left[2\log\mathcal{L}(y,C\mid\theta^C_{\mathrm{MAP}},\eta)+\Delta(\Sigma\mid\theta^\Sigma_{\mathrm{MLE}},\eta)\right]. \tag*{(45)}

The second term coincides with eq. (A.11). Under the assumption of normally-distributed data, the first term reads

E[2log⁡L(y,C∣θMAPC,η)]=−E[χC2(θMAPC)]−ndlog⁡2π−log⁡det⁡C.(46)\mathbb{E}\left[2\log\mathcal{L}(y,C\mid\theta^C_{\mathrm{MAP}},\eta)\right]=-\mathbb{E}\left[\chi^2_C(\theta^C_{\mathrm{MAP}})\right]-n_d\log2\pi-\log\det C. \tag*{(46)}

By expanding around the true value θ0\theta_0, the expectation value of the chi-square in the expression above can be written as

E[χC2(θMAPC)]=tr⁡[C−1Σ]−2E[δθTJT(θ0)C−1(y−η(θ0))]+E[δθTHC(θ0)δθ](47)\mathbb{E}\left[\chi_C^2\left(\theta_{\mathrm{MAP}}^C\right)\right] = \operatorname{tr}\left[C^{-1}\Sigma\right] - 2\mathbb{E}\left[\delta\theta^T J^T(\theta_0)C^{-1}\left(y-\eta(\theta_0)\right)\right] + \mathbb{E}\left[\delta\theta^T H_C(\theta_0)\delta\theta\right] \tag*{(47)}

where δθ=θMAPC−θ0\delta\theta= \theta_{\mathrm{MAP}}^C-\theta_0 and we used the definition in eq. (A.4). Note that for the first term on the r.h.s. we used the result for the expectation values of quadratic forms. The remaining two terms are computed analogously, by using the condition at the minimum of the augmented chi-square [13]. The condition at the minimum leads to the relation

θMAPC−θ0=HC−1(θ0)JT(θ0)C−1(y−η(θ0))+HC−1(θ0)Cθ−1(θ~−θ0).(48)\theta_{\mathrm{MAP}}^C-\theta_0 = \mathcal{H}_C^{-1}(\theta_0)J^T(\theta_0)C^{-1}\left(y-\eta(\theta_0)\right) + \mathcal{H}_C^{-1}(\theta_0)C_\theta^{-1}\left(\tilde{\theta}-\theta_0\right). \tag*{(48)}

Let us stress that the equation above differs from eq. (A.10) due to the presence of priors and because θMAPC\theta_{\mathrm{MAP}}^C does not directly minimize the chi-square of the data appearing in the bias, but rather it minimizes the augmented chi-square. In analogy to eq. (A.4), in eq. (B.4) we introduced

HC(θ)=HC(θ)+Cθ−1,(49)\mathcal{H}_C(\theta) = H_C(\theta) + C_\theta^{-1}, \tag*{(49)}

where CθC_\theta is the prior covariance matrix. If nπ<nηn_\pi< n_\eta, Cθ−1C_\theta^{-1} is understood to be the nη×nηn_\eta\times n_\eta matrix obtained by inverting the nπ×nπn_\pi\times n_\pi prior covariance matrix and setting to zero the rows and columns associated with those parameters without priors. By using eq. (B.4) in the expectation value in eq. (B.3), and by taking advantage of the well-known result for quadratic forms, the bias term in eq. (B.2) simplifies to

E[2log⁡L(y,C∣θMAPC,η)]=−ndlog⁡2π−log⁡det⁡C−tr⁡[C−1Σ]−tr⁡[C−1/2ΣC−1/2P0]+2tr⁡[C−1/2ΣC−1/2P1]−(θ~−θ0)TQ−1(θ~−θ0).(50)\mathbb{E}\left[2\log\mathcal{L}\left(y,C\mid\theta_{\mathrm{MAP}}^C,\eta\right)\right] = -n_d\log2\pi-\log\det C-\operatorname{tr}\left[C^{-1}\Sigma\right] - \operatorname{tr}\left[C^{-1/2}\Sigma C^{-1/2}\mathcal{P}_0\right] + 2\operatorname{tr}\left[C^{-1/2}\Sigma C^{-1/2}\mathcal{P}_1\right] - \left(\tilde{\theta}-\theta_0\right)^TQ^{-1}\left(\tilde{\theta}-\theta_0\right). \tag*{(50)}

Here, similarly to the projector in eq. (A.3), we introduced the matrices

P0=C−1/2JHC−1HCHC−1JTC−1/2∣θ0,(51)\mathcal{P}_0 = C^{-1/2}J\mathcal{H}_C^{-1}H_C\mathcal{H}_C^{-1}J^TC^{-1/2}\big|_{\theta_0}, \tag*{(51)}
P1=C−1/2JHC−1JTC−1/2∣θ0,(52)\mathcal{P}_1 = C^{-1/2}J\mathcal{H}_C^{-1}J^TC^{-1/2}\big|_{\theta_0}, \tag*{(52)}
Q−1=Cθ−1HC−1HCHC−1Cθ−1∣θ0.(53)Q^{-1} = C_\theta^{-1}\mathcal{H}_C^{-1}H_C\mathcal{H}_C^{-1}C_\theta^{-1}\big|_{\theta_0}. \tag*{(53)}

In practice, to compute these terms instead of using the true parameter vector θ0\theta_0, which is unknown, we use the estimator θMAPC\theta_{\mathrm{MAP}}^C. This substitution is justified when priors are either wide or centered close to the true values of the parameters. If priors are misspecified, the MAP estimator is biased and the resulting estimate of ε(θ~)\varepsilon(\tilde{\theta}) may be unreliable. However, the entire derivation relies on the assumption that the model, including priors, is correctly specified, which is checked through the quality of the fit before computing the RIBAIC. Fits with misspecified priors are therefore expected to be discarded based on their quality.

Finally, by combining the expression above with eq. (A.11) we obtain the explicit expression of the bias defined in eq. (28) and consequently the Information Criterion in eq. (29). As a final remark, note that by taking the limit of flat priors, both P0\mathcal{P}_0 and P1\mathcal{P}_1 reduce to the projector in eq. (A.3), while the term involving Q−1\mathcal{Q}^{-1} vanishes and the overall expression for the bias reproduces the one of the RIAIC derived in App. § 7.

Variance of Delta RIAIC

In this appendix we investigate the statistical properties of

Δi=RIAICi−AICi=δXi−E[δXi],(54)\Delta_i = \mathrm{RIAIC}_i - \mathrm{AIC}_i = \delta X_i - \mathrm{E}\left[\delta X_i\right], \tag*{(54)}

where we used the shorthand notation Δ≡ΔRIAIC\Delta\equiv\Delta_{\mathrm{RIAIC}} and, in contrast to the notation of the main text, we introduced the label ii which identifies the iith fitted dataset. Note that in the last step on the r.h.s. we explicitly used the expressions in eqs. (22) and (7) and introduced, for a fixed fitted dataset, δXi≡χCi2−χΣi2\delta X_i \equiv\chi^2_{C_i} - \chi^2_{\Sigma_i}. Here CiC_i and Σi\Sigma_i denote the regularized and the fully-correlated covariance matrices of the iith dataset respectively. Similarly to the main text, the expectation value on the r.h.s. is taken over the true probability distribution of the fitted dataset. From eq. (C.1), Δi\Delta_i has vanishing mean; therefore, given two fitted datasets, the resulting covariance between Δi\Delta_i and Δj\Delta_j is

Cov⁡[Δi,Δj]=E[ΔiΔj]=E[δXiδXj]−E[δXi]E[δXj],(55)\operatorname{Cov}\left[\Delta_i,\Delta_j\right] = \mathrm{E}\left[\Delta_i\Delta_j\right] = \mathrm{E}\left[\delta X_i\delta X_j\right] - \mathrm{E}\left[\delta X_i\right]\mathrm{E}\left[\delta X_j\right], \tag*{(55)}

where in the last step we used the definition in eq. (C.1) and expanded the expectation values. As before, we work under the assumptions that the model is correctly specified and that data follow a multivariate Gaussian distribution. Following the steps of Apps. § 7 and § 8, by expanding the chi-square variables in δXi\delta X_i and δXj\delta X_j around the true value η(θ0)\eta(\theta_0) and recalling that E[y]=η(θ0)\mathrm{E}[\boldsymbol{y}] = \eta(\theta_0) for both ii and jj, standard results for the second and fourth moments of a multivariate normal distribution lead to

Cov⁡[Δi,Δj]=2tr⁡[KiΣijKjΣji],(56)\operatorname{Cov}\left[\Delta_i,\Delta_j\right] = 2\operatorname{tr}\left[K_i\Sigma_{ij}K_j\Sigma_{ji}\right], \tag*{(56)}

where Σij=ΣjiT\Sigma_{ij} = \Sigma_{ji}^{\mathsf{T}} is an ndi×ndjn_{d_i} \times n_{d_j} matrix which contains the correlations between the two fitted datasets and, by using the notation of App. § 7,

Ki≡Ci−1/2[1−Pi]Ci−1/2−Σi−1/2[1−Σi−1/2JiHi−1JiTΣi−1/2]Σi−1/2,(57)K_i \equiv C_i^{-1/2}\left[1-\mathcal{P}_i\right]C_i^{-1/2} - \Sigma_i^{-1/2}\left[1-\Sigma_i^{-1/2}J_iH_i^{-1}J_i^{\mathsf{T}}\Sigma_i^{-1/2}\right]\Sigma_i^{-1/2}, \tag*{(57)}

where Pi\mathcal{P}_i is the projector in eq. (A.3) and JiJ_i is the Jacobian, both restricted to the iith dataset. The variance of Δi\Delta_i on the iith fitted subset is obtained from the diagonal term Cov⁡[Δi,Δi]\operatorname{Cov}[\Delta_i,\Delta_i] and can be written in the compact form

Var⁡[Δi]=2tr⁡{[(1−Pi)(Ci−1/2ΣiCi−1/2−1)]2}.(58)\operatorname{Var}\left[\Delta_i\right] = 2\operatorname{tr}\left\{\left[\left(1-\mathcal{P}_i\right)\left(C_i^{-1/2}\Sigma_iC_i^{-1/2}-1\right)\right]^2\right\}. \tag*{(58)}

Note that, as expected, in the correlated limit, i.e. for Ci=ΣiC_i = \Sigma_i, Δi\Delta_i has vanishing variance since it is identically zero.

Software implementation

A Python implementation of the Information Criteria introduced in this work is publicly available at https://github.com/laudid46/pyaic under the GPL-3.0 license.

References

  1. [1]H. Akaike. “A new look at the statistical model identification”. In: IEEE Trans. Automatic Control 19.6 (1974), pp. 716–723. DOI: 10.1109/TAC.1974.1100705.DOI
  2. [2]Hirotugu Akaike. “Information Theory and an Extension of the Maximum Likelihood Principle”. In: Selected Papers of Hirotugu Akaike. Ed. by Emanuel Parzen, Kunio Tanabe, and Genshiro Kitagawa. New York, NY: Springer New York, 1998, pp. 199–213. ISBN: 978-1-4612-1694-0. DOI: 10.1007/978-1-4612-1694-0_15.DOI
  3. [3]Mattia Bruno and Rainer Sommer. “On fits to correlated and auto-correlated data”. In: Comput. Phys. Commun. 285 (2023), p. 108643. DOI: 10.1016/j.cpc.2022.108643. arXiv: 2209.14188 [hep-lat].DOI
  4. [4]Joseph E. Cavanaugh and Andrew A. Neath. “The Akaike information criterion: Background, derivation, properties, application, interpretation, and refinements”. In: WIREs Computational Statistics 11.3 (2019), e1460. DOI: 10.1002/wics.1460.DOI
  5. [5]B. Efron. “Bootstrap Methods: Another Look at the Jackknife”. In: The Annals of Statistics 7.1 (1979), pp. 1 –26. DOI: 10.1214/aos/1176344552.DOI
  6. [6]Ulli Wolff. “Monte Carlo errors with less errors”. In: Comput. Phys. Commun. 156 (2004). [Erratum: Comput.Phys.Commun. 176, 383 (2007)], pp. 143–153. DOI: 10.1016/S0010-4655(03)00467–3. arXiv: hep-lat/0306017.arxiv.org/abs/hep-lat/0306017
  7. [7]S. Kullback and R. A. Leibler. “On Information and Sufficiency”. In: The Annals of Mathematical Statistics 22.1 (1951), pp. 79 –86. DOI: 10.1214/aoms/1177729694.DOI
  8. [8]Ethan T. Neil and Jacob W. Sitsison. “Improved information criteria for Bayesian model averaging in lattice field theory”. In: Phys. Rev. D 109.1 (2024), p. 014510. DOI: 10.1103/PhysRevD.109.014510. arXiv: 2208.14983 [stat.ME].DOI
  9. [9]Ethan T. Neil and Jacob W. Sitsison. “Model averaging approaches to data subset selection”. In: Phys. Rev. E 108.4 (2023), p. 045308. DOI: 10.1103/PhysRevE.108.045308. arXiv: 2305.19417 [stat.ME].DOI
  10. [10]Sz. Borsanyi et al. “Leading hadronic contribution to the muon magnetic moment from lattice QCD”. In: Nature 593.7857 (2021), pp. 51–55. DOI: 10.1038/s41586-021-03418-1. arXiv: 2002.12347 [hep-lat].DOI
  11. [11]William I. Jay and Ethan T. Neil. “Bayesian model averaging for analysis of lattice field theory results”. In: Phys. Rev. D 103 (2021), p. 114502. DOI: 10.1103/PhysRevD.103.114502. arXiv: 2008.01069 [stat.ME].DOI
  12. [12]Julien Frison. Towards fully bayesian analyses in Lattice QCD. 2023. arXiv: 2302.06550 [hep-lat].arxiv.org/abs/2302.06550
  13. [13]G. P. Lepage et al. “Constrained curve fitting”. In: Nucl. Phys. B Proc. Suppl. 106 (2002). Ed. by M. Muller-Preussker et al., pp. 12–20. DOI: 10.1016/S0920-5632(01)01638-3. arXiv: hep-lat/0110175.DOI
  14. [14]André Baião Raposo et al. “Preliminary study of the H dibaryon in N_f = 2 + 1 lattice QCD”. In: 42nd International Symposium on Lattice Field Theory. Feb. 2026. arXiv: 2603.00698 [hep-lat].arxiv.org/abs/2603.00698
  15. [15]Jorge Baeza-Ballesteros et al. “Two-nucleon scattering amplitudes at light quark masses from lattice QCD”. In: (Oct. 2026). arXiv: 2610.23332 [hep-lat].
  16. [16]Jorge Baeza-Ballesteros et al. “Quark-mass dependence of two-nucleon scattering”. In: (Nov. 2026). arXiv: 2611.07864 [hep-lat].
  17. [17]Simon Kuberski et al. “D and D_s decay constants in N_f = 2 + 1 QCD with Wilson fermions”. In: JHEP 07 (2024), p. 090. DOI: 10.1007/JHEP07(2024)090. arXiv: 2405.04506 [hep-lat].DOI
  18. [18]Peter Lepage. gplepage/gvar. https://github.com/gplepage/gvar. Apr. 2020.
  19. [19]Peter Lepage. gplepage/lsqfit. https://github.com/gplepage/lsqfit. Apr. 2020.

Paper details

vlnc:W_2e8s2p9e
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Physics and Astronomy

Not recorded for this paper

View statement
Looking for related papers…
Contents