WorksPhysics and Astronomy
A regularization-independent Information Criterion for correlated data
Abstract
We generalize the Akaike Information Criterion (AIC) to allow model and data subset selection when regularized fits are performed on correlated data. While the standard generalization to regularized fits is inherently dependent on the type of regularization applied to the data covariance matrix, our definition, the RIAIC, is, on average, regularization-independent. The regularization-independence is achieved by defining the new Information Criterion as an estimator of the model-dependent contribution to the correlated Kullback–Leibler divergence. The Bayesian extension is derived as well. We test the new criterion on lattice QCD data by performing regularized fits and show that it reproduces the selection properties of the standard correlated AIC, with fluctuations which are compatible with the expected statistical uncertainties. These results enable the application of subset and model selection techniques to cases where regularizations are applied to the covariance matrix of the fitted dataset.
Introduction
The goal of identifying the optimal description of a dataset plays a pivotal role in both experimental and theoretical data analysis. By optimal we refer to a parameterization that simultaneously maximizes the information extracted from the data while minimizing the complexity of the model itself. In addition to the optimal parameterization for a fixed dataset, comparisons of different data subsets are also necessary. More generally, model and data subset selection provides a valuable tool to assess and quantify systematic uncertainties arising from arbitrary choices on the model and on the fitted dataset.
Information-Theoretic Criteria (ICs) are a widely used approach to model selection problems. They are theoretically sound, since their definition is rooted in information theory through the Kullback–Leibler divergence, and, by construction, they represent a trade-off between quality of the fit and complexity of the description.
One commonly used IC is the Akaike Information Criterion (AIC) [1, 2]. The standard definition of the AIC explicitly assumes that correlations in the dataset under study are properly taken into account.
However, it is often the case that covariance matrices are estimated on a limited, and noisy, sample of data. This may lead to close-to-singular covariance matrices, for which minimization procedures, such as least squares methods, become impractical. In this context, a possible solution to extract additional information from the data is to perform fits by employing regularized versions of the covariance matrix. Typical regularizations include the application of SVD cuts or the limiting case of a fully uncorrelated fit with a diagonal covariance matrix.
While a proper treatment of the covariance matrix and the impact of regularizations can be addressed at the level of the quality of the fit, by redefining the expectation values of chi-square variables, see Ref. [3], accounting for these effects at the level of the AIC still represents a gap which prevents a reliable comparison of models and subsets in the case of regularized fits. In general, ICs are rooted in information theory and regularizations of the covariance matrix effectively modify the amount of information contained in the selected dataset, by removing poorly determined information. Similarly, in a Bayesian framework, the use of priors modifies the information used as input to the fit, by adding knowledge on the system under study. On the basis of these considerations, the application of any regularization to the data covariance matrix introduces a dependence on the regularization itself in the AIC. As such, a systematic comparison of models and datasets when fits are performed with regularized covariance matrices is still not possible. This is particularly relevant when the size of the fitted dataset is varied, since the regularization required to obtain a stable fit typically changes with it. A typical example in the analysis of lattice field theory data is represented by the search for the optimal fit window of a correlation function, where different windows may require different regularizations, mostly depending on the size of the window itself. Using the approach proposed here, and detailed in the following sections, the regularization of the covariance matrix, which is necessary to stabilize the fit, is incorporated into the AIC.
This work is organized as follows. In § 2 we briefly review the idea behind Information Criteria, by focusing on the definition of the Akaike Information Criterion and its extension to dataset selection. In the same section, we report the standard generalization of the AIC to regularized fits and we show that its extension to subset selection problems induces an explicit dependence on the regularization which makes it unreliable. In § 3 we propose a new Information Criterion which is explicitly built to be, on average, regularization-independent. Finally, in § 4 we explicitly carry out numerical tests on actual lattice data to validate the formulae derived in this work.
The main results of this work are represented by the expressions in eqs. (22) and (25), together with their Bayesian extension in eq. (29).
Throughout this work we often refer to examples or typical situations encountered in lattice field theory studies, e.g. the identification of the optimal fit window in the extraction of the energy spectrum from Euclidean lattice correlators. Nevertheless the formulae derived here apply to any situation in which the data follow a multivariate Gaussian distribution.
Information-Theoretic Criteria
Given a dataset , and a model function which depends on the parameter vector and, optionally, on some other variables, let us consider a set of candidate model probabilities, i.e. a set of probability density functions [4]
where is the parameter space for a given model function and . The true covariance matrix is in most cases unknown and it is usually replaced by the sample covariance matrix [5, 6] which provides a consistent and asymptotically unbiased estimator of .
Given a candidate probability density function and the data-generating unknown distribution of the dataset , we define the corresponding Kullback–Leibler (KL) divergence [7] as
where in the last line denotes the expectation value over the true probability distribution . Note that if and only if . Therefore the empirical description that best approximates the true probability distribution is found by minimizing the KL divergence, and eq. (2) can be interpreted as a measure of how large the deviations between the candidate and the true distribution are.
The first term appearing in eq. (2), is, by definition, independent of the candidate model. Therefore, minimizing the Kullback–Leibler divergence with respect to the parameter vector is equivalent to minimizing the model-dependent quantity
where the additional factor of 2 is conventional. Let us stress that the expression above depends on the data-generating distribution and it is, therefore, not directly accessible. However, given an estimator for the parameter vector , Information Criteria are defined as unbiased plug-in [8, 9] estimators of eq. (3) as
where is a bias term that depends on the dataset and on the parameter space associated with the candidate model and its expression depends on the choice of the plug-in estimator used in eq. (4). In eq. (4) the subscript indicates the model function. As usual, the bias is computed as the expectation value of the difference between the biased estimator and the target quantity, i.e.
As we will see in the following, the bias in eq. (5) usually plays the role of a penalty term which penalizes overly complex models and favors simpler parameterizations for a given dataset; in this sense, the bias naturally discourages overfitting.
Akaike Information Criterion
In this section we briefly review the standard definition of the Akaike Information Criterion (AIC) [1, 2]. For a detailed introduction to the topic we refer to Ref. [4]. In the following we work under the assumption that the model is correctly or over-specified. In practice, a model can be considered correctly specified if the corresponding fit is accepted based on its quality, e.g. if the corresponding -value is above a given threshold. For regularized fits the quality of the fit must be computed as in Ref. [3]. In this case the data-generating distribution belongs to the set and can be written as , where is the true value of the parameter vector.
The standard definition of the AIC assumes as plug-in estimator for the parameter vector the maximum likelihood estimator (MLE) which is asymptotically unbiased. Following the definitions given in the previous section, the negative log-likelihood evaluated at the MLE parameter defines an Information Criterion according to eq. (4) with bias computed as in eq. (5). In Refs. [1, 2] it was shown that the bias term has a particularly simple expression and depends only on the overall dimension of the parameter space , leading to
For a detailed derivation of the bias see Ref. [4]. Therefore an unbiased estimator for eq. (3) is given by the so-called Akaike Information Criterion [1, 2], which is defined as
Under the assumption that both the data-generating and the candidate model probabilities are multivariate Gaussian distributions, the AIC can be written, up to terms which are constant across all models, in the standard way as
where is the chi-square evaluated at the MLE parameters and the subscript makes explicit the covariance matrix used in its definition. Therefore, smaller values of the AIC correspond to smaller values of eq. (3) and, as a consequence, a better agreement between the candidate model and the true probability distribution. Note that, in general, it holds . Therefore parameterizations with a large number of parameters will be penalized, if the quality of the fit is similar. In this sense, the AIC is a trade-off between the quality of the fit and the complexity of the model.
Model and dataset selection
The problem of data subset selection, i.e. the problem of finding the most informative data subset which can be described by the simplest model, can be rephrased in terms of model selection problems as detailed in Refs. [8, 9].
The core idea is that a given dataset, which hereafter we refer to as the full dataset, can be split into two different subsets: one which is included in the actual fit and one which is excluded. Under the assumption that the fitted and non-fitted datasets are statistically independent, the total KL divergence and, as a consequence, eq. (3) factorize into a sum of two independent terms 2. This leads to a factorization of the corresponding AIC as
where the first term on the r.h.s. corresponds to the AIC of the fitted dataset where the model function has been used to describe the data and is the AIC of the non-fitted dataset3.
In the literature different definitions for have been proposed, see Refs. [10, 11] and Ref. [9] for a detailed comparison. The results presented here are independent of the chosen definition; nevertheless, in the following we will always assume that the perfect model definition is employed [11]. The generalization to other definitions is straightforward.
The perfect model is a model for the non-fitted dataset with as many parameters as data points, e.g. a polynomial interpolant. Therefore, under the Gaussian approximation of the non-fitted dataset, since the perfect model provides a vanishing chi-square, the AIC associated with the non-fitted subset reduces to
where the first term on the r.h.s. is twice the number of parameters of the perfect model, which is, by definition, equal to the number of data points of the non-fitted dataset and, most importantly, it does not depend on the explicit functional form of the perfect model. As a consequence, for normally-distributed data the AIC of the full dataset can be written as
where and we employed the fact that the fitted and non-fitted data subsets are statistically uncorrelated, i.e. . Finally, by neglecting terms which are common to all models and to the full dataset, it simplifies to the known result [9, 11]
For a specific full dataset and model function , eq. (12) can be used to extend model selection problems to subset selection ones. Specifically, the use of will automatically promote models which provide a good description of a data subset which is as large as possible, with a model depending on a number of parameters which is as small as possible.
Akaike Information Criterion and regularized fits
The definition of the AIC given in eq. (7) can be easily extended to the scenario where fits with regularized covariance matrices are performed on correlated, normally distributed, data. This result is reported in Ref. [12] and in this section we briefly review the framework in which it is derived.
As before, we assume that the model is correctly specified and the true probability distribution is a multivariate normal distribution , where is the true value of the parameter vector. If the covariance matrix is numerically ill-conditioned, it is common practice to perform the fit using a regularized matrix. Therefore, the plug-in estimator used to estimate eq. (3) is which is, unlike , obtained by maximizing the log-likelihood using a generic positive-definite matrix whose elements are , where is the size of the Monte Carlo sample. Note that, in general, these two estimators might differ and their difference depends on how much the regularized covariance matrix differs from .
In this context, we are testing model probabilities which are no longer in the set but in the much larger set which contains as the specific case of models based on correlated fits. By defining the estimator of the regularized model-dependent term of the KL-divergence as in eq. (3), the bias to be computed is
where, in analogy with the standard expression in eq. (7), we introduced the effective number of parameters as
where the projector is defined in Ref. [3] and satisfies , see eq. (A.3). By using this expression for the bias, the regularized version of the AIC, which hereafter we refer to as in order to distinguish it from the standard definition, reads
The expression in eq. (15) corresponds, up to constant terms, to the one reported in Ref. [12]. Notice that the definition of the AIC is inherently regularization-dependent and, in particular, under the assumption of normally-distributed data, its expectation value reads
where . Therefore, by applying different regularizations to the covariance matrix, the AIC will yield different values. This is not surprising, since eq. (15) is an estimate of how much the regularized candidate model and the true model probability differ. As a final remark, note that in the correlated limit, i.e. for , the expression in eq. (15) reproduces exactly the standard result in eq. (7).
Dataset selection and regularized fits
In this section we show that the definition given in eq. (15), which naturally arises when regularized fits are performed on correlated data, is incompatible with the type of dataset selection problems described in § 2.1.1. The typical case is encountered when some sort of regularization is applied to the covariance matrix of a fitted subset , while the covariance matrix of the non-fitted dataset is left untouched. In this case, the expression for is not altered with respect to eq. (10). This leads to
where . In general the term is no longer common to all the subsets of the full dataset, since explicitly depends on the regularization employed on the chosen fitted subset. Therefore, in model and data subset selection procedures which involve regularized fits, by removing terms which are common to the full dataset, the estimator to be used is
in contrast to the standard case of correlated fits, where the simpler formula in eq. (12) applies.
As mentioned in § 1, regularizing the covariance matrix induces a loss of information. In the expression above part of this loss of information is encoded in the different determinant appearing in the expression of the AIC and can be easily understood if the applied regularization involves an SVD cut on the smallest eigenvalues of in order to make it less ill-conditioned. In this case, it follows that and the definition in eq. (18) always promotes models and datasets with a regularized covariance matrix closer to . The crucial point is that if the null-hypothesis is not rejected, i.e. there is no statistical evidence that the model is misspecified, then is still a good estimator of the true parameter vector . This implies that the information lost in the regularization procedure only plays the role of stabilizing the fit and it is not needed to have a good estimator of the parameter vector. The main reason is that the removed information is poorly estimated since the smallest eigenvalues of the covariance matrix are usually affected by large errors.
Therefore, by construction, the AIC in eq. (18) does not allow one to compare subsets fitted with different regularizations and the standard trade-off between complexity and quality of the fit is no longer the main feature of the AIC. The regularization dependence of eqs. (15) and (18) and its effects are illustrated in § 4.2 on a fixed dataset and in § 4.3 and § 4.4 in the context of dataset and model selection, respectively.
Regularization-independent AIC
We tackle the problem outlined in the previous section by deriving an expression of the Akaike Information Criterion which is, on average, independent of the chosen regularization. This is obtained by defining an estimator of eq. (3) which is able to account for the loss of information due to the regularization of the covariance matrix.
The regularization dependence of the AIC arises from the fact that the expression in eq. (15) has been derived as the unbiased estimator of the regularized version of eq. (3), i.e. computed with covariance matrix instead of the standard . To cope with this issue, in the following we define a regularization-independent Information Criterion as an estimator of the correlated . This allows us to absorb the loss of information due to the regularization applied to the covariance matrix into the bias and, therefore, to get an expression which is, on average, regularization-independent. We now take the plug-in estimator ; therefore the Information Criterion reads
where RIAIC stands for regularization-independent AIC. The crucial step to absorb any regularization-dependent term is taking the regularized log-likelihood in the equation above as an estimator of eq. (3). The bias to be computed reads
Note that the second term on the r.h.s. is exactly the term appearing in the bias of the standard AIC in eq. (6) and is evaluated at the correlated MLE estimator, i.e. . This is in contrast to eq. (13) where it is evaluated at the regularized MLE.
If the model is correctly specified, under the assumption of normally-distributed data, see App. § 7 for the detailed derivation, the bias reads
where and have been introduced in eqs. (14) and (16). This leads to the expression of the regularization-independent AIC
Note that the regularization-dependent term appearing in the bias in eq. (21) exactly cancels out the contribution arising from the log-likelihood in eq. (19). Similarly to the AIC in eq. (15), the formulation in eq. (22) reproduces, in the correlated limit , the standard AIC in eq. (7). However, unlike the AIC, see eq. (16), the expectation value of eq. (22) is exactly the one of eq. (7), i.e.
and it only depends on the non-regularized dataset and on the number of parameters of the model. In this sense, eq. (22) is, on average, regularization-independent and deviations from eq. (7) might arise only due to fluctuations in the quality of the fit. The size of the statistical fluctuations can be quantified by computing the variance, for a given fitted model, of ; see eq. (C.5) for the full expression and App. § 9 for the derivation.
The idea behind eq. (22) is that, provided that the null-hypothesis has not been rejected, the maximum likelihood estimator is still a good estimator of the true parameter vector . Therefore, the regularization of the covariance matrix is only relevant for the convergence of the fit but the poorly-estimated information which is lost in the regularization step is irrelevant for the purpose of obtaining a reliable estimator of . In practice, care must be taken in choosing the regularization of the covariance matrix. Indeed, this argument relies on the fact that the regularization only removes poorly-estimated information; however, if the regularization removes information which is relevant for a correct estimation of the model parameters, the RIAIC may deviate from the corresponding correlated AIC.
The criterion in eq. (22), being on average regularization-independent, allows the comparison between models fitted on data subsets which are regularized in different ways; see § 4 for some detailed examples.
Extension to dataset selection
As shown in § 2.2.1, once we try to rephrase data subset selection problems in terms of model selection ones, the definition of the AIC fails to provide the usual trade-off criterion between quality of the fit and complexity; see § 4.3 for a detailed example. This issue is solved if the definition in eq. (22) is used instead. As before, we assume that the covariance matrix of the non-fitted dataset is not regularized. Under this assumption, the bias correction in eq. (21) absorbs the loss of information due to the regularization of the covariance matrix of the fitted subset and the depends only on the determinant of the original covariance matrix , i.e.
where, again, we used the perfect-model prescription for the AIC of the non-fitted dataset, but generalizations to other definitions are possible as well. Therefore, by removing terms which are common to the full dataset and all models we obtain the final expression
which does not depend on the log-determinant of the regularized covariance matrix, unlike eq. (18). This expression inherits all the properties of the RIAIC: it reproduces, in the correlated limit, the standard result in eq. (12) and its expectation value is regularization-independent.
Extension to the Bayesian framework
Assume that the a priori knowledge about some of the parameters of the model function is encoded in Gaussian priors, with , characterized by mean and covariance matrix . Then the posterior probability distribution, up to an overall normalization factor, reads
In Refs. [8, 11] different possible extensions of the AIC to the Bayesian framework are presented. Given the inherently frequentist nature of the AIC, here we generalize to our framework what the authors of Ref. [8] call BAIC, namely the Bayesian AIC; however, the extension to other proposals is straightforward. We use as plug-in estimator for the Information Criterion the maximum a posteriori (MAP) estimator obtained from a regularized fit, i.e. . In this way the regularization-independent BAIC (RIBAIC) reads
Similarly to the frequentist case in eq. (20), any regularization-dependent term is removed by the bias
It is worth noticing that, in contrast to the frequentist case, the estimator does not minimize the chi-square appearing in the log-likelihood in eq. (27), but rather the so-called augmented chi-square [13], which encompasses both the data and the prior contributions. Nevertheless, if the model, including the priors, is correctly specified, the difference between and is expected to be small, leading to a bias correction which is numerically close to the one in eq. (21). The derivation of the full bias is carried out in App. § 8 and leads, for normally-distributed data, to the expression
where we introduced the effective number of parameters
where is simply the Bayesian extension of the projector defined in eq. (A.3) and has a similar definition, see eqs. (B.7) and (B.8) and App. § 8 for the full derivation. Note that the last term appearing on the r.h.s. of eq. (29) encodes the residual bias induced by a misspecification of the priors. It reads
where the matrix is defined in eq. (B.9). As already remarked, the underlying assumption of the entire derivation is that the model, and the priors in a Bayesian framework, are correctly specified. In this scenario, provides a small correction and the overall bias in the Bayesian framework is expected to be numerically similar to the one in eq. (21).
As a final remark, note that in the limit of non-informative priors eq. (29) correctly reproduces the RIAIC in eq. (22), while it reproduces the standard AIC if the correlated limit is taken as well. Conversely, if only the correlated limit is taken, eq. (29) reproduces the expression of the BAIC derived in Ref. [8] up to small corrections induced by the difference between the MLE and the MAP estimators, since the expression in Ref. [8], by construction, assumes the equivalence of the two estimators. Similarly to eq. (25), the RIBAIC presented in this section can be readily generalized to dataset selection problems by including the standard term for the non-fitted dataset.
Numerical example
In this section we numerically test the Information Criteria discussed in the previous sections on data obtained from lattice QCD simulations. In particular, we compare the RIAIC and the AIC and use the value of the AIC obtained from the corresponding fully-correlated fit as a benchmark.
Setup
The Information Criteria are tested on the zero-momentum single-nucleon correlation function computed on the D251 ensemble generated by the CLS initiative and already presented in Refs. [14, 15, 16]. The correlator has been computed within the distillation framework on 32 timeslices on independent gauge field configurations; further details on the ensemble and the number of eigenvectors of the covariant three-dimensional Laplacian operator used are given in Ref. [15]. The subsequent statistical analysis employs the jackknife resampling method. In all the fits we keep fixed and vary the size of the fitted dataset by varying , i.e. the first timeslice included in the fit.
As fit ansatz we use a multi-state model parameterized as
In the following, a model is considered to be correctly specified if the -value of the corresponding fit is larger than a threshold, which is set a priori to the value 0.05; if the fit is regularized, the -value is computed as in Ref. [3]. Fits which do not satisfy this requirement are therefore discarded.
For each regularized fit, the sample covariance matrix of the fitted dataset is replaced by its regularized version . The regularized covariance matrix is obtained through an SVD cut which acts on the smallest eigenvalues of . In particular, eigenvalues which are smaller than , where is the largest eigenvalue of and is a threshold on the condition number of which is chosen a priori, are set to be equal to , such that . Conversely, if , no SVD cut is applied to . Note that we work under the same assumptions used in the derivations of the previous sections; therefore, no regularization is applied to the covariance matrix of the non-fitted dataset . In § 4.2 the threshold is varied in order to probe the regularization dependence of the criteria, while in the subsequent sections it is fixed a priori to the condition number of the covariance matrix of the dataset in , i.e. , which corresponds to the smallest fitted dataset considered in § 4.4.
The comparison among the different Information Criteria is carried out by employing the expressions in eq. (24) for the RIAIC, eq. (17) for the AIC, and eq. (11) for the standard AIC, with block diagonal covariance matrices, i.e. by neglecting correlations between the fitted and non-fitted subsets, as described in the previous sections.
For the chosen datasets, the resulting covariance matrices, although ill-conditioned, are not singular and fully-correlated fits are still feasible. This allows us to use the AIC obtained from the corresponding correlated fits as a benchmark to test whether the RIAIC reproduces its outcome when regularized fits are performed on the same dataset. In realistic scenarios, e.g. on larger datasets, where fully-correlated fits are not viable, the RIAIC then provides a regularization-independent alternative where the standard correlated AIC cannot be computed. Since the RIAIC is expected to be, on average, independent of the chosen regularization, small deviations compared to the AIC are expected if the overall quality of the fit is the same and if the fit parameters can still be determined reliably. Conversely, larger deviations are expected to appear for the AIC due to its inherent regularization dependence. In the subsequent analysis we, therefore, compare different criteria through the differences and through the model and subset they select.
The tests are organized as follows. In § 4.2 we study the regularization dependence of the criteria for a fixed dataset and model. In § 4.3 and § 4.4 the criteria are applied to dataset selection and to combined dataset and model selection, respectively. Finally, in § 4.5 we consider fits with Bayesian priors.
Regularization dependence
As a first numerical test we study the regularization dependence of the RIAIC and of the AIC for a fixed dataset and model. We fit the single-nucleon correlator with a single-exponential model, . As previously mentioned, the benchmark for the comparison is provided by the AIC obtained from correlated fits on the dataset. To this aim, we identify the optimal subset by performing fully-correlated fits with and different choices of . Fits with -value lower than 0.05 are discarded and the smallest AIC identifies as the optimal window for the chosen single-exponential model.
On the same data subset we apply several SVD cuts to its covariance matrix , such that ranges from 1 to , and perform the corresponding regularized fits. The quality of the fit in this case is assessed as in Ref. [3] and we employ the same threshold used for the correlated case.
In the left panel of Figure 1 we show as a function of . For both the AIC and the RIAIC reproduce the standard AIC, as expected. When stronger regularizations are applied, the increasingly deviates from the AIC, up to for the strongest regularization considered, and the deviation is expected to grow further for even stronger regularizations. This behavior is not surprising; indeed, the in eq. (15) is an unbiased estimator of the regularization-dependent , which measures the distance between the data-generating probability distribution, containing , and the fitted regularized distribution, containing . The SVD cut acts as a regulator by raising the smallest eigenvalues of ; this implies that , and, therefore, the is dominated by this term as the regularization becomes stronger, which is the behavior observed in Figure 1.
Figure 1. Left panel: deviations of the ICs under study compared to the standard AIC as a function of the regularization expressed in terms of . Orange points correspond to the RIAIC, while blue points correspond to the AIC. The orange band is the error computed as in App. C. Right panel: ground-state determinations for each fit as a function of . The horizontal gray band corresponds to the result obtained from a correlated fit. The fitted dataset has been kept fixed to the window and the model is the single-exponential one.
On the other hand, the RIAIC is an estimator of the correlated , which is, by definition, regularization-independent, and only contains the determinant of the covariance matrix . As a consequence, its value remains stable across the entire range of regularizations explored and exhibits a deviation which is always below one unit and compatible with zero within the error computed as in eq. (C.5), suggesting that such deviation is mainly due to statistical fluctuations of around its expectation value.
In the right panel of Figure 1 we show the obtained ground-state energy estimates. Over the whole range of , the correlated difference between the ground-state energy from a regularized fit and the one obtained from the corresponding correlated fit is compatible with zero within standard deviation, and its statistical error increases by at most . In particular, for , the standard deviation of the estimated ground-state energy obtained from the regularized fit differs by less than 5% from the correlated result. The latter result, together with the fact that deviations between the RIAIC and the AIC differ from zero by less than one standard deviation, confirms that, for moderate regularizations, the information removed by the SVD cut does not affect the overall final estimate of the ground-state energy. Conversely, at the same values of , is for the AIC and may lead to the wrong identification of the optimal subset or model, or, in general, affects the final estimate in model averaging procedures.
Dataset selection
We fit the single-nucleon correlator used in the previous section with a two-exponential model , keeping fixed and varying . As before, fits with a -value lower than 0.05 are discarded from the analysis. In addition, we exclude from the analysis fits where excited-state contamination cannot be resolved, i.e. when the associated parameters are compatible with zero within one standard deviation. For both correlated and regularized fits, this leads to restricting the analysis to fit windows with . The benchmark optimal fit range is identified through the AIC of fully-correlated fits and corresponds to the window starting at .
The AIC and the RIAIC are tested in the context of data subset selection, by repeating the same fits with the regularization described in § 4.1, i.e. with threshold on the condition number of given by . The strength of the regularization, , increases by lowering from at to at , i.e. within the range explored in § 4.2.
The effective energy of the single-nucleon correlator is shown in the left panel of Figure 2, together with the optimal correlated double-exponential fit starting at and the estimates of the ground-state energy selected by the ICs under study.
Figure 2. Left panel: the effective energy of the single-nucleon correlator. The lines and bands represent the optimal double-exponential model and the estimated ground-state energy selected by the AIC in a correlated fit (red) and by the RIAIC (orange) and the AIC (blue) in a regularized fit. Upper right panel: ground-state energies obtained for several choices of with correlated (red) and regularized (gray) fits. Middle right panel: plot with the values of the AIC (red), RIAIC (orange) and AIC (blue). Stars mark the optimal according to each criterion. The vertical lines represent the optimal identified using the AIC and the RIAIC (dashed line) and the AIC (dotted line). Lower right panel: the deviation of each IC from the AIC; the orange band is the error associated with the difference between the RIAIC and the AIC; see App. § 9.
In the middle right panel we show the values of the three Information Criteria under study for each . Red points correspond to the AIC obtained from correlated fits, used as a benchmark, while orange and blue points represent the RIAIC and the AIC, respectively. The RIAIC exhibits deviations of a few units from the AIC over the whole range of . An interpretation of such deviations is given by the lower panel on the right, where we explicitly show the difference for both the RIAIC and the AIC. The orange band is the error associated with the deviations of the RIAIC relative to the standard AIC, computed from eq. (C.5). In general, the difference between the RIAIC and the AIC deviates from zero by less than 1.8 standard deviations, and by 1.2 standard deviations for , which is identified as the optimal one by the RIAIC. The RIAIC exhibits a systematic shift to smaller values compared to the AIC. This behavior is understood from the fact that the values of obtained on different fit windows are strongly correlated, with correlation coefficients computed from eq. (C.3).
Ground-state energies obtained from regularized and correlated fits are shown in the upper right panel of Figure 2. These are compatible for all and, therefore, deviations between the AIC and the RIAIC can be traced back to fluctuations of around its expectation value, as discussed in § 4.2. Remarkably, since such deviations are approximately constant in , they do not affect the selection and the RIAIC identifies the same optimal window, starting at , obtained by applying the AIC to fully-correlated fits.
Conversely, for the AIC increases monotonically from to . As discussed in § 4.2, this behavior is due to the fact that the AIC is an estimator of the regularization-dependent . As a consequence, the AIC selects the largest among those which yield a valid fit, i.e. it promotes subsets which are less regularized and, therefore, loses the usual trade-off between quality of the fit and complexity of the description.
As a final remark, the identification of different optimal data subsets affects the estimate of the ground-state energy as shown in the upper right panel of Figure 2. The correlated difference between the determinations based on the RIAIC and the AIC is compatible with zero within less than 0.2 standard deviations and the two ground-state energy estimates have similar statistical errors. In contrast, the determination obtained with the AIC, despite being compatible with the estimate based on the AIC, has a statistical uncertainty which is almost twice as large.
Dataset and model selection
In this section we test one- and two-exponential models and in eq. (32) for different choices of to identify the optimal subset-model description of the data, based on the Information Criteria under study. As before, we use as benchmark the results based on the AIC obtained from correlated fits and we discard or accept models based on the same criteria discussed in § 4.3. In the correlated analysis, the double-exponential model is preferred for and the single-exponential one for . For no model is accepted based on our criteria. The reason is that, for such windows, single-exponential models are discarded based on their poor quality due to the residual excited-state contamination at low , while double-exponential models cannot clearly resolve the contribution from the first excited state. As in the previous section, the correlated analysis based on the AIC identifies the double-exponential model starting at as the optimal subset-model description. We then repeat the same analysis by performing regularized fits on each dataset in order to compare the outcome of the AIC and RIAIC. As before, we regularize the covariance matrix of a given dataset if its condition number is larger than a given threshold set to . For each we only select the model with the lowest IC.
In the middle right panel of Figure 3 we show the values of the three Information Criteria for each . Let us stress that, for each , both the RIAIC and the AIC identify the same optimal model obtained from the benchmark analysis based on the standard AIC. For , the double-exponential model is preferred, and the new analysis reproduces the results seen in § 4.3, with the RIAIC deviating from the AIC by a few units and the AIC deviating by up to units from the AIC. For all the ICs identify the single-exponential model as the optimal one and deviations from the AIC are smaller both for the RIAIC and the AIC due to the much milder regularization applied to the covariance matrix of the fitted dataset. The limiting case is achieved for , where no regularization is applied and the three ICs coincide. As a consequence, the RIAIC identifies the same subset-model description obtained by applying the AIC to correlated fits, i.e. the double-exponential model starting at , whereas the AIC selects the single-exponential model starting at . As discussed in § 4.2, this result is mainly due to the regularization dependence of the AIC which penalizes regularized subsets with low and promotes fit windows with larger values of where the regularization has smaller effects, i.e. the regularized covariance matrix is closer to the one with intact correlations.
Figure 3. As in Figure 2, but for simultaneous dataset and model selection. Squares and circles denote single- () and double-exponential () fits, respectively. For each , only the model with the lowest value of the corresponding Information Criterion is shown; the vertical lines mark the optimal subset-model description identified by the AIC and the RIAIC (dashed line) and by the AIC (dotted line).
The lower right panel shows that, for , the deviation of the RIAIC coincides with the one discussed in § 4.3, while for it is compatible with zero within less than 0.7 standard deviations. As expected, the statistical error shrinks as increases and vanishes at , where ; see eq. (C.5).
The impact of the different selections on the ground-state energy is shown in the upper right panel of Figure 3. Regularized and correlated fits yield compatible results for all both for single- and double-exponential models. The determination based on the RIAIC agrees with the one based on the AIC: their correlated difference is compatible with zero within less than 0.2 standard deviations and the two estimates have similar statistical errors. In contrast, the determination based on the AIC, obtained from the single-exponential fit starting at , results in a correlated difference which is standard deviations away from zero and a statistical uncertainty which is more than 1.5 times larger.
These results show that, when regularizations are applied to the covariance matrix, the AIC may lead to a different subset-model selection. On the other hand, the RIAIC, whose bias accounts for the regularization of the covariance matrix, is expected to reproduce, on average, the selection obtained by applying the AIC to correlated fits.
As a final remark, note that the small deviations between the RIAIC and the AIC may, in principle, lead to different results if the ICs are used as weights in model-averaging procedures. However, these differences are expected to be even larger if the AIC is used instead.
Bayesian fits and prior dependence
In this section we test the RIBAIC defined in eq. (29) on Bayesian fits to the same single-nucleon correlator analyzed in the previous numerical tests. As discussed in § 3.2, in the limit of non-informative priors, the RIBAIC is expected to reproduce the RIAIC in eq. (22) and, if the correlated limit is taken as well, i.e. , it is expected to reproduce the standard AIC. To this aim, we keep fixed both the model and the fitted dataset based on the results of § 4.3 and § 4.4, i.e. we fit the dataset starting at with the double-exponential model, and we vary the width of the priors.
We use Gaussian priors for all the model parameters, i.e. with diagonal prior covariance matrix and priors given by
where the second argument in here denotes the standard deviation. In the priors above, and are estimated from the effective energy at by assuming a single-state saturation at the selected timeslice and ; for a more precise estimate of the pion mass on this ensemble see Table II of Ref. [15] and Table 4 of Ref. [17]. The parameter is varied between and to probe the effects of constraining and non-informative priors. For each selection of priors, we perform both fully-correlated and regularized fits with covariance matrix and respectively, where is obtained by applying an SVD cut such that its condition number is equal to . The quality of the Bayesian fit is assessed through the -value associated with the augmented chi-square which includes the contribution of the priors [13].
In Figure 4 we show the results of the analysis for both the correlated and the regularized fits. For all the fits have -value larger than the selected threshold and the RIBAIC approaches the expected non-informative limit, i.e. the standard AIC in the case of fully-correlated fits and the RIAIC in the regularized case. Note that, in contrast to in the previous numerical tests, here denotes the difference between the RIBAIC and the corresponding expected limit for . In particular, for the deviations of the RIBAIC from the expected non-informative limit are already below and the ground-state energy estimates reported in the lower panels are stable and with a statistical error similar to the one extracted from the correlated fit. In this regime, since the MAP and the MLE estimators are numerically close, the correlated RIBAIC (left column) also reproduces the BAIC of Ref. [8] up to small corrections. In the middle panels we show the term as a function of for both the correlated (left) and the regularized (right) fits. As expected, when the width of the priors increases, monotonically decreases and, for priors with , it is already of the order of .
Figure 4. Results from Bayesian fits to the single-nucleon correlator with the double-exponential model on the window , as a function of the relative width of the priors. Left (right) column: correlated (regularized) fits. Upper panels: deviation of the RIBAIC from the AIC (RIAIC) of correlated (regularized) frequentist fits, ; the dashed lines mark the AIC (left) and the RIAIC (right), i.e. the expected limits for non-informative priors. Middle panels: the term in eq. (31). Lower panels: ground-state energy estimates. The red band corresponds to the result of the correlated frequentist fit. Open symbols in the gray regions denote fits with Bayesian -value lower than 0.05.
For , fits are prior dominated and do not provide stable results for the fit parameters. The ground-state energy estimate approaches the prior means as . Similarly, the RIBAIC increases and, in the same limit, is expected to saturate to the value of evaluated at the prior means. Note that, even though fits in this region are discarded based on their -value, the value of remains relatively small and does not signal the misspecification of the priors. The main reason, as discussed in App. § 8, is that the MAP estimator is biased if priors are misspecified and therefore the substitution is no longer justified and may lead to an unreliable estimate of the value of .
These results show that, for moderately wide priors, the RIBAIC of a Bayesian fit and the RIAIC of the corresponding frequentist fit on the same dataset are numerically equivalent. This is due to the fact that, under the assumption that both model and priors are correctly specified, these criteria are estimators of the same correlated quantity, independently of any type of regularization. As a consequence, they can be used in a unified framework, in which frequentist and Bayesian fits, possibly with different regularizations, are compared in model and dataset selection or combined in model-averaging procedures.
Summary
In this work, we have derived an extension of the Akaike Information Criterion which allows one to tackle model and data subset selection problems when regularized fits are performed on correlated data. The main result is represented by the RIAIC and is reported in eq. (22) for standard frequentist fits. The RIAIC allows the comparison between models and subsets when regularizations are applied to the covariance matrix of the fitted dataset. The generalization to maximum a posteriori fits, i.e. the RIBAIC, has been derived as well and is reported in eq. (29). In § 4, we tested the numerical and statistical properties of the RIAIC and of the RIBAIC. We found that, while the naive generalization of the AIC to the case of regularized fits, which throughout this work we called the AIC, systematically favors less regularized datasets, the RIAIC deviates from the AIC of the corresponding correlated fit only by a few units, consistently with the statistical fluctuations in around its expectation value; see App. § 9. In the numerical tests that we carried out, this led to the identification of the same optimal model and data subset obtained with the AIC, as expected since the two criteria have the same expectation value.
The RIAIC and the RIBAIC were obtained by defining the Information Criterion as an unbiased estimator of the correlated model-dependent contribution to the KL divergence. Equivalently, the expectation values of the RIAIC and of the RIBAIC are equal to the expectation value of the standard, correlated Akaike Information Criterion. By taking the correlated limit, both the RIAIC and the RIBAIC reproduce standard results in the literature, namely the AIC [1, 2] and, if the MAP and the MLE estimators are numerically close, the BAIC [8], respectively.
These criteria apply in the most general setting, where both regularizations of the covariance matrix and priors are included in the fit. We interpret the use of regularizations at the level of the covariance matrix and the inclusion of priors as a modification of the total amount of information encoded in the data we aim to model. In the first case, regularizing the covariance matrix has the role of removing poorly-estimated information. Conversely, the inclusion of priors provides additional information which effectively acts as a regulator of the fit. In both cases the regularization is properly absorbed into the bias of the Information Criteria in eqs. (22) and (29), leading to an overall result which is, on average, independent of the chosen regularization.
The estimators that have been derived are obtained starting from the Akaike Information Criterion. Nonetheless, the generalization to other Information-Theoretic Criteria is possible.
The application of these results encompasses not only data subset and model selection, but also model averaging procedures as detailed in Ref. [11]. In particular, as shown in § 4.5, for moderately wide priors, the RIBAIC reproduces the numerical value of the RIAIC. As a consequence, frequentist and Bayesian fits performed on the same dataset, with or without regularization, can be compared and combined within a unified framework, either in model and dataset selection or in model averaging procedures.
Acknowledgements
I thank my co-authors of Refs. [15, 16] and colleagues of the Baryon Scattering (BaSc) Collaboration for sharing the single-nucleon correlator used in this work, in particular F. Romero-López for computing perambulators on the D251 ensemble and J.R. Green for the Wick contractions. I would like to thank J. Bulava for comments on an early version of the manuscript and A. Baião Raposo, M. Bruno and M. Ginesi for useful discussions on the topic of this work. My work is supported by the European Research Council (ERC) consolidator grant StrangeScatt-101088506. Part of the analysis in § 4 was performed on Elysium, the HPC cluster of the Ruhr-Universität Bochum, subsidized by the DFG(INST213/1055-1).
The analysis of § 4.5 was performed using the gvar [18] and lsqfit [19] libraries.
RIAIC bias correction
Recall the expression of the bias appearing in the regularization-independent definition of the AIC in eq. (20)
The derivation in this section follows the presentation in Ref. [4] and generalizes it to the case of regularized fits for the first term appearing in the bias. The same steps, applied to the definition in § 2.2, provide the expression of the bias of the AIC in eq. (15). The first term of the equation above is readily computed by inserting the expression of the log-likelihood, which leads to
where is the regularized chi-square variable evaluated at . In the second line we have used the result from Ref. [3] for the expected value of the chi-square. The projector is defined as
where we have introduced the estimated inverse covariance matrix of the MLE parameters, associated with the regularized matrix , as
and is the Jacobian of the model function with respect to the model parameters . As in Ref. [3], the unknown is replaced in practice by the estimator . Therefore, by introducing the effective number of parameters as in eq. (14) and , the first term of the bias reads
The second term of the bias is evaluated at , i.e. the MLE parameter obtained from a correlated fit, which is in principle not accessible if is close-to-singular. By inserting the expression in eq. (3), it can be written as
By expanding the likelihood, the innermost expectation value reads
where the expectation value of the chi-square is taken before evaluating it at the MLE parameter . In order to compute the expectation value appearing on the r.h.s. of eq. (A.7) we Taylor-expand the model function around the true value . By using the standard result for the expectation value of quadratic forms and recalling that , the expected value on the r.h.s. reads
where we introduced . By combining the expression above with eqs. (A.6) and (A.7), the second term of the bias reads
where we defined according to eq. (A.4). The remaining expectation value can be readily computed by using the condition at the minimum for , which leads to the relation
By using this relation and the standard results for quadratic forms, the expectation value on the r.h.s. reduces to the total number of parameters of the model function [4, 12]. Therefore the second term of the bias in eq. (20) can be written in the compact form
Finally, by combining eqs. (A.2) and (A.11), the bias takes the form in eq. (21) and the final expression for the regularization-independent criterion is the one reported in eq. (22). To conclude this section, notice that in the correlated limit the trace involving the projector reduces to the total number of parameters and the expression of the bias reproduces the bias of the standard correlated AIC, which reads, up to model-independent terms, .
RIBAIC bias correction
The bias of the Bayesian extension in eq. (27) can be computed similarly to App. § 7. It reads
The second term coincides with eq. (A.11). Under the assumption of normally-distributed data, the first term reads
By expanding around the true value , the expectation value of the chi-square in the expression above can be written as
where and we used the definition in eq. (A.4). Note that for the first term on the r.h.s. we used the result for the expectation values of quadratic forms. The remaining two terms are computed analogously, by using the condition at the minimum of the augmented chi-square [13]. The condition at the minimum leads to the relation
Let us stress that the equation above differs from eq. (A.10) due to the presence of priors and because does not directly minimize the chi-square of the data appearing in the bias, but rather it minimizes the augmented chi-square. In analogy to eq. (A.4), in eq. (B.4) we introduced
where is the prior covariance matrix. If , is understood to be the matrix obtained by inverting the prior covariance matrix and setting to zero the rows and columns associated with those parameters without priors. By using eq. (B.4) in the expectation value in eq. (B.3), and by taking advantage of the well-known result for quadratic forms, the bias term in eq. (B.2) simplifies to
Here, similarly to the projector in eq. (A.3), we introduced the matrices
In practice, to compute these terms instead of using the true parameter vector , which is unknown, we use the estimator . This substitution is justified when priors are either wide or centered close to the true values of the parameters. If priors are misspecified, the MAP estimator is biased and the resulting estimate of may be unreliable. However, the entire derivation relies on the assumption that the model, including priors, is correctly specified, which is checked through the quality of the fit before computing the RIBAIC. Fits with misspecified priors are therefore expected to be discarded based on their quality.
Finally, by combining the expression above with eq. (A.11) we obtain the explicit expression of the bias defined in eq. (28) and consequently the Information Criterion in eq. (29). As a final remark, note that by taking the limit of flat priors, both and reduce to the projector in eq. (A.3), while the term involving vanishes and the overall expression for the bias reproduces the one of the RIAIC derived in App. § 7.
Variance of Delta RIAIC
In this appendix we investigate the statistical properties of
where we used the shorthand notation and, in contrast to the notation of the main text, we introduced the label which identifies the th fitted dataset. Note that in the last step on the r.h.s. we explicitly used the expressions in eqs. (22) and (7) and introduced, for a fixed fitted dataset, . Here and denote the regularized and the fully-correlated covariance matrices of the th dataset respectively. Similarly to the main text, the expectation value on the r.h.s. is taken over the true probability distribution of the fitted dataset. From eq. (C.1), has vanishing mean; therefore, given two fitted datasets, the resulting covariance between and is
where in the last step we used the definition in eq. (C.1) and expanded the expectation values. As before, we work under the assumptions that the model is correctly specified and that data follow a multivariate Gaussian distribution. Following the steps of Apps. § 7 and § 8, by expanding the chi-square variables in and around the true value and recalling that for both and , standard results for the second and fourth moments of a multivariate normal distribution lead to
where is an matrix which contains the correlations between the two fitted datasets and, by using the notation of App. § 7,
where is the projector in eq. (A.3) and is the Jacobian, both restricted to the th dataset. The variance of on the th fitted subset is obtained from the diagonal term and can be written in the compact form
Note that, as expected, in the correlated limit, i.e. for , has vanishing variance since it is identically zero.
Software implementation
A Python implementation of the Information Criteria introduced in this work is publicly available at https://github.com/laudid46/pyaic under the GPL-3.0 license.
References
- [1]H. Akaike. “A new look at the statistical model identification”. In: IEEE Trans. Automatic Control 19.6 (1974), pp. 716–723. DOI: 10.1109/TAC.1974.1100705.DOI
- [2]Hirotugu Akaike. “Information Theory and an Extension of the Maximum Likelihood Principle”. In: Selected Papers of Hirotugu Akaike. Ed. by Emanuel Parzen, Kunio Tanabe, and Genshiro Kitagawa. New York, NY: Springer New York, 1998, pp. 199–213. ISBN: 978-1-4612-1694-0. DOI: 10.1007/978-1-4612-1694-0_15.DOI
- [3]Mattia Bruno and Rainer Sommer. “On fits to correlated and auto-correlated data”. In: Comput. Phys. Commun. 285 (2023), p. 108643. DOI: 10.1016/j.cpc.2022.108643. arXiv: 2209.14188 [hep-lat].DOI
- [4]Joseph E. Cavanaugh and Andrew A. Neath. “The Akaike information criterion: Background, derivation, properties, application, interpretation, and refinements”. In: WIREs Computational Statistics 11.3 (2019), e1460. DOI: 10.1002/wics.1460.DOI
- [5]B. Efron. “Bootstrap Methods: Another Look at the Jackknife”. In: The Annals of Statistics 7.1 (1979), pp. 1 –26. DOI: 10.1214/aos/1176344552.DOI
- [6]Ulli Wolff. “Monte Carlo errors with less errors”. In: Comput. Phys. Commun. 156 (2004). [Erratum: Comput.Phys.Commun. 176, 383 (2007)], pp. 143–153. DOI: 10.1016/S0010-4655(03)00467–3. arXiv: hep-lat/0306017.arxiv.org/abs/hep-lat/0306017
- [7]S. Kullback and R. A. Leibler. “On Information and Sufficiency”. In: The Annals of Mathematical Statistics 22.1 (1951), pp. 79 –86. DOI: 10.1214/aoms/1177729694.DOI
- [8]Ethan T. Neil and Jacob W. Sitsison. “Improved information criteria for Bayesian model averaging in lattice field theory”. In: Phys. Rev. D 109.1 (2024), p. 014510. DOI: 10.1103/PhysRevD.109.014510. arXiv: 2208.14983 [stat.ME].DOI
- [9]Ethan T. Neil and Jacob W. Sitsison. “Model averaging approaches to data subset selection”. In: Phys. Rev. E 108.4 (2023), p. 045308. DOI: 10.1103/PhysRevE.108.045308. arXiv: 2305.19417 [stat.ME].DOI
- [10]Sz. Borsanyi et al. “Leading hadronic contribution to the muon magnetic moment from lattice QCD”. In: Nature 593.7857 (2021), pp. 51–55. DOI: 10.1038/s41586-021-03418-1. arXiv: 2002.12347 [hep-lat].DOI
- [11]William I. Jay and Ethan T. Neil. “Bayesian model averaging for analysis of lattice field theory results”. In: Phys. Rev. D 103 (2021), p. 114502. DOI: 10.1103/PhysRevD.103.114502. arXiv: 2008.01069 [stat.ME].DOI
- [12]Julien Frison. Towards fully bayesian analyses in Lattice QCD. 2023. arXiv: 2302.06550 [hep-lat].arxiv.org/abs/2302.06550
- [13]G. P. Lepage et al. “Constrained curve fitting”. In: Nucl. Phys. B Proc. Suppl. 106 (2002). Ed. by M. Muller-Preussker et al., pp. 12–20. DOI: 10.1016/S0920-5632(01)01638-3. arXiv: hep-lat/0110175.DOI
- [14]André Baião Raposo et al. “Preliminary study of the H dibaryon in N_f = 2 + 1 lattice QCD”. In: 42nd International Symposium on Lattice Field Theory. Feb. 2026. arXiv: 2603.00698 [hep-lat].arxiv.org/abs/2603.00698
- [15]Jorge Baeza-Ballesteros et al. “Two-nucleon scattering amplitudes at light quark masses from lattice QCD”. In: (Oct. 2026). arXiv: 2610.23332 [hep-lat].
- [16]Jorge Baeza-Ballesteros et al. “Quark-mass dependence of two-nucleon scattering”. In: (Nov. 2026). arXiv: 2611.07864 [hep-lat].
- [17]Simon Kuberski et al. “D and D_s decay constants in N_f = 2 + 1 QCD with Wilson fermions”. In: JHEP 07 (2024), p. 090. DOI: 10.1007/JHEP07(2024)090. arXiv: 2405.04506 [hep-lat].DOI
- [18]Peter Lepage. gplepage/gvar. https://github.com/gplepage/gvar. Apr. 2020.
- [19]Peter Lepage. gplepage/lsqfit. https://github.com/gplepage/lsqfit. Apr. 2020.



