Introduction
In the quest to provide rigorous quantitative constraints of high-dimension model-parameter spaces by comparing sophisticated computationally expensive numerical models to experimental results, one must sample the model-parameter space by running the full-model at large numbers of points. For example, sampling a 12-parameter model with only three points in each direction requires 3 12 ≈ 500,000 3^{12} \approx500{,}000 3 12 ≈ 500 , 000 runs. Thus, Markov Chain Monte Carlo (MCMC) searches, weighted by the likelihood that the model reproduces the experiment, often involve millions of full-model evaluations. If a single run of the full-model at one point in parameter space requires days of CPU time, the analysis becomes untenable. Model emulation provides a solution to this problem. If one can train an emulator to reproduce the full-model by training it on only a few hundred full-model runs or fewer, and if the emulator can provide a reasonable estimate of its uncertainty, the emulator can take the place of the full model in the MCMC search, and thus enable the analysis.
Motivation, context, and related work
This paper is motivated by the theory and practice of heavy-ion physics, a field where emulators have been applied to various large-scale analyses, often involving more than a dozen model parameters, with models that often require days of CPU time. Contextual references in heavy-ion physics abound: [1 , 2 , 3 , 4 , 5 , 6 , 7 , 8 , 9 , 10 , 11 , 12 , 13 , 14 , 15 , 16 , 17 , 18 , 19 , 20 , 21 , 22 , 23 , 24 , 25 , 26 , 27 , 28 , 29 , 30 , 31 , 32 , 33 , 34 , 35 , 36 , 37 , 38 , 39 , 40 , 41 , 42 , 43 ] . For example, in a global analysis of 14 model parameters [34 ] , Gaussian-processes (GP) [44 , 45 ] were used as emulators of the output from hybrid hydrodynamic/Boltzmann approaches with accuracies near one percent, after being trained on 1200 points. One might ask whether one can do as well, or better, with fewer training points. The field of heavy-ion physics is similar to many other physical or biological fields of application, in that the relation between the model parameters and model output tends to be roughly linear in a physical parameter range. For example, in high-energy Lead-Lead (Pb-Pb) collisions at the Large Hadron Collider (LHC), the mean momentum of outgoing protons rises monotonically with the stiffness of the equation of state. That dependence should not be perfectly linear, but one would expect that, within a reasonable prior parameter range, the dependence is well described by linear or quadratic functions, and that higher-order terms should become increasingly negligible. This expectation also applies to most any other observable with respect to nearly any other model parameter. For lower-energy collisions, where phase transitions are perhaps more relevant, one then imagines a dependence that has significant structure, e.g. a clear threshold w.r.t. some parameter or set of parameter hypersurfaces, with roughly linear behaviors in the regions determined by these surfaces. At the higher LHC energies, these complications would not occur, and the methods in this paper would apply directly to the entire physical parameter range.
As a quick sanity check, putting aside complete UQ considerations, imagining a scalar model, with N p a r s N_{\mathrm{pars}} N pars parameters, if it were nearly perfectly linear, a first-order polynomial (linear approximation) would provide an excellent fit, and would presumably result in acceptable UQ bands. One could then run the full model only N p a r s + 1 N_{\mathrm{pars}} + 1 N pars + 1 times, at points fairly regularly far apart, to constrain the linear approximation’s N p a r s N_{\mathrm{pars}} N pars slopes and the intercept. For a 12-parameter model, this would require only 13 full model runs. Imagining that the model was instead nearly perfectly quadratic, one would need only N p a r s ( N p a r s + 1 ) / 2 N_{\mathrm{pars}}(N_{\mathrm{pars}} + 1)/2 N pars ( N pars + 1 ) /2 additional full-model runs to constrain all the quadratic contributions. For the 12-parameter case, this would lead to 91 full model runs, with an expectation that this way of contraining the model would still lead to acceptable UQ bands. This is fewer than the number of full-model runs often used in recent analyses in heavy-ion physics, by orders of magnitude, for modeling scenarios where there is arguably little benefit, within physical parameter ranges, of a cubic approximation over a quadratic one.
Another way to say this, which is prudent and parsimonious, both in computational and modeling terms, is that in any domain of application, such as heavy-ion physics, where most models feature rather smooth responses to their parameters, is that one should reconsider polynomial fits, and investigate the degree to which they might, or might not, improve upon the GP approaches cited above. The first goal of this paper is to exploit the expectation that a polynomial fit should be adequate systematically, by presenting a way to construct a polynomial expansion where the importance of various terms in the expansion decreases with increasing order. In the physics literature, this expectation of adequacy (of a polynomial approximation) is often labeled as “convergence” of the approximation, where the asymptotic notion in this convergence is with respect to the degree of the polynomial as it increases to infinity, even though practitioners talk about convergence even if the degree is small, as long as the error in the approximation fulfills the practitioners’ “expectations”, which in this case should be understood as their stated or implicit tolerances. To be able to declare that approximations are “converging” in the sense that the errors are tolerable, it is best to quantify those expectations in the most transparent terms possible, which is often done in terms of a “convergence” parameter, Λ \Lambda Λ . Once one has agreed on such a quantitative framework, one can work to define and find an optimum set of expansion coefficients, along with a quantification of their uncertainty. Herein, the emulator is built on a polynomial expansion where the polynomial’s variables are the N p a r s N_{\mathrm{pars}} N pars values θ i / Λ \theta_i/\Lambda θ i /Λ , where the model parameters are θ i \theta_i θ i ; i = 1 , … , N p a r s i = 1,\ldots,N_{\mathrm{pars}} i = 1 , … , N pars , and are all scaled so that they vary of order unity within the prior. Thus, higher-order terms contribute as negative integer powers of the convergence parameter Λ \Lambda Λ .
Emulators defined by choosing polynomial fits based on least-square minimization algorithms have long been viewed as good classes of candidates. More modern approaches improve on such fits by providing estimates that exactly reproduce the training values when evaluated at the training points, and provide an estimate of the emulator’s uncertainty as a function of the vector of parameters θ \theta θ . These approaches which reproduce training values are typically based on GP assumptions, where the predictions are all based on an assumed form of the covariance between values Y Y Y and Y ′ Y' Y ′ of the model evaluated at two parameter-vector points θ \theta θ and θ ′ \theta' θ ′ . That correlation kernel might be a function of θ \theta θ and θ ′ \theta' θ ′ , but a particularly simple and popular form is a “Gaussian” covariance kernel, Cov 0 ( Y 1 , Y 2 ∣ σ A , Λ ) = σ A 2 e − ∣ θ − θ ′ ∣ 2 / ( 2 Λ 2 ) \operatorname{Cov}_0(Y_1,Y_2\mid\sigma_A,\Lambda) = \sigma_A^2 e^{-\lvert\theta-\theta'\rvert^2/(2\Lambda^2)} Cov 0 ( Y 1 , Y 2 ∣ σ A , Λ ) = σ A 2 e − ∣ θ − θ ′ ∣ 2 / ( 2 Λ 2 ) . Here, the parameters σ A \sigma_A σ A and Λ \Lambda Λ are referred to as hyper-parameters, to differentiate them from model parameters. They are used only to tune the emulator, and their tuning can be optimized based on training data. Here, the subscript 0 denotes the fact that the covariance is taken with respect to the prior distribution. It has been shown that any polynomial expansion, or any expansion over some set of basis functions, where the expansion’s terms have random coefficients, can be optimized to reproduce training points. These can then be equated to GP forms once one defines the appropriate correlation kernel [44 ] . Previous approaches have typically been constructed from sets of orthogonal basis functions, e.g. [46 ] , but this requirement is not necessary and is not enforced herein. The sum over polynomial terms with random coefficients is sometimes referred to as a “polynomial chaos” [47 ] . Once trained, one can optimize all the random coefficients for the polynomial expansion. One can then calculate the emulator values at any parameter vector θ \theta θ using those optimal coefficients. However, there are many more coefficients than training constraints, since one has the option to truncate the full Taylor series at an arbitrarily high degree. Fortunately, once one has written an equivalent form for the correlations, the emulator can much more efficiently be expressed in terms of the correlation kernel and one can avoid performing sums over all monomial terms. This is sometimes referred to as the kernel trick [44 ] .
Narrative summary of this paper’s contributions
In this work’s first part, we consider emulators as polynomial expansions with random coefficients, and we constrain the emulators’ form by requiring that the emulating behavior be invariant by rotation and by translation. Under this requirement, we find that the equivalent correlation kernel must then have a simple Gaussian form. Thus, if one believes that the model is equally sensitive to all parameters and that there is no reason to expand the polynomials about one point rather than around some other point, then the equivalent GP correlation kernel must be Gaussian, and described via the form involving the two hyper-parameters σ A \sigma_A σ A and Λ \Lambda Λ mentioned above. The polynomial description then becomes equivalent to a GP description with a correlation kernel with a simple Gaussian form. If the user believes that the sensitivity w.r.t. some model parameter should be less than that of another parameter, rather than applying different convergence parameters, one can scale the parameters so that the expected sensitivity is isotropic in the scaled parameter basis. Thus, the assumption of the model having a rotation-invariant behavior can reasonably apply to many problems after appropriate rescaling. The assumption of translation invariance is one of spatial stationarity across parameter space, which is also a strong modeling preference for most problems. In particular, it significantly reduces the statistically grounded UQ on any parameter, avoiding the risk of overfitting a model for UQ itself.
On the strength of the translation and rotation invariance assumption of the model’s parameter-space sensitivity, the single scalar convergence parameter, Λ \Lambda Λ , sets the relative mutual importance of terms of different orders in the polynomial expansion. We find that Λ \Lambda Λ is precisely the same as the correlation length in the GP interpretation with a Gaussian correlation kernel. For the polynomial expansion to “converge”, i.e. for its truncation to yield a tolerable error, within a few orders, at a minimum, one must have the convergence parameter Λ \Lambda Λ larger than the characteristic scale of the prior on parameter space, and this would typically not be enough for the truncation error to be tolerable. If the model parameters are scaled to vary roughly between − 1 -1 − 1 and + 1 +1 + 1 , then Λ \Lambda Λ should be significantly larger than unity if the expansion is to be well defined by a few terms, or equivalently, to be constrained via calibration by a few training points. After translating to the equivalent GP paradigm, this is equivalently stated as having the GP correlation length, which coincides with Λ \Lambda Λ , being significantly larger than the size of the prior.
Conversely, if one assumes a Gaussian correlation kernel in a GP context, the relations derived here identify the form of the corresponding polynomial expansion. Thus, one can use this form to generate random functions that are consistent with the Gaussian kernel, albeit with some cutoff in the order of the expansion’s truncation. This should be useful for testing emulation software, as it should allow one to build models that correspond to the desired correlation kernel and should, by construction, have uncertainties that are well understood and quantified.
Our work’s second thrust is to understand, for expansions which are rapidly converging (where only a small number of terms are needed to get tolerable truncation errors), how one should optimally choose the points at which the full model is run to train the emulator. We present a metric that represents the expected accuracy (expected truncation error) of the emulator as a function of the convergence parameter, Λ \Lambda Λ , and as a function of the positions of the training points. For a fixed Λ \Lambda Λ , optimized arrangements of training points are then generated for various cases involving either Gaussian or uniform priors on the parameters. For test cases, this procedure is found to be significantly superior to either random placement of the training points throughout the prior, or to Latin hyper-cube sampling [48 ] . In this setting of rapidly converging expansions, one might expect that the number of training points might be chosen to match that needed for linear, quadratic or cubic fits. For a 6-parameter example, that would require 7, 28 or 84 full-model runs respectively. Here, we show that the marginal improvement to the emulation decreases at each of the two corresponding thresholds, assuming that the training points are optimally chosen. In practice, this analysis would provide a user who has a clear understanding of their computational costs, and who has a clear quantification of their aversion to error risk, which can simply take the form of a worst-case scenario cost of tolerance, to run an elementary cost-benefit analysis.
By accomplishing the goals above, one should be able to gain a better understanding of how many model runs might be necessary to design an emulator to the desired accuracy, where the term “desired” can be built into a cost-benefit analysis of the aforementioned ilk. This decision framework will still depend on some pre-estimate or other choice of the convergence parameter Λ \Lambda Λ , but it turns out, from our testing, that the actual placement of the training points appears rather insensitive to that estimate for Λ \Lambda Λ , even though the accuracy is strongly sensitive to the estimate. For users who knows ahead of time that some of the model parameters are more likely to drive the functional behavior than others, we provide a seamless method to exploit that prior knowledge. This can assist with overcoming the challenges of high dimensionality. We find herein that such flexibility turns out to be critical in determining how confidently one might ultimately emulate models.
In the next section, we present the mathematical form of such an emulator, and a derivation of the optimum set of coefficients, given the training data. Closed expressions for the emulator uncertainty are also derived. The emulator’s form also accounts for models with noisy training, i.e. those for which point-by-point uncertainty is acknowledged and taken into account, i.e. where that uncertainty is propagated in a fashion consistent with the modeling. The physical arguments for choosing the specific emulator form are discussed. The equivalence with a GP emulator is established carefully. The subsequent section presents a tractable method to optimize the location of training points in the N p a r s N_{\mathrm{pars}} N pars -dimensional parameter space. For the case of Gaussian priors with N p a r s + 1 N_{\mathrm{pars}}+1 N pars + 1 training points, the placement follows a straightforward N p a r s N_{\mathrm{pars}} N pars -dimensional simplex, but for most cases the optimized solution does not have a simple symmetry. Instead, our method finds the optimal points numerically. In Section IV we test our emulator procedure by applying it to some models with random polynomial expansions or randomized trigonometric or exponential forms. This allows us to gain a solid quantified grasp for how well one might, in practice, not only emulate a function, but also understand how well one might trust the fidelity of our emulator’s quantitatively estimated uncertainty. This faithfulness is studied as a function of the dimensionality of the model’s parameter space and of the convergence parameter Λ \Lambda Λ .
Theoretical Foundation of Emulating Convergent Functions
The form and basic properties of our proposed class of emulators is described in detail in this section. The form is that of a polynomial expansion in terms of the model parameters, i.e. expanded as a multivariate power series with respect to the parameter vector, where the coefficient for each term in the polynomial is given an independent centered Gaussian (mean-zero normal) weight. Since centered Gaussian variables have a scale property with respect to their standard deviation, it is convenient and elementary to standardize the Gaussian coefficients by pulling out their scales as explicit deterministic emulation weights. Using these weights, our form, while general, is easily constrained to assume that terms of higher order should contribute less. For example, in the absence of training data, quadratic terms should be expected to contribute less than linear terms, and cubic terms should contribute less than quadratic terms. What separates the form presented here compared to previous studies is that we also constrain the expansion to behave invariantly by rotation and by translation in parameter space. Thus, moving the origin about which the polynomial expansion is constructed, or rotating the coordinate system about any such origin, should not affect the emulator.
In fact, in the next subsection we show that this natural constraint implies a precise and unique form for the deterministic emulator weights, well illustrating the intuition that weights should decrease as terms’ order increases. We shall see that the form of the resulting polynomial expansion depends only on two emulator parameters, σ A \sigma_A σ A , which sets the strength of the fluctuation of the pre-trained emulator at any point, and Λ \Lambda Λ , which drives the convergence of the polynomial expansion, as it is a power series expansion in terms of powers of the scaled parameter vector Λ − 1 θ ⃗ \Lambda^{-1}\vec{\theta} Λ − 1 θ . A reader might pause at this point to recognize that we will be dealing with N p a r s N_{\mathrm{pars}} N pars physical model parameters assembled in the vector θ ⃗ \vec{\theta} θ and with two scalar emulator parameters σ A \sigma_A σ A and Λ \Lambda Λ .
It is well known that expansions, including polynomial expansions, which are linear in their random coefficients, lead to emulators whose behavior can easily be expressed in terms of the pre-trained covariance kernel, C 0 ( θ ⃗ 1 , θ ⃗ 2 ) C_0(\vec{\theta}_1,\vec{\theta}_2) C 0 ( θ 1 , θ 2 ) . Both the emulator and its error are easily expressed in terms of C 0 C_0 C 0 , which allows one to avoid calculations where one must sum over the actual terms of the polynomial expansion. This result is known as the kernel trick [44 ] . As we said, this is not new. The new constructive element we introduce here, is that our assumption that all the random terms in the polynomial expansion are independent of each other, and our constraint of rotational and translational invariance, together imply a unique form for the expansion, and thus a unique form for the kernel. That unique form turns out to be that of an ordinary Gaussian kernel (i.e. a simple quadratic exponential decay) with Λ \Lambda Λ playing the role of characteristic length, in parameter space, of that quadratic exponential correlation-decay. By standardizing all parameters to be of the same magnitude, one then only requires a single characteristic length, i.e. just one scalar Λ \Lambda Λ .
Throughout this section the model parameters, θ ⃗ \vec{\theta} θ , are assumed to have been translated, rotated and scaled so that their expected sensitivities to a given step size Δ Θ \Delta\Theta ΔΘ are similar. For example, if a model parameter x i x_i x i has an initial prior, a i < x i < b i a_i < x_i < b_i a i < x i < b i , one defines a new translated and scaled variable
θ i = 2 θ max , i ( b i − a i ) ( x i − ( a i + b i ) / 2 ) . (1) \theta_i = \frac{2\theta_{\max,i}}{(b_i-a_i)}\left(x_i-(a_i+b_i)/2\right). \tag*{(1)} θ i = ( b i − a i ) 2 θ m a x , i ( x i − ( a i + b i ) /2 ) . (1)
where now the prior on θ i \theta_i θ i is uniform in the symmetric range ± θ max , i \pm\theta_{\max,i} ± θ m a x , i . The flexibility with choosing the scale θ max , i \theta_{\max,i} θ m a x , i is use so that, for each i i i , one would not expect the observable, y ( θ ⃗ ) y(\vec{\theta}) y ( θ ) , to be any more sensitive to steps of size Δ Θ \Delta\Theta ΔΘ in the direction of θ i \theta_i θ i than to steps of the same size in the direction of some other parameter θ j \theta_j θ j . Thus, if the sensitivity to θ 1 \theta_1 θ 1 was expected to be higher than the sensitivity to θ 2 \theta_2 θ 2 , one would reduce the ratio of θ max , 2 \theta_{\max,2} θ m a x , 2 to θ max , 1 \theta_{\max,1} θ m a x , 1 until the expected sensitivities were similar. All this can be attempted mathematically when partial derivatives of the model for y ( θ ⃗ ) y(\vec{\theta}) y ( θ ) can be estimated with some confidence, or it can be done empirically thanks to a limited number of full-model numerical evaluation, not emulation. Further, on the question of fixing issues with rotational dependence, to consider a simple example, if one expected y ( θ ⃗ ) y(\vec{\theta}) y ( θ ) to be more sensitive to a given direction in parameter space, e.g., θ 1 + θ 2 \theta_1+\theta_2 θ 1 + θ 2 , compared to some orthogonal direction, θ 1 − θ 2 \theta_1-\theta_2 θ 1 − θ 2 , one could rotate the coordinate system by 45 degrees, then scale the two dimensions accordingly. More generally, one can define a matrix, S i j S_{ij} S ij , describing the expected or empirical variability of y y y with respect to Δ θ i Δ θ j \Delta\theta_i\Delta\theta_j Δ θ i Δ θ j , then redefine a coordinate system where the new parameters were eigenvector of S S S , then scale the new parameters so that the eigenvalues of S S S were similar. One would then perform the model emulation in this new basis where the functions was not expected to vary more in any specific direction for the same step size, although the range of the prior could vary per directions. For the case where one had no information indicating that the model were more likely to vary across the prior in one direction than in another, all the values of θ max , i \theta_{\max,i} θ m a x , i would all be set to equal 1.
As we mentioned, when using a uniform prior on each parameter, we shall assume that the parameter θ i \theta_i θ i is distributed according to the following probability density:
P i ( θ i ) = { 1 2 θ max , i , − θ max , i < θ i < θ max , i , 0 , otherwise (2) P_i(\theta_i) =
\begin{cases}
\frac{1}{2\theta_{\max,i}}, & -\theta_{\max,i}<\theta_i<\theta_{\max,i},\\
0, & \text{otherwise}
\end{cases} \tag*{(2)} P i ( θ i ) = { 2 θ m a x , i 1 , 0 , − θ m a x , i < θ i < θ m a x , i , otherwise (2)
and as we just said, if all parameters are considered to be of equal importance, we let θ max , i = 1 \theta_{\max,i}=1 θ m a x , i = 1 for every i i i . If the model function y ( θ ⃗ ) y(\vec{\theta}) y ( θ ) is expected to vary mostly along a specific i i i th parameter’s direction, since everthing is invariant to a single scaling factor, we need only scale that parameter with θ max , i = 1 \theta_{\max,i}=1 θ m a x , i = 1 , and scale other parameters with appropriately chosen smaller values of θ max , j \theta_{\max,j} θ m a x , j , j ≠ i j\ne i j = i .
If the parameters have Gaussian priors, one can scale and translate each of them so that the new marginal priors are Gaussian of the form
P i ( θ i ) = 1 ( 2 π / β i ) 1 / 2 e − β i θ i 2 / 2 . (3) P_i(\theta_i)=\frac{1}{(2\pi/\beta_i)^{1/2}}e^{-\beta_i\theta_i^2/2}. \tag*{(3)} P i ( θ i ) = ( 2 π / β i ) 1/2 1 e − β i θ i 2 /2 . (3)
where the β i \beta_i β i ’s are chosen to place the parameter sensitivities on presumed equal footing. But the Gaussian case is easier to treat formally than the uniform case, because a multivariate Gaussian distribution has an intrinsic property of rotational invariance that the multivariate uniform distribution lacks. Therefore, if one insists on a specific multivariate Gaussian prior distribution on the entire vector θ ⃗ \vec{\theta} θ , with mean μ ⃗ \vec{\mu} μ and covariance matrix S S S , one then simply shifts the entire vector θ ⃗ \vec{\theta} θ by subtracting μ \mu μ , and one rotates and scales it simultaneously by multiplying it by a Cholesky square root L L L of S S S , i.e. a lower-triangular matrix L L L such that S = L L T S=LL^{\mathsf{T}} S = L L T . The new resulting multivariate prior is then the product of all the normal densities in (3) with each β i = 1 \beta_i=1 β i = 1 . These β i \beta_i β i ’s can then also be adjusted to make up for uneven sensitivities, similarly to what we suggested with the θ max , i \theta_{\max,i} θ m a x , i scales for uniform priors.
Again in the multivariate normal case, if we now believe that all the parameters are put on equal footing, but we want this multivariate prior to have the same dispersion properties as in the uniform case, one might want to match variances. The variance of the uniform distribution on [ − 1 , 1 ] [-1,1] [ − 1 , 1 ] being equal to 1 / 3 1/3 1/3 , the value of all the β i \beta_i β i ’s need only be set to equal 3 as well for the variances to match in the Gaussian and uniform priors cases. As we mentioned in the uniform prior case, if some given parameter (# i i i ) is expected to drive most of the variability that parameter is assigned β i = 3 \beta_i=3 β i = 3 and all other parameters are assigned lower values for their β j \beta_j β j ’s. Certainly, matching variances is not the only way to match dispersions. If instead one wanted a density in (3) with a β i \beta_i β i which matches the support of the uniform distribution [ − 1 , 1 ] [-1,1] [ − 1 , 1 ] at a confidence/credibility level of 95%, then one will want to choose not β i = 3 \beta_i=3 β i = 3 but rather closer to β i = 4 \beta_i=4 β i = 4 (strictly speaking, it would be closer to β i = 3.84 \beta_i=3.84 β i = 3.84 .) This is because the normal distribution has higher moments that increase slightly faster than those of the uniform distribution. In the sequel, we will stick with the variance-matching choice of β i = 3 \beta_i=3 β i = 3 for our expository purposes.
With our choice to use polynomial expansions in terms of θ i / Λ \theta_i/\Lambda θ i /Λ , as discussed above, it may be the case that the function to be emulated is known to be less sensitive to some model parameters than it is to others. Rather than assigning different Λ \Lambda Λ for different model parameters to reflect θ max , i \theta_{\max,i} θ m a x , i and β i − 1 / 2 \beta_i^{-1/2} β i − 1/2 difference in sensitivities, the prior scales θ max , i \theta_{\max,i} θ m a x , i and β i − 1 / 2 \beta_i^{-1/2} β i − 1/2 are modified from the values of θ max , i = 1 \theta_{\max,i}=1 θ m a x , i = 1 and β i − 1 / 2 = 1 / 3 \beta_i^{-1/2}=1/\sqrt{3} β i − 1/2 = 1/ 3 . This choice makes the mathematical description of the emulator simpler and maintains rotational invariance, while not forfeiting the ability to assume some parameters contribute in a more convergent manner than others. Because the emulator is a function of θ i / Λ \theta_i/\Lambda θ i /Λ , scaling θ i \theta_i θ i is mathematically identical to scaling Λ \Lambda Λ . Moreover, one can assume different prior forms, e.g. half Gaussians, exponentials, Laplace distributions, and these choices do not affect any of the derivations or results in this section. However, the choice of the parameter priors are critical in Section III, where methods to optimize the training-point placement are considered.
The remainder of this section includes several subsections briefly reviewing the methods needed to tune and train our proposed emulator. Training is accomplished by calculating the full function y ( θ ⃗ ) y(\vec{\theta}) y ( θ ) at N t r a i n N_{\mathrm{train}} N train predetermined locations θ a \theta_a θ a , a = 1 , 2 , … , N t r a i n a=1,2,\ldots,N_{\mathrm{train}} a = 1 , 2 , … , N train in model-parameter space. The training data is thus comprised of knowing y a = y ( θ ⃗ a ) y_a=y(\vec{\theta}_a) y a = y ( θ a ) for a = 1 , 2 , … , N t r a i n a=1,2,\ldots,N_{\mathrm{train}} a = 1 , 2 , … , N train . Expressions for the optimized coefficients are found in terms of the training data. After applying the aforementioned “kernel trick”, generating these coefficients is mostly unnecessary given that the emulator and its uncertainty can readily be expressed in terms of the training data and the covariance kernel, without ever performing (infinite) sums over all polynomial terms. Given that the resulting emulator is identical to one used in many GP treatments, our detailed mathematical and computational analysis of training and tuning might be viewed as unnecessary in practice. Nonetheless, it is insightful to understand what precise set of basis functions results in a GP emulator with a Gaussian covariance kernel.
One part of our analysis of training and tuning which does not appear in many GP treatments is the inclusion of the emulator uncertainty due to the uncertainty in choosing the emulation hyper-parameter Λ \Lambda Λ . Although this uncertainty term is not usually significant once one has a reasonable distribution of training points, it plays an important role in estimating the number of training points required for a desired emulation accuracy, and in optimizing training-point placement itself, a subject considered in detail in Section III. To summarize briefly, one of the conclusions of our analysis of optimal training point placement is that standard methods like random or quasi-random placement, or latin-hypercube placement, are clearly sub-optimal for our emulator class, and therefore are suboptimal for the standard GP emulators with Gaussian kernels.
Recall that our N p a r s N_{\mathrm{pars}} N pars -dimensional model parameters can be expressed as a vector θ ⃗ \vec{\theta} θ , and that we assume that the original model parameters have been rotated, shifted and scaled so that the prior ranges are defined as described above, with scale modulations to account for higher presumed sensitivities in some directions than others in parameter space. The first assumption for choosing the functional form is that it be analytic, so that it can be expanded as a polynomial in full parameter space, at least in the range of the prior. To avoid abusing the complex-analysis term “analytic” here, we can say that its meaning is the classical one, for the power series to have an infinite radius of convergence when the prior’s support is the full N p a r s N_{\mathrm{pars}} N pars -dimensional Euclidean space, and has a radius of convergence defining a sphere in that Euclidean space which contains the support of any prior when it is bounded. Here, we assume the following form for the emulator, Y ( θ ⃗ ) Y(\vec{\theta}) Y ( θ ) ,
Y ( θ ⃗ ) = ∑ n ⃗ d n ⃗ ( ∣ θ ⃗ ∣ 2 ) f K ( n ⃗ ) ( ∣ θ ⃗ ∣ 2 ) A n ⃗ ( θ 1 Λ ) n 1 ( θ 2 Λ ) n 2 ⋯ ( θ N Λ ) n N p a r s . (4) Y(\vec{\theta}) = \sum_{\vec{n}} d_{\vec{n}}(|\vec{\theta}|^2) f_{K(\vec{n})}(|\vec{\theta}|^2) A_{\vec{n}} \left(\frac{\theta_1}{\Lambda}\right)^{n_1} \left(\frac{\theta_2}{\Lambda}\right)^{n_2} \cdots\left(\frac{\theta_{N}}{\Lambda}\right)^{n_{N_{\mathrm{pars}}}} . \tag*{(4)} Y ( θ ) = n ∑ d n ( ∣ θ ∣ 2 ) f K ( n ) ( ∣ θ ∣ 2 ) A n ( Λ θ 1 ) n 1 ( Λ θ 2 ) n 2 ⋯ ( Λ θ N ) n N pars . (4)
Throughout the paper, the uppercase function Y ( θ ⃗ ) Y(\vec{\theta}) Y ( θ ) will refer to the emulator whereas the lowercase function y ( θ ⃗ ) y(\vec{\theta}) y ( θ ) refers to the evaluation of the full model. Here, K = K ( n ⃗ ) = ∣ n ⃗ ∣ = n 1 + n 2 + ⋯ + n N p a r s K = K(\vec{n}) = |\vec{n}| = n_1 + n_2 + \cdots+ n_{N_{\mathrm{pars}}} K = K ( n ) = ∣ n ∣ = n 1 + n 2 + ⋯ + n N pars is the order of the polynomial, d n ⃗ d_{\vec{n}} d n and f K ( n ⃗ ) f_{K(\vec{n})} f K ( n ) are functions of the squared norm ∣ θ ⃗ ∣ 2 |\vec{\theta}|^2 ∣ θ ∣ 2 . The vector n ⃗ \vec{n} n refers to the N p a r s N_{\mathrm{pars}} N pars -dimensional vector comprised of non-negative integers, ( n 1 , n 2 , ⋯ , n k ) (n_1,n_2,\cdots,n_k) ( n 1 , n 2 , ⋯ , n k ) .
The random coefficients A n ⃗ A_{\vec{n}} A n are considered to be independent in the absence of training data. Therefore, each polynomial term can be considered as a basis function. This set of basis functions is clearly non-orthogonal. The normalization and orthogonality properties are discussed in Sec. A. The form of the emulator class Y ( θ ⃗ ) Y(\vec{\theta}) Y ( θ ) appears rather general, in that all terms for a given K K K are independent. For the emulator class Y ( θ ⃗ ) Y(\vec{\theta}) Y ( θ ) to be an analytic function of emulator class θ ⃗ \vec{\theta} θ , and to avoid any unnecessary contortions, the functions f K f_K f K and d n ⃗ d_{\vec{n}} d n are best chosen to be polynomial expansions of the squared Euclidean norm ∣ θ ⃗ ∣ 2 |\vec{\theta}|^2 ∣ θ ∣ 2 . With this natural choice, the emulator Y ( θ ⃗ ) Y(\vec{\theta}) Y ( θ ) ’s current form above, where the infinite series is assumed to be absolutely summable within an appropriate radius of convergence, spans all possible analytic functions of Y ( θ ⃗ ) Y(\vec{\theta}) Y ( θ ) , i.e. functions that can be written as polynomials of θ ⃗ \vec{\theta} θ , or as uniform limits of these polynomials on compact sets in parameter space. However, this precludes explicit dependence on terms such as θ 1 / ∣ θ ⃗ ∣ \theta_1/|\vec{\theta}| θ 1 /∣ θ ∣ .
Our second assumption is about the random coefficients A n ⃗ A_{\vec{n}} A n . We assume that the coefficients are independent and identically distributed, mean-zero Gaussian random variables, with common variance σ A 2 \sigma_A^2 σ A 2 , i.e. with marginal density φ A n ⃗ ( a ) \varphi_{A_{\vec{n}}}(a) φ A n ( a ) and covariance structure given by
φ A n ⃗ ( a ) = 1 2 π σ A 2 e − a 2 / 2 σ A 2 , \varphi_{A_{\vec{n}}}(a) = \frac{1}{\sqrt{2\pi\sigma_A^2}} e^{-a^2/2\sigma_A^2}, φ A n ( a ) = 2 π σ A 2 1 e − a 2 /2 σ A 2 ,
Cov 0 ( A m ⃗ , A n ⃗ ) = σ A 2 δ m ⃗ , n ⃗ . (5) \operatorname{Cov}_0(A_{\vec{m}},A_{\vec{n}}) = \sigma_A^2 \delta_{\vec{m},\vec{n}} . \tag*{(5)} Cov 0 ( A m , A n ) = σ A 2 δ m , n . (5)
We use the subscript “0” to refer to the averaging (via expectations or covariances) over the prior, i.e. without any constraints from matching training data. The particular choice of using Gaussian coefficients does not affect any of the derivations of this section unless one is considering moments of higher order than quadratic, e.g a correlation of order Y 4 Y^4 Y 4 such as E 0 ( Y ( θ ⃗ a ) Y ( θ ⃗ b ) Y ( θ ⃗ c ) Y ( θ ⃗ d ) ) E_0(Y(\vec{\theta}_a)Y(\vec{\theta}_b)Y(\vec{\theta}_c)Y(\vec{\theta}_d)) E 0 ( Y ( θ a ) Y ( θ b ) Y ( θ c ) Y ( θ d )) . In Sec. II C it will be shown that each independent random coefficient A n ⃗ A_{\vec{n}} A n must have a Gaussian distribution if the higher-order moments are to all be independent of translation and rotation.
Any distribution that averages to zero and has some variance σ A 2 \sigma_A^2 σ A 2 would give the same behavior for moments involving only quadratic terms of Y Y Y . The choice that each random coefficient’s distribution is characterized by the same single scale parameter σ A \sigma_A σ A does not reduce the generality because d n ⃗ d_{\vec{n}} d n is still arbitrary up to this point.
Similarly, contrary to what intuition might dictate, the choice that these coefficients are all independent of one another does not fundamentally reduce the generality of the expression except for the fact that we choose to work a unique random coefficient per monomial, disallowing repeats among the n ⃗ \vec{n} n ’s. To illustrate this point, assume for a moment that one allows such repeats. Then the above form does not eliminate the possibility that different terms could be correlated. Indeed, as long as the overall joint distribution of the (infinite-dimensional) vector of all coefficients is assumed to be that of a Gaussian vector, a sum of independent terms can be achieved as follows. Any Gaussian vector, even with countably infinite dimension, has a covariance matrix that can be expressed for instance as its Cholesky decomposition. Then, its corresponding square root can be inverted and multiplied by the Gaussian vector, resulting in an explicit series of independent Gaussian terms for each n ⃗ \vec{n} n , if we allow repeats in the multi-indexes n ⃗ \vec{n} n . The flexibility afforded by the presence of the scaling d n ⃗ d_{\vec{n}} d n , and the allowance of repeats in the multi-indexes n ⃗ \vec{n} n , leads immediately to a countably infinite sum of independent terms, but with new values of the d n ⃗ d_{\vec{n}} d n ’s, and possibly more than one monomial term per n ⃗ \vec{n} n . The details are left to the reader.
If such repeats are not allowed, the fact that d n ⃗ d_{\vec{n}} d n can be any analytic function of ∣ θ ⃗ ∣ 2 |\vec{\theta}|^2 ∣ θ ∣ 2 does represent allowing correlations between polynomials of different order that differ only by powers of ∣ θ ⃗ ∣ 2 |\vec{\theta}|^2 ∣ θ ∣ 2 , but correlations between terms that have different angular behavior are forbidden if each n ⃗ \vec{n} n is assigned a single random coefficient. Some other treatments might instead expand in an orthonormal basis of functions, e.g. [46 ] . The basis functions are effectively the functions which are multiplied by the random coefficients to construct the emulator. Our functions here are simple products of various parameter components (monomials) multiplied by functions of ∣ θ ⃗ ∣ 2 |\vec{\theta}|^2 ∣ θ ∣ 2 , and are clearly not orthonormal. However, even with orthonormal terms one would be making some assumption that the terms are independent if each basis function is assigned a single random coefficient. It is not clear that orthonormality plays an important role in our application, but it is important to understand completeness. Our basis functions will become better defined below. The appendix, Sec. A, reviews the orthonormality and completeness properties of the basis functions used here.
The dependence on ∣ θ ⃗ ∣ 2 |\vec{\theta}|^2 ∣ θ ∣ 2 in the factors d n ⃗ d_{\vec{n}} d n and f K f_K f K is not superfluous. As mentioned, these terms provide the flexibility of correlating random coefficients for terms that differ only by factors of ∣ θ ⃗ ∣ 2 |\vec{\theta}|^2 ∣ θ ∣ 2 . However, the ∣ θ ⃗ ∣ |\vec{\theta}| ∣ θ ∣ -dependence is necessary if the variance is to be independent of the choice in positioning the origin. At the origin, only the first term, n ⃗ = 0 \vec{n}=0 n = 0 , contributes to the emulator or to its variance. If there were no ∣ θ ⃗ ∣ |\vec{\theta}| ∣ θ ∣ dependence, the variance away from the origin would have the same contribution from the n ⃗ = 0 \vec{n}=0 n = 0 term, but would also have contributions from other terms. Thus, to have a variance that is independent of position, the factors d n ⃗ d_{\vec{n}} d n and f K f_K f K must include some behavior that decreases with ∣ θ ⃗ ∣ |\vec{\theta}| ∣ θ ∣ so that as ∣ θ ⃗ ∣ |\vec{\theta}| ∣ θ ∣ increases and terms of higher K K K contribute more, terms of lower K K K contribute less. In the next subsection it will be shown that this dependence must be a Gaussian form.
Rotational and Translational Invariance
The third assumption will be that the fluctuation at some point θ ⃗ \vec{\theta} θ is independent of the direction of θ ⃗ \vec{\theta} θ . The fluctuation is
Var 0 ( Y ( θ ⃗ ) ) = ∑ n ⃗ f K 2 ( ∣ θ ⃗ ∣ 2 ) σ A 2 d n ⃗ 2 ( θ 1 2 n 1 Λ 2 ) ( θ 2 2 n 2 Λ 2 ) ⋯ ( θ N 2 n N Λ 2 ) . (6) \operatorname{Var}_0(Y(\vec{\theta})) = \sum_{\vec{n}} f_K^2(|\vec{\theta}|^2)\sigma_A^2 d_{\vec{n}}^2 \left(\frac{\theta_1^{2n_1}}{\Lambda^2}\right)\left(\frac{\theta_2^{2n_2}}{\Lambda^2}\right)\cdots\left(\frac{\theta_N^{2n_N}}{\Lambda^2}\right). \tag*{(6)} Var 0 ( Y ( θ )) = n ∑ f K 2 ( ∣ θ ∣ 2 ) σ A 2 d n 2 ( Λ 2 θ 1 2 n 1 ) ( Λ 2 θ 2 2 n 2 ) ⋯ ( Λ 2 θ N 2 n N ) . (6)
To be independent of direction, for a fixed norm the θ \theta θ dependence must be proportional to ∣ θ ⃗ ∣ 2 K |\vec{\theta}|^{2K} ∣ θ ∣ 2 K . Using the fact that
∣ θ ⃗ ∣ 2 K = ∑ n ⃗ , s . t . ∑ i n i = K K ! n 1 ! ⋯ n N ! ( θ 1 2 ) n 1 ( θ 2 2 ) n 2 ⋯ ( θ N p a r s 2 ) n N p a r s , (7) |\vec{\theta}|^{2K} = \sum_{\vec{n},\,s.t.\,\sum_i n_i=K} \frac{K!}{n_1!\cdots n_N!}(\theta_1^2)^{n_1}(\theta_2^2)^{n_2}\cdots(\theta_{N_{\mathrm{pars}}}^2)^{n_{N_{\mathrm{pars}}}}, \tag*{(7)} ∣ θ ∣ 2 K = n , s . t . ∑ i n i = K ∑ n 1 ! ⋯ n N ! K ! ( θ 1 2 ) n 1 ( θ 2 2 ) n 2 ⋯ ( θ N pars 2 ) n N pars , (7)
one can recognize that
d n ⃗ 2 ( ∣ θ ⃗ ∣ 2 ) ∝ 1 n 1 ! n 2 ! ⋯ n N p a r s ! . (8) d_{\vec{n}}^2(|\vec{\theta}|^2) \propto\frac{1}{n_1!n_2!\cdots n_{N_{\mathrm{pars}}}!}. \tag*{(8)} d n 2 ( ∣ θ ∣ 2 ) ∝ n 1 ! n 2 ! ⋯ n N pars ! 1 . (8)
It can also be some arbitrary function of K K K and ∣ θ ⃗ ∣ 2 |\vec{\theta}|^2 ∣ θ ∣ 2 , but that arbitrariness can be captured by f K ( ∣ θ ⃗ ∣ 2 ) f_K(|\vec{\theta}|^2) f K ( ∣ θ ∣ 2 ) . Thus, with complete generality, once the independence of direction is enforced one can set
d n ⃗ ( ∣ θ ⃗ ∣ 2 ) = 1 n 1 ! n 2 ! ⋯ n N p a r s ! . (9) d_{\vec{n}}(|\vec{\theta}|^2) = \frac{1}{\sqrt{n_1!n_2!\cdots n_{N_{\mathrm{pars}}}!}}. \tag*{(9)} d n ( ∣ θ ∣ 2 ) = n 1 ! n 2 ! ⋯ n N pars ! 1 . (9)
The number of ways one can arrange the K K K elements for a given n ⃗ \vec{n} n is K ! d n ⃗ 2 K!d_{\vec{n}}^{2} K ! d n 2 . The general form for the emulator, given the constraint that it is agnostic in regards to rotation in θ ⃗ \vec{\theta} θ space, is now
Y ( θ ⃗ ) = ∑ n ⃗ f K ( ∣ θ ⃗ ∣ 2 ) A n ⃗ n 1 ! n 2 ! ⋯ n N p a r s ! ( θ 1 Λ ) n 1 ( θ 2 Λ ) n 2 ⋯ ( θ N Λ ) n N p a r s . (10) Y(\vec{\theta}) = \sum_{\vec{n}} f_K(|\vec{\theta}|^2) \frac{A_{\vec{n}}}{\sqrt{n_1!n_2!\cdots n_{N_{\mathrm{pars}}}!}} \left(\frac{\theta_1}{\Lambda}\right)^{n_1}\left(\frac{\theta_2}{\Lambda}\right)^{n_2}\cdots\left(\frac{\theta_N}{\Lambda}\right)^{n_{N_{\mathrm{pars}}}}. \tag*{(10)} Y ( θ ) = n ∑ f K ( ∣ θ ∣ 2 ) n 1 ! n 2 ! ⋯ n N pars ! A n ( Λ θ 1 ) n 1 ( Λ θ 2 ) n 2 ⋯ ( Λ θ N ) n N pars . (10)
The fourth assumption is that the covariance for two different points should depend only on ∣ θ ⃗ 1 − θ ⃗ 2 ∣ 2 |\vec{\theta}_1-\vec{\theta}_2|^2 ∣ θ 1 − θ 2 ∣ 2 . This will fix the form for f K ( ∣ θ ⃗ ∣ 2 ) f_K(|\vec{\theta}|^2) f K ( ∣ θ ∣ 2 ) . The covariance is
Cov 0 ( Y ( θ ⃗ a ) , Y ( θ ⃗ b ) ) = σ A 2 ∑ n ⃗ 1 Λ 2 K f K ( ∣ θ ⃗ a ∣ 2 ) f K ( ∣ θ ⃗ b ∣ 2 ) n 1 ! n 2 ! ⋯ n N p a r s ! ( θ a , 1 θ b , 1 ) n 1 ( θ a , 2 θ b , 2 ) n 2 ⋯ ( θ a , N p a r s θ b , N p a r s ) n N p a r s = σ A 2 ∑ n ⃗ f K ( ∣ θ ⃗ a ∣ 2 ) f K ( ∣ θ ⃗ b ∣ 2 ) K ! ( θ ⃗ a ⋅ θ ⃗ b / Λ 2 ) K . (11) \begin{aligned}
\operatorname{Cov}_0(Y(\vec{\theta}_a),Y(\vec{\theta}_b)) &= \sigma_A^2 \sum_{\vec{n}} \frac{1}{\Lambda^{2K}}\frac{f_K(|\vec{\theta}_a|^2)f_K(|\vec{\theta}_b|^2)}{n_1!n_2!\cdots n_{N_{\mathrm{pars}}}!}(\theta_{a,1}\theta_{b,1})^{n_1}(\theta_{a,2}\theta_{b,2})^{n_2}\cdots(\theta_{a,N_{\mathrm{pars}}}\theta_{b,N_{\mathrm{pars}}})^{n_{N_{\mathrm{pars}}}} \\
&= \sigma_A^2 \sum_{\vec{n}} \frac{f_K(|\vec{\theta}_a|^2)f_K(|\vec{\theta}_b|^2)}{K!}(\vec{\theta}_a\cdot\vec{\theta}_b/\Lambda^2)^K. \tag*{(11)}
\end{aligned} Cov 0 ( Y ( θ a ) , Y ( θ b )) = σ A 2 n ∑ Λ 2 K 1 n 1 ! n 2 ! ⋯ n N pars ! f K ( ∣ θ a ∣ 2 ) f K ( ∣ θ b ∣ 2 ) ( θ a , 1 θ b , 1 ) n 1 ( θ a , 2 θ b , 2 ) n 2 ⋯ ( θ a , N pars θ b , N pars ) n N pars = σ A 2 n ∑ K ! f K ( ∣ θ a ∣ 2 ) f K ( ∣ θ b ∣ 2 ) ( θ a ⋅ θ b / Λ 2 ) K . (11)
The last step used the identity,
∣ θ ⃗ a ⋅ θ ⃗ b ∣ K = ∑ n ⃗ , s . t . ∑ i n i = K K ! n 1 ! ⋯ n N ! ( θ a , 1 θ b , 1 ) n 1 ( θ a , 2 θ b , 2 ) n 2 ⋯ ( θ a , N p a r s θ b , N p a r s ) n N p a r s , (12) |\vec{\theta}_a\cdot\vec{\theta}_b|^K = \sum_{\vec{n},\mathrm{s.t.}\sum_i n_i=K} \frac{K!}{n_1!\cdots n_N!}(\theta_{a,1}\theta_{b,1})^{n_1}(\theta_{a,2}\theta_{b,2})^{n_2}\cdots(\theta_{a,N_{\mathrm{pars}}}\theta_{b,N_{\mathrm{pars}}})^{n_{N_{\mathrm{pars}}}}, \tag*{(12)} ∣ θ a ⋅ θ b ∣ K = n , s.t. ∑ i n i = K ∑ n 1 ! ⋯ n N ! K ! ( θ a , 1 θ b , 1 ) n 1 ( θ a , 2 θ b , 2 ) n 2 ⋯ ( θ a , N pars θ b , N pars ) n N pars , (12)
If one were to choose
f K ( ∣ θ ⃗ ∣ 2 ) = e − ∣ θ ⃗ ∣ 2 / 2 Λ 2 , (13) f_K(|\vec{\theta}|^2)=e^{-|\vec{\theta}|^2/2\Lambda^2}, \tag*{(13)} f K ( ∣ θ ∣ 2 ) = e − ∣ θ ∣ 2 /2 Λ 2 , (13)
the resulting covariance would be
Cov 0 ( Y ( θ ⃗ a ) , Y ( θ ⃗ b ) ) = σ A 2 e − ∣ θ ⃗ a − θ ⃗ b ∣ 2 / 2 Λ 2 , (14) \operatorname{Cov}_0(Y(\vec{\theta}_a),Y(\vec{\theta}_b))=\sigma_A^2e^{-|\vec{\theta}_a-\vec{\theta}_b|^2/2\Lambda^2}, \tag*{(14)} Cov 0 ( Y ( θ a ) , Y ( θ b )) = σ A 2 e − ∣ θ a − θ b ∣ 2 /2 Λ 2 , (14)
which would satisfy the constraint of being a function of ∣ θ ⃗ a − θ ⃗ b ∣ |\vec{\theta}_a-\vec{\theta}_b| ∣ θ a − θ b ∣ . But, one can go further and show that this is the only such solution. To demonstrate this, one can write
f K ( ∣ θ ⃗ ∣ 2 ) = e ∣ θ ⃗ ∣ 2 / 2 Λ 2 + δ f K ( ∣ θ ⃗ ∣ 2 ) . (15) f_K(|\vec{\theta}|^2)=e^{|\vec{\theta}|^2/2\Lambda^2}+\delta f_K(|\vec{\theta}|^2). \tag*{(15)} f K ( ∣ θ ∣ 2 ) = e ∣ θ ∣ 2 /2 Λ 2 + δ f K ( ∣ θ ∣ 2 ) . (15)
Further, one can expand δ f K \delta f_K δ f K ,
δ f K ( ∣ θ ⃗ ∣ 2 ) = α K , 2 ∣ θ ⃗ ∣ 4 + α K , 3 ∣ θ ⃗ ∣ 6 + α K , 4 ∣ θ ⃗ ∣ 8 + ⋯ (16) \delta f_K(|\vec{\theta}|^2)=\alpha_{K,2}|\vec{\theta}|^4+\alpha_{K,3}|\vec{\theta}|^6+\alpha_{K,4}|\vec{\theta}|^8+\cdots \tag*{(16)} δ f K ( ∣ θ ∣ 2 ) = α K , 2 ∣ θ ∣ 4 + α K , 3 ∣ θ ∣ 6 + α K , 4 ∣ θ ∣ 8 + ⋯ (16)
The series begins for quartic terms because adding a quadratic term, beyond what is in the Gaussian in Eq. (15) , would violate the constraint that overall the quadratic term in Eq. (11) must behave as ∣ θ ⃗ a − θ ⃗ b ∣ 2 |\vec{\theta}_a-\vec{\theta}_b|^2 ∣ θ a − θ b ∣ 2 . The corrections due to δ f K \delta f_K δ f K of quartic behavior are
δ Cov 0 ( Y ( θ ⃗ a ) , Y ( θ ⃗ b ) ) = α 0 , 2 ∣ θ ⃗ a ∣ 4 + α 0 , 2 ∣ θ ⃗ b ∣ 4 + α 2 , 0 ( θ ⃗ a ⋅ θ ⃗ b ) 2 . (17) \delta\operatorname{Cov}_0(Y(\vec{\theta}_a),Y(\vec{\theta}_b))=\alpha_{0,2}|\vec{\theta}_a|^4+\alpha_{0,2}|\vec{\theta}_b|^4+\alpha_{2,0}(\vec{\theta}_a\cdot\vec{\theta}_b)^2. \tag*{(17)} δ Cov 0 ( Y ( θ a ) , Y ( θ b )) = α 0 , 2 ∣ θ a ∣ 4 + α 0 , 2 ∣ θ b ∣ 4 + α 2 , 0 ( θ a ⋅ θ b ) 2 . (17)
One can set α 0 , 2 = − α 2 , 0 / 2 \alpha_{0,2}=-\alpha_{2,0}/2 α 0 , 2 = − α 2 , 0 /2 to make this disappear when θ ⃗ 1 = θ ⃗ 2 \vec{\theta}_1=\vec{\theta}_2 θ 1 = θ 2 , but this correction is not proportional to ∣ θ ⃗ 1 − θ ⃗ 2 ∣ 4 |\vec{\theta}_1-\vec{\theta}_2|^4 ∣ θ 1 − θ 2 ∣ 4 . The same can be said for higher-order terms in θ \theta θ . Thus, δ f \delta f δ f must be zero and the form for f K ( ∣ θ ⃗ ∣ 2 ) f_K(|\vec{\theta}|^2) f K ( ∣ θ ∣ 2 ) is completely determined to match that of Eq. (15) if one insists that the covariance be a function of ∣ θ ⃗ 1 − θ ⃗ 2 ∣ 2 |\vec{\theta}_1-\vec{\theta}_2|^2 ∣ θ 1 − θ 2 ∣ 2 .
After enforcing the four criteria, the form for the emulator and covariance are now fixed to
Y ( θ ⃗ ) = e − ∣ θ ⃗ ∣ 2 / 2 Λ 2 ∑ n ⃗ A n ⃗ n 1 ! n 2 ! ⋯ n N p a r s ! ( θ 1 Λ ) n 1 ( θ 2 Λ ) n 2 ⋯ ( θ N Λ ) n N p a r s , (18) Y(\vec{\theta})=e^{-|\vec{\theta}|^2/2\Lambda^2}\sum_{\vec{n}}\frac{A_{\vec{n}}}{\sqrt{n_1!n_2!\cdots n_{N_{\mathrm{pars}}}!}}\left(\frac{\theta_1}{\Lambda}\right)^{n_1}\left(\frac{\theta_2}{\Lambda}\right)^{n_2}\cdots\left(\frac{\theta_N}{\Lambda}\right)^{n_{N_{\mathrm{pars}}}}, \tag*{(18)} Y ( θ ) = e − ∣ θ ∣ 2 /2 Λ 2 n ∑ n 1 ! n 2 ! ⋯ n N pars ! A n ( Λ θ 1 ) n 1 ( Λ θ 2 ) n 2 ⋯ ( Λ θ N ) n N pars , (18)
Cov 0 ( Y ( θ ⃗ a ) , Y ( θ ⃗ b ) ) = σ A 2 e − ∣ θ ⃗ a − θ ⃗ b ∣ 2 / 2 Λ 2 , \operatorname{Cov}_0(Y(\vec{\theta}_a),Y(\vec{\theta}_b))=\sigma_A^2e^{-|\vec{\theta}_a-\vec{\theta}_b|^2/2\Lambda^2}, Cov 0 ( Y ( θ a ) , Y ( θ b )) = σ A 2 e − ∣ θ a − θ b ∣ 2 /2 Λ 2 ,
Var 0 ( Y ( θ ⃗ ) ) = σ A 2 . \operatorname{Var}_0(Y(\vec{\theta}))=\sigma_A^2. Var 0 ( Y ( θ )) = σ A 2 .
In fact, for the four criteria mentioned above, satisfying the third criteria is automatically satisfied if the fourth is satisfied, because the fluctuation is simply the covariance evaluated at θ ⃗ 1 = θ ⃗ 2 \vec{\theta}_1=\vec{\theta}_2 θ 1 = θ 2 . Thus, only three criteria are required to constrain the form to that of Eq. (18) . The form for the emulator and covariances is fixed to that of Eq. (18) if the following choices or assumptions are made:
The function should be analytic throughout the prior.
At any point the function should vary according to the same Gaussian distribution.
The prior-to-constraint covariance, Cov 0 ( Y ( θ ⃗ a ) , Y ( θ ⃗ b ) ) \operatorname{Cov}_0(Y(\vec{\theta}_a), Y(\vec{\theta}_b)) Cov 0 ( Y ( θ a ) , Y ( θ b )) , should depend only on ∣ θ ⃗ 1 − θ ⃗ 2 ∣ 2 \lvert\vec{\theta}_1-\vec{\theta}_2\rvert^2 ∣ θ 1 − θ 2 ∣ 2 .
From here on, the covariance will be referred through the function C 0 ( θ ⃗ a , θ ⃗ b ) C_0(\vec{\theta}_a,\vec{\theta}_b) C 0 ( θ a , θ b ) , defined as
C 0 ( θ ⃗ a , θ ⃗ b ) ≡ 1 σ A 2 Cov 0 ( Y ( θ ⃗ a ) , Y ( θ ⃗ b ) ) = e − ∣ θ ⃗ a − θ ⃗ b ∣ 2 / 2 Λ 2 . (19) C_0(\vec{\theta}_a,\vec{\theta}_b) \equiv\frac{1}{\sigma_A^2}\operatorname{Cov}_0(Y(\vec{\theta}_a),Y(\vec{\theta}_b)) = e^{-\lvert\vec{\theta}_a-\vec{\theta}_b\rvert^2/2\Lambda^2}. \tag*{(19)} C 0 ( θ a , θ b ) ≡ σ A 2 1 Cov 0 ( Y ( θ a ) , Y ( θ b )) = e − ∣ θ a − θ b ∣ 2 /2 Λ 2 . (19)
The subscript zero again refers to the fact that this averaging is performed before one has applied any constraints from the training data.
Higher Moments and the Assumption that Coefficients have Gaussian Distributions
One might ask whether all other moments, e.g. E 0 ( Y 4 ( θ ⃗ ) ) E_0(Y^4(\vec{\theta})) E 0 ( Y 4 ( θ )) , are also independent of θ ⃗ \vec{\theta} θ . The answer is yes, and derives from the initial choice that the weights for the coefficients are Gaussian. For Gaussians centered at zero, all cumulants are zero except for the second cumulant, which is σ A 2 \sigma_A^2 σ A 2 . Thus, when averaging over the initial coefficients, the averaged fourth moment of the coefficients becomes
Cov 0 ( A i , A j , A m , A n ) = Cov 0 ( A i , A j ) Cov 0 ( A m , A n ) + Cov 0 ( A i , A m ) Cov 0 ( A j , A n ) + Cov 0 ( A i , A n ) Cov 0 ( A j , A m ) . (20) \operatorname{Cov}_0(A_i,A_j,A_m,A_n)=\operatorname{Cov}_0(A_i,A_j)\operatorname{Cov}_0(A_m,A_n)
+\operatorname{Cov}_0(A_i,A_m)\operatorname{Cov}_0(A_j,A_n)+\operatorname{Cov}_0(A_i,A_n)\operatorname{Cov}_0(A_j,A_m). \tag*{(20)} Cov 0 ( A i , A j , A m , A n ) = Cov 0 ( A i , A j ) Cov 0 ( A m , A n ) + Cov 0 ( A i , A m ) Cov 0 ( A j , A n ) + Cov 0 ( A i , A n ) Cov 0 ( A j , A m ) . (20)
For any distribution where E 0 ( A i ) = 0 E_0(A_i)=0 E 0 ( A i ) = 0 , the independent coefficients, Eq. (20) would be zero if the indices were not all the same, e.g. if i = j i=j i = j and m = n m=n m = n , but i ≠ m i\ne m i = m . Gaussian distributions are special in that this holds true even if i = j = m = n i=j=m=n i = j = m = n . If one expands Y ( θ ⃗ ) Y(\vec{\theta}) Y ( θ ) in terms of the coefficients and polynomials, one readily sees that this implies that
Cov 0 ( Y ( θ ⃗ a ) , Y ( θ ⃗ b ) , Y ( θ ⃗ c ) , Y ( θ ⃗ d ) ) = Cov 0 ( Y ( θ ⃗ a ) , Y ( θ ⃗ b ) ) Cov 0 ( Y ( θ ⃗ c ) , Y ( θ ⃗ d ) ) + Cov 0 ( Y ( θ ⃗ a ) , Y ( θ ⃗ c ) ) Cov 0 ( Y ( θ ⃗ b ) , Y ( θ ⃗ d ) ) + Cov 0 ( Y ( θ ⃗ a ) , Y ( θ ⃗ d ) ) Cov 0 ( Y ( θ ⃗ b ) , Y ( θ ⃗ c ) ) . (21) \operatorname{Cov}_0(Y(\vec{\theta}_a),Y(\vec{\theta}_b),Y(\vec{\theta}_c),Y(\vec{\theta}_d))=\operatorname{Cov}_0(Y(\vec{\theta}_a),Y(\vec{\theta}_b))\operatorname{Cov}_0(Y(\vec{\theta}_c),Y(\vec{\theta}_d))
+\operatorname{Cov}_0(Y(\vec{\theta}_a),Y(\vec{\theta}_c))\operatorname{Cov}_0(Y(\vec{\theta}_b),Y(\vec{\theta}_d))+\operatorname{Cov}_0(Y(\vec{\theta}_a),Y(\vec{\theta}_d))\operatorname{Cov}_0(Y(\vec{\theta}_b),Y(\vec{\theta}_c)). \tag*{(21)} Cov 0 ( Y ( θ a ) , Y ( θ b ) , Y ( θ c ) , Y ( θ d )) = Cov 0 ( Y ( θ a ) , Y ( θ b )) Cov 0 ( Y ( θ c ) , Y ( θ d )) + Cov 0 ( Y ( θ a ) , Y ( θ c )) Cov 0 ( Y ( θ b ) , Y ( θ d )) + Cov 0 ( Y ( θ a ) , Y ( θ d )) Cov 0 ( Y ( θ b ) , Y ( θ c )) . (21)
All covariances to order Y 4 Y^4 Y 4 can therefore be expressed in terms of covariances of order Y 2 Y^2 Y 2 . If the covariances of order Y 2 Y^2 Y 2 are independent of translations and rotations, the covariances of order Y 4 Y^4 Y 4 must then also satisfy this requirement. If all the coordinates of a covariance are rotated or translated, the covariance is unchanged. For the particular case where one considers all four angles to be equal to one another this gives
E 0 ( Y 4 ( θ ⃗ ) ) = 3 σ A 4 , (22) E_0(Y^4(\vec{\theta}))=3\sigma_A^4, \tag*{(22)} E 0 ( Y 4 ( θ )) = 3 σ A 4 , (22)
which is independent of θ ⃗ \vec{\theta} θ . In fact, because all cumulants disappear, covariances to any order, Cov 0 ( Y ( θ ⃗ 1 ) , ⋯ Y ( θ ⃗ n ) ) \operatorname{Cov}_0(Y(\vec{\theta}_1),\cdots Y(\vec{\theta}_n)) Cov 0 ( Y ( θ 1 ) , ⋯ Y ( θ n )) , can be expressed in terms of Cov 0 ( Y ( θ ⃗ i ) , Y ( θ ⃗ j ) ) = σ A 2 e − ∣ θ ⃗ i − θ ⃗ j ∣ 2 / 2 Λ 2 \operatorname{Cov}_0(Y(\vec{\theta}_i),Y(\vec{\theta}_j))=\sigma_A^2e^{-\lvert\vec{\theta}_i-\vec{\theta}_j\rvert^2/2\Lambda^2} Cov 0 ( Y ( θ i ) , Y ( θ j )) = σ A 2 e − ∣ θ i − θ j ∣ 2 /2 Λ 2 .
If one replaces the choice of Gaussian distributions for the coefficients with another distribution with the same second moments, with all the odd moments being zero, but with different cumulants for the higher even moments, the property that the covariances will only depend on relative coordinates will be broken. For example, if one considers the fourth moment,
E 0 ( A i 4 ) = 3 σ A 4 + κ . (23) E_0(A_i^4)=3\sigma_A^4+\kappa. \tag*{(23)} E 0 ( A i 4 ) = 3 σ A 4 + κ . (23)
Here, κ = 0 \kappa=0 κ = 0 for a Gaussian distribution. If one were to calculate the fourth moment of E E E with all four positions being equal, and with θ ⃗ \vec{\theta} θ being along the first axis, θ ⃗ = [ θ 1 , 0 , 0 … ] \vec{\theta}=[\theta_1,0,0\ldots] θ = [ θ 1 , 0 , 0 … ] ,
E 0 ( Y ( θ ⃗ ) 4 ) = 3 σ A 4 + κ e − 2 θ 1 2 / Λ 2 ∑ n 1 ( n ! ) 2 θ 1 4 n . (24) E_0(Y(\vec{\theta})^4)=3\sigma_A^4+\kappa e^{-2\theta_1^2/\Lambda^2}\sum_n\frac{1}{(n!)^2}\theta_1^{4n}. \tag*{(24)} E 0 ( Y ( θ ) 4 ) = 3 σ A 4 + κ e − 2 θ 1 2 / Λ 2 n ∑ ( n ! ) 2 1 θ 1 4 n . (24)
Clearly, this is not independent of θ 1 \theta_1 θ 1 , and is clearly not a function of relative coordinates, as all the relative coordinates are zero. The correlations are then clearly not invariant under translation and rotation. Thus, if one wishes all the higher-order correlations to depend only on relative coordinates and be independent of translation and rotation, the distribution for the coefficients must have zero cumulants. The distribution W ( A n ˉ ) W(A_{\bar{n}}) W ( A n ˉ ) in Eq. (5) should therefore be of a Gaussian form.
Comparison with Gaussian Process Emulators
Ultimately, an emulator constructed from the form proposed above depends only on the unconstrained correlations, C 0 ( θ ⃗ i , θ ⃗ j ) C_0(\vec{\theta}_i,\vec{\theta}_j) C 0 ( θ i , θ j ) , a result known as the kernel trick. The emulators then map perfectly to Gaussian process emulators, which are also perfectly defined once one knows C 0 ( θ ⃗ i , θ ⃗ j ) C_0(\vec{\theta}_i,\vec{\theta}_j) C 0 ( θ i , θ j ) . Here, the convergence parameter Λ \Lambda Λ is introduced to set the relative strength of order K + 1 K+1 K + 1 terms to order K K K terms, whereas in Gaussian process emulators the same parameter is introduced to set the correlation of the unconstrained correlation. Despite these differences in starting points and perspective, the resulting emulators are identical. This equivalence demonstrates how Gaussian process emulators can be equivalently viewed as polynomial expansions if one uses the basis assumption in Eq. (4) .
Gaussian process emulators often assume the correlation has a more general form,
C 0 ( θ ⃗ 1 , θ ⃗ 2 ) ∼ exp { − ( ∣ θ ⃗ 1 − θ ⃗ 2 ∣ 2 2 Λ 2 ) α } , (25) C_0(\vec{\theta}_1,\vec{\theta}_2) \sim\exp\left\{-\left(\frac{|\vec{\theta}_1-\vec{\theta}_2|^2}{2\Lambda^2}\right)^\alpha\right\}, \tag*{(25)} C 0 ( θ 1 , θ 2 ) ∼ exp { − ( 2 Λ 2 ∣ θ 1 − θ 2 ∣ 2 ) α } , (25)
where α \alpha α is an additionally adjustable hyper parameter. For α = 1 \alpha=1 α = 1 this corresponds precisely to the form discussed above. If the Taylor expansion for the emulator is analytic, one should expect that the kernel, C 0 ( θ ⃗ 1 , θ ⃗ 2 ) C_0(\vec{\theta}_1,\vec{\theta}_2) C 0 ( θ 1 , θ 2 ) , should also be analytic in terms of θ ⃗ \vec{\theta} θ . The form expressed in Eq. (25) will only be analytic if α \alpha α is an integer. Thus, the Gaussian form for the emulator in Eq. (18) should not come as a surprise. What is perhaps surprising is that α \alpha α cannot be some other integer.
Despite the fact that the emulators are identical in practice to GP forms, there is insight into viewing the emulator from the perspective of polynomial expansions. Once one considers the emulator form in terms of terms of a polynomial expansion, one also understands how many training points are needed to constrain all terms of a specific order. For example, for a linear fit, N p a r s + 1 N_{\rm pars}+1 N pars + 1 training points are needed to determine the intercept and slopes. For a quadratic fit, one would need ( N p a r s + 1 ) ( N p a r s + 2 ) / 2 (N_{\rm pars}+1)(N_{\rm pars}+2)/2 ( N pars + 1 ) ( N pars + 2 ) /2 terms, and for a cubic fit one would need ( N p a r s + 1 ) ( N p a r s + 2 ) ( N p a r s + 3 ) / 6 (N_{\rm pars}+1)(N_{\rm pars}+2)(N_{\rm pars}+3)/6 ( N pars + 1 ) ( N pars + 2 ) ( N pars + 3 ) /6 training points. Thus, it might motivate one to choose a number of training points to constrain all coefficients to a given order.
Tuning the Emulator
Due to the Gaussian weights, the most probable set of expansion coefficients would be when all A n ⃗ A_{\vec{n}} A n are zero. However, once one fits to N t r a i n N_{\rm train} N train training points one needs to find the set of coefficients that both maximizes the weight function, W ( A n ⃗ ) W(A_{\vec{n}}) W ( A n ) , while reproducing the N t r a i n N_{\rm train} N train training points. Whereas there might be dozens or a few hundred training points, there are an infinite number of coefficients. Thus, one needs to maximize the weight with N t r a i n N_{\rm train} N train constraints. Here, it is shown how one can analytically solve for the set of most probable coefficients, A n ⃗ A_{\vec{n}} A n , given the training values, y a = y ( θ ⃗ a ) y_a=y(\vec{\theta}_a) y a = y ( θ a ) , where θ ⃗ a \vec{\theta}_a θ a are the points at which the emulator was trained. For shorthand, we define the function T n ⃗ ( θ ⃗ ) T_{\vec{n}}(\vec{\theta}) T n ( θ ) to rewrite the emulator in Eq. (18) ,
Y ( θ ⃗ ) = ∑ n ⃗ A n ⃗ T n ⃗ ( θ ⃗ ) , (26) Y(\vec{\theta})=\sum_{\vec{n}} A_{\vec{n}}T_{\vec{n}}(\vec{\theta}), \tag*{(26)} Y ( θ ) = n ∑ A n T n ( θ ) , (26)
T n ⃗ ( θ ⃗ ) = 1 n 1 ! n 2 ! ⋯ n N p a r s ! ( θ 1 Λ ) n 1 ( θ 2 Λ ) n 2 ⋯ ( θ N Λ ) n N p a r s e − ∣ θ ⃗ ∣ 2 / 2 Λ 2 . T_{\vec{n}}(\vec{\theta})=\frac{1}{\sqrt{n_1!n_2!\cdots n_{N_{\rm pars}}!}}\left(\frac{\theta_1}{\Lambda}\right)^{n_1}\left(\frac{\theta_2}{\Lambda}\right)^{n_2}\cdots\left(\frac{\theta_N}{\Lambda}\right)^{n_{N_{\rm pars}}}e^{-|\vec{\theta}|^2/2\Lambda^2}. T n ( θ ) = n 1 ! n 2 ! ⋯ n N pars ! 1 ( Λ θ 1 ) n 1 ( Λ θ 2 ) n 2 ⋯ ( Λ θ N ) n N pars e − ∣ θ ∣ 2 /2 Λ 2 .
Here, T n ⃗ ( θ ⃗ ) T_{\vec{n}}(\vec{\theta}) T n ( θ ) is shorthand for the basis functions used to represent the emulator. None of the following derivations are dependent on this particular set of basis functions.
Evaluated at the N t r a i n N_{\rm train} N train training points, one can define the N t r a i n × N c o e f N_{\rm train}\times N_{\rm coef} N train × N coef matrix,
T a , n ⃗ ≡ T n ⃗ ( θ ⃗ a ) , (27) T_{a,\vec{n}}\equiv T_{\vec{n}}(\vec{\theta}_a), \tag*{(27)} T a , n ≡ T n ( θ a ) , (27)
where θ ⃗ a \vec{\theta}_a θ a labels the training points, 0 < a < N t r a i n 0<a<N_{\rm train} 0 < a < N train . The number of coefficients, N c o e f N_{\rm coef} N coef , approaches infinity.
The goal is to find the coefficients, A n ⃗ A_{\vec{n}} A n , that maximize the probability given the constraints of matching the training data. If the training data are to be exactly matched, the differential probability of a given set of coefficients is
d P ( A ⃗ , σ A , Λ ) ∼ ∏ n ⃗ d A n ⃗ ( 2 π σ A 2 ) 1 / 2 e − A n ⃗ 2 / 2 σ A 2 ∏ a = 1 N t r a i n δ [ y ( θ ⃗ a ) − Y ( θ ⃗ a ) ] . (28) dP(\vec{A},\sigma_A,\Lambda)\sim\prod_{\vec{n}}\frac{dA_{\vec{n}}}{(2\pi\sigma_{A}^2)^{1/2}}e^{-A_{\vec{n}}^2/2\sigma_A^2}\prod_{a=1}^{N_{\rm train}}\delta[y(\vec{\theta}_a)-Y(\vec{\theta}_a)]. \tag*{(28)} d P ( A , σ A , Λ ) ∼ n ∏ ( 2 π σ A 2 ) 1/2 d A n e − A n 2 /2 σ A 2 a = 1 ∏ N train δ [ y ( θ a ) − Y ( θ a )] . (28)
If the training constraints are not exact, the delta function should be replaced with some other form, perhaps a Gaussian. The Gaussian would be reasonable if the training had some sort of point-by-point noise, such as one would have if the full-model calculation involved some sort of sampling or Monte Carlo evaluations. However, more generally, one could also consider the fact that the training calculation might have some sort of error, e.g. that of missing physics. In that case the error might be correlated between points. Noise in the full-model calculations at points θ ⃗ a \vec{\theta}_a θ a and θ ⃗ b \vec{\theta}_b θ b can be characterized by some error matrix,
Cov ( y a , y b ) = σ A 2 Δ a b . (29) \operatorname{Cov}(y_a,y_b)=\sigma_A^2\Delta_{ab}. \tag*{(29)} Cov ( y a , y b ) = σ A 2 Δ ab . (29)
Here, y a = y ( θ ⃗ a ) y_a=y(\vec{\theta}_a) y a = y ( θ a ) , the full model value calculated at the training point a a a , and δ y a \delta y_a δ y a is the uncertainty of the full model due to point-by-point noise in the full model or any other source of error. The choice to scale out the factor of σ A 2 \sigma_A^2 σ A 2 in the definition is so that Δ a b \Delta_{ab} Δ ab is dimensionless. If the random noise is completely uncorrelated, Δ a b \Delta_{ab} Δ ab is diagonal. For example, if one were to recalculate the full model at exactly the same value of θ ⃗ a \vec{\theta}_a θ a , the new value of y a y_a y a could be different. This is often the case for models involving Monte Carlo calculation or if numerical noise is non-negligible. Given that the characteristic size of the observable is σ A \sigma_A σ A , the elements of the diagonal matrix would then be chosen according to the percentage random error. For example, if the random error were 1%, one would chose Δ a b = δ a b 10 − 4 \Delta_{ab}=\delta_{ab}10^{-4} Δ ab = δ ab 1 0 − 4 .
To account for the uncertainty represented by Δ \Delta Δ , one can relax the constraint imposed by the delta function by replacing it,
∏ a δ [ y a − Y ( θ ⃗ a ) ] → ∣ Δ ∣ − 1 / 2 ( 2 π ) N t r a i n / 2 exp { − 1 2 ( y a − Y ( θ ⃗ a ) ) Δ a b − 1 ( y b − Y ( θ ⃗ b ) ) } . (30) \prod_a \delta[y_a-Y(\vec{\theta}_a)]\to\frac{|\Delta|^{-1/2}}{(2\pi)^{N_{\rm train}/2}}\exp\left\{-\frac{1}{2}(y_a-Y(\vec{\theta}_a))\Delta_{ab}^{-1}(y_b-Y(\vec{\theta}_b))\right\}. \tag*{(30)} a ∏ δ [ y a − Y ( θ a )] → ( 2 π ) N train /2 ∣Δ ∣ − 1/2 exp { − 2 1 ( y a − Y ( θ a )) Δ ab − 1 ( y b − Y ( θ b )) } . (30)
The second step above exploits the fact that the Fourier transform of a Gaussian is a Gaussian. The r.h.s. above behaves like a delta function in the limit that the eigenvalues of Δ \Delta Δ approach infinity. Otherwise, in the basis where Δ \Delta Δ is diagonal this can be thought of as a finite-width modification to a delta function. Next, one can replace Y ( θ ⃗ ) Y(\vec{\theta}) Y ( θ ) in Eq. (30) using Eq. (26) . The altered δ \delta δ functions then become
∏ a δ [ y a − Y ( θ ⃗ a ) ] → ∫ ∏ a ( d k a 2 π ) exp { i k a ( y a − ∑ m ⃗ A m ⃗ T m ⃗ ( θ ⃗ a ) ) − ∑ b σ A 2 2 k a Δ a b k b } . (31) \prod_a\delta[y_a-Y(\vec{\theta}_a)]\to\int\prod_a\left(\frac{dk_a}{2\pi}\right)\exp\left\{ik_a\left(y_a-\sum_{\vec{m}}A_{\vec{m}}T_{\vec{m}}(\vec{\theta}_a)\right)-\sum_b\frac{\sigma_A^2}{2}k_a\Delta_{ab}k_b\right\}. \tag*{(31)} a ∏ δ [ y a − Y ( θ a )] → ∫ a ∏ ( 2 π d k a ) exp { i k a ( y a − m ∑ A m T m ( θ a ) ) − b ∑ 2 σ A 2 k a Δ ab k b } . (31)
Defining the net probability after integrating over the coefficients above, but not over σ A 2 \sigma_A^2 σ A 2 or Λ \Lambda Λ ,
Z ( σ A 2 , Λ ) ≡ ∫ ( ∏ n ⃗ d A n ⃗ ) d P ( A ⃗ , σ A , Λ ) ∏ n ⃗ d A n ⃗ = ∫ ( ∏ n ⃗ d A n ⃗ 2 π σ A 2 ) exp [ − ∑ i A i 2 2 σ A 2 ] ∫ d k 2 π exp [ i ∑ b k b ( y b − ∑ ℓ A ℓ T ℓ ( θ ⃗ b ) ) − σ A 2 2 ∑ a b k a Δ a b k b ] . (32) \begin{aligned}
\mathcal{Z}(\sigma_A^2,\Lambda)\equiv\int\left(\prod_{\vec{n}}dA_{\vec{n}}\right)\frac{dP(\vec{A},\sigma_A,\Lambda)}{\prod_{\vec{n}}dA_{\vec{n}}} \\
&=\int\left(\prod_{\vec{n}}\frac{dA_{\vec{n}}}{\sqrt{2\pi\sigma_A^2}}\right)\exp\left[-\frac{\sum_i A_i^2}{2\sigma_A^2}\right]\int\frac{dk}{2\pi}\exp\left[i\sum_b k_b\left(y_b-\sum_{\ell}A_\ell T_\ell(\vec{\theta}_b)\right)-\frac{\sigma_A^2}{2}\sum_{ab}k_a\Delta_{ab}k_b\right]. \tag*{(32)}
\end{aligned} Z ( σ A 2 , Λ ) ≡ ∫ ( n ∏ d A n ) ∏ n d A n d P ( A , σ A , Λ ) = ∫ ( n ∏ 2 π σ A 2 d A n ) exp [ − 2 σ A 2 ∑ i A i 2 ] ∫ 2 π d k exp [ i b ∑ k b ( y b − ℓ ∑ A ℓ T ℓ ( θ b ) ) − 2 σ A 2 ab ∑ k a Δ ab k b ] . (32)
The function Z ( σ A 2 , Λ ) \mathcal{Z}(\sigma_A^2,\Lambda) Z ( σ A 2 , Λ ) is the conditional probability of observing the training data given σ A \sigma_A σ A and Λ \Lambda Λ given the training data. From Eq. (28) one can see that Z ( σ A 2 , Λ ) \mathcal{Z}(\sigma_A^2,\Lambda) Z ( σ A 2 , Λ ) is normalized when integrated over all possible training data,
∫ ( ∏ a d y a ) Z ( σ A 2 , Λ ) = 1. (33) \int\left(\prod_a dy_a\right)\mathcal{Z}(\sigma_A^2,\Lambda)=1. \tag*{(33)} ∫ ( a ∏ d y a ) Z ( σ A 2 , Λ ) = 1. (33)
The average coefficients, and correlation of coefficients, for a given σ A \sigma_A σ A and Λ \Lambda Λ , become
E ( A n ⃗ ) = Z − 1 ∫ ( ∏ i ⃗ d A i ⃗ 2 π σ A 2 ) exp [ − ∑ j A j 2 2 σ A 2 ] A n ⃗ × ∫ ( ∏ a d k a 2 π ) exp [ i ∑ b k b ( y b − ∑ ℓ A ℓ T ℓ ( θ ⃗ b ) ) − σ A 2 2 ∑ a b k a Δ a b k b ] , Cov ( A m ⃗ , A n ⃗ ) = Z − 1 ∫ ( ∏ i ⃗ d A i ⃗ 2 π σ A 2 ) exp [ − ∑ j A j 2 2 σ A 2 − σ A 2 2 ∑ a b k a Δ a b k b ] × A m ⃗ A n ⃗ ∫ ( ∏ a d k a 2 π ) exp [ i ∑ b k b ( y b − ∑ ℓ A ℓ T ℓ ( θ ⃗ b ) ) ] , (34) \begin{aligned}
\mathop{\mathrm{E}}(A_{\vec{n}})&=\mathcal{Z}^{-1}\int\left(\prod_{\vec{i}}\frac{dA_{\vec{i}}}{\sqrt{2\pi\sigma_A^2}}\right)\exp\left[-\sum_j\frac{A_j^2}{2\sigma_A^2}\right]A_{\vec{n}}\\
&\quad\times\int\left(\prod_a\frac{dk_a}{2\pi}\right)\exp\left[i\sum_b k_b\left(y_b-\sum_{\ell}A_\ell T_\ell(\vec{\theta}_b)\right)-\frac{\sigma_A^2}{2}\sum_{ab}k_a\Delta_{ab}k_b\right],
\\
\operatorname{Cov}(A_{\vec{m}},A_{\vec{n}})&=\mathcal{Z}^{-1}\int\left(\prod_{\vec{i}}\frac{dA_{\vec{i}}}{\sqrt{2\pi\sigma_A^2}}\right)\exp\left[-\sum_j\frac{A_j^2}{2\sigma_A^2}-\frac{\sigma_A^2}{2}\sum_{ab}k_a\Delta_{ab}k_b\right]\\
&\quad\times A_{\vec{m}}A_{\vec{n}}\int\left(\prod_a\frac{dk_a}{2\pi}\right)\exp\left[i\sum_b k_b\left(y_b-\sum_{\ell}A_\ell T_\ell(\vec{\theta}_b)\right)\right], \tag*{(34)}
\end{aligned} E ( A n ) Cov ( A m , A n ) = Z − 1 ∫ i ∏ 2 π σ A 2 d A i exp [ − j ∑ 2 σ A 2 A j 2 ] A n × ∫ ( a ∏ 2 π d k a ) exp [ i b ∑ k b ( y b − ℓ ∑ A ℓ T ℓ ( θ b ) ) − 2 σ A 2 ab ∑ k a Δ ab k b ] , = Z − 1 ∫ i ∏ 2 π σ A 2 d A i exp [ − j ∑ 2 σ A 2 A j 2 − 2 σ A 2 ab ∑ k a Δ ab k b ] × A m A n ∫ ( a ∏ 2 π d k a ) exp [ i b ∑ k b ( y b − ℓ ∑ A ℓ T ℓ ( θ b ) ) ] , (34)
In this subsection, the means, variances and covariances refer to probabilities for assumed values of the hyper-parameters, σ A \sigma_A σ A and Λ \Lambda Λ .
After completing the square, the covariance can be written as
Cov ( A m ⃗ , A n ⃗ ) = Z − 1 ∫ ( ∏ a d k a 2 π ) e i ∑ b k b y b − σ A 2 2 ∑ a b k a Δ a b k b × ∫ ( ∏ i ⃗ d A i ⃗ ) A m ⃗ A n ⃗ exp [ − 1 2 σ A 2 ( A i ⃗ + i σ A 2 ∑ b T i ⃗ ( θ ⃗ b ) k b ) 2 − σ A 2 2 ( ∑ b , ℓ ⃗ T ℓ ⃗ ( θ ⃗ b ) k b ) 2 ] = Z − 1 ( ∫ ∏ a d k a 2 π ) exp [ − σ A 2 2 ( T m ⃗ ( θ ⃗ b ) k b ) 2 + i k b y b − σ A 2 2 k a Δ a b k b ] × ( ∫ ∏ i ⃗ d A i ⃗ ) ( A n ⃗ − i σ A 2 ∑ b T k ( θ ⃗ b ) k b ) ( A m ⃗ − i σ A 2 T m ⃗ ( θ ⃗ c ) k c ) exp [ − ∑ ℓ ⃗ A ℓ ⃗ 2 2 σ A 2 ] . (35) \begin{aligned}
\operatorname{Cov}(A_{\vec m},A_{\vec n})={}&Z^{-1}\int\left(\prod_a\frac{dk_a}{2\pi}\right)e^{i\sum_b k_by_b-\frac{\sigma_A^2}{2}\sum_{ab}k_a\Delta_{ab}k_b}\\
&\times\int\left(\prod_{\vec i}dA_{\vec i}\right)A_{\vec m}A_{\vec n}\exp\left[-\frac{1}{2\sigma_A^2}\left(A_{\vec i}+i\sigma_A^2\sum_bT_{\vec i}(\vec\theta_b)k_b\right)^2-\frac{\sigma_A^2}{2}\left(\sum_{b,\vec\ell}T_{\vec\ell}(\vec\theta_b)k_b\right)^2\right]\\
={}&Z^{-1}\left(\int\prod_a\frac{dk_a}{2\pi}\right)\exp\left[-\frac{\sigma_A^2}{2}\left(T_{\vec m}(\vec\theta_b)k_b\right)^2+ik_by_b-\frac{\sigma_A^2}{2}k_a\Delta_{ab}k_b\right]\\
&\times\left(\int\prod_{\vec i}dA_{\vec i}\right)(A_{\vec n}-i\sigma_A^2\sum_bT_k(\vec\theta_b)k_b)(A_{\vec m}-i\sigma_A^2T_{\vec m}(\vec\theta_c)k_c)\exp\left[-\sum_{\vec\ell}\frac{A_{\vec\ell}^2}{2\sigma_A^2}\right]. \tag*{(35)}
\end{aligned} Cov ( A m , A n ) = = Z − 1 ∫ ( a ∏ 2 π d k a ) e i ∑ b k b y b − 2 σ A 2 ∑ ab k a Δ ab k b × ∫ i ∏ d A i A m A n exp − 2 σ A 2 1 ( A i + i σ A 2 b ∑ T i ( θ b ) k b ) 2 − 2 σ A 2 b , ℓ ∑ T ℓ ( θ b ) k b 2 Z − 1 ( ∫ a ∏ 2 π d k a ) exp [ − 2 σ A 2 ( T m ( θ b ) k b ) 2 + i k b y b − 2 σ A 2 k a Δ ab k b ] × ∫ i ∏ d A i ( A n − i σ A 2 b ∑ T k ( θ b ) k b ) ( A m − i σ A 2 T m ( θ c ) k c ) exp − ℓ ∑ 2 σ A 2 A ℓ 2 . (35)
Similarly,
E ( A m ⃗ ) = − Z − 1 ∫ ( ∏ a d k a 2 π ) exp [ − ∑ b σ A 2 2 ( T m ⃗ ( θ ⃗ b ) k b ) 2 + i ∑ b k b y b − σ A 2 2 ∑ a b k a Δ a b k b ] × ∫ ( ∏ i ⃗ d A i ⃗ 2 π σ A 2 ) i σ A 2 ( ∑ c ⃗ T m ⃗ ( θ ⃗ c ) k c ) exp [ − ∑ ℓ ⃗ A ℓ ⃗ 2 2 σ A 2 ] . (36) \begin{aligned}
E(A_{\vec m})={}&-Z^{-1}\int\left(\prod_a\frac{dk_a}{2\pi}\right)\exp\left[-\sum_b\frac{\sigma_A^2}{2}\left(T_{\vec m}(\vec\theta_b)k_b\right)^2+i\sum_bk_by_b-\frac{\sigma_A^2}{2}\sum_{ab}k_a\Delta_{ab}k_b\right]\\
&\times\int\left(\prod_{\vec i}\frac{dA_{\vec i}}{\sqrt{2\pi\sigma_A^2}}\right)i\sigma_A^2\left(\sum_{\vec c}T_{\vec m}(\vec\theta_c)k_c\right)\exp\left[-\sum_{\vec\ell}\frac{A_{\vec\ell}^2}{2\sigma_A^2}\right]. \tag*{(36)}
\end{aligned} E ( A m ) = − Z − 1 ∫ ( a ∏ 2 π d k a ) exp [ − b ∑ 2 σ A 2 ( T m ( θ b ) k b ) 2 + i b ∑ k b y b − 2 σ A 2 ab ∑ k a Δ ab k b ] × ∫ i ∏ 2 π σ A 2 d A i i σ A 2 ( c ∑ T m ( θ c ) k c ) exp − ℓ ∑ 2 σ A 2 A ℓ 2 . (36)
It is convenient to define the function,
C 0 ( θ ⃗ 1 , θ ⃗ 2 ) ≡ ∑ i ⃗ T i ⃗ ( θ ⃗ 1 ) T i ⃗ ( θ ⃗ 2 ) = Cov 0 ( Y ( θ ⃗ 1 ) , Y ( θ ⃗ 2 ) ) σ A 2 . (37) \begin{aligned}
C_0(\vec\theta_1,\vec\theta_2)&\equiv\sum_{\vec i}T_{\vec i}(\vec\theta_1)T_{\vec i}(\vec\theta_2)\\
&=\frac{\operatorname{Cov}_0(Y(\vec\theta_1),Y(\vec\theta_2))}{\sigma_A^2}. \tag*{(37)}
\end{aligned} C 0 ( θ 1 , θ 2 ) ≡ i ∑ T i ( θ 1 ) T i ( θ 2 ) = σ A 2 Cov 0 ( Y ( θ 1 ) , Y ( θ 2 )) . (37)
The subscript zero, denotes that this is the average in the absence of training.
It is convenient to define the N t r a i n × N t r a i n N_{\rm train}\times N_{\rm train} N train × N train matrix,
B a b ≡ C 0 ( θ ⃗ a , θ ⃗ b ) + Δ a b , (38) B_{ab}\equiv C_0(\vec\theta_a,\vec\theta_b)+\Delta_{ab}, \tag*{(38)} B ab ≡ C 0 ( θ a , θ b ) + Δ ab , (38)
where θ ⃗ a \vec\theta_a θ a are the locations of the training points. Performing the integrals for Z ( σ A 2 , Λ ) Z(\sigma_A^2,\Lambda) Z ( σ A 2 , Λ ) in Eq. (32) ,
Z ( σ A 2 , Λ ) = σ A − N t r a i n ( det B ) − 1 / 2 exp [ − 1 2 σ A 2 ∑ a b y a B a b − 1 y b ] . (39) Z(\sigma_A^2,\Lambda)=\sigma_A^{-N_{\rm train}}(\det B)^{-1/2}\exp\left[-\frac{1}{2\sigma_A^2}\sum_{ab}y_aB_{ab}^{-1}y_b\right]. \tag*{(39)} Z ( σ A 2 , Λ ) = σ A − N train ( det B ) − 1/2 exp [ − 2 σ A 2 1 ab ∑ y a B ab − 1 y b ] . (39)
One can then find the moments of the integral over k k k ,
E ( k a ) = − i ∂ y a ln ( Z ) = i σ A 2 ∑ b B a b − 1 y b , Cov ( k a , k b ) = − ∂ y a ∂ y b ln ( Z ) = 1 σ A 2 B a b − 1 . (40) \begin{aligned}
E(k_a)&=-i\partial_{y_a}\ln(Z)=\frac{i}{\sigma_A^2}\sum_bB_{ab}^{-1}y_b,\\
\operatorname{Cov}(k_a,k_b)&=-\partial_{y_a}\partial_{y_b}\ln(Z)=\frac{1}{\sigma_A^2}B_{ab}^{-1}. \tag*{(40)}
\end{aligned} E ( k a ) Cov ( k a , k b ) = − i ∂ y a ln ( Z ) = σ A 2 i b ∑ B ab − 1 y b , = − ∂ y a ∂ y b ln ( Z ) = σ A 2 1 B ab − 1 . (40)
The averaged coefficients are then:
E ( A n ⃗ ) = ∑ a b T n ⃗ ( θ ⃗ a ) B a b − 1 y b , (41) E(A_{\vec n})=\sum_{ab}T_{\vec n}(\vec\theta_a)B_{ab}^{-1}y_b, \tag*{(41)} E ( A n ) = ab ∑ T n ( θ a ) B ab − 1 y b , (41)
Cov ( A m ⃗ , A n ⃗ ) = δ m ⃗ , n ⃗ σ A 2 − σ A 2 ∑ a b T m ⃗ ( θ ⃗ a ) B a b − 1 T n ⃗ ( θ ⃗ b ) . \operatorname{Cov}(A_{\vec m},A_{\vec n})=\delta_{\vec m,\vec n}\sigma_A^2-\sigma_A^2\sum_{ab}T_{\vec m}(\vec\theta_a)B_{ab}^{-1}T_{\vec n}(\vec\theta_b). Cov ( A m , A n ) = δ m , n σ A 2 − σ A 2 ab ∑ T m ( θ a ) B ab − 1 T n ( θ b ) .
The observables and their fluctuations due to the fluctuation in the coefficient are then:
E ( Y ( θ ⃗ ) ) = ∑ n ⃗ E ( A n ⃗ ) T n ⃗ ( θ ⃗ ) , = ∑ a b y a B a b − 1 C 0 ( θ ⃗ b , θ ⃗ a ) , V a r ( Y ( θ ⃗ ) ) = ∑ m ⃗ n ⃗ C o v ( A m ⃗ , A n ⃗ ) T m ⃗ ( θ ⃗ ) T n ⃗ ( θ ⃗ ) = σ A 2 [ C 0 ( θ ⃗ , θ ⃗ ) − ∑ a b C 0 ( θ ⃗ , θ ⃗ a ) B a b − 1 C 0 ( θ ⃗ b , θ ⃗ ) ] . (42) \begin{aligned}
\mathrm{E}(Y(\vec{\theta})) &= \sum_{\vec{n}} \mathrm{E}(A_{\vec{n}})T_{\vec{n}}(\vec{\theta}), \\
&= \sum_{ab} y_a B^{-1}_{ab} C_0(\vec{\theta}_b,\vec{\theta}_a), \\
\mathrm{Var}(Y(\vec{\theta})) &= \sum_{\vec{m}\vec{n}} \mathrm{Cov}(A_{\vec{m}},A_{\vec{n}})T_{\vec{m}}(\vec{\theta})T_{\vec{n}}(\vec{\theta}) \\
&= \sigma_A^2\left[C_0(\vec{\theta},\vec{\theta})-\sum_{ab}C_0(\vec{\theta},\vec{\theta}_a)B^{-1}_{ab}C_0(\vec{\theta}_b,\vec{\theta})\right]. \tag*{(42)}
\end{aligned} E ( Y ( θ )) Var ( Y ( θ )) = n ∑ E ( A n ) T n ( θ ) , = ab ∑ y a B ab − 1 C 0 ( θ b , θ a ) , = m n ∑ Cov ( A m , A n ) T m ( θ ) T n ( θ ) = σ A 2 [ C 0 ( θ , θ ) − ab ∑ C 0 ( θ , θ a ) B ab − 1 C 0 ( θ b , θ ) ] . (42)
The fact that the prediction for Y ( θ ⃗ ) Y(\vec{\theta}) Y ( θ ) and its uncertainty can be expressed so simply in terms of the correlation kernel C 0 C_0 C 0 without involving the coefficients is the well known kernel trick [44 ] . This enables one to build the emulator without ever bothering to calculate the individual expansion coefficients A n ⃗ A_{\vec{n}} A n .
After defining
χ a ≡ ∑ b B a b − 1 y b , (43) \chi_a \equiv\sum_b B^{-1}_{ab}y_b, \tag*{(43)} χ a ≡ b ∑ B ab − 1 y b , (43)
the emulated value and emulator uncertainty at some point θ ⃗ \vec{\theta} θ are
Y ( θ ⃗ ) = ∑ a χ a C 0 ( θ ⃗ a , θ ⃗ ) , (44) Y(\vec{\theta})=\sum_a\chi_a C_0(\vec{\theta}_a,\vec{\theta}), \tag*{(44)} Y ( θ ) = a ∑ χ a C 0 ( θ a , θ ) , (44)
V a r ( Y ( θ ⃗ ) ) = σ Y 2 ( θ ⃗ ) = σ A 2 C 0 ( θ ⃗ , θ ⃗ ) − σ A 2 ∑ a b C 0 ( θ ⃗ , θ ⃗ a ) B a b − 1 C 0 ( θ ⃗ b , θ ⃗ ) . \mathrm{Var}(Y(\vec{\theta}))=\sigma_Y^2(\vec{\theta})=\sigma_A^2 C_0(\vec{\theta},\vec{\theta})-\sigma_A^2\sum_{ab}C_0(\vec{\theta},\vec{\theta}_a)B^{-1}_{ab}C_0(\vec{\theta}_b,\vec{\theta}). Var ( Y ( θ )) = σ Y 2 ( θ ) = σ A 2 C 0 ( θ , θ ) − σ A 2 ab ∑ C 0 ( θ , θ a ) B ab − 1 C 0 ( θ b , θ ) .
The optimum value E ( Y ( θ ⃗ ) ) \mathrm{E}(Y(\vec{\theta})) E ( Y ( θ )) depends linearly on the training values, y a = 1 , N t r a i n y_{a=1,N_{\mathrm{train}}} y a = 1 , N train . The emulator’s uncertainty depends on θ ⃗ \vec{\theta} θ and vanishes at the training points if Δ a b = 0 \Delta_{ab}=0 Δ ab = 0 . It is linear in σ A \sigma_A σ A and has a more complicated dependence on Λ \Lambda Λ . It does not depend on the model values at those training points, y a y_a y a , although after tuning the choice of σ A \sigma_A σ A will depend on the training values. Not surprisingly, this is the exactly the same expression used for Gaussian process emulators, with the requirement that the form for the GP correlation kernel should be Gaussian.
Even though the procedure outlined above allows one to solve for all the coefficients, A n ⃗ A_{\vec{n}} A n , solving for the individual coefficients is not necessary because both the emulator and its uncertainty can be expressed, as shown in Eq. (44) , in terms of the bare correlation kernel, C 0 ( θ ⃗ 1 , θ ⃗ 2 ) C_0(\vec{\theta}_1,\vec{\theta}_2) C 0 ( θ 1 , θ 2 ) . The simple form is the same for some GP emulators, but rather than being arbitrarily assigned as is the case for the GP paradigm, in this perspective the form is motivated by the three simple assumptions: analyticity, independent Gaussian fluctuation of the polynomial terms, and invariance to translations and rotations. The fact that the emulator can be expressed simply in terms of the kernel, C 0 ( θ ⃗ 1 , θ ⃗ 2 ) C_0(\vec{\theta}_1,\vec{\theta}_2) C 0 ( θ 1 , θ 2 ) , is not unique to the polynomial expansion used here. Any form where Y ( θ ⃗ ) Y(\vec{\theta}) Y ( θ ) is expressed as a sum of terms, where each term is linear in some coefficient, A i A_i A i , that has some Gaussian distribution, can be expressed in the form of Eq. (44) . This is known as the kernel trick [44 ] and has been exploited in many previous studies involving polynomial expansions. For example, if one had chosen the form f K ( ∣ θ ⃗ ∣ 2 ) = 1 f_K(|\vec{\theta}|^2)=1 f K ( ∣ θ ∣ 2 ) = 1 , rather than being a Gaussian, the fluctuation would then not be independent of the norm of θ ⃗ \vec{\theta} θ , and the correlation would no longer depend only on ∣ θ ⃗ 1 − θ ⃗ 2 ∣ 2 |\vec{\theta}_1-\vec{\theta}_2|^2 ∣ θ 1 − θ 2 ∣ 2 , but one could still optimize the expansion. The resulting correlation kernel would change to C 0 ( θ ⃗ 1 , θ ⃗ 2 ) = e θ ⃗ 1 ⋅ θ ⃗ 2 / Λ 2 C_0(\vec{\theta}_1,\vec{\theta}_2)=e^{\vec{\theta}_1\cdot\vec{\theta}_2/\Lambda^2} C 0 ( θ 1 , θ 2 ) = e θ 1 ⋅ θ 2 / Λ 2 , but the form of Eq. (44) would not change.
Despite the fact that the emulator’s behavior is completely determined by the kernel, knowing the form for the polynomial expansion that satisfies the three aforementioned constraints is nonetheless useful for testing. In Sec. IV random test functions will be created using this form. Each function is characterized by σ A \sigma_A σ A , Λ \Lambda Λ and the set of random Gaussian coefficients. If one trains an emulator on such a test function, the emulator’s value of σ A \sigma_A σ A and Λ \Lambda Λ should be close to those used to generate the function. Such tests are presented in Sec. IV.
The fluctuation described in Eq. (44) were only those due to the fluctuation of the coefficients. This fluctuation was calculated ignoring the fact that the hyper-parameters σ A \sigma_A σ A and Λ \Lambda Λ are not precisely determined. An additional contribution to V a r ( Y ( θ ⃗ ) ) \mathrm{Var}(Y(\vec{\theta})) Var ( Y ( θ )) arises if one accounts for the variance in Λ \Lambda Λ . This is the subject of the next subsection.
Finding the Optimized Hyper-Parameters
This approach to emulation has two hyper parameters, the variance of the coefficients, σ A \sigma_A σ A , and the convergence parameter Λ \Lambda Λ . If one has sufficient training data, one can find their optimum values. The next subsection considers choosing σ A \sigma_A σ A for some fixed Λ \Lambda Λ and the following subsection considers choosing Λ \Lambda Λ while fixing σ A \sigma_A σ A .
Both σ A \sigma_A σ A and Λ \Lambda Λ are unknown, and the probability distribution for them is determined by the function Z ( σ A 2 , Λ ) Z(\sigma_A^2,\Lambda) Z ( σ A 2 , Λ ) , which can be considered as the conditional probability of observing the training data given σ A \sigma_A σ A and Λ \Lambda Λ . Restating Eq. (39) ,
Z ( σ A 2 , Λ ) = ∫ ∏ i d A i P ( A ⃗ , σ A 2 , Λ ) = σ A − N t r a i n ( det B ) − 1 / 2 exp [ − 1 2 σ A 2 y a B a b − 1 y b ] . (45) \mathcal{Z}(\sigma_A^2,\Lambda)=\int\prod_i dA_i P(\vec{A},\sigma_A^2,\Lambda)
= \sigma_A^{-N_{\mathrm{train}}}(\det B)^{-1/2}\exp\left[-\frac{1}{2\sigma_A^2}y_aB^{-1}_{ab}y_b\right]. \tag*{(45)} Z ( σ A 2 , Λ ) = ∫ i ∏ d A i P ( A , σ A 2 , Λ ) = σ A − N train ( det B ) − 1/2 exp [ − 2 σ A 2 1 y a B ab − 1 y b ] . (45)
By Bayes theorem the probability of σ A \sigma_A σ A and Λ \Lambda Λ is proportional to the product of Z Z Z and Q ( σ A 2 , Λ ) Q(\sigma_A^2,\Lambda) Q ( σ A 2 , Λ ) , the prior probability of σ A \sigma_A σ A and Λ \Lambda Λ . I.e. Q ( σ A 2 , Λ ) Q(\sigma_A^2,\Lambda) Q ( σ A 2 , Λ ) is the probability for σ A \sigma_A σ A and Λ \Lambda Λ in the absence of training data. The most appealing method to chose σ A \sigma_A σ A and Λ \Lambda Λ , would be to choose their average over the weight Z Q ZQ Z Q ,
σ ˉ A = 1 N ∫ d σ A d Λ σ A Z ( σ A 2 , Λ ) Q ( σ A 2 , Λ ) , \bar{\sigma}_A=\frac{1}{\mathcal{N}}\int d\sigma_A d\Lambda\ \sigma_A Z(\sigma_A^2,\Lambda)Q(\sigma_A^2,\Lambda), σ ˉ A = N 1 ∫ d σ A d Λ σ A Z ( σ A 2 , Λ ) Q ( σ A 2 , Λ ) ,
Λ ˉ = 1 N ∫ d σ A d Λ Λ Z ( σ A 2 , Λ ) Q ( σ A 2 , Λ ) , \bar{\Lambda}=\frac{1}{\mathcal{N}}\int d\sigma_A d\Lambda\ \Lambda Z(\sigma_A^2,\Lambda)Q(\sigma_A^2,\Lambda), Λ ˉ = N 1 ∫ d σ A d Λ Λ Z ( σ A 2 , Λ ) Q ( σ A 2 , Λ ) ,
N = ∫ d σ A d Λ Z ( σ A 2 , Λ ) Q ( σ A 2 , Λ ) . (46) \mathcal{N}=\int d\sigma_A d\Lambda Z(\sigma_A^2,\Lambda)Q(\sigma_A^2,\Lambda). \tag*{(46)} N = ∫ d σ A d Λ Z ( σ A 2 , Λ ) Q ( σ A 2 , Λ ) . (46)
Similarly, one can find expressions for the uncertainties of the hyper parameters by calculating higher-moment integrals. We define the covariance matrix W α , β W_{\alpha,\beta} W α , β ,
W σ A σ A = Var ( σ A ) , W σ A Λ = W Λ σ A = Cov ( σ A , Λ ) , W Λ Λ = Var ( Λ ) . (47) \begin{aligned}
W_{\sigma_A\sigma_A}&=\operatorname{Var}(\sigma_A),\\
W_{\sigma_A\Lambda}&=W_{\Lambda\sigma_A}=\operatorname{Cov}(\sigma_A,\Lambda),\\
W_{\Lambda\Lambda}&=\operatorname{Var}(\Lambda). \tag*{(47)}
\end{aligned} W σ A σ A W σ A Λ W ΛΛ = Var ( σ A ) , = W Λ σ A = Cov ( σ A , Λ ) , = Var ( Λ ) . (47)
Here, the averages, variances and covariances take into account the weights Z ( σ A 2 , Λ ) Q ( σ A 2 , Λ ) Z(\sigma_A^2,\Lambda)Q(\sigma_A^2,\Lambda) Z ( σ A 2 , Λ ) Q ( σ A 2 , Λ ) . Unfortunately, the functions Z ( σ A 2 , Λ ) Z(\sigma_A^2,\Lambda) Z ( σ A 2 , Λ ) approaches a constant as Λ → ∞ \Lambda\to\infty Λ → ∞ . This means that all moments, even the normalization, are strongly affected by the range of the prior, Q ( σ A 2 , Λ ) Q(\sigma_A^2,\Lambda) Q ( σ A 2 , Λ ) . In fact, if the prior has some range Λ max \Lambda_{\max} Λ m a x , the n n n th moment would scale as Λ max n \Lambda_{\max}^n Λ m a x n for large Λ max \Lambda_{\max} Λ m a x . Thus, this weight gives one pause in using moments of the distribution to represent best values or uncertainties.
As an alternative to integrating the expressions above over Λ \Lambda Λ in order to find the optimized values of σ A \sigma_A σ A and Λ \Lambda Λ , one can solve for the values of σ A \sigma_A σ A and Λ \Lambda Λ that maximize Z ( σ A 2 , Λ ) Z(\sigma_A^2,\Lambda) Z ( σ A 2 , Λ ) as alternatives to Λ ˉ \bar{\Lambda} Λ ˉ and σ ˉ A \bar{\sigma}_A σ ˉ A . Further, one can assign the uncertainty by calculating the curvatures, i.e. the second derivatives of Z ( σ A 2 , Λ ) Z(\sigma_A^2,\Lambda) Z ( σ A 2 , Λ ) evaluated at the values of σ A \sigma_A σ A and Λ \Lambda Λ for which the first derivatives vanish.
W α β = − ∂ 2 ln ( Z ) ∂ α ∂ β . (48) W_{\alpha\beta}=-\frac{\partial^2\ln(Z)}{\partial\alpha\partial\beta}. \tag*{(48)} W α β = − ∂ α ∂ β ∂ 2 ln ( Z ) . (48)
Essentially, this corresponds to assuming that ln ( Z ) \ln(Z) ln ( Z ) is quadratic in the parameters. Normally, choosing the point of the maximum and its curvature is a reasonable assumption whenever Z Z Z is a sharply peaked function. If Z Z Z were a Gaussian function of the parameters, which it is not, one could assume an arbitrarily wide prior, and the averages of parameters and their covariances would be equivalent to the position of the maximum and the curvature at the maximum.
Another advantage of using the alternative expressions for σ A \sigma_A σ A and Λ \Lambda Λ and their uncertainties are that the quantities are calculated quickly. A Newtons method algorithm required ≲ 10 \lesssim10 ≲ 10 evaluations of Z ( σ A 2 , Λ ) Z(\sigma_A^2,\Lambda) Z ( σ A 2 , Λ ) and its derivatives. Both Z Z Z and its derivatives are easily calculated and finding these points at which Z Z Z is maximized is robust. The calculations shown in this paper apply this alternative. As a test, in Sec. IV functions y ( θ ⃗ ) y(\vec{\theta}) y ( θ ) are built with polynomial coefficients chosen consistent with the form assumed by the emulator with specific values of σ A \sigma_A σ A and Λ \Lambda Λ . Emulators, Y ( θ ⃗ ) Y(\vec{\theta}) Y ( θ ) are then constructed with σ A \sigma_A σ A and Λ \Lambda Λ chosen according to the method described above. The chosen values of σ A \sigma_A σ A and Λ \Lambda Λ well match those used to construct y ( θ ⃗ ) y(\vec{\theta}) y ( θ ) for larger numbers training points.
Choosing the Characteristic Coefficient Width
First, we consider fixing the convergence parameter Λ \Lambda Λ and finding the optimum value of σ A \sigma_A σ A . This involves maximizing the probability Z ( σ A 2 , Λ ) Z(\sigma_A^2,\Lambda) Z ( σ A 2 , Λ ) , defined in Eq. (32) for a fixed Λ \Lambda Λ . The goal here is to find the value of σ A \sigma_A σ A that maximizes Z \mathcal{Z} Z . For the case where Λ = 0 \Lambda= 0 Λ = 0 or if Λ \Lambda Λ is independent of σ A \sigma_A σ A , one can find σ A \sigma_A σ A rather easily because B a b B_{ab} B ab does not depend on σ A \sigma_A σ A . In that case, setting the derivative w.r.t. σ A 2 \sigma_A^2 σ A 2 equal to zero,
0 = N t r a i n σ A 2 Z − Z σ A 4 y a B a b − 1 y b , (49) 0 = \frac{N_{\mathrm{train}}}{\sigma_A^2}\mathcal{Z} - \frac{\mathcal{Z}}{\sigma_A^4} y_a B_{ab}^{-1} y_b, \tag*{(49)} 0 = σ A 2 N train Z − σ A 4 Z y a B ab − 1 y b , (49)
σ A 2 = 1 N t r a i n y a B a b − 1 y b . (50) \sigma_A^2 = \frac{1}{N_{\mathrm{train}}} y_a B_{ab}^{-1} y_b. \tag*{(50)} σ A 2 = N train 1 y a B ab − 1 y b . (50)
Choosing the Convergence Parameter
Eq. Eq. (50) provides the means for choosing σ A \sigma_A σ A for a given value Λ \Lambda Λ . One can then plot Z ( σ A 2 , Λ ) Z(\sigma_A^2,\Lambda) Z ( σ A 2 , Λ ) , as given in Eq. (45) , as a function of Λ \Lambda Λ and choose the value of Λ \Lambda Λ that maximizes Z Z Z . For some value of Λ \Lambda Λ the expressions for B a b B_{ab} B ab and σ A \sigma_A σ A are easily found. One can also find analytic expressions for the derivatives of B B B with respect to Λ \Lambda Λ . This allows one to use a Newtons method to find the optimized values of σ A \sigma_A σ A and Λ \Lambda Λ with a few interactions. The uncertainties defined by Eq. (48) are then found analytically.
One should be careful in using smaller values of the convergence parameter, especially when there are a large number of parameters. The contribution to the variance of the emulator for a specific term in the polynomial expansion scales as 1 / Λ 2 n 1/\Lambda^{2n} 1/ Λ 2 n , if n n n is the order of the term. The number of terms of a given order scales as N p a r s n N_{\mathrm{pars}}^n N pars n . Thus, the importance of a given order relative to the previous order scales as N p a r s / Λ 2 N_{\mathrm{pars}}/\Lambda^2 N pars / Λ 2 . If Λ ≲ N p a r s 1 / 2 \Lambda\lesssim N_{\mathrm{pars}}^{1/2} Λ ≲ N pars 1/2 , the emulator uncertainty does not converge well for increasing order. For example, for four model parameters, one might find that any model characterized by Λ ≳ 2 \Lambda\gtrsim2 Λ ≳ 2 can be reasonably well emulated by a Taylor expansion, a model with 16 parameters might need Λ ≳ 4 \Lambda\gtrsim4 Λ ≳ 4 to converge similarly as function of the order K K K . The challenge of handling emulation with large numbers of parameters is not confined to emulators based on polynomial expansions, but can be a problem for all emulators. A discussion of these issues is given in Sec. III.
Contribution to the Emulator Uncertainty from the Uncertainty of the Hyper Parameters
The emulator uncertainty σ Y ( θ ⃗ ) \sigma_Y(\vec{\theta}) σ Y ( θ ) has contributions from the variance of the expansion coefficients for given values of σ A \sigma_A σ A and Λ \Lambda Λ , and also contributions from the fact that σ A \sigma_A σ A and Λ \Lambda Λ are not known. Here, we present a general discussion of how one might approximate the latter contribution. We consider a set of hyper parameters, h ⃗ \vec{h} h , which in this case would refer to σ A \sigma_A σ A and Λ \Lambda Λ .
In this derivation, E ( O ∣ h ⃗ ) \mathrm{E}(O|\vec{h}) E ( O ∣ h ) refers to the the average of O O O over all the possible expansion coefficients given the training data for a specific set of hyper parameters h ⃗ \vec{h} h . The notation E ( O ) \mathrm{E}(O) E ( O ) refers to the average once one has also averaged over h ⃗ \vec{h} h . Here, h ⃗ 0 \vec{h}_0 h 0 is the hyper parameter vector at which the probability P ( h ⃗ ) P(\vec{h}) P ( h ) is maximized, Y ˉ ( h ⃗ ) = E ( Y ∣ h ⃗ ) \bar{Y}(\vec{h}) = \mathrm{E}(Y|\vec{h}) Y ˉ ( h ) = E ( Y ∣ h ) , and Y ˉ 0 = E ( Y ∣ h ⃗ = h ⃗ 0 ) \bar{Y}_0 = \mathrm{E}(Y|\vec{h}=\vec{h}_0) Y ˉ 0 = E ( Y ∣ h = h 0 ) and Var 0 ( Y ) = Var ( Y ∣ h ⃗ = h ⃗ 0 ) \operatorname{Var}_0(Y) = \operatorname{Var}(Y|\vec{h}=\vec{h}_0) Var 0 ( Y ) = Var ( Y ∣ h = h 0 ) . The goal is to estimate the variance of the emulated value at some point in parameter space, θ ⃗ \vec{\theta} θ .
Var ( Y ) = ∫ d h ⃗ P ( h ⃗ ) Var ( Y ∣ h ⃗ ) = ∫ d h ⃗ P ( h ⃗ ) E ( [ ( Y − Y ˉ ( h ⃗ ) ) + ( Y ˉ ( h ⃗ ) − Y ˉ 0 ) ] 2 | h ⃗ ) = ∫ d h ⃗ P ( h ⃗ ) { E ( ( Y − Y ˉ ( h ⃗ ) ) 2 | h ⃗ ) + ( Y ˉ ( h ⃗ ) − Y ˉ 0 ) 2 } = ∫ d h ⃗ P ( h ⃗ ) { Var ( Y ∣ h ⃗ = h ⃗ 0 ) + ( h i − h 0 i ) ( h j − h 0 j ) 2 ∂ 2 ∂ h i ∂ h j Cov ( Y ∣ h ⃗ ) + ( h i − h 0 i ) ( h j − h 0 j ) ( ∂ Y ˉ ( h ⃗ ) ∂ h i ) ( ∂ Y ˉ ( h ⃗ ) ∂ h j ) + O ( h − h 0 ) 3 } ≈ Var 0 ( Y ) + [ ∂ Y ˉ ( h ⃗ ) ∂ h i ∂ Y ˉ ( h ⃗ ) ∂ h j ] W i j , (51) \begin{aligned}
\operatorname{Var}(Y) &= \int d\vec{h}\,P(\vec{h})\operatorname{Var}(Y|\vec{h}) \\
&= \int d\vec{h}\,P(\vec{h})\mathrm{E}\left(\left[(Y-\bar{Y}(\vec{h}))+(\bar{Y}(\vec{h})-\bar{Y}_0)\right]^2\middle|\vec{h}\right) \\
&= \int d\vec{h}\,P(\vec{h})\left\{\mathrm{E}\left((Y-\bar{Y}(\vec{h}))^2\middle|\vec{h}\right)+(\bar{Y}(\vec{h})-\bar{Y}_0)^2\right\} \\
&= \int d\vec{h}\,P(\vec{h})\left\{\operatorname{Var}(Y|\vec{h}=\vec{h}_0)+\frac{(h_i-h_{0i})(h_j-h_{0j})}{2}\frac{\partial^2}{\partial h_i\partial h_j}\operatorname{Cov}(Y|\vec{h}) \right.\\
&\qquad\left. +(h_i-h_{0i})(h_j-h_{0j})\left(\frac{\partial\bar{Y}(\vec{h})}{\partial h_i}\right)\left(\frac{\partial\bar{Y}(\vec{h})}{\partial h_j}\right)+\mathcal{O}(h-h_0)^3\right\} \\
&\approx\operatorname{Var}_0(Y)+\left[\frac{\partial\bar{Y}(\vec{h})}{\partial h_i}\frac{\partial\bar{Y}(\vec{h})}{\partial h_j}\right]W_{ij}, \tag*{(51)}
\end{aligned} Var ( Y ) = ∫ d h P ( h ) Var ( Y ∣ h ) = ∫ d h P ( h ) E ( [ ( Y − Y ˉ ( h )) + ( Y ˉ ( h ) − Y ˉ 0 ) ] 2 h ) = ∫ d h P ( h ) { E ( ( Y − Y ˉ ( h ) ) 2 h ) + ( Y ˉ ( h ) − Y ˉ 0 ) 2 } = ∫ d h P ( h ) { Var ( Y ∣ h = h 0 ) + 2 ( h i − h 0 i ) ( h j − h 0 j ) ∂ h i ∂ h j ∂ 2 Cov ( Y ∣ h ) + ( h i − h 0 i ) ( h j − h 0 j ) ( ∂ h i ∂ Y ˉ ( h ) ) ( ∂ h j ∂ Y ˉ ( h ) ) + O ( h − h 0 ) 3 } ≈ Var 0 ( Y ) + [ ∂ h i ∂ Y ˉ ( h ) ∂ h j ∂ Y ˉ ( h ) ] W ij , (51)
W i j = ∫ d h ⃗ P ( h ⃗ ) ( h i − h 0 i ) ( h j − h 0 j ) . W_{ij}=\int d\vec{h}\,P(\vec{h})(h_i-h_{0i})(h_j-h_{0j}). W ij = ∫ d h P ( h ) ( h i − h 0 i ) ( h j − h 0 j ) .
The approximation of the last step is built on the assumption that sufficient training data has been provided so that
P ( h ⃗ ) P(\vec{h}) P ( h ) is sufficiently narrow about h ⃗ 0 \vec{h}_0 h 0 that higher order terms in h ⃗ − h ⃗ 0 \vec{h}-\vec{h}_0 h − h 0 can be ignored.
Var ( Y ∣ h ⃗ ) \operatorname{Var}(Y\mid\vec{h}) Var ( Y ∣ h ) is small so that terms that scale both as Var ( Y ∣ h ⃗ ) \operatorname{Var}(Y\mid\vec{h}) Var ( Y ∣ h ) and ( h ⃗ − h ⃗ 0 ) 2 (\vec{h}-\vec{h}_0)^2 ( h − h 0 ) 2 can be ignored.
One of the remaining terms is small because Cov ( Y ∣ h ⃗ ~ ) \operatorname{Cov}(Y\mid\tilde{\vec{h}}) Cov ( Y ∣ h ~ ) is already small and additional powers of h ⃗ ~ − h ⃗ 0 \tilde{\vec{h}}-\vec{h}_0 h ~ − h 0 make it increasingly smaller compared to the first term. The neglected terms should all be increasingly smaller relative to the included terms as the number of training points is increased.
Applying the approximation of Eq. (52) to the calculations of the uncertainty of the emulator,
σ Y 2 ( θ ⃗ ) ≈ σ A 2 [ C 0 ( θ ⃗ , θ ⃗ ) − C 0 ( θ ⃗ , θ ⃗ a ) B a b − 1 C 0 ( θ ⃗ b , θ ⃗ ) ] + W σ A σ A ( ∂ Y ∂ σ A ) 2 + 2 W σ A Λ ( ∂ Y ∂ σ A ) ( ∂ Y ∂ Λ ) + W Λ Λ ( ∂ Y ∂ Λ ) 2 . (52) \begin{aligned}
\sigma_Y^2(\vec{\theta}) &\approx\sigma_A^2\left[C_0(\vec{\theta},\vec{\theta})-C_0(\vec{\theta},\vec{\theta}_a)B_{ab}^{-1}C_0(\vec{\theta}_b,\vec{\theta})\right] \\
&\quad+W_{\sigma_A\sigma_A}\left(\frac{\partial Y}{\partial\sigma_A}\right)^2+2W_{\sigma_A\Lambda}\left(\frac{\partial Y}{\partial\sigma_A}\right)\left(\frac{\partial Y}{\partial\Lambda}\right)+W_{\Lambda\Lambda}\left(\frac{\partial Y}{\partial\Lambda}\right)^2. \tag*{(52)}
\end{aligned} σ Y 2 ( θ ) ≈ σ A 2 [ C 0 ( θ , θ ) − C 0 ( θ , θ a ) B ab − 1 C 0 ( θ b , θ ) ] + W σ A σ A ( ∂ σ A ∂ Y ) 2 + 2 W σ A Λ ( ∂ σ A ∂ Y ) ( ∂ Λ ∂ Y ) + W ΛΛ ( ∂ Λ ∂ Y ) 2 . (52)
The r.h.s. of Eq. (52) is evaluated with σ A \sigma_A σ A and Λ \Lambda Λ chosen to maximize Z ( σ A , Λ ) Z(\sigma_A,\Lambda) Z ( σ A , Λ ) , and the covariance of the hyper-parameters, W W W , is approximated from the curvature of Z ( σ A 2 , Λ ) Z(\sigma_A^2,\Lambda) Z ( σ A 2 , Λ ) , as described in Eq. (48) . The first term represents the emulator’s uncertainty due to the uncertainty of the coefficients A n ⃗ A_{\vec{n}} A n for fixed σ A \sigma_A σ A and Λ \Lambda Λ , and the terms proportional to W W W provide the uncertainty due to the fact that σ A \sigma_A σ A and Λ \Lambda Λ are themselves uncertain. In the limit of large numbers of training points the range of Λ \Lambda Λ is well constrained around the optimum value and this becomes an excellent approximation, but the approximation becomes dubious when Λ \Lambda Λ is poorly constrained. Often, the contributions to the uncertainty from the terms proportional to W α β W_{\alpha\beta} W α β are ignored. In our studies it was found these corrections were relatively small when one chooses a reasonable set of training-point locations. However, these terms were found to be critical for optimizing the choice of locations of training points, θ ⃗ a = 1 … N t r a i n \vec{\theta}_{a=1\ldots N_{\rm train}} θ a = 1 … N train . For example, if all the training points were placed very close to the origin, the terms were found to be significant, but for optimized placement the terms were very small. In contrast the strength of the other terms was observed to be only weakly dependent of the training-point placement. Thus, these additional terms played an outsized role when optimizing training-point placement.
Restating Eq. (44) , the emulator depends on the training values as,
Y ( θ ⃗ ) = C 0 ( θ ⃗ , θ ⃗ a ) B a b − 1 y b , (53) Y(\vec{\theta})=C_0(\vec{\theta},\vec{\theta}_a)B_{ab}^{-1}y_b, \tag*{(53)} Y ( θ ) = C 0 ( θ , θ a ) B ab − 1 y b , (53)
and is independent of σ A \sigma_A σ A because C 0 C_0 C 0 and B B B depend only on Λ \Lambda Λ . Thus, ∂ σ A Y ( θ ⃗ ) = 0 \partial_{\sigma_A}Y(\vec{\theta})=0 ∂ σ A Y ( θ ) = 0 , and one need only worry about W Λ Λ W_{\Lambda\Lambda} W ΛΛ and can ignore W σ A σ A W_{\sigma_A\sigma_A} W σ A σ A or W σ A , Λ W_{\sigma_A,\Lambda} W σ A , Λ , even though one needs to calculate all the terms in W − 1 W^{-1} W − 1 before finding W W W .
The probability Z Z Z depends on σ A \sigma_A σ A , Λ \Lambda Λ , the location of the training points and the training values.
ln ( Z ) = − N t r a i n ln ( σ A ) − 1 2 ln ( ∣ B ∣ ) − 1 2 σ A 2 y a B a b − 1 y b . (54) \ln(Z)=-N_{\rm train}\ln(\sigma_A)-\frac{1}{2}\ln(|B|)-\frac{1}{2\sigma_A^2}y_aB_{ab}^{-1}y_b. \tag*{(54)} ln ( Z ) = − N train ln ( σ A ) − 2 1 ln ( ∣ B ∣ ) − 2 σ A 2 1 y a B ab − 1 y b . (54)
Because B B B is independent of σ A \sigma_A σ A ,
W σ A σ A − 1 = − N t r a i n σ A 2 + 3 σ A 4 y a B a b − 1 y b , (55) W_{\sigma_A\sigma_A}^{-1}=-\frac{N_{\rm train}}{\sigma_A^2}+\frac{3}{\sigma_A^4}y_aB_{ab}^{-1}y_b, \tag*{(55)} W σ A σ A − 1 = − σ A 2 N train + σ A 4 3 y a B ab − 1 y b , (55)
W σ A Λ − 1 ≈ 1 σ A 3 y a ( d d Λ B − 1 ) a b y b , W_{\sigma_A\Lambda}^{-1}\approx\frac{1}{\sigma_A^3}y_a\left(\frac{d}{d\Lambda}B^{-1}\right)_{ab}y_b, W σ A Λ − 1 ≈ σ A 3 1 y a ( d Λ d B − 1 ) ab y b ,
W Λ Λ − 1 ≈ 1 2 d 2 d Λ 2 ln ( ∣ B ∣ ) − 1 2 σ A 2 y a ( d d Λ ) 2 B a b − 1 y b . W_{\Lambda\Lambda}^{-1}\approx\frac{1}{2}\frac{d^2}{d\Lambda^2}\ln(|B|)-\frac{1}{2\sigma_A^2}y_a\left(\frac{d}{d\Lambda}\right)^2B_{ab}^{-1}y_b. W ΛΛ − 1 ≈ 2 1 d Λ 2 d 2 ln ( ∣ B ∣ ) − 2 σ A 2 1 y a ( d Λ d ) 2 B ab − 1 y b .
These expressions assume that near the optimum values, Z ( σ A , Λ ) Z(\sigma_A,\Lambda) Z ( σ A , Λ ) can be treated as if it were a Gaussian, so that the widths are determined by the curvature of Z ( σ A , Λ ) Z(\sigma_A,\Lambda) Z ( σ A , Λ ) around the optimum. To take the derivatives w.r.t. B − 1 B^{-1} B − 1 , one can use the identity
d d Λ B a b − 1 = − B a c − 1 ( d d Λ B c d ) B d b − 1 . (56) \frac{d}{d\Lambda}B_{ab}^{-1}=-B_{ac}^{-1}\left(\frac{d}{d\Lambda}B_{cd}\right)B_{db}^{-1}. \tag*{(56)} d Λ d B ab − 1 = − B a c − 1 ( d Λ d B c d ) B d b − 1 . (56)
After defining
B a b ′ = Λ 3 d d Λ B a b = ( ∣ θ ⃗ a − θ ⃗ b ∣ 2 ) B a b , (57) B'_{ab}=\Lambda^3\frac{d}{d\Lambda}B_{ab}=(|\vec{\theta}_a-\vec{\theta}_b|^2)B_{ab}, \tag*{(57)} B ab ′ = Λ 3 d Λ d B ab = ( ∣ θ a − θ b ∣ 2 ) B ab , (57)
B a b ′ ′ = Λ 3 d d Λ B a b ′ = ( ∣ θ ⃗ a − θ ⃗ b ∣ 4 ) B a b . B''_{ab}=\Lambda^3\frac{d}{d\Lambda}B'_{ab}=(|\vec{\theta}_a-\vec{\theta}_b|^4)B_{ab}. B ab ′′ = Λ 3 d Λ d B ab ′ = ( ∣ θ a − θ b ∣ 4 ) B ab .
one finds
W σ A σ A − 1 = N t r a i n 1 σ A 2 − 3 σ A 4 y a B a b − 1 y b , W σ A Λ − 1 = − 1 Λ 3 σ A 2 y a ( B − 1 B ′ B − 1 ) a b y b , W Λ Λ − 1 = 1 2 d 2 d Λ 2 ln ( ∣ B ∣ ) + 1 2 σ A 2 Λ 6 y a ( 2 B − 1 B ′ B − 1 B ′ B − 1 − B − 1 B ′ ′ B − 1 ) a b y b + 3 2 σ A 2 Λ 4 y a ( B − 1 B ′ B − 1 ) a b y b . (58) \begin{aligned}
W^{-1}_{\sigma_A\sigma_A} &= N_{\rm train}\frac{1}{\sigma_A^2}-\frac{3}{\sigma_A^4}y_aB^{-1}_{ab}y_b,\\
W^{-1}_{\sigma_A\Lambda} &= -\frac{1}{\Lambda^3\sigma_A^2}y_a\left(B^{-1}B'B^{-1}\right)_{ab}y_b,\\
W^{-1}_{\Lambda\Lambda} &= \frac{1}{2}\frac{d^2}{d\Lambda^2}\ln(|B|)+\frac{1}{2\sigma_A^2\Lambda^6}y_a\left(2B^{-1}B'B^{-1}B'B^{-1}-B^{-1}B''B^{-1}\right)_{ab}y_b\\
&\quad+\frac{3}{2\sigma_A^2\Lambda^4}y_a\left(B^{-1}B'B^{-1}\right)_{ab}y_b. \tag*{(58)}
\end{aligned} W σ A σ A − 1 W σ A Λ − 1 W ΛΛ − 1 = N train σ A 2 1 − σ A 4 3 y a B ab − 1 y b , = − Λ 3 σ A 2 1 y a ( B − 1 B ′ B − 1 ) ab y b , = 2 1 d Λ 2 d 2 ln ( ∣ B ∣ ) + 2 σ A 2 Λ 6 1 y a ( 2 B − 1 B ′ B − 1 B ′ B − 1 − B − 1 B ′′ B − 1 ) ab y b + 2 σ A 2 Λ 4 3 y a ( B − 1 B ′ B − 1 ) ab y b . (58)
To calculate ( d 2 / d Λ 2 ) ln ( ∣ B ∣ ) \left(d^2/d\Lambda^2\right)\ln(|B|) ( d 2 / d Λ 2 ) ln ( ∣ B ∣ ) , one can use the identity
d d Λ ∣ B ∣ = ∣ B ∣ tr ( B − 1 d B d Λ ) . (59) \frac{d}{d\Lambda}|B|=|B|\operatorname{tr}\left(B^{-1}\frac{dB}{d\Lambda}\right). \tag*{(59)} d Λ d ∣ B ∣ = ∣ B ∣ tr ( B − 1 d Λ d B ) . (59)
This leads to
d 2 d Λ 2 ln ( ∣ B ∣ ) = − tr ( B − 1 d B d Λ B − 1 d B d Λ ) + tr ( B − 1 d 2 B d Λ 2 ) . (60) \frac{d^2}{d\Lambda^2}\ln(|B|)=-\operatorname{tr}\left(B^{-1}\frac{dB}{d\Lambda}B^{-1}\frac{dB}{d\Lambda}\right)+\operatorname{tr}\left(B^{-1}\frac{d^2B}{d\Lambda^2}\right). \tag*{(60)} d Λ 2 d 2 ln ( ∣ B ∣ ) = − tr ( B − 1 d Λ d B B − 1 d Λ d B ) + tr ( B − 1 d Λ 2 d 2 B ) . (60)
The last piece required to calculate the emulator uncertainty in Eq. (52) is ∂ Λ Y ( θ ⃗ ) \partial_\Lambda Y(\vec{\theta}) ∂ Λ Y ( θ ) .
∂ Λ Y ( θ ⃗ ) = ∂ Λ C 0 ( θ ⃗ , θ ⃗ a ) B a b − 1 y b = { 1 Λ 3 ∣ θ ⃗ − θ ⃗ a ∣ 2 C 0 ( θ ⃗ , θ ⃗ a ) B a b − 1 − C 0 ( θ ⃗ , θ ⃗ a ) ( B − 1 [ d d Λ B ] B − 1 ) a b } y b . (61) \begin{aligned}
\partial_\Lambda Y(\vec{\theta})&=\partial_\Lambda C_0(\vec{\theta},\vec{\theta}_a)B^{-1}_{ab}y_b\\
&=\left\{\frac{1}{\Lambda^3}|\vec{\theta}-\vec{\theta}_a|^2C_0(\vec{\theta},\vec{\theta}_a)B^{-1}_{ab}-C_0(\vec{\theta},\vec{\theta}_a)\left(B^{-1}\left[\frac{d}{d\Lambda}B\right]B^{-1}\right)_{ab}\right\}y_b. \tag*{(61)}
\end{aligned} ∂ Λ Y ( θ ) = ∂ Λ C 0 ( θ , θ a ) B ab − 1 y b = { Λ 3 1 ∣ θ − θ a ∣ 2 C 0 ( θ , θ a ) B ab − 1 − C 0 ( θ , θ a ) ( B − 1 [ d Λ d B ] B − 1 ) ab } y b . (61)
Thus, to estimate the contribution to the emulator uncertainty due to the uncertainties in determining the hyper parameters one should perform the following steps.
First tune the emulator and find the optimum values of σ A \sigma_A σ A and Λ \Lambda Λ .
Find the 2 × 2 2 \times2 2 × 2 matrix W − 1 W^{-1} W − 1 by using Eqs. (58) with the matrices B B B , B ′ B' B ′ and B ′ ′ B'' B ′′ calculated using Λ \Lambda Λ from step 1. This also involves finding ( d 2 / d Λ 2 ) ln ( ∣ B ∣ ) \left(d^2/d\Lambda^2\right)\ln(|B|) ( d 2 / d Λ 2 ) ln ( ∣ B ∣ ) .
Invert W − 1 W^{-1} W − 1 to find W W W . Only W Λ Λ W_{\Lambda\Lambda} W ΛΛ will be needed because ∂ Y / ∂ σ A = 0 \partial Y/\partial\sigma_A=0 ∂ Y / ∂ σ A = 0 .
Find d Y / d Λ dY/d\Lambda d Y / d Λ from Eq. (61) and using W Λ Λ W_{\Lambda\Lambda} W ΛΛ calculate the contribution to σ Y 2 \sigma_Y^2 σ Y 2 in Eq. (52) .
The Emulator Uncertainty at Training Points
First, we demonstrate the well-known result that in the absence of point-by-point noise, i.e. Δ = 0 \Delta=0 Δ = 0 , the estimated uncertainty will zero. If one sets θ ⃗ = θ ⃗ c \vec{\theta}=\vec{\theta}_c θ = θ c , one of the training points, and if Δ = 0 \Delta=0 Δ = 0 , then
C 0 ( θ ⃗ a , θ ⃗ c ) = B a c , (62) C_0(\vec{\theta}_a,\vec{\theta}_c)=B_{ac}, \tag*{(62)} C 0 ( θ a , θ c ) = B a c , (62)
T m ( θ ⃗ c ) T m ( θ ⃗ c ) = B c c , no sum over c , T_m(\vec{\theta}_c)T_m(\vec{\theta}_c)=B_{cc},\ \text{no sum over }c, T m ( θ c ) T m ( θ c ) = B cc , no sum over c ,
and
Y ( θ ⃗ c ) = χ a C 0 ( θ ⃗ a , θ ⃗ c ) = B a b − 1 y b B a c = y c , (63) Y(\vec{\theta}_c)=\chi_aC_0(\vec{\theta}_a,\vec{\theta}_c)=B^{-1}_{ab}y_bB_{ac}=y_c, \tag*{(63)} Y ( θ c ) = χ a C 0 ( θ a , θ c ) = B ab − 1 y b B a c = y c , (63)
σ Y 2 ( θ ⃗ c ) ∣ Δ = 0 = σ A 2 B c c − σ A 2 B c a B a b − 1 B b c + W Λ Λ ( ∂ Y ∂ Λ ) 2 = W Λ Λ ( ∂ Y ∂ Λ ) 2 . \begin{aligned}
\sigma_Y^2(\vec{\theta}_c)\big|_{\Delta=0}&=\sigma_A^2B_{cc}-\sigma_A^2B_{ca}B^{-1}_{ab}B_{bc}+W_{\Lambda\Lambda}\left(\frac{\partial Y}{\partial\Lambda}\right)^2\\
&=W_{\Lambda\Lambda}\left(\frac{\partial Y}{\partial\Lambda}\right)^2.
\end{aligned} σ Y 2 ( θ c ) Δ = 0 = σ A 2 B cc − σ A 2 B c a B ab − 1 B b c + W ΛΛ ( ∂ Λ ∂ Y ) 2 = W ΛΛ ( ∂ Λ ∂ Y ) 2 .
Next, using Eq. (61) , one can see that evaluated at a training point, θ ⃗ c \vec{\theta}_c θ c , is
∂ Y ( θ ⃗ c ) ∂ Λ = [ B c a ′ B a b − 1 − B c a ( B − 1 B ′ B − 1 ) a b ] y b = 0. (64) \frac{\partial Y(\vec{\theta}_c)}{\partial\Lambda}=\left[B'_{ca}B^{-1}_{ab}-B_{ca}\left(B^{-1}B'B^{-1}\right)_{ab}\right]y_b=0. \tag*{(64)} ∂ Λ ∂ Y ( θ c ) = [ B c a ′ B ab − 1 − B c a ( B − 1 B ′ B − 1 ) ab ] y b = 0. (64)
Here, B ′ = ( d / d Λ ) B B' = (d/d\Lambda)B B ′ = ( d / d Λ ) B . Thus, if Δ = 0 \Delta= 0 Δ = 0 , there is no uncertainty when evaluated at the training points and the optimum value for the emulator exactly reproduces the training values.
When Δ ≠ 0 \Delta\ne0 Δ = 0 , the optimum is no longer constrained to exactly reproduce the full model at the training points, and the uncertainty at a training point, θ ⃗ c \vec{\theta}_c θ c is no longer zero,
Var ( Y ( θ ⃗ c ) ) = σ A 2 ( Δ c c − Δ c a B a b − 1 Δ b c ) , (no sum over c ). (65) \operatorname{Var}(Y(\vec{\theta}_c)) = \sigma_A^2 \left(\Delta_{cc} - \Delta_{ca}B^{-1}_{ab}\Delta_{bc}\right), \qquad\text{(no sum over }c\text{).} \tag*{(65)} Var ( Y ( θ c )) = σ A 2 ( Δ cc − Δ c a B ab − 1 Δ b c ) , (no sum over c ). (65)
If Δ \Delta Δ is small, the uncertainty of the emulator is simply given by σ A Δ 1 / 2 \sigma_A\Delta^{1/2} σ A Δ 1/2 , however it is somewhat reduced for larger Δ \Delta Δ .
Typically, the emulator will be applied thousands, or perhaps millions of times, all for different θ ⃗ \vec{\theta} θ . Before evaluating the emulator, it would be efficient to first calculate and store both B a b − 1 B^{-1}_{ab} B ab − 1 and χ a \chi_a χ a . Calculating the matrix B B B involves calculating N t r a i n ( N t r a i n + 1 ) / 2 N_{\mathrm{train}}(N_{\mathrm{train}} + 1)/2 N train ( N train + 1 ) /2 exponentials, which is extremely fast. One must then invert B B B , an N t r a i n × N t r a i n N_{\mathrm{train}} \times N_{\mathrm{train}} N train × N train matrix. Then, when considering a specific θ ⃗ \vec{\theta} θ , one calculates C 0 ( θ ⃗ a , θ ⃗ ) C_0(\vec{\theta}_a,\vec{\theta}) C 0 ( θ a , θ ) for each a a a according to Eq. (19) . Once these are stored, calculating Var ( Y ( θ ⃗ ) ) \operatorname{Var}(Y(\vec{\theta})) Var ( Y ( θ )) involves calculating a double sum with N t r a i n 2 N_{\mathrm{train}}^2 N train 2 elements.
The derivations above could have been performed finding the coefficients A n ⃗ A_{\vec{n}} A n that maximize the probability, Z ( A ⃗ ) Z(\vec{A}) Z ( A ) , rather than solving for the mean coefficients, but the expressions for Y ( θ ⃗ ) Y(\vec{\theta}) Y ( θ ) and σ Y ( θ ⃗ ) \sigma_Y(\vec{\theta}) σ Y ( θ ) end up identical to those provided above.
Summarizing the Emulation
Despite the fact that the emulator was built in terms of optimized expansion coefficients A n ⃗ A_{\vec{n}} A n , the entire emulator construction and application avoids any mention of the coefficients. Through the kernel trick, the emulator need only know the form for the unconstrained correlation, C 0 ( θ ⃗ 1 , θ ⃗ 2 ) C_0(\vec{\theta}_1,\vec{\theta}_2) C 0 ( θ 1 , θ 2 ) . Using the short hand definitions of the N t r a i n × N t r a i n N_{\mathrm{train}} \times N_{\mathrm{train}} N train × N train matrix and N t r a i n N_{\mathrm{train}} N train -dimensional vector χ a \chi_a χ a ,
B a b ≡ C 0 ( θ ⃗ a , θ ⃗ b ) + Δ a b , (66) B_{ab} \equiv C_0(\vec{\theta}_a,\vec{\theta}_b) + \Delta_{ab}, \tag*{(66)} B ab ≡ C 0 ( θ a , θ b ) + Δ ab , (66)
χ a ≡ B a b − 1 y b , \chi_a \equiv B^{-1}_{ab}y_b, χ a ≡ B ab − 1 y b ,
one finds expressions for the following quantities,
σ A 2 = 1 N t r a i n χ a y a , (67) \sigma_A^2 = \frac{1}{N_{\mathrm{train}}}\chi_a y_a, \tag*{(67)} σ A 2 = N train 1 χ a y a , (67)
Z ( σ A 2 , Λ ) = σ A − N t r a i n ∣ B ∣ − 1 / 2 exp [ − 1 2 σ A 2 y a B a b − 1 y b ] = σ A − N t r a i n ∣ B ∣ − 1 / 2 e − 1 / ( 2 N t r a i n ) when σ a is optimized , Y ( θ ⃗ ) = χ a C 0 ( θ ⃗ a , θ ⃗ ) , σ Y 2 ( θ ⃗ ) = σ A 2 [ C 0 ( θ ⃗ , θ ⃗ ) − C 0 ( θ ⃗ , θ ⃗ a ) B a b − 1 C 0 ( θ ⃗ b , θ ⃗ ) ] + W Λ Λ ( ∂ Y ∂ Λ ) 2 . \begin{aligned}
Z(\sigma_A^2,\Lambda) &= \sigma_A^{-N_{\mathrm{train}}}|B|^{-1/2}\exp\left[-\frac{1}{2\sigma_A^2}y_aB^{-1}_{ab}y_b\right] \\
&= \sigma_A^{-N_{\mathrm{train}}}|B|^{-1/2}e^{-1/(2N_{\mathrm{train}})} \quad\text{when }\sigma_a\text{ is optimized}, \\
Y(\vec{\theta}) &= \chi_a C_0(\vec{\theta}_a,\vec{\theta}), \\
\sigma_Y^2(\vec{\theta}) &= \sigma_A^2\left[C_0(\vec{\theta},\vec{\theta}) - C_0(\vec{\theta},\vec{\theta}_a)B^{-1}_{ab}C_0(\vec{\theta}_b,\vec{\theta})\right] + W_{\Lambda\Lambda}\left(\frac{\partial Y}{\partial\Lambda}\right)^2.
\end{aligned} Z ( σ A 2 , Λ ) Y ( θ ) σ Y 2 ( θ ) = σ A − N train ∣ B ∣ − 1/2 exp [ − 2 σ A 2 1 y a B ab − 1 y b ] = σ A − N train ∣ B ∣ − 1/2 e − 1/ ( 2 N train ) when σ a is optimized , = χ a C 0 ( θ a , θ ) , = σ A 2 [ C 0 ( θ , θ ) − C 0 ( θ , θ a ) B ab − 1 C 0 ( θ b , θ ) ] + W ΛΛ ( ∂ Λ ∂ Y ) 2 .
Aside from the last term, proportional to W Λ Λ W_{\Lambda\Lambda} W ΛΛ , the other pieces are no different than what finds with GP emulators with Gaussian correlation kernels.
Outlining the procedure for tuning,
Choose the locations of the N t r a i n N_{\mathrm{train}} N train training points, θ ⃗ a \vec{\theta}_a θ a .
Choose a value of Λ \Lambda Λ .
Evaluate the full model at the training points, i.e. determine y a = 1 , N t r a i n y_{a=1,N_{\mathrm{train}}} y a = 1 , N train .
Define the noise matrix Δ a b \Delta_{ab} Δ ab . For example, if the noise is uncorrelated, and is approximately one percent of the signal, one would choose Δ a b = α 2 δ a b \Delta_{ab} = \alpha^2\delta_{ab} Δ ab = α 2 δ ab , with α = 0.01 \alpha= 0.01 α = 0.01 .
Calculate and store B B B , B i n v B_{\mathrm{inv}} B inv , χ \chi χ .
Calculate σ A 2 \sigma_A^2 σ A 2 according to Eq. (50) .
Calculate Z ( σ A 2 , Λ ) Z(\sigma_A^2,\Lambda) Z ( σ A 2 , Λ ) .
Repeat steps 2-7 for different Λ \Lambda Λ and keep the value of Λ \Lambda Λ , and its corresponding σ A \sigma_A σ A , that maximized Z Z Z .
Once both Λ \Lambda Λ and σ A \sigma_A σ A are determined, calculate and store W Λ Λ W_{\Lambda\Lambda} W ΛΛ .
To this point, the most numerically taxing part of the procedure is inverting B B B and calculating its determinant. One might need to repeat the steps, as mentioned in (8) roughly a dozen times to best find Λ \Lambda Λ .
Once this is finished, one can evaluate the emulator and its uncertainty at any θ ⃗ \vec{\theta} θ . At this point one needs to merely calculate a vector C 0 ( θ ⃗ a , θ ⃗ ) C_0(\vec{\theta}_a,\vec{\theta}) C 0 ( θ a , θ ) for a = 1 − N t r a i n a=1-N_{\mathrm{train}} a = 1 − N train , then contract the vector with χ \chi χ to obtain Y ( θ ⃗ ) Y(\vec{\theta}) Y ( θ ) or contract with B − 1 B^{-1} B − 1 to find the squared emulator uncertainty.
Choosing the Optimum Training Points
For emulating a convergent function, optimizing one’s choice of training points involves choosing points that allow accurate determination of the lowest-order expansion coefficients. If one has N p a r s N_{\mathrm{pars}} N pars model parameters, one needs at least N t r a i n = N p a r s + 1 N_{\mathrm{train}}=N_{\mathrm{pars}}+1 N train = N pars + 1 points to fit all the up-to-order-one coefficients, i.e. the slope in N p a r s N_{\mathrm{pars}} N pars dimensions and the intercept. To obtain the coefficients up to second order, one would need at least ( N p a r s + 1 ) ( N p a r s + 2 ) / 2 (N_{\mathrm{pars}}+1)(N_{\mathrm{pars}}+2)/2 ( N pars + 1 ) ( N pars + 2 ) /2 training values. Thus, it might seem natural to choose a set of either N p a r s + 1 N_{\mathrm{pars}}+1 N pars + 1 or ( N p a r s + 1 ) ( N p a r s + 2 ) / 2 (N_{\mathrm{pars}}+1)(N_{\mathrm{pars}}+2)/2 ( N pars + 1 ) ( N pars + 2 ) /2 training points with optimized positions.
One would like to find an algorithm than optimizes the constraining power of the training points for arbitrary priors, for an arbitrary numbers of model parameters, and for an arbitrary number of training points. This needs to be performed before the full model is run, and thus before any of the training values, y a y_a y a , are known. This means that the hyper-parameters, Λ \Lambda Λ and σ A \sigma_A σ A , are also as yet undetermined. In this section, algorithms are presented that accomplish this goal, but as will be shown, depend on an estimate for Λ \Lambda Λ . Given Λ \Lambda Λ , an optimized placement of the training points is found. Fortunately, the placement will not be strongly dependent on Λ \Lambda Λ . Unfortunately, the overall uncertainties are strongly sensitive to Λ \Lambda Λ , so choosing the number of training points required for a specific accuracy can be problematic.
An accuracy metric is defined based on the expected squared uncertainty averaged over the prior. If the priors are all some combination of Gaussian or uniform forms, analytic expressions are found for this metric. Although these expressions are rather messy, they enable a rather rapid numerically optimized positioning of the training points. The next subsection presents the analytic expressions for the accuracy metric. The analytic forms require that the prior can be expressed in terms of a product of either uniform or Gaussian priors for each model parameter. The next subsection provides a case study of how the accuracy behaves with the number of training points, with the number of parameters, and as a function of the estimated convergence parameter. For context, results are compared to Latin hyper-cube sampling [48 ] .
Certain numbers of training points allow the placement to satisfy symmetries of the prior, if such symmetries exist. If the prior is uniform within a N t r a i n N_{\mathrm{train}} N train –dimensional cube, the training points might be placed at the corners of an interior cube if N t r a i n = 2 N p a r s N_{\mathrm{train}}=2^{N_{\mathrm{pars}}} N train = 2 N pars , or perhaps adding an additional point at the center. For spherically symmetric priors with N t r a i n = N p a r s + 1 N_{\mathrm{train}}=N_{\mathrm{pars}}+1 N train = N pars + 1 , one might place the points in a simplex. The additional improvement for these specific values is rather modest, and is discussed in Sec. III C.
Expressing the Overall Emulator Accuracy
To optimize the location of the training points, we define the accuracy metric as being proportional to the emulator’s uncertainty squared, σ Y 2 ( θ ⃗ ) \sigma_Y^2(\vec{\theta}) σ Y 2 ( θ ) , averaged over the prior, P ( θ ⃗ ) P(\vec{\theta}) P ( θ ) , then divided by σ A 2 \sigma_A^2 σ A 2 . One might, or might not, be able to make solid arguments that averaging σ Y \sigma_Y σ Y to some other power might be more appropriate. This choice has the benefit of allowing much of the averaging to be performed analytically, and seems to provide a reasonable balance of weighing areas with larger uncertainty relative to areas with lower uncertainty.
The goal is thus to calculate the expected value of σ Y 2 ( θ ⃗ ) \sigma_Y^2(\vec{\theta}) σ Y 2 ( θ ) , then average this weighted by the prior, P ( θ ⃗ ) P(\vec{\theta}) P ( θ ) . This averaging over θ ⃗ \vec{\theta} θ will be denoted with the notation, E P ( σ Y 2 ) \mathrm{E}_P(\sigma_Y^2) E P ( σ Y 2 ) . Because the hyper-parameters are not yet determined at the time when the training points are chosen, the goal will be to minimize E P ( σ Y 2 ) \mathrm{E}_P(\sigma_Y^2) E P ( σ Y 2 ) for some assumed values of σ A \sigma_A σ A and Λ \Lambda Λ .
E P ( σ Y 2 ) ≡ ∫ d θ ⃗ P ( θ ⃗ ) σ Y 2 ( θ ⃗ ) = ∫ d θ ⃗ P ( θ ⃗ ) { σ A 2 [ 1 − C 0 ( θ ⃗ , θ ⃗ a ) B a b − 1 C 0 ( θ ⃗ b , θ ⃗ ) ] + ( ∂ Y ∂ Λ ) 2 W Λ Λ } . (68) \begin{aligned}
\mathrm{E}_P(\sigma_Y^2) \equiv\int d\vec{\theta}P(\vec{\theta})\sigma_Y^2(\vec{\theta}) \\
&= \int d\vec{\theta}P(\vec{\theta})\left\{\sigma_A^2\left[1-C_0(\vec{\theta},\vec{\theta}_a)B^{-1}_{ab}C_0(\vec{\theta}_b,\vec{\theta})\right]+\left(\frac{\partial Y}{\partial\Lambda}\right)^2W_{\Lambda\Lambda}\right\}. \tag*{(68)}
\end{aligned} E P ( σ Y 2 ) ≡ ∫ d θ P ( θ ) σ Y 2 ( θ ) = ∫ d θ P ( θ ) { σ A 2 [ 1 − C 0 ( θ , θ a ) B ab − 1 C 0 ( θ b , θ ) ] + ( ∂ Λ ∂ Y ) 2 W ΛΛ } . (68)
The expressions from Sec. (II) for W Λ Λ W_{\Lambda\Lambda} W ΛΛ and ( ∂ Y / ∂ Λ ) 2 (\partial Y/\partial\Lambda)^2 ( ∂ Y / ∂ Λ ) 2 in Eqs. (58) and Eq. (61) depend on the training values through the product y a y b y_a y_b y a y b . Given that the positions of the training points are not yet known, the quantities y a y b / σ a 2 y_a y_b/\sigma_a^2 y a y b / σ a 2 must be replaced with their unconstrained expectation,
y a y b σ A 2 → B a b = [ C 0 ( θ ⃗ a , θ ⃗ b ) + α 2 δ a b ] . (69) \frac{y_a y_b}{\sigma_A^2} \to B_{ab} = \left[C_0(\vec{\theta}_a,\vec{\theta}_b) + \alpha^2\delta_{ab}\right]. \tag*{(69)} σ A 2 y a y b → B ab = [ C 0 ( θ a , θ b ) + α 2 δ ab ] . (69)
The expressions for W W W , which are independent of θ ⃗ \vec{\theta} θ , then become
W σ A σ A − 1 = 2 N t r a i n σ A 2 , W σ A Λ − 1 = − 1 Λ 3 σ A Tr ( B ′ B − 1 ) , W Λ Λ − 1 = + 1 2 Λ 6 Tr ( B ′ B − 1 B ′ B − 1 ) . (70) \begin{aligned}
W^{-1}_{\sigma_A\sigma_A} &= \frac{2N_{\mathrm{train}}}{\sigma_A^2},\\
W^{-1}_{\sigma_A\Lambda} &= -\frac{1}{\Lambda^3\sigma_A}\operatorname{Tr}(B'B^{-1}),\\
W^{-1}_{\Lambda\Lambda} &= +\frac{1}{2\Lambda^6}\operatorname{Tr}(B'B^{-1}B'B^{-1}). \tag*{(70)}
\end{aligned} W σ A σ A − 1 W σ A Λ − 1 W ΛΛ − 1 = σ A 2 2 N train , = − Λ 3 σ A 1 Tr ( B ′ B − 1 ) , = + 2 Λ 6 1 Tr ( B ′ B − 1 B ′ B − 1 ) . (70)
One needs to invert W − 1 W^{-1} W − 1 to find W Λ Λ W_{\Lambda\Lambda} W ΛΛ , which will be independent of σ A \sigma_A σ A . The factor ( ∂ Λ Y ) 2 (\partial_\Lambda Y)^2 ( ∂ Λ Y ) 2 depends on θ ⃗ \vec{\theta} θ and becomes
( ∂ Λ Y ) 2 ( θ ⃗ ) = σ A 2 Λ 6 ∑ a b { C 0 ( θ ⃗ b , θ ⃗ ) ∣ θ ⃗ b − θ ⃗ ∣ 2 ∣ θ ⃗ − θ ⃗ a ∣ 2 C 0 ( θ ⃗ , θ ⃗ a ) B a b − 1 } − σ A 2 Λ 6 ∑ a b { C 0 ( θ ⃗ b , θ ⃗ ) ∣ θ ⃗ − θ ⃗ a ∣ 2 C 0 ( θ ⃗ , θ ⃗ a ) ( B − 1 B ′ B − 1 ) a b } − σ A 2 Λ 6 ∑ a b { C 0 ( θ ⃗ b , θ ⃗ ) ∣ θ ⃗ − θ ⃗ b ∣ 2 C 0 ( θ ⃗ , θ ⃗ a ) ( B − 1 B ′ B − 1 ) a b } + σ A 2 Λ 6 ∑ a b { C 0 ( θ ⃗ b , θ ⃗ ) C 0 ( θ ⃗ , θ ⃗ a ) ( B − 1 B ′ B − 1 B ′ B − 1 ) a b } . (71) \begin{aligned}
(\partial_\Lambda Y)^2(\vec{\theta}) ={}& \frac{\sigma_A^2}{\Lambda^6}\sum_{ab}\left\{C_0(\vec{\theta}_b,\vec{\theta})|\vec{\theta}_b-\vec{\theta}|^2|\vec{\theta}-\vec{\theta}_a|^2C_0(\vec{\theta},\vec{\theta}_a)B^{-1}_{ab}\right\}\\
&-\frac{\sigma_A^2}{\Lambda^6}\sum_{ab}\left\{C_0(\vec{\theta}_b,\vec{\theta})|\vec{\theta}-\vec{\theta}_a|^2C_0(\vec{\theta},\vec{\theta}_a)(B^{-1}B'B^{-1})_{ab}\right\}\\
&-\frac{\sigma_A^2}{\Lambda^6}\sum_{ab}\left\{C_0(\vec{\theta}_b,\vec{\theta})|\vec{\theta}-\vec{\theta}_b|^2C_0(\vec{\theta},\vec{\theta}_a)(B^{-1}B'B^{-1})_{ab}\right\}\\
&+\frac{\sigma_A^2}{\Lambda^6}\sum_{ab}\left\{C_0(\vec{\theta}_b,\vec{\theta})C_0(\vec{\theta},\vec{\theta}_a)(B^{-1}B'B^{-1}B'B^{-1})_{ab}\right\}. \tag*{(71)}
\end{aligned} ( ∂ Λ Y ) 2 ( θ ) = Λ 6 σ A 2 ab ∑ { C 0 ( θ b , θ ) ∣ θ b − θ ∣ 2 ∣ θ − θ a ∣ 2 C 0 ( θ , θ a ) B ab − 1 } − Λ 6 σ A 2 ab ∑ { C 0 ( θ b , θ ) ∣ θ − θ a ∣ 2 C 0 ( θ , θ a ) ( B − 1 B ′ B − 1 ) ab } − Λ 6 σ A 2 ab ∑ { C 0 ( θ b , θ ) ∣ θ − θ b ∣ 2 C 0 ( θ , θ a ) ( B − 1 B ′ B − 1 ) ab } + Λ 6 σ A 2 ab ∑ { C 0 ( θ b , θ ) C 0 ( θ , θ a ) ( B − 1 B ′ B − 1 B ′ B − 1 ) ab } . (71)
The expression for σ Y 2 \sigma_Y^2 σ Y 2 above has a simple dependence w.r.t σ A \sigma_A σ A . Every term is proportional to σ A 2 \sigma_A^2 σ A 2 . In contrast, the Λ \Lambda Λ dependence is complicated. To optimize the positions of the training points, θ ⃗ a = 1 … N t r a i n \vec{\theta}_{a=1\ldots N_{\mathrm{train}}} θ a = 1 … N train , one must minimize the r.h.s. of Eq. (68) . Unfortunately, without any input from full-model runs, one can only guess at Λ \Lambda Λ . Before training all values of σ A 2 \sigma_A^2 σ A 2 are considered equally likely. Thus, because of the simple proportionality to σ A 2 \sigma_A^2 σ A 2 , one can choose any σ A \sigma_A σ A and it will not affect the choice of training-point locations to minimize E P ( σ Y 2 ) \mathbf{E}_P(\sigma_Y^2) E P ( σ Y 2 ) . Thus, as a metric for the expected accuracy, we will choose
A c c u r a c y m e t r i c = E P ( σ Y 2 ) σ A 2 , (72) \mathrm{Accuracy\ metric} = \frac{\mathbf{E}_P(\sigma_Y^2)}{\sigma_A^2}, \tag*{(72)} Accuracy metric = σ A 2 E P ( σ Y 2 ) , (72)
which is independent of σ A \sigma_A σ A and depends only on Λ \Lambda Λ and on the location of the training points. The goal will be to find locations of the training points that minimize the metric after guessing at a value for Λ \Lambda Λ .
Once Λ \Lambda Λ is chosen, one can integrate Eq. (68) over the prior to calculate the accuracy. Because the matrices, B B B , B ′ B' B ′ and B ′ ′ B'' B ′′ , defined in Eqs. (38) and (57) , depend only on Λ \Lambda Λ and on the locations of the training points, one can express the metric in terms of the integrals, I a b I_{ab} I ab , J a b J_{ab} J ab and K a b K_{ab} K ab defined as:
E P ( σ Y 2 ) σ A 2 = Tr ( I − I B − 1 ) + W Λ Λ Λ 6 Tr { K B − 1 + J B − 1 B ′ B − 1 + I B − 1 B ′ B − 1 B ′ B − 1 } , (73) \frac{\mathbf{E}_P(\sigma_Y^2)}{\sigma_A^2}
= \operatorname{Tr}(\mathbb{I}-IB^{-1})+\frac{W_{\Lambda\Lambda}}{\Lambda^6}\operatorname{Tr}\left\{KB^{-1}+JB^{-1}B'B^{-1}+IB^{-1}B'B^{-1}B'B^{-1}\right\}, \tag*{(73)} σ A 2 E P ( σ Y 2 ) = Tr ( I − I B − 1 ) + Λ 6 W ΛΛ Tr { K B − 1 + J B − 1 B ′ B − 1 + I B − 1 B ′ B − 1 B ′ B − 1 } , (73)
I a b ≡ ∫ d θ ⃗ P ( θ ⃗ ) C 0 ( θ ⃗ a , θ ⃗ ) C 0 ( θ ⃗ , θ ⃗ b ) , I_{ab}\equiv\int d\vec{\theta}\,P(\vec{\theta})C_0(\vec{\theta}_a,\vec{\theta})C_0(\vec{\theta},\vec{\theta}_b), I ab ≡ ∫ d θ P ( θ ) C 0 ( θ a , θ ) C 0 ( θ , θ b ) ,
J a b ≡ J a b ; a + J a b ; b , J_{ab}\equiv J_{ab;a}+J_{ab;b}, J ab ≡ J ab ; a + J ab ; b ,
J a b ; a ≡ ∫ d θ ⃗ P ( θ ⃗ ) C 0 ( θ ⃗ a , θ ⃗ ) ∣ θ ⃗ a − θ ⃗ ∣ 2 C 0 ( θ ⃗ , θ ⃗ b ) , J_{ab;a}\equiv\int d\vec{\theta}\,P(\vec{\theta})C_0(\vec{\theta}_a,\vec{\theta})|\vec{\theta}_a-\vec{\theta}|^2C_0(\vec{\theta},\vec{\theta}_b), J ab ; a ≡ ∫ d θ P ( θ ) C 0 ( θ a , θ ) ∣ θ a − θ ∣ 2 C 0 ( θ , θ b ) ,
K a b ≡ ∫ d θ ⃗ P ( θ ⃗ ) C 0 ( θ ⃗ a , θ ⃗ ) ∣ θ ⃗ a − θ ⃗ ∣ 2 ∣ θ ⃗ − θ ⃗ b ∣ 2 C 0 ( θ ⃗ , θ ⃗ b ) . K_{ab}\equiv\int d\vec{\theta}\,P(\vec{\theta})C_0(\vec{\theta}_a,\vec{\theta})|\vec{\theta}_a-\vec{\theta}|^2|\vec{\theta}-\vec{\theta}_b|^2C_0(\vec{\theta},\vec{\theta}_b). K ab ≡ ∫ d θ P ( θ ) C 0 ( θ a , θ ) ∣ θ a − θ ∣ 2 ∣ θ − θ b ∣ 2 C 0 ( θ , θ b ) .
Here, a a a and b b b run from 1 to N t r a i n N_{\mathrm{train}} N train . The N t r a i n × N t r a i n N_{\mathrm{train}}\times N_{\mathrm{train}} N train × N train matrices I I I , J J J and K K K carry all the relevant information about the prior.
If the prior P ( θ ⃗ ) P(\vec{\theta}) P ( θ ) can be written as product of priors for individual model parameters,
P ( θ ⃗ ) = ∏ i P i ( θ i ) , (74) P(\vec{\theta}) = \prod_i P_i(\theta_i), \tag*{(74)} P ( θ ) = i ∏ P i ( θ i ) , (74)
one can express the integrals over the prior in terms of integrals over individual parameters.
I a b = ∏ i I a b ; i , J a b = I a b ∑ i J a b ; a i + J a b ; b i I a b ; i , K a b = J a b ; a i J a b ; b i I a b + I a b ∑ i { K a b ; i I a b ; i − J a b ; a i J a b ; b i I a b ; i 2 } . (75) \begin{aligned}
I_{ab} &= \prod_i I_{ab;i},\\
J_{ab} &= I_{ab}\sum_i \frac{J_{ab;ai}+J_{ab;bi}}{I_{ab;i}},\\
K_{ab} &= \frac{J_{ab;ai}J_{ab;bi}}{I_{ab}}+I_{ab}\sum_i\left\{\frac{K_{ab;i}}{I_{ab;i}}-\frac{J_{ab;ai}J_{ab;bi}}{I_{ab;i}^{2}}\right\}. \tag*{(75)}
\end{aligned} I ab J ab K ab = i ∏ I ab ; i , = I ab i ∑ I ab ; i J ab ; ai + J ab ; bi , = I ab J ab ; ai J ab ; bi + I ab i ∑ { I ab ; i K ab ; i − I ab ; i 2 J ab ; ai J ab ; bi } . (75)
Here, the integrals over single parameters are:
I a b ; i = ∫ d θ i P i ( θ i ) exp { − ( θ i − θ a i ) 2 2 Λ 2 − ( θ i − θ b i ) 2 2 Λ 2 } , J a b ; a i = ∫ d θ i P i ( θ i ) ( θ i − θ a i ) 2 exp { − ( θ i − θ a i ) 2 2 Λ 2 − ( θ i − θ b i ) 2 2 Λ 2 } , K a b ; i = ∫ d θ i P i ( θ i ) ( θ i − θ a i ) 2 ( θ i − θ b i ) 2 exp { − ( θ i − θ a i ) 2 2 Λ 2 − ( θ i − θ b i ) 2 2 Λ 2 } . (76) \begin{aligned}
I_{ab;i} &= \int d\theta_i\,P_i(\theta_i)\exp\left\{-\frac{(\theta_i-\theta_{ai})^2}{2\Lambda^2}-\frac{(\theta_i-\theta_{bi})^2}{2\Lambda^2}\right\},\\
J_{ab;ai} &= \int d\theta_i\,P_i(\theta_i)(\theta_i-\theta_{ai})^2\exp\left\{-\frac{(\theta_i-\theta_{ai})^2}{2\Lambda^2}-\frac{(\theta_i-\theta_{bi})^2}{2\Lambda^2}\right\},\\
K_{ab;i} &= \int d\theta_i\,P_i(\theta_i)(\theta_i-\theta_{ai})^2(\theta_i-\theta_{bi})^2\exp\left\{-\frac{(\theta_i-\theta_{ai})^2}{2\Lambda^2}-\frac{(\theta_i-\theta_{bi})^2}{2\Lambda^2}\right\}. \tag*{(76)}
\end{aligned} I ab ; i J ab ; ai K ab ; i = ∫ d θ i P i ( θ i ) exp { − 2 Λ 2 ( θ i − θ ai ) 2 − 2 Λ 2 ( θ i − θ bi ) 2 } , = ∫ d θ i P i ( θ i ) ( θ i − θ ai ) 2 exp { − 2 Λ 2 ( θ i − θ ai ) 2 − 2 Λ 2 ( θ i − θ bi ) 2 } , = ∫ d θ i P i ( θ i ) ( θ i − θ ai ) 2 ( θ i − θ bi ) 2 exp { − 2 Λ 2 ( θ i − θ ai ) 2 − 2 Λ 2 ( θ i − θ bi ) 2 } . (76)
After calculating the integrals I a b ; i I_{ab;i} I ab ; i analytically, the integrals J a b ; a i J_{ab;ai} J ab ; ai , J a b ; b i J_{ab;bi} J ab ; bi and K a b ; i K_{ab;i} K ab ; i can all be generated by taking derivatives of I a b ; i I_{ab;i} I ab ; i w.r.t. γ = 1 / Λ 2 \gamma= 1/\Lambda^2 γ = 1/ Λ 2 .
One also needs an expression for W W W . This is found in the same way as for Eqs. Eqs. (58) . In those expressions W i n v W_{\rm inv} W inv was defined in terms of B a b B_{ab} B ab , where B a b B_{ab} B ab was expressed in terms of the training values through the matrix y a y b y_a y_b y a y b , whereas in this case B a b B_{ab} B ab is estimated from the pre-constrained covariance, defined in Eq. (69) .
If a specific model parameter, θ i \theta_i θ i , has a Gaussian prior,
P i ( θ = θ i ) = 1 ( 2 π / β i ) 1 / 2 e − β i θ i 2 / 2 , (77) P_i(\theta=\theta_i)=\frac{1}{(2\pi/\beta_i)^{1/2}}e^{-\beta_i\theta_i^2/2}, \tag*{(77)} P i ( θ = θ i ) = ( 2 π / β i ) 1/2 1 e − β i θ i 2 /2 , (77)
one can integrate Eqs. (76) and find
I a b ; i = β i λ i exp { − 1 2 λ i [ γ 2 ( θ a − θ b ) 2 + β i γ ( θ a 2 + θ b 2 ) ] } , (78) I_{ab;i}=\sqrt{\frac{\beta_i}{\lambda_i}}\exp\left\{-\frac{1}{2\lambda_i}\left[\gamma^2(\theta_a-\theta_b)^2+\beta_i\gamma(\theta_a^2+\theta_b^2)\right]\right\}, \tag*{(78)} I ab ; i = λ i β i exp { − 2 λ i 1 [ γ 2 ( θ a − θ b ) 2 + β i γ ( θ a 2 + θ b 2 ) ] } , (78)
J a b ; a i = I a b ; i { 1 λ i + ∣ θ ⃗ a − θ ⃗ b ∣ 2 γ 2 + β i γ λ i 2 + ( θ a 2 + θ b 2 ) β i 2 2 λ i 2 + β i 2 λ i ( θ a 2 − θ b 2 ) } , J a b ; b i = I a b ; i { 1 λ i + ∣ θ ⃗ a − θ ⃗ b ∣ 2 γ 2 + β i γ λ i 2 + ( θ a 2 + θ b 2 ) β i 2 2 λ i 2 − β i 2 λ i ( θ a 2 − θ b 2 ) } , K a b ; i = J a b ; a i J a b ; b i I a b ; i + I a b ; i { 2 λ i 2 + 2 β i 2 ( θ a 2 + θ b 2 ) λ i 3 − ( 4 γ 2 + 2 β i λ i ) ( θ a − θ b ) 2 λ i 3 } . \begin{aligned}
J_{ab;ai} &= I_{ab;i}\left\{\frac{1}{\lambda_i}+|\vec{\theta}_a-\vec{\theta}_b|^2\frac{\gamma^2+\beta_i\gamma}{\lambda_i^2}+(\theta_a^2+\theta_b^2)\frac{\beta_i^2}{2\lambda_i^2}+\frac{\beta_i}{2\lambda_i}(\theta_a^2-\theta_b^2)\right\},\\
J_{ab;bi} &= I_{ab;i}\left\{\frac{1}{\lambda_i}+|\vec{\theta}_a-\vec{\theta}_b|^2\frac{\gamma^2+\beta_i\gamma}{\lambda_i^2}+(\theta_a^2+\theta_b^2)\frac{\beta_i^2}{2\lambda_i^2}-\frac{\beta_i}{2\lambda_i}(\theta_a^2-\theta_b^2)\right\},\\
K_{ab;i} &= \frac{J_{ab;ai}J_{ab;bi}}{I_{ab;i}}+I_{ab;i}\left\{\frac{2}{\lambda_i^2}+\frac{2\beta_i^2(\theta_a^2+\theta_b^2)}{\lambda_i^3}-\frac{(4\gamma^2+2\beta_i\lambda_i)(\theta_a-\theta_b)^2}{\lambda_i^3}\right\}.
\end{aligned} J ab ; ai J ab ; bi K ab ; i = I ab ; i { λ i 1 + ∣ θ a − θ b ∣ 2 λ i 2 γ 2 + β i γ + ( θ a 2 + θ b 2 ) 2 λ i 2 β i 2 + 2 λ i β i ( θ a 2 − θ b 2 ) } , = I ab ; i { λ i 1 + ∣ θ a − θ b ∣ 2 λ i 2 γ 2 + β i γ + ( θ a 2 + θ b 2 ) 2 λ i 2 β i 2 − 2 λ i β i ( θ a 2 − θ b 2 ) } , = I ab ; i J ab ; ai J ab ; bi + I ab ; i { λ i 2 2 + λ i 3 2 β i 2 ( θ a 2 + θ b 2 ) − λ i 3 ( 4 γ 2 + 2 β i λ i ) ( θ a − θ b ) 2 } .
where λ i ≡ 2 γ + β i \lambda_i\equiv2\gamma+\beta_i λ i ≡ 2 γ + β i and γ = 1 / Λ 2 \gamma=1/\Lambda^2 γ = 1/ Λ 2 .
Similarly, one can perform the integrals for uniform priors. In this case the prior is
P i ( θ = θ i ) = { 1 2 θ max , i , − θ max , i < θ < θ max , i , 0 , otherwise . (79) P_i(\theta=\theta_i)=
\begin{cases}
\frac{1}{2\theta_{\max,i}}, & -\theta_{\max,i}<\theta<\theta_{\max,i},\\
0, & \text{otherwise}.
\end{cases} \tag*{(79)} P i ( θ = θ i ) = { 2 θ m a x , i 1 , 0 , − θ m a x , i < θ < θ m a x , i , otherwise . (79)
After making the definitions
θ ˉ ≡ θ a + θ b 2 , Δ θ ≡ θ a − θ b 2 , θ + ≡ θ max , i − θ ˉ , θ − ≡ − θ max , i − θ ˉ , p ≡ 1 2 θ max , i e − γ Δ θ 2 , q ≡ 1 2 π γ [ erf ( γ θ + ) − erf ( γ θ − ) ] , r ≡ θ + e − γ θ + 2 − θ − e − γ θ − 2 , (80) \begin{aligned}
\bar{\theta} &\equiv\frac{\theta_a+\theta_b}{2},\\
\Delta\theta&\equiv\frac{\theta_a-\theta_b}{2},\\
\theta_+ &\equiv\theta_{\max,i}-\bar{\theta},\\
\theta_- &\equiv-\theta_{\max,i}-\bar{\theta},\\
p &\equiv\frac{1}{2\theta_{\max,i}}e^{-\gamma\Delta\theta^2},\\
q &\equiv\frac{1}{2}\sqrt{\frac{\pi}{\gamma}}\left[\operatorname{erf}(\sqrt{\gamma}\theta_+)-\operatorname{erf}(\sqrt{\gamma}\theta_-)\right],\\
r &\equiv\theta_+e^{-\gamma\theta_+^2}-\theta_-e^{-\gamma\theta_-^2}, \tag*{(80)}
\end{aligned} θ ˉ Δ θ θ + θ − p q r ≡ 2 θ a + θ b , ≡ 2 θ a − θ b , ≡ θ m a x , i − θ ˉ , ≡ − θ m a x , i − θ ˉ , ≡ 2 θ m a x , i 1 e − γ Δ θ 2 , ≡ 2 1 γ π [ erf ( γ θ + ) − erf ( γ θ − ) ] , ≡ θ + e − γ θ + 2 − θ − e − γ θ − 2 , (80)
the integrals of interest are
I a b ; i = p q , J a b ; a i = ( 1 2 γ − Δ θ 2 ) I a b ; i − 1 2 γ p r + p e − γ θ + 2 Δ θ γ − P e − γ θ − 2 Δ θ γ , J a b ; b i = ( 1 2 γ − Δ θ 2 ) I a b ; i − 1 2 γ p r − p e − γ θ + 2 Δ θ γ + P e − γ θ − 2 Δ θ γ , K a b ; i = 1 2 ( J a b ; a i + J a b ; b i ) ( 1 2 γ + Δ θ 2 ) + I a b ; i ( 1 2 γ 2 − 2 Δ θ 2 γ ) − p e − γ θ + 2 θ + ( 1 2 γ 2 − 3 Δ θ 2 2 γ + θ + 2 2 γ ) + p e − γ θ − 2 θ − ( 1 2 γ 2 − 3 Δ θ 2 2 γ + θ − 2 2 γ ) . (81) \begin{aligned}
I_{ab;i} &= pq,\\
J_{ab;ai} &= \left(\frac{1}{2\gamma}-\Delta\theta^2\right)I_{ab;i}-\frac{1}{2\gamma}pr+pe^{-\gamma\theta_+^2}\frac{\Delta\theta}{\gamma}-Pe^{-\gamma\theta_-^2}\frac{\Delta\theta}{\gamma},\\
J_{ab;bi} &= \left(\frac{1}{2\gamma}-\Delta\theta^2\right)I_{ab;i}-\frac{1}{2\gamma}pr-pe^{-\gamma\theta_+^2}\frac{\Delta\theta}{\gamma}+Pe^{-\gamma\theta_-^2}\frac{\Delta\theta}{\gamma},\\
K_{ab;i} &= \frac{1}{2}(J_{ab;ai}+J_{ab;bi})\left(\frac{1}{2\gamma}+\Delta\theta^2\right)\\
&\quad+I_{ab;i}\left(\frac{1}{2\gamma^2}-\frac{2\Delta\theta^2}{\gamma}\right)\\
&\quad-pe^{-\gamma\theta_+^2}\theta_+\left(\frac{1}{2\gamma^2}-\frac{3\Delta\theta^2}{2\gamma}+\frac{\theta_+^2}{2\gamma}\right)\\
&\quad+pe^{-\gamma\theta_-^2}\theta_-\left(\frac{1}{2\gamma^2}-\frac{3\Delta\theta^2}{2\gamma}+\frac{\theta_-^2}{2\gamma}\right). \tag*{(81)}
\end{aligned} I ab ; i J ab ; ai J ab ; bi K ab ; i = pq , = ( 2 γ 1 − Δ θ 2 ) I ab ; i − 2 γ 1 p r + p e − γ θ + 2 γ Δ θ − P e − γ θ − 2 γ Δ θ , = ( 2 γ 1 − Δ θ 2 ) I ab ; i − 2 γ 1 p r − p e − γ θ + 2 γ Δ θ + P e − γ θ − 2 γ Δ θ , = 2 1 ( J ab ; ai + J ab ; bi ) ( 2 γ 1 + Δ θ 2 ) + I ab ; i ( 2 γ 2 1 − γ 2Δ θ 2 ) − p e − γ θ + 2 θ + ( 2 γ 2 1 − 2 γ 3Δ θ 2 + 2 γ θ + 2 ) + p e − γ θ − 2 θ − ( 2 γ 2 1 − 2 γ 3Δ θ 2 + 2 γ θ − 2 ) . (81)
For most of the results presented in this section Gaussian priors were considered.
Dependence on the number of parameters, the number of training points and the convergence parameter
The accuracy metric, defined in Eq. (72) , was optimized by performing a simple search through the parameter space. For some initial configuration of training points E P ( σ Y 2 ) / σ A 2 \mathbf{E}_{P}(\sigma_Y^2)/\sigma_A^2 E P ( σ Y 2 ) / σ A 2 was calculated. A new set of training points was then chosen by taking a small random step in parameter space for each training points. If the new uncertainty was reduced, the step was accepted. This is by no means a particularly efficient method, but the method was sufficiently fast for N p a r s ≲ 10 N_{\mathrm{pars}}\lesssim10 N pars ≲ 10 , as the optimized values were found within a few minutes or less. Larger numbers of parameters required increasing amounts of patience.
The difficulty in emulating high-dimension parameter spaces increases more than linearly with the number of model parameters. This is because the number of expansion coefficients of order K K K is
N c o e f f i c i e n t s ( K = 0 ) = 1 , N c o e f f i c i e n t s ( K = 1 ) = N p a r s , N c o e f f i c i e n t s ( K = 2 ) = N p a r s ( N p a r s + 1 ) / 2 , N c o e f f i c i e n t s ( K = 3 ) = N p a r s ( N p a r s + 1 ) ( N p a r s + 2 ) / 6 , ⋮ (82) \begin{aligned}
N_{\mathrm{coefficients}}(K=0)&=1,\\
N_{\mathrm{coefficients}}(K=1)&=N_{\mathrm{pars}},\\
N_{\mathrm{coefficients}}(K=2)&=N_{\mathrm{pars}}(N_{\mathrm{pars}}+1)/2,\\
N_{\mathrm{coefficients}}(K=3)&=N_{\mathrm{pars}}(N_{\mathrm{pars}}+1)(N_{\mathrm{pars}}+2)/6,\\
&\vdots \tag*{(82)}
\end{aligned} N coefficients ( K = 0 ) N coefficients ( K = 1 ) N coefficients ( K = 2 ) N coefficients ( K = 3 ) = 1 , = N pars , = N pars ( N pars + 1 ) /2 , = N pars ( N pars + 1 ) ( N pars + 2 ) /6 , ⋮ (82)
Further, the number of coefficients of order less than or equal to K K K , is
N c o e f f i c i e n t s ( K ≤ 0 ) = 1 , N c o e f f i c i e n t s ( K ≤ 1 ) = N p a r s + 1 , N c o e f f i c i e n t s ( K ≤ 2 ) = ( N p a r s + 1 ) ( N p a r s + 2 ) / 2 , N c o e f f i c i e n t s ( K ≤ 3 ) = ( N p a r s + 1 ) ( ( N p a r s + 2 ) ( N p a r s + 3 ) / 6 , ⋮ (83) \begin{aligned}
N_{\mathrm{coefficients}}(K \leq0) &= 1,\\
N_{\mathrm{coefficients}}(K \leq1) &= N_{\mathrm{pars}} + 1,\\
N_{\mathrm{coefficients}}(K \leq2) &= (N_{\mathrm{pars}} + 1)(N_{\mathrm{pars}} + 2)/2,\\
N_{\mathrm{coefficients}}(K \leq3) &= (N_{\mathrm{pars}} + 1)((N_{\mathrm{pars}} + 2)(N_{\mathrm{pars}} + 3)/6,\\
&\vdots \tag*{(83)}
\end{aligned} N coefficients ( K ≤ 0 ) N coefficients ( K ≤ 1 ) N coefficients ( K ≤ 2 ) N coefficients ( K ≤ 3 ) = 1 , = N pars + 1 , = ( N pars + 1 ) ( N pars + 2 ) /2 , = ( N pars + 1 ) (( N pars + 2 ) ( N pars + 3 ) /6 , ⋮ (83)
Thus, the number of terms of order K + 1 K + 1 K + 1 is larger than the number of terms of order K K K by a factor of ( N p a r s + K ) / K (N_{\mathrm{pars}} + K)/K ( N pars + K ) / K . If the number of parameters is large, there are large numbers of parameters of higher order. For example for N p a r s = 12 N_{\mathrm{pars}} = 12 N pars = 12 there are 364 cubic terms, and for N p a r s = 20 N_{\mathrm{pars}} = 20 N pars = 20 there are 1540 cubic terms. Because the importance of higher order terms falls as 1 / Λ 2 n 1/\Lambda^{2n} 1/ Λ 2 n , emulating high-dimensional functions requires larger values of the convergence parameter to arrive at the same accuracy. Figure 1 shows the accuracy metric as a function of the number of parameters for fixed Λ = 3 \Lambda= 3 Λ = 3 and for a number of training points set to correspond to the number of expansion parameters of order two or less, N t r a i n = ( N p a r s + 1 ) ( N p a r s + 2 ) / 2 N_{\mathrm{train}} = (N_{\mathrm{pars}} + 1)(N_{\mathrm{pars}} + 2)/2 N train = ( N pars + 1 ) ( N pars + 2 ) /2 .
Figure 1. The accuracy metric as a function of the number of model parameters assuming Gaussian priors. The convergence parameter was set to Λ = 3 \Lambda= 3 Λ = 3 , the point-by-point noise was set to α = 0.01 \alpha= 0.01 α = 0.01 and the number of training points was set to match the number of expansion coefficient of quadratic order or less. This illustrates the difficulty to accurately emulate higher dimensional functions even if the number of training points increases as N t r a i n = ( N p a r s + 1 ) ( N p a r s + 2 ) / 2 N_{\mathrm{train}} = (N_{\mathrm{pars}}+1)(N_{\mathrm{pars}}+2)/2 N train = ( N pars + 1 ) ( N pars + 2 ) /2 .
It should be emphasized that this measure of the uncertainty is proportional to E P ( σ Y 2 ) \mathbf{E}_{P}(\sigma_Y^2) E P ( σ Y 2 ) , not to E P ( σ Y ) \mathbf{E}_{P}(\sigma_Y) E P ( σ Y ) . Thus, to get to 10% accuracy, one might want E P ( σ Y 2 ) / σ A 2 \mathbf{E}_{P}(\sigma_Y^2)/\sigma_A^2 E P ( σ Y 2 ) / σ A 2 to be close to 0.01.
If Λ \Lambda Λ is higher, it is easier to emulate functions because the contribution to E P ( σ Y 2 ) \mathbf{E}_{P}(\sigma_Y^2) E P ( σ Y 2 ) of terms of order n n n falls roughly as 1 / Λ 2 1/\Lambda^{2} 1/ Λ 2 . Figure 2 shows how the expected accuracy falls as a function of Λ \Lambda Λ . For this illustration the number of parameters was set to 6 and the number of training points was again set to correspond to the number of expansion coefficients of order n = 2 n = 2 n = 2 or less. Figure 2 also illustrates the dependence on the point-by-point noise-strength, α \alpha α . The effect of α \alpha α only sets in when the accuracy metric falls to near α 2 \alpha^2 α 2 . However, given that there are multiple training points are considered the expected uncertainty can fall below α 2 \alpha^2 α 2 .
Figure 2. The expected accuracy of the emulator falls rapidly with the convergence parameter Λ \Lambda Λ . For this illustration N p a r s = 6 N_{\mathrm{pars}} = 6 N pars = 6 and the number of training points was set to 28, the number of expansion coefficients of order n = 2 n = 2 n = 2 or less. Calculations were repeated for different values of the point-by-point noise, α \alpha α . As expected, the impact of the point-by-point noise sets in as the accuracy of the emulator improves.
The accuracy should increase with more training points. This is illustrated for the case of six model parameters with Λ = 3.0 \Lambda= 3.0 Λ = 3.0 and α = 0.01 \alpha= 0.01 α = 0.01 in Figure 3 . The behavior underscores the value of choosing a number of training points corresponding to the number of independent coefficients for a linear, quadratic or cubic expansion. For six model parameters, those numbers of coefficients are N t r a i n = 7 N_{\mathrm{train}} = 7 N train = 7 , 28 and 84 respectively. Once one of these thresholds is passed, the marginal value of an additional training point decreases. Then, as one approaches the next threshold, the marginal value increases again, though not to the degree of those points below the previous threshold.
Figure 3. For all Gaussian priors, with N p a r s = 6 N_{\mathrm{pars}} = 6 N pars = 6 , Λ = 3 \Lambda= 3 Λ = 3 , and α = 0.01 \alpha= 0.01 α = 0.01 , the expected mean accuracy improves with increasing the number of training points. The minimum number of training points required for linear, quadratic and cubic fits are 7, 28 and 82. After passing one of these thresholds the marginal improvement for additional training points decreases. For N = 8 N = 8 N = 8 , one can place the points in a simplex with an additional point at the center, and the symmetry of this configuration provides an especially large improvement. This behavior might inspire the modeler to choose a number of training points equal to one of the thresholds. The optimized placement provides significant improvement compared to Latin hyper-cube sampling.
The principal source of structure, or kinks, in Figure 3 is due to whether the number of training points passes one of the thresholds mentioned above. However, there is a second less prominent source of structure that occurs when the number of training points corresponds to a symmetry. Given that this example was for Gaussian priors for each parameter, with the same widths (relative to Λ \Lambda Λ ) for each prior, the prior had a spherical symmetry. Examples of arrangements with symmetries are simplexes and cubes. These might also include an extra point at the origin. Such structures vary with dimensionality. For this case, N p a r s = 6 N_{\mathrm{pars}} = 6 N pars = 6 , the uncertainty for N t r a i n = 8 N_{\mathrm{train}} = 8 N train = 8 appears noticeably less. On inspection, this turned out to be a simplex with an extra point at the center. Thus, one might well choose N t r a i n = 8 N_{\mathrm{train}} = 8 N train = 8 , even though it is one above the threshold. A smaller kink is seen at N t r a i n = 44 N_{\mathrm{train}} = 44 N train = 44 . But this structure is not so prominent as to drive one’s decision to run more points.
For many cases, one finds that the minimization procedure might settle on a false minimum. The searches were initialized by either randomly placing the training points according to the prior weights, or using Latin hyper-cube samples [48 ] . For Latin hyper-cube sampling each dimension of parameter space was divided into N t r a i n N_{\mathrm{train}} N train equally probable slices. For each parameter, the value for each training point was set randomly forced to lie in a different slice. Figure 4 shows how the accuracy metric improved with the number of random steps. Several chains are displayed to illustrate the progress of the search, and its dependence on the initial configuration. Settling on a false minimum can affect the estimated uncertainty by at most a few percent, and can lead to a bit of jitter for the optimized values. One can find the true minimum by repeating the minimization search several times, then choosing the best configuration. The number of random steps required to reach a desired accuracy increases with dimensionality of the model parameter space. The simple method applied here seems fine for cases where N p a r s N_{\mathrm{pars}} N pars is near or less than a dozen, but if one had two dozen parameters, it might warrant a more sophisticated search algorithm.
Figure 4. The search procedure provides most of the improvement for the first few thousand steps. For this example, N p a r s = 6 N_{\mathrm{pars}} = 6 N pars = 6 , Λ = 3 \Lambda= 3 Λ = 3 and α = 0.01 \alpha= 0.01 α = 0.01 the number of training points was set to 28. Several initial configurations were considered. Those where the points were placed randomly (blue circles) tended to start at roughly three times the optimized value, whereas those that were initialized with Latin hyper-cube sampling tended to start at a bit less than double the optimized uncertainty. The bands roughly represent the spread of initial accuracies, and the fall of the bands to the optimal values represents the improvement one expects to gain through the optimization procedure.
Also shown in Figure 3 and Figure 4 are the accuracy metrics for Latin hyper-cube sampling. As this is not an optimized choice, there is more noticeable jitter in the curves. One can see that the optimization procedure reduces the expected uncertainty by nearly a factor of two at the quadratic threshold and by nearly a factor of five at the cubic threshold. Depending on the desired accuracy, one might need 50% more runs with Latin hyper-cube sampling to achieve the same accuracy as would be obtained with an optimized placement. Figure 4 also illustrates the improvement compared to random placement of the training points. In that case the training points were placed independently, according to the prior, with no correlation between points.
While performing the calculations of E P ( σ Λ 2 ) / σ A 2 E_{\mathrm{P}}(\sigma_{\Lambda}^{2})/\sigma_{A}^{2} E P ( σ Λ 2 ) / σ A 2 considered here, the contributions to the uncertainty can be split into two pieces as expressed in Eq. (73) . Whereas the first term in Eq. (73) represented the uncertainty assuming Λ \Lambda Λ is fixed at its most probable value, the second term represents the emulator uncertainty coming from Λ \Lambda Λ varying, with the weight Z Z Z . Remarkably, this second contribution was never more than a few percent of the total expected uncertainty for the optimized placement, which makes it appear largely negligible. However, that fraction was often much larger when the optimization procedure was first started, perhaps a third of the uncertainty. Then as the training-point placement approached the optimal arrangement, this second contribution shrunk. It was observed that the first contribution, the part assuming Λ \Lambda Λ was fixed, varied less with the training point placement compared to the reduction in the first contribution. Thus, the second contribution was rather instrumental in driving the placement of the training points, despite the fact that it is the lesser contribution. This would suggest that while this contribution is crucial for choosing the location of the training points, but that once that optimization is completed, and one has built an emulator, that contribution can be largely neglected when estimating the emulator’s uncertainty.
Fine Structure of Point Placement
Here, fine structure refers to the fact that certain numbers of training points might allow the placement to satisfy symmetries of the prior. If the prior is Gaussian in all directions, and is rotationally symmetric, then one might expect the optimum choice to exploit the symmetry. For the case of N t r a i n = N p a r s + 1 N_{\mathrm{train}} = N_{\mathrm{pars}} + 1 N train = N pars + 1 , one can place the training points in an N p a r s N_{\mathrm{pars}} N pars dimensional simplex, a simplex being the placement of all points at the same radius, with the same distance between each pair. In two dimensions a simplex is an equilateral triangle, while for three dimensions the simplex is a tetrahedron. With this placement, there are a set of rotations under which each point can be rotated to another point in the structure without changing the structure. A few other arrangements may have similar symmetries. For example, one could have N p a r s + 1 N_{\mathrm{pars}} + 1 N pars + 1 training points placed in a simplex, while one additional point might be placed in the center. For three dimensions, a buckyball configuration with N t r a i n = 60 N_{\mathrm{train}} = 60 N train = 60 has a symmetry. But these symmetries are hard to come by, so in general it is not at all obvious how to place training points in an arbitrary number of dimensions, unless the number of training points happens to match some specific value. Further, even if the number of training points were to match that of a simplex, or of a simplex plus one extra point, one would have to determine the optimum radius of the simplex, which would necessitate a numerical search. Finally, in most cases, not all the model parameters have Gaussian priors, and if they were Gaussian, they might not have identical spreads. Another such symmetry is that where points are placed at the corners of an N p a r s N_{\mathrm{pars}} N pars –dimensional hyper-cube. This would require 2 N p a r s 2^{N_{\mathrm{pars}}} 2 N pars points. Again, another point might be added to the center. This arrangement would seem relevant for either the case where the prior was purely Gaussian with a spherical symmetry, or if all the priors were uniform with the same widths.
Thus, unless one has a symmetry, and additionally if the number of training points happens to allow placement according to that symmetry, the placement of the points is difficult to predict. The radius of a training point is simply the point’s distance from the origin,
∣ θ ⃗ a ∣ = ( ∑ i < N p a r s θ a , i 2 ) 1 / 2 . (84) \lvert\vec{\theta}_{a}\rvert= \left(\sum_{i<N_{\mathrm{pars}}}\theta_{a,i}^{2}\right)^{1/2}. \tag*{(84)} ∣ θ a ∣ = i < N pars ∑ θ a , i 2 1/2 . (84)
For the case with N p a r s = 12 N_{\mathrm{pars}} = 12 N pars = 12 parameters, Figure 5 shows the radii of the various positions. For two cases, all the priors were Gaussian, with an r.m.s. radius of 1 / 3 1/\sqrt{3} 1/ 3 . The number of training points was set to N t r a i n = 91 N_{\mathrm{train}} = 91 N train = 91 , which corresponds to the quadratic-fit threshold for N p a r s = 12 N_{\mathrm{pars}} = 12 N pars = 12 . As can be seen the optimized placement is for most of the points to be a nearly the same radius, whereas one point is placed near the origin. When this same example was worked out for N p a r s = 6 N_{\mathrm{pars}} = 6 N pars = 6 , one point was at the center, while the others were all positioned at exactly the same radius, but for N p a r s = 12 N_{\mathrm{pars}} = 12 N pars = 12 there was some small variation in the radii of the points away from the center. Two values of Λ \Lambda Λ were considered in Figure 5 , Λ = 2.5 \Lambda= 2.5 Λ = 2.5 and Λ = 4.0 \Lambda= 4.0 Λ = 4.0 . One can see that the radii moved outward for larger Λ \Lambda Λ . Figure 5 also shows the radii for 12 model parameters where each parameter had a uniform prior with minimum and maximum values of − 1 -1 − 1 and + 1 +1 + 1 . That distribution is at somewhat smaller radius, and without a point placed at the origin.
Figure 5. The distribution of the 91 optimized radii, ∣ θ ⃗ a ∣ |\vec{\theta}_a| ∣ θ a ∣ , for the optimized placement of 91 training points in the 12-dimensional model-parameter space. For this case N t r a i n = 91 N_{\mathrm{train}} = 91 N train = 91 . If all model parameters had Gaussian priors with the same width relative to Λ \Lambda Λ , the radii tend to organize with one point in the center while the others have similar radius spread over different directions. For larger Λ \Lambda Λ (Λ = 4 \Lambda= 4 Λ = 4 , green) larger radii were preferred compared to smaller Λ \Lambda Λ (Λ = 2.5 \Lambda= 2.5 Λ = 2.5 , red). For uniform priors (Λ = 2.5 \Lambda= 2.5 Λ = 2.5 , blue) radii were smaller with a similarly broad distribution, and there was not one training point in the middle.
Overcoming the Pernicious Nature of High Dimensions
The overarching goal of this manuscript is to understand how to best emulate functions that can be described well with polynomials not extending to high order, or equivalently, are well described by Taylor expansions. In Sec. II it was described how this can equivalently be posed in terms of Gaussian process approaches by requiring that the correlation length is larger than the size of the prior. Either way, emulation depended on defining a characteristic scale, Λ \Lambda Λ . For smaller Λ \Lambda Λ , more training points are needed. In high dimensionality parameter spaces, it becomes more difficult to effectively cover the prior. In Figure 1 , it was shown that even if the number of training points increased with number of quadratic coefficients, the accuracy of the emulator still fell as a function of N p a r s N_{\mathrm{pars}} N pars . The discussion here aims to provide several perspectives from which can appreciate the challenge of building high-dimensional emulators.
As discussed earlier and shown with Eqs. (82) and (83) , the number of polynomial terms of order n + 1 n+1 n + 1 exceeds the number of terms of order n n n by a factor proportional to the dimensionality of the parameter space, N p a r s N_{\mathrm{pars}} N pars . The variance due to the terms of order n + 1 n+1 n + 1 compared to those of order n n n roughly grows as N p a r s / Λ 2 N_{\mathrm{pars}}/\Lambda^2 N pars / Λ 2 . One can roughly state that the value of Λ \Lambda Λ that is necessary to give a certain accuracy should increase as N p a r s \sqrt{N_{\mathrm{pars}}} N pars . Thus, accurate emulation will be more difficult in higher dimensions as illustrated in Figure 1 . Another perspective from which one can gain the same conclusion is to consider the characteristic size of the parameter space. The characteristic r.m.s. sizes increase as N p a r s 1 / 2 N_{\mathrm{pars}}^{1/2} N pars 1/2 . The unconstrained covariance behaves as e − ( ∣ θ ⃗ i − θ ⃗ j ∣ ) 2 / 2 Λ 2 e^{-(|\vec{\theta}_i-\vec{\theta}_j|)^2/2\Lambda^2} e − ( ∣ θ i − θ j ∣ ) 2 /2 Λ 2 , so to maintain the same characteristic correlations, Λ 2 \Lambda^2 Λ 2 must be increased proportional to N p a r s N_{\mathrm{pars}} N pars , or again, the Λ \Lambda Λ must increase as N p a r s \sqrt{N_{\mathrm{pars}}} N pars .
In higher dimensions it is increasingly critical that the convergence parameter be longer than the system. Otherwise, one would need to sample the function in each corner of the model-parameter phase space. For uniform priors, there are 2 N p a r s 2^{N_{\mathrm{pars}}} 2 N pars corners to the phase space. For 12 dimensions this requires 4096 training points, and for 18 dimensions, 262,144 training points. Thus, it can be untenable to place training points in each corner of the space.
One can also see the problem with placing the training points in a spherically symmetric fashion. The hyper-volume of the parameter-space hyper-cube is 2 N 2^N 2 N , whereas the volume of an N N N -dimensional hyper-sphere of radius R = 1 R = 1 R = 1 is
V s p h e r e = Ω N ∫ 0 R d r r N − 1 = Ω N R N N . (85) V_{\mathrm{sphere}} = \Omega_N \int_0^R dr\,r^{N-1} = \Omega_N \frac{R^N}{N}. \tag*{(85)} V sphere = Ω N ∫ 0 R d r r N − 1 = Ω N N R N . (85)
The solid angle, Ω N \Omega_N Ω N in N N N dimensions is
Ω N = 2 π N / 2 Γ ( N / 2 ) (86) \Omega_N = \frac{2\pi^{N/2}}{\Gamma(N/2)} \tag*{(86)} Ω N = Γ ( N /2 ) 2 π N /2 (86)
and after putting this together, the fraction of the hyper-cube’s volume that is within the hyper-sphere is
V s p h e r e V c u b e = { ( π / 2 ) N / 2 N ! ! , N = 2 , 4 , 6 , 8 ⋯ ( π / 2 ) ( N − 1 ) / 2 N ! ! , N = 3 , 5 , 7 , ⋯ (87) \frac{V_{\mathrm{sphere}}}{V_{\mathrm{cube}}} =
\begin{cases}
\frac{(\pi/2)^{N/2}}{N!!}, & N = 2, 4, 6, 8 \cdots\\
\frac{(\pi/2)^{(N-1)/2}}{N!!}, & N = 3, 5, 7, \cdots
\end{cases} \tag*{(87)} V cube V sphere = { N !! ( π /2 ) N /2 , N !! ( π /2 ) ( N − 1 ) /2 , N = 2 , 4 , 6 , 8 ⋯ N = 3 , 5 , 7 , ⋯ (87)
In two dimensions, the ratio is π / 4 \pi/4 π /4 , and in three dimensions it is π / 6 \pi/6 π /6 . In 10 dimensions it is 2.5 × 10 − 3 2.5 \times10^{-3} 2.5 × 1 0 − 3 . For high dimensions only a small fraction of the parameter space can ever lie inside a sphere used to place points. And, if the model is expensive, it may not be tenable to run the full model inside every corner.
All the perspectives above emphasize the same conclusion. The only way a high-dimensional space can be well emulated is if the behavior is highly convergent. The parameter Λ \Lambda Λ must be large compared to the characteristic size, which is N p a r s / 3 \sqrt{N_{\mathrm{pars}}/3} N pars /3 . For 12 parameters this suggests Λ > 2 \Lambda> 2 Λ > 2 , and for 18 parameters this suggests Λ > 2.45 \Lambda> 2.45 Λ > 2.45 . If Λ \Lambda Λ is near or below this limit, emulation may require a large number of training points.
For large numbers of model parameters it is crucial that one pause to consider whether some model parameters are more or less important. As stated in the beginning of Sec. II this can be accomplished by scaling the model parameters. The model parameters with uniform priors can be scaled such that their priors have defined by θ max , i = s i \theta_{\max,i}=s_i θ m a x , i = s i , where s i < 1 s_i < 1 s i < 1 can be thought of as a suppression factor. The r.m.s. of the prior is then s i / 3 s_i/\sqrt{3} s i / 3 . The parameters with Gaussian priors can be scaled so the r.m.s. of their variance is s i / 3 s_i/\sqrt{3} s i / 3 . Scaling the range of the priors by s i s_i s i is mathematically equivalent to scaling the convergence parameters, i.e. setting Λ i = Λ / s i \Lambda_i=\Lambda/s_i Λ i = Λ/ s i . Scaling the priors, rather than scaling the convergence parameters, allows emulation to be simpler, as there is then only one convergence parameter. One could scale the parameters so that the most important parameters had θ max , i = 1 \theta_{\max,i}=1 θ m a x , i = 1 , or β i = 3 \beta_i=3 β i = 3 . One would set Λ \Lambda Λ to a value appropriate for those parameters, then scale θ max , i \theta_{\max,i} θ m a x , i and β i \beta_i β i for the remaining parameters to account for the suppression of the expected sensitivity of y ( θ ⃗ ) y(\vec{\theta}) y ( θ ) to the other parameters.
If warranted, assigning suppression factors could be an effective means of dealing with high dimensionality. For example if Λ = 2 \Lambda=2 Λ = 2 , it would be difficult to accurately emulate a 12-dimensional parameter space. However, if all but a couple of parameters were suppressed with s i = 1 / 2 s_i=1/2 s i = 1/2 , the less critical parameters would not contribute much beyond their linear contribution. For example, if only three parameters were given full strength, then training-point placement might optimize so that the 2 3 = 8 2^3=8 2 3 = 8 corners of the three-dimensional space were covered. This is much easier than attempting to cover 2 12 = 4096 2^{12}=4096 2 12 = 4096 corners of parameter space. The software developed for this study was specifically designed to allow the user to assign suppression factors to each model parameter.
One might ask whether certain parameters might be deemed less important after the training data is created. That is certainly possible, and the emulator’s uncertainty would fall whenever one would reduce s i s_i s i . However, assigning s i ≠ 1 s_i \ne1 s i = 1 also affects the optimized choice of training-point location. Thus, if the reduced importance is understood before the training data is generated that understanding can be exploited in the optimization of the training-point placement.
Testing the Emulator and Training Point Optimization
Overview of Testing
For this study software was developed according to the methods described in the previous sections. The first piece of software generated a list of optimized training points as described in Sec. III. The input for this code is a list of model parameters and their priors, plus an estimate of the convergence parameter, Λ \Lambda Λ , and the point-by-point noise parameter α \alpha α . Each model parameter was assumed to have an independent prior, either a Gaussian form with a given centroid and width, or a uniform distribution between some minimum and maximum value. These parameters were then translated and scaled so that their priors were centered around zero. By default, the widths were set according to the definitions in Eqs. (2) and (3) so that the priors would have the same variance as a uniform distribution with − 1 < θ < 1 -1 < \theta< 1 − 1 < θ < 1 even if the prior distribution is Gaussian. An option was added to allow the width for a given parameter, θ i \theta_i θ i , to be scaled down by a factor s i s_i s i if one believed a certain direction in parameter space was expected to contribute relatively less to the overall variance of the function. The software then provides a list of training points, θ ⃗ a \vec{\theta}_a θ a , with a = 1 − N t r a i n a=1-N_{\mathrm{train}} a = 1 − N train . Some tests of this procedure were provided in the previous section.
The second piece of software was the emulator discussed in Sec. II. The emulator assumed the forms for Y ( θ ⃗ ) Y(\vec{\theta}) Y ( θ ) and its uncertainty as described in Eq. (67) . This includes the contribution to the emulator uncertainty from the uncertainty in knowing Λ \Lambda Λ . The goal of this section is to study and test the emulator, and the theoretical foundations of both the emulator and the training-point optimization. The emulator’s mean prediction, Y ( θ ⃗ ) Y(\vec{\theta}) Y ( θ ) , is exactly that of a typical GP emulator with a Gaussian kernel, despite the fact that the form was generated from a different perspective than the usual presentation of GP emulators. Thus, the tests here will focus on the following questions.
If a model is built based on a polynomial expansion with specific values of Λ \Lambda Λ and σ A \sigma_A σ A , how well will the tuned emulator reproduce those values?
In the limits of Λ ≳ 2 \Lambda\gtrsim2 Λ ≳ 2 , can one not only reproduce a model, but can it also provide a meaningful estimate of its uncertainty?
Is the contribution to the emulator uncertainty from the uncertainty in Λ \Lambda Λ significant?
Did the estimate of the emulator’s uncertainty averaged over the prior, E P ( σ Y 2 ) / σ A 2 \mathop{\mathbf{E}}\nolimits_P(\sigma_Y^2)/\sigma_A^2 E P ( σ Y 2 ) / σ A 2 , from the training point selection before any model runs were processed, match that of the tuned emulator?
A subsection is devoted to each of the four questions above.
In order to test the models, two model types were created. The first type, Type-A, is based on creating polynomials with random coefficients, where the variance of the coefficients is consistent with the assumed form of the emulator for assumed values of σ A \sigma_A σ A and Λ \Lambda Λ . The second, Type-B, is a sum of randomized trigonometric and exponential functions. Test functions were thus generated of both types, and for each function an emulator was constructed. By creating thousands of such models, and comparing emulators trained upon them, one can test how accurately the emulator represents its uncertainty. One can also test whether the estimates of accuracy used for the optimization of training-point placement are consistent with the accuracy actually exhibited by the trained emulator.
Type-A Model
The form of the expansion is the same as what was assumed when designing the emulator:
F ( θ ⃗ ) = e − ∣ θ ⃗ ∣ 2 / 2 Λ 2 ∑ n ⃗ A n ⃗ n 1 ! n 2 ! ⋯ n N p a r s ! ( θ 1 Λ ) n 1 ( θ 2 Λ ) n 2 ⋯ ( θ N p a r s Λ ) n N p a r s . (88) F(\vec{\theta}) = e^{-|\vec{\theta}|^2/2\Lambda^2}\sum_{\vec{n}}\frac{A_{\vec{n}}}{\sqrt{n_1!n_2!\cdots n_{N_{\mathrm{pars}}}!}}\left(\frac{\theta_1}{\Lambda}\right)^{n_1}\left(\frac{\theta_2}{\Lambda}\right)^{n_2}\cdots\left(\frac{\theta_{N_{\mathrm{pars}}}}{\Lambda}\right)^{n_{N_{\mathrm{pars}}}} . \tag*{(88)} F ( θ ) = e − ∣ θ ∣ 2 /2 Λ 2 n ∑ n 1 ! n 2 ! ⋯ n N pars ! A n ( Λ θ 1 ) n 1 ( Λ θ 2 ) n 2 ⋯ ( Λ θ N pars ) n N pars . (88)
Here, the coefficients A n ⃗ A_{\vec{n}} A n are assumed to have a Gaussian distribution with variance σ A 2 \sigma_A^2 σ A 2 . To create the model, one chooses the dimensionality, N p a r s N_{\mathrm{pars}} N pars , the variance σ A 2 \sigma_A^2 σ A 2 and the convergence parameter Λ \Lambda Λ . The coefficients A n ⃗ A_{\vec{n}} A n are then randomly set according to a Gaussian distribution, with the maximum order set to K max = 5 K_{\max}=5 K m a x = 5 . Because this is the same form assumed by the emulator, aside from limiting K ≤ K max K\leq K_{\max} K ≤ K m a x , one can test whether the emulator, Y ( θ ⃗ ) Y(\vec{\theta}) Y ( θ ) , will choose optimum values of the hyper-parameters, σ A \sigma_A σ A and Λ \Lambda Λ , that match those used to create the full model, F ( θ ⃗ ) F(\vec{\theta}) F ( θ ) . A different choice for the coefficients represented a different model. For this study thousands of Type-A models were considered.
Type-B Model
The second functional form was constructed by summing over randomly generated sums of trigonometric and hyperbolic functions. The observable was a sum of six functions, each constructed from six model parameters.
F ( θ ⃗ ) = D 0 cos ( t 0 / Λ 0 ) + D 1 sin ( t 1 / Λ 1 ) + D 2 tan − 1 ( t 2 / Λ 2 ) + D 3 cosh ( t 3 / Λ 3 ) + D 4 sinh ( t 3 / Λ 4 ) + D 5 tanh − 1 ( s 5 ) (89) F(\vec{\theta}) = D_0\cos(t_0/\Lambda_0)+D_1\sin(t_1/\Lambda_1)+D_2\tan^{-1}(t_2/\Lambda_2)+D_3\cosh(t_3/\Lambda_3)+D_4\sinh(t_3/\Lambda_4)+D_5\tanh^{-1}(s_5) \tag*{(89)} F ( θ ) = D 0 cos ( t 0 / Λ 0 ) + D 1 sin ( t 1 / Λ 1 ) + D 2 tan − 1 ( t 2 / Λ 2 ) + D 3 cosh ( t 3 / Λ 3 ) + D 4 sinh ( t 3 / Λ 4 ) + D 5 tanh − 1 ( s 5 ) (89)
s 5 ≡ t 5 Λ 5 2 + t 5 2 , t i ≡ ∑ j r i j θ j / 6 + r i j k θ k θ j / λ j k . (90) s_5 \equiv\frac{t_5}{\sqrt{\Lambda_5^2+t_5^2}},
\qquad
t_i \equiv\sum_j r_{ij}\theta_j/\sqrt{6}+r_{ijk}\theta_k\theta_j/\lambda_{jk}. \tag*{(90)} s 5 ≡ Λ 5 2 + t 5 2 t 5 , t i ≡ j ∑ r ij θ j / 6 + r ij k θ k θ j / λ j k . (90)
Here, r i r_i r i and r i j r_{ij} r ij and r i j k r_{ijk} r ij k are random numbers taken from a uniform distribution between ± 1 \pm1 ± 1 . The values Λ i \Lambda_i Λ i and λ j k \lambda_{jk} λ j k were random numbers between 2.0 and 5.0. Each of the coefficients D i D_i D i were randomly chosen according to a Gaussian distribution with the same variance. Again, thousands of such models were generated to perform tests.
Choosing the Hyper-Parameters
In order to test the algorithms from Sec. (II) for choosing the hyper parameters, σ A \sigma_A σ A and Λ \Lambda Λ , 10 5 10^5 1 0 5 Type-A models were constructed. Each model was constructed using σ A = 100 \sigma_A = 100 σ A = 100 and Λ = 3 \Lambda= 3 Λ = 3 . Because the type-A models use the form assumed by the emulator, one might expect the algorithm to choose values of σ A \sigma_A σ A and Λ \Lambda Λ that match the values used to generate the models. There should be some fluctuation due to the coefficients for the models having been chosen with random distributions. The distribution of extracted hyper parameters for the 10 5 10^5 1 0 5 models is displayed in Figure 6 .
Figure 6. 10 5 10^5 1 0 5 models were constructed based on a expansion with random coefficients chosen according to the form assumed by the emulator using Λ = 3 \Lambda= 3 Λ = 3 and σ A = 100 \sigma_A = 100 σ A = 100 . The hyper parameters chosen by the tuning methods from Sec. II F are compared to the values used to build the model (red stars). For each model the extracted hyper-parameters varied significantly, but if one averages the extracted hyper-parameters over the 10 5 10^5 1 0 5 models, the average values come within a percent of Λ = 3 \Lambda= 3 Λ = 3 and σ A = 100 \sigma_A = 100 σ A = 100 . A strong band structure was found in that if the extracted value of σ A \sigma_A σ A was low, the value of Λ \Lambda Λ also tended to be low. Both panels exhibit some bimodal behavior in the distribution.
Distributions were found for two cases, one with 6-parameter models, and one with 12-parameter models. For the 6-parameter models, the emulators were trained at 28 points in parameter space, and for the 12-parameter model tuning involved 91 training points. These numbers correspond to the number coefficients that have K max ≤ 2 K_{\max} \le2 K m a x ≤ 2 . On average, the hyper parameters well matched those of the model (shown by the red stars), σ A = 100 \sigma_A = 100 σ A = 100 and Λ = 3 \Lambda= 3 Λ = 3 . Averaging over the distribution, the extracted hyper parameters matched those used to generate the model to within 1%. However, the distribution was rather wide, with widths of a few tens of percent. A strong correlation was observed between the extracted values of σ A \sigma_A σ A and Λ \Lambda Λ , as one can see a strong band structure. Larger Λ \Lambda Λ is highly correlated with larger σ A \sigma_A σ A . For both cases, the distributions exhibit some bimodal behavior. Being that the emulator uncertainty is proportional to σ A \sigma_A σ A and falls with increasing Λ \Lambda Λ , the overall uncertainty did not vary greatly from one instance to another despite the rather broad range of the distribution. The variance of the extracted hyper parameters decreased for the case with larger numbers of model parameters.
If one extracts small values of Λ \Lambda Λ , the expansion is poorly converging. For that reason, whenever the extracted value fell below Λ = N p a r s / 3 \Lambda= \sqrt{N_{\rm pars}/3} Λ = N pars /3 , the emulator would set Λ \Lambda Λ to that value. This was a rare occurrence for this case. A second issue can arise for large Λ \Lambda Λ , especially for larger numbers of parameters. The tuning procedure involves inverting the matrix B B B and finding its determinant. The form for B B B is
B a b = C 0 ( θ ⃗ a , θ ⃗ b ) + Δ a b , Δ a b = α 2 δ a b . (91) \begin{aligned}
B_{ab} &= C_0(\vec{\theta}_a,\vec{\theta}_b) + \Delta_{ab},\\
\Delta_{ab} &= \alpha^2\delta_{ab}. \tag*{(91)}
\end{aligned} B ab Δ ab = C 0 ( θ a , θ b ) + Δ ab , = α 2 δ ab . (91)
The term with n ⃗ = 0 \vec{n} = 0 n = 0 provides a contribution of unity to each element of the N p a r s × N p a r s N_{\rm pars} \times N_{\rm pars} N pars × N pars matrix. Thus, ignoring the other contributions, one would have B a b = 1 B_{ab} = 1 B ab = 1 , and B B B would be highly singular. Contributions from other n ⃗ \vec{n} n behave as Λ − ∣ n ⃗ ∣ \Lambda^{-|\vec{n}|} Λ − ∣ n ∣ , and keep the matrix from being singular, but numerically, overcoming the nearness to the singularity is more difficult for large Λ \Lambda Λ or for higher dimensionality of the N p a r s × N p a r s N_{\rm pars} \times N_{\rm pars} N pars × N pars matrix. Further, some choices of training points can accidentally lead to singular, or nearly singular, matrices. Fortunately, adding point-by-point noise, α ≠ 0 \alpha\ne0 α = 0 , also eliminates the singularity. The same issues can plague GP emulators. In that case the noise term above is referred to as a “nugget” [44 ] .
Predicting The Uncertainty
To address questions (2) and (3) from the beginning of this section, both Type-A and Type-B models were applied. For each model emulators were trained from model values at the training points. The training points were chosen according to method outlined in Sec. III. For this study, the parameter space was 12-dimensional with Gaussian priors. The training involved evaluating the model at 91 optimized points, 91 being the threshold for a quadratic fit. Figure 7 presents this comparison for 500 of these points from a variety of different models, and for a variety of random points in model-parameter space. The emulator’s uncertainty for each point is also displayed. The percentage of emulations within one unit of its stated uncertainty should be approximately 68.3% for Gaussian errors. For the Type-A models the success rate was indeed 68%, while for the random assorted functions, the success rate was 88%.
Figure 7. Emulator predictions and their uncertainties (red circles) are compared to model calculations (black squares) at 500 random points in parameter space. Models involved 12 parameters and were trained at 91 points. For the left-side panel, the model was a polynomial with random coefficients, type-A, chosen with the emulator’s assumed form with Λ = 3 \Lambda= 3 Λ = 3 , while in the right-side panel the model is a sum of random assorted trigonometric and exponential functions, type-B as described by Eq. (89) .
When calculating the uncertainties at the 500 random points used to generate Figure 7 , the contribution to the uncertainty from the uncertainty in determining Λ \Lambda Λ was included. That contribution, related to term proportional
to W Λ Λ W_{\Lambda\Lambda} W ΛΛ in Eq. (67) was rarely more than a few percent of the total uncertainty. This was consistent with the expectations for that contribution, which were calculated when the choice of training points was being optimized.
Testing the Preconstrained Estimates of the Uncertainty
While choosing the training points, and before any full-model runs were performed, estimates were made of the expected emulator’s uncertainty averaged over the prior. Figure 9 illustrates the degree to which that prediction was successful. Ten thousand models of type-A were randomly generated using N p a r s = 12 N_{\mathrm{pars}} = 12 N pars = 12 with Gaussian priors. The coefficients were random but chosen with weights assuming Λ = 3 \Lambda= 3 Λ = 3 . The distribution’s average indeed matched the predicted value, but the width of the distribution is remarkably wide. The bimodality seen in Figure 6 is also manifest here. An island of fits with smaller σ A \sigma_A σ A and smaller Λ \Lambda Λ in Figure 6 is seen in Figure 9 as an additional island of values of larger ( σ Y / σ A ) 2 (\sigma_Y/\sigma_A)^2 ( σ Y / σ A ) 2 . Even though these models tended to have larger values of σ Y / σ A \sigma_Y/\sigma_A σ Y / σ A , their overall uncertainties were similar to those from the bulk of the distribution because σ A \sigma_A σ A tended to be smaller. Hence, the distribution of accurate predictions in Figure 8 was not particularly wide despite the widths of the distributions in Figure 6 and Figure 9 .
Figure 8. Ten thousand models were created, each a polynomial expansion with random coefficients chosen to match the form assumed by the emulator. After the emulator was tuned, each model was evaluated at 1000 random points within the prior and compared to the emulator prediction. For each model the percentage was found for the prediction being within ± σ Y \pm\sigma_Y ± σ Y . If the emulator were to perfectly estimate its uncertainty, the distribution would be peaked around 68.3%. For the models of type A the emulator tended to predict its uncertainty rather well, whereas for type B the emulator tended to modestly overpredict its uncertainty.
Figure 9. When optimizing the choice of training points, the procedure produced estimates of the prior-averaged uncertainty relative to σ A 2 \sigma_A^2 σ A 2 . That value is represented by the green dashed line. From tuning 10,000 randomly generated type-A models with N p a r s = 12 N_{\mathrm{pars}} = 12 N pars = 12 , a distribution of the accuracy metrics, E P ( σ Y 2 ) / σ A 2 E_P(\sigma_Y^2)/\sigma_A^2 E P ( σ Y 2 ) / σ A 2 , is displayed here. The distribution is remarkably wide, but the average indeed matches the description. The large width mainly comes from the fact that the value of Λ \Lambda Λ varies, and that the estimated uncertainty is strongly sensitive to Λ \Lambda Λ .
Summary
The goal of this study was to consider the emulation of functions which are expected to be well described by polynomials with a few orders in the expansion. From this perspective, questions were addressed about what forms of the emulator one should use, how one should choose the training points, and whether one can trust the emulator’s statement of its uncertainty. In particular, the motivation is driven by the need to consider models with high-dimensional parameter spaces that, due to their numerical cost, should be trained with optimum efficiency.
In Sec. II it was found that if the following criteria were set, then the form of the emulator would be that of a Gaussian process with a Gaussian kernel. The criteria were:
The function should be analytic throughout the prior.
At any point the function should fluctuate according to a Gaussian distribution.
The prior-to-constraint covariance should depend only on ∣ θ ⃗ 1 − θ ⃗ 2 ∣ 2 \lvert\vec{\theta}_1-\vec{\theta}_2\rvert^2 ∣ θ 1 − θ 2 ∣ 2 .
The polynomial form corresponding to these criteria was found to be:
Y ( θ ⃗ ) = ∑ n ⃗ N c o e f f A n ⃗ T n ⃗ ( θ ⃗ ) , (92) Y(\vec{\theta}) = \sum_{\vec{n}}^{N_{\mathrm{coeff}}} A_{\vec{n}} \mathcal{T}_{\vec{n}}(\vec{\theta}), \tag*{(92)} Y ( θ ) = n ∑ N coeff A n T n ( θ ) , (92)
T n ⃗ ( θ ⃗ ) = 1 n 1 ! n 2 ! ⋯ n N p a r s ! ( θ 1 Λ ) n 1 ( θ 2 Λ ) n 2 ⋯ ( θ N Λ ) n N p a r s e − ∣ θ ⃗ ∣ 2 / 2 Λ 2 . \mathcal{T}_{\vec{n}}(\vec{\theta}) = \frac{1}{\sqrt{n_1!n_2!\cdots n_{N_{\mathrm{pars}}}!}}\left(\frac{\theta_1}{\Lambda}\right)^{n_1}\left(\frac{\theta_2}{\Lambda}\right)^{n_2}\cdots\left(\frac{\theta_N}{\Lambda}\right)^{n_{N_{\mathrm{pars}}}}e^{-|\vec{\theta}|^2/2\Lambda^2}. T n ( θ ) = n 1 ! n 2 ! ⋯ n N pars ! 1 ( Λ θ 1 ) n 1 ( Λ θ 2 ) n 2 ⋯ ( Λ θ N ) n N pars e − ∣ θ ∣ 2 /2 Λ 2 .
Expressions were found for all expansion coefficients in terms of the training data. The actual coefficients were not needed given that the emulator predictions and uncertainties could all be expressed in terms of the unconstrained covariance,
Cov 0 ( Y ( θ ⃗ 1 ) , Y ( θ ⃗ 2 ) ) = σ A 2 e − ∣ θ ⃗ 1 − θ ⃗ 2 ∣ 2 / 2 Λ 2 . (93) \operatorname{Cov}_0(Y(\vec{\theta}_1),Y(\vec{\theta}_2))=\sigma_A^2e^{-|\vec{\theta}_1-\vec{\theta}_2|^2/2\Lambda^2}. \tag*{(93)} Cov 0 ( Y ( θ 1 ) , Y ( θ 2 )) = σ A 2 e − ∣ θ 1 − θ 2 ∣ 2 /2 Λ 2 . (93)
In practice, the emulator behaves exactly as a GP emulator with a Gaussian kernel, and with the convergence parameter, Λ \Lambda Λ , being the same as the correlation-length hyper-parameter in the GP paradigm. Whereas a large convergence parameter, Λ \Lambda Λ , relative to the prior size, forces the polynomial to converge rapidly, within the GP paradigm one can equivalently state that the correlation length, Λ \Lambda Λ , should well exceed the size of the prior.
The emulator’s uncertainty was shown to have two contributions. The first comes from the fact that the expansion coefficients, A n ⃗ A_{\vec{n}} A n , are constrained by the training but are still very much unknown given that there are typically many more expansion coefficients than training constraints. This first contribution is identical to that which is typically used for GP emulators. The second contribution comes from the fact that the hyper-parameters, σ A \sigma_A σ A and Λ \Lambda Λ , are not precisely known. Although there are expressions for the most likely values, the likelihood for varying σ \sigma σ and Λ \Lambda Λ is characterized by widths. Because the width W Λ Λ ≠ 0 W_{\Lambda\Lambda}\ne0 W ΛΛ = 0 and because the prediction can be sensitive to Λ \Lambda Λ , this leads to a second contribution to the emulator uncertainty. For cases with reasonably large numbers of training points, this second contribution turned out to be relatively small but played an outsized role in the optimization of training-point placement.
In Sec. III, a procedure was presented for optimizing the placement of training points. The metric for accuracy was the squared uncertainty averaged over the prior, E P ( σ Y 2 ) / σ A 2 \mathbb{E}_P(\sigma_Y^2)/\sigma_A^2 E P ( σ Y 2 ) / σ A 2 . Aside from the location of the training points, the accuracy metric also depended on a guess for what value of Λ \Lambda Λ might be found once the actual training was performed. The measure was expressed analytically for Gaussian or uniform priors. Even with a rather inefficient search procedure, optimized configurations of training points were readily found, although for very high dimensions the procedure might take hours. The optimization was driven mainly by the second source of uncertainty, that due to Λ \Lambda Λ varying. However, once an optimized placement was chosen, this second source of uncertainty was found to be rather negligible.
It was found that the optimized configuration would typically improve on simple Latin hyper-cube sampling by a factor of nearly two or more as quantified by the accuracy metric. This approach might thus reduce the number of full-model training runs by perhaps a third. The marginal improvement of an additional full-model run was found to fall when the number of training points passed thresholds related to the number of expansion coefficients of a given order. The lesson gained here is that it could be advantageous to choose a number of full-model runs that exactly correspond to the number of coefficients up to a certain order. For example, for N p a r s = 12 N_{\mathrm{pars}}=12 N pars = 12 , there are 91 coefficients of quadratic order or less. The expected accuracy improved significantly more while increasing the number of runs to 91, and improved less with additional runs.
The measure of expected accuracy, E P ( σ Y 2 ) / σ A 2 \mathbb{E}_P(\sigma_Y^2)/\sigma_A^2 E P ( σ Y 2 ) / σ A 2 , was found to be highly sensitive to the convergence parameter, Λ \Lambda Λ and to the dimensionality of the model-parameter space. Sensitivity to the point-by-point noise was small as long as the noise was small compared to the other variation of the model. Although the estimate of the accuracy depended strongly on Λ \Lambda Λ , the actual optimum locations of the training points changed only modestly with Λ \Lambda Λ . The sensitivity to the dimensionality is a prime lesson to be taken from this study. Even if the number of training points is increased to match the increased number of coefficient of a given order as N p a r s N_{\mathrm{pars}} N pars increases, the expected accuracy falls. The challenge of emulating high-dimension spaces was discussed in detail at the end of Sec. III.
To test the methods discussed above, two simple models were considered. The first was a model based on the expansion in Eq. (92) , using random expansion coefficients. This precisely coincides with the form assumed by the emulator. The second form was a sum of random trigonometric and exponential forms. Thousands of models were constructed for each class. The models were each then used to train and build emulators. Emulator predictions were then compared to model values at randomly selected points throughout the prior. For the first class of models the emulator provided an outstanding estimate of its accuracy. This was expected given that the model’s form matched that assumed by the emulator. For the case of randomized trigonometric and exponential functions, the emulator modestly understated its accuracy.
For the first type of model, one would expect the emulator’s value of σ A \sigma_A σ A and Λ \Lambda Λ to match those used to define the model. On average the extracted values matched well, but the distribution of extracted values was large. Despite the fact that σ A \sigma_A σ A and Λ \Lambda Λ might stray far from the values used to generate the functional data, and despite the fact that the ratio σ Y / σ A \sigma_Y/\sigma_A σ Y / σ A often strayed far from the predictions provided by the training-point-placement procedure, the emulators largely provided good estimates of their uncertainty. This was driven by the fact that extracted σ A \sigma_A σ A and Λ \Lambda Λ values were highly correlated. Thus, if the extracted σ A \sigma_A σ A was much higher than the value used to generate the training data, the extracted values of Λ \Lambda Λ were also quite high. The two taken together, then led to rather modestly varying uncertainties.
An important lesson to taken from this study is that high-dimension parameter spaces are difficult to emulate unless the functions are highly convergent. However, in most instances one expects many of the model parameters to contribute weakly. A strategy was provided in Sec. III D to exploit this fact by simply assigning suppression factors to some subset of model parameters. The suppression factors would allow the procedure to account for the relatively weaker sensitivity to some parameters by scaling their priors to narrower ranges, while assuming all parameters contributed according to a single convergence parameter.
Aside from the lessons gained from this study, the work points to places where the procedures could be improved or expanded. The main opportunity for improvement would be in coding a more efficient procedure for optimizing the training points. Steepest descent algorithms might accelerate the search by multiple factors of two. A primary opportunity for expanding the functionality would be to consider more classes of priors when optimizing the training-point placement. The current software considered Gaussian and uniform priors. One could consider half Gaussians, or even or odd functions.
Finally, one needs to ask the question as to whether one should perform full model runs simultaneously or sequentially. Here, by doing a significant number simultaneously, one can maximize the accuracy for that number of runs as measured over the entire prior. However, once the emulator is tuned well enough, one could perform a Markov Chain Mote Carlo (MCMC) exploration of space and identify those regions of the parameter space which are likely. One could then augment the training data with model runs specifically aimed to best constrain predictions for regions that are relevant to the goal of determining the model parameters. The additional points would likely be located in the likely regions of parameter space. Given that one has little concrete knowledge of where the likely region of parameter space lies before the training, it might make sense to simultaneously perform a first set of predictions to constrain the linear or quadratic behavior without comparing to experiment. Then, after determining the likely regions of parameter space by comparing to data, one might perform additional runs in a more sequential manner. Unfortunately, it is difficult to say whether the first set of runs should be small, e.g. only enough for a linear fit, or whether one might opt for enough for a quadratic fit. The decision might rest on numerous factors. This might include how convergent the functions were expected to be, their numerical cost, and whether one can easily perform some number of full-model runs simultaneously. Unfortunately, their convergence would not be well understood until after the first round of model runs. Choosing an optimized strategy for most cases should be quite beneficial and allow one to avoid performing unnecessary full-model runs.
Properties of the Polynomial Basis Functions
The polynomial expansion presented in Sec. II can be considered as an expansion in basis functions. The purpose of this appendix is to examine the orthonormality and completeness properties of those basis functions.
The polynomial expansion, for N = N p a r s N = N_{\mathrm{pars}} N = N pars , used for the studies here is of the form
Y ( θ ⃗ ) = e − ∣ θ ⃗ ∣ 2 / 2 Λ 2 ∑ n ⃗ A n ⃗ n 1 ! n 2 ! ⋯ n N ! ( θ 1 Λ ) n 1 ( θ 2 Λ ) n 2 ⋯ ( θ N Λ ) n N , (94) Y(\vec{\theta}) = e^{-|\vec{\theta}|^2/2\Lambda^2} \sum_{\vec{n}} \frac{A_{\vec{n}}}{\sqrt{n_1!n_2!\cdots n_N!}} \left(\frac{\theta_1}{\Lambda}\right)^{n_1} \left(\frac{\theta_2}{\Lambda}\right)^{n_2} \cdots\left(\frac{\theta_N}{\Lambda}\right)^{n_N}, \tag*{(94)} Y ( θ ) = e − ∣ θ ∣ 2 /2 Λ 2 n ∑ n 1 ! n 2 ! ⋯ n N ! A n ( Λ θ 1 ) n 1 ( Λ θ 2 ) n 2 ⋯ ( Λ θ N ) n N , (94)
where A n ⃗ A_{\vec{n}} A n were independent coefficients, each obeying a Gaussian distribution,
W ( A n ⃗ ) = 1 2 π σ A 2 e − A n ⃗ 2 / σ A 2 , (95) W(A_{\vec{n}}) = \frac{1}{\sqrt{2\pi\sigma_A^2}}e^{-A_{\vec{n}}^2/\sigma_A^2}, \tag*{(95)} W ( A n ) = 2 π σ A 2 1 e − A n 2 / σ A 2 , (95)
E 0 ( A m ⃗ , A n ⃗ ) = σ A 2 δ m ⃗ n ⃗ . E_0(A_{\vec{m}},A_{\vec{n}}) = \sigma_A^2\delta_{\vec{m}\vec{n}}. E 0 ( A m , A n ) = σ A 2 δ m n .
This can be considered as an expansion in terms of basis functions, ϕ n ⃗ \phi_{\vec{n}} ϕ n defined as
ϕ n ⃗ ( θ ⃗ ) = 1 ( 2 π Λ 2 ) N / 4 e − ∣ θ ⃗ ∣ 2 / 2 Λ 2 1 n 1 ! n 2 ! ⋯ n N ! ( θ 1 Λ ) n 1 ( θ 2 Λ ) n 2 ⋯ ( θ N Λ ) n N . (96) \phi_{\vec{n}}(\vec{\theta}) = \frac{1}{(2\pi\Lambda^2)^{N/4}}e^{-|\vec{\theta}|^2/2\Lambda^2}\frac{1}{\sqrt{n_1!n_2!\cdots n_N!}}\left(\frac{\theta_1}{\Lambda}\right)^{n_1}\left(\frac{\theta_2}{\Lambda}\right)^{n_2}\cdots\left(\frac{\theta_N}{\Lambda}\right)^{n_N}. \tag*{(96)} ϕ n ( θ ) = ( 2 π Λ 2 ) N /4 1 e − ∣ θ ∣ 2 /2 Λ 2 n 1 ! n 2 ! ⋯ n N ! 1 ( Λ θ 1 ) n 1 ( Λ θ 2 ) n 2 ⋯ ( Λ θ N ) n N . (96)
Here, the basis functions are labeled by n ⃗ \vec{n} n , which is a list of any N N N non-negative integers. With this definition the emulator is
Y ( θ ⃗ ) = ( 2 π Λ 2 ) N / 4 ∑ n ⃗ A n ⃗ ϕ n ⃗ ( θ ⃗ ) . (97) Y(\vec{\theta}) = (2\pi\Lambda^2)^{N/4} \sum_{\vec{n}} A_{\vec{n}}\phi_{\vec{n}}(\vec{\theta}). \tag*{(97)} Y ( θ ) = ( 2 π Λ 2 ) N /4 n ∑ A n ϕ n ( θ ) . (97)
The prefactor, 1 / ( 2 π Λ 2 ) N / 4 1/(2\pi\Lambda^2)^{N/4} 1/ ( 2 π Λ 2 ) N /4 , is defined for convenience.
First, the orthogonality and normalization properties are:
Z m ⃗ n ⃗ = ∫ d θ 1 ⋯ d θ N ϕ m ⃗ ( θ ⃗ ) ϕ n ⃗ ( θ ⃗ ) = ∏ j = 1 N z m j n j , z m j n j = 1 ( 2 π Λ 2 ) 1 / 2 1 m j ! n j ! ∫ d θ e − θ 2 / Λ 2 ( θ Λ ) m j + n j = { 1 ( 2 π ) 1 / 2 1 m j ! n j ! Γ ( [ m j + n j + 1 ] / 2 ) , m j + n j is even 0 , m j + n j is odd . (98) \begin{aligned}
Z_{\vec{m}\vec{n}} &= \int d\theta_1 \cdots d\theta_N \, \phi_{\vec{m}}(\vec{\theta})\phi_{\vec{n}}(\vec{\theta}) \\
&= \prod_{j=1}^{N} z_{m_jn_j}, \\
z_{m_jn_j} &= \frac{1}{(2\pi\Lambda^2)^{1/2}}\frac{1}{\sqrt{m_j!n_j!}}\int d\theta\, e^{-\theta^2/\Lambda^2}\left(\frac{\theta}{\Lambda}\right)^{m_j+n_j} \\
&= \begin{cases}
\frac{1}{(2\pi)^{1/2}}\frac{1}{\sqrt{m_j!n_j!}}\Gamma([m_j+n_j+1]/2), & m_j+n_j\text{ is even} \\
0, & m_j+n_j\text{ is odd}.
\end{cases} \tag*{(98)}
\end{aligned} Z m n z m j n j = ∫ d θ 1 ⋯ d θ N ϕ m ( θ ) ϕ n ( θ ) = j = 1 ∏ N z m j n j , = ( 2 π Λ 2 ) 1/2 1 m j ! n j ! 1 ∫ d θ e − θ 2 / Λ 2 ( Λ θ ) m j + n j = { ( 2 π ) 1/2 1 m j ! n j ! 1 Γ ([ m j + n j + 1 ] /2 ) , 0 , m j + n j is even m j + n j is odd . (98)
The basis functions are neither normalized nor orthogonal, although the normalizations are of order unity. For the purposes of emulation it is not clear that orthonormality properties play an important role.
On the other hand, understanding completeness determines whether the basis can capture all facets of the function it is attempting to represent. The completeness properties are defined by
∑ n ⃗ ϕ n ⃗ ( θ ⃗ a ) ϕ n ⃗ ( θ ⃗ b ) = 1 ( 2 π Λ 2 ) N / 2 e − ∣ θ ⃗ a ∣ 2 / 2 Λ 2 e − ∣ θ ⃗ b ∣ 2 / 2 Λ 2 × ∑ n ⃗ 1 n 1 ! ⋯ n N ! ( θ a 1 θ b 1 Λ 2 n 1 ) ⋯ ( θ a N θ b N Λ 2 n N ) = 1 ( 2 π Λ 2 ) N / 2 e − ∣ θ ⃗ a − θ ⃗ b ∣ 2 / 2 Λ 2 . (99) \begin{aligned}
\sum_{\vec{n}} \phi_{\vec{n}}(\vec{\theta}_a)\phi_{\vec{n}}(\vec{\theta}_b)
&= \frac{1}{(2\pi\Lambda^2)^{N/2}}e^{-|\vec{\theta}_a|^2/2\Lambda^2}e^{-|\vec{\theta}_b|^2/2\Lambda^2} \\
&\quad\times\sum_{\vec{n}}\frac{1}{n_1!\cdots n_N!}
\left(\frac{\theta_{a1}\theta_{b1}}{\Lambda^{2n_1}}\right) \cdots
\left(\frac{\theta_{aN}\theta_{bN}}{\Lambda^{2n_N}}\right) \\
&= \frac{1}{(2\pi\Lambda^2)^{N/2}}e^{-|\vec{\theta}_a-\vec{\theta}_b|^2/2\Lambda^2}. \tag*{(99)}
\end{aligned} n ∑ ϕ n ( θ a ) ϕ n ( θ b ) = ( 2 π Λ 2 ) N /2 1 e − ∣ θ a ∣ 2 /2 Λ 2 e − ∣ θ b ∣ 2 /2 Λ 2 × n ∑ n 1 ! ⋯ n N ! 1 ( Λ 2 n 1 θ a 1 θ b 1 ) ⋯ ( Λ 2 n N θ a N θ b N ) = ( 2 π Λ 2 ) N /2 1 e − ∣ θ a − θ b ∣ 2 /2 Λ 2 . (99)
For a complete basis the right-hand side would have been a delta function. Instead, it is a smeared delta function. It integrates to unity but has a non-zero width, Λ \Lambda Λ . This would satisfy the completeness relation if Λ \Lambda Λ were allowed to approach zero, but in that limit one would likely require an infinite number of terms in the expansion.
This basis can then be considered as being incomplete in that it cannot well represent small scale features, i.e. those with characteristic scales below Λ \Lambda Λ . For perfect completeness to hold over an infinite volume one would expect to require a continuous set of coefficients that cover an infinite extent. A similar example would be a Fourier transform, where instead of the infinite sets of integers, n ⃗ \vec{n} n , one has the vector k ⃗ \vec{k} k which is distributed continuously. If one confined the vectors k ⃗ \vec{k} k to integer multiples of some step vectors, Δ k i \Delta k_i Δ k i , one would be unable to represent features below the inverse size of the step vectors, and if one cut off the expansion at some maximum, k max k_{\max} k m a x , one would be unable to describe structures of sizes below 1 / k max 1/k_{\max} 1/ k m a x . For the problem here, Λ \Lambda Λ sets the minimum scale at which one would expect features for the full function, and the maximum scale at which one can represent the function is determined by whether the sum over n ⃗ \vec{n} n is cut off at some maximum order.
Above, for a given rank K K K , the basis functions include independent terms for all n 1 + n 2 + ⋯ n N p a r s = K n_1+n_2+\cdots n_{N_{\mathrm{pars}}}=K n 1 + n 2 + ⋯ n N pars = K . One might have instead chosen to express the polynomials as all terms i 1 , i 2 , i K i_1,i_2,i_K i 1 , i 2 , i K , so that Y ( θ ⃗ ) Y(\vec{\theta}) Y ( θ ) would then be:
Y ( θ ⃗ ) = ∑ K , i 1 ⋯ i K A i 1 ⋯ i K 1 K ! θ i 1 θ i 2 ⋯ θ i K . (100) Y(\vec{\theta}) = \sum_{K,i_1\cdots i_K} A_{i_1\cdots i_K}\frac{1}{K!}\theta_{i_1}\theta_{i_2}\cdots\theta_{i_K}. \tag*{(100)} Y ( θ ) = K , i 1 ⋯ i K ∑ A i 1 ⋯ i K K ! 1 θ i 1 θ i 2 ⋯ θ i K . (100)
Here, 1 ≤ i j ≤ N p a r s 1\le i_j\le N_{\mathrm{pars}} 1 ≤ i j ≤ N pars . In this form, there are more coefficients. In the first form there was a term proportional to θ 1 θ 2 \theta_1\theta_2 θ 1 θ 2 , and in this form there are three separate terms with the same product, θ 1 θ 2 \theta_1\theta_2 θ 1 θ 2 , θ 2 θ 1 \theta_2\theta_1 θ 2 θ 1 and θ 1 θ 2 θ 1 \theta_1\theta_2\theta_1 θ 1 θ 2 θ 1 . This alternate form has redundant coefficients, but those coefficients have less strength. In the previous form N ! / ( n 1 ! n 2 ! n 3 … ) N!/(n_1!n_2!n_3\ldots) N ! / ( n 1 ! n 2 ! n 3 … ) was set to N ! / ( n 1 ! n 2 ! n 3 … ) N!/(n_1!n_2!n_3\ldots) N ! / ( n 1 ! n 2 ! n 3 … ) , which equals the number of identical terms contributing in the new form. Because having d d d independent gaussians is the same as having a single gaussian of strength d \sqrt{d} d , the end result is the same correlation. Thus, the alternate form in Eq. (100) leads to all the same results as that presented in Sec. (II).
Acknowledgments
This work was supported by the Department of Energy Office of Science through grant number DE-FG02-03ER41259 and by the National Science Foundation’s Cyberinfrastructure for Sustained Scientific Innovation program through grant number OAC-2004601. The authors express gratitude to Simon Mak and Daniel Phillips for generously sharing their insight into the subject.
References
(1979).