WorksComputer Science
Condensed Stein Variational Gradient Descent for Uncertainty Quantification of Neural Networks
Abstract
We propose a Stein variational gradient descent method to concurrently sparsify, train, and provide uncertainty quantification of a complexly parameterized model such as a neural network. It employs a graph reconciliation and condensation process to reduce complexity and increase similarity in the Stein ensemble of parameterizations. Therefore, the proposed condensed Stein variational gradient (cSVGD) method provides uncertainty quantification on parameters, not just outputs. Furthermore, the parameter reduction speeds up the convergence of the Stein gradient descent as it reduces the combinatorial complexity by aligning and differentiating the sensitivity to parameters. These properties are demonstrated with an illustrative example and an application to a representation problem in solid mechanics.
Introduction
In the context of uncertainty quantification (UQ) the curse of dimensionality, whereby quantification efficiency degrades drastistically with parameter dimension, is particular extreme with highly parameterized models such as neural networks (NNs). Fortunately, in many cases, these models are overparameterized in the sense that the number of parameters can be reduced with negligible effects on accuracy and sometimes improvements in generalization [1]. Furthermore, NNs often have parameterizations that have fungible parameters such that permutations of the values and connections lead to equivalent output responses. This suggests methods that simultaneously sparsify and characterize the uncertainty of a model, while handling and taking advantage of the symmetries inherent in the model, are potentially advantageous approaches.
Although Markov chain Monte Carlo (MCMC) methods [2] have been the reference standard to generate samples for UQ methods, they can be temperamental and do not scale well for high dimensional models. More recently, there has been widespread use of variational inference methods (VI), which cast the parameter posterior sampling problem as an optimization of a surrogate posterior guided by a suitable objective, such as the Kullback-Liebler (KL) divergence between the predictive posterior and true posterior induced by the data. In particular, there is now a family of model ensemble methods based on Stein’s identity [3], such as Stein variational gradient descent (SVGD) [4], projected SVGD [5], and Stein variational Newton’s method [6]. These methods have advantages over MCMC methods by virtue of propagating in parallel a coordinated ensemble of particles that represent the empirical posterior.
In our recent previous work [7], we proposed a sequential sparsification-uncertainty quantification method based on regularization and Stein gradient flow. We showed that sparsification by methods [8] was able to capture the nonlinear nature of the reduced parameter manifold, which subsequently was explored by a Stein ensemble of particles representing likely model parameterizations. The sequential application of sparsification and then SVGD displayed advantages over established methods [5] on a number of physical representation problems.
The concurrent sparsification-uncertainty quantification method presented in this work, termed cSVGD, has a number of advantages over our sequential method. Foremost, the new method enables parameter alignment and leads to more interpretable parameter UQ, versus solely providing output uncertainty estimates. This also enables the Stein method to avoid spurious repulsion or lack of repulsion due to parameter permutations. Furthermore, there is an improvement in computational speed due to the reduction in the complexity of the models in the Stein ensemble. Parameter UQ and computational speed-ups are enabled by reconciling the parameters by importance across the Stein ensemble and condensing the common representation of the model as the gradient flow and sparsification progress. The method has a firm theoretical foundation based on the introduction of sparsifying priors into the Stein variational gradient flow formalism [4].
In the next section, § 2, we give an overview of related work that we build upon. Then, in § 3, we develop the concurrent sparsifying Stein gradient descent (cSVGD) algorithm for feedforward neural networks (FFNNs). In § 4, we illustrate the general behavior of the algorithm, in particular, the interplay of the hyperparameters controlling the prior and the Stein kernel. In § 5, we demonstrate the performance of the method on a problem of scientific interest: determining a hyperelastic potential from stress-strain data. Finally, in § 6, we conclude with a summary of findings and avenues for future work.
Related work
To provide an efficient alternative to MCMC sampling Liu and Wang [4, 9] introduced Stein variational gradient descent (SVGD) which propagates a coordinated ensemble of parameter realizations that approximate the posterior distribution. Newton [6, 10] and projected [5] variants soon followed. Our previous work [7] combined sparsification and SVGD in sequence to fully and efficiently explore the submanifold of influential parameters. That work relied on , , regularizations that induced parameter sparsity, such as the work by Louizos et al. [8] which developed a smoothed regularization. Others, such as McCulloch et al. [11], have used this type of regularization in scientific machine learning. Another notable example of sparse parameter identification is the Sparse Identification of Nonlinear Dynamical systems (SINDy) framework [12]. In a probabilistic, Bayesian context these regularizations have analogs in sparsifying priors, such as the Cauchy prior introduced in Bayesian compressive sensing and Least Absolute Shrinkage and Selection Operator (LASSO) [13, 14, 15, 16].
Our augmentation of sparsifying priors in the Stein flow with model graph condensation to reduce complexity has some similarity to the well-established field of active set methods [17, 18, 19, 20]. These methods rely on transformations and/or heuristics to reduce the permutational complexity of identifying the active constraints in a large set of inequality constraints [20] or the optimum in a non-smooth problem [19].
Concurrent sparsification and Stein variational gradient descent
First, we develop Stein variational gradient descent (SVGD) with a sparsifying prior as the basis for the proposed algorithm. Then, we show how we achieve parameter alignment by treating the Stein ensemble as an ensemble of graphs with a common graph. Finally, we summarize the overall algorithm.
Bayesian parameter uncertainty quantification
Given a model with inputs , parameters , and data , where is the number of data samples; Bayes rule expressed as:
provides the posterior distribution on the parameters given data . The formulation is also dependent on choices for the likelihood and the prior distribution of parameters . These distributions determine the posterior and the evidence .
For example, with the assumption of a Gaussian likelihood
and a sparsifying prior
the negative log posterior becomes
where is the data noise variance and acts as a penalty parameter on the regularizating objective . Note the remaining additive terms in Eq. (4) do not depend on the parameters and, hence, do not affect the gradient of , otherwise known as the score. Here, and in the following, we use to denote
i.e. is related to the norm by . Figure 1 illustrates this family of posteriors; notice that as the prior approximates the prior.
Figure 1. Log posteriors formed from a Gaussian likelihood and priors from the exponential family scaled by . As is increased (top to bottom), the posterior resembles the prior mode more than the likelihood, and as is reduced (right to left), the contours of the prior transform from diamond-shaped to more cruciform-shaped. Note the values of are relative to the particular likelihood chosen for this illustration.
Specifically, we employ priors in the exponential family
where
and is the Gamma function.
Variational inference
As an alternative to sampling the posterior distribution with standard but computationally expensive Markov chain Monte Carlo (MCMC) methods [21], variational inference (VI) reformulates the task of approximating the posterior as an optimization problem on the parameters of a surrogate posterior. In VI, the optimization problem for the surrogate is typically based on the Kullback–Leibler (KL) divergence:
measuring the difference between the surrogate and true posterior associated with the data D. Given that appears in a logarithm in Eq. (8), the evidence does not need to be evaluated (since the it is a constant) in the search for the optimum .
The SVGD algorithm [4] is a VI method based on the Stein identity [3]
where
for a smooth function and a generic probability density . The Stein operator depends on the density only through the score function which is independent of the normalization of and thus avoids the evaluation of the evidence.
If we let be an approximate density under a perturbation mapping of the random variable , then the expectation of is related to the gradient of the Kullback–Leibler (KL) divergence between and [4 Thm 3.1]:
This provides the basis for a gradient-based method to update the approximate posterior density.
The SVGD algorithm utilizes an ensemble of particles in the parameter space. After randomly initializing the parameter ensemble, SVGD updates the particles via gradient-descent steps
where is a step size parameter, is the iteration counter, and
is the parameter gradient which depends on the choice of the kernel . Note the sum is over all particles and by definition. We will call the kernel smoothed posterior apparent in Eq. (13) the pseudopotential.
In the form of the gradient in Eq. (14) we see that the first term depends solely on the likelihood and kernel, while the second term depends only on the prior and the kernel. While, in the form of the gradient in Eq. (15), the first term emphasizes data fit through the likelihood, the second term encourages sparsification through the prior, and the third term penalizes similar parameterizations through the repulsive kernel.
More specifically, the terms in Eq. (15) can be interpreted as: (a) a kernel-smoothed gradient that drives the particles toward the best-fit region according to the likelihood, (b) a sparsifying force whose magnitude depends on that is present in the definition of the prior density in Eq. (6), and (c) a repulsion term that keeps the particles distinct. For the second term, the scaling is explicit in , but for the third term, it is implicit in the bandwidth of the kernel. A traditional Stein flow with an uninformative prior involves only (a) and (c), i.e., , while a gradient flow to find a sparse maximum a posteriori (MAP) point involves only (a) and (b) with the bandwidth such that the kernel is flat and the gradient in (c) is zero. Depending on the number of fungible parameters and the number of negligible parameters, the sparsification and repulsion terms can conflict with their objectives, which we will explore in the next section, Sec. 4.
We use an exponential family of kernels similar to the prior family described in Eq. (3), where a sparsifying prior of the form , was utilized:
with the idea that lower exponential kernels will repel more strongly along coordinate directions and potentially add sparsification. These kernels have the gradient
For this family, the second term in Eq. (14) simplifies to
using the identity that the score of a product is the sum of the scores
So, in this manner, the (kernel smoothed) gradient of the log kernel and log prior directly affect the Stein flow, which is guided by the data through the likelihood.
Graph condensation
We focus on complex models, such as neural networks, with many parameters, some of which are redundant or insignificant.
The FFNNs model have a structure based on layers composed of nodes. The output is the composition of trainable affine transforms and preselected element-wise nonlinear transforms :
where is the output of the first layer and the parameters of the model are consisting of weight matrices and bias vectors . Due to the fungibility of parameters in a layer, multiple permutations of the same parameters produce identical output. In fact, nothing in the (regularized or unregularized) optimization process breaks this degeneracy. In an ensemble of parameter realizations, forward propagation is unhindered but uncertainties of parameters are confounded by this permutational symmetry, in the sense that parameters playing similar roles in are not in identical nodes of the NN. Furthermore, this condensation ameliorates the potential for spurious particle repulsion in the Stein ensemble.
Hence, we reconcile the parameterizations by maximizing their similarity. First, we represent the NNs as directed graphs with nodes associated with biases and edges with weights . In a graph representing an FFNN all nodes in a layer are fungible so that permutations for all the hidden layers (i.e. excluding input and output layers )
produce models with identical output. A particular version of the similarity objective is to find permutations that attain a minimum distance across the ensemble:
where the permutations for realization are fixed at the identity and the pairwise distance is
In Eq. (23) we have ignored biases, which will be appropriate for the model we use in the demonstration but, in general, may not be sufficient to distinguish parameterizations.
In this work, we employ a heuristic algorithm based on sorting and pruning to obtain a solution to the parameter reconciliation problem. First, we identify and remove edges associated with insignificant (near zero) weights; then we remove disconnected nodes i.e., those with only zero incoming or outgoing weights. Second, we sort the nodes in each layer based on criterion importance.
which is the influence of node in layer on all nodes in the subsequent layer and are components of the weight matrix. Third we put all graphs in the ensemble on a common graph that is defined by the maximum nodes for each layer across the ensemble. Graphs that do not use all nodes in this template are padded to fit. Alg. 1 summarizes the algorithm and provides more details.
Algorithm 1: Graph condense
Input: A set of directed graphs representing FFNN realizations in a ensemble
while ensemble is not changing topologically do
(1) Prune and sort each graph
for graph do
Prune insignificant edges,
Prune nodes that do not have at least one incoming edge and outgoing edge with non-zero weights
Sort nodes in each layer by importance Eq. (24) based on their outgoing edge weights
Reorder nodes in each layer and remap edge connections
end
(2) Determine the common graph
Define common graph template as a fully connected layer-to-layer graph with nodes in layer
for layer do
Determine maximum number of active nodes of layer across ensemble
end
(3) Reconcile ensemble
for graph do
Insert into template with node padding as needed
end
end
Result: A set of directed graphs sharing a common graph structure with quasi-optimal edge similarity
Algorithm: SVGD with concurrent sparsification.
The overall algorithm is summarized in Alg. 2. Beyond the choice of prior order and kernel order , the hyperparameters are: (a) the initial penalty , (b) the kernel width , and (c) the number of particles. The initial penalty should be chosen low enough to only slightly perturb the posterior so as to promote initial likelihood driven convergence. The kernel width can be chosen empirically, for example, with Silverman’s rule [22], or adaptively. The dimensionality of the parameter space and the computational budget guide the choice of the number of parameters. Paramount is to scale to achieve a balance between the competing objectives of sparsification and accuracy.
At the convergence of Alg. 2, the particles collectively approximate the posterior distribution empirically:
where is the number of realizations/particles. A kernel density estimate (KDE) of the particles enables resampling of the posterior.
Algorithm 2: Stein variational gradient descent with concurrent sparsification.
Input: A set of initial particles , score function , kernel , step-size , initial penalty
for stage do
for iteration do
Compute gradient from Eq. (15)
Update parameters
If converged then break
end
compact models: clip inactive edges, put all particles on common graph using Alg. 1
adapt sparsifying penalty : if accuracy (MSE) has not decreased from previous stage increase , else terminate
end
at termination, revert to minimal to remove the effect of the prior and increase accuracy with the fixed model form.
Result: A set of particles that approximates the target posterior
Illustrative example
We illustrate the behavior of the Stein ensemble under the combined influence of a Gaussian likelihood, a sparsifying exponential prior and a repulsive exponential kernel. Specifically, the chosen likelihood is a multivariate normal (MVN) with mean and precision :
The prior is associated with the multiplier , and the kernel with bandwidth . An ensemble of particles is more than sufficient to recover the posterior mean and covariance structure in the 3 dimensional parameter space. Given the low precision of , we expect the algorithm to sparsify . Note by sparsifying we can not expect a good estimate of the covariance for that parameter, which illustrates the competing sparsification and repulsion objectives.
We measure error in the predicted posterior using distance between the empirical mean and covariance of the particle ensemble and true moments of the MVN. Specifically, we use the Bhattacharyya distance
where , which is sensitive to discrepancies in the mean and some aspects of the covariance. We measure sparsity with a norm of across the ensemble.
Figure 2 shows, in the absence of data, how a particle, under the influence of the kernel effects of another particle, is perturbed only if the other particle is also close to the origin. In this illustration, the gradient is only from the prior, as in Eq. (18), and the contour plot depicts the pseudopotential corresponding to , , and . For practical demonstration in the next section, § 5, we modify Alg. 2 to mask kernel effects when the particles are close to parameter axes.
Figure 2. Illustration of the Stein gradient (red arrows) and pseudopotential (contours) for prior and kernel, one particle (blue dot) in the field of another (green X) which is fixed: (left) fixed particle is at the origin, (middle) fixed particle is near the origin, (right) fixed particle is away from the origin. The multiplier and the bandwidth .
Figure 3 quantifies the interplay between kernel and prior over wide range of the multiplier and bandwidth values for selected priors and kernels . Clearly, there are similar trends for all prior and kernel selections; mainly, distribution error increases and sparsity error (the opposite of sparsity) decreases with increased prior multiplier i.e. there is a tradeoff between accuracy and sparsity that is dependent on the kernel width and the multiplier. For distribution accuracy there appears to be an optimal kernel width. Also, most trends are sigmoidal, with clustering the curves indicating insensitivity to the kernel bandwidth for some choices, particularly for the parameter sparsity with the kernel. Note, for the following physical application, we will only use the kernel and use a threshold to reduce near zero parameters to zero (so that parameter count becomes the appropriate measure of sparsity).
Figure 3. Error distribution (a, Bhatacharyya distance between the posterior and the true distribution) and sparsity (b, norm of ) with respect to increasing multiplier . Colors indicate kernel bandwidth . Each panel represents a different choice of prior order and kernel order .
Lastly, Figure 4 shows the actual arrangement of the Stein particles relative to the contours of the posterior for two values of the multiplier and multiple values of the kernel bandwidth . Clearly, the higher values of the bandwidth encourage the particles to explore more of the posterior landscape, while the higher value of allows the particles to constrict more on the dimension, as seen by the high density of particles near and hence sparsify the model.
Figure 4. Converged particles for various bandwidths using a prior with multiplier (a,left) and (b,right). Colors indicate kernel bandwidth.
Application to material constitutive modeling
Constructing a model of the response of a hyperelastic material from data is a canonical problem in solid mechanics [23]. In this case the (second Piola-Kirchhoff) stress determined by a potential :
so that the response is energy conservative i.e.
over a cycle of (the Lagrange) strain . To ensure a well-behaved boundary value problems based on the model (28), it is sufficient that is polyconvex [24]; hence, an input convex neural network (ICNN) is a particular FFNN that is well-suited to this problem [25, 26, 27, 28, 29, 30, 31, 32, 33]. Furthermore, if we assume material isotropy, the inputs of can be reduced to three independent scalar invariants of strain so that
Neural Network architecture
For this demonstration, we construct a physics-informed ICNN model that maps between three input invariants and the output stress through Eq. (30). As in Ref. [34], a generalized ICNN network is a fully connected neural network with a skip connection and constraints on specific weights. For an output and corresponding input , the ICNN with layers is the stack:
with weights and , activation functions and . The weights and biases form the set of trainable parameters . The output is convex with respect to the input because the weights are constrained to be non-negative, and the activation functions are chosen to be convex and non-decreasing [34].
As in Ref. [7], we modify the ICNN architecture to obtain better training and prediction. For computational efficiency, we ignore bias terms and assign the same weight per layer for while having the weight values to be non-negative. We employ softplus as the activation functions . Specifically, we initialize the ICNN architecture
with layers and nodes for each layer and two hidden layers for each particle. The model predicts the potential , and then the stress is calculated using automatic differentiation (PyTorch [35]). We constrain the stress to be zero at the reference:
where and is a constant that provides stress normalization, as in Ref. [1].
Data Generation
For this demonstration, we use training points and 1000 testing points, which are unseen during training. The input space for the training and testing data is generated by uniformly sampling the deformation space [F]:
For the training data, we limit the range to . Test set is formed from a parameterized deformation
where . Given these inputs we evaluate the same truth model as in our previous work Ref. [7]
with , , , and , and introduce multiplicative noise to the outputs to generate the input-output pairs
Here acts as a noise level.
Performance evaluation
Pushforward posteriors can be generated from the Stein ensemble of realizations via
We use the Wasserstein-1 () distance to compare the push-forward posteriors :
where the cumulative distribution functions (CDFs) associated with the probability density functions are defined empirically from the particle samples pushed forward through the model and data, respectively. Being a distance, a smaller implies that the two distributions are more similar. We apply the distance to the predictions at every time and also produce a single similarity measure by summing the values across all output times.
Graph condense
Figure 5 illustrates how the graphs representing the ensemble of FFNNs evolve under the Stein flow for prior with multiplier . The bandwidth for the kernel was adapted using where is the median distance between particles. Three randomly selected particles in an ensemble of are shown over a sequence of iterations progressing from left, where the models are clearly complex, to the right, where they are evidently sparse. All particles are distinct but become more similar as the gradient descent and graph condensation iterations progress. It is apparent that the method is sampling the diversity of likely parameterizations while removing permutational variations that artificially increase parameter variance. Note that any particular realization/particle can become more complex as it can fill out common graph for ensemble during a stage of the SVGD.
Figure 5. Sequence of graphs for 3 randomly selected particles during the iteration process. Initial models are on the left, and the final on the right. Nodes are colored by importance Eq. (24), and edges are colored by weight . All plotted values are normalized per layer.
Figure 6 illustrates the evolution of interparticle distances for a ensemble, as calculated by Eq. (23). Since the initial weights are randomized, the particles start out effectively equidistant from each other. Then we see that the particles have a transient where they attain a maximum dispersion. Eventually through the SVGD with graph condensation we see that subsets of particles grow close to each other but there is still diversity in the parameterizations.
Figure 6. Evolution of graph distances of the ensemble during SVGD (blue pixels indicate zero and yellow pixels indicate maximum value). Note this illustration uses particles.
Figure 7 shows the final distributions of , using kernel density estimation (KDE) of the particle samples. The input to first hidden layer weight matrix is not constrained to be positive by the ICNN formulation and has number of weight values that are widely centered at zero and a few that have distinct positive peaks. The middle weight matrix has a mixture of zero and non-zero centered distributions indicating the connections in between the two hidden layers are sparse. Interestingly, the last hidden layer to output weight matrix has strongly ordered weights, although some still have significant zero centered modes. The fact that most distributions are unimodal or bimodal with one peak at zero indicates the importance ranking is effective in aligning the weights; however, there are likely better metrics to sort and align the parameterizations.
Figure 7. Final kernel density estimates (KDE) of the distribution of weights: (a) , (b) , (c) . Weights connect layers and and are the components of which are abbreviated as in the legends of the plots. Note narrow distributions centered at zero are omitted for clarity.
Lastly Figure 8 provides evidence that a sparsifying SVGD, shown in Alg. 2, with graph condense, presented in Alg. 1, is more effective than without graph condensation at reducing model complexity across a wide range of prior choices. This comes with no reduction in accuracy and an attendant speed-up that is proportional to the complexity reduction. We found that the computational cost of an iteration is roughly proportional to the parameter count of the models in the Stein ensemble, hence, as the models simplify the gradient descent iterations speed up. Without graph condense, near-zero parameters are allowed to re-emerge, whereas with graph condensation the particles in the subsequent stages of the SVGD work within a restricted weight graph / template.
Figure 8. Accuracy (a, distance) and sparsity (b, parameter count) for various priors with and without graph condensation. Note initial parameter count is 1020.
Convergence
We explored the sensitivity of our method’s performance with respect to various hyperparameters.
Fig. 9 compares the accuracy, as measured by the distance, and sparsity, in terms of parameter count, of the Stein ensemble over the gradient descent iterations for various prior orders for (the noise level in Eq. (36)). As expected, the trends in accuracy and complexity are complementary. The most complex models are also the most accurate; however, the behavior for all priors is similar. The prior () is an outlier since it is generally not considered a prior that strongly induces parameter sparsity (it does force parameters to zero but has no corners in its level sets), as evidenced by the flat stages in the parameter counts. However, there are abrupt decreases in parameter counts when the graph condensation is applied based on a weight tolerance. This effect in sparsification is apparent with the higher priors and become more smooth for the lower priors. For all priors, we observe that as , sparsity is induced earlier in the iteration process, but this also delays the error convergence. The influence of on model complexity is evident from the final number of active weights at the last iteration: for , the corresponding active weights are . This highlights how smaller values of lead to stronger sparsity, whereas larger values result in more complex models with more active parameters. Consequently, as , computational time decreases due to the algorithm operating on a smaller/condensed graph with fewer active weights, whereas priors and require more iterations to converge compared to , as they involve training a larger number of active weights. For this case study, we consider an ensemble of 10 particles. The choice of enables the algorithm to explore the Pareto front between accuracy and sparsity objectives, balancing the trade-offs inherent to these competing goals.
Fig. 10 compares the convergence of the Stein ensemble for various sizes . Sparsity is relatively insensitive to the ensemble size, while accuracy is noticeably affected. In this analysis, where we consider the prior and noise level of 10%, we observe that for 100 particles, there is some evidence of overfitting or crowding behavior, likely due to the repulsive kernels not allowing all particles to occupy the high-probability region of the posterior. Allowing the kernel width to vary adaptively may reduce this sensitivity. Nevertheless, for , the results are fairly robust, yielding final model sizes on the order of 10 parameters.
Fig. 11 compares convergence for increasing noise levels . Again sparsity is relatively insensitive to increased noise, but accuracy suffers, as expected. The errors increase approximately linearly with multiplicative noise. Fig. 12a provides a detailed view of the distances as a function of the input, deformation , and noise. The errors are zero at due to multiplicative noise and model construction, Eq. (32), and generally increase linearly away from . Fig. 12b shows that the concurrent SVGD method proposed in this work can outperform the sequential sparsify and then propagate a Stein ensemble developed in our previous work [7]. In all cases, the model was initialized with the same number of weights: 1020 parameters. The effect of graph condensation significantly enhances the model’s efficiency. In our previous work [7], without graph condensation, the + Stein sequential method utilized 1005 parameters. In contrast, graph condensation in this work reduces the number of parameters to 127, achieving similar accuracy while significantly decreasing computational time. Hence, operating on a condensed graph with fewer active weights allows the algorithm to achieve lower computational complexity and improved efficiency.
All these studies were performed with a fixed penalty . This was done for simplicity and clarity in the comparison of trends. Since there is an optimal for each choice of the prior order, kernel order, and size, as well as trade-offs between accuracy and complexity, in the next section, we demonstrate the effectiveness of using an adaptive penalty scheme. For all the cases, including the adaptive case, a threshold of is employed to exclude negligible weights, thereby improving computational efficiency. The neural network models in this work were trained on a single NVIDIA RTX A6000 GPU.
Adaptive penalty
We assume the existence of an optimal kernel and prior in terms of a particular balance of sparsity and accuracy. In the previous demonstrations, we have presented a limited exploration focusing primarily on trends with changes in the hyperparameters. In this section, we compare the empirical study of the penalty to a control scheme that starts with a low so as not to strongly influence accuracy and then increases till accuracy suffers and finally reverts to the initial value with the sparsification fixed by the graph condensation after 4000 iterations. In this analysis, we fix at 0.5 and number of particles to be 10.
Fig. 13 compares the performance of a higher than optimal , a near optimal and lower than optimal penalty with the adaptive penalty scheme outlined in Alg. 2. Clearly, the simple adaptive scheme promotes both accuracy and sparsity that is near optimal across a wide range of noise levels .
Figure 9. Convergence of accuracy (a) and sparsity (b) for various prior orders with 10% noise. Note initial parameter count for all models is 1020.
Figure 10. Convergence of accuracy (a) and sparsity (b) for various ensemble sizes with 10% noise. Note initial parameter count for all models is 1020.
Conclusion
We developed an enhancement of the Stein variational gradient descent scheme that reduces the complexity of the models in the ensemble via a sparsifying prior. It is based a parameter alignment and condensation procedure ameliorates the potential for spurious particle repulsion in the Stein ensemble. The balance between accuracy and complexity goals is tunable via the choice of the penalty parameter and the prior order. Generally, the performance is not overly sensitive to these choices. We also observed some performance dependence on the repulsive kernel shape and size. Significantly, the reduction in complexity leads to iteration speed up due to the descent operating in a smaller parameter space, whereby the extra cost of the graph sorting and reconfiguring is more than offset by this efficiency.
In future work, we intend to explore other sorting criteria in the graph condense algorithm and replacing the like priors with smoother Huber-like priors [36] to possibly improve convergence. Also, more sophisticated penalty and kernel width adaptation will likely improve performance. We will apply the method to other neural network architectures, particularly large, multicomponent models, where the graph condensation can be done on components of the neural network, either in sequence or guided by expert knowledge or some model selection scheme. For the the large-scale deployment, we also foresee the need for optimization of the data structures and algorithms for efficient sorting and model reconfiguration.
(a) Accuracy
(b) Sparsity
Figure 11. Convergence of accuracy (a) and sparsity (b) for various noise levels . Note initial parameter count for all models is 1020. We consider the prior and number of particles 10.
(a) Noise levels
(b) Method comparison
Figure 12. Pushforward errors as measured by the distance as a function of deformation : (a) for various multiplicative noise levels, and (b) comparison of methods for 10% noise level. In all cases, the model was initialized with 1,020 parameters. Note the high-frequency noise in the curves has been filtered with a moving average to help distinguish the trends.
Figure 13. Comparison of (a) accuracy and (b) sparsity for fixed and adaptive penalty.
Acknowledgements
GAP and NB were supported by the SciAI Center, and funded by the Office of Naval Research (ONR), under Grant Number N00014-23-1-2729. REJ and CS were supported by the U.S. Department of Energy, Advanced Scientific Computing program. CS was also supported by the Scientific Discovery through Advanced Computing (SciDAC) program through the FASTMath Institute.
Sandia National Laboratories is a multi-mission laboratory managed and operated by National Technology & Engineering Solutions of Sandia, LLC (NTESS), a wholly owned subsidiary of Honeywell International Inc., for the U.S. Department of Energy’s National Nuclear Security Administration (DOE/NNSA) under contract DE-NA0003525. This written work is authored by an employee of NTESS. The employee, not NTESS, owns the right, title and interest in and to the written work and is responsible for its contents. Any subjective views or opinions that might be expressed in the written work do not necessarily represent the views of the U.S. Government. The publisher acknowledges that the U.S. Government retains a non-exclusive, paid-up, irrevocable, world-wide license to publish or reproduce the published form of this written work or allow others to do so, for U.S. Government purposes. The DOE will provide public access to results of federally sponsored research in accordance with the DOE Public Access Plan.
This research utilized the Delta advanced computing and data resource, which is supported by the National Science Foundation (award OAC 2005572) and the State of Illinois. Delta is a collaborative initiative between the University of Illinois Urbana-Champaign and its National Center for Supercomputing Applications (NCSA). This work leveraged the Delta system at NCSA through allocation MTH240015, provided by the Advanced Cyberinfrastructure Coordination Ecosystem: Services & Support (ACCESS) program, supported by National Science Foundation grants #2138259, #2138286, #2138307, #2137603, and #2138296.
References
- [1]Jan Niklas Fuhg, Reese Edward Jones, and Nikolaos Bouklas. Extreme sparsification of physics-augmented neural networks for interpretable model discovery in mechanics. Computer Methods in Applied Mechanics and Engineering, 426:116973, 2024.arxiv.org/abs/2310.03652
- [2]Roger Ghanem, David Higdon, Houman Owhadi, et al. Handbook of uncertainty quantification, volume 6. Springer New York, 2017.
- [3]Michael Stein. Large sample properties of simulations using latin hypercube sampling. Technometrics, 29(2):143–151, 1987.DOI
- [4]Qiang Liu and Diliin Wang. Stein variational gradient descent: A general purpose bayesian inference algorithm. Advances in neural information processing systems, 29, 2016.arxiv.org/abs/1608.04471
- [5]Peng Chen, Keyi Wu, Joshua Chen, Tom O’Leary-Roseberry, and Omar Ghattas. Projected stein variational newton: A fast and scalable bayesian inference method in high dimensions. Advances in Neural Information Processing Systems, 32, 2019.arxiv.org/abs/1901.08659
- [6]Gianluca Detommaso, Tiangang Cui, Youssef Marzouk, Alessio Spantini, and Robert Scheichl. A stein variational newton method. Advances in Neural Information Processing Systems, 31, 2018.
- [7]Govinda Anantha Padmanabha, Jan Niklas Fuhg, Cosmin Safta, Reese E Jones, and Nikolaos Bouklas. Improving the performance of stein variational inference through extreme sparsification of physically-constrained neural network models. Computer Methods in Applied Mechanics and Engineering, 432:117359, 2024.arxiv.org/abs/2407.00761
- [8]Christos Louizos, Max Welling, and Diederik P Kingma. Learning sparse neural networks through l_0 regularization. arXiv preprint arXiv:1712.01312, 2017.arxiv.org/abs/1712.01312
- [9]Qiang Liu. Stein variational gradient descent as gradient flow. Advances in neural information processing systems, 30, 2017.arxiv.org/abs/1704.07520
- [10]Alex Leviyev, Joshua Chen, Yifei Wang, Omar Ghattas, and Aaron Zimmerman. A stochastic stein variational newton method. arXiv preprint arXiv:2204.09039, 2022.arxiv.org/abs/2204.09039
- [11]Jeremy A McCulloch, Skyler R St. Pierre, Kevin Linka, and Ellen Kuhl. On sparse regression, lp-regularization, and automated model discovery. International Journal for Numerical Methods in Engineering, 125(14):e7481, 2024.
- [12]Steven L Brunton, Joshua L Proctor, and J Nathan Kutz. Discovering governing equations from data by sparse identification of nonlinear dynamical systems. Proceedings of the national academy of sciences, 113(15):3932–3937, 2016.DOI
- [13]Robert Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology, 58(1):267–288, 1996.
- [14]Shihao Ji, Ya Xue, and Lawrence Carin. Bayesian compressive sensing. IEEE Transactions on signal processing, 56(6):2346–2356, 2008.DOI
- [15]S Derin Babacan, Rafael Molina, and Aggelos K Katsaggelos. Bayesian compressive sensing using laplace priors. IEEE Transactions on image processing, 19(1):53–63, 2009.DOI
- [16]Dror Baron, Shriram Sarvotham, and Richard G Baraniuk. Bayesian compressive sensing via belief propagation. IEEE Transactions on Signal Processing, 58(1):269–280, 2009.arxiv.org/abs/0812.4627
- [17]Adrian S Lewis. Active sets, nonsmoothness, and sensitivity. SIAM Journal on Optimization, 13(3):702–725, 2002.DOI
- [18]Hans Joachim Ferreau, Hans Georg Bock, and Moritz Diehl. An online active set strategy to overcome the limitations of explicit mpc. International Journal of Robust and Nonlinear Control: IFAC-Affiliated Journal, 18(8):816–830, 2008.
- [19]Zaiwen Wen, Wotao Yin, Hongchao Zhang, and Donald Goldfarb. On the convergence of an active-set method for ℓ1 minimization. Optimization Methods and Software, 27(6):1127–1146, 2012.
- [20]Sidhant Misra, Line Roald, and Yeesian Ng. Learning for constrained optimization: Identifying optimal active constraint sets. INFORMS Journal on Computing, 34(1):463–480, 2022.
- [21]Charles J Geyer. Introduction to markov chain monte carlo. Handbook of markov chain monte carlo, 201160222(45):22, 2011.
- [22]B. W. Silverman. Density Estimation for Statistics and Data Analysis. Chapman and Hall, 1986.
- [23]Jan N Fuhg, Govinda Anantha Padmanabha, Nikolaos Bouklas, Bahador Bahmani, WaiChing Sun, Nikolaos N Vlassis, Moritz Flaschel, Pietro Carrara, and Laura De Lorenzis. A review on data-driven constitutive laws for solids. Archives of Computational Methods in Engineering, pages 1–43, 2024.
- [24]John M Ball. Convexity conditions and existence theorems in nonlinear elasticity. Archive for rational mechanics and Analysis, 63:337–403, 1976.
- [25]Vahidullah Tac, Francisco Sahli Costabal, and Adrian B Tepole. Data-driven tissue mechanics with polyconvex neural ordinary differential equations. Computer Methods in Applied Mechanics and Engineering, 398:115248, 2022.
- [26]Peiyi Chen and Johann Guilleminot. Polyconvex neural networks for hyperelastic constitutive models: A rectification approach. Mechanics Research Communications, 125:103993, 2022.
- [27]Faisal As’ad, Philip Avery, and Charbel Farhat. A mechanics-informed artificial neural network approach in data-driven constitutive modeling. International Journal for Numerical Methods in Engineering, 123(12):2738–2759, 2022.
- [28]Kailai Xu, Daniel Z Huang, and Eric Darve. Learning constitutive relations using symmetric positive definite neural networks. Journal of Computational Physics, 428:110072, 2021.
- [29]Dominik K Klein, Mauricio Fernández, Robert J Martin, Patrizio Neff, and Oliver Weeger. Polyconvex anisotropic hyperelasticity with neural networks. Journal of the Mechanics and Physics of Solids, 159:104703, 2022.DOI
- [30]Dominik K Klein, Fabian J Roth, Iman Valizadeh, and Oliver Weeger. Parametrized polyconvex hyperelasticity with physics-augmented neural networks. Data-Centric Engineering, 4:e25, 2023.DOI
- [31]Karl A Kalina, Philipp Gebhart, Jörg Brummund, Lennart Linden, WaiChing Sun, and Markus Kästner. Neural network-based multiscale modeling of finite strain magneto-elasticity with relaxed convexity criteria. Computer Methods in Applied Mechanics and Engineering, 421:116739, 2024.arxiv.org/abs/2308.16311
- [32]Jan N Fuhg, Nikolaos Bouklas, and Reese E Jones. Learning hyperelastic anisotropy from data via a tensor basis neural network. Journal of the Mechanics and Physics of Solids, 168:105022, 2022.
- [33]Jan N Fuhg, Lloyd van Wees, Mark Obstalekci, Paul Shade, Nikolaos Bouklas, and Matthew Kasemer. Machine-learning convex and texture-dependent macroscopic yield from crystal plasticity simulations. Materialia, 23:101446, 2022.
- [34]Brandon Amos, Lei Xu, and J Zico Kolter. Input convex neural networks. In International Conference on Machine Learning, pages 146–155. PMLR, 2017.arxiv.org/abs/1609.07152
- [35]Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.arxiv.org/abs/1912.01703
- [36]Peter J Huber. Robust estimation of a location parameter. In Breakthroughs in statistics: Methodology and distribution, pages 492–518. Springer, 1992.









