betaFeedback

WorksComputer Science

Condensed Stein Variational Gradient Descent for Uncertainty Quantification of Neural Networks

Abstract

We propose a Stein variational gradient descent method to concurrently sparsify, train, and provide uncertainty quantification of a complexly parameterized model such as a neural network. It employs a graph reconciliation and condensation process to reduce complexity and increase similarity in the Stein ensemble of parameterizations. Therefore, the proposed condensed Stein variational gradient (cSVGD) method provides uncertainty quantification on parameters, not just outputs. Furthermore, the parameter reduction speeds up the convergence of the Stein gradient descent as it reduces the combinatorial complexity by aligning and differentiating the sensitivity to parameters. These properties are demonstrated with an illustrative example and an application to a representation problem in solid mechanics.

Introduction

In the context of uncertainty quantification (UQ) the curse of dimensionality, whereby quantification efficiency degrades drastistically with parameter dimension, is particular extreme with highly parameterized models such as neural networks (NNs). Fortunately, in many cases, these models are overparameterized in the sense that the number of parameters can be reduced with negligible effects on accuracy and sometimes improvements in generalization [1]. Furthermore, NNs often have parameterizations that have fungible parameters such that permutations of the values and connections lead to equivalent output responses. This suggests methods that simultaneously sparsify and characterize the uncertainty of a model, while handling and taking advantage of the symmetries inherent in the model, are potentially advantageous approaches.

Although Markov chain Monte Carlo (MCMC) methods [2] have been the reference standard to generate samples for UQ methods, they can be temperamental and do not scale well for high dimensional models. More recently, there has been widespread use of variational inference methods (VI), which cast the parameter posterior sampling problem as an optimization of a surrogate posterior guided by a suitable objective, such as the Kullback-Liebler (KL) divergence between the predictive posterior and true posterior induced by the data. In particular, there is now a family of model ensemble methods based on Stein’s identity [3], such as Stein variational gradient descent (SVGD) [4], projected SVGD [5], and Stein variational Newton’s method [6]. These methods have advantages over MCMC methods by virtue of propagating in parallel a coordinated ensemble of particles that represent the empirical posterior.

In our recent previous work [7], we proposed a sequential sparsification-uncertainty quantification method based on L0L_0 regularization and Stein gradient flow. We showed that sparsification by L0L_0 methods [8] was able to capture the nonlinear nature of the reduced parameter manifold, which subsequently was explored by a Stein ensemble of particles representing likely model parameterizations. The sequential application of sparsification and then SVGD displayed advantages over established methods [5] on a number of physical representation problems.

The concurrent sparsification-uncertainty quantification method presented in this work, termed cSVGD, has a number of advantages over our sequential method. Foremost, the new method enables parameter alignment and leads to more interpretable parameter UQ, versus solely providing output uncertainty estimates. This also enables the Stein method to avoid spurious repulsion or lack of repulsion due to parameter permutations. Furthermore, there is an improvement in computational speed due to the reduction in the complexity of the models in the Stein ensemble. Parameter UQ and computational speed-ups are enabled by reconciling the parameters by importance across the Stein ensemble and condensing the common representation of the model as the gradient flow and sparsification progress. The method has a firm theoretical foundation based on the introduction of sparsifying priors into the Stein variational gradient flow formalism [4].

In the next section, § 2, we give an overview of related work that we build upon. Then, in § 3, we develop the concurrent sparsifying Stein gradient descent (cSVGD) algorithm for feedforward neural networks (FFNNs). In § 4, we illustrate the general behavior of the algorithm, in particular, the interplay of the hyperparameters controlling the prior and the Stein kernel. In § 5, we demonstrate the performance of the method on a problem of scientific interest: determining a hyperelastic potential from stress-strain data. Finally, in § 6, we conclude with a summary of findings and avenues for future work.

Related work

To provide an efficient alternative to MCMC sampling Liu and Wang [4, 9] introduced Stein variational gradient descent (SVGD) which propagates a coordinated ensemble of parameter realizations that approximate the posterior distribution. Newton [6, 10] and projected [5] variants soon followed. Our previous work [7] combined sparsification and SVGD in sequence to fully and efficiently explore the submanifold of influential parameters. That work relied on LpL_p, p∈[0,1]p \in[0, 1], regularizations that induced parameter sparsity, such as the work by Louizos et al. [8] which developed a smoothed L0L_0 regularization. Others, such as McCulloch et al. [11], have used this type of regularization in scientific machine learning. Another notable example of sparse parameter identification is the Sparse Identification of Nonlinear Dynamical systems (SINDy) framework [12]. In a probabilistic, Bayesian context these regularizations have analogs in sparsifying priors, such as the Cauchy prior introduced in Bayesian compressive sensing and Least Absolute Shrinkage and Selection Operator (LASSO) [13, 14, 15, 16].

Our augmentation of sparsifying priors in the Stein flow with model graph condensation to reduce complexity has some similarity to the well-established field of active set methods [17, 18, 19, 20]. These methods rely on transformations and/or heuristics to reduce the permutational complexity of identifying the active constraints in a large set of inequality constraints [20] or the optimum in a non-smooth problem [19].

Concurrent sparsification and Stein variational gradient descent

First, we develop Stein variational gradient descent (SVGD) with a sparsifying prior as the basis for the proposed algorithm. Then, we show how we achieve parameter alignment by treating the Stein ensemble as an ensemble of graphs with a common graph. Finally, we summarize the overall algorithm.

Bayesian parameter uncertainty quantification

Given a model y^(x;θ)\hat{y}(\mathbf{x};\boldsymbol{\theta}) with inputs x\mathbf{x}, parameters θ=(θ1,θ2,…,θn)\boldsymbol{\theta} = (\theta_1, \theta_2, \ldots, \theta_n), and data D={⟨xi,yi⟩}i=1ND\mathcal{D} = \{\langle x_i, y_i\rangle\}_{i=1}^{N_D}, where NDN_D is the number of data samples; Bayes rule expressed as:

π(θ∣D)=π(D∣θ)π(θ)π(D)(1)\pi(\theta\mid\mathcal{D}) = \frac{\pi(\mathcal{D} \mid\theta)\pi(\theta)}{\pi(\mathcal{D})} \tag*{(1)}

provides the posterior distribution π(θ∣D)\pi(\theta\mid\mathcal{D}) on the parameters θ\theta given data D\mathcal{D}. The formulation is also dependent on choices for the likelihood π(D∣θ)\pi(\mathcal{D} \mid\theta) and the prior distribution of parameters π(θ)\pi(\theta). These distributions determine the posterior and the evidence π(D)\pi(\mathcal{D}).

For example, with the assumption of a Gaussian likelihood

π(D∣θ)∝exp⁡(−12σ2∣y−y^(x;θ)∣2)(2)\pi(\mathbf{D}\mid\theta) \propto\exp\left(-\frac{1}{2\sigma^2}\lvert y-\hat{y}(x;\theta)\rvert^2\right) \tag*{(2)}

and a sparsifying prior

π(θ)∝exp⁡(−λ∣θ∣α),α∈(0,1](3)\pi(\theta) \propto\exp\left(-\lambda\lvert\theta\rvert^\alpha\right),\quad\alpha\in(0,1] \tag*{(3)}

the negative log posterior becomes

−log⁡π(θ∣D)=12σ2∣y−y^(x;θ)∣2+λ∣θ∣α+…,(4)-\log\pi(\theta\mid\mathbf{D})=\frac{1}{2\sigma^2}\lvert y-\hat{y}(x;\theta)\rvert^2+\lambda\lvert\theta\rvert^\alpha+\ldots, \tag*{(4)}

where σ2\sigma^2 is the data noise variance and λ\lambda acts as a penalty parameter on the regularizating objective ∣θ∣α\lvert\theta\rvert^\alpha. Note the remaining additive terms in Eq. (4) do not depend on the parameters θ\theta and, hence, do not affect the gradient of log⁡π(θ∣D)\log\pi(\theta\mid\mathbf{D}), otherwise known as the score. Here, and in the following, we use ∣θ∣α\lvert\theta\rvert^\alpha to denote

∣θ∣α=∑i∣θi∣α(5)\lvert\theta\rvert^\alpha=\sum_i\lvert\theta_i\rvert^\alpha \tag*{(5)}

i.e. ∣θ∣α\lvert\theta\rvert^\alpha is related to the LαL_\alpha norm by ∣θ∣α=(∥θ∥α)α\lvert\theta\rvert^\alpha=(\lVert\theta\rVert_\alpha)^\alpha. Figure 1 illustrates this family of posteriors; notice that as α→0\alpha\to0 the prior approximates the L0L_0 prior.

Figure 1. Log posteriors formed from a Gaussian likelihood and priors from the α\alpha exponential family scaled by λ\lambda. As λ\lambda is increased (top to bottom), the posterior resembles the prior mode more than the likelihood, and as α\alpha is reduced (right to left), the contours of the prior transform from diamond-shaped to more cruciform-shaped. Note the values of λ\lambda are relative to the particular likelihood chosen for this illustration.

Specifically, we employ priors in the exponential family

π(θ)=λc1(α)exp⁡(−λαc2(α)∣θ∣α),(6)\pi(\theta)=\lambda c_1(\alpha)\exp\left(-\lambda^\alpha c_2(\alpha)\lvert\theta\rvert^\alpha\right), \tag*{(6)}

where

c1(α)=α[Γ(3/α)]1/22[Γ(1/α)]3/2andc2(α)=[Γ(3/α)Γ(1/α)]α/2.(7)c_1(\alpha)=\frac{\alpha\left[\Gamma(3/\alpha)\right]^{1/2}}{2\left[\Gamma(1/\alpha)\right]^{3/2}} \quad\text{and}\quad c_2(\alpha)=\left[\frac{\Gamma(3/\alpha)}{\Gamma(1/\alpha)}\right]^{\alpha/2}. \tag*{(7)}

and Γ\Gamma is the Gamma function.

Variational inference

As an alternative to sampling the posterior distribution with standard but computationally expensive Markov chain Monte Carlo (MCMC) methods [21], variational inference (VI) reformulates the task of approximating the posterior as an optimization problem on the parameters of a surrogate posterior. In VI, the optimization problem for the surrogate q(θ)q(\theta) is typically based on the Kullback–Leibler (KL) divergence:

q∗(θ)=arg⁡min⁡φ KL⁡(q(θ;φ)∥π(θ∣D))=arg⁡min⁡φ Eq[log⁡q(θ;φ)−log⁡p(θ∣D)],(8)\begin{aligned} q^*(\theta) = \underset{\varphi}{\arg\min}\ \operatorname{KL}(q(\theta;\varphi)\Vert\pi(\theta\mid\mathrm{D})) \\ &= \underset{\varphi}{\arg\min}\ \mathbb{E}_q[\log q(\theta;\varphi)-\log p(\theta\mid\mathrm{D})], \tag*{(8)} \end{aligned}

measuring the difference between the surrogate qq and true posterior π(θ∣D)\pi(\theta\mid\mathrm{D}) associated with the data D. Given that p(θ∣D)p(\theta\mid\mathrm{D}) appears in a logarithm in Eq. (8), the evidence does not need to be evaluated (since the it is a constant) in the search for the optimum q∗q^*.

The SVGD algorithm [4] is a VI method based on the Stein identity [3]

Eθ∼π[Sπ[ϕ(θ)]]=0,(9)\mathbb{E}_{\theta\sim\pi}[\mathcal{S}_{\pi}[\phi(\theta)]]=0, \tag*{(9)}

where

Sπ[ϕ(θ)]=∇θ[log⁡π(θ)]⊗ϕ(θ)+∇θϕ(θ)=∇θ[π(θ)ϕ(θ)]π(θ)(10)\mathcal{S}_{\pi}[\phi(\theta)] = \nabla_{\theta}[\log\pi(\theta)]\otimes\phi(\theta)+\nabla_{\theta}\phi(\theta)=\frac{\nabla_{\theta}[\pi(\theta)\phi(\theta)]}{\pi(\theta)} \tag*{(10)}

for a smooth function ϕ\phi and a generic probability density π(θ)\pi(\theta). The Stein operator Sπ\mathcal{S}_{\pi} depends on the density π\pi only through the score function ∇θ[log⁡π(θ)]\nabla_{\theta}[\log\pi(\theta)] which is independent of the normalization of π\pi and thus avoids the evaluation of the evidence.

If we let qq be an approximate density under a perturbation mapping of the random variable θ↦θ+ϵϕ\theta\mapsto\theta+\epsilon\phi, then the expectation of tr⁡(Aπ[ϕ(θ)])\operatorname{tr}(\mathcal{A}_{\pi}[\phi(\theta)]) is related to the gradient of the Kullback–Leibler (KL) divergence between qq and π\pi [4 Thm 3.1]:

∇ϵDKL(qθ+ϵϕ∥π)∣ϵ=0=−Eq[tr⁡(Aπ[ϕ(θ)])].(11)\left.\nabla_{\epsilon}D_{\mathrm{KL}}(q_{\theta+\epsilon\phi}\Vert\pi)\right|_{\epsilon=0}=-\mathbb{E}_q[\operatorname{tr}(\mathcal{A}_{\pi}[\phi(\theta)])]. \tag*{(11)}

This provides the basis for a gradient-based method to update the approximate posterior density.

The SVGD algorithm utilizes an ensemble of NrN_r particles {θ(i)}i=1Nr\{\theta_{(i)}\}_{i=1}^{N_r} in the parameter space. After randomly initializing the parameter ensemble, SVGD updates the particles via gradient-descent steps

θa(k+1)=θa(k)+ϵg(θ(i)(k)),(12)\theta_a^{(k+1)}=\theta_a^{(k)}+\epsilon g(\theta_{(i)}^{(k)}), \tag*{(12)}

where ϵ\epsilon is a step size parameter, kk is the iteration counter, and

g(θ)=1Nr∑b=1Nr∇θb(k)(κ(θb(k),θ)π(θb(k)∣D))π(θb(k)∣D)(13)g(\theta)=\frac{1}{N_r}\sum_{b=1}^{N_r}\frac{\nabla_{\theta_b^{(k)}}\left(\kappa(\theta_b^{(k)},\theta)\pi(\theta_b^{(k)}\mid\mathrm{D})\right)}{\pi(\theta_b^{(k)}\mid\mathrm{D})} \tag*{(13)}
=1Nr∑b=1Nr[κ(θb(k),θ)∇θb(k)π(D∣θb(k))π(D∣θb(k))+∇θb(k)(κ(θb(k),θ)π(θb(k)))π(θb(k))](14)=\frac{1}{N_r}\sum_{b=1}^{N_r}\left[\kappa(\theta_b^{(k)},\theta)\frac{\nabla_{\theta_b^{(k)}}\pi(\mathrm{D}\mid\theta_b^{(k)})}{\pi(\mathrm{D}\mid\theta_b^{(k)})}+\frac{\nabla_{\theta_b^{(k)}}\left(\kappa(\theta_b^{(k)},\theta)\pi(\theta_b^{(k)})\right)}{\pi(\theta_b^{(k)})}\right] \tag*{(14)}
=1Nr∑b=1Nr[κ(θb(k),θ)(∇θb(k)[log⁡π(D∣θb(k))]+∇θb(k)[log⁡π(θb(k))])+∇θb(k)κ(θb(k),θ)](15)=\frac{1}{N_r}\sum_{b=1}^{N_r}\left[\kappa(\theta_b^{(k)},\theta)\left(\nabla_{\theta_b^{(k)}}[\log\pi(\mathrm{D}\mid\theta_b^{(k)})]+\nabla_{\theta_b^{(k)}}[\log\pi(\theta_b^{(k)})]\right)+\nabla_{\theta_b^{(k)}}\kappa(\theta_b^{(k)},\theta)\right] \tag*{(15)}

is the parameter gradient which depends on the choice of the kernel κ(⋅,⋅)\kappa(\cdot,\cdot). Note the sum is over all particles and ∇θκ(θ,θ)≡0\nabla_{\theta}\kappa(\theta,\theta)\equiv0 by definition. We will call the kernel smoothed posterior κ(θb(k),θ)π(θb(k)∣D)\kappa(\theta_b^{(k)},\theta)\pi(\theta_b^{(k)}\mid\mathrm{D}) apparent in Eq. (13) the pseudopotential.

In the form of the gradient in Eq. (14) we see that the first term depends solely on the likelihood and kernel, while the second term depends only on the prior and the kernel. While, in the form of the gradient in Eq. (15), the first term emphasizes data fit through the likelihood, the second term encourages sparsification through the prior, and the third term penalizes similar parameterizations through the repulsive kernel.

More specifically, the terms in Eq. (15) can be interpreted as: (a) a kernel-smoothed gradient that drives the particles toward the best-fit region according to the likelihood, (b) a sparsifying force whose magnitude depends on λ\lambda that is present in the definition of the prior density in Eq. (6), and (c) a repulsion term that keeps the particles distinct. For the second term, the scaling is explicit in λ\lambda, but for the third term, it is implicit in the bandwidth of the kernel. A traditional Stein flow with an uninformative prior involves only (a) and (c), i.e., λ=0\lambda= 0, while a gradient flow to find a sparse maximum a posteriori (MAP) point involves only (a) and (b) with the bandwidth γ→∞\gamma\to\infty such that the kernel is flat and the gradient in (c) is zero. Depending on the number of fungible parameters and the number of negligible parameters, the sparsification and repulsion terms can conflict with their objectives, which we will explore in the next section, Sec. 4.

We use an exponential family of kernels similar to the prior family described in Eq. (3), where a sparsifying prior of the form π(θ)∝exp⁡(−λ∣θ∣α)\pi(\theta) \propto\exp(-\lambda|\theta|^\alpha), α∈(0,1]\alpha\in(0,1] was utilized:

κ(θ′,θ)=exp⁡(−1γβ∣θ′−θ∣β),(16)\kappa(\theta',\theta) = \exp\left(-\frac{1}{\gamma^\beta}|\theta'-\theta|^\beta\right), \tag*{(16)}

with the idea that lower β\beta exponential kernels will repel more strongly along coordinate directions and potentially add sparsification. These kernels have the gradient

∇θ′κ(θ′,θ)=1γ∣θ−θ′∣β−1sign⁡(θ−θ′)κ(θ′,θ)(17)\nabla_{\theta'}\kappa(\theta',\theta) = \frac{1}{\gamma}|\theta-\theta'|^{\beta-1}\operatorname{sign}(\theta-\theta')\kappa(\theta',\theta) \tag*{(17)}

For this family, the second term in Eq. (14) simplifies to

∇θb(k)(κ(θb(k),θ)π(θb(k)))π(θb(k))=exp⁡(−γ∣θb(k)−θ∣β)∇θb(k)(−γ∣θb(k)−θ∣β+λ∣θb(k)∣α)(18)\frac{\nabla_{\theta_b^{(k)}}\left(\kappa(\theta_b^{(k)},\theta)\pi(\theta_b^{(k)})\right)}{\pi(\theta_b^{(k)})} = \exp\left(-\gamma|\theta_b^{(k)}-\theta|^\beta\right)\nabla_{\theta_b^{(k)}}\left(-\gamma|\theta_b^{(k)}-\theta|^\beta+\lambda|\theta_b^{(k)}|^\alpha\right) \tag*{(18)}

using the identity that the score of a product is the sum of the scores

∇x(a(x)b(x))a(x)b(x)=∇xa(x)a(x)+∇xb(x)b(x).(19)\frac{\nabla_x(a(x)b(x))}{a(x)b(x)} = \frac{\nabla_x a(x)}{a(x)}+\frac{\nabla_x b(x)}{b(x)}. \tag*{(19)}

So, in this manner, the (kernel smoothed) gradient of the log kernel and log prior directly affect the Stein flow, which is guided by the data through the likelihood.

Graph condensation

We focus on complex models, such as neural networks, with many parameters, some of which are redundant or insignificant.

The FFNNs model have a structure based on layers ℓ\ell composed of nodes. The output is the composition of trainable affine transforms Wℓ⋅+bℓW_\ell\cdot+ b_\ell and preselected element-wise nonlinear transforms aℓ(⋅)a_\ell(\cdot):

y=NN⁡(x;θ)=aNℓ(WNℓ(…a0(W0x+b0)…)+bNℓ)(20)y = \operatorname{NN}(x;\theta) = a_{N_\ell}\left(W_{N_\ell}\left(\ldots a_0(W_0x+b_0)\ldots\right)+b_{N_\ell}\right) \tag*{(20)}

where y1y_1 is the output of the first layer (ℓ=1)(\ell=1) and the parameters of the model are θ={Wℓ,bℓ}ℓ=1Nℓ\theta= \{W_\ell,b_\ell\}_{\ell=1}^{N_\ell} consisting of weight matrices WℓW_\ell and bias vectors bℓb_\ell. Due to the fungibility of parameters in a layer, multiple permutations of the same parameters produce identical output. In fact, nothing in the (regularized or unregularized) optimization process breaks this degeneracy. In an ensemble of parameter realizations, forward propagation is unhindered but uncertainties of parameters are confounded by this permutational symmetry, in the sense that parameters playing similar roles in are not in identical nodes of the NN. Furthermore, this condensation ameliorates the potential for spurious particle repulsion in the Stein ensemble.

Hence, we reconcile the parameterizations by maximizing their similarity. First, we represent the NNs as directed graphs G={N,E}\mathcal{G}=\{\mathcal{N},\mathcal{E}\} with nodes N\mathcal{N} associated with biases bℓ=[bi]ℓ\mathbf{b}_\ell=[b_i]_\ell and edges E\mathcal{E} with weights W=[WIJ]ℓ\mathbf{W}=[W_{IJ}]_\ell. In a graph representing an FFNN all nodes in a layer ℓ\ell are fungible so that permutations Pℓ\mathbf{P}_\ell for all the hidden layers ℓ∈[2,Nℓ−1]\ell\in[2,N_\ell-1] (i.e. excluding input ℓ=1\ell=1 and output layers ℓ=Nℓ\ell=N_\ell)

Wℓ′=Pℓ+1TWℓPℓandbℓ′=Pℓ+1Tbℓ(21)W'_\ell= P_{\ell+1}^{T}W_\ell P_\ell\quad\text{and}\quad b'_\ell= P_{\ell+1}^{T}b_\ell \tag*{(21)}

produce models with identical output. A particular version of the similarity objective is to find permutations that attain a minimum distance across the ensemble:

argmin⁡{Pa} ∥d(NNa,NNb)∥a∈[1,Nr] and b∈[a+1,Nr](22)\underset{\{P_a\}}{\operatorname{argmin}}\ \lVert d(\mathrm{NN}_a,\mathrm{NN}_b)\rVert\quad a \in[1,N_r] \text{ and } b \in[a+1,N_r] \tag*{(22)}

where the permutations Pa={Pℓ,a}\mathbf{P}_a = \{\mathbf{P}_{\ell,a}\} for realization a=1a = 1 are fixed at the identity and the pairwise distance is

d(NNa,NNb)=∑ℓ∥Wℓ,a−Wℓ,b∥22,(23)d(\mathcal{NN}_a,\mathcal{NN}_b) = \sqrt{\sum_{\ell} \left\lVert\mathbf{W}_{\ell,a} - \mathbf{W}_{\ell,b} \right\rVert_2^2}, \tag*{(23)}

In Eq. (23) we have ignored biases, which will be appropriate for the model we use in the demonstration but, in general, may not be sufficient to distinguish parameterizations.

In this work, we employ a heuristic algorithm based on sorting and pruning to obtain a solution to the parameter reconciliation problem. First, we identify and remove edges associated with insignificant (near zero) weights; then we remove disconnected nodes i.e., those with only zero incoming or outgoing weights. Second, we sort the nodes in each layer based on criterion importance.

sℓ,j=∑i[Wℓ]ij(24)s_{\ell,j} = \sum_i [W_{\ell}]_{ij} \tag*{(24)}

which is the influence of node jj in layer ℓ\ell on all nodes in the subsequent layer and [Wℓ]ij[W_{\ell}]_{ij} are components of the wℓw_{\ell} weight matrix. Third we put all graphs in the ensemble on a common graph that is defined by the maximum nodes for each layer across the ensemble. Graphs that do not use all nodes in this template are padded to fit. Alg. 1 summarizes the algorithm and provides more details.

Algorithm 1: Graph condense

Input: A set of directed graphs {Ga}\{G_a\} representing FFNN realizations in a ensemble

while ensemble {Ga}\{G_a\} is not changing topologically do

(1) Prune and sort each graph

for graph G∈{Ga}G \in\{G_a\} do

Prune insignificant edges, [Wℓ]ij<ϵ[W_{\ell}]_{ij} < \epsilon

Prune nodes that do not have at least one incoming edge and outgoing edge with non-zero weights

Sort nodes in each layer by importance Eq. (24) based on their outgoing edge weights

Reorder nodes in each layer and remap edge connections

end

(2) Determine the common graph

Define common graph template G‾\overline{G} as a fully connected layer-to-layer graph with nℓn_{\ell} nodes in layer ℓ∈[0,Nℓ]\ell\in[0, N_{\ell}]

for layer ℓ∈[2,Nℓ−1]\ell\in[2, N_{\ell} - 1] do

Determine maximum number of active nodes nℓn_{\ell} of layer ℓ\ell across ensemble {Ga}\{G_a\}

end

(3) Reconcile ensemble

for graph G∈{Ga}G \in\{G_a\} do

Insert GG into template G‾\overline{G} with node padding as needed

end

end

Result: A set of directed graphs {Ga}\{G_a\} sharing a common graph structure with quasi-optimal edge similarity

Algorithm: SVGD with concurrent sparsification.

The overall algorithm is summarized in Alg. 2. Beyond the choice of prior order α\alpha and kernel order β\beta, the hyperparameters are: (a) the initial penalty λ0\lambda_0, (b) the kernel width γ\gamma, and (c) the NrN_r number of particles. The initial penalty should be chosen low enough to only slightly perturb the posterior so as to promote initial likelihood driven convergence. The kernel width can be chosen empirically, for example, with Silverman’s rule [22], or adaptively. The dimensionality of the parameter space and the computational budget guide the choice of the number of parameters. Paramount is to scale λ\lambda to achieve a balance between the competing objectives of sparsification and accuracy.

At the convergence of Alg. 2, the particles collectively approximate the posterior distribution empirically:

π(θ∣D)≈q(θ)=∑a=1Nrδ(θ−θa)(25)\pi(\theta\mid\mathcal{D}) \approx q(\theta) = \sum_{a=1}^{N_r} \delta(\theta- \theta_a) \tag*{(25)}

where NrN_r is the number of realizations/particles. A kernel density estimate (KDE) of the particles enables resampling of the posterior.

Algorithm 2: Stein variational gradient descent with concurrent sparsification.

Input: A set of initial particles {θa(0)}a=1Nr\{\theta_a^{(0)}\}_{a=1}^{N_r}, score function ∇log⁡π(θ∣D)\nabla\log\pi(\theta\mid D), kernel κ(θ,θ′)\kappa(\theta,\theta'), step-size {ϵ}\{\epsilon\}, initial penalty λ(0)\lambda^{(0)}

for stage ss do

&nbsp;&nbsp;&nbsp;&nbsp;for iteration kk do

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Compute gradient ga(k)g_a^{(k)} from Eq. (15)

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Update parameters θa(k+1)←θa(k)+ϵg(θa(k))\theta_a^{(k+1)} \leftarrow\theta_a^{(k)}+\epsilon g(\theta_a^{(k)})

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;If converged then break

&nbsp;&nbsp;&nbsp;&nbsp;end

&nbsp;&nbsp;&nbsp;&nbsp;compact models: clip inactive edges, put all particles on common graph using Alg. 1

&nbsp;&nbsp;&nbsp;&nbsp;adapt sparsifying penalty λ\lambda: if accuracy (MSE) has not decreased from previous stage increase λ\lambda, else terminate

end

at termination, revert to minimal λ\lambda to remove the effect of the prior and increase accuracy with the fixed model form.

Result: A set of particles {θa}a=1Nr\{\theta_a\}_{a=1}^{N_r} that approximates the target posterior

Illustrative example

We illustrate the behavior of the Stein ensemble under the combined influence of a Gaussian likelihood, a sparsifying α\alpha exponential prior and a repulsive β\beta exponential kernel. Specifically, the chosen likelihood is a multivariate normal (MVN) with mean μ\mu and precision Σ−1\boldsymbol{\Sigma}^{-1}:

μ=(1,2,3)andΣ−1=(210120000.0025).(26)\mu=(1,2,3)\quad\text{and}\quad\boldsymbol{\Sigma}^{-1}= \begin{pmatrix} 2 & 1 & 0\\ 1 & 2 & 0\\ 0 & 0 & 0.0025 \end{pmatrix}. \tag*{(26)}

The α\alpha prior is associated with the multiplier λ\lambda, and the β\beta kernel with bandwidth γ\gamma. An ensemble of Nparticles=128N_{\mathrm{particles}}=128 particles is more than sufficient to recover the posterior mean and covariance structure in the 3 dimensional parameter space. Given the low precision of θ3\theta_3, we expect the algorithm to sparsify θ3\theta_3. Note by sparsifying θ3\theta_3 we can not expect a good estimate of the covariance for that parameter, which illustrates the competing sparsification and repulsion objectives.

We measure error in the predicted posterior using distance between the empirical mean and covariance of the particle ensemble and true moments of the MVN. Specifically, we use the Bhattacharyya distance

d(N′,N)=18(μ′−μ)′Σˉ−1(μ′−μ)+12log⁡(det⁡Σˉdet⁡Σ′det⁡Σ)(27)d(\mathcal{N}',\mathcal{N})=\frac{1}{8}(\mu'-\mu)^{\prime}\bar{\boldsymbol{\Sigma}}^{-1}(\mu'-\mu)+\frac{1}{2}\log\left(\frac{\det\bar{\boldsymbol{\Sigma}}}{\sqrt{\det\boldsymbol{\Sigma}'\det\boldsymbol{\Sigma}}}\right) \tag*{(27)}

where Σˉ=12(Σ′+Σ)\bar{\boldsymbol{\Sigma}}=\frac{1}{2}(\boldsymbol{\Sigma}'+\boldsymbol{\Sigma}), which is sensitive to discrepancies in the mean and some aspects of the covariance. We measure sparsity with a L1L_1 norm of θ3\theta_3 across the ensemble.

Figure 2 shows, in the absence of data, how a particle, under the influence of the kernel effects of another particle, is perturbed only if the other particle is also close to the origin. In this illustration, the gradient is only from the prior, as in Eq. (18), and the contour plot depicts the pseudopotential corresponding to α=1\alpha=1, β=2\beta=2, and γ=1\gamma=1. For practical demonstration in the next section, § 5, we modify Alg. 2 to mask kernel effects when the particles are close to parameter axes.

Image not extracted

Figure 2. Illustration of the Stein gradient (red arrows) and pseudopotential (contours) for α=1\alpha= 1 prior and β=2\beta= 2 kernel, one particle (blue dot) in the field of another (green X) which is fixed: (left) fixed particle is at the origin, (middle) fixed particle is near the origin, (right) fixed particle is away from the origin. The multiplier λ=1\lambda= 1 and the bandwidth γ=1/10\gamma= 1/10.

Figure 3 quantifies the interplay between kernel and prior over wide range of the multiplier λ\lambda and bandwidth γ\gamma values for selected priors α∈{1/4,1,2}\alpha\in\{1/4,1,2\} and kernels β∈{1,2}\beta\in\{1,2\}. Clearly, there are similar trends for all prior and kernel selections; mainly, distribution error increases and sparsity error (the opposite of sparsity) decreases with increased prior multiplier λ\lambda i.e. there is a tradeoff between accuracy and sparsity that is dependent on the kernel width and the multiplier. For distribution accuracy there appears to be an optimal kernel width. Also, most trends are sigmoidal, with clustering the curves indicating insensitivity to the kernel bandwidth γ\gamma for some choices, particularly for the parameter sparsity with the β=2\beta=2 kernel. Note, for the following physical application, we will only use the β=2\beta=2 kernel and use a threshold to reduce near zero parameters to zero (so that parameter count becomes the appropriate measure of sparsity).

Figure 3. Error distribution (a, Bhatacharyya distance between the posterior and the true distribution) and sparsity (b, L1L_1 norm of w3w_3) with respect to increasing multiplier λ\lambda. Colors indicate kernel bandwidth γ\gamma. Each panel represents a different choice of prior order α\alpha and kernel order β\beta.

Lastly, Figure 4 shows the actual arrangement of the Stein particles relative to the contours of the posterior for two values of the multiplier λ\lambda and multiple values of the kernel bandwidth γ\gamma. Clearly, the higher values of the bandwidth encourage the particles to explore more of the posterior landscape, while the higher value of λ\lambda allows the particles to constrict more on the θ3\theta_3 dimension, as seen by the high density of particles near θ3=0\theta_3 = 0 and hence sparsify the model.

Figure 4. Converged particles for various bandwidths using a α=1\alpha= 1 prior with multiplier λ=0.1\lambda= 0.1 (a,left) and λ=1.0\lambda= 1.0 (b,right). Colors indicate kernel bandwidth.

Application to material constitutive modeling

Constructing a model of the response of a hyperelastic material from data is a canonical problem in solid mechanics [23]. In this case the (second Piola-Kirchhoff) stress S\mathbf{S} determined by a potential Φ\Phi:

S=∂EΦ(28)\mathbf{S} = \partial_{\mathbf{E}}\Phi \tag*{(28)}

so that the response is energy conservative i.e.

∮ES⋅E˙ dt=0(29)\oint_{\mathbf{E}} \mathbf{S} \cdot\dot{\mathbf{E}}\,\mathrm{d}t = 0 \tag*{(29)}

over a cycle of (the Lagrange) strain E\mathbf{E}. To ensure a well-behaved boundary value problems based on the model (28), it is sufficient that Φ\Phi is polyconvex [24]; hence, an input convex neural network (ICNN) is a particular FFNN that is well-suited to this problem [25, 26, 27, 28, 29, 30, 31, 32, 33]. Furthermore, if we assume material isotropy, the inputs of Φ\Phi can be reduced to three independent scalar invariants of strain {I1,I2,I3}\{I_1, I_2, I_3\} so that

S=∂EΦ=∑i=1,2,3∂IiΦ ∂EIi(30)\mathbf{S} = \partial_{\mathbf{E}}\Phi= \sum_{i=1,2,3} \partial_{I_i}\Phi\,\partial_{\mathbf{E}}I_i \tag*{(30)}

Neural Network architecture

For this demonstration, we construct a physics-informed ICNN model that maps between three input invariants and the output stress through Eq. (30). As in Ref. [34], a generalized ICNN network is a fully connected neural network with a skip connection and constraints on specific weights. For an output y=Φ\mathbf{y} = \Phi and corresponding input x=(Ii,i=1,2,3)\mathbf{x} = (I_i, i = 1, 2, 3), the ICNN with NℓN_\ell layers is the stack:

h1=σ1(V1x+b1)hk=σk(Vkx+Wkhk−1+bk)k=2,…,Nℓ−1y=VNℓx+WNℓhNℓ−1+bNℓ(31)\begin{aligned} \mathbf{h}_1 &= \sigma_1(\mathbf{V}_1\mathbf{x}+\mathbf{b}_1) \\ \mathbf{h}_k &= \sigma_k(\mathbf{V}_k\mathbf{x}+\mathbf{W}_k\mathbf{h}_{k-1}+\mathbf{b}_k) \qquad k = 2,\ldots,N_\ell-1 \\ \mathbf{y} &= \mathbf{V}_{N_\ell}\mathbf{x}+\mathbf{W}_{N_\ell}\mathbf{h}_{N_\ell-1}+\mathbf{b}_{N_\ell} \tag*{(31)} \end{aligned}

with weights Wk\mathbf{W}_k and Vk\mathbf{V}_k, activation functions σk\sigma_k and k∈[1,Nℓ]k \in[1,N_\ell]. The weights and biases form the set of trainable parameters θ={Wk,Vk,bk}\theta= \{\mathbf{W}_k, \mathbf{V}_k, \mathbf{b}_k\}. The output is convex with respect to the input because the weights Wk\mathbf{W}_k are constrained to be non-negative, and the activation functions σk\sigma_k are chosen to be convex and non-decreasing [34].

As in Ref. [7], we modify the ICNN architecture to obtain better training and prediction. For computational efficiency, we ignore bias terms and assign the same weight Vk=Wk\mathbf{V}_k = \mathbf{W}_k per layer for Vk\mathbf{V}_k while having the weight values Wk\mathbf{W}_k to be non-negative. We employ softplus as the activation functions σk\sigma_k. Specifically, we initialize the ICNN architecture

with Nℓ=3N_{\ell}=3 layers and nℓ=30n_{\ell}=30 nodes for each layer and two hidden layers for each particle. The model predicts the potential Φ\Phi, and then the stress is calculated using automatic differentiation (PyTorch [35]). We constrain the stress to be zero at the reference:

Φ^(I1,I2,I3)=NN(I1,I2,I3)−NN(3,3,1)−Φ0(I3)(32)\hat{\Phi}(I_{1}, I_{2}, I_{3}) = \mathrm{NN}(I_{1}, I_{2}, I_{3}) - \mathrm{NN}(3, 3, 1) - \Phi_{0}(I_{3}) \tag*{(32)}

where Φ0(I3)=n(I3−1)\Phi_{0}(I_{3}) = n(\sqrt{I_{3}} - 1) and nn is a constant that provides stress normalization, as in Ref. [1].

Data Generation

For this demonstration, we use ND=80N_{D}=80 training points and 1000 testing points, which are unseen during training. The input space for the training and testing data is generated by uniformly sampling the deformation space [F]:

F=I+H where H∼U[−δ,δ].(33)\mathbf{F} = \mathbf{I} + \mathbf{H} \text{ where } \mathbf{H} \sim\mathcal{U}[-\delta,\delta]. \tag*{(33)}

For the training data, we limit the range to δ=0.2\delta=0.2. Test set is formed from a parameterized deformation

F=diag⁡(1+δ,1+δ,1+δ)(34)\mathbf{F} = \operatorname{diag}(1+\delta,\sqrt{1+\delta},\sqrt{1+\delta}) \tag*{(34)}

where δ∈[−0.4,0.4]\delta\in[-0.4,0.4]. Given these inputs we evaluate the same truth model as in our previous work Ref. [7]

Ψ(I1,I2,J)=−ϑ12Jmlog⁡(1−I1−3Jm)−ϑ2log⁡(I2J)+ϑ3(12(J2−1)−log⁡J),(35)\Psi(I_{1}, I_{2}, J) = -\frac{\vartheta_{1}}{2}J_{m}\log\left(1-\frac{I_{1}-3}{J_{m}}\right)-\vartheta_{2}\log\left(\frac{I_{2}}{J}\right)+\vartheta_{3}\left(\frac{1}{2}(J^{2}-1)-\log J\right), \tag*{(35)}

with J=I3J=\sqrt{I_{3}}, Jm=77.931J_{m}=77.931, ϑ1=2.4195\vartheta_{1}=2.4195, ϑ2=−0.75\vartheta_{2}=-0.75 and ϑ3=1.20975\vartheta_{3}=1.20975, and introduce multiplicative noise to the outputs to generate the input-output pairs

D=(Ei,ζϵSi) where ϵ∼N(1,I)(36)D=(\mathbf{E}_{i},\zeta\epsilon\mathbf{S}_{i}) \text{ where } \epsilon\sim\mathcal{N}(1,\mathbf{I}) \tag*{(36)}

Here ζ\zeta acts as a noise level.

Performance evaluation

Pushforward posteriors can be generated from the Stein ensemble of realizations {θb}b=1Nr\{\theta_{b}\}_{b=1}^{N_{r}} via

Sa=S^(E;θa) where θa∈{θb}b=1Nr(37)S_{a}=\hat{S}(\mathbf{E};\theta_{a}) \text{ where } \theta_{a}\in\{\theta_{b}\}_{b=1}^{N_{r}} \tag*{(37)}

We use the Wasserstein-1 (W1W_{1}) distance to compare the push-forward posteriors π(S)=∑aδ(S−Sa)\pi(S)=\sum_{a}\delta(S-S_{a}):

W1(π(S^),π(S))=∫∣CDF⁡(S^)−CDF⁡(S)∣ dS(38)W_{1}(\pi(\hat{S}),\pi(S))=\int\left|\operatorname{CDF}(\hat{S})-\operatorname{CDF}(S)\right|\,\mathrm{d}S \tag*{(38)}

where the cumulative distribution functions (CDFs) associated with the probability density functions π(S)\pi(S) are defined empirically from the particle samples pushed forward through the model and data, respectively. Being a distance, a smaller W1\mathcal{W}_1 implies that the two distributions are more similar. We apply the W1\mathcal{W}_1 distance to the predictions at every time and also produce a single similarity measure by summing the values across all output times.

Graph condense

Figure 5 illustrates how the graphs representing the ensemble of FFNNs evolve under the Stein flow for α=1/4\alpha= 1/4 prior with multiplier λ=0.05\lambda= 0.05. The bandwidth for the β=2\beta= 2 kernel was adapted using γ=dˉ/2log⁡(Nr+1)\gamma= \sqrt{\bar{d}/2\log(N_r + 1)} where dˉ\bar{d} is the median distance between particles. Three randomly selected particles in an ensemble of Nr=10N_r = 10 are shown over a sequence of iterations progressing from left, where the models are clearly complex, to the right, where they are evidently sparse. All particles are distinct but become more similar as the gradient descent and graph condensation iterations progress. It is apparent that the method is sampling the diversity of likely parameterizations while removing permutational variations that artificially increase parameter variance. Note that any particular realization/particle can become more complex as it can fill out common graph for ensemble during a stage of the SVGD.

Image not extracted

Figure 5. Sequence of graphs for 3 randomly selected particles during the iteration process. Initial models are on the left, and the final on the right. Nodes are colored by importance Eq. (24), and edges are colored by weight WijW_{ij}. All plotted values are normalized per layer.

Figure 6 illustrates the evolution of interparticle distances for a Nr=20N_r = 20 ensemble, as calculated by Eq. (23). Since the initial weights are randomized, the particles start out effectively equidistant from each other. Then we see that the particles have a transient where they attain a maximum dispersion. Eventually through the SVGD with graph condensation we see that subsets of particles grow close to each other but there is still diversity in the parameterizations.

Image not extracted

Figure 6. Evolution of graph distances of the ensemble during SVGD (blue pixels indicate zero and yellow pixels indicate maximum value). Note this illustration uses Nr=20N_r = 20 particles.

Figure 7 shows the final distributions of WℓW_\ell, ℓ∈0,1,2\ell\in0, 1, 2 using kernel density estimation (KDE) of the particle samples. The input to first hidden layer weight matrix W0W_0 is not constrained to be positive by the ICNN formulation and has number of weight values that are widely centered at zero and a few that have distinct positive peaks. The middle weight matrix W2W_2 has a mixture of zero and non-zero centered distributions indicating the connections in between the two hidden layers are sparse. Interestingly, the last hidden layer to output weight matrix W2W_2 has strongly ordered weights, although some still have significant zero centered modes. The fact that most distributions are unimodal or bimodal with one peak at zero indicates the importance ranking is effective in aligning the weights; however, there are likely better metrics to sort and align the parameterizations.

Image not extracted

Figure 7. Final kernel density estimates (KDE) of the distribution of weights: (a) W0W_0, (b) W1W_1, (c) W2W_2. Weights WkW_k connect layers ℓ=k\ell= k and ℓ=k+1\ell= k + 1 and [Wk]ij[W_k]_{ij} are the components of WkW_k which are abbreviated as WijW_{ij} in the legends of the plots. Note narrow distributions centered at zero are omitted for clarity.

Lastly Figure 8 provides evidence that a sparsifying SVGD, shown in Alg. 2, with graph condense, presented in Alg. 1, is more effective than without graph condensation at reducing model complexity across a wide range of prior choices. This comes with no reduction in accuracy and an attendant speed-up that is proportional to the complexity reduction. We found that the computational cost of an iteration is roughly proportional to the parameter count of the models in the Stein ensemble, hence, as the models simplify the gradient descent iterations speed up. Without graph condense, near-zero parameters are allowed to re-emerge, whereas with graph condensation the particles in the subsequent stages of the SVGD work within a restricted weight graph / template.

Image not extracted

Figure 8. Accuracy (a, W1\mathcal{W}_1 distance) and sparsity (b, parameter count) for various priors with and without graph condensation. Note initial parameter count is 1020.

Convergence

We explored the sensitivity of our method’s performance with respect to various hyperparameters.

Fig. 9 compares the accuracy, as measured by the W1W_1 distance, and sparsity, in terms of parameter count, of the Stein ensemble over the gradient descent iterations for various prior orders α\alpha for ς=10%\varsigma= 10\% (the noise level in Eq. (36)). As expected, the trends in accuracy and complexity are complementary. The most complex models are also the most accurate; however, the behavior for all priors α≤1\alpha\le1 is similar. The L2L_2 prior (α=2\alpha= 2) is an outlier since it is generally not considered a prior that strongly induces parameter sparsity (it does force parameters to zero but has no corners in its level sets), as evidenced by the flat stages in the parameter counts. However, there are abrupt decreases in parameter counts when the graph condensation is applied based on a weight tolerance. This effect in sparsification is apparent with the higher α\alpha priors and become more smooth for the lower α\alpha priors. For all priors, we observe that as α→0\alpha\to0, sparsity is induced earlier in the iteration process, but this also delays the error convergence. The influence of α\alpha on model complexity is evident from the final number of active weights at the last iteration: for α=[0.25,0.5,0.75,1,2]\alpha= [0.25, 0.5, 0.75, 1, 2], the corresponding active weights are [17,22,25,29,127][17, 22, 25, 29, 127]. This highlights how smaller values of α\alpha lead to stronger sparsity, whereas larger values result in more complex models with more active parameters. Consequently, as α→0\alpha\to0, computational time decreases due to the algorithm operating on a smaller/condensed graph with fewer active weights, whereas priors α=1\alpha= 1 and α=2\alpha= 2 require more iterations to converge compared to α<1\alpha< 1, as they involve training a larger number of active weights. For this case study, we consider an ensemble of 10 particles. The choice of α\alpha enables the algorithm to explore the Pareto front between accuracy and sparsity objectives, balancing the trade-offs inherent to these competing goals.

Fig. 10 compares the convergence of the Stein ensemble for various sizes NrN_r. Sparsity is relatively insensitive to the ensemble size, while accuracy is noticeably affected. In this analysis, where we consider the prior α=0.5\alpha= 0.5 and noise level of 10%, we observe that for 100 particles, there is some evidence of overfitting or crowding behavior, likely due to the repulsive kernels not allowing all particles to occupy the high-probability region of the posterior. Allowing the kernel width to vary adaptively may reduce this sensitivity. Nevertheless, for Nr≤50N_r \le50, the results are fairly robust, yielding final model sizes on the order of 10 parameters.

Fig. 11 compares convergence for increasing noise levels ς\varsigma. Again sparsity is relatively insensitive to increased noise, but accuracy suffers, as expected. The errors increase approximately linearly with multiplicative noise. Fig. 12a provides a detailed view of the W1W_1 distances as a function of the input, deformation F11F_{11}, and noise. The errors are zero at F11=0F_{11} = 0 due to multiplicative noise and model construction, Eq. (32), and generally increase linearly away from F11=0F_{11} = 0. Fig. 12b shows that the concurrent SVGD method proposed in this work can outperform the sequential sparsify and then propagate a Stein ensemble developed in our previous work [7]. In all cases, the model was initialized with the same number of weights: 1020 parameters. The effect of graph condensation significantly enhances the model’s efficiency. In our previous work [7], without graph condensation, the L2L_2+ Stein sequential method utilized 1005 parameters. In contrast, graph condensation in this work reduces the number of parameters to 127, achieving similar accuracy while significantly decreasing computational time. Hence, operating on a condensed graph with fewer active weights allows the algorithm to achieve lower computational complexity and improved efficiency.

All these studies were performed with a fixed penalty λ=0.05\lambda= 0.05. This was done for simplicity and clarity in the comparison of trends. Since there is an optimal λ\lambda for each choice of the prior order, kernel order, and size, as well as trade-offs between accuracy and complexity, in the next section, we demonstrate the effectiveness of using an adaptive penalty scheme. For all the cases, including the adaptive case, a threshold of 10−310^{-3} is employed to exclude negligible weights, thereby improving computational efficiency. The neural network models in this work were trained on a single NVIDIA RTX A6000 GPU.

Adaptive penalty

We assume the existence of an optimal kernel and prior in terms of a particular balance of sparsity and accuracy. In the previous demonstrations, we have presented a limited exploration focusing primarily on trends with changes in the hyperparameters. In this section, we compare the empirical study of the penalty λ\lambda to a control scheme that starts with a low λ=0.01\lambda= 0.01 so as not to strongly influence accuracy and then increases λ\lambda till accuracy suffers and finally reverts λ\lambda to the initial value with the sparsification fixed by the graph condensation after 4000 iterations. In this analysis, we fix α\alpha at 0.5 and number of particles to be 10.

Fig. 13 compares the performance of a higher than optimal λ=0.1\lambda= 0.1, a near optimal λ=0.05\lambda= 0.05 and lower than optimal penalty λ=0.01\lambda= 0.01 with the adaptive penalty scheme outlined in Alg. 2. Clearly, the simple adaptive scheme promotes both accuracy and sparsity that is near optimal across a wide range of noise levels ς\varsigma.

Figure 9. Convergence of accuracy (a) and sparsity (b) for various prior orders α\alpha with 10% noise. Note initial parameter count for all models is 1020.

Figure 10. Convergence of accuracy (a) and sparsity (b) for various ensemble sizes NrN_r with 10% noise. Note initial parameter count for all models is 1020.

Conclusion

We developed an enhancement of the Stein variational gradient descent scheme that reduces the complexity of the models in the ensemble via a sparsifying prior. It is based a parameter alignment and condensation procedure ameliorates the potential for spurious particle repulsion in the Stein ensemble. The balance between accuracy and complexity goals is tunable via the choice of the penalty parameter and the prior order. Generally, the performance is not overly sensitive to these choices. We also observed some performance dependence on the repulsive kernel shape and size. Significantly, the reduction in complexity leads to iteration speed up due to the descent operating in a smaller parameter space, whereby the extra cost of the graph sorting and reconfiguring is more than offset by this efficiency.

In future work, we intend to explore other sorting criteria in the graph condense algorithm and replacing the LαL_\alpha like priors with smoother Huber-like priors [36] to possibly improve convergence. Also, more sophisticated penalty λ\lambda and kernel width γ\gamma adaptation will likely improve performance. We will apply the method to other neural network architectures, particularly large, multicomponent models, where the graph condensation can be done on components of the neural network, either in sequence or guided by expert knowledge or some model selection scheme. For the the large-scale deployment, we also foresee the need for optimization of the data structures and algorithms for efficient sorting and model reconfiguration.

(a) Accuracy

(b) Sparsity

Figure 11. Convergence of accuracy (a) and sparsity (b) for various noise levels ς\varsigma. Note initial parameter count for all models is 1020. We consider the prior α=0.5\alpha= 0.5 and number of particles 10.

(a) Noise levels

(b) Method comparison

Figure 12. Pushforward errors as measured by the W1W_1 distance as a function of deformation F11F_{11}: (a) for various multiplicative noise levels, and (b) comparison of methods for 10% noise level. In all cases, the model was initialized with 1,020 parameters. Note the high-frequency noise in the W1W_1 curves has been filtered with a moving average to help distinguish the trends.

Figure 13. Comparison of (a) accuracy and (b) sparsity for fixed and adaptive penalty.

Acknowledgements

GAP and NB were supported by the SciAI Center, and funded by the Office of Naval Research (ONR), under Grant Number N00014-23-1-2729. REJ and CS were supported by the U.S. Department of Energy, Advanced Scientific Computing program. CS was also supported by the Scientific Discovery through Advanced Computing (SciDAC) program through the FASTMath Institute.

Sandia National Laboratories is a multi-mission laboratory managed and operated by National Technology & Engineering Solutions of Sandia, LLC (NTESS), a wholly owned subsidiary of Honeywell International Inc., for the U.S. Department of Energy’s National Nuclear Security Administration (DOE/NNSA) under contract DE-NA0003525. This written work is authored by an employee of NTESS. The employee, not NTESS, owns the right, title and interest in and to the written work and is responsible for its contents. Any subjective views or opinions that might be expressed in the written work do not necessarily represent the views of the U.S. Government. The publisher acknowledges that the U.S. Government retains a non-exclusive, paid-up, irrevocable, world-wide license to publish or reproduce the published form of this written work or allow others to do so, for U.S. Government purposes. The DOE will provide public access to results of federally sponsored research in accordance with the DOE Public Access Plan.

This research utilized the Delta advanced computing and data resource, which is supported by the National Science Foundation (award OAC 2005572) and the State of Illinois. Delta is a collaborative initiative between the University of Illinois Urbana-Champaign and its National Center for Supercomputing Applications (NCSA). This work leveraged the Delta system at NCSA through allocation MTH240015, provided by the Advanced Cyberinfrastructure Coordination Ecosystem: Services & Support (ACCESS) program, supported by National Science Foundation grants #2138259, #2138286, #2138307, #2137603, and #2138296.

References

  1. [1]Jan Niklas Fuhg, Reese Edward Jones, and Nikolaos Bouklas. Extreme sparsification of physics-augmented neural networks for interpretable model discovery in mechanics. Computer Methods in Applied Mechanics and Engineering, 426:116973, 2024.arxiv.org/abs/2310.03652
  2. [2]Roger Ghanem, David Higdon, Houman Owhadi, et al. Handbook of uncertainty quantification, volume 6. Springer New York, 2017.
  3. [3]Michael Stein. Large sample properties of simulations using latin hypercube sampling. Technometrics, 29(2):143–151, 1987.DOI
  4. [4]Qiang Liu and Diliin Wang. Stein variational gradient descent: A general purpose bayesian inference algorithm. Advances in neural information processing systems, 29, 2016.arxiv.org/abs/1608.04471
  5. [5]Peng Chen, Keyi Wu, Joshua Chen, Tom O’Leary-Roseberry, and Omar Ghattas. Projected stein variational newton: A fast and scalable bayesian inference method in high dimensions. Advances in Neural Information Processing Systems, 32, 2019.arxiv.org/abs/1901.08659
  6. [6]Gianluca Detommaso, Tiangang Cui, Youssef Marzouk, Alessio Spantini, and Robert Scheichl. A stein variational newton method. Advances in Neural Information Processing Systems, 31, 2018.
  7. [7]Govinda Anantha Padmanabha, Jan Niklas Fuhg, Cosmin Safta, Reese E Jones, and Nikolaos Bouklas. Improving the performance of stein variational inference through extreme sparsification of physically-constrained neural network models. Computer Methods in Applied Mechanics and Engineering, 432:117359, 2024.arxiv.org/abs/2407.00761
  8. [8]Christos Louizos, Max Welling, and Diederik P Kingma. Learning sparse neural networks through l_0 regularization. arXiv preprint arXiv:1712.01312, 2017.arxiv.org/abs/1712.01312
  9. [9]Qiang Liu. Stein variational gradient descent as gradient flow. Advances in neural information processing systems, 30, 2017.arxiv.org/abs/1704.07520
  10. [10]Alex Leviyev, Joshua Chen, Yifei Wang, Omar Ghattas, and Aaron Zimmerman. A stochastic stein variational newton method. arXiv preprint arXiv:2204.09039, 2022.arxiv.org/abs/2204.09039
  11. [11]Jeremy A McCulloch, Skyler R St. Pierre, Kevin Linka, and Ellen Kuhl. On sparse regression, lp-regularization, and automated model discovery. International Journal for Numerical Methods in Engineering, 125(14):e7481, 2024.
  12. [12]Steven L Brunton, Joshua L Proctor, and J Nathan Kutz. Discovering governing equations from data by sparse identification of nonlinear dynamical systems. Proceedings of the national academy of sciences, 113(15):3932–3937, 2016.DOI
  13. [13]Robert Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology, 58(1):267–288, 1996.
  14. [14]Shihao Ji, Ya Xue, and Lawrence Carin. Bayesian compressive sensing. IEEE Transactions on signal processing, 56(6):2346–2356, 2008.DOI
  15. [15]S Derin Babacan, Rafael Molina, and Aggelos K Katsaggelos. Bayesian compressive sensing using laplace priors. IEEE Transactions on image processing, 19(1):53–63, 2009.DOI
  16. [16]Dror Baron, Shriram Sarvotham, and Richard G Baraniuk. Bayesian compressive sensing via belief propagation. IEEE Transactions on Signal Processing, 58(1):269–280, 2009.arxiv.org/abs/0812.4627
  17. [17]Adrian S Lewis. Active sets, nonsmoothness, and sensitivity. SIAM Journal on Optimization, 13(3):702–725, 2002.DOI
  18. [18]Hans Joachim Ferreau, Hans Georg Bock, and Moritz Diehl. An online active set strategy to overcome the limitations of explicit mpc. International Journal of Robust and Nonlinear Control: IFAC-Affiliated Journal, 18(8):816–830, 2008.
  19. [19]Zaiwen Wen, Wotao Yin, Hongchao Zhang, and Donald Goldfarb. On the convergence of an active-set method for ℓ1 minimization. Optimization Methods and Software, 27(6):1127–1146, 2012.
  20. [20]Sidhant Misra, Line Roald, and Yeesian Ng. Learning for constrained optimization: Identifying optimal active constraint sets. INFORMS Journal on Computing, 34(1):463–480, 2022.
  21. [21]Charles J Geyer. Introduction to markov chain monte carlo. Handbook of markov chain monte carlo, 201160222(45):22, 2011.
  22. [22]B. W. Silverman. Density Estimation for Statistics and Data Analysis. Chapman and Hall, 1986.
  23. [23]Jan N Fuhg, Govinda Anantha Padmanabha, Nikolaos Bouklas, Bahador Bahmani, WaiChing Sun, Nikolaos N Vlassis, Moritz Flaschel, Pietro Carrara, and Laura De Lorenzis. A review on data-driven constitutive laws for solids. Archives of Computational Methods in Engineering, pages 1–43, 2024.
  24. [24]John M Ball. Convexity conditions and existence theorems in nonlinear elasticity. Archive for rational mechanics and Analysis, 63:337–403, 1976.
  25. [25]Vahidullah Tac, Francisco Sahli Costabal, and Adrian B Tepole. Data-driven tissue mechanics with polyconvex neural ordinary differential equations. Computer Methods in Applied Mechanics and Engineering, 398:115248, 2022.
  26. [26]Peiyi Chen and Johann Guilleminot. Polyconvex neural networks for hyperelastic constitutive models: A rectification approach. Mechanics Research Communications, 125:103993, 2022.
  27. [27]Faisal As’ad, Philip Avery, and Charbel Farhat. A mechanics-informed artificial neural network approach in data-driven constitutive modeling. International Journal for Numerical Methods in Engineering, 123(12):2738–2759, 2022.
  28. [28]Kailai Xu, Daniel Z Huang, and Eric Darve. Learning constitutive relations using symmetric positive definite neural networks. Journal of Computational Physics, 428:110072, 2021.
  29. [29]Dominik K Klein, Mauricio Fernández, Robert J Martin, Patrizio Neff, and Oliver Weeger. Polyconvex anisotropic hyperelasticity with neural networks. Journal of the Mechanics and Physics of Solids, 159:104703, 2022.DOI
  30. [30]Dominik K Klein, Fabian J Roth, Iman Valizadeh, and Oliver Weeger. Parametrized polyconvex hyperelasticity with physics-augmented neural networks. Data-Centric Engineering, 4:e25, 2023.DOI
  31. [31]Karl A Kalina, Philipp Gebhart, Jörg Brummund, Lennart Linden, WaiChing Sun, and Markus Kästner. Neural network-based multiscale modeling of finite strain magneto-elasticity with relaxed convexity criteria. Computer Methods in Applied Mechanics and Engineering, 421:116739, 2024.arxiv.org/abs/2308.16311
  32. [32]Jan N Fuhg, Nikolaos Bouklas, and Reese E Jones. Learning hyperelastic anisotropy from data via a tensor basis neural network. Journal of the Mechanics and Physics of Solids, 168:105022, 2022.
  33. [33]Jan N Fuhg, Lloyd van Wees, Mark Obstalekci, Paul Shade, Nikolaos Bouklas, and Matthew Kasemer. Machine-learning convex and texture-dependent macroscopic yield from crystal plasticity simulations. Materialia, 23:101446, 2022.
  34. [34]Brandon Amos, Lei Xu, and J Zico Kolter. Input convex neural networks. In International Conference on Machine Learning, pages 146–155. PMLR, 2017.arxiv.org/abs/1609.07152
  35. [35]Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.arxiv.org/abs/1912.01703
  36. [36]Peter J Huber. Robust estimation of a location parameter. In Breakthroughs in statistics: Methodology and distribution, pages 492–518. Springer, 1992.

Paper details

vlnc:W_j7kv7e54
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents