betaFeedback

WorksPhysics and Astronomy

Grand canonical generative diffusion model for crystalline phases and grain boundaries

Abstract

The diffusion model has emerged as a powerful tool for generating atomic structures for materials science. This work calls attention to the deficiency of current particle-based diffusion models, which represent atoms as a point cloud, in generating even the simplest ordered crystalline structures. The problem is attributed to particles being trapped in local minima during the score-driven simulated annealing of the diffusion process, similar to the physical process of force-driven simulated annealing. We develop a solution, the grand canonical diffusion model, which adopts an alternative voxel-based representation with continuous rather than fixed number of particles. The method is applied towards generation of several common crystalline phases as well as the technologically important and challenging problem of grain boundary structures.

Introduction

The ability to traverse and sample from the space of atomic structures is a ubiquitous and indispensable component of computational materials science. For example, to accelerate materials discovery, inverse design of materials starts with target material properties and seeks compositions and structures that satisfy the specifications [1]. High-throughput materials screening and data mining pipelines need to explore novel structures beyond known datasets to broaden their scope of search [2, 3]. Mesoscale or coarse-grained materials modeling, e.g. on materials strength, increasingly resort to atomistic simulations such as molecular dynamics on structures that correspond to a coarse-level configuration for improved parametrization and verification [4]. Property-prediction efforts such as interatomic potential development need to sample new structures outside the training set to ensure transferability of the trained models [5, 6]. While important and useful, this inverse mapping from property to structure can be more challenging than the straightforward forward map or structure-property relationships. The inverse map is often one-to-many, as there can be many configurations in the high-dimensional structure space that satisfy the desired criterion. It may even be ill defined if there is no such structure that exactly meets the goal. Well-known traditional approaches to generate atomic structures include molecular dynamics and Monte-Carlo to sample from a statistical distribution (often Boltzmann), and structure optimization/prediction methods, a special case for sampling low energy structures, such as simulated annealing [7], genetic algorithm [8, 9], and basin hopping [10]. They require knowledge of a potential energy surface to drive the sampling or optimization and are typically computationally expensive for large systems.

The rapid ascent of generative artificial intelligence techniques, especially probabilistic methods such as normalizing flow [11] and diffusion probabilistic model (DPM) [12, 13, 14], has brought fresh tools to the longstanding challenge of structure generation [15, 16, 17, 18]. Among its numerous achievements [19], the diffusion model has been applied to generating materials such crystal and molecular structures. While the initial efforts have been focused on generating molecules, especially organic molecules [20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31], recently there has been an increasing number of works targeting crystalline solids [32, 33, 34, 35, 36]. DPM formulates the inverse map as a well-defined conditional probabilistic problem: to generate/sample candidate structures conditioned on a given criterion. In the training phase, it learns the training dataset’s probability distribution, more specifically the score ∂xlog⁡P(x)\partial_x \log P(x) or the derivative of log probability with respect to (high dimensional) data xx. This is done through score matching, typically denoising score matching [37, 38]. In the inference phase, one integrates a diffusion equation starting from random initial points in the data space under the driving force of the learned score towards denoised outcomes according to the target distribution. This equation of motion can be either a deterministic ordinary differential equation (ODE) or a stochastic differential equation (SDE) formulated as a Langevin equation [14]. When it comes to structure generation with real atomic coordinates xx, the above process looks conspicuously similar to the well-known simulated annealing (SA) algorithm for simulating physical systems driven by atomic forces (rather than score) [7]. The similarity and connection to statistical mechanics were of course explicitly called out and cited as an aspiration to DPM [14, 38, 39].

Given SA’s crucial role in the diffusion model, it is interesting to investigate whether its limitations in physical simulation are also present in DPM. A well-known issue is that SA can get trapped in local minima and miss the long-range ordered crystalline phase as the correct global minimum, at least not without introducing defects in the sampled structures [40]. Indeed, we have observed the same problem in particle-based DPM. The failure was recently noted for crystalline diamond in Ref. [41]. Figure 1a shows a particle-based DPM trained on a simple cubic (SC) crystal structure fails to generate SC starting from random uniform positions, the asymptotic distribution of indistinguishable particle with large i.i.d. displacements ∼N(0,σ2)\sim\mathcal{N}(0,\sigma^2) according to the variance exploding noise schedule of DPM. In contrast, our grand canonical diffusion model was able to generate crystalline structures without any problem (details in Results), as shown in Figure 1b. Figure 2 quantitatively illustrates the effects of initial randomness on SC generation with this particle-based DPM. Initialized with up to 20% Gaussian displacements added to SC (left), it can successfully generate ordered SC evidenced by sharp radial distribution function (RDF) peaks (right). At 30% initial noise, structural defects emerge. With 50% or more, the SC structure is “forgotten”, the RDF is featureless and the initial structures are essentially random uniform (left), leading to substantial RDF broadening in the generated structures (right). In short, a particle DPM can generate ordered SC only when the cubic topology is present in the initial structure. Otherwise, it cannot tolerate initial randomness and gets trapped in local minima.

Comparison of diffusion models for simple cubic structures and grain boundary phase transition

Figure 1. Comparison of diffusion models for simple cubic structures: (a) particle-based generator with fixed number of particles, and (b) voxel-based grand canonical generator with continuously adjustable number of particles. (c) Grain boundary phase transition in tungsten Σ5(100)\Sigma5(100) twist boundary. The two GB phases with different atomic densities [n]=1/5,2/5[n] = 1/5, 2/5 are shown in blue and red, respectively. The bulk atoms of the misoriented upper and lower crystals are shown in gray.

Figure 2. Effects of different levels of randomness σ\sigma in initial structures on particle diffusion models. σ=0\sigma= 0 (≳0.5\gtrsim0.5) means unperturbed (random) initial structure. Radial distribution functions of simple cubic structure (lattice constant 1) averaged over ten 4×4×44 \times4 \times4 supercells are shown. Left: initial structures of ideal SC+Gaussian displacements N(0,σ2)\mathcal{N}(0,\sigma^2); right: generated with diffusion.

Grain Boundary Crystalline phases, if already known and available as a training set, are relatively simple. More interesting and technologically relevant are crystalline structures with defects. Most engineering materials are composed of many crystallites or grains separated by interfaces called grain boundaries (GBs), which are the primary goal of this paper (see Figure 1c for an example). These interfaces, inherited from synthesis and processing, greatly influence many important properties of polycrystalline materials including mechanical, electrical, thermal, corrosion resistance, diffusion, and optical properties. Recently, it has been recognized that even though GBs are effectively the disruptions of perfect crystalline order, they have a well-defined and ordered structure [42, 43]. The five microscopic degrees of freedom associated with the orientation of the abutting crystals make the space of possible GB structures extraordinarily vast [44]. However, even for a fixed crystal orientation and the GB plane, the interface structure can also exist in multiple states or GB phases characterized by the different atomic structures of the boundary. First-order GB phase transitions have been directly observed by high-resolution electron microscopy [45] and atomistic simulations [42]. Upon a transition, the boundary’s structure and properties change in a discontinuous manner which in turn leads to abrupt changes in properties of polycrystalline materials. Computational tools are needed that can efficiently predict the structure and properties of GBs to discover how to manipulate them with chemical composition, temperature, or pressure.

Even in the simplest elemental metals, GB structure prediction can be extremely challenging and require advanced grand canonical optimization [46]. GBs of the same orientation can exist in many metastable states of different relative translations and different number of atoms separated by large energy barriers, which prevents the interface from finding its ground state on the short timescale of simulations or even experiments. It is generally insufficient to simply take two perfect half crystals and join them together to obtain the correct ground state. Instead, advanced structure sampling methods such as genetic algorithms [46] and simulated annealing [43] have been applied to predict GBs by inserting or removing atoms from the GB core followed by atomic position optimization. These computationally intense algorithms require systematic exploration of a complex structural space, and their efficiency can be improved tremendously by proposing better initial candidate structures and proposing intelligent mutation moves aimed at improving the current configurations [46]. Such searches explore different metastable states and often generate hundreds or thousands of candidate structures before they find the ground state. This generated rich information about the different GB atomic environments is usually not utilized in the follow-up searches, and for every new boundary, the search is performed from scratch. It is well known, however, that many different boundaries are not completely different since they may be composed of the same dislocations with the same dislocation core structure [42, 46, 47]. Therefore, it is desirable to develop AI algorithms that could learn the GB structural space to accelerate structure searches and identify hidden relations for GBs. Given the aforementioned difficulty of generating simple crystalline phases, GBs with both ordered and disordered atoms are expected to be challenging.

The main contributions of this paper are

  • To call attention to the deficiency of current particle-based diffusion models in generating even the simplest ordered crystalline structures, and to explain the reason for the failure.

  • To provide a solution—the grand canonical diffusion model—with an alternative voxel-based representation and variable number of particles.

  • To apply the model for generation of grain boundary structures, which are important in many materials problems.

Method

Related work: Voxel-based generative model for atomic structure

Pinheiro et al designed a voxel-based model for molecule generation [48]. There are a few important distinctions. First, ref. [48] did not use a diffusion model. Second, Pinheiro et al employed a discrete grid and involved an extra step to convert the grid to atomic coordinates. Here, we have smeared the positions, which leads to the ability to sample continuous coordinates, and we focus on solid materials.

Particle-based DPM and GNN score model

We adopted a baseline particle-based GNN diffusion model from Ref. [41]. Throughout this work we are concerned with single-element materials, and we will regard particle/atom type encoding as fixed and optional. The variance-exploding noise schedule was used, under which perturbed atomic coordinates approach a random uniform distribution. This enables one to start the denoising trajectory from a random structure sampled from a uniform distribution over the coordinate space. Attempts were made to improve structure sampling and avoid local minima, including testing both the ODE and the SDE, increasing the integration steps to 1000, and the restart sampling algorithm [49], but local minima persisted. More details can be found in the Appendix.

Grand canonical voxel diffusion model

Particle-to-voxel encoder. An encoder EN\mathrm{EN} converts an input structure or point cloud R={r}R = \{r\} into a voxel array C=EN(R)C = \mathrm{EN}(R) by smearing the coordinates of each particle (Figure 3a). One assumes NN particles, with coordinate raαr_{a\alpha} for particle (atom) aa in direction α=1,2,3\alpha= 1, 2, 3, and spatial partitions {VI}\{V_I\} with each partition VIV_I representing a region of the simulation cell Ω=⋃IVI\Omega= \bigcup_I V_I without overlap (VI∩VJ=∅V_I \cap V_J = \varnothing for I≠JI \ne J). Here, each partition VI=Vijk={r∣r∈V,(i−12)L≤r1<(i+12)L,…}V_I = V_{ijk} = \{r \mid r \in V, (i - \frac{1}{2})L \le r_1 < (i + \frac{1}{2})L, \ldots\} is a cube of linear size LL centered at (iL,jL,kL)(iL, jL, kL). Together they form a regular grid. Each particle is smeared from a point or delta distribution into a uniform distribution U(r−ra)U(r-r_a) within a size-LL cube, and contribute P(a,I)P(a,I) to partition VIV_I. P(a,I)P(a,I) is the proportion or relative volume of smeared “cubic” particle aa within VIV_I,

Grand canonical diffusion model with voxel representation

Figure 3. Grand canonical diffusion model with voxel representation. (a) Particle to voxel encoder by smearing using linear weights and no learnable parameters. For clarity a 2D slice is also shown with red dots designating particle positions. (b) Learnable voxel to particle coordinate decoder. Both the training and inference stages are shown. (c) Voxel-space diffusion using a convolutional U-Net.

P(a,I)=∫VIU(r−ra) dr=∏α=13P(raα,Iα),(1)P(a,I) = \int_{V_I} U(r-r_a)\,\mathrm{d}r = \prod_{\alpha=1}^{3} P(r_{a\alpha}, I_\alpha), \tag*{(1)}
P(r,i)={1−∣r−iL∣/L,if ∣r−iL∣/L<10,otherwiseP(r,i) = \begin{cases} 1-\lvert r-iL\rvert/L, & \text{if } \lvert r-iL\rvert/L < 1 \\ 0, & \text{otherwise} \end{cases}

The above formulation is trivially generalized for periodic boundary conditions, which are often adopted for crystal structures. If a particle is exactly at the center of a partition, it contributes wholly 1 to that partition, as shown in the lower left corner of 2D inset in Figure 3a. More generally, each particle is smeared into up to 8 (4) neighbor voxels in 3D (pixels in 2D). The total voxel array CC is summed over all particles

CI=∑aP(a,I).(2)C_I = \sum_a P(a,I). \tag*{(2)}

To avoid associating any voxel with more than one particle and obscuring the voxel-based generator and the decoder, LL is preferably a small fraction of the shortest bond distance. There is a compromise in the choice of LL: smaller LL means less smearing and more precise locations, but more voxels and larger memory footprint, which becomes a problem for large 3D cells.

Our encoder design, with a grid-size hyperparameter LL and no learnable parameters, encodes the particle coordinates continuously into the voxel array CC, as is evident in Eqs. (1),(2). This allows for the voxel-space generator to learn on a continuous voxel representation without discontinuity.

Additionally, a data-augmentation step is adopted to enforce translational equivariance of our approach. By construction the encoder is not exactly translationally equivariant: if all atoms are translated by integer multiplets of LL, then the array CC is the same after translation. Otherwise, translation by fractional multiplets of LL does modify CC, as seen in the 2D inset in Figure 3a. In the training step random translation vectors from a uniform distribution U(0,L)U(0,L) are sampled. The same data-augmentation step is adopted throughout the paper to train the decoder and the voxel diffusion model.

Voxel-to-particle decoder Since the coordinates were continuously encoded into voxel values, a dedicated decoder DE\mathrm{DE} based on a 3D convolutional neural network (CNN) was trained to map voxel arrays into coordinates (Figure 3b). It consists of three modules, each being a single-layer ConvNext model [50] with 333^3 filters and 64 hidden features: a denoiser DEden\mathrm{DE}_{\mathrm{den}} to reduce noise in the input voxel array, a binary classifier DEcls\mathrm{DE}_{\mathrm{cls}} to predict whether a voxel is associated with or occupied by any particle, and a relative coordinate regression model DEcor\mathrm{DE}_{\mathrm{cor}} that predict the cartesian coordinate of any associated particle. The decoder was trained with three corresponding loss functions (lower part of Figure 3b). First, we introduce a voxel value denoising loss inspired by denoising score matching [37]

Lden=∥EN(R)−DEden(EN(R+σ))∥22,(3)L_{\mathrm{den}} = \lVert\mathrm{EN}(R) - \mathrm{DE}_{\mathrm{den}}(\mathrm{EN}(R+\sigma)) \rVert_2^2, \tag*{(3)}

where structure RR is sampled from the same training atomic structures as the generator model, and σ∼N(0,σmax⁡2)\sigma\sim\mathcal{N}(0,\sigma_{\max}^2) is i.i.d. Gaussian noise of magnitude σmax⁡\sigma_{\max}. In the decoder training stage, random noises are added to the encoded (smeared) voxel configuration, and the denoising module DEden\mathrm{DE}_{\mathrm{den}} and loss LdenL_{\mathrm{den}} help with the quality and stability of decoding. Second, a classification loss employing binary cross entropy

Lcls=BCE(EN(R)>0,DEcls(DEden(EN(R)+σ)))(4)L_{\mathrm{cls}} = \mathrm{BCE}(\mathrm{EN}(R)>0,\mathrm{DE}_{\mathrm{cls}}(\mathrm{DE}_{\mathrm{den}}(\mathrm{EN}(R)+\sigma))) \tag*{(4)}

helps identify the voxels with particle occupancy in them, i.e. non-vanishing and substantial overlap P(a,I)P(a,I). The classifier was trained against the ground truth logical label EN(R)>pthre\mathrm{EN}(R)>p_{\mathrm{thre}}. Up to 8 voxels can be partially occupied by a particle and the occupancies are 1/8 in the most even case, so the threshold pthrep_{\mathrm{thre}} for substantial overlap was deliberately chosen to a small 0.05. Thirdly, a coordinate reconstruction loss

Lcor=∥int⁡(DEcls(DEden(EN(R)+σ)))(Δr−DEcor(DEden(EN(R)+σ)))∥22(5)L_{\mathrm{cor}} = \left\lVert\operatorname{int}\left(\mathrm{DE}_{\mathrm{cls}}\left(\mathrm{DE}_{\mathrm{den}}\left(\mathrm{EN}(\mathcal{R})+\sigma\right)\right)\right)\left(\Delta r-\mathrm{DE}_{\mathrm{cor}}\left(\mathrm{DE}_{\mathrm{den}}\left(\mathrm{EN}(\mathcal{R})+\sigma\right)\right)\right)\right\rVert_2^2 \tag*{(5)}

recovers the coordinate of the potential particle relative to the partition’s center. Δr\Delta r is the relative displacement between the associated particle and center of a partition. Partitions without a predicted overlapping particle are disregarded in this loss by int, which converts true/false to 1/0. The total loss is

L=ER,σ,t[(Lden+Lcls+Lcor)|R+t,σ],(6)L = \mathbb{E}_{\mathcal{R},\sigma,t}\left[\left(L_{\mathrm{den}}+L_{\mathrm{cls}}+L_{\mathrm{cor}}\right)\middle|\mathcal{R}+t,\sigma\right], \tag*{(6)}

where vector t\mathbf{t} is sampled from uniform tα∼U(0,L)t_{\alpha}\sim U(0,L) to translate all atoms in each structure R\mathcal{R} such that the smeared voxels are approximately equivariant under translation. For inference (upper part of Figure 3b), the input voxel array C\mathcal{C} is first denoised into C′=DEden(C)\mathcal{C}^{\prime}=\mathrm{DE}_{\mathrm{den}}(\mathcal{C}), partitions with particles are identified by the classifier DEcls(C′)\mathrm{DE}_{\mathrm{cls}}(\mathcal{C}^{\prime}), and then coordinates of a particle is found for each identified partition ijkijk as (iL,jL,kL)+DEcor(C′)(iL,jL,kL)+\mathrm{DE}_{\mathrm{cor}}(\mathcal{C}^{\prime}). This algorithm tend to conservatively return more than one closely distributed coordinates for each atom due to smearing. Finally, duplicates were removed by a refinement process, which finds the cluster of coordinates based on the euclidean distance between pairs and return the averaged coordinate for each cluster. A visualization of the decoder’s training and inference can be found in Figure 3b.

Voxel-space diffusion model. With the encoder and decoder to map between particles and voxels, a voxel-space diffusion model was trained that adopts a convolutional U-Net, largely following Ref. [51]. To treat 3D voxel data, 2D operators such as convolution, pooling, normalization, up-sampling and down-sampling were replaced with their 3D counterparts. Periodic boundary conditions were adopted for crystalline phases, while special hybrid boundary conditions were used for GBs, which were periodic in the xyxy directions but capped with vacuum in both ends of the zz direction. Therefore we applied periodic padding in xyxy and zero padding in zz. The SiLU activation function [52] and variance-preserving noise schedule were adopted. We scaled voxels C←C/s\mathcal{C}\leftarrow\mathcal{C}/s by s=0.02s=0.02, which is approximately the standard deviation of the smeared voxel values, most of which are zero. The solver3 variant of DPM-Solver [53] was employed with 60 function evaluations.

Model training. All learnable networks were trained with an AdamW optimizer [54] with weight decay of 2×10−52\times10^{-5} and learning rate 3×10−43\times10^{-4}. The learning rate was halved upon validation loss plateau until it reaches a minimum of 10−510^{-5}. A single Nvidia Tesla V100 GPU is used for training.

Grain boundary structure dataset

In this work we investigate a representative high-angle, high-energy Σ5(100)\Sigma5(100) twist boundary in refractory metal tungsten (W). Tungsten was chosen as a representative body-centered cubic (BCC) material and it is also a leading candidate for plasma-facing material (PFM) in fusion energy devices [55]. Tungsten GBs are known to be prone to brittle fracture [56], which motivates their investigation in this work. The twist boundary was obtained by a 36.87° rotation of the (100)-oriented W grains around a common [1] axis with the GB plane perpendicular to the axis of rotation. The interaction between the tungsten atoms were modeled using an embedded-atom method (EAM) potential by Zhou, et al. [57]. GB structure search was performed using GRand canonical Interface Predictor (GRIP) [58]. During the structure search, the tool systematically explores all the available GB microscopic degrees of freedom which include relative grain translations and different atomic densities at the boundary core by deleting and inserting a certain number of GB atoms to minimize the GB energy. The GB energy, EgbE_{\mathrm{gb}}, is calculated as:

Egb=(Etotalgb−NtotalgbEcohbulk)/Aplanegb,(7)E_{\mathrm{gb}}=\left(E_{\mathrm{total}}^{\mathrm{gb}}-N_{\mathrm{total}}^{\mathrm{gb}}E_{\mathrm{coh}}^{\mathrm{bulk}}\right)/A_{\mathrm{plane}}^{\mathrm{gb}}, \tag*{(7)}

where EtotalgbE_{\mathrm{total}}^{\mathrm{gb}} is the total energy of the region containing the GB, EcohbulkE_{\mathrm{coh}}^{\mathrm{bulk}} is the energy per atom in the perfect W BCC crystal, and AplanegbA_{\mathrm{plane}}^{\mathrm{gb}} is the GB area. The total number of nonequivalent atomic densities that should be sampled is given by the number of atoms in one bulk atomic plane parallel to the boundary because a complete removal of one plane of atoms coupled with the relevant translation is guaranteed to return the boundary to an identical state. As a result, this number of GB atoms, [n][n], is measured as a fraction of the total number of atoms in one bulk atomic plane. If NtotalN_{\mathrm{total}} is the total number of atoms in the bulk crystal and NplanebulkN_{\mathrm{plane}}^{\mathrm{bulk}} is the total number of atoms in one bulk atomic plane, the fraction of GB atoms is

[n]=(Ntotal mod⁡ Nplanebulk)/Nplanebulk∈[0,1)(8)[n] = \left(N_{\mathrm{total}} \ \operatorname{mod}\ N_{\mathrm{plane}}^{\mathrm{bulk}}\right) / N_{\mathrm{plane}}^{\mathrm{bulk}} \in[0,1) \tag*{(8)}

The results of GB search with GRIP are summarized in Figure 6. Each blue point corresponds to EgbE_{\mathrm{gb}} for a given atom fraction [n][n]. Here we generated 300 distinct structures with different energy and [n][n]. The search reveals two minima at [n]=1/5[n] = 1/5 and [n]=2/5[n] = 2/5 which correspond to the ground state and the metastable phase of this boundary, respectively (Figure 1c). A previous study [47] demonstrated first-order structural phase transitions between these two GB phases at finite temperature when point defects were inserted in the ground-state (Figure 1c). All other structures generated by GRIP have higher energy. Some of them are distorted versions of the two phases containing point defects, while other boundaries have completely different structures. While they are all mechanically stable at 0 K, at finite temperature they are expected to evolve toward the two GB phases shown in Figure 1c.

Grain boundary energy plot and exemplar structures

Figure 6. Grain boundary energy after relaxation and exemplar structures. (a) GB energy is plotted versus the number of atoms [n][n] in the boundary core. GB structure search using GRIP tool (blue) and the developed diffusion model (red) both predict the ground state at [n]=1/5[n] = 1/5 and the second metastable phase at [n]=2/5[n] = 2/5 shown in Figure 1c. Diffusion model is trained with GRIP generated structures with EgbE_{\mathrm{gb}} in the range of 2.6 J m−2≤Egb≤2.8 J m−22.6\ \mathrm{J\,m^{-2}} \le E_{\mathrm{gb}} \le2.8\ \mathrm{J\,m^{-2}}. (b) An exemplary structure used in training, the DPM generated structure and predicted relaxed ground state structure.

Metrics of generated structures. To quantitatively characterize the generated structures, we use the Steinhardt order parameter [59] as the atomic descriptors, as well as radial distribution functions. For GBs, we focus on the energy EgbE_{\mathrm{gb}}, as the goal is a generative model that can learn the complex GB atomic environments from a training set and efficiently predict new GB candidates that will be relaxed by energy minimization, omitting the need for costly global optimization. This final minimization step is always necessary as generation of the exact ground state is unrealistic.

Experiments

Simple ordered crystalline structures

Separate particle- and voxel-based DPM models were trained for each of simple cubic (SC), FCC and BCC phases with one ideal 4×4×44 \times4 \times4 conventional cell as the training set. LL was chosen to partition the supercell into 24324^3, 20320^3 and 24324^3 voxels for SC, BCC and FCC, respectively. Figure 4 provides the ground truth and grand canonical generated (without relaxation) crystal structures for SC, BCC and FCC phases. The Steinhardt order parameters and radial distributions both show that the generated structures are very close to the reference structure. In contrast, the particle-based baseline DPM produce structures that deviate significantly (purple points and curves).

Figure 4. Ideal and generated structure for SC, BCC and FCC phases, and the corresponding Steinhardt order parameters and radial distribution functions averaged over ten 434^3 cells.

Conditional grain boundary generation by in-painting

To develop the grand canonical diffusion model for GB, we first smeared the W atoms using cube size L=0.834L = 0.834 Å, resulting in a 12×12×7212 \times12 \times72 voxel array per structure. Then we trained the voxel-space diffuser, a 3D U-Net consisting of 2 levels and 3 residual convolutional blocks per level. The training lasts 30,000 steps with a batch size of 4.

During inference, we adopt a conditional generation strategy by in-painting. Concretely, the initial input to the diffusion model is a voxel array encoded from perfect W BCC crystal grains on both ends of zz direction, each taking 12×12×1812 \times12 \times18 voxels and kept fixed during diffusion (Figure 5). The middle of the array is subject to diffusion through the simulated annealing process to generate voxels in the GB region. Finally, the whole array is decoded into particles as described previously.

Figure 5. Conditional grain boundary generation via in-painting.

Trained on the full GB dataset, our model can successfully generate the low energy states (details in Appendix Fig. S.3). To show that it can not only memorize but also predict unseen structures, we built a training set consisting of GB structures with 2.6 J m−2≤Egb≤2.8 J m−22.6\ \mathrm{J\,m^{-2}} \leq E_{\mathrm{gb}} \leq2.8\ \mathrm{J\,m^{-2}}, shown in Figure 6 as solid blue circles within the two dashed lines. The remaining data points are not used for training, depicted as blue open circles. Candidate GB structures were generated by the trained diffusion model and relaxed with conjugate gradient minimization using LAMMPS [60]. The convergence criteria are 10−510^{-5} for relative energy and 10−5 eV A˚−110^{-5}\ \mathrm{eV\,\mathring{A}^{-1}} for forces, with a maximum of 10510^5 evaluations. Note that since the model is grand canonical, it automatically generates candidate structures with different [n]∈[0,1)[n] \in[0,1).

The results of the GB structure prediction are shown as red dots in Figure 6a, next to the data points generated by GRIP. Remarkably, the diffusion model search predicted both the ground state at [n]=1/5[n] = 1/5 as well as the metastable GB phase [n]=2/5[n] = 2/5 (see Figure 1c), which were excluded from the training set. Moreover, within the entire interval of the GB atomic densities, the distribution of generated EgbE_{\mathrm{gb}} for each fraction [n][n] concentrates near the lowest energy configurations, illustrating that the generative model produces better GB candidates than those optimized by GRIP. Figure 6b illustrates examples of a high energy, metastable GB structure from training set with Egb=2.649 J m−2E_{\mathrm{gb}} = 2.649\ \mathrm{J\,m^{-2}} and [n]=3/10[n] = 3/10, a GB candidate structure as generated by the diffusion model, and the ground state predicted by the search with the diffusion model. While the metastable structure and the ground state are not identical, the diffusion model learns from the metastable configurations enough to generate a realistic GB structure closely resembling the ground state, making the structure prediction possible.

Discussions

In conclusion, we presented an alternative diffusion model with a voxel representation with continuously adjustable number of particles. Structures were sampled using simulated annealing in the space of voxel values, whereby creation and annihilation of particles occurs in a grand canonical ensemble, rather than moving a fixed number of particles in real coordinate space. Our method circumvents the challenge of global optimization of atomic structures directly. Instead, particles can be added or deleted continuously. The grand canonical diffusion model was tested successfully on crystalline phases including simple cubic, face centered cubic, and body centered cubic, as well as grain boundary structures in tungsten. Ground state GB structures withheld from the model can be discovered, suggesting that the grand canonical diffusion model was able to learn generalizable structural features with both order and disorder. Future work will explore application of our method for materials design and generative modeling of different types of grain boundaries, other defects such as dislocations, and bulk phases in alloys.

Limitations. The presented method should be considered a first step towards generative models of crystalline or crystalline + defect structures in a grand canonical setting. There are a few notable technical limitations. (1) The adopted CNN architecture is only applicable to orthorhombic unit cells, and the lattice constants have to be integer multiples of the voxel linear dimension LL chosen at training time. (2) Since the data representations are voxel values rather than particle coordinates, conditional generation is less straightforward than in a particle representation. Usually a guiding likelihood P(y∣r)P(y|r) assumes known particle coordinates rr, and the noising voxel configuration VtV_t needs to be decoded first to coordinates rt=D(Vt)r_t = D(V_t), which is not well defined for t>0t > 0. (3) Translational equivariance is valid only at integer multiples of LL but rather enforced through data augmentation during training. Rotational equivariance is not attempted in this work and can likewise be introduced approximately. (4) The current method takes many voxels to represent each particle and has a big memory footprint. Future work is needed to resolve or mitigate such issues, such as adoption of more flexible score model architectures that allows different unit cell shapes and dimensions, vision transformers for conditional generation, equivariant constructs, and latent diffusion methods that are more memory efficient. The developed method can be used in areas such as materials design and grain boundary engineering. Any negative societal impact associated with those applications can be applied to our method.

Acknowledgements

This work was performed under the auspices of the U.S. Department of Energy by Lawrence Livermore National Laboratory under Contract DE-AC52-07NA27344. This work was funded by the Laboratory Directed Research and Development (LDRD) Program at LLNL under project tracking code 22-ERD-016 and 23-SI-006. Computing support for this work came from the Lawrence Livermore National Laboratory institutional computing facility. TF was supported by the U.S. DOE, Office of Science under an Office of Fusion Energy Sciences Early Career Award. Computing support for this work came from the Lawrence Livermore National Laboratory Institutional Computing Grand Challenge program.

Appendix / supplemental material

Figure S.1. Carbon structure generation by simulated annealing using pretrained particle-based diffusion model from Ref. [41]. Amorphous carbon (a), and diamond starting from diamond + larger (b) or small (c) perturbations. Amorphous structures have been successfully generated with random initial positions and real-space simulated annealing. Diamond can only be generated starting with slightly perturbed diamond, while initialization with large perturbations or random positions lead to significant amount of defects.

Details about particle-based GNN score model. The GNN score model sθ(rt,t)s_\theta(r_t,t) is trained with dataset consisted of crystalline structures containing atomic coordinates r0r_0 and unit cell dimensions A\mathbf{A}. The score is learned via the denoising score matching loss

rt=αtr0+σtϵ, ϵ∼N(0,I),r_t = \alpha_t r_0 + \sigma_t \epsilon,\ \epsilon\sim\mathcal{N}(0,\mathbf{I}),
LDSM=Er0,ϵ,t[∥sθ(rt,t)σt+ϵ∥2].(9)\mathcal{L}_{\mathrm{DSM}} = \mathbb{E}_{r_0,\epsilon,t}\left[\left\|s_\theta(r_t,t)\sigma_t+\epsilon\right\|^2\right]. \tag*{(9)}

where A\mathbf{A} is implicitly accounted for by the atomic graph representation. We apply the variance-exploding noise schedule such that αt\alpha_t is fixed to 1 and σt\sigma_t linearly increases from 0 (at t=0t=0) to some maximum value kk (at t=1t=1), which is set to 0.4 in this work. Under this variance-exploding scheme, we observe that the perturbed atomic coordinates Rt\mathbf{R}_t approach a random uniform distribution. Therefore, this enables one to start the denoising trajectory from a random structure sampled from a uniform distribution over the coordinate space.

The GNN architecture consists of three components: an Encoder, a Processor, and a Decoder. The input atomic graph is denoted as (V,E)(V,E), where VV is a set of node features {hi}\{h_i\} for node/atom ii, and EE is a set of edge/bond features {eij}\{e_{ij}\} from node ii to node jj. The set of edges EE is determined by a cutoff distance rcr_c, within which a pair of nodes form an edge connection.

The Encoder is responsible for embedding node and edge features for the subsequent Processor operation. The node features are embedded as

hi←MH(onehot⁡(zi))+MT(GaussianRFF⁡(t)),(10)h_i \leftarrow M_H(\operatorname{onehot}(z_i)) + M_T(\operatorname{GaussianRFF}(t)), \tag*{(10)}

where MHM_H and MTM_T are multilayer perceptrons (MLPs), and GaussianRFF is a shortname for the Gaussian Fourier features [61]. Note that the atom type information zz is optional in this work because we only focus on single-element systems in this paper. Regardless, we define the Encoder as above so that it can accommodate multi-element systems. The edge features eije_{ij} are embedded as

eij←ME(rij⊕rˉij),(11)e_{ij} \leftarrow M_E(r_{ij} \oplus\bar{r}_{ij}), \tag*{(11)}

where MEM_E is an MLP, rijr_{ij} are edge vectors computed as rj−ri\mathbf{r}_j-\mathbf{r}_i, rˉij\bar{r}_{ij} are edge lengths, and ⊕\oplus denotes concatenation. MHM_H, MEM_E, and MTM_T consist of two dense linear layers, with SiLU activation function after the first layer and layer normalization after the second.

The Processor is responsible for graph message-passing (convolutions). We used the Processor from MeshGraphNets [62], which we refer to for further details of the model. Note that the original work considers two sets of edges (mesh-space edges and world-space edges), whereas there is only one in our work.

Voxel representation of crystalline phases

Figure S.2. Voxel representation of crystalline phases: simple cubic (top row), BCC (middle) and FCC (bottom row). Left column: Encoded voxel arrays of ideal structures. Right: generated configurations by grand canonical voxel-based DPMs separately trained on crystalline structure.

The Decoder is responsible for mapping node features to score outputs. We use a simple MLP MOM_{O} as the Decoder:

y^i←MO(hi).(12)\hat{y}_{i} \leftarrow M_{O}(h_{i}). \tag*{(12)}

MOM_{O} has the same architecture as that of MHM_{H}, MEM_{E}, and MTM_{T}, except that there is no layer normalization at the end.

Scatter plot of grain boundary energy and three exemplar structures labelled Training E = 2.297, Generated, and Gen Relaxed E = 2.296

Figure S.3. Same as Figure 6 but using all ground truth structures without holdout in training the grand canonical DPM. Grain boundary energy after relaxation (left) and exemplar structures (right).

References

  1. [1]Sanchez-Lengeling, B. & Aspuru-Guzik, A. Inverse molecular design using machine learning: Generative models for matter engineering. Science 361, 360–365 (2018).
  2. [2]Fischer, C. C., Tibbetts, K. J., Morgan, D. & Ceder, G. Predicting crystal structure by merging data mining with quantum mechanics. Nature Materials 5, 641–646. http://www.nature.com/doifinder/10.1038/nmat1691 (2006).pubmed.ncbi.nlm.nih.gov/16845417
  3. [3]Jain, A. et al. Commentary: The Materials Project: A materials genome approach to accelerating materials innovation. APL Materials 1, 110002. http://link.aip.org/link/AMPADS/v1/i1/p011002/s1&Aggoi (2013).DOI
  4. [4]Bertin, N., Bulatov, V. V. & Zhou, F. Learning dislocation dynamics mobility laws from large-scale MD simulations. arXiv: 2309.14450. http://arxiv.org/abs/2309.14450 (Sept. 2023).
  5. [5]Deringer, V. L., Caro, M. A. & Csányi, G. Machine Learning Interatomic Potentials as Emerging Tools for Materials Science. Advanced Materials 31, 1–16. ISSN: 15214095 (2019).DOI
  6. [6]Mishin, Y. Machine-learning interatomic potentials for materials science. Acta Materialia 214, 116980. ISSN: 13596454. arXiv: 2102.06163. https://doi.org/10.1016/j.actamat.2021.116980 (2021).
  7. [7]Pannettier, J., Bassas-Alsina, J., Rodriguez-Carvajal, J. & Caignaert, V. Prediction of crystal structures from crystal chemistry rules by simulated annealing. Nature 346, 343–345. ISSN: 0028-0836. https://www.nature.com/articles/346343a0 (July 1990).
  8. [8]Deaven, D. M. & Ho, K. M. Molecular geometry optimization with a genetic algorithm. Physical Review Letters 75, 288–291. ISSN: 00319007. arXiv: 9506004 [mtrl-th] (1995).pubmed.ncbi.nlm.nih.gov/10059656
  9. [9]Oganov, A. R., Lyakhov, A. O. & Valle, M. How evolutionary crystal structure prediction works-and why. Accounts of Chemical Research 44, 227–237. ISSN: 00014842 (2011).DOI
  10. [10]Wales, D. J. & Doye, J. P. K. Global optimization by basin-hopping and the lowest energy structures of Lennard-Jones clusters containing up to 110 atoms. Journal Of Physical Chemistry A 101, 5111–5116 (1997).DOI
  11. [11]Altieri, N. & Duvenaud, D. Variational Inference with Gradient Flows. NIPS Workshop 37, 3–6 (2015).DOI
  12. [12]Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N. & Ganguli, S. Deep unsupervised learning using nonequilibrium thermodynamics in International Conference on Machine Learning (2015), 2256–2265.
  13. [13]Ho, J., Jain, A. & Abbeel, P. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems 33, 6840–6851 (2020).arxiv.org/abs/2006.11239
  14. [14]Song, Y. et al. Score-based generative modeling through stochastic differential equations. arXiv:2011.13456 (2020).arxiv.org/abs/2011.13456
  15. [15]Noh, J. et al. Inverse design of solid-state materials via a continuous representation. Matter 1, 1370–1384 (2019).DOI
  16. [16]Yao, Z. et al. Inverse design of nanoporous crystalline reticular materials with deep generative models. Nature Machine Intelligence 3, 76–86 (2021).DOI
  17. [17]Kim, B., Lee, S. & Kim, J. Inverse design of porous materials using artificial neural networks. Science advances 6, eaax9324 (2020).DOI
  18. [18]Noh, J., Gu, G. H., Kim, S. & Jung, Y. Machine-enabled inverse design of inorganic solid materials: promises and challenges. Chemical Science 11, 4871–4881 (2020).DOI
  19. [19]Yang, L. et al. Diffusion models: A comprehensive survey of methods and applications (2022).arxiv.org/abs/2209.00796
  20. [20]Hoogeboom, E., Satorras, V. G., Vignac, C. & Welling, M. Equivariant diffusion for molecule generation in 3d in International conference on machine learning (2022), 8867–8887.arxiv.org/abs/2203.17003
  21. [21]Jing, B., Corso, G., Chang, J., Barzilay, R. & Jaakkola, T. Torsional diffusion for molecular conformer generation. Advances in Neural Information Processing Systems 35, 24240–24253 (2022).arxiv.org/abs/2206.01729
  22. [22]Xu, M. et al. Geodiff: A geometric diffusion model for molecular conformation generation. arXiv: arxiv: 2203.02923.arxiv.org/abs/2203.02923
  23. [23]Wu, K. E. et al. Protein structure generation via folding diffusion. Nature Communications 15, 1059. ISSN: 2041-1723. arXiv: 2209.15611. http://arxiv.org/abs/2209.15611%20https://www.nature.com/articles/s41467-024-45051-2 (Feb. 2024).
  24. [24]Weiss, T. et al. Guided Diffusion for Inverse Molecular Design (2023).DOI
  25. [25]Duan, C., Du, Y., Jia, H. & Kulik, H. J. Accurate transition state generation with an object-aware equivariant elementary reaction diffusion model. Nature Computational Science 3, 1045–1055. ISSN: 2662-8457. http://arxiv.org/abs/2304.06174%20https://www.nature.com/articles/s43588-023-00563-7 (Dec. 2023).
  26. [26]Ingraham, J. B. et al. Illuminating protein space with a programmable generative model. Nature, 2022.12.01.518682. ISSN: 0028-0836. https://www.nature.com/articles/s41586-023-06728-8 (Nov. 2023).
  27. [27]Watson, J. L. et al. De novo design of protein structure and function with RFdiffusion. Nature. ISSN: 1476-4687. http://www.ncbi.nlm.nih.gov/pubmed/37433327 (2023).
  28. [28]Lu, J., Zhong, B. & Tang, J. Score-based Enhanced Sampling for Protein Molecular Dynamics, 1–26. arXiv: 2306.03117. http://arxiv.org/abs/2306.03117 (2023).
  29. [29]Corso, G., Stärk, H., Jing, B., Barzilay, R. & Jaakkola, T. DiffDock: Diffusion Steps, Twists, and Turns for Molecular Docking, 1–33. arXiv: 2210.01776. http://arxiv.org/abs/2210.01776 (2022).
  30. [30]Jing, B. et al. EigenFold: Generative Protein Structure Prediction with Diffusion Models, 1–10. arXiv: 2304.02198. http://arxiv.org/abs/2304.02198 (Apr. 2023).
  31. [31]Hsu, T., Sadigh, B., Bulatov, V. & Zhou, F. Score Dynamics: Scaling Molecular Dynamics with Picoseconds Time Steps via Conditional Diffusion Model. Journal of Chemical Theory and Computation 20, 2335–2348. ISSN: 1549-9618. https://pubs.acs.org/doi/10.1021/acs.jctc.3c01361 (Mar. 2024).
  32. [32]Xie, T., Fu, X., Ganea, O.-E., Barzilay, R. & Jaakkola, T. Crystal diffusion variational autoencoder for periodic material generation. eprint: arXiv:2110.06197 (2021).arxiv.org/abs/2110.06197
  33. [33]Jiao, R. et al. Crystal structure prediction by joint equivariant diffusion. Advances in Neural Information Processing Systems 36 (2024).arxiv.org/abs/2309.04475
  34. [34]Zeni, C. et al. Mattergen: a generative model for inorganic materials design. arXiv preprint arXiv:2312.03687 (2023).
  35. [35]Yang, M. et al. Scalable diffusion for materials generation. arXiv preprint arXiv:2311.09235 (2023).arxiv.org/abs/2311.09235
  36. [36]Pakornchote, T. et al. Diffusion probabilistic models enhance variational autoencoder for crystal structure generative modeling. Scientific Reports 14, 1275 (2024).
  37. [37]Vincent, P. A connection between score matching and denoising autoencoders. Neural computation 23, 1661–1674 (2011).
  38. [38]Song, Y. & Ermon, S. Generative modeling by estimating gradients of the data distribution in NeurIPS 32 (2019). arXiv: 1907.05600.
  39. [39]Dockhorn, T., Vahdat, A. & Kreis, K. Score-Based Generative Modeling With Critically-Damped Langevin Diffusion. ICLR 2022 - 10th International Conference on Learning Representations. arXiv: 2112.07068 (2022).arxiv.org/abs/2112.07068
  40. [40]Bollweg, W., Maurer, H. & Kroll, H. in Developments in Global Optimization (eds Bomze, I. M., Csendes, T., Horst, R. & Pardalos, P. M.) 253–288 (Springer US, Boston, MA, 1997). ISBN: 978-1-4757-2600-8. https://doi.org/10.1007/978-1-4757-2600-8_16.
  41. [41]Kwon, H. et al. Spectroscopy-Guided Discovery of Three-Dimensional Structures of Disordered Materials with Diffusion Models. arXiv: arXiv:2312.05472.arxiv.org/abs/2312.05472
  42. [42]Frolov, T., Olmsted, D. L., Asta, M. & Mishin, Y. Structural phase transformations in metallic grain boundaries. Nature Communications 4, 1–7 (2013).DOI
  43. [43]von Alfthan, S., Haynes, P. D., Kaski, K. & Sutton, A. P. Are the Structures of Twist Grain Boundaries in Silicon Ordered at 0 K? Physical Review Letters 96, 055505 (2006).pubmed.ncbi.nlm.nih.gov/16486949
  44. [44]Sutton, A. P. & Balluffi, R. W. Interfaces in Crystalline Materials 1st. ISBN: 978-0-19-921106-7. https://global.oup.com/academic/product/interfaces-in-crystalline-materials-9780199211067?cc=us&lang=en&# (Oxford University Press, New York, 1995).
  45. [45]Meiners, T., Frolov, T., Rudd, R. E., Dehm, G. & Liebscher, C. H. Observations of grain-boundary phase transformations in an elemental metal. Nature 579, 375–378 (2020).DOI
  46. [46]Zhu, Q., Samanta, A., Li, B., Rudd, R. E. & Frolov, T. Predicting Phase Behavior of Grain Boundaries with Evolutionary Search and Machine Learning. Nature Communications 9, 467 (2018).DOI
  47. [47]Frolov, T. et al. Grain boundary phases in bcc metals. Nanoscale 10, 8253–8268 (2018).DOI
  48. [48]Pinheiro, P. O. et al. 3D molecule generation by denoising voxel grids in NeurIPS (2023).arxiv.org/abs/2306.07473
  49. [49]Xu, Y. et al. Restart Sampling for Improving Generative Processes. arXiv preprint arXiv:2306.14878 (2023).arxiv.org/abs/2306.14878
  50. [50]Liu, Z. et al. A ConvNet for the 2020s in CVPR (2022). arXiv: 2201.03545. http://arxiv.org/abs/2201.03545.
  51. [51]Saharia, C. et al. Palette: Image-to-Image Diffusion Models in Proceedings of ACM SIGGRAPH (2021). ISBN: 978145031432. arXiv: 2111.05826. http://arxiv.org/abs/2111.05826.
  52. [52]Hendrycks, D. & Gimpel, K. Gaussian Error Linear Units (GELUs). arXiv: 1606.08415. http://arxiv.org/abs/1606.08415 (2016).
  53. [53]Lu, C. et al. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. Advances in Neural Information Processing Systems 35, 5775–5787 (2022).arxiv.org/abs/2206.00927
  54. [54]Loshchilov, I. & Hutter, F. Decoupled weight decay regularization. 7th International Conference on Learning Representations, ICLR 2019. arXiv: 1711.05101 (2019).arxiv.org/abs/1711.05101
  55. [55]Marian, J. et al. Recent Advances in Modeling and Simulation of the Exposure and Response of Tungsten to Fusion Energy Conditions. Nuclear Fusion 57, 092008 (2017).DOI
  56. [56]Talignani, A. et al. A Review on Additive Manufacturing of Refractory Tungsten and Tungsten Alloys. Additive Manufacturing 58, 103009 (2022).DOI
  57. [57]Zhou, X., Johnson, R. A. & Wadley, H. N. G. Misfit-energy-increasing dislocations in vapor-deposited CoFe/NiFe multilayers. Physical Review B 69, 144113 (2004).DOI
  58. [58]Chen, E., Heo, T. W., Wood, B. C., Asta, M. & Frolov, T. Grand Canonically Optimized Grain Boundary Phases in Hexagonal Close-Packed Titanium 2024. arXiv: 2404.04230.DOI
  59. [59]Steinhardt, P. J., Nelson, D. R. & Ronchetti, M. Bond-orientational order in liquids and glasses. Physical Review B 28, 784 (1983).DOI
  60. [60]Thompson, A. P. et al. LAMMPS - a Flexible Simulation Tool for Particle-Based Materials Modeling at the Atomic, Meso, and Continuum Scales. Computer Physics Communications 271, 108171 (2022).DOI
  61. [61]Tancik, M. et al. Fourier features let networks learn high frequency functions in low dimensional domains. Advances in Neural Information Processing Systems 33, 7537–7547 (2020).arxiv.org/abs/2006.10739
  62. [62]Pfaff, T., Fortunato, M., Sanchez-Gonzalez, A. & Battaglia, P. W. Learning mesh-based simulation with graph networks. eprint: arXiv:2010.03409 (2020).arxiv.org/abs/2010.03409

Paper details

vlnc:W_pgta9j2b
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Physics and Astronomy

Not recorded for this paper

View statement
Looking for related papers…
Contents