betaFeedback

WorksPhysics and Astronomy

CrysAtom: Distributed Representation of Atoms for Crystal Property Prediction

Abstract

Application of artificial intelligence (AI) has been ubiquitous in the growth of research in the areas of basic sciences. Frequent use of machine learning (ML) and deep learning (DL) based methodologies by researchers has resulted in significant advancements in the last decade. These techniques led to notable performance enhancements in different tasks such as protein structure prediction, drug-target binding affinity prediction, and molecular property prediction. In material science literature, it is well-known that crystalline materials exhibit topological structures. Such topological structures may be represented as graphs and utilization of graph neural network (GNN) based approaches could help encoding them into an augmented representation space. Primarily, such frameworks adopt supervised learning techniques targeted towards downstream property prediction tasks on the basis of electronic properties (formation energy, bandgap, total energy, etc.) and crystalline structures. Generally, such type of frameworks rely highly on the handcrafted atom feature representations along with the structural representations. In this paper, we propose an unsupervised framework namely, CrysAtom, using untagged crystal data to generate dense vector representation of atoms, which can be utilized in existing GNN-based property predictor models to accurately predict important properties of crystals. Empirical results show that our dense representation embeds chemical properties of atoms and enhance the performance of the baseline property predictor models significantly.

Introduction

In recent years, there has been a significant surge in applying machine learning (ML) algorithms across various disciplines, including material science and chemistry, where ML advancements are leveraged to address domain-specific challenges [1, 2, 3, 4], such as property prediction, molecule generation, and discovery of key descriptors for CO2\mathrm{CO}_{2} activation [5, 6]. Despite the reliance on density functional theory (DFT) simulations in early material science works [7], their resource-intensive nature prompted a shift towards ML-based strategies to replace high-latency computational processes with efficient approximations [8, 9]. ML techniques depend on handcrafted features, whereas deep learning algorithms learn feature representations, mitigating the need for domain-expert intervention [7]. In material science, crystal structures play a significant role for most of the downstream tasks [5]. Since majority of the crystals are available in nature as three-dimensional (3D) structures, they are

Preprint. Preliminary work. initially transformed into graphs by preserving their periodic invariance [5]. Such graphs are used in graph neural network based frameworks for solving different downstream property prediction tasks [5, 10, 11, 12].

For the downstream tasks, atoms are commonly initialized using a one-hot sparse representation, leading to suboptimal performance [9, 13]. In contrast, distributed representations encapsulate richer semantic and structural information [14, 15]. Atom2Vec [13] proposed a singular value decomposition (SVD) based distributed atom vector representation using handcrafted feature vectors, requiring domain knowledge. To alleviate this problem, SkipAtom [9] introduced a skip-gram [15] based dense representation of the atoms by learning the required feature representation from the dataset. However, neither consider structural representation of the crystal materials, that can be harnessed by neural network-based models for improving dense representations. To mitigate this challenge, we investigate the feasibility of utilizing the graph structure information into the neural network framework towards generating distributed atom vector representations. On this note, we propose a novel auto-encoder-decoder based framework, namely Crystal Atom Vector Extractor (CrysAtom), to learn distributed representations of molecular atoms (shown in Figure 1) by introducing a fusion mechanism by combining Self-Supervised Learning (SSL) and Unsupervised Learning (UL)-based techniques. The key distinction of our approach from existing work [16] lies in the adoption of a novel fusion mechanism, characterized by a generalized SSL loss for pretraining task. Unlike previous methods, our SSL techniques are uniquely generalized, requiring no external information such as space groups. Furthermore, we utilize our proposed distributed vector representation for different downstream property prediction tasks and analyze its performance.

Figure 1. Workflow showing the downstream property prediction tasks with the (a) application of the distributed vector representation obtained from CrysAtom framework, and (b) application of sparse atom vector representation.

Our Contributions.

  1. To the best of our knowledge, we are the first to investigate the feasibility of applying an auto-encoder-decoder based graph neural network approach to obtain a domain-independent generic distributed representation of atoms.

  1. We assess the quality of our distributed representation by comparing it with other existing state-of-the-art (SOTA) representations of atoms (Atom2Vec and SkipAtom) in multiple property prediction tasks.

  1. We use two popular benchmark materials datasets to show that our distributed representation of atoms obtained from CrysAtom helps gain substantial improvement in performance for several property predictor models (CGCNN, ALIGNN) over their vanilla (from 5.21% to 21.92%), distilled and fine-tuned versions.

  1. Moreover, the property-tagged dataset suffers from error bias, as it is theoretically derived from DFT. We successfully mitigate this issue using a small set of experimental data in the training setup.

Related Work

In recent times, data-driven approach specifically the graph neural network based frameworks [5, 17, 18] played a crucial role to conduct the property prediction task by utilizing the topological structures of the crystal materials. Earlier studies [5, 10, 19, 20, 21] did not comprise of the periodic invariance properties. Later, Matformer [11], applied a periodic graph transformer based framework by employing both periodic invariance and periodic pattern encoding strategy. Similarly, PotNet [12] used interatomic potentials for the property prediction tasks. Additionally, there are several works which employed UL [7, 16] and SSL [22, 23] strategies to apply the pretraining task followed by task-

NotationTerms
uuSource node
vvTarget node
ViV_iSet of nodes present in the unit cell
nuvn_{uv}No. of edges between uu and vv
EiE_iA multiset of node pairs
χi\chi_iNode features
FiF_iCollection of edge weights
ssLength of one bond
rrRadius
MMCross-correlation matrix
READOUTGlobal pooling function
RandomVector representations drawn from R\mathbb{R} using normal distribution
IITarget cross-correlation matrix
LBTL_{\mathrm{BT}}Barlow Twins Loss
LFRL_{\mathrm{FR}}Node feature reconstruction loss
LCRL_{\mathrm{CR}}Local connectivity reconstruction loss

caption: Important notations used in the paper.

:::

specific finetuning. All the discussed works employed sparse one-hot representations to initialize the node level feature representation. On this note, we propose our novel CrysAtom framework towards generating distributed atom vector representation by incorporating graph structure as the necessary input representation into the encoder module and also by introducing generalized unsupervised contrastive loss, which does not require any external information such as space group, as described in the work [16].

Methodology

This section commences by outlining the general idea of our proposed neural framework, namely CrysAtom, for generating dense vector representation of chemical atoms. Subsequently, a detailed discussion about the different components of our proposed framework follows. An overview of our proposed CrysAtom framework has been shown in Figure 4. Table 1 summarizes important terms and corresponding notations used in this work.

Figure 4. Schematic diagram illustrating the architecture of our proposed framework, CrysAtom, highlighting the key components. The diagram encapsulates the novel mechanisms underlying CrysAtom’s functionality.

CrysAtom

In this work, we propose a novel encoder-decoder based neural framework, CrysAtom, to generate dense vector representations2 of the chemical atoms, which can be used to enhance the performance of SOTA downstream neural property predictor models present in the literature of material science [5, 7, 11, 16]. To generate the dense vector representation, we first consider a collection of untagged crystal graphs Dut={Gi}D_{ut} = \{G_i\} collected from the well-known materials database, which serves as input to our proposed CrysAtom model (fθf_\theta). Similar to some of the state-of-the-art methods, popularly known for generating dense vector representations of atoms, such as SkipAtom and Atom2Vec, we use untagged data for generating dense vectors with better generalization capabilities. Subsequently, it generates the dense vector representation of the chemical atoms by learning the intrinsic structural and chemical patterns from the input representation of the crystal graphs. Additionally, to establish the effectiveness of the vector representations generated by CrysAtom, we conduct extensive theoretical and empirical analysis. For empirical analysis, we consider chemical property prediction, which is an important and challenging task in material science literature [7, 16].

Crystal Graph Representation. As proposed by Xie et. al. [5], we use crystal graph structures D={Gi=(Vi,Ei,χi,Fi)}D = \{G_i = (V_i, E_i, \chi_i, F_i)\} to represent crystalline materials. A crystal lattice is formed by repeating a unit cell in all three dimensions as shown in Figure 2. GiG_i is an undirected weighted multigraph that represents unit cell of the crystal structure. ViV_i denotes a set of nodes (atoms) present in the unit cell and Ei={(u,v,nuv)}E_i = \{(u, v, n_{uv})\} denotes a set of triples, where each triple consists of three entries - a pair of nodes and the number of edges between them. It signifies that vv is an atom that appears nuvn_{uv} times in the nearby cells around uu. The nearby cells are defined as the cells that are within a distance of radius rr from uu. Therefore, rr is a hyper-parameter for this model and the set of nearby cells may be expressed as,

Figure 2. This illustration showcases a periodic crystal structure, featuring a point cloud of atoms arranged in repeating patterns. The image includes a magnified view of a unit cell, clearly delineating the lattice vectors L=[l1,l2,l3]L = [l_1, l_2, l_3], highlighting the fundamental building blocks of the crystal’s geometric arrangement.

Nr(u)={v∈Vi∣dist⁡(u,v)≤r},\mathcal{N}_r(u) = \{v \in V_i \mid\operatorname{dist}(u,v) \le r\},

where ViV_i is the set of atoms present in the crystal graph GiG_i of crystal ii and Nr(u)\mathcal{N}_r(u) is the set of nearby cells for uu given rr. In the later part of the paper, we have often used the terms nodes and atoms interchangeably. The pictorial representation of creating multigraph from crystal structure has been shown in Figure 3. χi\chi_i represents node features i.e., features which comprehensively represent the chemical properties of an atom, such as atomic volume, electron affinity, etc. Finally, FiF_i represents a collection of edge weights between a pair of atoms in a crystal graph. In other words, Fi={{sn}(u,v)∣(u,v)∈Ei}F_i = \{ \{s^n\}_{(u,v)} \mid(u,v) \in E_i\} is a set that represents a collection of bond length values between each pair of nodes (u,v)(u,v) that are connected by an edge in EiE_i. The bond length is denoted as sns^n, where ss is the length of one bond and nn is the number of bonds (hence, the number of edges) between uu and vv. We consider bond length as a measure of distance from one atom to other atoms in close proximity. In the next section, we explain our proposed methodology for generating dense vector representation of atoms and subsequently analyze its effectiveness in property predictor models.

Figure 3. Illustration of the multigraph representation of a crystal. We use blue arrows, l1l_1 and l2l_2, for the crystal structure in 2D. We use circles of different colors to represent different atoms and green lines to denote periodic boundaries. We use Ni\mathcal{N}_i as the neighborhood set of node ii, and use yellow and blue arrows to specify the captured atomic interactions from the multigraph of the given crystal. (a) A crystal structure with periodic patterns l1l_1 and l2l_2 is shown in 2D. (b) Atomic interactions between the yellow nodes jj and the center red node ii, captured by the multigraph of the crystal. (c) The corresponding multigraph. All the periodic duplicates of jj in the crystal are mapped to a single node jj in the multigraph.

Atom vector formation

In this part, we discuss the architecture of the proposed CrysAtom model, as shown Figure 4. It consists of an auto-encoder with an SSL framework that leverages the correlations in the input to learn robust and generalizable dense vector representation of atoms.

Encoder. We develop our auto-encoder module by employing crystal graph convolutional neural network (CGCNN) [5]. We utilize CGCNN to encode the chemical and structural information of a crystal graph GG. It encodes information of the ll-hop neighborhood for each node by applying following equations:

z(u,v)nl−1=xul−1⊕xvl−1⊕s(u,v)n(1)z_{(u,v)_n}^{l-1}=x_u^{l-1}\oplus x_v^{l-1}\oplus s_{(u,v)}^n \tag*{(1)}
xul=xul−1+∑v,nσ(z(u,v)nl−1Θcl−1+bcl−1)⊙g(z(u,v)nl−1Θsl−1+bsl−1)x_u^l=x_u^{l-1}+\sum_{v,n}\sigma\left(z_{(u,v)_n}^{l-1}\Theta_c^{l-1}+b_c^{l-1}\right)\odot g\left(z_{(u,v)_n}^{l-1}\Theta_s^{l-1}+b_s^{l-1}\right)

where, ll is the number of CGCNN layers, xul−1x_u^{l-1} denotes the embedding of node uu by aggregating the l−1l-1 hop neighborhood information. The embedding of node uu is initialized to a transformed node feature vector, i.e., it is a function of the atom uu’s chemical features such as xu0=χuΘχx_u^0=\chi_u\Theta_\chi where Θχ\Theta_\chi is the list of trainable parameters of the transformation network and χu\chi_u is the input node feature vector. s(u,v)n∈Fus_{(u,v)}^n\in F_u represents the overall bond length between atoms uu and vv. The operator ⊕\oplus denotes concatenation and ⊙\odot denotes element-wise multiplication. Here, Θcl−1\Theta_c^{l-1}, Θsl−1\Theta_s^{l-1}, bcl−1b_c^{l-1}, bsl−1b_s^{l-1} are the convolution matrix, self-weight matrix, convolution bias, self-bias of (l−1)(l-1)th layer convolution, respectively. σ\sigma is a nonlinear transformation function, used to generate a real value in [0,1][0,1] indicating the edge importance and gg is a feed-forward network. Finally, we collect local information at each node after aggregating the information from the neighborhood (xulx_u^l). We denote the set of trainable parameters for this encoder as Θe\Theta_e for future reference.

Decoder. The encoder encodes the chemical properties of an atom into a latent vector space xx by learning the structural and chemical information. Subsequently, the decoder tries to decode the vectors from xx, thereby enhancing the encoding capability of the encoder. As mentioned earlier, the crystal properties depend on the local chemical environment and the overall conformation of the repeating crystal cell structure. Therefore, we build our decoder framework to reconstruct two important features to capture the properties of local chemical environment: (a) the node features, which are the chemical properties of individual atoms, and (b) the local connectivity, which is the relative position of the nodes with respect to their local neighbors. We employ the node feature reconstruction strategy by computing the following equations,

χ^u=ΘxTxul+bx(2)\hat{\chi}_u=\Theta_x^T x_u^l+b_x \tag*{(2)}
LFR=−χu⋅log⁡(χ^u)−(1−χu)⋅log⁡(1−χ^u)(3)\mathcal{L}_{FR}=-\chi_u\cdot\log(\hat{\chi}_u)-(1-\chi_u)\cdot\log(1-\hat{\chi}_u) \tag*{(3)}

where Θx\Theta_x, bxb_x and LFR\mathcal{L}_{FR} are the trainable weights, biases and the feature reconstruction loss, respectively.

Furthermore, we reconstruct the global topological information to generate the connectivity and the periodicity information of the crystal structures by employing a bilinear transformation strategy, with the help of the following equations,

xuvl=σ(xulΘblTxvl+bbl)(4)x_{uv}^l=\sigma(x_u^l\Theta_{bl}^T x_v^l+b_{bl}) \tag*{(4)}
LCR=arg⁡max⁡neβk(xuv,n)∑neβk(xuv,n)(5)\mathcal{L}_{CR}=\arg\max_n\frac{e^{\beta_k(x_{uv},n)}}{\sum_n e^{\beta_k(x_{uv},n)}} \tag*{(5)}

where Θbl\Theta_{bl}, bblb_{bl} and LCR\mathcal{L}_{CR} are the trainable weights and biases and connection reconstruction loss, respectively. The function σ\sigma is an activation function that maps the input to a value between 0 and 1. The output representation from the initial bilinear transformation layer (Θbl\Theta_{bl} and bblb_{bl}) is passed to another linear transformation layer followed by a softmax activation. The term βk\beta_k represents a feed-forward neural network with kk layers, which generates a logit vector of length nn (as mentioned previously, nn is the number of edges between two atoms) with the help of a softmax function. We

Algorithm 1 Training procedure and chemical vector extraction

Input: $\mathcal{D} = \{G_i = (V_i, E_i, \chi_i, F_i)\}$ dataset used for creating the chemical atom vector
Output: Generalized dense atom vector representations ($A$)

1 begin
2 Initialize $\Theta_e$ ▷ Parameters of the encoder
3 Initialize $\Theta_d$ ▷ Parameters of the decoder
4 $\mathcal{L}_{FR}$ ▷ Mean squared error loss (Equation 3)
5 $\mathcal{L}_{CR}$ ▷ Connection reconstruction loss (Equation 5)
6 $\mathcal{L}_{BT}$ ▷ Barlow Twins loss (Equation 7)
7 $A$ ▷ Set of atom vectors
8 $S$ ▷ Set of common atoms in one batch
9 $N$ ▷ Number of epochs
10 $G_j$ ▷ Batch graph (collection of 128 crystal structures)
11 $\mathcal{D}$ ▷ Graph data
12 for $i \leftarrow 1$ to $N$ do
13 for $G_j \in \mathcal{D}$ do
14 ▷ $j$-th batch with randomly selected 128 crystals
   $H_{emd} = f_e(G_j, \Theta_e)$
   ▷ $H_{emd}$: hidden representation of $G_j$
15 $A_{Femd}, A_{dje} = f_d(H_{emd}, \Theta_d)$
   ▷ $A_{Femd}$: output of the decoder, $A_{dje}$: constructed adjacency matrix
16 $G_1, G_2 \leftarrow G_j$
   ▷ $G_1, G_2$: augmented representations of $G_j$
17 $\mathcal{X}^{G_1}, \mathcal{X}^{G_2} = f_e(G_1, G_2)$
   ▷ $\mathcal{X}^{G_1}, \mathcal{X}^{G_2}$: augmented embeddings of $G_1, G_2$
18 $\mathcal{L}_j = \alpha\mathcal{L}_{FR} + \beta\mathcal{L}_{CR} + \gamma\mathcal{L}_{BT}$
19 $\Theta = \Theta_e \oplus \Theta_d$
20 $\Theta = \Theta - \alpha\nabla\mathcal{L}$
21 $A_j \leftarrow f_{Unbatch}(H_{emd})$
   ▷ $f_{Unbatch}$ generates atom representations
22 $\forall s \in S, S_j^s = 0, \mathit{counts}_s = 0$
   ▷ $S_j^s$: cumulative representation of atom $s$ in $A_j$
23 $\forall A_j^s \in A_j, \mathit{counts}_s ++, S_j^s = S_j^s + A_j^s$
24 $\forall s \in S, A_j^s = \frac{S_j^s}{\mathit{counts}_s}$
   ▷ Final representation of atom $s$ for batch $j$
25 if $\mathcal{L}_j \leq \mathcal{L}_{j-1}$ then
26 $A_j = A_j$
27 else
28 $A_j = A_{j-1}$
29 return $A$

denote the set of trainable parameters for this decoder as Θd\Theta_d for future reference.

SSL framework. Here, the SSL framework uses correlation between the actual input graph and its augmented versions to learn robust and generalized representation from the unlabeled data [23]. It provides an additional boost in terms of performance. We employ two types of augmentation techniques such as atom masking and edge masking. Atom masking randomly masks 10% of the atoms in the crystal, while edge masking randomly masks 10% of the edge features between adjacent atoms.

XG=READOUT(xul)(6)\mathcal{X}_G = READOUT(x_u^l) \tag*{(6)}

where READOUT interpretes to a global pooling function. Augmented graph representations are generated from the same crystalline structures. These augmented representations are used towards identity matrix formation. Given the problem formulation, we identified Barlow Twins [24] (BT) loss as a suitable loss function to reconstruct the graph representation of the crystals. Hereby, we utilize Barlow Twins loss function, which is based on the redundancy reduction principle by H. Barlow [25, 26]. We apply this loss function to the cross-correlation matrix that is formed from the embeddings produced by the encoder module of the auto-encoder.

LBT≜∑i(1−Mii2)⏟invariance term+λ∑i∑j≠iMij2⏟redundancy reduction term(7)\mathcal{L}_{BT} \triangleq\underbrace{\sum_i (1-\mathcal{M}_{ii}^{2})}_{\text{invariance term}}+\lambda\underbrace{\sum_i\sum_{j\ne i}\mathcal{M}_{ij}^{2}}_{\text{redundancy reduction term}} \tag*{(7)}
Mij≜∑bXb,iG1Xb,jG2(Xb,iG1)2(Xb,jG2)2(8)\mathcal{M}_{ij}\triangleq\frac{\sum_b \mathcal{X}_{b,i}^{G_1}\mathcal{X}_{b,j}^{G_2}}{\sqrt{\left(\mathcal{X}_{b,i}^{G_1}\right)^2}\sqrt{\left(\mathcal{X}_{b,j}^{G_2}\right)^2}} \tag*{(8)}

Equation 7 describes the Barlow Twins loss function, which is used in our SSL block. It considers the cross-correlation matrix M\mathcal{M} of embeddings from two augmented instances, which is computed by Equation 8. The parameter λ\lambda (set to 0.0051 in the original paper [24]) is a positive constant that balances the first and second terms of the loss function. In this study, we apply Equations (1) and (6), to obtain the two augmented embeddings, namely XG1\mathcal{X}^{G_1} and XG2\mathcal{X}^{G_2}. Furthermore, we utilize Equation (8) to derive the cross-correlation matrix, wherein bb represents the batch index and i,ji,j indicate the vector dimension of the projected output. Overall, the deep auto-encoder architecture is trained in an end-to-end fashion to optimize the loss function (Ltrain\mathcal{L}_{train}) as shown in Equation (9).

Ltrain=αLFR+βLCR⏟loss for UL+γLBT⏟loss for SSL(9)\mathcal{L}_{train}=\underbrace{\alpha\mathcal{L}_{FR}+\beta\mathcal{L}_{CR}}_{\text{loss for UL}}+\underbrace{\gamma\mathcal{L}_{BT}}_{\text{loss for SSL}} \tag*{(9)}

LFR\mathcal{L}_{FR}, LCR\mathcal{L}_{CR} are the reconstruction losses for node features and local connectivity, respectively. LBT\mathcal{L}_{BT} is the Barlow Twins loss and α,β,γ\alpha,\beta,\gamma are the weighting coefficients for each loss. The sequential steps for the training process and extraction of dense vector representation are stated in Algorithm 1. The number of parameters used by CrysAtom is 5.5 MB and the running time of each epoch of the training (using Algorithm 1) is approximately 30 minutes.

Atom Feature Vector Extraction. The atom vector extraction strategy in Algorithm 1 focuses on deriving atom feature vectors from the latent feature representations generated by the encoder module during each training epoch. For a batch graph GjG_j, the hidden representation HemdH_{emd} is obtained using the encoder function fef_e:

Hemd=fe(Gj,Θe)H_{emd}=f_e(G_j,\Theta_e)

Subsequently, the function fUnbatchf_{Unbatch} is applied to HemdH_{emd} to generate individual atom representation AjA_j.

Aj=fUnbatch(Hemd)A_j=f_{Unbatch}(H_{emd})

Here, fUnbatchf_{Unbatch} maps the latent feature vectors HemdH_{emd} to their corresponding atom feature vectors. If HemdH_{emd} is a hidden representation of size n×dn\times d, where nn is the number of nodes (atoms) and dd is the dimensionality of the feature vector, fUnbatchf_{Unbatch} effectively extracts the important features to generate the distributed vector representation for each atom.

For each atom type ss in the set of common atoms SS, the proposed algorithm computes a cumulative representation SjsS_j^s and calculates countsscounts_s as,

Sjs=∑i=1nI{Aji=s}⋅AjiS_j^s=\sum_{i=1}^{n}\mathbb{I}_{\{A_j^i=s\}}\cdot A_j^i
countss=∑i=1nI{Aji=s}counts_s=\sum_{i=1}^{n}\mathbb{I}_{\{A_j^i=s\}}

Here, I{Aji=s}\mathbb{I}_{\{A_j^i=s\}} is an indicator function that is 1 if the atom AjiA_j^i is of type ss and 0 otherwise. Lastly, the final representation for each atom type ss, in the batch jj, is computed by averaging the cumulative representations.

Ajs=∑i=1nI{Aji=s}⋅Aji∑i=1nI{Aji=s}A_j^s=\frac{\sum_{i=1}^{n}\mathbb{I}_{\{A_j^i=s\}}\cdot A_j^i}{\sum_{i=1}^{n}\mathbb{I}_{\{A_j^i=s\}}}

In this way, our algorithm ensures that the atom vectors are consistently updated and refined through the training epochs, ultimately storing the generalized dense atom vector representations in AA.

Downstream Property Prediction Task

The objective of this study is to integrate the proposed atom feature vectors into a SOTA property predictor model to enhance the performance of the downstream task. The following steps are used for applying the dense vector representations to the downstream property prediction tasks. We extract the atom vectors from our novel CrysAtom framework. Subsequently, we train SOTA property predictor model (PψP_{\psi}) by providing property-tagged training data Dt={Gi,yi}D_t = \{G_i, y_i\} as well as the generated feature vector as initial node feature representation. Here, we consider CGCNN [5], CrysXPP [7], ALIGNN [10], Matformer [11] and PotNet [12] as the baseline property predictors due to its SOTA performance in property prediction tasks. Training setup and hyper-parameter details are stated in Table 2.

Training Setup/ Hyper-parameter details/ Computational ResourcesValue
Convolution LayersFive convolution layers [5]
Epochs110
OptimizerAdam [27]
Learning Rate0.03
Embedding Dimension200 (can be 50 and 100-dimensional dense vectors)
Batch Size128
Weightage for α\alpha0.25
Weightage for β\beta0.25
Weightage for γ\gamma0.50 (for convex sum)
OptimizerAdam [27]
Epochs1000
Learning RateDefault learning rates used in the vanilla versions
Batch Size64
Seed Value123
Train, Valid, Test Splits80%, 10%, 10%
Implementation FrameworkPyTorch
Computational Resourcesone NVIDIA A6000 48GB GPU and one NVIDIA A100 80GB GPU

Table 2. Summary of hyper-parameter details in CrysAtom (top), CrysAtom variant of all SOTA models (middle) and computational resources (bottom) used in this work.

Results

In this section, we first describe the details of our dataset used in our experiments, and then we follow up with the research questions in context to the task of atom vector generation, and the analytical discussion towards addressing those research questions.

Datasets

We use 139K unlabeled crystal graphs from the Materials Project (MP)3 to obtain the required dense vector representation of atoms. For our downstream property prediction, as suggested by the Yan et. al [11], we consider the datasets MP 2018.6.13 and JARVIS-DFT 2021.8.124 [28], to investigate the chemical rationality of our proposed vector representations. MP 2018.6.1 contains 69,239 materials with four properties, formation energy, bandgap (OPT), bulk modulus (Kv) and shear modulus (Gv), whereas the JARVIS-DFT dataset contains 55,723 materials with seven properties such as formation energy, bandgap (OPT), total energy, ehull, bandgap (MBJ), bulk modulus (Kv) and shear modulus (Gv). Details of each of these datasets are given in Table 3 and details of this properties is provided in Table 4. Here, all these properties in Materials Project and JARVIS-DFT datasets are based on DFT calculations of crystal. To investigate how our dense vector representation mitigates DFT errors, we take a small dataset OQMD-EXP [29] containing 1,500 materials, consisting of experimental data for formation energy.

TaskDatasetsGraph Num.Structural Info.Properties CountData Type
Dense Vector CreationMP139K✓×DFT Calculated
Property PredictionMP 2018.6.169K✓4DFT Calculated
Property PredictionJARVIS-DFT55K✓7DFT Calculated
Property PredictionOQMD-EXP1.5K✓1Experimental

Table 3. Datasets used for both dense vector creation and downstream tasks.

PropertyUnitData-size
Formation EnergyeV/(atom)eV/(atom)69239
Bandgap (OPT)eVeV69239
Bulk Modulus (Kv)GPa5450
Shear Modulus (Gv)GPa5450
Formation EnergyeV/(atom)eV/(atom)55723
Bandgap (OPT)eVeV55723
Total_EnergyeV/(atom)eV/(atom)55723
EhulleVeV55371
Bandgap (MBJ)eVeV18172
Bulk Modulus (Kv)GPa19680
Shear Modulus (Gv)GPa19680

Table 4. Summary of different crystal properties in Materials Project (top) and JARVIS-DFT (bottom) datasets.

Research Questions

We pose a few important research questions (RQs), which are central to our research work.

RQ-1: Effectiveness of CrysAtom for downstream property prediction task. How effectively does CrysAtom aid the existing neural property predictors to achieve SOTA performance? Furthermore, to what extent does CrysAtom improve the performance of the distilled, pre-trained, and fine-tuned versions of the neural property predictors?

RQ-2: Robust dense vector representation. How does our dense vector representation CrysAtom fare when compared to the popular vector representations for atoms, such as, Atom2Vec or SkipAtom?

RQ-3: Removing DFT bias. How effectively does our proposed crystal atom vector representation mitigate DFT error bias, leading to SOTA results in existing neural property predictors?

RQ-4: Preserving periodic properties. How well does CrysAtom capture the periodic properties and the chemical significance of the atom? Does it also account for the elements displaying aberrant behavior with respect to its position in the periodic table, such as, Hydrogen (H), Helium (He)?

Discussions

Downstream Task Analysis. In relation to RQ-1, we compare five different SOTA frameworks for crystal property prediction such as CGCNN [5], CrysXPP [7], ALIGNN [10], Matformer [11] and PotNet [12]. To train these methods for property prediction, we use the 200-dimensional dense vectors obtained for each atom using Algorithm 1. These vectors serve as input atom features, which are initialized as non-trainable node features. For each property, we trained on 80%, validated on 10% and evaluated on 10% of the data.

In Table 5, we report MAE score (lower the MAE, higher the improvement) for the property prediction task. We observe that the SOTA models, when trained using CrysAtom generated vector representation as input, outperform their counterparts on the Materials Project and JARVIS-DFT datasets. Specifically, the average improvements of vanilla SOTA models such as CGCNN, CrysXPP, ALIGNN, Matformer, and PotNet are 21.92%, 23.40%, 8.63%, 6.49%, and 5.21%, respectively. These improvements are significant, considering the overall architecture of these property predictor models remain unchanged, while in the input space, we introduce CrysAtom generated feature vectors to train these models on various downstream tasks. Additionally, it is to be noted that the average relative improvement across all properties for ALIGNN (8.63%), Matformer (6.49%) and PotNet (5.21%) is lesser compared to CGCNN (21.92%) and CrysXPP (23.40%). The likely reason is that ALIGNN, Matformer, and PotNet are more complex models with higher parameter counts (97.8 MB for ALIGNN and 68.42 MB for Matformer) compared to CGCNN and CrysXPP. ALIGNN learns three-body interactions, Matformer captures periodic invariance, and PotNet incorporates interatomic potentials. Conversely, CGCNN and CrysXPP use simpler encoder architectures, primarily applying GCN layers to multi-graph crystal structures. Complex models often learn intrinsic features of crystal structures, so introducing atom vector representation alone doesn’t significantly enhance performance. In contrast, simpler models benefit more from dense features as inputs for downstream tasks. Another interesting observation from Table 5 is that all SOTA models achieve an average improvement of 17.03% in formation energy prediction, when trained using CrysAtom generated atom vector representation. This improvement is likely because the formation energy of a crystal, defined as the difference between the energy of a unit cell composed of NN chemical species and the sum of their chemical potentials (with units of eV/atomeV/atom), depends on its node features [16]. In contrast to that, However, the improvement is suboptimal for mechanical properties like bulk (9.99%) and shear modulus (7.18%), as these depend more on structural information such as lattice structure and symmetry [30] than on chemical properties.

PropertyCGCNNCGCNN (CrysAtom)CrysXPPCrysXPP (CrysAtom)ALIGNNALIGNN (CrysAtom)MatformerMatformer (CrysAtom)PotNetPotNet (CrysAtom)
Formation Energy0.0390.0280.0410.0300.0260.0230.0210.0190.0190.018
Bandgap (OPT)0.3880.2700.3470.2620.2710.2520.2110.2060.2040.193
Bulk Modulus (Kv)0.0540.0500.0800.0480.0510.0430.0430.0400.0400.038
Shear Modulus (Gv)0.0870.0820.1050.0820.0780.0720.0730.0710.0650.064
Formation Energy0.0630.0400.0620.0410.0330.0310.0330.0300.0290.028
Bandgap (OPT)0.2000.1430.1900.1420.1420.1330.1370.1350.1270.120
Total Energy0.0780.0430.0720.0440.0370.0350.0350.0310.0320.029
Ehull0.1700.1240.1390.1210.0760.0660.0640.0570.0550.049
Bandgap (MBJ)0.4100.3330.3780.3480.3100.2800.3000.2900.2700.240
Bulk Modulus (Kv)14.4712.3713.6113.1010.4010.1911.2110.8510.119.98
Shear Modulus (Gv)11.7510.4511.2010.449.869.3910.769.859.239.13

Table 5. Summary of the results (MAE) of different properties in Materials Project (top) and JARVIS-DFT (bottom). Model M is the vanilla variant of a SOTA model and M(CrysAtom) is a variant of the SOTA model with CrysAtom dense vectors as input. The best performance has been highlighted in bold.

Comparison with Existing Distilled and Pre-trained Models. To address the second part of RQ-1, we investigate the efficacy of utilizing a fixed atom feature vector representation in input space rather than applying resource intensive approaches such as knowledge distillation and task specific fine-tuning. As the encoder module of CGCNN, CrysGNN and CrysXPP are variants of GCN, we have shown performance comparison between these frameworks (as shown in Table 6), where we have used CrysAtom generated dense vector representation only for CGCNN. Additionally, we consider pre-trained GNN [31], which is widely used for pre-training property predictor models. We pre-train GNN [16, 31] on 800K untagged crystal data and fine-tune it on seven different properties, as shown in Table 6. For fine-tuned CrysGNN framework, we consider the pre-trained encoder of CrysGNN followed by a feed-forward neural network to predict a specific property. Similarly, for distilled CGCNN framework, we apply knowledge distillation using pre-trained CrysGNN model [16]. We observe that CGCNN (CrysAtom) outperforms fine-tuned CrysGNN, CrysXPP and Pre-trained GNN with a significant margin over all properties. We also notice that CGCNN (CrysAtom) outperforms distilled CGCNN by a large margin for formation energy, bandgap (OPT), total energy, bandgap (MBJ) and shear modulus. However, it produces comparable results for ehull and bulk modulus (Kv). The reason behind it could be use of a relatively small dataset of 139K untagged crystals to generate a fixed vector representation of atoms using CrysAtom model, whereas the pre-training of CrysGNN is done on a large dataset of size 800K.

PropertyCGCNN (CrysAtom)Distilled CGCNNFine-tuned CrysGNNCrysXPP Pre-trained GNN
Formation Energy0.040 (-14.9)0.0470.0560.0620.764
Bandgap (OPT)0.143 (-10.6)<u>0.160</u>0.1830.1900.688
Total Energy0.043 (-18.9)0.0530.0690.0721.451
Ehull0.124 (+2.4)0.1210.1300.1391.112
Bandgap (MBJ)0.333 (-2.1)0.3400.3710.3781.493
Bulk Modulus (Kv)12.37 (+0.5)12.3113.4213.6120.34
Shear Modulus (Gv)10.45 (-3.9)10.8711.0711.2016.51

Table 6. Comparison of prediction performance (MAE) for the seven properties in JARVIS-DFT between CrysAtom version of the CGCNN, Distilled CGCNN and other SOTA pre-trained/fine-tuned models. The best results have been shown in bold and the second best results have been underlined. Percentage of decrease in MAE for CrysAtom, with respect to the best performing model from the rest, has been mentioned within the bracket.

Comparison with Existing Dense Representations. In RQ-2, we compare performance of our dense vector representation, generated by CrysAtom, against the existing dense representations of atoms. Here, we have considered CGCNN as the encoder module to conduct necessary experiments. We train CGCNN using our 200-dimensional atom vector representations, including Random⁵, Atom2Vec and SkipAtom on seven properties from the JARVIS-DFT dataset and reported MAE values in Table 7. Our results show that CGCNN, when combined with CrysAtom, significantly outperforms SkipAtom, Atom2Vec, and Random aided versions across all the properties. The Random version performs the worst, as its dense representations fail to capture essential chemical features. The Atom2Vec version also underperforms, due to its inability to capture the topological complexity of crystal materials. The SkipAtom version, which captures atomic context better than SVD-based methods, performs slightly better than Atom2Vec but still falls short due to its limitations in comprehending complex structures.

PropertyCGCNN (CrysAtom)CGCNN (SkipAtom)CGCNN (Atom2Vec)CGCNN (Random)
Formation Energy0.040 (-34.4)0.0610.0700.075
Bandgap (OPT)0.143 (-27.0)0.1960.2510.263
Total Energy0.043 (-38.6)0.0700.0760.089
Ehull0.124 (-18.9)0.1530.1600.164
Bandgap (MBJ)0.333 (-20.7)0.4200.5290.569
Bulk Modulus (Kv)12.37 (-13.9)14.3615.4115.99
Shear Modulus (Gv)10.45 (-9.8)11.5812.0913.52

Table 7. Comparison of prediction performance (MAE) for the seven properties in JARVIS-DFT between variants of CGCNN with different dense vector representations, namely, CrysAtom, SkipAtom, Atom2Vec and Random. The best results have been shown in bold and the second best results have been underlined. Percentage of decrease in MAE for CrysAtom, with respect to the best performing model from the rest, has been mentioned within the bracket.

Removal of DFT Error Bias using CrysAtom. In this section we discuss RQ-3 to understand how we can remove DFT error bias using experimental data with the help of dense vector representations of atoms obtained from CrysAtom. One of the fundamental issues in material science is that the experimental data instances, as described in Section 4.1, for crystal properties are scarce [7]. Hence, existing SOTA models highly rely on DFT calculated data to train its parameters. However, mathematical approximations in DFT calculation lead to erroneous prediction (error bias) in contrast to the actual experimental data [16]. Hence, DFT error bias is a common problem present in the existing SOTA frameworks. Das et al. [7] have shown that pre-training plays a significant role in mitigating error bias when fine-tuned with experimental data. Consequently, we investigate whether DFT error bias in SOTA models can be reduced with the help of our novel atom vector representation, using a small set of experimental data instances. Here, we consider OQMD-EXP [29] dataset to conduct the relevant experiments for formation energy prediction task. We train all SOTA models and their CrysAtom variants with the complete DFT data in addition to a part of the experimental data. We report the MAE of different SOTA models and its CrysAtom variants in Table 8, where the evaluation is performed on gold standard experimental data. From Table 8, we can conclude that DFT bias error in the SOTA models have reduced with the application of CrysAtom generated vectors, consistently.

Experiment SettingsCGCNNCGCNN (CrysAtom)CrysXPPCrysXPP (CrysAtom)ALIGNNALIGNN (CrysAtom)MatformerMatformer (CrysAtom)PotNetPotNet (CrysAtom)
Train on DFT Test on Experimental0.2650.241 (-9.06)0.2430.222 (-8.6)0.2200.212 (-3.6)0.2180.213 (-2.3)0.2170.211 (-2.8)
Train on DFT and 20 % Experimental Test on 80 % Experimental0.1440.111 (-22.9)0.1380.115 (-16.7)0.0990.093 (-6.1)0.0980.094 (-4.1)0.0970.093 (-4.1)
Train on DFT and 80 % Experimental Test on 20 % Experimental0.0940.072 (-23.4)0.0870.071 (-18.4)0.0730.068 (-6.8)0.0720.069 (-4.2)0.0700.067 (-4.3)

Table 8. MAE of predicting experimental values by different SOTA models and their CrysAtom versions with full DFT data and different percentages of experimental data for formation energy in OQMD-EXP dataset. Percentage of decrease in MAE for CrysAtom is mentioned in bracket.

Preserving Periodic Properties of Elements using CrysAtom. In this section, we focus on RQ-4 to determine if the proposed dense representations embed equivalent atomic information. To understand the chemical significance of these vector representations, we visualize them using a lower-dimensional projection⁶. The illustration in Figure 5 significantly aids in qualitatively examining the chemical properties of the vector representations in relation to the periodic table5. Our illustration shows that group-I alkali metals (Li, Na, K, Rb, Cs) form a single cluster, indicating our vector representation captures their similarity in terms of chemical properties, such as high reactivity and single valence electron [33, 34, 35, 36]. Similarly, group-II alkaline earth metals (Ca, Sr, Mg, Ba) cluster together, indicating their lesser reactivity and electrical conductivity. Additionally, our representation, though not illustrating Be’s alkaline properties, captures its diagonal relationship with Al, indicating their tendency to form covalent bonds and tetrahedral structures [33, 34, 35, 36]. All reactive non-metals (C, N, O, F, Cl, Br, I, P, Se, and S) form a single cluster. This cluster spans groups VIA (halogens) and VIIA (chalcogens) of the periodic table due to their shared properties like reactivity and anion formation. Within this cluster, O and P are close in the embedding space, both binding with Hydrogen to form water and phosphine, respectively. Additionally, P and C exhibit a diagonal relationship [34, 36]. The figure shows that metalloid elements like Silicon (Si), Germanium (Ge), Boron (B), Antimony (Sb), and Tellurium (Te) cluster together, indicating that our vector representation captures their high semiconductivity and semi-metallic nature. Additionally, Boron and Silicon’s similarity, in terms of electronegativity, allows them to form covalent bonds, showcasing their diagonal relationship. Lead (Pb), Tin (Sn), and Thallium (Tl) also form a cluster, representing post-transition metals, with Pb and Sn in close proximity due to the inert pair effect. Our representation, groups all noble gases (Ne, Ar, Kr, Xe) and captures properties of transition metals (V, Cr, Fe, Co, Ni, Cu). Lanthanides and Actinides (e.g., La, Ce, Pr, Nd, Sm, Tb, Eu, Gd, Dy, Np, Ho, Pu, Pm, Pa, U, Th, Ac, Yb, Lu, Er) cluster due to their radioactive nature. Notably, Pb and Sn also exhibit diagonal relationship properties [33, 34, 35, 36].

Figure 5. Dimensionally-reduced atomic vectors were obtained from 200-dimensional vectors. Subsequently, these vectors were further reduced to two dimensions using t-SNE [32] for visualization. This plot shows an approximate position of the atomic vector representations of atoms5 in two dimensional Euclidean space.

Investigating the Positional Validity of Hydrogen and Helium. To investigate the second part of RQ-4, boundary cases are examined to illustrate the efficacy of the vector representation utilized in the study. Specifically, an evaluation is conducted to determine the precision of the vector representation in positioning Hydrogen (H) and Helium (He) atoms relative to their positions on the periodic table. The positioning of Hydrogen presents a contentious issue owing to its dual characteristics as both metals and non-metals, thus posing challenges in its classification within a particular metal group [36, 37]. Hydrogen’s electron configuration entails a single electron in its outermost shell, resembling alkali metals in group I, although it has the ability to generate a negative ion similar to halogens in group VIIA. Hence, Hydrogen is occasionally classified alongside alkali metals and at other times with halogens. Nevertheless, as depicted in Figure 5, Hydrogen does not align precisely with either group, underscoring its exceptional characteristics [36, 37]. Similarly, Helium, an inert gas possessing two electrons in its outer shell, diverges from the remaining noble gases in group VIIIA, and instead, it fits more appropriately within group II alongside the alkaline earth metals. The classification of Helium is further supported by Figure 5, which illustrates its unique properties [36].

Ablation Study

In this section, we demonstrate the effect of variation in the dimension of the dense vector representation on its capability to encode the chemical properties of an atom. We also analyze the influence of combining UL and SSL on CrysAtom performance and efficiency of different variants of CrysAtom aided SOTA models by designing the following set of ablation studies:

  1. Does increase in dimensions of the vector representation entail better performance in downstream property prediction task?

  1. Does hybrid learning strategy perform well when UL is aided with SSL?

  1. What is efficiency of different CrysAtom versions of SOTA models?

Impact of Increasing Dimensions. Results in Table 9 leads to the conclusion that increase in the dimension of the dense vectors improves the performance of the property predictor model (here, CGCNN). Our conclusion is aligned with earlier findings by Antunes et.al [9]. However, increasing dimension size of the vector representation beyond a certain point may lead to exponential growth in computational resources and may prove to be an impediment in terms of model training.

PropertyCGCNN (200-dim CrysAtom)CGCNN (100-dim CrysAtom)CGCNN (50-dim CrysAtom)
Formation Energy0.0400.0440.046
Bandgap (OPT)0.1430.1600.165
Total Energy0.0430.0490.053
Ehull0.1240.1270.130
Bandgap (MBJ)0.3330.3490.356
Bulk Modulus (Kv)12.3713.2913.60
Shear Modulus (Gv)10.4510.7510.99

Table 9. Performance comparison (MAE) of various versions of CGCNN equipped with different dimensional vector representations (such as 50-dim, 100-dim, and 200-dim) obtained from CrysAtom. The best results have been shown in bold and the second best results have been underlined.

Impact of Combining UL and SSL. Table 10 demonstrates how effectively we can combine our UL and SSL based approaches in JARVIS-DFT dataset for seven properties. We observe that combination of SSL and UL based frameworks equipped with Barlow Twins [24] loss helps the existing property predictors significantly to achieve SOTA performance on downstream task, when compared with NTXent [38] and VICREG [39] losses. The common intuition is that, while training a model using Barlow Twins loss, it generates the hard negative samples by masking the nodes and edges internally for a graph to capture the overall topological structure. Whereas, SSL loss functions such as NTXent, VICREG take necessary negative samples by considering external properties such as space group information. SSL strategies, in isolation, are not used for generative tasks, thereby preventing us from using such strategies for generation of dense vector representations. However, UL is applied together with SSL, leading to a generative loss for generating a dense vector representation as a subtask of graph generation.

PropertyUL + SSL (BT)UL + SSL (VICREG)UL + SSL (NTXent)UL
Formation Energy0.0400.0420.0420.043
Bandgap (OPT)0.1430.1560.1550.158
Total Energy0.0430.0450.0470.048
Ehull0.1240.1300.1260.127
Bandgap (MBJ)0.3330.3690.3400.345
Bulk Modulus (Kv)12.3713.2013.2013.24
Shear Modulus (Gv)10.4510.6510.6810.71

Table 10. Performance comparison of our CrysAtom (MAE) framework by introducing different SSL loss functions such as BT, VICREG, NTXent and UL (without using SSL loss). Here we use CGCNN as our encoder model to conduct the experiments. The best results have been shown in bold and the second best results have been underlined.

Efficiency of CrysAtom Variant of the SOTA Models. Table 11 shows comparison of CGCNN (CrysAtom), CrysXPP (CrysAtom), ALIGNN (CrysAtom), Matformer (CrysAtom) and PotNet (CrysAtom) in terms of training time per epoch, total training time, total testing time and model complexity for formation energy on JARVIS-DFT dataset. We clearly observe that CGCNN (CrysAtom) is the fastest model among all the CrysAtom variant of SOTA models. It uses less number of parameters compared to the other models. This led us to consider CGCNN as our encoder model to conduct the necessary experiments in ablation study.

MethodTime/EpochTotal Training TimeTotal Testing TimeModel Para.
CGCNN (CrysAtom)0.189 s3.12 h0.04 s1.1 MB
CrysXPP (CrysAtom)0.195 s3.25 h0.07 s1.1 MB
ALIGNN (CrysAtom)140.4 s39 h80.4 s97.8 MB
Matformer (CrysAtom)80.4 s22 h1.04 s68.42 MB
PotNet (CrysAtom)42 s11 h31s42.9 MB

Table 11. Training time per epoch, total training time, total testing time, and model complexity compared with CGCNN (CrysAtom), CrysXPP (CrysAtom), ALIGNN (CrysAtom), Matformer (CrysAtom) and PotNet (CrysAtom) for formation energy on JARVIS-DFT dataset.

Conclusion

In this study, we introduce a novel framework, CrysAtom, designed to create dense vector representations for crystal atoms. These vectors play a pivotal role for different graph neural network-based crystal property prediction tasks. Our approach uniquely combines Unsupervised Learning (UL) and Self-Supervised Learning (SSL) techniques to generate these dense representations. Additionally, we propose a novel way to extract the generalized feature vector representation from the latent space of the encoder module of CrysAtom framework. Our empirical results demonstrate that CrysAtom significantly enhances the performance of existing neural property predictors. Experiments show that our proposed framework generates a robust and unbiased dense vector representation for atoms, effectively capturing periodic properties and chemical significance of atoms. Future directions for extending our work could be incorporating many-body interactions as a part of the system, aiming to achieve performance improvements across complex state-of-the-art models.

References

  1. [1]John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Žídek, Anna Potapenko, et al. Highly accurate protein structure prediction with alphafold. Nature, 596(7873):583–589, 2021.DOI
  2. [2]Nongnuch Artrith, Keith T Butler, François-Xavier Coudert, Seungwu Han, Oxandr Isayev, Anubhav Jain, and Aron Walsh. Best practices in machine learning for chemistry. Nature chemistry, 13(6):505–508, 2021.DOI
  3. [3]Kamal Choudhary, Brian DeCost, Chi Chen, Anubhav Jain, Francesca Tavazza, Ryan Cohn, Cheol Woo Park, Alok Choudhary, Ankit Agrawal, Simon JL Billinge, et al. Recent advances and applications of deep learning methods in materials science. npj Computational Materials, 8(1):59, 2022.DOI
  4. [4]Shrimon Mukherjee, Madhusudan Ghosh, and Partha Basuchowdhuri. Deepglstm: deep graph convolutional network and lstm based approach for predicting drug-target binding affinity. In Proceedings of the 2022 SIAM International Conference on Data Mining (SDM), pages 729–737. SIAM, 2022.
  5. [5]Tian Xie and Jeffrey C Grossman. Crystal graph convolutional neural networks for an accurate and interpretable prediction of material properties. Physical review letters, 120(14):145301, 2018.DOI
  6. [6]Varadarajan Rengaraj, Sebastian Jost, Franz Bethke, Christian Plessl, Hossein Mirhosseini, Andrea Walther, and Thomas D Kühne. A two-step machine learning method for predicting the formation energy of ternary compounds. Computation, 11(5):95, 2023.DOI
  7. [7]Kishalay Das, Bidisha Samanta, Pawan Goyal, Seung-Cheol Lee, Satadeep Bhattacharjee, and Niloy Ganguly. Crysxpp: An explainable property predictor for crystalline materials. npj Computational Materials, 8(1):43, 2022.arxiv.org/abs/2104.10869
  8. [8]Daniel W Davies, Keith T Butler, Adam J Jackson, Andrew Morris, Jarvist M Frost, Jonathan M Skelton, and Aron Walsh. Computational screening of all stoichiometric inorganic materials. Chem, 1(4):617–627, 2016.pubmed.ncbi.nlm.nih.gov/27790643
  9. [9]Luis M Antunes, Ricardo Grau-Crespo, and Keith T Butler. Distributed representations of atoms and materials for machine learning. npj Computational Materials, 8(1):44, 2022.arxiv.org/abs/2107.14664
  10. [10]Kamal Choudhary and Brian DeCost. Atomistic line graph neural network for improved materials property predictions. npj Computational Materials, 7(1):185, 2021.DOI
  11. [11]Keqiang Yan, Yi Liu, Yuchao Lin, and Shuiwang Ji. Periodic graph transformers for crystal material property prediction. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.arxiv.org/abs/2209.11807
  12. [12]Yuchao Lin, Keqiang Yan, Youzhi Luo, Yi Liu, Xiaoning Qian, and Shuiwang Ji. Efficient approximations of complete interatomic potentials for crystal property prediction. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 21260–21287. PMLR, 23–29 Jul 2023.arxiv.org/abs/2306.10045
  13. [13]Quan Zhou, Peizhe Tang, Shenxiu Liu, Jinbo Pan, Qimin Yan, and Shou-Cheng Zhang. Learning atoms for materials discovery. Proceedings of the National Academy of Sciences, 115(28): E6411–E6417, 2018.DOI
  14. [14]Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781, 2013.arxiv.org/abs/1301.3781
  15. [15]Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532–1543, 2014.DOI
  16. [16]Kishalay Das, Bidisha Samanta, Pawan Goyal, Seung-Cheol Lee, Satadeep Bhattacharjee, and Niloy Ganguly. Crysgnn: Distilling pre-trained knowledge to enhance property prediction for crystalline materials. arXiv preprint arXiv:2301.05852, 2023.arxiv.org/abs/2301.05852
  17. [17]Steph-Yves Louis, Yong Zhao, Alireza Nasiri, Xiran Wang, Yuqi Song, Fei Liu, and Jianjun Hu. Graph convolutional neural networks with global attention for improved materials property prediction. Physical Chemistry Chemical Physics, 22(32):18141–18148, 2020.DOI
  18. [18]Anthony Yu-Tung Wang, Steven K Kauwe, Ryan J Murdock, and Taylor D Sparks. Compositionally restricted attention-based network for materials property predictions. Npj Computational Materials, 7(1):77, 2021.DOI
  19. [19]Chi Chen, Weike Ye, Yunxing Zuo, Chen Zheng, and Shyue Ping Ong. Graph networks as a universal machine learning framework for molecules and crystals. Chemistry of Materials, 31(9):3564–3572, 2019.
  20. [20]Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 15750–15758, 2021.arxiv.org/abs/2011.10566
  21. [21]Zhantao Chen, Nina Andrejevic, Tess Smidt, Zhiwei Ding, Qian Xu, Yen-Ting Chi, Quynh T Nguyen, Ahmet Alatas, Jing Kong, and Mingda Li. Direct prediction of phonon density of states with euclidean neural networks. Advanced Science, 8(12):2004214, 2021.
  22. [22]Hannes Stärk, Dominique Beaini, Gabriele Corso, Prudencio Tossou, Christian Dallago, Stephan Günnemann, and Pietro Liò. 3d infomax improves gnns for molecular property prediction. In International Conference on Machine Learning, pages 20479–20502. PMLR, 2022.arxiv.org/abs/2110.04126
  23. [23]Rishikesh Magar, Yuyang Wang, and Amir Barati Farimani. Crystal twins: self-supervised learning for crystalline material property prediction. npj Computational Materials, 8(1):231, 2022.arxiv.org/abs/2205.01893
  24. [24]Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. In International Conference on Machine Learning, pages 12310–12320. PMLR, 2021.arxiv.org/abs/2103.03230
  25. [25]Horace B Barlow et al. Possible principles underlying the transformation of sensory messages. Sensory communication, 1(01):217–233, 1961.
  26. [26]Horace Barlow. Redundancy reduction revisited. Network: computation in neural systems, 12 (3):241, 2001. 6pubmed.ncbi.nlm.nih.gov/11563528
  27. [27]Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014. 8arxiv.org/abs/1412.6980
  28. [28]Kamal Choudhary, Kevin F Garrity, Andrew CE Reid, Brian DeCost, Adam J Biacchi, Angela R Hight Walker, Zachary Trautt, Jason Hattrick-Simpers, A Gilad Kusne, Andrea Centrone, et al. The joint automated repository for various integrated simulations (jarvis) for data-driven materials design. npj computational materials, 6(1):173, 2020. 9DOI
  29. [29]Scott Kirklin, James E Saal, Bryce Meredig, Alex Thompson, Jeff W Doak, Muratahan Aykol, Stephan Rühl, and Chris Wolverton. The open quantum materials database (oqmd): assessing the accuracy of dft formation energies. npj Computational Materials, 1(1):1–15, 2015. 9, 11DOI
  30. [30]Peter T Bobrowsky and Brian Marker. Encyclopedia of engineering geology. Springer Berlin, 2018. 10
  31. [31]Weihua Hu, Bowen Liu, Joseph Gomes, Marinka Zitnik, Percy Liang, Vijay Pande, and Jure Leskovec. Strategies for pre-training graph neural networks. In International Conference on Learning Representations, 2020. 10arxiv.org/abs/1905.12265
  32. [32]Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008. 12
  33. [33]James E Huheey, Ellen A Keiter, Richard L Keiter, and Okhil K Medhi. Inorganic chemistry: principles of structure and reactivity. Pearson Education India, 2006. 12
  34. [34]Gary L Miessler. Inorganic chemistry. Pearson Education India, 2008. 12
  35. [35]John Emsley. Nature’s building blocks: an AZ guide to the elements. Oxford University Press, USA, 2011.
  36. [36]Eric Scerri. The periodic table: its story and its significance. Oxford University Press, 2019. 12
  37. [37]John S Rigden. Hydrogen: the essential element, 2003. 12
  38. [38]Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597–1607. PMLR, 2020. 13arxiv.org/abs/2002.05709
  39. [39]Adrien Bardes, Jean Ponce, and Yann LeCun. Vicreg: Variance-invariance-covariance regular-ization for self-supervised learning. arXiv preprint arXiv:2105.04906, 2021. 13arxiv.org/abs/2105.04906

Paper details

vlnc:W_x7jbnacm
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
  • Physics and Astronomy

Not recorded for this paper

View statement
Looking for related papers…
Contents