betaFeedback

WorksComputer Science

EquiformerV2: Improved Equivariant Transformer for Scaling to Higher-Degree Representations

Abstract

Equivariant Transformers such as Equiformer have demonstrated the efficacy of applying Transformers to the domain of 3D atomistic systems. However, they are limited to small degrees of equivariant representations due to their computational complexity. In this paper, we investigate whether these architectures can scale well to higher degrees. Starting from Equiformer, we first replace SO(3)SO(3) convolutions with eSCN convolutions to efficiently incorporate higher-degree tensors. Then, to better leverage the power of higher degrees, we propose three architectural improvements – attention re-normalization, separable S2S^2 activation and separable layer normalization. Putting this all together, we propose EquiformerV2, which outperforms previous state-of-the-art methods on large-scale OC20 dataset by up to 9%9\% on forces, 4%4\% on energies, offers better speed-accuracy trade-offs, and 2×2\times reduction in DFT calculations needed for computing adsorption energies. Additionally, EquiformerV2 trained on only OC22 dataset outperforms GemNet-OC trained on both OC20 and OC22 datasets, achieving much better data efficiency. Finally, we compare EquiformerV2 with Equiformer on QM9 and OC20 S2EF-2M datasets to better understand the performance gain brought by higher degrees.

Introduction

In recent years, machine learning (ML) models have shown promising results in accelerating and scaling high-accuracy but compute-intensive quantum mechanical calculations by effectively accounting for key features of atomic systems, such as the discrete nature of atoms, and Euclidean and permutation symmetries ([21, 69, 25, 17, 3, 35, 60, 52, 41, 30]). By bringing down computational costs from hours or days to fractions of seconds, these methods enable new insights in many applications such as molecular simulations, material design and drug discovery. A promising class of ML models that have enabled this progress is equivariant graph neural networks (GNNs) ([55, 63, 28, 16, 3, 4, 33, 39] (Musielewicz et al. 2022)).

Equivariant GNNs treat 3D atomistic systems as graphs, and incorporate inductive biases such that their internal representations and predictions are equivariant to 3D translations, rotations and optionally inversions. Specifically, they build up equivariant features of each node as vector spaces of irreducible representations (or irreps) and have interactions or message passing between nodes based on equivariant operations such as tensor products. Recent works on equivariant Transformers, specifically Equiformer ([33]), have shown the efficacy of applying Transformers ([61]), which have previously enjoyed widespread success in computer vision ([6, 11, 56]), language ([10, 5]), and graphs ([13, 66, 50] (Kreuzer et al. 2021)), to this domain of 3D atomistic systems.

A bottleneck in scaling Equiformer as well as other equivariant GNNs is the computational complexity of tensor products, especially when we increase the maximum degree of irreps LmaxL_{max}. This limits these models to use small values of LmaxL_{max} (e.g., Lmax≤3L_{max} \le3), which consequently limits their performance.

Higher degrees can better capture angular resolution and directional information, which is critical to accurate prediction of atomic energies and forces ([3, 70, 39]). To this end, eSCN (Passaro & Zitnick [39]) recently proposes efficient convolutions to reduce SO(3)SO(3) tensor products to SO(2)SO(2) linear operations, bringing down the computational cost from O(Lmax⁡3)O(L_{\max}^{3}) to O(Lmax⁡2)O(L_{\max}^{2}) and enabling scaling to larger values of Lmax⁡L_{\max} (e.g., Lmax⁡L_{\max} up to 8). However, except using efficient convolutions for higher Lmax⁡L_{\max}, eSCN still follows SEGNN-like message passing network (Brandstetter et al. [4]) design, and Equiformer has been shown to improve upon SEGNN. More importantly, this ability to use higher Lmax⁡L_{\max} challenges whether the previous design of equivariant Transformers can scale well to higher-degree representations.

In this paper, we are interested in adapting eSCN convolutions for higher-degree representations to equivariant Transformers. We start with Equiformer (Liao & Smidt [33]) and replace SO(3)SO(3) convolutions with eSCN convolutions. We find that naively incorporating eSCN convolutions does not result in better performance than the original eSCN model. Therefore, to better leverage the power of higher degrees, we propose three architectural improvements – attention re-normalization, separable S2S^{2} activation and separable layer normalization. Putting these all together, we propose EquiformerV2, which is developed on large and diverse OC20 dataset (Chanussot* et al. [7]). Experiments on OC20 show that EquiformerV2 outperforms previous state-of-the-art methods with improvements of up to 9% on forces and 4% on energies, and offers better speed-accuracy trade-offs compared to existing invariant and equivariant GNNs. Additionally, when used in the AdsorbML algorithm (Lan et al. [30]) for performing adsorption energy calculations, EquiformerV2 achieves the highest success rate and 2×2\times reduction in DFT calculations to achieve comparable adsorption energy accuracies as previous methods. Furthermore, EquiformerV2 trained on only OC22 (Tran* et al. [58]) dataset outperforms GemNet-OC (Gasteiger et al. [19]) trained on both OC20 and OC22 datasets, achieving much better data efficiency. Finally, we compare EquiformerV2 with Equiformer on QM9 dataset ([43, 45]) and OC20 S2EF-2M dataset to better understand the performance gain of higher degrees and the improved architecture.

Background

We present background relevant to this work here and discuss related works in Sec. B.

SE(3)/E(3)SE(3)/E(3)-Equivariant Neural Networks

We discuss the relevant background of SE(3)/E(3)SE(3)/E(3)-equivariant neural networks here. Please refer to Sec. A in appendix for more details of equivariance and group theory.

Including equivariance in neural networks can serve as a strong prior knowledge, which can therefore improve data efficiency and generalization. Equivariant neural networks use equivariant irreps features built from vector spaces of irreducible representations (irreps) to achieve equivariance to 3D rotation. Specifically, the vector spaces are (2L+1)(2L + 1)-dimensional, where degree LL is a non-negative integer. LL can be intuitively interpreted as the angular frequency of the vectors, i.e., how fast the vectors rotate with respect to a rotation of the coordinate system. Higher LL is critical to tasks sensitive to angular information like predicting forces ([3, 70, 39]). Vectors of degree LL are referred to as type-LL vectors, and they are rotated with Wigner-D matrices D(L)D^{(L)} when rotating coordinate systems. Euclidean vectors r⃗\vec{r} in R3\mathbb{R}^{3} can be projected into type-LL vectors by using spherical harmonics Y(L)(r⃗∥r⃗∥)Y^{(L)}\left(\frac{\vec{r}}{\lVert\vec{r}\rVert}\right). We use order mm to index the elements of type-LL vectors, where −L≤m≤L-L \le m \le L. We concatenate multiple type-LL vectors to form an equivariant irreps feature ff. Concretely, ff has CLC_L type-LL vectors, where 0≤L≤Lmax⁡0 \le L \le L_{\max} and CLC_L is the number of channels for type-LL vectors. In this work, we mainly consider CL=CC_L = C, and the size of ff is (Lmax⁡+1)2×C(L_{\max} + 1)^2 \times C. We index ff by channel ii, degree LL, and order mm and denote as fm,i(L)f^{(L)}_{m,i}.

Equivariant GNNs update irreps features by passing messages of transformed irreps features between nodes. To interact different type-LL vectors during message passing, we use tensor products, which generalize multiplication to equivariant irreps features. Denoted as ⊗L1,L2L3\otimes^{L_3}_{L_1,L_2}, the tensor product uses Clebsch-Gordan coefficients to combine type-L1L_1 vector f(L1)f^{(L_1)} and type-L2L_2 vector g(L2)g^{(L_2)} and produces type-L3L_3 vector h(L3)h^{(L_3)}:

hm3(L3)=(f(L1)⊗L1,L2L3g(L2))m3=∑m1=−L1L1∑m2=−L2L2C(L1,m1)(L2,m2)(L3,m3)fm1(L1)gm2(L2)(1)h^{(L_3)}_{m_3} = \left(f^{(L_1)} \otimes^{L_3}_{L_1,L_2} g^{(L_2)}\right)_{m_3} = \sum_{m_1=-L_1}^{L_1} \sum_{m_2=-L_2}^{L_2} C^{(L_3,m_3)}_{(L_1,m_1)(L_2,m_2)} f^{(L_1)}_{m_1} g^{(L_2)}_{m_2} \tag*{(1)}

where m1m_1 denotes order and refers to the m1m_1-th element of f(L1)f^{(L_1)}. Clebsch-Gordan coefficients C(L1,m1)(L2,m2)(L3,m3)C^{(L_3,m_3)}_{(L_1,m_1)(L_2,m_2)} are non-zero only when ∣L1−L2∣≤L3≤∣L1+L2∣\lvert L_1 - L_2\rvert\le L_3 \le\lvert L_1 + L_2\rvert and thus restrict output vectors to be of certain degrees. We typically discard vectors with L>Lmax⁡L > L_{\max}, where Lmax⁡L_{\max} is a hyper-parameter, to prevent vectors of increasingly higher dimensions. In many works, message passing is implemented as equivariant convolutions, which perform tensor products between input irreps features x(L1)x^{(L_1)} and spherical harmonics of relative position vectors Y(L2)(r⃗∥r⃗∥)Y^{(L_2)}\left(\frac{\vec{r}}{\lVert\vec{r}\rVert}\right).

Equiformer

Equiformer ([33]) is an SE(3)/E(3)SE(3)/E(3)-equivariant GNN that combines the inductive biases of equivariance with the strength of Transformers. First, Equiformer replaces scalar node features with equivariant irreps features to incorporate equivariance. Next, it performs equivariant operations on these irreps features and equivariant graph attention for message passing. These operations include tensor products and equivariant linear operations, equivariant layer normalization ([1]) and gate activation ([63]). For stronger expressivity in the attention compared to typical Transformers, Equiformer uses non-linear functions for both attention weights and message passing. Additionally, Equiformer incorporates regularization techniques commonly used by Transformers, e.g., dropout ([53]) to attention weights ([62]) and stochastic depth ([24]) to the outputs of equivariant graph attention and feed forward networks.

eSCN Convolution

eSCN convolutions ([39]) are proposed to use SO(2)SO(2) linear operations for efficient tensor products. We provide an outline and intuition for that method here, please refer to Sec. A.3 and their work ([39]) for mathematical details.

A traditional SO(3)SO(3) convolution interacts input irreps features xmi(Li)x^{(L_i)}_{m_i} and spherical harmonic projections of relative positions Ymf(Lf)(r⃗ij)Y^{(L_f)}_{m_f}(\vec{r}_{ij}) with an SO(3)SO(3) tensor product with Clebsch-Gordan coefficients C(Li,mi),(Lf,mf)(Lo,mo)C^{(L_o,m_o)}_{(L_i,m_i),(L_f,m_f)}. The projection Ymf(Lf)(r⃗ij)Y^{(L_f)}_{m_f}(\vec{r}_{ij}) becomes sparse if we rotate the relative position vector r⃗ij\vec{r}_{ij} with a rotation matrix DijD_{ij} to align with the direction of L=0L=0 and m=0m=0, which corresponds to the zz axis traditionally but the yy axis in the conventions of e3nn (Geiger et al., [20]). Concretely, given Dijr⃗ijD_{ij}\vec{r}_{ij} aligned with the yy axis, Ymf(Lf)(Dijr⃗ij)≠0Y^{(L_f)}_{m_f}(D_{ij}\vec{r}_{ij}) \ne0 only for mf=0m_f=0. If we consider only mf=0m_f=0, C(Li,mi),(Lf,mf)(Lo,mo)C^{(L_o,m_o)}_{(L_i,m_i),(L_f,m_f)} can be simplified, and C(Li,mi),(Lf,0)(Lo,mo)≠0C^{(L_o,m_o)}_{(L_i,m_i),(L_f,0)} \ne0 only when mi=±mom_i=\pm m_o. Therefore, the original expression depending on mim_i, mfm_f, and mom_o is now reduced to only depend on mom_o. This means we are no longer mixing all integer values of mim_i and mfm_f, and outputs of order mom_o are linear combinations of inputs of order ±mo\pm m_o. eSCN convolutions go one step further and replace the remaining non-trivial paths of the SO(3)SO(3) tensor product with an SO(2)SO(2) linear operation to allow for additional parameters of interaction between ±mo\pm m_o without breaking equivariance.

EquiformerV2

Starting from Equiformer (Liao & Smidt, [33]), we first use eSCN convolutions to scale to higher-degree representations (§ 3.1). Then, we propose three architectural improvements, which yield further performance gain when using higher degrees: attention re-normalization (§ 3.2), separable S2S^2 activation (§ 3.3) and separable layer normalization (§ 3.4). Figure 1 illustrates the overall architecture of EquiformerV2 and the differences from Equiformer.

Overview of EquiformerV2

Figure 1. Overview of EquiformerV2. We highlight the differences from Equiformer (Liao & Smidt, [33]) in red. For (b), (c), and (d), the left figure is the original module in Equiformer, and the right figure is the revised module in EquiformerV2. Input 3D graphs are embedded with atom and edge-degree embeddings and processed with Transformer blocks, which consist of equivariant graph attention and feed forward networks. “⊗\otimes” denotes multiplication, “⊕\oplus” denotes addition, and ∑\sum within a circle denotes summation over all neighbors. “DTP” denotes depth-wise tensor products used in Equiformer. Gray cells indicate intermediate irreps features.

Incorporating eSCN Convolutions for Higher Degrees

The computational complexity of SO(3)SO(3) tensor products used in traditional SO(3)SO(3) convolutions scale unfavorably with Lmax⁡L_{\max}. Because of this, it is impractical for Equiformer to use Lmax⁡>2L_{\max}>2 for large-scale datasets like OC20. Since higher Lmax⁡L_{\max} can better capture angular information and are correlated with model expressivity (Batzner et al., [3]), low values of Lmax⁡L_{\max} can lead to limited performance on certain tasks such as predicting forces. Therefore, we replace original tensor products with eSCN convolutions for efficient tensor products, enabling Equiformer to scale up Lmax⁡L_{\max} to 6 or 8 on OC20 dataset. Equiformer uses equivariant graph attention for message passing. The attention consists of depth-wise tensor products, which mix information across different degrees, and linear layers, which mix information between channels of the same degree. Since eSCN convolutions mix information across both degrees and channels, we replace the SO(3)SO(3) convolution, which involves one depth-wise tensor product layer and one linear layer, with a single eSCN convolutional layer, which consists of a rotation matrix DijD_{ij} and an SO(2)SO(2) linear layer as shown in Figure 1b.

Attention Re-normalization

Equivariant graph attention in Equiformer uses tensor products to project node embeddings xix_i and xjx_j, which contain vectors of degrees from 0 to Lmax⁡L_{\max}, to scalar features fij(0)f_{ij}^{(0)} and applies non-linear functions to fij(0)f_{ij}^{(0)} for attention weights aija_{ij}. We propose attention re-normalization and introduce one additional layer normalization (LN) (Ba et al., [1]) before non-linear functions. Specifically, given fij(0)f_{ij}^{(0)}, we first apply LN and then use one leaky ReLU layer and one linear layer to calculate zij=waLeakyReLU⁡(LN⁡(fij(0)))z_{ij}=w_a\operatorname{LeakyReLU}(\operatorname{LN}(f_{ij}^{(0)})) and aij=softmax⁡j(zij)=exp⁡(zij)∑k∈N(i)exp⁡(zik)a_{ij}=\operatorname{softmax}_j(z_{ij})=\frac{\exp(z_{ij})}{\sum_{k\in\mathcal{N}(i)}\exp(z_{ik})}, where waw_a is a learnable vector of the same dimension as fij(0)f_{ij}^{(0)}. The motivation is similar to ViT-22B (Dehghani et al., [9]), where they find that they need an additional layer normalization to stabilize training when increasing widths. When we scale up Lmax⁡L_{\max}, we effectively increase the number of input channels for calculating fij(0)f_{ij}^{(0)}. By normalizing fij(0)f_{ij}^{(0)}, the additional LN can make sure the inputs to subsequent non-linear functions and softmax operations still lie within the same range as lower Lmax⁡L_{\max} is used. This empirically improves the performance as shown in Table 1a.

Table 1. Ablation studies of EquiformerV2. We report mean absolute errors (MAE) for forces in meV/Å and energy in meV, and lower is better. All models are trained on the 2M split of the OC20 S2EF dataset, and errors are averaged over the four validation sub-splits. The base model setting is marked in gray.

Separable S2S^2 Activation

The gate activation (Weiler et al., [63]) used by Equiformer applies sigmoid activation to scalar features to obtain non-linear weights and then multiply irreps features of degree >0>0 with non-linear weights to add non-linearity to equivariant features. The activation only accounts for the interaction from vectors of degree 0 to those of degree >0>0 and can be sub-optimal when we scale up Lmax⁡L_{\max}. To better mix the information across degrees, SCN (Zitnick et al., [70]) and eSCN adopt S2S^2 activation (Cohen et al., [8]). The activation first converts vectors of all degrees to point samples on a sphere for each channel, applies unconstrained functions FF to those samples, and finally convert them back to vectors. Specifically, given an input irreps feature x∈R(Lmax⁡+1)2×Cx \in\mathbb{R}^{(L_{\max}+1)^2 \times C}, the output is y=G−1(F(G(x)))y = G^{-1}(F(G(x))), where GG denotes the conversion from vectors to point samples on a sphere, FF can be typical SiLU activation (Elfwing et al., [14]; Ramachandran et al., [42]) or typical MLPs, and G−1G^{-1} is the inverse of GG. We provide more details of S2S^2 activation in Sec. A.4.

However, we find that directly replacing the gate activation with S2S^2 activation in Equiformer results in large gradients and training instability (Index 3 in Table 1a). To address the issue, we propose separable S2S^2 activation, which separates activation for vectors of degree 0 and those of degree >0> 0. Similar to gate activation, we have more channels for vectors of degree 0. As shown in Figure 2c, we apply a SiLU activation to the first part of vectors of degree 0, and the second part of vectors of degree 0 are used for S2S^2 activation along with vectors of higher degrees. After S2S^2 activation, we concatenate the first part of vectors of degree 0 with vectors of degrees >0> 0 as the final output and ignore the second part of vectors of degree 0. Separating the activation for vectors of degree 0 and those of degree >0> 0 prevents large gradients, enabling using more expressive S2S^2 activation for better performance. Additionally, we use separable S2S^2 activation in feed forward networks (FFNs). Figure 2 illustrates the differences between gate activation, S2S^2 activation and separable S2S^2 activation.

Illustration of different activation functions

Figure 2. Illustration of different activation functions. GG denotes conversion from vectors to point samples on a sphere, FF can typically be a SiLU activation or MLPs, and G−1G^{-1} is the inverse of GG.

Separable Layer Normalization

Equivariant layer normalization used by Equiformer normalizes vectors of different degrees independently. However, it potentially ignores the relative importance of different degrees since the relative magnitudes between different degrees become the same after the normalization. Therefore, instead of performing normalization to each degree independently, motivated by the separable S2S^2 activation mentioned above, we propose separable layer normalization (SLN), which separates normalization for vectors of degree 0 and those of degrees >0> 0. Mathematically, let x∈R(Lmax⁡+1)2×Cx \in\mathbb{R}^{(L_{\max}+1)^2 \times C} denote an input irreps feature of maximum degree Lmax⁡L_{\max} and CC channels, and xm,i(L)x^{(L)}_{m,i} denote the LL-th degree, mm-th order and ii-th channel of xx. SLN calculates the output yy as follows. For L=0L=0, y(0)=γ(0)∘(x(0)−μ(0)σ(0))+β(0)y^{(0)} = \gamma^{(0)} \circ\left(\frac{x^{(0)}-\mu^{(0)}}{\sigma^{(0)}}\right) + \beta^{(0)}, where μ(0)=1C∑i=1Cx0,i(0)\mu^{(0)} = \frac{1}{C}\sum_{i=1}^{C}x^{(0)}_{0,i} and σ(0)=1C∑i=1C(x0,i(0)−μ(0))2\sigma^{(0)} = \sqrt{\frac{1}{C}\sum_{i=1}^{C}(x^{(0)}_{0,i}-\mu^{(0)})^2}.

For L>0L > 0, y(L)=γ(L)∘(x(L)−μ(L>0)σ(L>0))y^{(L)} = \gamma^{(L)} \circ\left(\frac{x^{(L)}-\mu^{(L>0)}}{\sigma^{(L>0)}}\right), where σ(L>0)=1Lmax⁡∑L=1Lmax⁡(σ(L))2\sigma^{(L>0)} = \sqrt{\frac{1}{L_{\max}}\sum_{L=1}^{L_{\max}}(\sigma^{(L)})^2} and σ(L)=1C∑i=1C12L+1∑m=−LL(xm,i(L))2\sigma^{(L)} = \sqrt{\frac{1}{C}\sum_{i=1}^{C}\frac{1}{2L+1}\sum_{m=-L}^{L}(x^{(L)}_{m,i})^2}. γ(0),γ(L),β(0)∈RC\gamma^{(0)},\gamma^{(L)},\beta^{(0)} \in\mathbb{R}^{C} are learnable parameters, μ(0)\mu^{(0)} and σ(0)\sigma^{(0)} are mean and standard deviation of vectors of degree 0, σ(L)\sigma^{(L)} and σ(L>0)\sigma^{(L>0)} are root mean square values (RMS), and ∘\circ denotes element-wise product. Figure 3 compares how μ(0)\mu^{(0)}, σ(0)\sigma^{(0)}, σ(L)\sigma^{(L)} and σ(L>0)\sigma^{(L>0)} are calculated in equivariant layer normalization and SLN. Preserving the relative magnitudes between degrees >0> 0 improves performance as shown in Table 1a.

Illustration of how statistics are calculated in different normalizations

Figure 3. Illustration of how statistics are calculated in different normalizations. “std” denotes standard deviation, and “RMS” denotes root mean square.

Overall Architecture

Equivariant Graph Attention. Figure 1b illustrates equivariant graph attention after the above modifications. Given node embeddings xix_i and xjx_j, we first concatenate them along the channel dimension and then rotate them with rotation matrices DijD_{ij} based on their relative positions or edge directions r⃗ij\vec{r}_{ij}. We replace depth-wise tensor products and linear layers between xix_i, xjx_j and fijf_{ij} with a single SO(2)SO(2) linear layer. To consider the information of relative distances ∥r⃗ij∥\lVert\vec{r}_{ij}\rVert, we transform ∥r⃗ij∥\lVert\vec{r}_{ij}\rVert with a radial function to obtain edge distance embeddings and then multiply the edge distance embeddings with concatenated node embeddings before the first SO(2)SO(2) linear layer. We split the outputs fijf_{ij} of the first SO(2)SO(2) linear layer into two parts. The first part is scalar features fij(0)f_{ij}^{(0)}, which only contains vectors of degree 0, and the second part is irreps features fij(L)f_{ij}^{(L)} and includes vectors of all degrees up to Lmax⁡L_{\max}. As mentioned in § 3.2, we first apply an additional LN to fij(0)f_{ij}^{(0)} and then follow the design of Equiformer by applying one leaky ReLU layer, one linear layer and a final softmax layer to obtain attention weights aija_{ij}. As for value vijv_{ij}, we replace the gate activation with separable S2S^2 activation with F′F' being a single SiLU activation and then apply the second SO(2)SO(2) linear layer. While in Equiformer, the message mijm_{ij} sent from node jj to node ii is mij=aij×vijm_{ij}=a_{ij}\times v_{ij}, here we need to rotate aij×vija_{ij}\times v_{ij} back to original coordinate frames and the message mijm_{ij} becomes Dij−1(aij×vij)D_{ij}^{-1}(a_{ij}\times v_{ij}). Finally, we can perform hh parallel equivariant graph attention functions given fijf_{ij}. The hh different outputs are concatenated and projected with a linear layer to become the final output yiy_i. Parallelizing attention functions and concatenating can be implemented with “Reshape”.

Feed Forward Network. As shown in Figure 1d, we replace the gate activation with separable S2S^2 activation. The function FF consists of a two-layer MLP, with each linear layer followed by SiLU, and a final linear layer.

Embedding. This module consists of atom embedding and edge-degree embedding. The former is the same as that in Equiformer. For the latter, as depicted in the right branch in Figure 1c, we replace original linear layers and depth-wise tensor products with a single SO(2)SO(2) linear layer followed by a rotation matrix Dij−1D_{ij}^{-1}. Similar to equivariant graph attention, we consider the information of relative distances by multiplying the outputs of the SO(2)SO(2) linear layer with edge distance embeddings.

Radial Basis and Radial Function. We represent relative distances ∥r⃗ij∥\lVert\vec{r}_{ij}\rVert with a finite radial basis like Gaussian radial basis functions [48] to capture their subtle changes. We transform radial basis with a learnable radial function to generate edge distance embeddings. The function consists of a two-layer MLP, with each linear layer followed by LN and SiLU, and a final linear layer.

Output Head. To predict scalar quantities like energy, we use a feed forward network to transform irreps features on each node into a scalar and then sum over all nodes. For predicting atom-wise forces, we use a block of equivariant graph attention and treat the output of degree 1 as our predictions.

Experiments

OC20 Dataset

Our main experiments focus on the large and diverse OC20 dataset (Chanussot* et al., [7]). Please refer to Sec. D.1 for details on the dataset. We first conduct ablation studies on EquiformerV2 trained on the 2M split of OC20 S2EF dataset (§ 4.1.1). Then, we report the results of training All and All+MD splits (§ 6.0.1). Additionally, we investigate the performance of EquiformerV2 when used in the AdsorbML algorithm (Lan et al., [30]) (§ 6.0.2). Please refer to Sec. C and D for details of models and training.

Ablation Studies

Architectural Improvements. In Table 1a, we start with incorporating eSCN convolutions into Equiformer for higher-degree representations (Index 1) and then ablate the three proposed architectural improvements. First, with attention re-normalization, energy mean absolute errors (MAE) improve by 2.4%, while force MAE are about the same (Index 1 and 2). Second, we replace the gate activation with S2S^2 activation, but that does not converge (Index 3). With the proposed S2S^2 activation, we stabilize training and successfully leverage more expressive activation to improve force MAE (Index 4). Third, replacing equivariant layer normalization with separable layer normalization further improves force MAE (Index 5). We note that simply incorporating eSCN convolutions into Equiformer and using higher degrees (Index 1) do not result in better performance than the original eSCN baseline (Index 8), and that the proposed architectural changes are necessary. We additionally compare the performance gain of architectural improvements with that of training for longer. Attention re-normalization (Index 1 and 2) improves energy MAE by the same amount as increasing training epochs from 12 to 20 (Index 5 and 7). The improvement of SLN (Index 4 and 5) in force MAE is about 40% of that of increasing training epochs from 12 to 20 (Index 5 and 7), and SLN is about 6% faster in training. We also conduct the ablation studies on the QM9 dataset and summarize the results in Sec. Sec. F.3.

IndexAttention re-normalizationActivationNormalizationEpochsforcesenergy
1×\timesGateLN1221.85286
2✓\checkmarkGateLN1221.86279
3✓\checkmarkS2S^2LN12didn’t converge
4✓\checkmarkSep. S2S^2LN1220.77285
5✓\checkmarkSep. S2S^2SLN1220.46285
6✓\checkmarkSep. S2S^2LN2020.02276
7✓\checkmarkSep. S2S^2SLN2019.72278
8eSCN baseline1221.3294

Table 1a. Architectural improvements. Attention re-normalization improves energies, and separable S2S^2 activation (“Sep. S2S^2”) and separable layer normalization (“SLN”) improve forces.

LmaxL_{max}EpochseSCN forceseSCN energyEquiformerV2 forcesEquiformerV2 energy
61221.329420.46285
62020.629019.78280
63020.128519.42278
81221.329620.46279
820--19.95273

Table 1b. Training epochs. Training for more epochs consistently leads to better results.

LmaxL_{max}eSCN forceseSCN energyEquiformerV2 forcesEquiformerV2 energy
422.229121.37284
621.329420.46285
821.329620.46279

Table 1c. Degrees LmaxL_{max}. Higher degrees are consistently helpful.

MmaxM_{max}eSCN forceseSCN energyEquiformerV2 forcesEquiformerV2 energy
221.329420.46285
321.229520.24284
421.229820.24282
6--20.26278

Table 1d. Orders MmaxM_{max}. Higher orders mainly improve energy predictions.

LayerseSCN forceseSCN energyEquiformerV2 forcesEquiformerV2 energy
822.430621.18293
1221.329420.46285
1620.528320.11282

Table 1e. Number of Transformer blocks. Adding more blocks can help both force and energy predictions.

Scaling of Parameters

In Tables 1c, 1d, 1e, we vary the maximum degree LmaxL_{max}, the maximum order MmaxM_{max}, and the number of Transformer blocks and compare with equivalent eSCN variants. Across all experiments, EquiformerV2 performs better than its eSCN counterparts. Besides, while one might intuitively expect higher resolutions and larger models to perform better, this is only true for EquiformerV2, not eSCN. For example, increasing LmaxL_{max} from 6 to 8 or MmaxM_{max} from 3 to 4 degrades the performance of eSCN on energy predictions but helps that of EquiformerV2. In Table 1b, we show that increasing the training epochs from 12 to 30 epochs can consistently improve performance.

Speed-Accuracy Trade-offs

We compare trade-offs between inference speed or training time and force MAE among prior works and EquiformerV2 and summarize the results in Figure 4a and Figure 4b. EquiformerV2 achieves the lowest force MAE across a wide range of inference speed and training cost.

Trade-offs between inference speed and validation force MAE and between training cost and validation force MAE

Figure 4. EquiformerV2 achieves better accuracy trade-offs both in terms of inference speed as well as training cost. All models are trained on the S2EF-2M split and measured on V100 GPUs.

Main Results

Table 2 reports results on the test splits for all the three tasks of OC20 averaged over all four sub-splits. Models are trained on either OC20 S2EF-All or S2EF-All+MD splits. All test results are computed via the EvalAI evaluation server1. We train EquiformerV2 of two sizes, one with 153M parameters and the other with 31M parameters. When trained on the S2EF-All+MD split, EquiformerV2 (λE=4\lambda_E=4, 153M) improves previous state-of-the-art S2EF energy MAE by 4%, S2EF force MAE by 9%, IS2RS Average Forces below Threshold (AFbT) by absolute 4% and IS2RE energy MAE by 4%. In particular, the improvement in force predictions is significant. Going from SCN to eSCN, S2EF test force MAE improves from 17.2 meV/Å to 15.6 meV/Å, largely due to replacing approximate equivariance in SCN with strict equivariance in eSCN during message passing. Similarly, by scaling up the degrees of representations in Equiformer, EquiformerV2 (λE=4\lambda_E=4, 153M) further improves force MAE to 14.2 meV/Å, which is similar to the gain of going from SCN to eSCN. Additionally, the smaller EquiformerV2 (λE=4\lambda_E=4, 31M) improves upon previously best results for all metrics except IS2RS AFbT and achieves comparable training throughput to the fastest GemNet-OC-L-E. Although the training time of EquiformerV2 is higher here, we note that this is because training EquiformerV2 for longer keeps improving performance and that we already demonstrate EquiformerV2 achieves better trade-offs between force MAE and speed.

Training setModelNumber of parametersTraining time (GPU-days)Throughput Samples / GPU sec.S2EF validation Energy MAE (meV)S2EF validation Force MAE (meV/Å)S2EF test Energy MAE (meV)S2EF test Force MAE (meV/Å)IS2RS test AFbT (%)IS2RS test ADwT (%)IS2RE test Energy MAE (meV)
OC20 S2EF-AllSchNet (Schütt et al., [48])9.1M194—54956.854054.7—14.4749
OC20 S2EF-AllDimeNet++-L-F+E (Gasteiger et al., [17])10.7M16004.651532.848031.321.751.7559
OC20 S2EF-AllSpinConv (Shuaibi et al., [51])8.5M2756.037141.233629.716.753.6437
OC20 S2EF-AllGemNet-dT (Klicpera et al., [27])32M49225.831527.229224.227.658.7400
OC20 S2EF-AllGemNet-OC (Gasteiger et al., [19])39M33618.324421.723320.735.360.3355
OC20 S2EF-AllSCN L=8L=8 K=20K=20 (Zitnick et al., [70])271M645———24417.740.367.1330
OC20 S2EF-AlleSCN L=6L=6 K=20K=20 (Passaro & Zitnick, [39])200M6002.9——24217.148.565.7341
OC20 S2EF-AllEquiformerV2 (λE=2\lambda_E=2, 153M)153M13681.823615.722914.853.069.0316
OC20 S2EF-All+MDGemNet-OC-L-E (Gasteiger et al., [19])56M6407.523922.123021.0———
OC20 S2EF-All+MDGemNet-OC-L-F (Gasteiger et al., [19])216M7653.225220.024119.040.660.4—
OC20 S2EF-All+MDGemNet-OC-L-F+E (Gasteiger et al., [19])—————————348
OC20 S2EF-All+MDSCN L=6L=6 K=16K=16 (4-tap 2-band) (Zitnick et al., [70])168M414———22817.843.364.9328
OC20 S2EF-All+MDSCN L=8L=8 K=20K=20 (Zitnick et al., [70])271M1280———23717.243.667.5321
OC20 S2EF-All+MDeSCN L=6L=6 K=20K=20 (Passaro & Zitnick, [39])200M5682.924317.122815.650.366.7323
OC20 S2EF-All+MDEquiformerV2 (λE=4\lambda_E=4, 31M)31M7057.123216.322815.547.668.3315
OC20 S2EF-All+MDEquiformerV2 (λE=2\lambda_E=2, 153M)153M13681.823014.622713.855.469.8311
OC20 S2EF-All+MDEquiformerV2 (λE=4\lambda_E=4, 153M)153M15711.822715.021914.254.469.4309

Table 2. OC20 results on S2EF validation and test splits, and IS2RS and IS2RE test splits when trained on OC20 S2EF-All or S2EF-All+MD splits. Throughput is reported as the number of structures processed per GPU-second during training and measured on V100 GPUs. λE\lambda_E is the coefficient of the energy loss.

AdsorbML Results

Lan et al. [30] recently proposes the AdsorbML algorithm, wherein they show that recent state-of-the-art GNNs can achieve more than 1000× speedup over DFT relaxations at computing adsorption energies within a 0.1eV margin of DFT results with an 87% success rate. This is done by using OC20-trained models to perform structure relaxations for an average 90 configurations of an adsorbate placed on a catalyst surface, followed by DFT single-point calculations for the top-kk structures with lowest predicted relaxed energies, as a proxy for calculating the global energy minimum or adsorption energy. We refer readers to Sec. D.4 and the work (Lan et al., [30]) for more details. We benchmark AdsorbML with EquiformerV2, and Table 3 shows that EquiformerV2 (λE=4\lambda_E=4, 153M) improves over SCN by a significant margin, with 8% and 5% absolute improvements at k=1k=1 and k=2k=2, respectively. Moreover, EquiformerV2 (λE=4\lambda_E=4, 153M) at k=2k=2 is more accurate at adsorption energy calculations than all the other models even at k=5k=5, thus requiring at least 2×2\times fewer DFT calculations. Since the speedup is with respect to using DFT for structure relaxations and that ML models are much faster than DFT, the speedup is dominated by the final DFT single-point calculations and ML models with the same value of kk have roughly the same speedup. To better understand the speed-accuracy trade-offs of different models, we also report average GPU-seconds of running one structure relaxation in Table 3. Particularly, EquiformerV2 (λE=4\lambda_E=4, 31M) improves upon previous methods while being 3.7× to 9.8× faster than GemNet-OC-MD-Large and SCN, respectively.

ModelGPU-seconds per relaxationk=1k = 1 Successk=1k = 1 Speedupk=2k = 2 Successk=2k = 2 Speedupk=3k = 3 Successk=3k = 3 Speedupk=4k = 4 Successk=4k = 4 Speedupk=5k = 5 Successk=5k = 5 Speedup
SchNet (Schütt et al., [48])-2.77%4266.133.91%2155.364.32%1458.774.73%1104.885.04%892.79
DimeNet++ (Gasteiger et al., [17])-5.34%4271.237.61%2149.788.84%1435.2110.07%1081.9610.79%865.20
PaiNN (Schütt et al., [49])-27.44%4089.7733.61%2077.6536.69%1395.5538.64%1048.6339.57%840.44
GemNet-OC (Gasteiger et al., [19])9.468.76%4185.1877.29%2087.1180.78%1392.5181.50%1046.8582.94%840.25
GemNet-OC-MD (Gasteiger et al., [19])9.468.76%4182.0478.21%2092.2781.81%1404.1183.25%1053.3684.38%841.64
GemNet-OC-MD-Large (Gasteiger et al., [19])45.073.18%4078.7679.65%2065.1583.25%1381.3985.41%1041.5086.02%834.46
SCN-MD-Large (Zitnick et al., [70])120.077.80%3974.2184.28%1989.3286.33%1331.4387.36%1004.4087.77%807.00
EquiformerV2 (λE=4\lambda_E = 4, 31M)12.284.17%3983.4187.15%1992.6487.87%1331.3588.69%1000.6289.31%802.95
EquiformerV2 (λE=4\lambda_E = 4, 153M)45.085.41%4001.7188.90%2012.4790.54%1352.0891.06%1016.3191.57%815.87

Table 3. AdsorbML results with EquiformerV2 (λE=4\lambda_E = 4, 31M) and (λE=4\lambda_E = 4, 153M) trained on S2EF-All+MD from Table 2. We visualize the speed-accuracy trade-offs of different models in Figure 5.

OC22 Dataset

Dataset. The OC22 dataset (Tran* et al., [58]) focuses on oxide electrocatalysis. One crucial difference between OC22 and OC20 is that the energies in OC22 are DFT total energies. DFT total energies are harder to predict but are the most general and closest to a DFT surrogate, offering the flexibility to study property prediction beyond adsorption energies. Similar to OC20, the tasks in OC22 are S2EF-Total and IS2RE-Total. We train models on the OC22 S2EF-Total dataset and evaluate them on energy and force MAE on the S2EF-Total validation and test splits. We use the trained models to perform structural relaxations and predict relaxed energy. Relaxed energy predictions are evaluated on the IS2RE-Total test split.

Training Details. Please refer to Section E.1 for details on architectures, hyper-parameters and training time.

Results. We train two EquiformerV2 models with different energy coefficients λE\lambda_E and force coefficients λF\lambda_F. We follow the practice of OC22 models and use linear reference (Tran* et al., [58]). The results are summarized in Table 4. EquiformerV2 improves upon previous models on all the tasks. EquiformerV2 (λE=4\lambda_E = 4, λF=100\lambda_F = 100) trained on only OC22 achieves better results on all the tasks than GemNet-OC trained on both OC20 and OC22. We note that OC22 contains about 8.4M structures and OC20 contains about 130M structures, and therefore EquiformerV2 demonstrates significantly better data efficiency. Additionally, the performance gap between eSCN and EquiformerV2 is larger than that on OC20, suggesting that more complicated structures can benefit more from the proposed architecture. When trained on OC20 S2EF-All+MD, EquiformerV2 (λE=4\lambda_E = 4, 153M) improves upon eSCN by 4% on energy MAE and 9% on force MAE. For OC22, EquiformerV2 (λE=4\lambda_E = 4, λF=100\lambda_F = 100) improves upon eSCN by 18.9% on average energy MAE and 8.9% on average force MAE.

ModelTraining SetLinear referenceNumber of parametersS2EF-Total validation Energy MAE (meV) IDS2EF-Total validation Energy MAE (meV) OODS2EF-Total validation Force MAE (meV/Å) IDS2EF-Total validation Force MAE (meV/Å) OODS2EF-Total test Energy MAE (meV) IDS2EF-Total test Energy MAE (meV) OODS2EF-Total test Force MAE (meV/Å) IDS2EF-Total test Force MAE (meV/Å) OODIS2RE-Total test Energy MAE (meV) IDIS2RE-Total test Energy MAE (meV) OOD
GemNet-OC (Gasteiger et al., [19])OC2239M5451011304037482929.439.613291584
GemNet-OC (Gasteiger et al., [19])OC22✓39M----357105730.040.0--
GemNet-OC (Gasteiger et al., [19])OC22 + OC2039M464859273431168926.934.212001534
eSCN (Passaro & Zitnick, [39])OC22✓200M----35078923.835.7--
EquiformerV2 (λE=1\lambda_E = 1, λF=1\lambda_F = 1)OC22✓122M34358024.4234.31182.8677.422.9835.5710841444
EquiformerV2 (λE=4\lambda_E = 4, λF=100\lambda_F = 100)OC22✓122M43362922.8830.70263.7659.821.5832.6511191440

Table 4. OC22 results on S2EF-Total validation and test splits and IS2RE-Total test split.

Comparison with Equiformer

QM9 Dataset. We follow the setting of Equiformer and train EquiformerV2 on the QM9 dataset (Ramakrishnan et al., [43]; Ruddigkeit et al., [45]) and summarize the results in Table 5. The performance gain of using higher degrees and the improved architecture is not as significant as that on OC20 and OC22 datasets. This, however, is not surprising. The training set of QM9 contains only 110k examples, which is much less than OC20 S2EF-2M with 2M examples and OC22 with 8.4M examples. Moreover, QM9 has much less numbers of atoms in each example and much less diverse atom types, and each example has less angular variations. Nevertheless, EquiformerV2 achieves better results than Equiformer on 9 out of the 12 tasks and is therefore the overall best performing model. We additionally train EquiformerV2 with Noisy Nodes (Godwin et al., [22]) to better understand the gain of higher degrees in Sec. F.1.

ModelTask Unitsαa03\alpha_{a_0^3}Δε\Delta\varepsilon meVεHOMO\varepsilon_{\mathrm{HOMO}} meVεLUMO\varepsilon_{\mathrm{LUMO}} meVμ\mu DCνC_\nu cal/mol KGG meVHH meVRa022R^2_{a_0^2}UU meVU0U_0 meVZPVE meV
DimeNet++ [17]0.044332520.030.02387.331661.21
EGNN [47]†0.071482925.029.0311212.10612111.55
PaiNN [49]0.045462820.012.0247.355.98.0665.835.851.28
TorchMD-NET [54]0.059362018.011.0267.626.16.0336.386.151.84
SphereNet [34]0.046322318.026.02186.292761.12
SEGNN [4]†0.060422421.023.0311516.66013151.62
EQGAT [31]0.053322016.011.0242324.38225252.00
Equiformer [33]0.046301514.011.0237.636.63.2516.746.591.26
EquiformerV20.050291413.010.0237.576.22.1866.496.171.47

Table 5. Mean absolute error results on QM9 test set. † denotes using different data partitions.

OC20 S2EF-2M Dataset. We use similar configurations (e.g., numbers of blocks and numbers of channels) and train Equiformer on OC20 S2EF-2M dataset for the same number of epochs as training EquiformerV2. We vary Lmax⁡L_{\max} for both Equiformer and EquiformerV2 and compare the results in Table 6. For Lmax⁡=2L_{\max} = 2, EquiformerV2 is 2.3× faster than Equiformer since EquiformerV2 uses eSCN convolutions for efficient SO(3)SO(3) convolutions. Additionally, EquiformerV2 achieves better force MAE and similar energy MAE, demonstrating the effectiveness of the proposed improved architecture. For Lmax⁡>2L_{\max} > 2, we encounter out-of-memory errors when training Equiformer even after we reduce the number of blocks and use the batch size = 1. In contrast, We can easily train EquiformerV2 with Lmax⁡L_{\max} up to 6. When increasing Lmax⁡L_{\max} from 2 to 4, EquiformerV2 achieves lower energy MAE and significantly lower force MAE than Equiformer and requires 1.4× less training time. The comparison suggests that complicated datasets have more performance to gain from using more expressive models, enabling better performance and lower computational cost.

ModelLmax⁡L_{\max}Energy MAE (meV)↓Force MAE (meV/Å)↓Training time (GPU-hours)↓
Equiformer [33]229727.571365
Equiformer [33]3OOMOOMOOM
EquiformerV2229826.24600
EquiformerV2428421.37966
EquiformerV2628520.461412

Table 6. Comparison on OC20 S2EF-2M dataset. Errors are averaged over the four validation sub-splits. “OOM” denotes out-of-memory error, and we cannot use Lmax⁡=3L_{\max} = 3 for Equiformer.

Discussion. One limitation of EquiformerV2 is that the performance gains brought by scaling to higher degrees and the proposed architectural improvements can depend on datasets and tasks. For small datasets like QM9, the performance gain is not significant. We additionally compare Equiformer and EquiformerV2 on OC20 IS2RE dataset in Sec. D.5. For different tasks, the improvements are also different, and force predictions benefit more from better expressivity than energy predictions. We note that the first issue can be mitigated by first pre-training on large datasets like OC20 and PCQM4Mv2 [38] optionally via denoising [22, 67] and then fine-tuning on smaller datasets. The second issue might be mitigated by combining DFT with ML models. For example, AdsorbML uses ML forces for structural relaxations and a single-point DFT for calculating the final relaxed energies.

Conclusion

In this work, we investigate how equivariant Transformers can be scaled up to higher degrees of equivariant representations. We start by replacing SO(3)SO(3) convolutions in Equiformer with eSCN convolutions, and then we propose three architectural improvements to better leverage the power of higher degrees – attention re-normalization, separable S2S^2 activation and separable layer normalization. With these modifications, we propose EquiformerV2, which outperforms state-of-the-art methods on all the tasks on the OC20 and OC22 datasets, improves speed-accuracy trade-offs, and achieves the best success rate when used in AdsorbML. We also compare EquiformerV2 with Equiformer to better understand the performance gain brought by higher degrees and the improved architecture.

Ethics Statement

EquiformerV2 achieves more accurate approximation of quantum mechanical calculations and demonstrates one further step toward being able to replace DFT compute force fields with machine learned ones for practical applications in chemistry and material science. We hope these promising results will encourage the community to make further progress in applications like material design and drug discovery, rather than use these methods for adversarial purposes. We note that these methods only facilitate the identification of molecules or materials with specific properties; there remain substantial hurdles to synthesize and deploy such molecules or materials at scale. Finally, we note that the proposed method is general and can be applied to different problems like protein structure prediction (Lee et al., [32]) as long as inputs can be modeled as 3D graphs.

Reproducibility Statement

We include details on architectures, hyper-parameters and training time in Sec. D.2 (OC20), Sec. E.1 (OC22) and Sec. F.2 (QM9).

The code for reproducing the results of EquiformerV2 trained on OC20 S2EF-2M and QM9 datasets is available at https://github.com/atomicarchitects/equiformer_v2.

Acknowledgement

We thank Larry Zitnick and Saro Passaro for helpful discussions. We also thank Muhammed Shuaibi for helping with the DFT evaluations for AdsorbML (Lan et al., [30]). We acknowledge the MIT SuperCloud and Lincoln Laboratory Supercomputing Center (Reuther et al., [44]) for providing high performance computing and consultation resources that have contributed to the research results reported within this paper.

Yi-Lun Liao and Tess Smidt were supported by DOE ICDI grant DE-SC0022215.

Additional background

We first provide relevant mathematical background on group theory and equivariance. We note that most of the content is adapted from Equiformer (Liao & Smidt, [33]) and that these works ([68, 12]) have more in-depth and pedagogical discussions. Then, we provide mathematical details of eSCN convolutions.

Group theory

Definition of Groups. A group is an algebraic structure that consists of a set GG and a binary operator ∘:G×G→G\circ: G \times G \to G. Typically denoted as GG, groups satisfy the following four axioms:

  1. Closure: g∘h∈Gg \circ h \in G for all g,h∈Gg, h \in G.

  2. Identity: There exists an identity element e∈Ge \in G such that g∘e=e∘g=gg \circ e = e \circ g = g for all g∈Gg \in G.

  3. Inverse: For each g∈Gg \in G, there exists an inverse element g−1∈Gg^{-1} \in G such that g∘g−1=g−1∘g=eg \circ g^{-1} = g^{-1} \circ g = e.

  4. Associativity: g∘h∘i=(g∘h)∘i=g∘(h∘i)g \circ h \circ i = (g \circ h) \circ i = g \circ(h \circ i) for all g,h,i∈Gg, h, i \in G.

In this work, we consider 3D Euclidean symmetry, and relevant groups are:

  1. The Euclidean group in three dimensions E(3)E(3): 3D rotation, translation and inversion.

  2. The special Euclidean group in three dimensions SE(3)SE(3): 3D rotation and translation.

  3. The orthogonal group in three dimensions O(3)O(3): 3D rotation and inversion.

  4. The special orthogonal group in three dimensions SO(3)SO(3): 3D rotation.

Since eSCN (Passaro & Zitnick, [39]) and this work only consider equivariance to 3D rotation and invariance to 3D translation but not inversion, we mainly discuss SE(3)SE(3)-equivariance in the main text and in appendix and note that more details of E(3)E(3)-equivariance can be found in the work of Equiformer (Liao & Smidt, [33]).

Group Representations. Given a vector space XX, the way a group GG acts on XX is given by the group representation DXD_X. DXD_X is parameterized by g∈Gg \in G, with DX(g):X→XD_X(g): X \to X. Group representations DXD_X are invertible matrices, and group transformations, or group actions, take the form of matrix multiplications. This definition of group representations satisfies the requirements of groups, including associativity, D(g)D(h)=D(g∘h)D(g)D(h)=D(g\circ h) for all g,h∈Gg,h \in G. We say that the two group representations D(g)D(g) and D′(g)D'(g) are equivalent if there exists a change-of-basis N×NN \times N matrix PP such that P−1D(g)P=D′(g)P^{-1}D(g)P=D'(g) for all g∈Gg \in G. D(g)D(g) is reducible if D′(g)D'(g) is block diagonal for all g∈Gg \in G, meaning that D′(g)D'(g) acts on multiple independent subspaces of the vector space. Otherwise, the representation D(g)D(g) is said to be irreducible. Irreducible representations, or irreps, are a class of representations that are convenient for composing different group representations. Specifically, for the case of SO(3)SO(3), Wigner-D matrices are irreducible representations, and we can express any group representation of SO(3)SO(3) as a direct sum (concatenation) of Wigner-D matrices [68, 12, 20]:

D(g)=P−1(⨁iD(Li)(g))P=P−1(D(L0)(g)D(L1)(g)⋯)P(2)D(g)=P^{-1}\left(\bigoplus_i D^{(L_i)}(g)\right)P=P^{-1}\begin{pmatrix}D^{(L_0)}(g)&D^{(L_1)}(g)&\cdots\end{pmatrix}P \tag*{(2)}

where D(Li)(g)D^{(L_i)}(g) are Wigner-D matrices of degree LiL_i.

Equivariance

A function ff mapping between vector spaces XX and YY is equivariant to a group of transformations GG if for any input x∈Xx \in X, output y∈Yy \in Y and group element g∈Gg \in G, we have f(DX(g)x)=DY(g)f(x)=DY(g)yf(D_X(g)x)=D_Y(g)f(x)=D_Y(g)y, where DX(g)D_X(g) and DY(g)D_Y(g) are transformation matrices or group representations parametrized by gg in XX and YY. Additionally, ff is invariant when DY(g)D_Y(g) is an identity matrix for any g∈Gg \in G.

As neural networks comprise many composable operations, equivariant neural networks comprise many equivariant operations to maintain the equivariance of input, intermediate, and output features. Incorporating equivariance as a strong prior knowledge can help improve data efficiency and generalization of neural networks [3, 41, 15]. In this work, we achieve equivariance to 3D rotation by operating on vector spaces of SO(3)SO(3) irreps, incorporate invariance to 3D translation by acting on relative positions, but do not consider inversion.

ESCN Convolution

Message passing is used to update equivariant irreps features and is typically implemented as SO(3)SO(3) convolutions. A traditional SO(3)SO(3) convolution interacts input irrep features xmi(Li)x_{m_i}^{(L_i)} and spherical harmonic projections of relative positions Ymf(Lf)(r⃗ts)Y_{m_f}^{(L_f)}(\vec{r}_{ts}) with an SO(3)SO(3) tensor product with Clebsch-Gordan coefficients C(Li,mi),(Lf,mf)(Lo,mo)C_{(L_i,m_i),(L_f,m_f)}^{(L_o,m_o)}. Since tensor products are compute-intensive, eSCN convolutions [39] are proposed to reduce the complexity of tensor products when they are used in SO(3)SO(3) convolutions. Rotating the input irreps features xmi(Li)x_{m_i}^{(L_i)} based on the relative position vectors r⃗ts\vec{r}_{ts} simplifies the tensor products and enables reducing SO(3)SO(3) convolutions to SO(2)SO(2) linear operations. Below we provide the mathematical details of SO(3)SO(3) convolutions built from tensor products and how rotation can reduce their computational complexity.

Tensor products interact type-LiL_i vector x(Li)x^{(L_i)} and type-LfL_f vector f(Lf)f^{(L_f)} to produce type-LoL_o vector y(Lo)y^{(L_o)} with Clebsch-Gordan coefficients C(Li,mi),(Lf,mf)(Lo,mo)C_{(L_i,m_i),(L_f,m_f)}^{(L_o,m_o)}. Clebsch-Gordan coefficients C(Li,mi),(Lf,mf)(Lo,mo)C_{(L_i,m_i),(L_f,m_f)}^{(L_o,m_o)} are non-zero only when ∣Li−Lo∣≤Lf≤∣Li+Lo∣\lvert L_i-L_o\rvert\le L_f \le\lvert L_i+L_o\rvert. Each non-trivial combination of Li⊗Lf→LoL_i \otimes L_f \to L_o is called a path, and each path is independently equivariant and can be assigned a learnable weight wLi,Lf,Low_{L_i,L_f,L_o}.

We consider the message mtsm_{ts} sent from source node ss to target node tt in an SO(3)SO(3) convolution. The LoL_o-th degree of mtsm_{ts} can be expressed as:

mts(Lo)=∑Li,LfwLi,Lf,Lo(xs(Li)⊗Li,LfY(Lf)(r^ts))(3)m_{ts}^{(L_o)}=\sum_{L_i,L_f}w_{L_i,L_f,L_o}\left(x_s^{(L_i)}\otimes_{L_i,L_f}Y^{(L_f)}(\hat{r}_{ts})\right) \tag*{(3)}

where xsx_s is the irreps feature at source node ss, xs(Li)x_s^{(L_i)} denotes the LiL_i-th degree of xsx_s, and r^ts=r⃗ts∣r⃗ts∣\hat{r}_{ts} = \frac{\vec{r}_{ts}}{\lvert\vec{r}_{ts}\rvert}.

The spherical harmonic projection of relative positions Y(Lf)(r^ts)Y^{(L_f)}(\hat{r}_{ts}) becomes sparse if we rotate r^ts\hat{r}_{ts} with a rotation matrix RtsR_{ts} to align with the direction of L=0L = 0 and m=0m = 0, which corresponds to the z axis traditionally but the y axis in the conventions of e3nn ([20]). Concretely, given Rtsr^tsR_{ts}\hat{r}_{ts} aligned with the y axis, Ymf(Lf)(Rtsr^ts)≠0Y_{m_f}^{(L_f)}(R_{ts}\hat{r}_{ts}) \ne0 only for mf=0m_f = 0. Without loss of equivariance, we re-scale Y0(Lf)(Rtsr^ts)Y_0^{(L_f)}(R_{ts}\hat{r}_{ts}) to be one. Besides, we denote D(Li)(Rts)=D(Li)D^{(L_i)}(R_{ts}) = D^{(L_i)} and D(Lo)(Rts)=D(Lo)D^{(L_o)}(R_{ts}) = D^{(L_o)} as Wigner-D matrices of degrees LiL_i and LoL_o based on rotation matrix RtsR_{ts}, respectively, and we define D(Li)xs(Li)=x~s(Li)D^{(L_i)}x_s^{(L_i)} = \tilde{x}_s^{(L_i)}. Therefore, by rotating xs(Li)x_s^{(L_i)} and Y(Lf)Y^{(L_f)} based on r^ts\hat{r}_{ts}, we can simplify Eq. (3) as follows:

mts(Lo)=(D(Lo)(Rts))−1∑Li,LfwLi,Lf,Lo(D(Li)(Rts)xs(Li)⊗Li,LfY(Lf)(Rtsr^ts))=(D(Lo))−1∑Li,LfwLi,Lf,Lo⨁mo(∑mi,mf(D(Li)xs(Li))miC(Li,mi),(Lf,mf)(Lo,mo)(Y(Lf)(Rtsr^ts))mf)=(D(Lo))−1∑Li,LfwLi,Lf,Lo⨁mo(∑mi(D(Li)xs(Li))miC(Li,mi),(Lf,0)(Lo,mo))=(D(Lo))−1∑Li,LfwLi,Lf,Lo⨁mo(∑mi(x~s(Li))miC(Li,mi),(Lf,0)(Lo,mo))(4)\begin{aligned} m_{ts}^{(L_o)} &= \left(D^{(L_o)}(R_{ts})\right)^{-1}\sum_{L_i,L_f}w_{L_i,L_f,L_o}\left(D^{(L_i)}(R_{ts})x_s^{(L_i)}\otimes_{L_i,L_f}Y^{(L_f)}(R_{ts}\hat{r}_{ts})\right)\\ &= \left(D^{(L_o)}\right)^{-1}\sum_{L_i,L_f}w_{L_i,L_f,L_o}\bigoplus_{m_o}\left(\sum_{m_i,m_f}\left(D^{(L_i)}x_s^{(L_i)}\right)_{m_i}C^{(L_o,m_o)}_{(L_i,m_i),(L_f,m_f)}\left(Y^{(L_f)}(R_{ts}\hat{r}_{ts})\right)_{m_f}\right)\\ &= \left(D^{(L_o)}\right)^{-1}\sum_{L_i,L_f}w_{L_i,L_f,L_o}\bigoplus_{m_o}\left(\sum_{m_i}\left(D^{(L_i)}x_s^{(L_i)}\right)_{m_i}C^{(L_o,m_o)}_{(L_i,m_i),(L_f,0)}\right)\\ &= \left(D^{(L_o)}\right)^{-1}\sum_{L_i,L_f}w_{L_i,L_f,L_o}\bigoplus_{m_o}\left(\sum_{m_i}\left(\tilde{x}_s^{(L_i)}\right)_{m_i}C^{(L_o,m_o)}_{(L_i,m_i),(L_f,0)}\right) \tag*{(4)} \end{aligned}

where ⨁\bigoplus denotes concatenation. Additionally, given mf=0m_f = 0, Clebsch-Gordan coefficients C(Li,mi),(Lf,0)(Lo,mo)C^{(L_o,m_o)}_{(L_i,m_i),(L_f,0)} are sparse and are non-zero only when mi=±mom_i = \pm m_o, which further simplifies Eq. (4):

mts(Lo)=(D(Lo))−1∑Li,LfwLi,Lf,Lo⨁mo((x~s(Li))moC(Li,mo),(Lf,0)(Lo,mo)+(x~s(Li))−moC(Li,−mo),(Lf,0)(Lo,mo))(5)m_{ts}^{(L_o)} = \left(D^{(L_o)}\right)^{-1}\sum_{L_i,L_f}w_{L_i,L_f,L_o}\bigoplus_{m_o}\left(\left(\tilde{x}_s^{(L_i)}\right)_{m_o}C^{(L_o,m_o)}_{(L_i,m_o),(L_f,0)}+\left(\tilde{x}_s^{(L_i)}\right)_{-m_o}C^{(L_o,m_o)}_{(L_i,-m_o),(L_f,0)}\right) \tag*{(5)}

By re-ordering the summations and concatenation in Eq. (5), we have:

(D(Lo))−1∑Li⨁mo((x~s(Li))mo∑Lf(wLi,Lf,LoC(Li,mo),(Lf,0)(Lo,mo))+(x~s(Li))−mo∑Lf(wLi,Lf,LoC(Li,−mo),(Lf,0)(Lo,mo)))(6)\left(D^{(L_o)}\right)^{-1}\sum_{L_i}\bigoplus_{m_o}\left(\left(\tilde{x}_s^{(L_i)}\right)_{m_o}\sum_{L_f}\left(w_{L_i,L_f,L_o}C^{(L_o,m_o)}_{(L_i,m_o),(L_f,0)}\right)+\left(\tilde{x}_s^{(L_i)}\right)_{-m_o}\sum_{L_f}\left(w_{L_i,L_f,L_o}C^{(L_o,m_o)}_{(L_i,-m_o),(L_f,0)}\right)\right) \tag*{(6)}

Instead of using learnable parameters for wLi,Lf,Low_{L_i,L_f,L_o}, eSCN proposes to parametrize w~mo(Li,Lo)\tilde{w}_{m_o}^{(L_i,L_o)} and w~−mo(Li,Lo)\tilde{w}_{-m_o}^{(L_i,L_o)} as below:

w~mo(Li,Lo)=∑LfwLi,Lf,LoC(Li,mo),(Lf,0)(Lo,mo)=∑LfwLi,Lf,LoC(Li,−mo),(Lf,0)(Lo,−mo)for m≥0\tilde{w}_{m_o}^{(L_i,L_o)}=\sum_{L_f}w_{L_i,L_f,L_o}C^{(L_o,m_o)}_{(L_i,m_o),(L_f,0)}=\sum_{L_f}w_{L_i,L_f,L_o}C^{(L_o,-m_o)}_{(L_i,-m_o),(L_f,0)}\qquad\text{for }m\geq0
w~−mo(Li,Lo)=∑LfwLi,Lf,LoC(Li,mo),(Lf,0)(Lo,mo)=−∑LfwLi,Lf,LoC(Li,−mo),(Lf,0)(Lo,mo)for m>0(7)\tilde{w}_{-m_o}^{(L_i,L_o)}=\sum_{L_f}w_{L_i,L_f,L_o}C^{(L_o,m_o)}_{(L_i,m_o),(L_f,0)}=-\sum_{L_f}w_{L_i,L_f,L_o}C^{(L_o,m_o)}_{(L_i,-m_o),(L_f,0)}\qquad\text{for }m>0 \tag*{(7)}

The parametrization of w~mo(Li,Lo)\tilde{w}_{m_o}^{(L_i,L_o)} and w~−mo(Li,Lo)\tilde{w}_{-m_o}^{(L_i,L_o)} enables removing the summation over LfL_f and further simplifies the computation. By combining Eq. (6) and Eq. (7), we have:

mts(Lo)=(D(Lo))−1∑Li⨁mo(yts(Li,Lo))mom_{ts}^{(L_o)}=\left(D^{(L_o)}\right)^{-1}\sum_{L_i}\bigoplus_{m_o}\left(y_{ts}^{(L_i,L_o)}\right)_{m_o}
(yts(Li,Lo))mo=w~mo(Li,Lo)(x~s(Li))mo−w~−mo(Li,Lo)(x~s(Li))−mofor mo>0(yts(Li,Lo))−mo=w~−mo(Li,Lo)(x~s(Li))mo+w~mo(Li,Lo)(x~s(Li))−mofor mo>0(yts(Li,Lo))mo=w~mo(Li,Lo)(x~s(Li))mofor mo=0(8)\begin{aligned} \left(y_{ts}^{(L_i,L_o)}\right)_{m_o}&=\tilde{w}_{m_o}^{(L_i,L_o)}\left(\tilde{x}_s^{(L_i)}\right)_{m_o}-\tilde{w}_{-m_o}^{(L_i,L_o)}\left(\tilde{x}_s^{(L_i)}\right)_{-m_o}&&\text{for }m_o>0\\ \left(y_{ts}^{(L_i,L_o)}\right)_{-m_o}&=\tilde{w}_{-m_o}^{(L_i,L_o)}\left(\tilde{x}_s^{(L_i)}\right)_{m_o}+\tilde{w}_{m_o}^{(L_i,L_o)}\left(\tilde{x}_s^{(L_i)}\right)_{-m_o}&&\text{for }m_o>0\\ \left(y_{ts}^{(L_i,L_o)}\right)_{m_o}&=\tilde{w}_{m_o}^{(L_i,L_o)}\left(\tilde{x}_s^{(L_i)}\right)_{m_o}&&\text{for }m_o=0 \tag*{(8)} \end{aligned}

The formulation of yts(Li,Lo)y_{ts}^{(L_i,L_o)} coincides with performing SO(2)SO(2) linear operations [64, 39]. Additionally, eSCN convolutions can further simplify the computation by considering only a subset of mom_o components in Eq. (8), i.e., ∣mo∣≤Mmax⁡\lvert m_o\rvert\le M_{\max}.

In summary, efficient SO(3)SO(3) convolutions can be achieved by first rotating irreps features xs(Li)x_s^{(L_i)} based on relative position vectors r⃗ts\vec{r}_{ts} and then performing SO(2)SO(2) linear operations on rotated features. The key idea is that rotation simplifies the computation as in Eq. (4), Eq. (5), Eq. (7), and Eq. (8). Please refer to their work [39] for more details. We note that eSCN convolutions consider only simplifying the case of taking tensor products between input irreps features and spherical harmonic projections of relative position vectors. eSCN convolutions do not simplify general cases such as taking tensor products between input irreps features and themselves [2] since the relative position vectors used to rotate irreps features are not clearly defined.

S2S^2 Activation

S2S^2 activation was first proposed in Spherical CNNs [8]. Our implementation of S2S^2 activation is the same as that in e3nn [20], SCN [70] and eSCN [39]. Basically, we uniformly sample a fixed set of points on a unit sphere along the dimensions of longitude, parametrized by α∈[0,2π)\alpha\in[0, 2\pi), and latitude, parametrized by β∈[0,π)\beta\in[0,\pi). We set the resolutions RR of α\alpha and β\beta to be 18 when Lmax⁡=6L_{\max}=6, meaning that we will have 324 (= 18 ×\times 18) points. Once the points are sampled, they are kept the same during training and inference, and therefore there is no randomness. For each point on the unit sphere, we compute the spherical harmonics projection of degrees up to Lmax⁡L_{\max}. We consider an equivariant feature of CC channels and each channel contains vectors of all degrees from 0 to Lmax⁡L_{\max}. When performing S2S^2 activation, for each channel and for each sampled point, we first compute the inner product between the vectors of all degrees contained in one channel of the equivariant feature and the spherical harmonics projections of a sampled point. This results in R×R×CR \times R \times C values, where the first two dimensions, R×RR \times R, correspond to grid resolutions and the last dimension corresponds to channels. They can be viewed as 2D grid feature maps and treated as scalars, and we can apply any standard or typical activation functions like SiLU or use standard linear layers performing feature aggregation along the channel dimension. After applying these functions, we project back to vectors of all degrees by multiplying those values with their corresponding spherical harmonics projections of sampled points. The process is the same as performing a Fourier transform, applying some functions and then performing an inverse Fourier transform.

Moreover, since the inner products between one channel of vectors of all degrees and the spherical harmonics projections of sampled points sum over all degrees, the conversion to 2D grid feature maps implicitly considers the information of all degrees. Therefore, S2S^2 activation, which converts equivariant features into 2D grid feature maps, uses the information of all degrees to determine the non-linearity. In contrast, gate activation only uses vectors of degree 0 to determine the non-linearity of vectors of higher degrees. For tasks such as force predictions, where the information of degrees is critical, S2S^2 activation can be better than gate activation since S2S^2 activation uses all degrees to determine non-linearity.

Although there is sampling on a sphere, the works [8, 39] mention that as long as the number of samples, or resolution RR, is high enough, the equivariance error can be close to zero. Furthermore, eSCN [39] empirically computes such errors in Figure 9 in their latest manuscript and shows that the errors of using Lmax⁡=6L_{\max}=6 and R=18R=18 are close to 0.2%, which is similar to the equivariance errors of tensor products in e3nn [20]. We note that the equivariance errors in e3nn are due to numerical precision.

Related Works

SE(3)/E(3)SE(3)/E(3)-Equivariant GNNs

Equivariant neural networks [55, 28, 63, 16, 36, 57, 3, 26, 49, 47, 60, 4, 54, 31, 2, 33, 39] (Musaeian et al. 2022) use equivariant irreps features built from vector spaces of irreducible representations (irreps) to achieve equivariance to 3D rotation [55, 63, 28]. They operate on irreps features with equivariant operations like tensor products. Previous works differ in equivariant operations used in their networks and how they combine those operations. TFN [55] and NequIP [3] use equivariant graph convolution with linear messages built from tensor products, with the latter utilizing extra equivariant gate activation [63]. SEGNN [4] introduces non-linearity to messages passing [21, 46] with equivariant gate activation, and the non-linear messages improve upon linear messages. SE(3)-Transformer [16] adopts equivariant dot product attention [61] with linear messages. Equiformer [33] improves upon previously mentioned equivariant GNNs by combining MLP attention and non-linear messages. Equiformer additionally introduces equivariant layer normalization and regularizations like dropout [53] and stochastic depth [24]. However, the networks mentioned above rely on compute-intensive SO(3)\mathrm{SO}(3) tensor products to mix the information of vectors of different degrees during message passing, and therefore they are limited to small values for maximum degrees Lmax⁡L_{\max} of equivariant representations. SCN [70] proposes rotating irreps features based on relative position vectors and identifies a subset of spherical harmonics coefficients, on which they can apply unconstrained functions. They further propose relaxing the requirement for strict equivariance and apply typical functions to rotated features during message passing, which trades strict equivariance for computational efficiency and enables using higher values of Lmax⁡L_{\max}. eSCN [39] further improves upon SCN by replacing typical functions with SO(2)\mathrm{SO}(2) linear layers for rotated features and imposing strict equivariance during message passing. However, except using more efficient operations for higher Lmax⁡L_{\max}, SCN and eSCN mainly adopt the same network design as SEGNN, which is less performant than Equiformer. In this work, we propose EquiformerV2, which includes all the benefits of the above networks by incorporating eSCN convolutions into Equiformer and adopts three additional architectural improvements.

Invariant GNNs

Prior works (Schütt et al., 2017; Xie & Grossman, 2018; Unke & Meuwly, 2019; Gasteiger et al., 2020b;a; Qiao et al., 2020; Liu et al., 2022; Shuaibi et al., 2021; Klicpera et al., 2021; Sriram et al., 2022; Gasteiger et al., 2022) extract invariant information from 3D atomistic graphs and operate on the resulting graphs augmented with invariant features. Their differences lie in leveraging different geometric features such as distances, bond angles (3 atom features) or dihedral angles (4 atom features). SchNet [48] models interaction between atoms with only relative distances. DimeNet series [18 a] use triplet representations of atoms to incorporate bond angles. SphereNet [34] and GemNet [27, 19] further include dihedral angles by considering quadruplet representations. However, the memory complexity of triplet and quadruplet representations of atoms do not scale well with the number of atoms, and this requires additional modifications like interaction hierarchy used by GemNet-OC [19] for large datasets like OC20 [7]. Additionally, for the task of predicting DFT calculations of energies and forces on the large-scale OC20 dataset, invariant GNNs have been surpassed by equivariant GNNs recently.

Details of Architecture

In this section, we define architectural hyper-parameters like maximum degrees and numbers of channels in certain layers in EquiformerV2, which are used to specify the detailed architectures in Sec. D.2, Sec. D.5, Sec. E.1 and Sec. F.2. Besides, we note that eSCN [39] and this work mainly consider SE(3)\mathrm{SE}(3)-equivariance.

We denote embedding dimensions as dembedd_{embed}, which defines the dimensions of most irreps features. Specifically, the output irreps features of all modules except the output head in Figure 1a have dimension dembedd_{embed}. For separable S2S^2 activation as illustrated in Figure 2c, we denote the resolution of point samples on a sphere as RR, which can depend on maximum degree LmaxL_{max}, and denote the unconstrained functions after projecting to point samples as FF.

For equivariant graph attention in Figure 1b, the input irreps features xix_i and xjx_j have dimension dembedd_{embed}. The dimension of the irreps feature fij(L)f_{ij}^{(L)} is denoted as dattn_hiddend_{attn\_hidden}. Equivariant graph attention can have hh parallel attention functions. For each attention function, we denote the dimension of the scalar feature fij(0)f_{ij}^{(0)} as dattn_alphad_{attn\_alpha} and denote the dimension of the value vector, which is in the form of irreps features, as dattn_valued_{attn\_value}. For the separable S2S^2 activation used in equivariant graph attention, the resolution of point samples is RR, and we use a single SiLU activation for FF. We share the layer normalization in attention re-normalization across all hh attention functions but have different hh linear layers after that. The last linear layer projects the dimension back to dembedd_{embed}. The two intermediate SO(2)SO(2) linear layers operate with maximum degree LmaxL_{max} and maximum order MmaxM_{max}.

For feed forward networks (FFNs) in Figure 1d, we denote the dimension of the output irreps features of the first linear layer as dffnd_{ffn}. For the separable S2S^2 activation used in FFNs, the resolution of point samples is RR, and FF consists of a two-layer MLP, with each linear layer followed by SiLU, and a final linear layer. The linear layers have the same number of channels as dffnd_{ffn}.

For radial functions, we denote the dimension of hidden scalar features as dedged_{edge}. For experiments on OC20, same as eSCN ([39]), we use Gaussian radial basis to represent relative distances and additionally embed the atomic numbers at source nodes and target nodes with two scalar features of dimension dedged_{edge}. The radial basis and the two embeddings of atomic numbers are fed to the radial function to generate edge distance embeddings.

The maximum degree of irreps features is denoted as LmaxL_{max}. All irreps features have degrees from 0 to LmaxL_{max} and have CC channels for each degree. We denote the dimension as (Lmax,C)(L_{max}, C). For example, irreps feature xirrepsx_{irreps} of dimension (6,128)(6, 128) has maximum degree 6 and 128 channels for each degree. The dimension of scalar feature xscalarx_{scalar} can be expressed as (0,Cscalar)(0, C_{scalar}).

Following Equiformer ([33]), we apply dropout ([53]) to attention weights and stochastic depth ([24]) to outputs of equivariant graph attention and feed forward networks. However, we do not apply dropout or stochastic depth to the output head.

Details of Experiments on OC20

Detailed Description of OC20 Dataset

The large and diverse OC20 dataset ([7]) (Creative Commons Attribution 4.0 License) consists of 1.2M DFT relaxations for training and evaluation, computed with the revised Perdew-Burke-Ernzerhof (RPBE) functional ([23]). Each structure in OC20 has an adsorbate molecule placed on a catalyst surface, and the core task is Structure to Energy Forces (S2EF), which is to predict the energy of the structure and per-atom forces. Models trained for the S2EF task are evaluated on energy and force mean absolute error (MAE). These models can in turn be used for performing structure relaxations by using the model’s force predictions to iteratively update the atomic positions until a relaxed structure corresponding to a local energy minimum is found. These relaxed structure and energy predictions are evaluated on the Initial Structure to Relaxed Structure (IS2RS) and Initial Structure to Relaxed Energy (IS2RE) tasks. The “All” split of OC20 contains 1.3M training structures spanning 56 elements, the “2M” split consists of 38M structures, and the “2M” split has 2M structures. For validation and test splits, there are four sub-splits containing in-distribution adsorbates and catalysts (ID), out-of-distribution adsorbates (OOD Ads), out-of-distribution catalysts (OOD Cat), and out-of-distribution adsorbates and catalysts (OOD Both).

Training Details

Hyper-Parameters. We summarize the hyper-parameters for the base model setting on OC20 S2EF-2M dataset and the main results on OC20 S2EF-All and S2EF-All+MD datasets in Table 7. For the ablation studies on OC20 S2EF-2M dataset, when trained for 20 or 30 epochs as in Table 1b, we increase the learning rate from 2×10−42 \times10^{-4} to 4×10−44 \times10^{-4}. When using Lmax=8L_{max}=8 as in Table 1c, we increase the resolution of point samples RR from 18 to 20. We vary LmaxL_{max} and the widths for speed-accuracy trade-offs in Figure 4. Specifically, we first decrease LmaxL_{max} from 6 to 4. Then, we multiply hh and the number of channels of (dembed,dattn_hidden,dffn)(d_{embed}, d_{attn\_hidden}, d_{ffn}) by 0.75 and 0.5. We train all models for 30 epochs. The same strategy to scale down eSCN models is adopted for fair comparisons.

Hyper-parametersBase model setting on S2EF-2MEquiformerV2 (31M) on S2EF-All+MDEquiformerV2 (153M) on S2EF-All/S2EF-All+MD
OptimizerAdamWAdamWAdamW
Learning rate schedulingCosine learning rate with linear warmupCosine learning rate with linear warmupCosine learning rate with linear warmup
Warmup epochs0.10.010.01
Maximum learning rate2×10−42 \times10^{-4}4×10−44 \times10^{-4}4×10−44 \times10^{-4}
Batch size64512256 for S2EF-All, 512 for S2EF-All+MD
Number of epochs1231
Weight decay1×10−31 \times10^{-3}1×10−31 \times10^{-3}1×10−31 \times10^{-3}
Dropout rate0.10.10.1
Stochastic depth0.050.10.1
Energy coefficient λE\lambda_E242 for S2EF-All, 2, 4 for S2EF-All+MD
Force coefficient λF\lambda_F100100100
Gradient clipping norm threshold100100100
Model EMA decay0.9990.9990.999
Cutoff radius (Å)121212
Maximum number of neighbors202020
Number of radial bases600600600
Dimension of hidden scalar features in radial functions dedged_{\mathrm{edge}}(6, 128)(6, 128)(6, 128)
Maximum degree Lmax⁡L_{\max}646
Maximum order Mmax⁡M_{\max}223
Number of Transformer blocks12820
Embedding dimension dembedd_{\mathrm{embed}}(6, 128)(4, 128)(6, 128)
fij(L)f_{ij}^{(L)} dimension dattn_hiddend_{\mathrm{attn\_hidden}}(6, 64)(4, 64)(6, 64)
Number of attention heads hh888
fij(0)f_{ij}^{(0)} dimension dattn_alphad_{\mathrm{attn\_alpha}}(0, 64)(0, 64)(0, 64)
Value dimension dattn_valued_{\mathrm{attn\_value}}(6, 16)(4, 16)(6, 16)
Hidden dimension in feed forward networks dffnd_{ffn}(6, 128)(4, 128)(6, 128)
Resolution of point samples RR181818

Table 7. Hyper-parameters for the base model setting on OC20 S2EF-2M dataset and the main results on OC20 S2EF-All and S2EF-All+MD datasets.

Training Time, Inference Speed and Numbers of Parameters. Table 8 summarizes the training time, inference speed and numbers of parameters of models in Table 1a (Index 1, 2, 3, 4, 5), Table 1c, Table 1d and Table 2. V100 GPUs with 32GB are used to train all models. We use 16 GPUs to train each individual model on S2EF-2M dataset, 64 GPUs for S2EF-All, 64 GPUs for EquiformerV2 (31M) on S2EF-All+MD, and 128 GPUs for EquiformerV2 (153M) on S2EF-All+MD.

Training setAttention re-normalizationActivationNormalizationLmax⁡L_{\max}Mmax⁡M_{\max}Number of Transformer blocksTraining time (GPU-hours)Inference speed (Samples / GPU sec.)Number of parameters
S2EF-2M✗GateLN621296519.0691.06M
S2EF-2M✓GateLN621299819.0791.06M
S2EF-2M✓S2S^2LN6212147612.8081.46M
S2EF-2M✓Sep. S2S^2LN6212150512.5183.16M
S2EF-2M✓Sep. S2S^2SLN6212141213.2283.16M
S2EF-2M✓Sep. S2S^2SLN421296519.8644.83M
S2EF-2M✓Sep. S2S^2SLN821227097.86134.28M
S2EF-2M✓Sep. S2S^2SLN6312162311.9295.11M
S2EF-2M✓Sep. S2S^2SLN641227067.98102.14M
S2EF-2M✓Sep. S2S^2SLN661230527.13106.63M
S2EF-All✓Sep. S2S^2SLN6320204996.08153.60M
S2EF-All+MD (λE=2\lambda_E = 2)✓Sep. S2S^2SLN6320328346.08153.60M
S2EF-All+MD (λE=4\lambda_E = 4)✓Sep. S2S^2SLN4281693129.2131.06M
S2EF-All+MD (λE=4\lambda_E = 4)✓Sep. S2S^2SLN6320376926.08153.60M

Table 8. Training time, inference speed and numbers of parameters of different models trained on OC20 S2EF-2M, S2EF-All and S2EF-All+MD datasets. All numbers are measured on V100 GPUs with 32GB.

Details of Running Relaxations

A structural relaxation is a local optimization where atom positions are iteratively updated based on forces to minimize the energy of the structure. We perform ML relaxations using the LBFGS optimizer (quasi-Newton) implemented in the Open Catalyst Github repository (Chanussot* et al., [7]). The structural relaxations for OC20 IS2RE and IS2RS tasks are allowed to run for 200 steps or until the maximum predicted force per atom Fmax⁡≤0.02F_{\max} \leq0.02 eV/Å, and the relaxations for AdsorbML are allowed to run for 300 steps or until Fmax⁡≤0.02F_{\max} \leq0.02 eV/Å. These settings are chosen to be consistent with prior works. We run relaxations on V100 GPUs with 32GB. The computational cost of running relaxations with EquiformerV2 (153M) for OC20 IS2RE and IS2RS tasks is 1011 GPU-hours, and that of running ML relaxations for AdsorbML is 1075 GPU-hours. The time for running relaxations with EquiformerV2 (31M) is 240 GPU-hours for OC20 IS2RE and IS2RS and 298 GPU-hours for AdsorbML.

Details of AdsorbML

We run the AdsorbML algorithm on the OC20-Dense dataset in accordance with the procedure laid out in the paper [30], which is summarized here:

  1. Run ML relaxations on all initial structures in the OC20-Dense dataset. There are around 1000 different adsorbate-surface combinations with about 90 adsorbate placements per combination, and therefore we have roughly 90k structures in total.

  2. Remove invalid ML relaxed structures based on physical constraints and rank the other ML relaxed structures in order of lowest to highest ML predicted energy.

  3. Take the top kk ML relaxed structures with the lowest ML predicted energies for each adsorbate-surface combination and run DFT single-point calculations. The single-point calculations are performed on the ML relaxed structures to improve the energy predictions without running a full DFT relaxation and are run with VASP using the same setting as the original AdsorbML experiments. As shown in Table 3, we vary kk from 1 to 5.

  4. Compute success and speedup metrics based on our lowest DFT single-point energy per adsorbate-surface combination and the DFT labels provided in the OC20-Dense dataset.

To better understand the speed-accuracy trade-offs of different models, we compare the AdsorbML success rate averaged over kk from 1 to 5 and average GPU-seconds of running one structure relaxation in Figure 5. We visualize some examples of relaxed structures from eSCN [39], EquiformerV2 and DFT in Figure 6.

Speed-accuracy trade-offs of different models when used in the AdsorbML algorithm

Figure 5. Speed-accuracy trade-offs of different models when used in the AdsorbML algorithm.

Qualitative examples of the initial configuration of an adsorbate on a catalyst surface and corresponding relaxed configurations

Figure 6. Qualitative examples of the initial configuration of an adsorbate on a catalyst surface (column 1), and corresponding relaxed configurations obtained from eSCN [39] (column 2), EquiformerV2 (column 3), and DFT (column 4). All examples are selected from the OC20-Dense dataset [30]. We show top-down views of each structure, with dashed lines showing the boundary of the unit cell repeating in the xx and yy directions.

Additional Comparison with Equiformer on OC20 IS2RE

Training Details. We follow the same setting as Equiformer [33] and train two EquiformerV2 models on OC20 IS2RE dataset without and with IS2RS auxiliary task. We use the same radial basis function as Equiformer. When IS2RS auxiliary task is adopted, we use a linearly decayed weight for loss associated with IS2RS, which starts at 15 and decays to 1 and adopt Noisy Nodes data augmentation [22]. The hyper-parameters are summarized in Table 9. We train EquiformerV2 on OC20 IS2RE with 16 V100 GPUs with 32GB and train on OC20 IS2RE with IS2RS auxiliary task and Noisy Nodes data augmentation with 32 V100 GPUs. The training costs are 574 and 2075 GPU-hours, and the numbers of parameters are 36.03M and 95.24M.

EquiformerV2 on OC20 IS2REEquiformerV2 on OC20 IS2RE with IS2RS auxiliary task and Noisy Nodes
OptimizerAdamWAdamW
Learning rate schedulingCosine learning rate with linear warmupCosine learning rate with linear warmup
Warmup epochs22
Maximum learning rate4×10−44 \times10^{-4}5×10−45 \times10^{-4}
Batch size3264
Number of epochs2040
Weight decay1×10−31 \times10^{-3}1×10−31 \times10^{-3}
Dropout rate0.20.2
Stochastic depth0.050.1
Gradient clipping norm threshold100100
Model EMA decay0.9990.999
Cutoff radius (Å)5.05.0
Maximum number of neighbors5050
Number of radial basis128128
Dimension of hidden scalar features in radial functions dedged_{edge}(0,64)(0, 64)(0,64)(0, 64)
Maximum degree Lmax⁡L_{\max}66
Maximum order Mmax⁡M_{\max}22
Number of Transformer blocks615
Embedding dimension dembedd_{embed}(6,128)(6, 128)(6,128)(6, 128)
fij(L)f_{ij}^{(L)} dimension dattn_hiddend_{attn\_hidden}(6,64)(6, 64)(6,64)(6, 64)
Number of attention heads hh88
fij(0)f_{ij}^{(0)} dimension dattn_alphad_{attn\_alpha}(0,64)(0, 64)(0,64)(0, 64)
Value dimension dattn_valued_{attn\_value}(6,16)(6, 16)(6,16)(6, 16)
Hidden dimension in feed forward networks dffnd_{ffn}(6,128)(6, 128)(6,128)(6, 128)
Resolution of point samples RR1818

Table 9. Hyper-parameters for OC20 IS2RE dataset.

Results. The comparison is shown in Table 10. Without IS2RS auxiliary task, EquiformerV2 overfits the training set due to higher degrees and achieves worse results than Equiformer. However, with IS2RS auxiliary task and Noisy Nodes data augmentation, EquiformerV2 achieves better energy MAE. The different rankings of models under different settings is also found in Noisy Nodes (Godwin et al., 2022), where they mention a node-level auxiliary task can prevent overfitting and enable more expressive models to perform better.

IS2RE directIS2RE direct with IS2RS auxiliary task and Noisy Nodes
ModelIDOOD AdsOOD CatOOD BothAverageIDOOD AdsOOD CatOOD BothAverage
Equiformer (Liao & Smidt, [33])508.8627.1505.1554.5548.9415.6497.6416.5434.4441.0
EquiformerV2516.1704.1524.5636.5595.3400.4459.0406.2401.8416.9

Table 10. OC20 IS2RE results on the validation split. With IS2RS auxiliary task and Noisy Nodes (Godwin et al., [22]) data augmentation, EquiformerV2 achieves better energy MAE.

Details of Experiments on OC22

Training Details

The hyper-parameters for OC22 dataset is summarized in Table 11. We use 32 V100 GPUs with 32GB and train two EquiformerV2 models with different energy coefficients λE\lambda_E and force coefficients λF\lambda_F. The number of parameters is 121.53M, and the training cost of each model is 4552 GPU-hours. The time for running relaxations for OC22 IS2RE is 38 GPU-hours.

Hyper-parametersValue or description
OptimizerAdamW
Learning rate schedulingCosine learning rate with linear warmup
Warmup epochs0.1
Maximum learning rate2×10−42 \times10^{-4}
Batch size128
Number of epochs6
Weight decay1×10−31 \times10^{-3}
Dropout rate0.1
Stochastic depth0.1
Energy coefficient λE\lambda_E1, 4
Force coefficient λF\lambda_F1, 100
Gradient clipping norm threshold50
Model EMA decay0.999
Cutoff radius (Å)12
Maximum number of neighbors20
Number of radial bases600
Dimension of hidden scalar features in radial functions dedged_{\mathrm{edge}}(0, 128)
Maximum degree LmaxL_{\mathrm{max}}6
Maximum order MmaxM_{\mathrm{max}}2
Number of Transformer blocks18
Embedding dimension dembedd_{\mathrm{embed}}(6, 128)
fij(L)f_{ij}^{(L)} dimension dattn_hiddend_{\mathrm{attn\_hidden}}(6, 64)
Number of attention heads hh8
fij(0)f_{ij}^{(0)} dimension dattn_alphad_{\mathrm{attn\_alpha}}(0, 64)
Value dimension dattn_valued_{\mathrm{attn\_value}}(6, 16)
Hidden dimension in feed forward networks dffnd_{\mathrm{ffn}}(6, 128)
Resolution of point samples RR18

Table 11. Hyper-parameters for OC22 dataset.

Details of Experiments on QM9

Additional Results of Training with Noisy Nodes

Similar to Sec. D.5, we train EquiformV2 on QM9 with Noisy Nodes (Godwin et al., [22]) to show that the performance gain brought by using higher degrees can be larger when trained with a node-level auxiliary task and data augmentation. In Table 12, we summarize the results and compare with previous works using Noisy Nodes and pre-training via denoising (Zaidi et al., [67]). When trained with Noisy Nodes, EquiformerV2 performs better than Equiformer on more tasks. Specifically, without Noisy Nodes, EquiformerV2 is better than Equiformer on 9 out of the 12 tasks, similar on 1 task, and worse on the other 2 tasks. With Noisy Nodes, EquiformerV2 achieves better MAE on 10 tasks, similar on 1 task, and worse on 1 task. Additionally, we note that EquiformerV2 with Noisy Nodes is overall better than GNS-TAT with both pre-training and Noisy Nodes (Zaidi et al., [67]) on 9 out of the 12 tasks even though EquiformerV2 is not pre-trained on PCQM4Mv2 dataset, which is more than 30×30\times larger than QM9. This shows that a more expressive model can match the performance with significantly less data.

ModelTaskα\alphaΔε\Delta\varepsilonεHOMO\varepsilon_{\mathrm{HOMO}}εLUMO\varepsilon_{\mathrm{LUMO}}μ\muCvC_vGGHHR2R^2UUU0U_0ZPVE
Unitsa03\text{a}_0^3meVmeVmeVDcal/mol KmeVmeVa02\text{a}_0^2meVmeVmeV
Equiformer (Liao & Smidt, [33]).046301514.011.0237.636.63.2516.746.591.26
Equiformer (Liao & Smidt, [33]) + NN†.04026.413.713.0.011.0205.494.61.2354.814.611.18
GNS-TAT + NN (Zaidi et al., [67]).04725.717.317.1.021.0227.416.420.656.396.391.080
GNS-TAT + NN + pretraining (Zaidi et al., [67])‡.04022.014.914.7.016.0206.905.790.445.765.761.018
EquiformerV2.050291413.010.0237.576.22.1866.496.171.47
EquiformerV2 + NN.03924.212.211.4.009.0205.344.24.1824.284.341.21

Table 12. Mean absolute error results on QM9 test set when trained with Noisy Nodes (Godwin et al., [22]). † denotes that the results are produced by this work. ‡ denotes using different data partitions. “NN” denotes Noisy Nodes, and “pretraining” denotes pretraining on PCQM4Mv2 dataset (Nakata & Shimazaki, [38]). The performance gain from Equiformer to EquiformerV2 becomes larger when trained with Noisy Nodes.

Training Details

We follow the data partition of Equiformer. For the tasks of μ\mu, α\alpha, εHOMO\varepsilon_{\mathrm{HOMO}}, εLUMO\varepsilon_{\mathrm{LUMO}}, Δε\Delta\varepsilon, and CvC_v, we use batch size = 64, the number of epochs = 300, learning rate = 5×10−45 \times10^{-4}, Gaussian radial basis functions with the number of bases = 128, the number of Transformer blocks = 6, weight decay = 5×10−35 \times10^{-3}, and dropout rate = 0.2 and use mixed precision for training. For the task of R2R^2, we use batch size = 48, the number of epochs = 300, learning rate = 1.5×10−41.5 \times10^{-4}, Gaussian radial basis functions with the number of bases = 128, the number of Transformer blocks = 5, weight decay = 5×10−35 \times10^{-3}, and dropout rate = 0.1 and use single precision for training. For the task of ZPVE, we use batch size = 48, the number of epochs = 300, learning rate = 1.5×10−41.5 \times10^{-4}, Gaussian radial basis functions with the number of bases = 128, the number of Transformer blocks = 5, weight decay = 5×10−35 \times10^{-3}, and dropout rate = 0.2 and use single precision for training. For the task of GG, HH, UU, and U0U_0, we use batch size = 48, the number of epochs = 300, learning rate = 1.5×10−41.5 \times10^{-4}, Gaussian radial basis functions with the number of bases = 128, the number of Transformer blocks = 5, weight decay = 0.0, and dropout rate = 0.0 and use single precision for training. Other hyper-parameters are the same across all the tasks, and we summarize them in Table 13. We use a single A6000 GPU and train different models for different tasks. The training costs are 72 GPU-hours for mixed precision training and 137 GPU-hours for single precision training. The number of parameters are 11.20M for 6 blocks and 9.35M for 5 blocks.

Hyper-parametersValue or description
OptimizerAdamW
Learning rate schedulingCosine learning rate with linear warmup
Warmup epochs5
Maximum learning rate1.5×10−4,5×10−41.5 \times10^{-4}, 5 \times10^{-4}
Batch size48, 64
Number of epochs300
Weight decay0.0,5×10−30.0, 5 \times10^{-3}
Dropout rate0.0,0.1,0.20.0, 0.1, 0.2
Stochastic depth0.0,0.050.0, 0.05
Cutoff radius (Å)5.0
Maximum number of neighbors500
Number of radial bases128
Dimension of hidden scalar features in radial functions dedged_{\mathrm{edge}}(0,64)(0, 64)
Maximum degree Lmax⁡L_{\max}4
Maximum order Mmax⁡M_{\max}4
Number of Transformer blocks5, 6
Embedding dimension dembedd_{\mathrm{embed}}(4,96)(4, 96)
fij(l)f^{(l)}_{ij} dimension dattn_hiddend_{\mathrm{attn\_hidden}}(4,48)(4, 48)
Number of attention heads hh4
fij(0)f^{(0)}_{ij} dimension dattn_alphad_{\mathrm{attn\_alpha}}(0,64)(0, 64)
Value dimension dattn_valued_{\mathrm{attn\_value}}(4,24)(4, 24)
Hidden dimension in feed forward networks dffnd_{\mathrm{ffn}}(4,96)(4, 96)
Resolution of point samples RR18
Noise standard deviation σdenoise\sigma_{\mathrm{denoise}}0.02
Denoising coefficient λdenoise\lambda_{\mathrm{denoise}}0.1
Denoising probability pdenoisep_{\mathrm{denoise}}0.5
Corrupt ratio rdenoiser_{\mathrm{denoise}}0.125, 0.25

Table 13. Hyper-parameters for QM9 dataset.

As for training with Noisy Nodes as mentioned in Sec. F.1, we add noise to atomic coordinates and incorporate a node-level auxiliary task of denoising atomic coordinates. We thus introduce four additional hyper-parameters, which are noise standard deviation σdenoise\sigma_{\mathrm{denoise}}, denoising coefficient λdenoise\lambda_{\mathrm{denoise}}, denoising probability pdenoisep_{\mathrm{denoise}} and corrupt ratio rdenoiser_{\mathrm{denoise}}. The noise standard deviation σdenoise\sigma_{\mathrm{denoise}} denotes the standard deviation of Gaussian noise added to each xyz component of atomic coordinates. The denoising coefficient λdenoise\lambda_{\mathrm{denoise}} controls the relative importance of the auxiliary task compared to the original task. The denoising probability pdenoisep_{\mathrm{denoise}} denotes the probability of adding noise to atomic coordinates and optimizing for both the auxiliary task and the original task. Using pdenoise<1p_{\mathrm{denoise}} < 1 enables taking original atomistic structures without any noise as inputs and optimizing for only the original task for some training iterations. The corrupt ratio rdenoiser_{\mathrm{denoise}} denotes the ratio of the number of atoms, which we add noise to and denoise, to the total number of atoms. Using rdenoise<1r_{\mathrm{denoise}} < 1 allows only adding noise to and denoising a subset of atoms within a structure. For the task of R2R^2, we use σdenoise=0.02\sigma_{\mathrm{denoise}} = 0.02, λdenoise=0.1\lambda_{\mathrm{denoise}} = 0.1, pdenoise=0.5p_{\mathrm{denoise}} = 0.5 and rdenoise=0.125r_{\mathrm{denoise}} = 0.125. For other tasks, we use σdenoise=0.02\sigma_{\mathrm{denoise}} = 0.02, λdenoise=0.1\lambda_{\mathrm{denoise}} = 0.1, pdenoise=0.5p_{\mathrm{denoise}} = 0.5 and rdenoise=0.25r_{\mathrm{denoise}} = 0.25. We share the above hyper-parameters for training EquiformerV2 and Equiformer, and we add one additional block of equivariant graph attention for the auxiliary task. We slightly tune other hyper-parameters when trained with Noisy Nodes. For Equiformer, we additionally use stochastic depth = 0.05 for the tasks of α\alpha, Δε\Delta\varepsilon, εHOMO\varepsilon_{\mathrm{HOMO}}, εLUMO\varepsilon_{\mathrm{LUMO}}, and CνC_\nu. As for EquiformerV2, we additionally use stochastic depth = 0.05 for the tasks of μ\mu, Δε\Delta\varepsilon, εHOMO\varepsilon_{\mathrm{HOMO}}, εLUMO\varepsilon_{\mathrm{LUMO}}, and CνC_\nu. We increase the number of blocks from 5 to 6 and increase the batch size from 48 to 64 for the tasks of GG, HH, UU, and U0U_0. We increase the learning rate from 1.5×10−41.5 \times10^{-4} to 5×10−45 \times10^{-4} and increase the number of blocks from 5 to 6 for the task of R2R^2.

Ablation Study on Architectural Improvements

We conduct ablation studies on the proposed architectural improvements using the task of Δε\Delta\varepsilon on QM9 and compare with Equiformer baseline Liao & Smidt, [33]. The results are summarized in Table 14. The comparison between Index 0 and Index 1 shows that directly increasing Lmax⁡L_{\max} from 2 to 4 and using eSCN convolutions degrade the performance. This is due to overfitting since the QM9 dataset is smaller, and each structure in QM9 has fewer atoms, less diverse atom types and much less angular variations than OC20 and OC22. Comparing Index 1 and Index 2, attention re-normalization clearly improves the MAE result. Although using S2S^2 activation is stable here (Index 3) unlike OC20, it results in higher error than using gate activation (Index 2) and the Equiformer baseline (Index 0). When using the proposed separable S2S^2 activation (Index 4), we achieve lower error than using gate activation (Index 2). We can further reduce the error by using the proposed separable layer normalization (Index 5). Comparing Index 0 and Index 5, we note that the proposed architectural improvements are necessary to achieve better results than the baseline when using higher degrees on QM9. Overall, these ablation results follow the same trends as OC20.

IndexAttention re-normalizationActivationNormalizationLmax⁡L_{\max}Δε\Delta\varepsilon MAE (meV)
0Equiformer baseline229.98
1✗GateLN430.46
2✓GateLN429.51
3✓S2S^2LN430.23
4✓Sep. S2S^2LN429.31
5✓Sep. S2S^2SLN429.03

Table 14. Ablation studies on the proposed architectural improvements using the task of Δε\Delta\varepsilon of the QM9 dataset.

References

  1. [1]Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization. arxiv preprint arxiv:1607.06450, 2016.arxiv.org/abs/1607.06450
  2. [2]Ilyes Batatia, David Peter Kovacs, Gregor N. C. Simm, Christoph Ortner, and Gabor Csanyi. MACE: Higher order equivariant message passing neural networks for fast and accurate force fields. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
  3. [3]Simon Batzner, Albert Musaelian, Lixin Sun, Mario Geiger, Jonathan P. Mailoa, Mordechai Kornbluth, Nicola Molinari, Tess E. Smidt, and Boris Kozinsky. E(3)-equivariant graph neural networks for data-efficient and accurate interatomic potentials. Nature Communications, 13(1), May 2022. doi: 10.1038/s41467-022-29939-5. URL https://doi.org/10.1038/s41467-022-29939-5.
  4. [4]Johannes Brandstetter, Rob Hesselink, Elise van der Pol, Erik J Bekkers, and Max Welling. Geometric and physical quantities improve e(3) equivariant message passing. In International Conference on Learning Representations (ICLR), 2022. URL https://openreview.net/forum?id=_xwr8g0BeV1.arxiv.org/abs/2110.02905
  5. [5]Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Advances in Neural Information Processing Systems (NeurIPS), 2020.arxiv.org/abs/2005.14165
  6. [6]Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European Conference on Computer Vision (ECCV), 2020.arxiv.org/abs/2005.12872
  7. [7]Lowik Chanussot*, Abhishek Das*, Siddharth Goyal*, Thibaut Lavril*, Muhammed Shuaibi*, Morgane Riviere, Kevin Tran, Javier Heras-Domingo, Caleb Ho, Weihua Hu, Aini Palizhati, Anuroop Sriram, Brandon Wood, Junwoong Yoon, Devi Parikh, C. Lawrence Zitnick, and Zachary Ulissi. Open catalyst 2020 (oc20) dataset and community challenges. ACS Catalysis, 2021. doi: 10.1021/acscatal.0c04525.DOI
  8. [8]Taco S. Cohen, Mario Geiger, Jonas Köhler, and Max Welling. Spherical CNNs. In International Conference on Learning Representations (ICLR), 2018. URL https://openreview.net/forum?id=Hkbd5xZRb.arxiv.org/abs/1801.10130
  9. [9]Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdulmohsin, Rodolphe Jenatton, Lucas Beyer, Michael Tschannen, Anurag Arnab, Xiao Wang, Carlos Riquelme, Matthias Minderer, Joan Puigcerver, Utku Evci, Manoj Kumar, Sjoerd van Steenkiste, Gamaleldin F. Elsayed, Aravindh Mahendran, Fisher Yu, Avital Oliver, Fantine Huot, Jasmijn Bastings, Mark Patrick Collier, Alexey Gritsenko, Vignesh Birodkar, Cristina Vasconcelos, Yi Tay, Thomas Mensink, Alexander Kolesnikov, Filip Pavetić, Dustin Tran, Thomas Kipf, Mario Lučić, Xiaohua Zhai, Daniel Keysers, Jeremiah Harmsen, and Neil Houlsby. Scaling vision transformers to 22 billion parameters. arXiv preprint arXiv:2302.05442, 2023.arxiv.org/abs/2302.05442
  10. [10]Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. arxiv preprint arxiv:1810.04805, 2019.arxiv.org/abs/1810.04805
  11. [11]Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations (ICLR), 2021. URL https://openreview.net/forum?id=YicbFdNTTy.
  12. [12]Mildred S Dresselhaus, Gene Dresselhaus, and Ado Jorio. Group theory. Springer, Berlin, Germany, 2008 edition, March 2007.
  13. [13]Vijay Prakash Dwivedi and Xavier Bresson. A generalization of transformer networks to graphs. arxiv preprint arXiv:2012.09699, 2020.arxiv.org/abs/2012.09699
  14. [14]Stefan Elfwing, Eiji Uchibe, and Kenji Doya. Sigmoid-weighted linear units for neural network function approximation in reinforcement learning. arXiv preprint arXiv:1702.03118, 2017.arxiv.org/abs/1702.03118
  15. [15]Nathan Frey, Ryan Sklaski, Simon Axelrod, Siddharth Samsi, Rafael Gomez-Bombarelli, Connor Coley, and Vijay Pandeppally. Neural scaling of deep chemical models. ChemRxiv, 2022. doi: 10.26434/chemrxiv-2022-3s512.DOI
  16. [16]Fabian Fuchs, Daniel E. Worrall, Volker Fischer, and Max Welling. Se(3)-transformers: 3d roto-translation equivariant attention networks. In Advances in Neural Information Processing Systems (NeurIPS), 2020.arxiv.org/abs/2006.10503
  17. [17]Johannes Gasteiger, Shankari Giri, Johannes T. Margraf, and Stephan G,nnemann. Fast and uncertainty-aware directional message passing for non-equilibrium molecules. In Machine Learning for Molecules Workshop, NeurIPS, 2020a.arxiv.org/abs/2011.14115
  18. [18]Johannes Gasteiger, Janek Groß, and Stephan G,nnemann. Directional message passing for molecular graphs. In International Conference on Learning Representations (ICLR), 2020b.arxiv.org/abs/2003.03123
  19. [19]Johannes Gasteiger, Muhammed Shuaibi, Anuroop Sriram, Stephan G,nnemann, Zachary Ulissi, C Lawrence Zitnick, and Abhishek Das. GemNet-OC: Developing Graph Neural Networks for Large and Diverse Molecular Simulation Datasets. Transactions on Machine Learning Research (TMLR), 2022.arxiv.org/abs/2204.02782
  20. [20]Mario Geiger, Tess Smidt, Alby M., Benjamin Kurt Miller, Wouter Boomsma, Bradley Dice, Kostiantyn Lapchevskyi, Maurice Weiler, Michal Tyszkiewicz, Simon Batzner, Dylan Madisetti, Martin Uhrin, Jes Frellsen, Nuri Jung, Sophia Sanborn, Mingjian Wen, Josh Rackers, Marcel Rød, and Michael Bailey. e3nn/e3nn: 2022-04-13, April 2022. URL https://doi.org/10.5281/zenodo.6459381.
  21. [21]Justin Gilmer, Samuel S. Schoenholz, Patrick F. Riley, Oriol Vinyals, and George E. Dahl. Neural message passing for quantum chemistry. In International Conference on Machine Learning (ICML), 2017.arxiv.org/abs/1704.01212
  22. [22]Jonathan Godwin, Michael Schaarschmidt, Alexander L Gaunt, Alvaro Sanchez-Gonzalez, Yulia Rubanova, Petar Velickovic, James Kirkpatrick, and Peter Battaglia. Simple GNN regularisation for 3d molecular property prediction and beyond. In International Conference on Learning Representations (ICLR), 2022. URL https://openreview.net/forum?id=1wWwek3o1b.arxiv.org/abs/2106.07971
  23. [23]B. Hammer, L. B. Hansen, and J. K. Nørskov. Improved adsorption energetics within density-functional theory using revised perdew-burke-ernzerhof functionals. Phys. Rev. B, 1999.DOI
  24. [24]Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q. Weinberger. Deep networks with stochastic depth. In European Conference on Computer Vision (ECCV), 2016.arxiv.org/abs/1603.09382
  25. [25]Weile Jia, Han Wang, Mohan Chen, Denghui Lu, Lin Lin, Roberto Car, Weinan E, and Linfeng Zhang. Pushing the limit of molecular dynamics with ab initio accuracy to 100 million atoms with machine learning. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC ’20. IEEE Press, 2020.arxiv.org/abs/2005.00223
  26. [26]Bowen Jing, Stephan Eismann, Patricia Suriana, Raphael John Lamarre Townshend, and Ron Dror. Learning from protein structure with geometric vector perceptrons. In International Conference on Learning Representations (ICLR), 2021. URL https://openreview.net/forum?id=1YLJDvSx6J4.arxiv.org/abs/2009.01411
  27. [27]Johannes Klicpera, Florian Becker, and Stephan Günnemann. Gemnet: Universal directional graph neural networks for molecules. In Advances in Neural Information Processing Systems (NeurIPS), 2021.arxiv.org/abs/2106.08903
  28. [28]Risi Kondor, Zhen Lin, and Shubhendu Trivedi. Clebsch–gordan nets: a fully fourier space spherical convolutional neural network. In Advances in Neural Information Processing Systems 32, pp. 10117–10126, 2018.arxiv.org/abs/1806.09231
  29. [30]Janice Lan, Aini Palizhati, Muhammed Shuaibi, Brandon M Wood, Brook Wander, Abhishek Das, Matt Uyttendaele, C Lawrence Zitnick, and Zachary W Ulissi. AdsorbML: Accelerating adsorption energy calculations with machine learning. arXiv preprint arXiv:2211.16486, 2022.
  30. [31]Tuan Le, Frank Noé, and Djork-Arné Clevert. Equivariant graph attention networks for molecular property prediction. arXiv preprint arXiv:2202.09891, 2022.arxiv.org/abs/2202.09891
  31. [32]Jae Hyeon Lee, Payman Yadollahpour, Andrew Watkins, Nathan C. Frey, Andrew Leaver-Fay, Stephen Ra, Kyunghyun Cho, Vladimir Gligorijevic, Aviv Regev, and Richard Bonneau. Equifold: Protein structure prediction with a novel coarse-grained structure representation. bioRxiv, 2022. doi: 10.1101/2022.10.07.511322. URL https://www.biorxiv.org/content/early/2022/10/08/2022.10.07.511322.DOI
  32. [33]Yi-Lun Liao and Tess Smidt. Equiformer: Equivariant graph attention transformer for 3d atomistic graphs. In International Conference on Learning Representations (ICLR), 2023. URL https://openreview.net/forum?id=KwmPfARgOTD.arxiv.org/abs/2206.11990
  33. [34]Yi Liu, Limei Wang, Meng Liu, Yuchao Lin, Xuan Zhang, Bora Oztekin, and Shuiwang Ji. Spherical message passing for 3d molecular graphs. In International Conference on Learning Representations (ICLR), 2022. URL https://openreview.net/forum?id=giysRXs0t9r.
  34. [35]Denghui Lu, Han Wang, Mohan Chen, Lin Lin, Roberto Car, Weinan E, Weile Jia, and Linfeng Zhang. 86 pflops deep potential molecular dynamics simulation of 100 million atoms with ab initio accuracy. Computer Physics Communications, 259:107624, 2021. ISSN 0010-4655. doi: https://doi.org/10.1016/j.cpc.2020.107624. URL https://www.sciencedirect.com/science/article/pii/S001046552030299X.
  35. [36]Benjamin Kurt Miller, Mario Geiger, Tess E. Smidt, and Frank Noé. Relevance of rotationally equivariant convolutions for predicting molecular properties. arXiv preprint arXiv:2008.08461, 2020.arxiv.org/abs/2008.08461
  36. [38]Maho Nakata and Tomomi Shimazaki. Pubchemqc project: A large-scale first-principles electronic structure database for data-driven chemistry. Journal of chemical information and modeling, 57 6: 1300–1308, 2017.DOI
  37. [39]Saro Passaro and C Lawrence Zitnick. Reducing SO(3) Convolutions to SO(2) for Efficient Equivariant GNNs. In International Conference on Machine Learning (ICML), 2023.arxiv.org/abs/2302.03655
  38. [41]Joshua A. Rackers, Lucas Tecot, Mario Geiger, and Tess E. Smidt. A recipe for cracking the quantum scaling limit with machine learned electron densities. Machine Learning: Science and Technology, 4(1):015027, feb 2023. doi: 10.1088/2632-2153/acb314. URL https://dx.doi.org/10.1088/2632-2153/acb314.
  39. [42]Prajit Ramachandran, Barret Zoph, and Quoc V. Le. Searching for activation functions. arXiv preprint arXiv:1710.05941, 2017.arxiv.org/abs/1710.05941
  40. [43]Raghunathan Ramakrishnan, Pavlo O Dral, Matthias Rupp, and O Anatole von Lilienfeld. Quantum chemistry structures and properties of 134 kilo molecules. Scientific Data, 1, 2014.
  41. [44]Albert Reuther, Jeremy Kepner, Chansup Byun, Siddharth Samsi, William Arcand, David Bestor, Bill Bergeron, Vijay Gadepally, Michael Houle, Matthew Hubbell, Michael Jones, Anna Klein, Lauren Milechin, Julia Mullen, Andrew Prout, Antonio Rosa, Charles Yee, and Peter Michaleas. Interactive supercomputing on 40,000 cores for machine learning and data analysis. In 2018 IEEE High Performance extreme Computing Conference (HPEC), pp. 1–6. IEEE, 2018.DOI
  42. [45]Lars Ruddigkeit, Ruud van Deursen, Lorenz C. Blum, and Jean-Louis Reymond. Enumeration of 166 billion organic small molecules in the chemical universe database gdb-17. Journal of Chemical Information and Modeling, 52(11):2864–2875, 2012. doi: 10.1021/ci300415d. URL https://doi.org/10.1021/ci300415d. PMID: 23088335.
  43. [46]Alvaro Sanchez-Gonzalez, Jonathan Godwin, Tobias Pfaff, Rex Ying, Jure Leskovec, and Peter W. Battaglia. Learning to simulate complex physics with graph networks. In International Conference on Machine Learning (ICML), 2020.
  44. [47]Víctor Garcia Satorras, Emiel Hoogeboom, and Max Welling. E(n) equivariant graph neural networks. In International Conference on Machine Learning (ICML), 2021.
  45. [48]K. T. Schütt, P.-J. Kindermans, H. E. Sauceda, S. Chmiela, A. Tkatchenko, and K.-R. Müller. Schnet: A continuous-filter convolutional neural network for modeling quantum interactions. In Advances in Neural Information Processing Systems (NeurIPS), 2017.
  46. [49]Kristof T. Schütt, Oliver T. Unke, and Michael Gastegger. Equivariant message passing for the prediction of tensorial properties and molecular spectra. In International Conference on Machine Learning (ICML), 2021.arxiv.org/abs/2102.03150
  47. [50]Yu Shi, Shuxin Zheng, Guolin Ke, Yifei Shen, Jiacheng You, Jiyan He, Shengjie Luo, Chang Liu, Di He, and Tie-Yan Liu. Benchmarking graphormer on large-scale molecular modeling datasets. arxiv preprint arxiv:2203.04810, 2022.arxiv.org/abs/2203.04810
  48. [51]Muhammed Shuaibi, Adeesh Kolluru, Abhishek Das, Aditya Grover, Anuroop Sriram, Zachary Ulissi, and C. Lawrence Zitnick. Rotation invariant graph neural networks using spin convolutions. arxiv preprint arxiv:2106.09575, 2021.arxiv.org/abs/2106.09575
  49. [52]Anuroop Sriram, Abhishek Das, Brandon M Wood, and C. Lawrence Zitnick. Towards training billion parameter graph neural networks for atomic simulations. In International Conference on Learning Representations (ICLR), 2022.arxiv.org/abs/2203.09697
  50. [53]Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(56):1929–1958, 2014. URL http://jmlr.org/papers/v15/srivastava14a.html.
  51. [54]Philipp Thölke and Gianni De Fabritiis. Equivariant transformers for neural network based molecular potentials. In International Conference on Learning Representations (ICLR), 2022. URL https://openreview.net/forum?id=NZhqz9wrRB.arxiv.org/abs/2202.02541
  52. [55]Nathaniel Thomas, Tess E. Smidt, Steven Kearnes, Lusann Yang, Li Li, Kai Kohlhoff, and Patrick Riley. Tensor field networks: Rotation- and translation-equivariant neural networks for 3d point clouds. arXiv preprint arXiv:1802.08219, 2018.arxiv.org/abs/1802.08219
  53. [56]Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. arXiv preprint arXiv:2012.12877, 2020.arxiv.org/abs/2012.12877
  54. [57]Raphael J. L. Townshend, Brent Townshend, Stephan Eismann, and Ron O. Dror. Geometric prediction: Moving beyond scalars. arXiv preprint arXiv:2006.14163, 2020.arxiv.org/abs/2006.14163
  55. [58]Richard Tran*, Janice Lan*, Muhammed Shuaibi*, Brandon Wood*, Siddharth Goyal*, Abhishek Das, Javier Heras-Domingo, Adeesh Kolluru, Ammar Rizvi, Nima Shoghi, Anuroop Sriram, Zachary Ulissi, and C. Lawrence Zitnick. The open catalyst 2022 (oc22) dataset and challenges for oxide electrocatalysis. arXiv preprint arXiv:2206.08917, 2022.DOI
  56. [60]Oliver Thorsten Unke, Mihail Bogojeski, Michael Gastegger, Mario Geiger, Tess Smidt, and Klaus Robert Muller. SE(3)-equivariant prediction of molecular wavefunctions and electronic densities. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems (NeurIPS), 2021. URL https://openreview.net/forum?id=auGY2UQfhSu.arxiv.org/abs/2106.02347
  57. [61]Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems (NeurIPS), 2017.arxiv.org/abs/1706.03762
  58. [62]Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph attention networks. In International Conference on Learning Representations (ICLR), 2018. URL https://openreview.net/forum?id=rJXMpikCZ.arxiv.org/abs/1710.10903
  59. [63]Maurice Weiler, Mario Geiger, Max Welling, Wouter Boomsma, and Taco Cohen. 3D Steerable CNNs: Learning Rotationally Equivariant Features in Volumetric Data. In Advances in Neural Information Processing Systems 32, pp. 10402–10413, 2018.arxiv.org/abs/1807.02547
  60. [64]Daniel E. Worrall, Stephan J. Garbin, Daniyar Turmukham betov, and Gabriel J. Brostow. Harmonic networks: Deep translation and rotation equivariance. arXiv preprint arXiv:1612.04642, 2016.arxiv.org/abs/1612.04642
  61. [66]Chengxuan Ying, Tianle Cai, Shengjie Luo, Shuxin Zheng, Guolin Ke, Di He, Yanming Shen, and Tie-Yan Liu. Do transformers really perform badly for graph representation? In Advances in Neural Information Processing Systems (NeurIPS), 2021. URL https://openreview.net/forum?id=Oew0ooxFwDa.
  62. [67]Sheheryar Zaidi, Michael Schaarschmidt, James Martens, Hyunjik Kim, Yee Whye Teh, Alvaro Sanchez-Gonzalez, Peter Battaglia, Razvan Pascanu, and Jonathan Godwin. Pre-training via denoising for molecular property prediction. In International Conference on Learning Representations (ICLR), 2023.arxiv.org/abs/2206.00133
  63. [68]A. Zee. Group Theory in a Nutshell for Physicists. Princeton University Press, USA, 2016.
  64. [69]Linfeng Zhang, Jiequn Han, Han Wang, Roberto Car, and Weinan E. Deep potential molecular dynamics: A scalable model with the accuracy of quantum mechanics. Phys. Rev. Lett., 120: 143001, Apr 2018. doi: 10.1103/PhysRevLett.120.143001. URL https://link.aps.org/doi/10.1103/PhysRevLett.120.143001.DOI
  65. [70]Larry Zitnick, Abhishek Das, Adeesh Kolluru, Janice Lan, Muhammed Shuaibi, Anuroop Sriram, Zachary Ulissi, and Brandon Wood. Spherical channels for modeling atomic interactions. In Advances in Neural Information Processing Systems (NeurIPS), 2022.arxiv.org/abs/2206.14331

Paper details

vlnc:W_dnh4ppfx
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents