betaFeedback

WorksAgricultural and Biological Sciences

ET-Flow: Equivariant Flow-Matching for Molecular Conformer Generation

Abstract

Predicting low-energy molecular conformations given a molecular graph is an important but challenging task in computational drug discovery. Existing state-of-the-art approaches either resort to large scale transformer-based models that diffuse over conformer fields, or use computationally expensive methods to generate initial structures and diffuse over torsion angles. In this work, we introduce Equivariant Transformer Flow (ET-Flow). We showcase that a well-designed flow matching approach with equivariance and harmonic prior alleviates the need for complex internal geometry calculations and large architectures, contrary to the prevailing methods in the field. Our approach results in a straightforward and scalable method that directly operates on all-atom coordinates with minimal assumptions. With the advantages of equivariance and flow matching, ET-Flow significantly increases the precision and physical validity of the generated conformers, while being a lighter model and faster at inference. Code is available https://github.com/shenoynikhil/ETFlow.

Introduction

Generating low-energy 3D representations of molecules, called conformers, from the molecular graph is a fundamental task in computational chemistry as the 3D structure of a molecule is responsible for several biological, chemical and physical properties [23, 52, 51, 22, 5]. Conventional approaches to molecular conformer generation consist of stochastic and systematic methods. While stochastic methods such as Molecular Dynamics (MD) accurately generate conformations, they can be slow, cost-intensive, and have low sample diversity [54, 7, 13, 24, 47]. Systematic (rule-based) methods [25, 10, 37, 43, 12, 34] that rely on torsional profiles and knowledge base of fragments are much faster but become less accurate

Correspondence to: {majdi.hassan,jungyoon.lee}@umontreal.ca and [email protected] with larger molecules. Therefore, there has been an increasing interest in developing scalable and accurate generative modeling methods in molecular conformer generation.

Existing machine learning based approaches use diffusion models [26, 58] to sample diverse and high quality samples given access to low-energy conformations. Prior methods typically fall into two categories: diffusing the atomic coordinates in the Cartesian space [67, 65] or diffusing along the internal geometry such as pairwise distances, bond angles, and torsion angles [21, 27].

Early approaches based on diffusion [53, 42, 67] faced challenges such as lengthy inference and training times as well as having lower accuracy compared to cheminformatics methods. Torsional Diffusion [27] was the first to outperform cheminformatics methods by diffusing only on torsion angles after producing an initial conformer with the cheminformatics tool RDKit. This reliance on RDKit structures instead of employing an end-to-end approach comes with several limitations, such as restricting the tool to applications where the local structures produced by RDKit are of sufficient accuracy. Unlike prior approaches, the current state-of-the-art MCF [65] proposes a domain-agnostic approach by learning to diffuse over functions by scaling transformers and learning soft inductive bias from the data [69]. Consequently, it comes with drawbacks such as high computational demands due to large number of parameters, limited sample efficiency from a lack of inductive biases like euclidean symmetries, and potential difficulties in scenarios with sparse data — a common challenge in this field.

In this paper, we propose Equivariant Transformer Flow (ET-Flow), a simple yet powerful flow-matching model designed to generate low-energy 3D structures of small molecules with minimal assumptions. We utilize flow matching [40, 41] (Albergo et al. 2023), which enables the learning of arbitrary probability paths beyond diffusion paths, enhancing both training and inference efficiency compared to conventional diffusion generative models. Departing from traditional equivariant architectures like EGNN [50], we adopt an Equivariant Transformer [61] to better capture geometric features. Additionally, our method integrates a Harmonic Prior [28, 60], leveraging the inductive bias that atoms connected by a bond should be in close proximity. We further optimize our flow matching objective by initially conducting rotational alignment on the harmonic prior, thereby constructing shorter probability paths between source and target distributions at minimal computational cost.

Our contributions can be summarized as follows:

  1. We obtain state-of-the-art precision for molecule conformer prediction, resulting in more physically realistic and reliable molecules for practitioners. We improve upon the previous methods by a large margin on ensemble property prediction.

  2. We highlight the effectiveness of incorporating equivariance and more informed priors in generating physically-grounded molecules in our simple yet well-engineered method.

  3. Our parameter-efficient model requires orders of magnitude fewer sampling steps than GeoDiff (Xu et al., [67]) and has significantly fewer parameters than MCF (Wang et al., [65]).

Background

Diffusion Generative Models. Diffusion models ([58, 57, 26]) enables a high-quality and diverse sampling from an unknown data distribution by approximating the Stochastic Differential Equation(SDE) that maps a simple density i.e. Gaussian to the unknown data density. Concretely, it involves training a neural network to learn the score, represented as ∇xlog⁡pt(x)\nabla_x \log p_t(x) of the diffused data. During inference, the model generates sample by iteratively solving the reverse SDE. However, diffusion models have inherent drawbacks, as they (i) require on longer training times (ii) are restricted to specific probability paths and (iii) depend on the use of complicated tricks to speed up sampling ([57, 68]).

Flow Matching. Flow Matching ([40, 41] (Albergo et al. 2023)) provides a general framework to learn Continuous normalizing flows (CNFs) while improving upon diffusion models in simplicity, generality, and inference speed in several applications. Through simple regression against the vector field reminiscent of the score-matching objective in diffusion models, Flow matching has enabled a fast, simulation-free training of CNFs. Several subsequent studies have then expanded the scope of flow matching objective to manifolds ([11]), arbitrary sources ([46]), and conditional flow matching with arbitrary transport maps and optimal couplings between source and target samples ([63]).

Molecular Conformer Generation. Various machine learning (ML) based approaches ([32, 39, 15, 56, 53, 42, 66, 21, 67, 27, 65]) have been developed to improve upon the limitations of conventional methods, among which the most advanced are TorsionDiff ([27]) and Molecular Conformer Fields (MCF) ([65]). TorsionDiff designs a diffusion model on the torsion angles while incorporating the local structure from RDKit ETKDG ([49]). MCF trains a diffusion model over functions that map elements from the molecular graph to points in 3D space.

Equivariant Architectures for Atomistic Systems. Inductive biases play an important role in generalization and sample efficiency. In the case of 3D atomistic modelling, one example of a useful inductive bias is the euclidean group SO(3)SO(3) which represents rotation equivariance in 3D space. Recently, various equivariant architectures ([17]) have been developed that act on both Cartesian ([50, 61, 55, 16, 19]) and spherical basis ([44, 9, 20, 38, 45, 3, 62]). For molecular conformer generation, initial methods like ConfGF, DGSM utilize invariant networks as they act upon inter-atomic distances, whereas the use of equivariant GNNs have been used in GeoDiff ([67]) and Torsional Diffusion ([27]). GeoDiff utilizes EGNN ([50]), a Cartesian basis equivariant architecture while Torsional Diffusion uses Tensor Field Networks ([62]) to output pseudoscalars.

Method

We design ET-Flow, a scalable equivariant model that generates energy-minimized conformers given a molecular graph. In this section, we layout the framework to achieve this objective by detailing the generative process in flow matching, the rotation alignment between distributions, stochastic sampling, and finally the architecture details.

Preliminaries We define notation that we use throughout this paper. Inputs are continuous atom positions x∈RN×3x \in\mathbb{R}^{N \times3} where NN is the number of atoms. We use the notation vt(x)v_t(x) interchangeably with v(t,x)v(t, x) for vector field.

Flow Matching

The aim is to learn a time-dependent vector field vt(x):RN×3×[0,1]→RN×3v_t(x) : \mathbb{R}^{N \times3} \times[0, 1] \to\mathbb{R}^{N \times3} associated with the transport map Xt:RN×3×[0,1]→RN×3X_t : \mathbb{R}^{N \times3} \times[0, 1] \to\mathbb{R}^{N \times3} that pushes forward samples from a base distribution ρ0\rho_0, often an easy-to-sample distribution, to samples from a more complex target distribution ρ1\rho_1, the low-energy conformations of a molecule. This can be defined as an ordinary differential equation (ODE),

X˙t(x)=vt(Xt(x)),Xt=0=x0,(1)\dot{X}_t(x) = v_t(X_t(x)), \qquad X_{t=0} = x_0, \tag*{(1)}

where x0∼ρ0x_0 \sim\rho_0. We can construct the vtv_t via a time-differentiable interpolation between samples from ρ0\rho_0 and ρ1\rho_1 that gives rise to a probability path ρt\rho_t that we can easily sample [40, 41, 63] (Albergo and Vanden-Eijnden 2023). The general interpolation between samples x0∼ρ0x_0 \sim\rho_0 and x1∼ρ1x_1 \sim\rho_1 can be defined as:

It(x0,x1)=αtx1+βtx0.(2)I_t(x_0, x_1) = \alpha_t x_1 + \beta_t x_0. \tag*{(2)}

Given this interpolant that couples x0x_0 and x1x_1, we can define the conditional probability path as ρt(x∣x0,x1)=N(x∣It(x0,x1),σt2I)\rho_t(x \mid x_0, x_1) = \mathcal{N}(x \mid I_t(x_0, x_1), \sigma_t^2 I), and the vector field can be computed as vt(x)=∂tρt(x∣x0,x1)v_t(x) = \partial_t \rho_t(x \mid x_0, x_1) which has the following form

vt(x)=α˙tx1+β˙tx0+σ˙tzz∼N(0,I).(3)v_t(x) = \dot{\alpha}_t x_1 + \dot{\beta}_t x_0 + \dot{\sigma}_t z \qquad z \sim\mathcal{N}(0, I). \tag*{(3)}

Here we use α˙t\dot{\alpha}_t as a shorthand notation for ∂tαt\partial_t \alpha_t, and similarly we apply the same notation to β\beta and σ\sigma. In our work, we use linear interpolation where αt=t\alpha_t = t, βt=1−t\beta_t = 1 - t, and σt=σt(1−t)\sigma_t = \sigma\sqrt{t(1 - t)}, resulting in the vector field

vt(x)=x1−x0+1−2t2t(1−t)z.(4)v_t(x) = x_1 - x_0 + \frac{1 - 2t}{2\sqrt{t(1 - t)}}z. \tag*{(4)}

Now, we can define the objective function for learning a vector field vθ(x)v_\theta(x) that generates a probability path ρt\rho_t between a base density ρ0\rho_0 and the target density ρ1\rho_1 as,

L=Et∼U(0,1),x∼ρt(x0,x1)∥v(t,x)−vθ(t,x)∥2.(5)\mathcal{L} = \mathbb{E}_{t \sim\mathcal{U}(0,1), x \sim\rho_t(x_0,x_1)} \lVert v(t, x) - v_\theta(t, x) \rVert^2. \tag*{(5)}

For training, we sample (i) x0∼ρ0x_0 \sim\rho_0, x1∼ρ1x_1 \sim\rho_1, and t∼U(0,1)t \sim\mathcal{U}(0, 1), (ii) interpolate according to Equation (2), (iii) add noise from a standard Gaussian, and (iv) minimize the loss defined in Equation (5). For sampling, we sample x0∼ρ0x_0 \sim\rho_0 and integrate from t=0t = 0 to t=1t = 1 using the Euler’s method. At each time-step, the Euler solver iteratively predicts the vector field for xtx_t and updates its position xt+Δt=xt+vθ(t,x)Δtx_{t+\Delta t} = x_t + v_\theta(t, x)\Delta t. More details on the training and sampling algorithms are provided in Appendix § 9.

Alignment

Several previous works [63, 33, 29, 59] demonstrate that constructing a straighter path between base distribution ρ0\rho_0 and target distribution ρ1\rho_1 minimizes the transport costs and improves performance. In our work, we reduce the transport costs between samples from the harmonic prior ρ0\rho_0 and samples from the data distribution ρ1\rho_1 by rotationally aligning them using the Kabsch algorithm [30] similar to [33, 29]. This approach leads to faster convergence and reduces the path length between atoms by leveraging the similarity in "shape" of the samples as seen in Figure 1a without incurring high computational cost.

Overview of ET-Flow and its architecture

Figure 1. (a) Overview of ET-Flow. The model predicts a conditional vector field v⃗θ\vec{v}_{\theta} using interpolated positions (xtx_t), molecular structure (GG), and time-step (tt). Samples are drawn from the harmonic prior (x0∼p0x_0 \sim p_0) and then rotationally aligned with the samples from data (x1∼p1x_1 \sim p_1). A conditional probability path is constructed between pairs of x0x_0 and x1x_1, and xtx_t is then sampled from this path at a random time tt. (b) The ET-Flow architecture consists of a representation module based on the TorchMD-NET architecture [61] and an equivariant vector output module. For detailed architecture and input preprocessing information, see Section A.1.

Stochastic Sampling

We employ a variant of the stochastic sampling technique inspired by [31]. Specifically, we inject noise at each time step to construct an intermediate state, evaluate the vector field from the intermediate state, and then perform the deterministic ODE step from the noisy state. The original method utilizes a second-order integration, which averages the denoiser output at the noisy intermediate state and the state at the next time step after integration.

In our experiment, we use the stochastic sampler without this second-order correction term, which empirically provided a performance boost comparable to the second-order method. We apply stochastic sampling only during the final part of the integration steps, specifically within the range t∈[0.8,1.0]t \in[0.8, 1.0]. This helps prevent drifting towards overpopulated density regions and improves the quality of the samples [31]. Stochastic sampling has improved both diversity and accuracy of the generated conformers, measured by Coverage and Average Minimum RMSD (AMR) respectively as shown in Table 1. Detailed information on the stochastic sampling algorithm is provided in [algorithm B](algorithm B).

RecallPrecision
Coverage ↑\uparrowAMR ↓\downarrowCoverage ↑\uparrowAMR ↓\downarrow
meanmedianmeanmedianmeanmedianmeanmedian
GeoDiff42.1037.800.8350.80924.9014.501.1361.090
GeoMol44.6041.400.8750.83443.0036.400.9280.841
Torsional Diff.72.7080.000.5820.56555.2056.900.7780.729
MCF - S (13M)79.487.50.5120.49257.457.60.7610.715
MCF - B (62M)84.091.50.4270.40264.066.20.6670.605
MCF - L (242M)84.792.20.3900.24766.871.30.6180.530
ET-Flow (8.3M)79.5384.570.4520.41974.3881.040.5410.470
ET-Flow - SS (8.3M)79.6284.630.4390.40675.1981.660.5170.442
ET-Flow - SO(3)SO(3) (9.1M)78.1883.330.4800.45967.2771.150.6370.567

Table 1. Molecule conformer generation results on GEOM-DRUGS (δ=0.75\delta= 0.75Å). ET-Flow - SS is ET-Flow with stochastic sampling and ET-Flow - SO(3)SO(3) is ET-Flow using the SO(3)SO(3) architecture for chirality correction. For ET-Flow, ET-Flow-SS and ET-Flow-SO(3)SO(3), we sample conformations over 50 time-steps.

Stochastic sampling procedure used in inference

Figure 2. Stochastic sampling procedure used in inference. Noise is added to the positions xtx_t indicated by the purple line, resulting in x^t\hat{x}_t. Then, the model predicts the vector field v^t\hat{v}_t from x^t\hat{x}_t instead of xtx_t indicted by the yellow line and updates x^t\hat{x}_t using v^t\hat{v}_t to get xt+1x_{t+1}.

Chirality Correction

While generating conformations, it is necessary to take account of the stereochemistry of atoms bonded to four distinct groups also referred to as tetrahedral chiral centers. To generate conformations with the correct chirality, we propose a simple post hoc trick as done in GeoMol [21]. We compare the oriented volume (OV) (Equation (6)) of the generated conformation and the required orientation with the RDKit tags. In the case of a mismatch, we simply flip the conformation against the zz-axis. This correction step can be efficiently performed as a batched operation since it involves a simple comparison with the required RDKit tags and an inversion of position if necessary.

OV⁡(p1,p2,p3,p4)=sign⁡((1111x1x2x3x4y1y2y3y4z1z2z3z4)).(6)\operatorname{OV}(p_1,p_2,p_3,p_4)=\operatorname{sign}\left(\begin{pmatrix} 1 & 1 & 1 & 1 \\ x_1 & x_2 & x_3 & x_4 \\ y_1 & y_2 & y_3 & y_4 \\ z_1 & z_2 & z_3 & z_4 \end{pmatrix}\right). \tag*{(6)}

We also consider an alternative approach for chirality correction. Instead of using the post hoc correction with our O(3)O(3) equivariant architecture, we slightly tweak our architecture to make it SO(3)SO(3) equivariant by introducing a cross product term in the update layers. We compare these methods on both the GEOM-DRUGS and GEOM-QM9 dataset in Table 1 and Table 2. Our base method (ET-Flow) corresponds to using the post hoc correction whereas the SO(3)SO(3) variant is referred by ET-Flow-SO(3)SO(3). We empirically observe that using an additional chirality correction step is not only computationally efficient, but also performs better. We provide details on the architectural modification and proof of SO(3)SO(3) equivariance in Section A.1 and Section C.1 respectively.

RecallPrecision
Coverage ↑\uparrowAMR ↓\downarrowCoverage ↑\uparrowAMR ↓\downarrow
meanmedianmeanmedianmeanmedianmeanmedian
CGCF69.4796.150.4250.37438.2033.330.7110.695
GeoDiff76.50100.000.2970.22950.0033.501.5240.510
GeoMol91.50100.000.2250.19387.60100.000.2700.241
Torsional Diff.92.80100.000.1780.14792.70100.000.2210.195
MCF95.0100.000.1030.04493.7100.000.1190.055
ET-Flow96.47100.000.0730.04794.05100.000.0980.039
ET-Flow - SO(3)SO(3)95.98100.000.0760.03092.10100.000.1100.047

Table 2. Molecule conformer generation results on GEOM-QM9 (δ=0.5\delta= 0.5Å). ET-Flow - SO(3)SO(3) is ET-Flow using the SO(3)SO(3) architecture for chirality correction. For both ET-Flow and ET-Flow-SO(3)SO(3), we sample conformations over 50 time-steps.

Architecture

ET-Flow (Figure 1b) consists of two main components: (1) a representation module based on the equivariant transformer architecture from TorchMD-NET Thölke and De Fabritiis [61] and (2) the equivariant vector output module. In the representation module, an embedding layer encodes the inputs (atomic positions, atomic numbers, atom features, bond features and the time-step) into a set of invariant features. Initial equivariant features are constructed using normalized edge vectors where the edges are constructed using a radius graph of 10 angstrom and the bonds from the 2D molecular graph. Then, a series of equivariant attention-based layers update both the invariant and equivariant features using a multi-head attention mechanism. Finally, the vector field is produced by the output layer, which updates the equivariant features using gated equivariant blocks [52]. Given that TorchMD-NET was originally designed for modeling neural network potentials, we implement several modifications to its architecture to better suit generative modeling, as detailed in Section A.1.

Experiments

We empirically evaluate ET-Flow by comparing the generated and ground-truth conformers in terms of distance-based RMSD (§ 4.2) and chemical property based metrics (§ 4.4). We present the general experimental setups in § 4.1. The implementation details are provided in Appendix § 8.

Experimental Setup

Dataset: We conduct our experiments on the GEOM dataset ([4]), which offers curated conformer ensembles produced through meta-dynamics in CREST ([48]). Our primary focus is on GEOM-DRUGS, the most extensive and pharmacologically relevant subset comprising 304k drug-like molecules, each with an average of 44 atoms. We use a train/validation/test (243473/30433/1000) split as provided in ([21]) Additionally, we train and test model on GEOM-QM9, a subset of smaller molecules with an average of 11 atoms. Finally, in order to assess the model’s ability to generalize to larger molecules, we evaluate the model trained on GEOM-DRUGS on a GEOM-XL dataset, a subset of large molecules with more than 100 atoms. The results for GEOM-QM9 and GEOM-XL can be found in the Appendix § 11.

Evaluation: Our evaluation methodology is similar to that of ([27]). First, we look at RMSD based metrics like Coverage and Average Minimum RMSD (AMR) between generated and ground truth conformer ensembles. For this, we generate 2K2K conformers for a molecule with KK ground truth conformers. Second, we look at chemical similarity using properties like Energy (EE), dipole moment (μ\mu), HOMO-LUMO gap (Δϵ\Delta\epsilon) and the minimum energy (Emin⁡E_{\min}) calculated using xTB ([8]).

Baselines: We benchmark ET-Flow against leading approaches outlined in § 2. Specifically, we assess the performance of GeoMol ([21]), GeoDiff ([67]), Torsional Diffusion ([27]), and MCF ([65]). Notably, the most recent among these, MCF, has demonstrated superior performance across evaluation metrics compared to its predecessors. It’s worth mentioning that GeoDiff initially utilized a limited subset of the GEOM-DRUGS dataset; thus, for a fair comparison, we consider its re-evaluated performance as presented in ([27]).

Ensemble RMSD

As shown in Table 1 and Table 2, ET-Flow outperforms all preceding methodologies and demonstrates competitive performance with the previous state-of-the-art, MCF ([65]). Despite being significantly smaller with only 8.3M parameters, ET-Flow shows a substantial improvement in the quality of generated conformers, as evidenced by superior Precision metrics across all MCF models, including the largest MCF-L. When compared to MCF-S, which is closer in size, ET-Flow achieves markedly better Precision while the impact on Recall is less significant and limited to Recall Coverage. Notably, our Recall AMR remains competitive with much bigger MCF-B, underscoring the inherent advantage of our method in accurately predicting overall structures.

Coverage Threshold Plots

We compare the coverage metrics of ET-Flow against Torsional diffusion Jing et al. [27] and MCF Wang et al. [65] against a wide range of thresholds on the GEOM DRUGS dataset in Figure 3. ET-Flow consistently outperforms previous methods in precision-based metrics. In terms of recall, our approach demonstrates better performance than Torsional Diffusion across all thresholds. Despite MCF performing better at higher thresholds, ET-Flow outperforms in the lower thresholds, underscoring its proficiency in generating accurate conformer predictions.

Recall and Precision Coverage plots on GEOM-DRUGS

Figure 3. Recall and Precision Coverage result on GEOM-DRUGS as a function of the threshold distance. ET-Flow outperforms TorsionDiff by a large margin especially in a lower threshold region. We emphasize the better performance of ET-Flow at lower thresholds in both Recall and Precision metrics.

Ensemble Properties

RMSD provides a geometric measure for assessing ensemble quality, but it is also essential to consider the chemical similarity between generated and ground truth ensembles. For a random 100-molecule subset of the test set of GEOM-DRUGS, if a molecule has KK ground truth conformers, we generate a minimum of 2K2K and a maximum of 32 conformers per molecule. These conformers are then relaxed using GFN2-xTB Bannwarth et al. [8], and the Boltzmann-weighted properties of the generated and ground truth ensembles are compared. Specifically, using xTB Bannwarth et al. 2019), we compute properties such as energy (EE), dipole moment (μ\mu), HOMO-LUMO gap (Δϵ\Delta\epsilon), and the minimum energy (Emin⁡E_{\min}). Table 3 illustrates the median errors for ET-Flow and the baselines, highlighting our method’s capability to produce chemically accurate ensembles. Notably, we achieve significant improvements over both TorsionDiff and MCF across all evaluated properties.

EEμ\muΔϵ\Delta\epsilonEmin⁡E_{\min}
OMEGA0.680.660.680.69
GeoDiff0.310.350.890.39
GeoMol0.420.340.590.40
Torsional Diff.0.220.350.540.13
MCF0.68±0.060.68\pm0.060.28±0.050.28\pm0.050.63±0.050.63\pm0.050.04±0.000.04\pm0.00
ET-Flow0.18±0.010.18\pm0.010.18±0.010.18\pm0.010.35±0.060.35\pm0.060.02±0.000.02\pm0.00

Table 3. Median averaged errors of ensemble properties between sampled and generated conformers (EE, Δϵ\Delta\epsilon, Emin⁡E_{\min} in kcal/mol, and μ\mu in debye).

Inference Steps Ablation

In Table 1, our sampling process with ET-Flow utilizes 50 inference steps. To evaluate the method’s performance under constrained computational resources, we conducted an ablation study by progressively reducing the number of inference steps. Specifically, we sample for 5, 10 and 20 time-steps. The results on GEOM-DRUGS are presented in Table 4. We observed minimal performance degradation with a decrease in the number of steps. Notably, ET-Flow demonstrates high efficiency, maintaining performance across all precision and recall metrics even with as few as 5 inference steps. Interestingly, ET-Flow with 5 steps still achieves superior precision metrics compared to all existing methods. This underscores ET-Flow’s ability to generate high-quality conformations while operating within limited computational budgets.

RecallPrecision
Coverage ↑AMR ↓Coverage ↑AMR ↓
meanmedianmeanmedianmeanmedianmeanmedian
ET-Flow (5 Steps)77.8482.210.4760.44374.0380.80.550.474
ET-Flow (10 Steps)79.0584.000.4510.41574.6481.380.5330.457
ET-Flow (20 Steps)79.2984.040.4490.41374.8981.320.5310.454
ET-Flow (50 Steps)79.5384.570.4520.41974.3881.040.5410.470

Table 4. Ablation over number of inference steps on GEOM-DRUGS (δ=0.75\delta= 0.75Å). Performance of ET-Flow at 5 steps is competent across all metrics while also retaining state-of-the-art performance on precision metrics when compared with previous methods.

Sampling Efficiency

We demonstrate the ability of ET-Flow to generate samples efficiently. We evaluate the inference time per molecule over varying number of time steps and report the average time across 1000 random samples from the test set of GEOM-DRUGS. Figure 4 shows that ET-Flow outperforms Torsional diffusion Jing et al., [27] in inference across all time steps. While ET-Flow may not achieve the fastest raw inference times (potentially due to MCF variants benefiting from optimized CUDA kernels for attention), it maintains competitive speeds while ensuring higher precision. We suspect that concurrent work on improving equivariant operations with optimized CUDA kernels Lee et al., [36] should lead to similar efficiency gains as seen in transformer-based architectures.

Sampling efficiency plots showing Cov-1 and AMR-P means versus time for different models and numbers of steps

Figure 4. Sampling efficiency as a measure of the quality of Inference time with respect to the number of time steps on GEOM-DRUGS.

ET-Flow effectively balances performance and speed, making it ideal for tasks that require high sample quality with efficient computation. With the ability to generate high-quality samples in fewer time steps, e.g., 5 time steps, as indicated in Table 4, ET-Flow is well-suited for scenarios demanding a large number of samples, as fewer steps lead to lower inference time per molecule. Additionally, we encountered difficulties running MCF-L for 20 and 50 steps, so those results have not been included. In summary, ET-Flow demonstrates efficient sampling, balancing precision and speed, making it highly effective for generating high-quality molecular samples while remaining competitive in inference time.

Conclusion

In this paper, we present our simple and scalable method ET-Flow, which utilizes an equivariant transformer with flow matching to achieve state-of-the-art performance on multiple molecular conformer generation benchmarks. By incorporating inductive biases, such as equivariance, and enhancing probability paths with a harmonic prior and RMSD alignment, we significantly improve the precision of the generated molecules, and consequently generate more physically plausible molecules. Importantly, our approach maintains parameter and speed efficiency, making it not only effective but also accessible for practical high-throughput applications.

Limitations And Future Works

While ET-Flow demonstrates competitive performance in molecular conformer generation, there are areas where it can be enhanced. One such area is the recall metrics, which capture the diversity of generated conformations. Another area is the use of an additional chirality correction step that is used to predict conformations with the desired chirality. Moreover, although our performance on the GEOM-XL dataset is comparable to MCF-S and TorsionDiff, there is still room for improvement.

We propose three future directions here. First, we observe during experiments that a well-designed sampling process incorporating stochasticity can enhance the quality and diversity of generated samples. An extension of our current approach could involve using Stochastic Differential Equations (SDEs), which utilize both vector field and score in the integration process, potentially improving the diversity of samples. Second, we propose to scale the number of parameters of ET-Flow, which has not only been shown to be useful across different domains of deep learning, but has also shown to be useful in molecular conformer generation for MCF [65]. Third, to better handle the chirality problem, we aim to explore alternatives for incorporating SO(3)SO(3)-equivariance into the model in the future.

Acknowledgements

The authors sincerely thank Cristian Gabellini, Jiarui Ding, and the NeurIPS reviewers for the insightful discussions and feedback. Resources used in completing this research were provided by Valence Labs. Furthermore, we acknowledge a grant for student supervision received by Mila - Quebec’s AI institute - and financed by the Quebec ministry of Economy.

Implementation Details

Architecture

The ET-Flow architecture (Figure 5) consists of 2 major components, a representation layer and an output layer. For the representation layer, we use a modified version of the embedding and equivariant attention-based update layers from the equivariant transformer architecture of TorchMD-NET (Thölke and De Fabritiis, [61]). The output layer utilizes the gated equivariant blocks from [52]. We highlight our modifications over the original TorchMD-NET architecture with this color. These modifications enable stabilized training since we use a larger network than the one proposed in the TorchMD-NET (Thölke and De Fabritiis, 2022}) paper. Additionally, since our input structures are interpolations between structures sampled from a prior and actual conformations, it is important to ensure our network is numerically stable when the interpolations contain two atoms very close to each other.

Overall architecture of ET-Flow, equivariant attention layer, and multi-head attention block

Figure 5. (a) Overall Architecture of ET-Flow consisting of 2 components, (1) Representation Layer based on TorchMD-NET Thölke and De Fabritiis [61] and (2) Equivariant Output Layer from [52]. (b) Equivariant Attention Layer with all the operations involved, (c) Multi-Head Attention block modified with the LayerNorm.

Embedding Layer: The embedding layer maps each atom’s physical and chemical properties into a learned representation space, capturing both local atomic features and geometric neighborhood information. For the ii-th atom in a molecule with NN atoms, we compute an invariant embedding xix_i through the following process:

zi=embed⁡int(zi)(7)z_i = \operatorname{embed}^{\mathrm{int}}(z_i) \tag*{(7)}
hi=MLP⁡(hi)(8)h_i = \operatorname{MLP}(h_i) \tag*{(8)}

where ziz_i is the atomic number and hih_i represents atomic attributes (detailed in Appendix § 8). The MLP projects atomic attributes into a feature vector of dimension dhd_h.

Next, we compute a neighborhood embedding nin_i that captures local atomic environment:

ni=∑j=1Nembed⁡nbh(zj)⋅g(dij,lij).(9)n_i = \sum_{j=1}^{N} \operatorname{embed}^{\mathrm{nbh}}(z_j) \cdot g(d_{ij}, l_{ij}). \tag*{(9)}

Here, embed⁡nbh(zj)\operatorname{embed}^{\mathrm{nbh}}(z_j) provides a separate embedding for neighboring atomic numbers, dijd_{ij} is the distance between atoms ii and jj, and lijl_{ij} encodes edge features (either from a radius-based graph or molecular bonds). The interaction function g(dij,lij)g(d_{ij}, l_{ij}) combines distance and edge information:

g(dij,lij)=WF[ϕ(dij)e1RBF(dij),…,ϕ(dij)eKRBF(dij),lij](10)g(d_{ij}, l_{ij}) = W^{\mathrm{F}}\left[\phi(d_{ij})e^{\mathrm{RBF}}_1(d_{ij}), \ldots, \phi(d_{ij})e^{\mathrm{RBF}}_K(d_{ij}), l_{ij}\right] \tag*{(10)}

where ekRBFe^{\mathrm{RBF}}_k are KK exponential radial basis functions following Unke and Meuwly [64], and ϕ(dij)\phi(d_{ij}) is a smooth cutoff function:

ϕ(dij)={12(cos⁡(πdijdcutoff)+1),if dij≤dcutoff0,otherwise(11)\phi(d_{ij}) = \begin{cases} \frac{1}{2}\left(\cos\left(\frac{\pi d_{ij}}{d_{\mathrm{cutoff}}}\right)+1\right), & \text{if } d_{ij} \le d_{\mathrm{cutoff}} \\ 0, & \text{otherwise} \end{cases} \tag*{(11)}

Finally, we combine all features into the atom’s embedding through a linear projection:

xi=WC[embed⁡int(zi),hi,t,ni](12)x_i = W^C \left[\operatorname{embed}^{\mathrm{int}}(z_i), h_i, t, n_i\right] \tag*{(12)}

where tt represents the time-step, and [⋅,⋅][\cdot,\cdot] denotes concatenation. The resulting embedding xi∈Rdx_i \in\mathbb{R}^d serves as input to subsequent layers of the network.

Attention Mechanism: The multi-head dot-product attention operation uses atom features xix_i, atom attributes hih_i, time-step tt and inter-atomic distances dijd_{ij} to compute attention weights. The input atom-level features xix_i are mixed with the atom attributes hih_i and the time-step tt using an MLP and then normalized using a LayerNorm (Ba et al., [6]). To compute the attention matrix, the inter-atomic distances dijd_{ij} are projected into two dimensional filters DKD^K and DVD^V as:

DK=σ(WDKeRBF(dij)+bDK)D^K = \sigma\left(W^{D^K}e^{RBF}(d_{ij}) + b^{D^K}\right)
DV=σ(WDVeRBF(dij)+bDV)(13)D^V = \sigma\left(W^{D^V}e^{RBF}(d_{ij}) + b^{D^V}\right) \tag*{(13)}

The atom level features are then linearly projected along with a LayerNorm operation to derive the query QQ and key KK vectors. The value vector VV is computed with only the linear projection of atom-level features. Applying LayerNorm on QQ, KK vectors (also referred to as QK-Norm) has proven to stabilize un-normalized values in the attention matrix (Dehghani et al., [14]; Esser et al., [18]) when scaling networks to large number of parameters. The QQ and KK vectors are then used along with the distance filter DKD^K for a dot-product operation over the feature dimension:

Q=LayerNorm⁡(WQxi),K=LayerNorm⁡(WKxi),V=WVxi(14)Q = \operatorname{LayerNorm}(W^Qx_i), \quad K = \operatorname{LayerNorm}(W^Kx_i), \quad V = W^Vx_i \tag*{(14)}
dot⁡(Q,K,DK)=∑k=1FQk⋅Kk⋅DkK.(15)\operatorname{dot}(Q,K,D^K) = \sum_{k=1}^{F} Q_k \cdot K_k \cdot D^K_k. \tag*{(15)}

The attention matrix is derived by passing the above dot-product operation matrix through a non-linearity and weighting it using a cosine cutoff ϕ(dij)\phi(d_{ij}) (similar to the embedding layer) which ensures the attention weights are non-zero only when two atoms are within a specified cutoff:

A=SiLU⁡(dot⁡(Q,K,DK))⋅ϕ(dij).(16)A = \operatorname{SiLU}(\operatorname{dot}(Q,K,D^K)) \cdot\phi(d_{ij}). \tag*{(16)}

Using the value vector VV and the distance filter DVD_V, we derive 3 equally sized filters by splitting along the feature dimension,

sij1,sij2,sij3=split⁡(Vj⋅DijV).(17)s_{ij}^{1}, s_{ij}^{2}, s_{ij}^{3} = \operatorname{split}(V_j \cdot D_{ij}^{V}). \tag*{(17)}

A linear projection is then applied to combine the attention matrix and the vectors sij3s_{ij}^{3} to derive an atom level feature yi=WO(∑jAij⋅sij3)y_i = W^O\left(\sum_j A_{ij} \cdot s_{ij}^{3}\right). The output of the attention operation are yiy_i (an atom level feature) and two scalar filters sij1s_{ij}^{1} and sij2s_{ij}^{2} (edge-level features).

Update Layer: The update layer computes interactions between atoms in the attention block and uses the outputs to update the scalar feature xix_i and the vector feature v⃗i\vec{v}_i. First, the scalar feature output yiy_i from the attention mechanism is split into three features (qi1q_i^1, qi2q_i^2, qi3q_i^3), out of which qi1q_i^1 and qi2q_i^2 are used for the scalar feature update as,

Δxi=qi1+qi2⋅⟨U1v⃗i⋅U2v⃗i⟩(18)\Delta x_i = q_i^1 + q_i^2 \cdot\langle U_1\vec{v}_i \cdot U_2\vec{v}_i\rangle \tag*{(18)}

where ⟨U1v⃗i⋅U2v⃗i⟩\langle U_1\vec{v}_i \cdot U_2\vec{v}_i\rangle is the inner product between linear projections of vector features v⃗i\vec{v}_i with matrices U1,U2U_1, U_2.

The edge vector update consists of two components. First, we compute a vector w⃗i\vec{w}_i, which for each atom is computed as a weighted sum of vector features and a clamped-norm of the edge vectors over all neighbors:

w⃗i=∑jNsij1⋅v⃗j+sij2⋅r⃗i−r⃗jmax⁡(∥r⃗i−r⃗j∥,ϵ)(19)\vec{w}_i = \sum_{j}^{N} s_{ij}^{1} \cdot\vec{v}_j + s_{ij}^{2} \cdot\frac{\vec{r}_i - \vec{r}_j}{\max(\lVert\vec{r}_i - \vec{r}_j \rVert,\epsilon)} \tag*{(19)}
Δv⃗i=w⃗i+qi3⋅U3v⃗i(20)\Delta\vec{v}_i = \vec{w}_i + q_i^3 \cdot U_3\vec{v}_i \tag*{(20)}

where U1U_1 and U3U_3 are projection matrices over the feature dimension of the vector feature v⃗i\vec{v}_i. In this layer, we clamp the minimum value of the norm (to ϵ=0.01\epsilon= 0.01) to prevent numerically large values in cases where positions of two atoms are sampled too close from the prior.

SO(3)SO(3) Update Layer: We also design an SO(3)SO(3) equivariant architecture by adding an additional cross product term in Equation (19) as follows,

w⃗i=∑jNsij1⋅v⃗j+sij2⋅r⃗i−r⃗jmax⁡(∥r⃗i−r⃗j∥,ϵ)+sij4⋅(v⃗j×r⃗i−r⃗jmax⁡(∥r⃗i−r⃗j∥,ϵ)),(21)\vec{w}_i = \sum_{j}^{N} s_{ij}^{1} \cdot\vec{v}_j + s_{ij}^{2} \cdot\frac{\vec{r}_i - \vec{r}_j}{\max(\lVert\vec{r}_i - \vec{r}_j \rVert,\epsilon)} + s_{ij}^{4} \cdot\left(\vec{v}_j \times\frac{\vec{r}_i - \vec{r}_j}{\max(\lVert\vec{r}_i - \vec{r}_j \rVert,\epsilon)}\right), \tag*{(21)}

where sij4s_{ij}^{4} is derived by modifying the split operation Equation (17) in the attention layer where the value vector V\mathbf{V} and distance filter DVD_{V} is projected into 4 equally sized filters instead of 3.

Output Layer: The output layer consists of Gated Equivariant Blocks from Schütt et al. [52]. Given atom scalar xix_i and vector features v⃗i\vec{v}_i, the updates in each block is defined as,

xi,updated,w⃗i=split⁡(MLP⁡([xi,U1v⃗i]))(22)x_{i,\mathrm{updated}}, \vec{w}_i = \operatorname{split}(\operatorname{MLP}([x_i, U_1\vec{v}_i])) \tag*{(22)}
v⃗i,updated=(U2v⃗i)⋅w⃗i(23)\vec{v}_{i,\mathrm{updated}} = (U_2\vec{v}_i) \cdot\vec{w}_i \tag*{(23)}

Here, U1U_1 and U2U_2 are linear projection matrices that act along feature dimension. Our modification is to use LayerNorm in the MLP to improve training stability.

Input Featurization

Atomic features (or Node Features) are computed using RDKit Landrum et al. [35] features as described in Table 5. For computing edge features and edge index, we use a combination of global (radius based edges) and local (molecular graph edges) similar to Jing et al. [27].

NameDescriptionRange
chiralityChirality Tag{unspecified,tetrahedral CW & CCW,other}\{\text{unspecified}, \text{tetrahedral CW \& CCW}, \text{other}\}
degreeNumber of bonded neighbors{x:0≤x≤10,x∈Z}\{x : 0 \le x \le10, x \in\mathbb{Z}\}
chargeFormal charge of atom{x:−5≤x≤5,x∈Z}\{x : -5 \le x \le5, x \in\mathbb{Z}\}
num_HTotal Number of Hydrogens{x:0≤x≤8,x∈Z}\{x : 0 \le x \le8, x \in\mathbb{Z}\}
number_radical_eNumber of Radical Electrons{x:0≤x≤4,x∈Z}\{x : 0 \le x \le4, x \in\mathbb{Z}\}
hybrizationHybrization type{sp,sp2,sp3,sp3d,sp3d2,other}\{\mathrm{sp}, \mathrm{sp}^{2}, \mathrm{sp}^{3}, \mathrm{sp}^{3}\mathrm{d}, \mathrm{sp}^{3}\mathrm{d}^{2}, \text{other}\}
aromaticWhether on a aromatic ring{True,False}\{\text{True}, \text{False}\}
in_ringWhether in a ring{True,False}\{\text{True}, \text{False}\}

Table 5. Atomic features included in ET-Flow.

Evaluation Metrics

Following the approaches of Ganea et al. [21], Xu et al. [67], Jing et al. [27], we utilize Average Minimum RMSD (AMR) and Coverage (COV) to assess the performance of molecular conformer generation. Here, CgC_g denotes the set of generated conformations, and CrC_r denotes the set of reference conformations. For both AMR and COV, we calculate and report Recall (R) and Precision (P). Recall measures the extent to which the generated conformers capture the ground-truth conformers, while Precision indicates the proportion of generated conformers that are accurate. The specific formulations for these metrics are detailed in the following equations:

AMR-R⁡(Cg,Cr)=1∣Cr∣∑R∈Crmin⁡R^∈CgRMSD⁡(R,R^)\operatorname{AMR-R}(C_g,C_r) = \frac{1}{|C_r|} \sum_{R \in C_r} \min_{\hat{R} \in C_g} \operatorname{RMSD}(R,\hat{R})
COV-R⁡(Cg,Cr)=1∣Cr∣∣{R∈Cr∣RMSD⁡(R,R^)<δ, R^∈Cg}∣\operatorname{COV-R}(C_g,C_r) = \frac{1}{|C_r|} \left|\{R \in C_r \mid\operatorname{RMSD}(R,\hat{R}) < \delta,\ \hat{R} \in C_g\}\right|
AMR-P⁡(Cr,Cg)=1∣Cg∣∑R^∈Cgmin⁡R∈CrRMSD⁡(R^,R)\operatorname{AMR-P}(C_r,C_g) = \frac{1}{|C_g|} \sum_{\hat{R} \in C_g} \min_{R \in C_r} \operatorname{RMSD}(\hat{R},R)
COV-P⁡(Cr,Cg)=1∣Cg∣∣{R^∈Cg∣RMSD⁡(R^,R)<δ, R∈Cr}∣\operatorname{COV-P}(C_r,C_g) = \frac{1}{|C_g|} \left|\{\hat{R} \in C_g \mid\operatorname{RMSD}(\hat{R},R) < \delta,\ R \in C_r\}\right|

A lower AMR score signifies improved accuracy, while a higher COV score reflects greater diversity in the generative model. Following Jing et al. [27], the threshold δ\delta is set to 0.50.5Å for GEOM-QM9 and 0.750.75Å for GEOM-DRUGS.

Training Details and Hyperparameters

For GEOM-DRUGS, we train ET-Flow for a fixed 250 epochs with a batch size of 64 and 5000 training batches per epoch per GPU on 8 A100 GPUs. For the learning rate, we use the Adam Optimizer with a cosine annealing learning rate which goes from a maximum of 10−310^{-3} to a minimum 10−710^{-7} over 250 epochs with a weight decay of 10−1010^{-10}. For GEOM-QM9, we train ET-Flow for 200 epochs with a batch size of 128, and use all of the training dataset per epoch on 4 A100 GPUs. We use the cosine annealing learning rate schedule with maximum of 8⋅10−48 \cdot10^{-4} to minimum of 10−710^{-7} over 100 epochs, post which the maximum is reduced by a factor of 0.05. We select checkpoints based on the lowest validation error.

Hyper-parameterET-Flow
num_layers20
hidden_channels160
num_heads8
neighbor_embeddingTrue
cutoff_lower0.0
cutoff_higher10.0
node_attr_dim8
edge_attr_dim1
reduce_opTrue
activationSiLU
attn_activationSiLU
# param8.3M

Table 6. Hyperparameters for ET-Flow

Training and Sampling Algorithm

The following algorithms go over the pseudo-code for the training and sampling procedure. For each molecule, we use up to 30 conformations with the highest boltzmann weights as provided by CREST [48] similar to that of [27].

Algorithm 1: Training procedure

Input: molecules [G0,…,GN][G_0,\ldots,G_N] each with true conformers [CG,1,…CG,KG][C_{G,1},\ldots C_{G,K_G}], the harmonic prior ρ0\rho_0, learning rate α\alpha, number of epochs NeN_e, initialized vector field vθv_\theta

Output: trained flow matching model vθv_\theta

for i←1i \leftarrow1 to NeN_e do

for GG in [G0,…,GN][G_0,\ldots,G_N] do

Sample t∼U[0,1]t \sim U[0,1] and C1∈[CG,1,…CG,KG]C_1 \in[C_{G,1},\ldots C_{G,K_G}];

Sample prior C0∼ρ0(G)C_0 \sim\rho_0(G);

Align C0←RMSDAlign⁡(C0,C1)C_0 \leftarrow\operatorname{RMSDAlign}(C_0,C_1);

Sample Ct=tC1+(1−t)C0+σ2t(1−t)z,z∼N(0,I)C_t = tC_1 + (1-t)C_0 + \sigma^2 t(1-t)z,\quad z \sim\mathcal{N}(0,I);

Construct vector field ut←x1−x0+1−2t2t(1−t)zu_t \leftarrow x_1-x_0+\frac{1-2t}{2\sqrt{t(1-t)}}z;

Compute loss L←∥vθ(t,Ct)−ut∥2\mathcal{L} \leftarrow\lVert v_\theta(t,C_t)-u_t\rVert^2;

Take gradient step θ←θ−α∇θL\theta\leftarrow\theta-\alpha\nabla_\theta\mathcal{L};

Algorithm 2: Inference procedure

Input: molecular graph GG, number conformers KK, number of sampling steps NN

Output: predicted conformers [C1,…CK][C_1,\ldots C_K]

for CC in [C1,…CK][C_1,\ldots C_K] do

sample prior C^∼ρ0(G)\hat{C} \sim\rho_0(G);

for n←0n \leftarrow0 to N−1N-1 do

Set t←nNt \leftarrow\frac{n}{N};

Set Δt←1N\Delta t \leftarrow\frac{1}{N};

Predict v^=vθ(t,C^)\hat{v} = v_\theta(t,\hat{C});

Update C^=C^+v^Δt\hat{C} = \hat{C}+\hat{v}\Delta t

Algorithm 3: Stochastic Sampler

Input: molecular graph GG, number conformers KK, number of sampling steps NN, stochasticity level churnchurn, stochastic sampling range [tmin,tmax][t_{min}, t_{max}]

Output: predicted conformers [C^1,…C^K][\hat{C}_1,\ldots\hat{C}_K]

for C^\hat{C} in [C^1,…C^K][\hat{C}_1,\ldots\hat{C}_K] do

sample prior C^∼ρ0(G)\hat{C} \sim\rho_0(G);

for n←0n \leftarrow0 to N−1N-1 do

Set t←nNt \leftarrow\frac{n}{N};

Set Δt←1N\Delta t \leftarrow\frac{1}{N};

Set γ←churnN\gamma\leftarrow\frac{churn}{N};

if t∈[tmin,tmax]t \in[t_{min},t_{max}] then

Sample ϵ∼N(0,I)\epsilon\sim\mathcal{N}(0,I);

Δt^←γ(1−t)\widehat{\Delta t} \leftarrow\gamma(1-t);

t^←max⁡(t−Δt^,0)\hat{t} \leftarrow\max(t-\widehat{\Delta t},0);

C^←C^+Δt^t2−t^2ϵ\hat{C} \leftarrow\hat{C}+\widehat{\Delta t}\sqrt{t^2-\hat{t}^2}\epsilon;

Predict v^=vθ(t^,C^)\hat{v}=v_\theta(\hat{t},\hat{C});

Set Δt←Δt+Δt^\Delta t \leftarrow\Delta t+\widehat{\Delta t};

else

Predict v^=vθ(t,C^)\hat{v}=v_\theta(t,\hat{C});

Update C^=C^+v^Δt\hat{C}=\hat{C}+\hat{v}\Delta t

Proofs

Designing SO(3) Equivariance

We show that we can modify the architecture in Section A.1 (Equation (18)) to produce a final vector output that satisfies rotation equivariance and reflection asymmetry. Let v⃗1\vec{v}_1 and v⃗2\vec{v}_2 be linearly independent non-zero vectors ∥v⃗1∥>0\lVert\vec{v}_1\rVert> 0, ∥v⃗2∥>0\lVert\vec{v}_2\rVert> 0, and ss be a scalar. We implement SO(3) equivariance by adding a vector with a cross product. We show that vector v⃗=v⃗1+s(v⃗1×v⃗2)\vec{v}=\vec{v}_1+s(\vec{v}_1\times\vec{v}_2), where v⃗1×v⃗2\vec{v}_1\times\vec{v}_2 denotes cross product of v⃗1\vec{v}_1 and v⃗2\vec{v}_2, satisfies anti-symmetry while maintaining rotation equivariance as follows,

Rv⃗1+s(Rv⃗1×Rv⃗2)=R(v⃗1)+sR(v⃗1×v⃗2)(24)R\vec{v}_1+s(R\vec{v}_1\times R\vec{v}_2)=R(\vec{v}_1)+sR(\vec{v}_1\times\vec{v}_2) \tag*{(24)}
=R(v⃗1+s(v⃗1×v⃗2))(25)=R(\vec{v}_1+s(\vec{v}_1\times\vec{v}_2)) \tag*{(25)}
−v⃗1+s(−v⃗1×−v⃗2)=−v⃗1+s(v⃗1×v⃗2)(26)-\vec{v}_1+s(-\vec{v}_1\times-\vec{v}_2)=-\vec{v}_1+s(\vec{v}_1\times\vec{v}_2) \tag*{(26)}
≠−(v⃗1+s(v⃗1×v⃗2))(27)\ne-(\vec{v}_1+s(\vec{v}_1\times\vec{v}_2)) \tag*{(27)}

This concludes the proof for rotation equivariance and reflection anti-symmetry.

Additional Results

Design Choice Ablations

We conduct a series of ablation studies to assess the influence of each component in the ET-Flow. Particularly, we re-run the experiments with (1) O(3) equivariance without chirality correction, (2) Absence of Alignment, (3) Gaussian Prior as a base distribution. We demonstrate that improving probability paths and utilizing an expressive equivariant architecture with correct symmetries are key components for ET-Flow to achieve state of the art performance. The ablations were ran with reduced settings (50 epochs; lr =1e−4=1e-4; 4 A100 gpus). Results are shown in Table D.1.

Results on GEOM-XL

We now assess how well a model trained on GEOM-DRUGS generalises to unseen molecules with large numbers of atoms, using the GEOM-XL dataset containing a total of 102 molecules. This provides insights into the model’s capacity to tackle larger molecules and out-of-distribution tasks. Upon executing the checkpoint provided by Torsional Diffusion, we encountered 27 failed cases for generation likely due to RDKit failures, similar to the observations in MCF albeit with slightly different exact numbers. In both experiments involving all 102 molecules and a subset of 75 molecules, ET-Flow achieves performance comparable to Torsional Diffusion and MCF-S, but falls short of matching the performance of MCF-B and MCF-L. It’s worth noting that MCF-B and MCF-L are significantly larger models, potentially affording them an advantage in generalization tasks. As part of our future work, we plan to scale up our model and conduct further tests to explore its performance in this regard.

RecallPrecision
Coverage ↑\uparrowAMR ↓\downarrowCoverage ↑\uparrowAMR ↓\downarrow
meanmedianmeanmedianmeanmedianmeanmedian
ET-Flow75.3782.350.5570.52958.9060.870.7420.690
ET-Flow (O(3)O(3))72.7479.210.5760.55654.8454.110.7940.739
ET-Flow (w/o Alignment)68.6774.710.6220.61147.0944.250.8700.832
ET-Flow (Gaussian Prior)66.5373.010.6400.62544.4140.880.9030.864

Table 7. Ablation results on GEOM-DRUGS.

AMR-P ↓\downarrowAMR-R ↓\downarrow# mols
meanmedianmeanmedian
GeoDiff2.922.623.353.15-
GeoMol2.472.393.303.14-
Tor. Diff.2.051.862.942.78-
MCF - S2.221.973.172.81102
MCF - B2.011.703.032.64102
MCF - L1.971.602.942.43102
ET-Flow (ours)2.311.933.312.84102
Tor. Diff.1.931.862.842.7177
MCF - S2.021.872.92.6977
MCF - B1.711.612.692.4477
MCF - L1.641.512.572.2677
ET-Flow (ours)2.001.802.962.6375

Table 8. Generalization results on GEOM-XL.

Additional Out-of-Distribution Results

To further evaluate the generalization performance of ET-Flow, we conduct two more out-of-distribution experiments in addition to GEOM-XL. First, we test the model on scaffold-based splits of the GEOM-QM9 and GEOM-DRUGS dataset, which offers a more challenging alternative to the standard random split. We split the datasets based on Murcko scaffolds of the molecules into an 80:10:10 ratio for train, validation, and test sets. We evaluate our method on 1000 randomly sampled molecules from the resulting test set. The second experiment involves training the model on GEOM-DRUGS and assessing its performance on GEOM-QM9, a dataset with significantly smaller molecules. This experiment complements the generalization task to larger molecules in GEOM-XL by assessing ability for ET-Flow to generalize to smaller molecules. The results, presented in Table 9, indicate that the model’s performance degrades only marginally on the scaffold-based split. Furthermore, the model demonstrates robust performance on GEOM-QM9 even when trained on GEOM-DRUGS.

RecallPrecision
Coverage ↑\uparrowAMR ↓\downarrowCoverage ↑\uparrowAMR ↓\downarrow
meanmedianmeanmedianmeanmedianmeanmedian
ET-Flow (QM9 RS)96.47100.000.0730.04794.05100.000.0980.039
ET-Flow (QM9 SS)95.00100.000.0830.02990.25100.000.1240.053
ET-Flow (DRUGS →\to QM9)86.68100.000.2180.16068.6975.300.3690.317
ET-Flow (DRUGS RS)79.5384.570.4520.41974.3881.040.5410.470
ET-Flow (DRUGS SS)76.0680.650.6440.54567.8374.190.5110.473

Table 9. Additional OOD results. We use RS and SS to indicate Random Split and Scaffold Split respectively.

Visualizations

Figure 6 shows randomly selected examples of sampled conformers from ET-Flow for GEOM-DRUGS. The left column is the reference molecule from the ground truth, and the remaining columns are samples generated with 50 sampling steps. Figure 7 showcases the ability for ET-Flow to generate quality samples with fewer sampling steps.

Examples of conformers of ground truth and ET-Flow for GEOM-DRUGS

Figure 6. Examples of conformers of ground truth and ET-Flow for GEOM-DRUGS.

Examples of conformers of ground truth and ET-Flow for different number of sampling steps

Figure 7. Examples of conformers of ground truth and ET-Flow for different number of sampling steps.

References

  1. [3]Brandon Anderson, Truong Son Hy, and Risi Kondor. Cormorant: Covariant molecular neural networks. Advances in neural information processing systems, 32, 2019.arxiv.org/abs/1906.04015
  2. [4]Simon Axelrod and Rafael Gomez-Bombarelli. Geom, energy-annotated molecular conformations for property prediction and molecular generation. Scientific Data, 9(1):185, 2022.DOI
  3. [5]Simon Axelrod and Rafael Gomez-Bombarelli. Molecular machine learning with conformer ensembles. Machine Learning: Science and Technology, 4(3):035025, 2023.arxiv.org/abs/2012.08452
  4. [6]Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450, 2016.
  5. [7]Andrew J Ballard, Stefano Martiniani, Jacob D Stevenson, Sandeep Somani, and David J Wales. Exploiting the potential energy landscape to sample free energy. Wiley Interdisciplinary Reviews: Computational Molecular Science, 5(3):273–289, 2015.DOI
  6. [8]Christoph Bannwarth, Sebastian Ehlert, and Stefan Grimme. Gfn2-xtb—an accurate and broadly parametrized self-consistent tight-binding quantum chemical method with multipole electrostatics and density-dependent dispersion contributions. Journal of Chemical Theory and Computation, 15(3):1652–1671, 2019. doi: 10.1021/acs.jctc.8b01176. URL https://doi.org/10.1021/acs.jctc.8b01176. PMID: 30741547.
  7. [9]Ilyes Batatia, David P Kovacs, Gregor Simm, Christoph Ortner, and Gábor Csányi. Mace: Higher order equivariant message passing neural networks for fast and accurate force fields. Advances in Neural Information Processing Systems, 35:11423–11436, 2022.
  8. [10]Evan E Bolton, Sunghwan Kim, and Stephen H Bryant. Pubchem3d: conformer generation. Journal of cheminformatics, 3:1–16, 2011.DOI
  9. [11]Ricky T. Q. Chen and Yaron Lipman. Flow matching on general geometries, 2024.arxiv.org/abs/2302.03660
  10. [12]Jason C Cole, Oliver Korb, Patrick McCabe, Murray G Read, and Robin Taylor. Knowledge-based conformer generation using the Cambridge structural database. Journal of Chemical Information and Modeling, 58(3):615–629, 2018.DOI
  11. [13]Marco De Vivo, Matteo Masetti, Giovanni Bottegoni, and Andrea Cavalli. Role of molecular dynamics and related methods in drug discovery. Journal of medicinal chemistry, 59(9):4035–4061, 2016.DOI
  12. [14]Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Peter Steiner, Mehdi Caron, Robert Geirhos, Ibrahim Alabdulmohsen, et al. Scaling vision transformers to 22 billion parameters. In International Conference on Machine Learning, pages 7480–7512. PMLR, 2023.arxiv.org/abs/2302.05442
  13. [15]Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real nvp. arXiv preprint arXiv:1605.08803, 2016.arxiv.org/abs/1605.08803
  14. [16]Weitao Du, He Zhang, Yuanqi Du, Qi Meng, Wei Chen, Nanning Zheng, Bin Shao, and Tie-Yan Liu. Se (3) equivariant graph neural networks with complete local frames. In International Conference on Machine Learning, pages 5583–5608. PMLR, 2022.arxiv.org/abs/2110.14811
  15. [17]Alexandre Duval, Simon V Mathis, Chaitanya K Joshi, Victor Schmidt, Santiago Miret, Fragkiskos D Malliaros, Taco Cohen, Pietro Liò, Yoshua Bengio, and Michael Bronstein. A hitchhiker’s guide to geometric gnns for 3d atomic systems. arXiv preprint arXiv:2312.07511, 2023.arxiv.org/abs/2312.07511
  16. [18]Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. arXiv preprint arXiv:2403.03206, 2024.arxiv.org/abs/2403.03206
  17. [19]Thorben Frank, Oliver Unke, and Klaus-Robert Müller. So3krates: Equivariant attention for interactions on arbitrary length-scales in molecular systems. Advances in Neural Information Processing Systems, 35:29400–29413, 2022.arxiv.org/abs/2205.14276
  18. [20]Fabian Fuchs, Daniel Worrall, Volker Fischer, and Max Welling. Se (3)-transformers: 3d roto-translation equivariant attention networks. Advances in neural information processing systems, 33:1970–1981, 2020.arxiv.org/abs/2006.10503
  19. [21]Octavian Ganea, Lagnajit Pattanaik, Connor Coley, Regina Barzilay, Klavs Jensen, William Green, and Tommi Jaakkola. Geomol: Torsional geometric generation of molecular 3d conformer ensembles. Advances in Neural Information Processing Systems, 34:13757–13769, 2021.arxiv.org/abs/2106.07802
  20. [22]Johannes Gasteiger, Janek Groβ, and Stephan Günnemann. Directional message passing for molecular graphs. arXiv preprint arXiv:2003.03123, 2020.arxiv.org/abs/2003.03123
  21. [23]Cristiano RW Guimarães, Alan M Mathiowetz, Marina Shalaeva, Gilles Goetz, and Spiros Liras. Use of 3d properties to characterize beyond rule-of-5 property space for passive permeation. Journal of chemical information and modeling, 52(4):882–890, 2012.DOI
  22. [24]Paul CD Hawkins. Conformation generation: the state of the art. Journal of chemical information and modeling, 57(8):1747–1756, 2017.DOI
  23. [25]Paul CD Hawkins, A Geoffrey Skillman, Gregory L Warren, Benjamin A Ellingson, and Matthew T Stahl. Conformer generation with omega: algorithm and validation using high quality structures from the protein databank and cambridge structural database. Journal of chemical information and modeling, 50(4):572–584, 2010.DOI
  24. [26]Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840–6851, 2020.arxiv.org/abs/2006.11239
  25. [27]Bowen Jing, Gabriele Corso, Jeffrey Chang, Regina Barzilay, and Tommi Jaakkola. Torsional diffusion for molecular conformer generation. Advances in Neural Information Processing Systems, 35:24240–24253, 2022.arxiv.org/abs/2206.01729
  26. [28]Bowen Jing, Ezra Erives, Peter Pao-Huang, Gabriele Corso, Bonnie Berger, and Tommi Jaakkola. Eigenfold: Generative protein structure prediction with diffusion models. arXiv preprint arXiv:2304.02198, 2023.arxiv.org/abs/2304.02198
  27. [29]Bowen Jing, Bonnie Berger, and Tommi Jaakkola. Alphafold meets flow matching for generating protein ensembles, 2024.arxiv.org/abs/2402.04845
  28. [30]W. Kabsch. A solution for the best rotation to relate two sets of vectors. Acta Crystallographica Section A, 32(5):922–923, Sep 1976. doi: 10.1107/S0567739476001873. URL https://doi.org/10.1107/S0567739476001873.
  29. [31]Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models, 2022.arxiv.org/abs/2206.00364
  30. [32]Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.arxiv.org/abs/1312.6114
  31. [33]Leon Klein, Andreas Krämer, and Frank Noé. Equivariant flow matching. Advances in Neural Information Processing Systems, 36, 2024.
  32. [34]David Lagorce, Tania Pencheva, Bruno O Villoutreix, and Maria A Miteva. Dg-ammos: A new tool to generate 3d conformation of small molecules using distance geometry and automated molecular mechanics optimization for in silico screening. BMC Chemical Biology, 9:1–10, 2009.DOI
  33. [35]Greg Landrum et al. Rdkit: A software suite for cheminformatics, computational chemistry, and predictive modeling. Greg Landrum, 8(31.10):5281, 2013.
  34. [36]Kin Long Kelvin Lee, Mikhail Galkin, and Santiago Miret. Deconstructing equivariant representations in molecular systems. In AI for Accelerated Materials Design - NeurIPS 2024, 2024. URL https://openreview.net/forum?id=pshyLoyzRn.arxiv.org/abs/2410.08131
  35. [37]Jiabo Li, Tedman Ehlers, Jon Sutter, Shikha Varma-O’Brien, and Johannes Kirchmair. Caesar: a new conformer generation algorithm based on recursive buildup and local rotational symmetry consideration. Journal of chemical information and modeling, 47(5):1923–1932, 2007.
  36. [38]Yi-Lun Liao, Brandon Wood, Abhishek Das, and Tess Smidt. Equiformerv2: Improved equivariant transformer for scaling to higher-degree representations. arXiv preprint arXiv:2306.12059, 2023.arxiv.org/abs/2306.12059
  37. [39]Leo Liberti, Carlile Lavor, Nelson Maculan, and Antonio Mucherino. Euclidean distance geometry and applications. SIAM review, 56(1):3–69, 2014.arxiv.org/abs/1205.0349
  38. [40]Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint arXiv:2210.02747, 2022.arxiv.org/abs/2210.02747
  39. [41]Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. arXiv preprint arXiv:2209.03003, 2022.arxiv.org/abs/2209.03003
  40. [42]Shitong Luo, Chence Shi, Minkai Xu, and Jian Tang. Predicting molecular conformation via dynamic graph score matching. Advances in Neural Information Processing Systems, 34:19784–19795, 2021.
  41. [43]Maria A Mitev, Frederic Guyon, and Pierre Tuffi; 1/2ry. Frog2: Efficient 3d conformation ensemble generator for small compounds. Nucleic acids research, 38(suppl_2):W622–W627, 2010.
  42. [44]Albert Musaelian, Simon Batzner, Anders Johansson, Lixin Sun, Cameron J Owen, Mordechai Kornbluth, and Boris Kozinsky. Learning local equivariant representations for large-scale atomistic dynamics. Nature Communications, 14(1):579, 2023.DOI
  43. [45]Saro Passaro and C Lawrence Zitnick. Reducing so (3) convolutions to so (2) for efficient equivariant gnns. In International Conference on Machine Learning, pages 27420–27438. PMLR, 2023.
  44. [46]Aram-Alexandre Pooladian, Heli Ben-Hamu, Carles Domingo-Enrich, Brandon Amos, Yaron Lipman, and Ricky T. Q. Chen. Multisample flow matching: Straightening flows with minibatch couplings, 2023.arxiv.org/abs/2304.14772
  45. [47]Philipp Pracht, Fabian Bohle, and Stefan Grimme. Automated exploration of the low-energy chemical space with fast quantum chemical methods. Physical Chemistry Chemical Physics, 22(14):7169–7192, 2020.
  46. [48]Philipp Pracht, Stefan Grimme, Christoph Bannwarth, Fabian Bohle, Sebastian Ehlert, Gereon Feldmann, Johannes Gorges, Marcel Müller, Tim Neudecker, Christoph Plett, et al. Crest—a program for the exploration of low-energy molecular chemical space. The Journal of Chemical Physics, 160(11), 2024.
  47. [49]Sereina Riniker and Gregory A Landrum. Better informed distance geometry: using what we know to improve conformation generation. Journal of chemical information and modeling, 55(12):2562–2574, 2015.DOI
  48. [50]Victor Garcia Satorras, Emiel Hoogeboom, and Max Welling. E (n) equivariant graph neural networks. In International conference on machine learning, pages 9323–9332. PMLR, 2021.
  49. [51]Kristof Schütt, Oliver Unke, and Michael Gastegger. Equivariant message passing for the prediction of tensorial properties and molecular spectra. In International Conference on Machine Learning, pages 9377–9388. PMLR, 2021.arxiv.org/abs/2102.03150
  50. [52]Kristof T Schütt, Huziel E Sauceda, P-J Kindermans, Alexandre Tkatсhenko, and K-R Müller. Schnet—a deep learning architecture for molecules and materials. The Journal of Chemical Physics, 148(24), 2018.DOI
  51. [53]Chence Shi, Shitong Luo, Minkai Xu, and Jian Tang. Learning gradient fields for molecular conformation generation. In International conference on machine learning, pages 9558–9568. PMLR, 2021.arxiv.org/abs/2105.03902
  52. [54]Jihyun Shim and Alexander D MacKರೆll Jr. Computational ligand-based rational design: role of conformational sampling and force fields in model development. MedChemComm, 2(5):356–370, 2011.pubmed.ncbi.nlm.nih.gov/21716805
  53. [55]Guillem Simeon and Gianni De Fabritiis. Tensornet: Cartesian tensor representations for efficient learning of molecular potentials. Advances in Neural Information Processing Systems, 36, 2024.arxiv.org/abs/2306.06482
  54. [56]Gregor NC Simm and José Miguel Hernández-Lobato. A generative model for molecular distance geometry. arXiv preprint arXiv:1909.11459, 2019.arxiv.org/abs/1909.11459
  55. [57]Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020.arxiv.org/abs/2010.02502
  56. [58]Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.arxiv.org/abs/1907.05600
  57. [59]Yuxuan Song, Jingjing Gong, Minkai Xu, Ziyao Cao, Yanyan Lan, Stefano Ermon, Hao Zhou, and Wei-Ying Ma. Equivariant flow matching with hybrid probability transport for 3d molecule generation. Advances in Neural Information Processing Systems, 36, 2024.DOI
  58. [60]Hannes Stark, Bowen Jing, Regina Barzilay, and Tommi Jaakkola. Harmonic prior self-conditioned flow matching for multi-ligand docking and binding site design. In NeurIPS 2023 AI for Science Workshop, 2023.arxiv.org/abs/2310.05764
  59. [61]Philipp Thölke and Gianni De Fabritiis. Torchmd-net: Equivariant transformers for neural network based molecular potentials. arXiv preprint arXiv:2202.02541, 2022.arxiv.org/abs/2202.02541
  60. [62]Nathaniel Thomas, Tess Smidt, Steven Kearnes, Lusann Yang, Li Li, Kai Kohlhoff, and Patrick Riley. Tensor field networks: Rotation-and translation-equivariant neural networks for 3d point clouds. arXiv preprint arXiv:1802.08219, 2018.arxiv.org/abs/1802.08219
  61. [63]Alexander Tong, Nikolay Malkin, Guillaume Huguet, Yanlei Zhang, Jarrid Rector-Brooks, Kilian Fatras, Guy Wolf, and Yoshua Bengio. Conditional flow matching: Simulation-free dynamic optimal transport. arXiv preprint arXiv:2302.00482, 2(3), 2023.
  62. [64]Oliver T Unke and Markus Meuwly. Physnet: A neural network for predicting energies, forces, dipole moments, and partial charges. Journal of chemical theory and computation, 15(6):3678–3693, 2019.
  63. [65]Yuyang Wang, Ahmed A. Elhag, Navdeep Jaitly, Joshua M. Susskind, and Miguel Angel Bautista. Swallowing the bitter pill: Simplified scalable conformer generation, 2024.arxiv.org/abs/2311.17932
  64. [66]Minkai Xu, Shitong Luo, Yoshua Bengio, Jian Peng, and Jian Tang. Learning neural generative dynamics for molecular conformation generation. arXiv preprint arXiv:2102.10240, 2021.arxiv.org/abs/2102.10240
  65. [67]Minkai Xu, Lantao Yu, Yang Song, Chence Shi, Stefano Ermon, and Jian Tang. Geodiff: A geometric diffusion model for molecular conformation generation. arXiv preprint arXiv:2203.02923, 2022.arxiv.org/abs/2203.02923
  66. [68]Qinsheng Zhang and Yongxin Chen. Fast sampling of diffusion models with exponential integrator. arXiv preprint arXiv:2204.13902, 2022.arxiv.org/abs/2204.13902
  67. [69]Peiye Zhuang, Samira Abnar, Jiatao Gu, Alex Schwing, Joshua M Susskind, and Miguel Angel Bautista. Diffusion probabilistic fields. In The Eleventh International Conference on Learning Representations, 2022.arxiv.org/abs/2303.00165

Paper details

vlnc:W_95qayxtp
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 8

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Agricultural and Biological Sciences

Not recorded for this paper

View statement
Looking for related papers…
Contents