betaFeedback

WorksComputer Science

Using pretrained graph neural networks with token mixers as geometric featurizers for conformational dynamics

Abstract

Identifying informative low-dimensional features that characterize dynamics in molecular simulations remains a challenge, often requiring extensive manual tuning and system-specific knowledge. Here, we introduce geom2vec, in which pretrained graph neural networks (GNNs) are used as universal geometric featurizers. By pretraining equivariant GNNs on a large dataset of molecular conformations with a self-supervised denoising objective, we obtain transferable structural representations that are useful for learning conformational dynamics without further fine-tuning. We show how the learned GNN representations can capture interpretable relationships between structural units (tokens) by combining them with expressive token mixers. Importantly, decoupling training the GNNs from training for downstream tasks enables analysis of larger molecular graphs (such as small proteins at all-atom resolution) with limited computational resources. In these ways, geom2vec eliminates the need for manual feature selection and increases the robustness of simulation analyses.

Introduction

Molecular dynamics simulations can provide atomistic insight into complex reaction dynamics, but their high dimensionality makes them hard to interpret. Analyzing simulations thus relies on identifying low-dimensional representations (features), but care is needed in choosing them because they can strongly impact conclusions[1, 2, 3, 4]. Often features are selected manually, but doing so relies on system-specific intuition. Because developing intuition is often the goal of simulations, many researchers instead have turned to machine-learning methods for constructing features that capture observed variance (e.g., principal component analysis)[5, 6, 7] or decorrelate slowly according to the simulation data (e.g., the variational approach for Markov processes, VAMP)[8, 9, 10, 11, 12]. While these objectives are not always aligned with the reaction of interest[3, 4, 13, 14], such features can serve as useful intermediaries for computing reaction-specific statistics[14, 15] that provide a principled way for evaluating mechanisms[16, 17, 18].

Generally the inputs to the machine-learning methods above are internal coordinates such as distances between selected atoms and dihedral angles because they are invariant to translations and rotations of the system. However, the nonlocal nature of these coordinates and/or their effects (e.g., the rotation of a dihedral angle in a polymer backbone) can make the resulting features both ineffective and hard to interpret, and these issues become more significant with system size[16, 17]. Additionally, it is not obvious how to represent permutationally invariant species such as solvent molecules with internal coordinates; recently introduced machine learning approaches for treating such species do not scale well with system size[19].

Because atoms and their interactions (through bond or through space) can be viewed as the nodes and edges of graphs, molecular information can be readily encoded in graph representations (e.g., graph neural networks, GNNs). Importantly, graph representations can be constructed in ways that respect the symmetries of molecular systems, with translational, rotational, and permutational invariance and equivariance. Equivariance allows, for example, GNNs to output forces that rotate with the system, and appears to improve learning[20, 21, 22]. Owing to both their conceptual appeal and their performance, GNNs now dominate machine learning for force fields and molecular property prediction[22, 23, 24, 25, 26]. They also are being successfully used to learn representations of larger molecules for tasks such as protein structure prediction, design, fold classification, and function prediction[27, 28, 29, 30].

The tasks above concern prediction of static properties, including structures. Because molecular dynamics trajectories consist of sequences of structures, GNNs should be useful for identifying features for computing reaction statistics, and several groups have combined GNNs with VAMP[8] to learn metastable states and relaxation time scales of both materials and biomolecules[31, 32, 33, 34, 35]. These groups report improved variational scores, convergence for shorter lag times, and more interpretable learned representations relative to VAMPnets based on fully connected networks. However, existing GNNs for analyzing dynamics do not readily scale to large numbers of atoms, so the graphs in these studies are small, either because the molecules are small, or only a subset of atoms (e.g., the Cα_\alpha atoms of proteins) are used as inputs. Furthermore, training these GNNs is computationally costly, limiting the number of architectures that can be explored and their use for other types of analyses.

The key idea of this paper is that GNNs can be pretrained using independent structural data prior to their use to analyze dynamics, thus decoupling GNN training from training for downstream tasks. Pretraining has transformed other domains such as natural language processing (NLP) and computer vision, enabling high-dimensional latent representations of words (“tokens”)[36] or images (“patches”)[37] to be learned by self-supervised, auto-regressive training. In NLP, word2vec pioneered the idea of using learnable vector representations for words by assigning them based on the word itself and its surrounding context[38]; these representations could then be used for diverse downstream tasks. Inspired by word2vec, we propose geom2vec, an approach that leverages pretrained GNNs to learn transferable vector representations for molecular geometries.

Various pretraining strategies have been tried in molecular contexts[27, 28, 30, 39, 40, 41], but in contrast to complex pretraining and encoding schemes devised for specific classes of molecules[27, 30], we use a scheme that can be applied generally. Building on the idea that corrupting data with noise and training a model to reconstruct the original data (denoising) can lead to learning meaningful representations for generative models[42, 43], Zaidi et al. [40] showed that denoising atomic coordinates of structures of organic molecules significantly improved GNN performance on a number of molecular property prediction benchmarks.

Here, we show that this simple pretraining scheme also enables analysis of molecular dynamics simulations. Specifically, we pretrain GNNs using the same denoising objective and a dataset of structures of organic molecules obtained with density functional theory[44] and analyze protein molecular dynamics simulations with the resulting representations. We consider two tasks: learning slowly decorrelating modes with VAMP[8] and identifying metastable states with the state predictive information bottleneck (SPIB) framework[45, 46]. We show that neural networks trained using the GNN representations can readily take all non-hydrogen atoms in a small protein as inputs, enabling, for example, discovery of side chain dynamics that are important for folding. By decoupling learning molecular representations from training for specific tasks, our method naturally accommodates alternative pretraining schemes[27, 47] and datasets[48] (e.g., ones specific to particular classes of molecules), as well as other possible tasks[49, 50, 51, 52].

Methods

The basic idea of our method is to pretrain a GNN using a suitable task (here, denoising molecular coordinates) and then to use it with the resulting parameter values fixed as a feature encoder for other (downstream) tasks, as summarized in Figure 1. In this section, we provide an overview of the network architecture that we use and then describe its elaboration for the pretraining and downstream tasks; further details are provided in the Appendices. We refer to the workflow of transforming the atomic coordinates to representation vectors (i.e., features) and the use of those vectors as “geom2vec.”

Figure 1. The geom2vec workflow. (a) A GNN encoder is pretrained using a denoising objective on a dataset of structures of diverse molecules. (b) Geometric representations for configurations from molecular simulations are obtained by performing inference with the pretrained GNN encoder. (c) The representations are used as inputs to a downstream task head (here, a VAMPnet or SPIB), which is trained separately.

Network architecture

As noted above, our goal is learn a mapping from Cartesian coordinates to representation vectors via a GNN. There are many existing GNN architectures from which to choose[23]. Here, we use the ViSNet[53] architecture, which is built on TorchMD-ET[54]. These are both equivariant geometric graph transformers with modified attention mechanisms that suppress interactions between distant atoms; ViSNet goes beyond TorchMD-ET in using (standard and improper) dihedral angle information in its internal representations. We take the activations after the last message-passing layer as the representations for downstream computations. These representations include three-dimensional vector features, which change appropriately with molecular translation and rotation, and one-dimensional scalar features, which are invariant to molecular translation and rotation. That is,

fGNN:R3N→R(1+3d)N,(1)f_{\mathrm{GNN}} : \mathbb{R}^{3N} \to\mathbb{R}^{(1+3d)N}, \tag*{(1)}

where NN is the number of atoms, and dd is the number of features associated with each atom (equal to the dimension of the last update layer). We represent the combined vector (vi∈R3d\mathbf{v}_i \in\mathbb{R}^{3d}) and scalar (xi∈Rdx_i \in\mathbb{R}^d) features for atom ii by hi∈R(1+3)dh_i \in\mathbb{R}^{(1+3)d}. We choose ViSNet because it was previously shown to give accurate molecular property predictions and accurate conformational distributions when used to learn a potential for molecular dynamics simulations.[53] We briefly introduce the TorchMD-ET and ViSNet architectures in Appendices A 2 and A 3, and we refer interested readers to the original publications and Ref. 55 for further details.

Pretraining by denoising

For the pretraining, we draw random displacements from a multivariate Gaussian distribution and add them to the Cartesian coordinates of the molecules in the training set; we then train the GNN to predict the displacements. This process is designed to encourage the network to learn representations that capture the geometry of the molecular conformations, and it can be viewed as learning a force field with energy minima close to the training set geometries[40]. We chose this objective because structural data are more readily available than energetic data, especially for large molecules such as proteins.

Here, we use the OrbNet Denali dataset, which consists of 215,000 molecules and complexes (with an average of 45 atoms) with 2.3 million conformations sampled from molecular trajectories[44]. We randomly select 10,000 conformations for validation and use the remainder for training.

Following Zaidi et al. [40], we pretrain the model by passing the (1+3)d(1+3)d features for each atom (graph node) from the ViSNet architecture through a gated equivariant block (Algorithm 1) that combines the dd scalar and vector features to obtain a three-dimensional vector that represents the predictions for the displacement of that atom. We train for a fixed number of epochs and save the parameters resulting in the lowest mean squared error (MSE) between the predicted and added atom displacements for the validation set; for this comparison, we normalize the added atom displacements to have zero mean and unit variance. Further details, including hyperparameters, are given in Appendix D. Depending on the network architecture, choice of hyperparameters, and graphics card available, the one-time pretraining can take from several hours to a few days.

Use of the representations

In this section, we discuss the operational details of using the pretrained GNN for downstream tasks in general (summarized in Figure 1). The specific downstream tasks that we use for our numerical demonstrations are described in Section III.

Atom selection

We first select the atoms of interest. For example, when analyzing simulations with explicit solvent, we may select only the solute coordinates. Given the molecular coordinates, the pretrained GNN yields a learned representation hih_i for each selected atom ii. One can choose to base the calculations for the downstream tasks on the atomic representation vectors and/or the sums over their dd scalar and vector elements, but in most cases we reduce the size of the graph by coarse-graining it.

Coarse-graining

Let S={S1,…,SM}S = \{S_1, \ldots, S_M\} be a partition of the atoms into MM disjoint subsets, where each subset SmS_m contains atoms belonging to a structural unit, such as a functional group or monomer in a polymer, depending on the system. We pool the atomic representations for each SmS_m:

hˉSm=∑i∈Smhi.(2)\bar{\mathbf{h}}_{S_m} = \sum_{i \in S_m} \mathbf{h}_i. \tag*{(2)}

Each resulting coarse-grained representation vector hˉSm\bar{\mathbf{h}}_{S_m} represents the geometric information of the atoms within its structural unit in an average sense. Following the NLP literature, we refer to the coarse-grained representations as structural “tokens.”

How to partition the atoms is the user’s choice, but many systems have a natural structure. For example, here we study proteins and pool the representations for the atoms in each amino acid. In this case, the number of nodes in the graph is reduced by an order of magnitude. Coarse-graining reduces the computational cost and memory, and it facilitates both learning long-range relationships and interpretation of the results (e.g., attention maps).

Feature combination

To use the coarse-grained tokens in a learning task, we must combine their information in a useful fashion. How best to do so depends on the structural properties of interest, but we can generally lump approaches into two categories:

  • Pooling: We sum (or average) the coarse-grained tokens to obtain a single, global representation of the molecular system hˉ∈R(1+3)d\bar{\mathbf{h}} \in\mathbb{R}^{(1+3)d}. Note that this approach is equivalent to pooling the atomic representation vectors hi\mathbf{h}_i.

  • Token mixing: This approach applies a learnable mixing operation to the coarse-grained tokens to capture their interactions and dependencies[56]. Token mixers allow for greater expressivity than direct pooling and can capture complex interactions between structural units.

In this work, we employ two basic token mixers: (1) a standard transformer architecture[36], which we refer to as “SubFormer,”[57] and (2) an MLP-mixer architecture[58], which we refer to as “SubMixer.” Because the GNNs here employ a message passing architecture (Appendix A), the resulting features typically do not encode global geometric information. We show that this issue can be addressed by combining SubFormer and SubMixer with a special token that encodes global information[59] such as pairwise distances. Alternatively, they can be combined with geometric vector perceptrons (GVPs), equivariant GNNs introduced for biomolecular modeling[60] to learn expressive positional encodings (see Algorithms 2 and 3 in Appendices B and C).

Output layers

Ultimately, we combine the features from the different graph nodes and any graph-wide information (e.g., the CLS token of the transformer) and use an MLP to output quantities specific to a downstream task. Here, we consider downstream tasks that require only scalar quantities, so we input only the scalar features to the MLP, but generally scalar (invariant) and vector (equivariant) quantities can be input and output. We summarize the overall scheme in Algorithm 3.

Downstream tasks

To assess whether the geom2vec representations are useful for learning protein dynamics, we apply them to learning slowly decorrelating modes with VAMP[8] and identifying metastable states with the state predictive information bottleneck (SPIB) framework[45]. As described previously, we apply the pretrained GNN to the coordinates from molecular dynamics trajectories and then use the resulting features as inputs to the desired task without further fine-tuning the GNN parameters. In this section, we briefly describe the two downstream tasks mathematically.

VAMPnets

Let Xt\mathbf{X}_t be a Markov process and define the correlation functions

C00=EX0∼μ[χ0(h~(X0))χ0T(h~(X0))](3)C_{00} = \mathbb{E}_{\mathbf{X}_0 \sim\mu}\left[\chi_0(\tilde{\mathbf{h}}(\mathbf{X}_0))\chi_0^{\mathsf{T}}(\tilde{\mathbf{h}}(\mathbf{X}_0))\right] \tag*{(3)}
C0τ=EX0∼μ[χ0(h~(X0))χτT(h~(Xτ))](4)C_{0\tau} = \mathbb{E}_{\mathbf{X}_0 \sim\mu}\left[\chi_0(\tilde{\mathbf{h}}(\mathbf{X}_0))\chi_\tau^{\mathsf{T}}(\tilde{\mathbf{h}}(\mathbf{X}_\tau))\right] \tag*{(4)}
Cττ=EX0∼μ[χτ(h~(Xτ))χτT(h~(Xτ))],(5)C_{\tau\tau} = \mathbb{E}_{\mathbf{X}_0 \sim\mu}\left[\chi_\tau(\tilde{\mathbf{h}}(\mathbf{X}_\tau))\chi_\tau^{\mathsf{T}}(\tilde{\mathbf{h}}(\mathbf{X}_\tau))\right], \tag*{(5)}

where χ0\chi_0 and χτ\chi_\tau are vectors of functions and the expectation is over trajectories initialized from an arbitrary distribution μ\mu. In our case, Xt\mathbf{X}_t represents the molecular coordinates at time tt and h~(X)\tilde{\mathbf{h}}(\mathbf{X}) are the molecular features from a pretrained GNN. The variational approach for Markov processes (VAMP)[8, 9, 10, 11, 12] states that χ0\chi_0 and χτ\chi_\tau represent the slowest decorrelating modes (or collective variables; CVs) of the system when maximizing the VAMP-2 score

VAMP-2⁡=∥C00−1/2C0τCττ−1/2∥F2,(6)\operatorname{VAMP-2} = \left\|C_{00}^{-1/2}C_{0\tau}C_{\tau\tau}^{-1/2}\right\|_{\mathrm{F}}^2, \tag*{(6)}

where the subscript F denotes the Frobenius norm.

Operationally, the components of χ0\chi_0 and χτ\chi_\tau are learned from data by parameterized functions (e.g., neural networks in VAMPnets[8, 10], the output of geom2vec in our case) and maximizing (6). VAMPnets require one to specify the output dimension dod_o a priori. For the benchmark systems that we consider, we choose dod_o based on previous results in the literature.

As discussed below (Section IV D) we split each dataset into training and validation sets, evaluating the validation score every 10 training steps. Each training step or validation step, we randomly draw a batch of trajectory-frame pairs spaced by τ\tau to compute (6). We found that a large batch size (at least 1000 and usually 5000 for the examples here) was required to achieve a high validation score. With smaller batch sizes, we encountered numerical instabilities when inverting the correlation matrices in the VAMP-2 loss function (6). A large batch size is also needed to minimize variance in the late phase of training because neural network outputs exhibit large changes between metastable states, where fewer trajectory-frame pairs contribute. To prevent overfitting, we apply an early stopping criterion[12]: we stop training when the training VAMP score does not increase for 500 batches or the validation VAMP score does not increase for 10 batches. Further training details are given in Table S3.

HyperparameterChignolinVillinTrp-cage
Batch size50005000/10005000/1000
Learning rate0.00020.00020.0002
OptimizerAdamAtan281AdamAtan2AdamAtan2
Maximum epochs202020
Training patience5500500
Validation patience21010
Validation interval55050
MLP hidden dimension256128/64128/64
MLP activation functionSiLUSiLUSiLU
Trajectory stride124
Training lag time (ns)42010
Embedding dimension (dd)256128/64128/64
Dropout0.20.20.2
Batch normalizationFalseFalseFalse
Output dimension (dod_o)234
ArchitecturesMLP/SubFormer/SubMixerMLP/SubFormer/SubMixerMLP/SubFormer/SubMixer
GVPNoYesNo
Expansion factor222
Global tokenFalseTrueFalse

Table S3. Hyperparameters for VAMPnets. For GVP variants tested on villin, the architecture is consistent with the ones used in SPIB tasks.

State Predictive Information Bottleneck (SPIB)

In the information bottleneck (IB) framework, an encoder-decoder setup is used to learn a low-dimensional (latent) representation z\mathbf{z} that minimizes the information from a high-dimensional input x\mathbf{x} while maximizing the information about a target y\mathbf{y}. The associated loss function is

LIB=I(z,y)−βI(x,z),(7)\mathcal{L}_{\mathrm{IB}} = I(\mathbf{z},\mathbf{y}) - \beta I(\mathbf{x},\mathbf{z}), \tag*{(7)}

where II refers to the mutual information between two random variables:

I(A,B)=∫p(A,B)ln⁡p(A,B)p(A)p(B) dA dB,(8)I(A,B) = \int p(A,B)\ln\frac{p(A,B)}{p(A)p(B)}\,dA\,dB, \tag*{(8)}

and the parameter β\beta controls the tradeoff between prediction accuracy and the complexity of the latent representation.

In the state predictive information bottleneck (SPIB) extension of IB[45], the inputs are molecular features at time tt, h~(Xt)\tilde{h}(X_t), and the targets are state labels sts_t that indicate the state of the system at time tt. The latent representation and state labels are learned simultaneously by predicting the state labels at time t+τt+\tau given the molecular features at time tt.

To learn the latent representation z\mathbf{z}, SPIB maximizes the loss function

LSPIB=EX0∼μ,z∼pθ(z∣h~(X0))[ln⁡qθ(sτ∣z)−βln⁡pθ(z∣h~(X0))rθ(z)].(9)\mathcal{L}_{\mathrm{SPIB}} = \mathbb{E}_{X_0\sim\mu,z\sim p_\theta(z\mid\tilde{h}(X_0))}\left[\ln q_\theta(s_\tau\mid z)-\beta\ln\frac{p_\theta(z\mid\tilde{h}(X_0))}{r_\theta(z)}\right]. \tag*{(9)}

The encoder generates the latent representation z\mathbf{z} from the input with probability pθ(z∣h~(X))=N(z;μθ(h~(X)),Σθ(h~(X)))p_\theta(z\mid\tilde{h}(X))=\mathcal{N}(z;\mu_\theta(\tilde{h}(X)),\Sigma_\theta(\tilde{h}(X))), which is a multivariate normal distribution with learned mean μθ(h~(X))\mu_\theta(\tilde{h}(X)) and learned covariance Σθ(h~(X))\Sigma_\theta(\tilde{h}(X)). The decoder takes the latent representation z\mathbf{z} and returns the probability qθ(s∣z)q_\theta(s\mid z) of each state label ss; qθ(s∣z)q_\theta(s\mid z) is represented by a neural network with output dimension equal to the number of possible state labels. The quantity rθ(z)r_\theta(z) is a prior. The state labels are updated during training as follows:

sτ=argmax⁡sqθ(s∣μθ(h~(Xτ))).(10)s_\tau= \operatorname*{argmax}_{s}q_\theta(s\mid\mu_\theta(\tilde{h}(X_\tau))). \tag*{(10)}

It is important to note that (10) allows the number of distinct states that are populated to fluctuate (unpopulated states are ignored).

We follow Wang and Tiwary[45] and use a variational mixture of posteriors for the prior61:

rθ(z)=∑iωipθ(z∣ui)∑iωi,(11)r_\theta(z)=\frac{\sum_i\omega_i p_\theta(z\mid u_i)}{\sum_i\omega_i}, \tag*{(11)}

where ω\omega and uu are learned parameters. In this work, we prepared the initial state labels by performing kk-means clustering on the CVs learned from VAMPnets based on distances between Cα_\alpha atoms with k=100k=100 clusters. Training reduces the number of distinct states that are populated to the estimated number of metastable states. Further training details are given in Table S2.

HyperparameterShared
Batch size1000
Learning rate0.0002
OptimizerAdamAtan2
Training patience5
MLP hidden dimension64
MLP activation functionSiLU
Embedding dimension (dd)64
Dropout0.2
Batch normalizationFalse
Label refinement frequency5
ArchitecturesSubFormer/SubMixer
GVPYes
# of GVP layers3
# of transformer/mixer layers3
Expansion factor2
Global tokenTrue
Villin
Trajectory stride2
Training lag time (ns)20
Trp-cage
Trajectory stride4
Training lag time (ns)10

Table S2. Hyperparameters for SPIB

Systems studied

We examine the performance of geom2vec for analyzing data from long molecular dynamics simulations of three well-characterized fast-folding proteins (chignolin, trp-cage, and villin)62. The data for each system is a single, unbiased simulation, which we assume approximately samples the equilibrium distribution. In this section, we introduce each system and briefly describe its structure and dynamics.

Chignolin

Chignolin is a 10-residue fast-folding protein with sequence YYDPETGTWY. The folded state consists of three β\beta-hairpin structures that are distinguished by hydrogen bonding between the threonine side chains and their dihedral angles, which interconvert on the nanosecond timescale63. The trajectory that we analyze is 106 μ\mus long at 340 K and saved every 0.2 ns62. For VAMPnet fitting, we choose do=3d_o=3.

Trp-cage

Trp-cage is a 20-residue fast-folding protein64; here we study the K8A mutant with sequence DAYAQWLADG-GPSSGRPPPS. Its secondary structure consists of an α\alpha-helix (residues 2–9), a short 3₁₀-helix (residues 11–14), and a polyproline II helix (residues 17–19); the protein takes its name from Trp6, which is in the core of the folded state. The trajectory is 208 μ\mus long at 290 K and saved every 0.2 ns62. Previous studies of this trajectory generally identified the folded and unfolded states, with varying numbers of intermediates and misfolded states[46] [65]. For VAMPnet fitting, we choose do=4d_o=4.

Villin

The 35-residue villin headpiece subdomain (HP35)66 is a fast-folding protein with sequence LSDEDFKA-VFGMTRSAFANLPLWKQQNLKKEKGLF where L refers to the unnatural amino acid norleucine. The K65nL/N68H/K70nL mutant was engineered to fold more rapidly67. The secondary structure of villin consists of three α\alpha-helices at residues 3–10, 14–19, and 22–32. Villin has a hydrophobic core centered on residues Phe6, Phe10, and Phe17. The trajectory that we study is 125 μ\mus long at 360 K and saved every 0.2 ns62. Previous studies typically identified three states: a folded state, an unfolded state, and a misfolded state[12]. Wang et al.68 proposed two primary folding pathways, where either the C-terminus or the N-terminus folds first, ultimately leading to the native state. Additionally, a cooperative hydrophobic interaction may facilitate a third folding pathway. For VAMPnet fitting, we choose do=3d_o = 3.

Training-validation split

Although previous studies used a random split[8, 33, 50], we observe that, due to the strong correlation between successive structures sampled by molecular dynamics simulations, a random split allows networks to achieve high validation scores even when they have memorized the training data rather than learned useful abstractions from it; the models then perform poorly on independent data. Consequently, for all our numerical experiments, we split the data into training and validation sets by time. That is, we select the first 50% of the long trajectory for training and the remainder for validation. If we had access to multiple independent trajectories, randomly choosing trajectories for training and validation would also be appropriate. Some studies split the trajectory into equal segments and draw random segments for training and validation[46] [65] (kk-fold cross-validation). When there are only two segments, this approach is identical to ours. When every structure is its own segment, one recovers the random split. Intermediate numbers of segments result in intermediate amounts of correlation between the training and validation sets. In cases where we vary the amount of training data, we first select the first 50% of the trajectory and then divide only this half of the trajectory into segments that we draw randomly for training; the second half of trajectory is used as holdout validation set. This approach is fundamentally different from cross-validation and minimizes the correlation between the training and validation datasets.

Results

VAMPNets

For each system and token mixer architecture pair that we consider, we independently train three VAMPnets using different random number generator seeds (and the training-validation split described in Section IV D). We report the training and validation VAMP-2 scores for the different token mixer architectures for each of the three systems in Figure S1 and Figure S2. For chignolin, GNNs with pooling (summing), SubMixer, and SubFormer reach approximately the same maximum validation score. GNNs with SubMixer and SubFormer require fewer epochs to reach the convergence criteria, but they require more computational time per epoch, as we discuss in Section VI. For both villin and trp-cage, the token mixers generally outperform pooling.

Figure S1. Training curves for VAMPnets. Each curve represents the mean of three training runs. Shaded regions indicate standard error over three runs.

Validation curves for VAMPnets

Figure S2. Validation curves for VAMPnets. Each curve represents the mean of three training runs. Shaded regions indicate standard error over three runs.

Figure 2 displays the VAMP-2 scores for VAMPNets trained with different training dataset sizes, varying from 5% to 50% of the available data. For chignolin, the GNNs clearly outperform a multilayer perceptron (MLP) that takes distances between pairs of CαC_{\alpha} atoms as inputs; there is not a significant difference between pooling and the mixers considered. For trp-cage and villin, the GNN with pooling consistently achieves the lowest scores. The distance-based MLP and GNN with SubMixer perform comparably, presumably because the distances between pairs of CαC_{\alpha} atoms are sufficient to describe the folding (in contrast to chignolin, as we discuss below). For villin, we combined SubMixer and SubFormer with GVP and augmented them with a global token; this enables the GNNs to outperform the distance-based MLP. The improvement is particularly striking for SubFormer. The models with GVP are more expressive because they use the equivariant features at the token mixing stage and directly mix them with global features after message-passing. We note that the VAMP scores that we achieve are lower than published ones owing to our choice of the training-validation split (Section IV D) and output dimension dod_o. While the amount of data does not significantly impact the performance for chignolin, the trp-cage and villin results suggest that additional data would permit achieving higher scores.

Figure 2. VAMPnets with various geom2vec architectures. The amount of training data was varied by dividing the training data into 20 trajectory segments of equal length and then randomly selecting the indicated fraction for training. The validation set is held fixed as the second half of each trajectory. Error bars show standard errors over three independent runs.

To visualize the results, we build histograms of learned CV-value pairs, which we convert to potentials of mean force (PMFs) by taking the negative logarithm (Figure 3, Figure S3, Figure S4, and Figure S5). We also show average values of CVs as functions of physically-motivated coordinates (Figure 4, Figure S6, and Figure S7).

Figure 3. Potential of mean force (PMF) of chignolin as a function of the first two CVs learned by a VAMPnet trained with SubMixer. Contours are drawn every 1 kcal/mol. See Figure S3, Figure S4, and Figure S5 for corresponding plots for other architectures and proteins.

Figure 4. Chignolin VAMPnet (with SubMixer) CVs as a function of two physical coordinates: the fraction of native contacts and the χ1\chi_1 side chain dihedral angle of Thr6 (left) or Thr8 (right). Qˉ\bar{Q} is the fraction of native contacts smoothed with a 1-ns moving window centered on each time point. We define native contacts as two residues that are three or more positions apart in sequence and have at least one distance between non-hydrogen atoms that is less than 4.5 Å in the crystal structure (5AWL69). See Figure S6 and Figure S7 for analogous plots for trp-cage and villin.

Figure S3. PMFs as a function of VAMP CVs for chignolin. From top to bottom, VAMPnets were trained with no token mixer (Sum), SubMixer, or SubFormer. Each column shows the result from a single training run. Contours are drawn every 1 kcal/mol.

Figure S4. PMFs as a function of VAMP CVs for trp-cage. From top to bottom, VAMPnets were trained with no token mixer (Sum), SubMixer, or SubFormer. Each column shows the result from a single training run. Contours are drawn every 1 kcal/mol.

Figure S5. PMFs as a function of VAMP CVs for villin. From top to bottom, VAMPnets were trained with no token mixer (Sum), SubMixer, SubMixer-GVP, SubFormer, or SubFormer-GVP. Each column shows the result from a single training run. Contours are drawn every 1 kcal/mol.

Figure S6. Trp-cage VAMPnet (with SubMixer) CVs as a function of two physical coordinates: Cα RMSD of helix 1 (residues 2–9) and Cα RMSD of helix 2 (residues 11–14). The Cα RMSDs were computed with respect to the PDB structure 2JOF64.

Figure S7. Villin VAMPnet (with SubFormer) CVs as a function of two physical coordinates: Cα RMSD of helices 1 and 2 (residues 3–10 and 14–19), and Cα RMSD of helices 2 and 3 (residues 14–19 and 22–32). The Cα RMSDs were computed with respect to the PDB structure 2F4K67.

The advantages of the GNN architecture are well illustrated by the results for chignolin. A previous machine-learning study of chignolin identified two slow CVs, one for the folding-unfolding transition and one distinguishing competing folded states63. Our VAMPNets appear to recover these two CVs (Figure 3 and Figure S3; results shown are with SubMixer), distinguishing folded and unfolded states with CV 1 and four folded states with CV 2. To understand the physical differences between the folded states, we plot CVs 1 and 2 as functions of the fraction of native contacts and the χ1\chi_1 side chain dihedral of Thr6 or Thr8 (Figure 4). The fraction of native contacts clearly correlates with CV 1, consistent with earlier studies. CV 2 distinguishes the folded states by the configurations of Thr6 and Thr8 side chains, which can each occupy two rotamers, yielding four possible folded states. These side chain dynamics could not be detected by VAMPnets that take backbone internal coordinates as inputs, as is common, or even GNNs limited to backbone atoms (Bonati, Piccini, and Parrinello63 included distances to side chain atoms and then manually curated the inputs). This makes clear the usefulness of the pre-training approach that we take here, which enables treating all the atoms with an architecture that supports both scalar and vector features.

SPIB

Trp-cage

Figure 5 shows a Markov state model based on the states learned for trp-cage with a lag time of 20 ns. There are 13 metastable states. State S11S_{11} represents the folded ensemble with a well-defined structure. In states S1S_{1} and S2S_{2} the α\alpha-helix is folded, and the 3103_{10}-helix and polyproline II helix are unfolded. In contrast, in state S8S_{8} the 3103_{10}-helix is folded, and the α\alpha-helix and polyproline II helix are unfolded.

SPIB for trp-cage: PMF and learned Markov State Model

Figure 5. SPIB for trp-cage. All results are obtained from a GNN with SubFormer-GVP token mixer. (top left) PMF as a function of the first two information bottleneck coordinates (IBs). Contours are drawn every 1 kcal/mol. (top right) Same contours colored by SPIB assigned labels. (bottom) Learned Markov State Model. The highlighted structures are chosen randomly from the trajectory. The N-terminus is violet and the C-terminus is red.

helix is folded, and the α\alpha-helix and polyproline II helix are unfolded. State S5S_{5} represents a fully unfolded state, which acts as a hub that connects all intermediate states and the folded state, S11S_{11}. States S3S_{3}, S4S_{4}, S6S_{6}, S7S_{7}, S9S_{9}, S10S_{10}, and S12S_{12} are also largely unfolded and differ with regard to the specific conformations of the helices.

We show attention maps of the SubFormer blocks averaged over all structures in Figure 6 and over the structures in each state in Figure S8, Figure S9, and Figure S10. The attention maps are averaged over all heads of the transformer, yielding (M+1)×(M+1)(M + 1) \times(M + 1) matrices, where for trp-cage M=20M = 20 represents the number of sequence positions (tokens), and the additional row and column correspond to a global token that encodes pairwise distances between the CαC_{\alpha} atoms[57, 59]. The learned attention maps can be interpreted in terms of the structure. The residues that are most consistently activated across states and layers are Trp6 (W6) and Asp9–Ser14 (D9–S14), which roughly correspond to the 3103_{10}-helix. Tyr3 (Y3), Gln5 (Q5), Gly15 (G15), and Arg16 (R16) are activated in selected states. The attention thus appears to track the packing of residues around Trp6. Across all three layers, the global token remains highly active, consistent with the fact that the metastable states of trp-cage are reasonably well-characterized by the distances between the CαC_{\alpha} atoms[14] [65]. The fact that tokens other than the global token participate in the attention mechanism again underscores the ability of GNNs to go beyond distances between CαC_{\alpha} atoms.

SPIB SubFormer-GVP attention

Figure 6. SPIB SubFormer-GVP attention. (left) A typical fully folded trp-cage structure (classified as S11S_{11}) with the central tryptophan residue (Trp6/W6) highlighted. (right three plots) Log-scaled averaged attention maps from three layers of the SubFormer block in the SubFormer-GVP architecture. The sequence is indicated by one-letter amino acid codes, and ∗\ast represents a global token that encodes pairwise distances between the CαC_{\alpha} atoms. Tokens from query and key projections are colored according to the row-wise and column-wise sum of layer-wise attention weights. Results shown are for all structures in the trajectory. Results for individual states are in Figure S8, Figure S9, and Figure S10.

Figure S8. Log-scaled attention weight heatmaps for trp-cage SPIB states 0 to 4 from three layers of SubFormer-GVP. Each subplot displays attention weights with color-coded tick labels based on normalized sums. Colorbars indicate log-scaled attention values.

Figure S9. Log-scaled attention weight heatmaps for trp-cage SPIB states 5 to 9 from three layers of SubFormer-GVP. Each subplot displays attention weights with color-coded tick labels based on normalized sums. Colorbars indicate log-scaled attention values.

Figure S10. Log-scaled attention weight heatmaps for trp-cage SPIB states 10 to 12 from three layers of SubFormer-GVP. Each subplot displays attention weights with color-coded tick labels based on normalized sums. Colorbars indicate log-scaled attention values.

Villin

Figure 7 shows a Markov state model based on the states learned for villin HP35 with a lag time of 10 ns. There are 11 metastable states. State S9S_{9} represents the fully folded structure, in which all three α\alpha-helices are folded and packed compactly. States S1S_1 and S3S_3 correspond to fully unfolded states. In states S0S_0 and S2S_2 helix 1 is folded, suggesting a pathway in which folding initiates at the N-terminus, while in states S5S_5 and S6S_6 helix 3 is folded, suggesting a pathway in which folding initiates at the C-terminus. Both of these pathways are discussed in the literature (see Ref. 68 and references therein).

SPIB for villin: PMF coordinates, state labels, and learned Markov State Model

Figure 7. SPIB for villin. All results are obtained from a GNN with SubFormer-GVP token mixer. (top left) PMF as a function of the first two information bottleneck coordinates (IBs). Contours are drawn every 11 kcal/mol. (top right) Same contours colored by SPIB assigned labels. (bottom) Learned Markov State Model. The highlighted structures are chosen randomly from the trajectory. The N-terminus is violet and the C-terminus is red.

The attention maps (Figure S11, S12, and S13) exhibit patterns that correspond to features of the folded structure. Notably, the attention consistently focuses on the tokens representing Val9 (V9), Gly11 (G11), Met12 (M12), Arg14 (R14), Pro21 (P21), and Trp23 (W23). These residues correspond roughly to the turns between helices. In the attention maps for states 0, 2, 5, and 6, the tokens corresponding to helix 3 feature prominently; the tokens corresponding to helix 1 are also activated in state 0. The attention maps thus suggest that the network tracks the folding and packing of the helices.

Figure S11. Log-scaled attention weight heatmaps for villin SPIB states 0 to 3 from three layers of SubFormer-GVP. Each subplot displays attention weights with color-coded tick labels based on normalized sums. Colorbars indicate log-scaled attention values.

It is interesting to compare our attention maps with those of Ghorbani et al. 33 and Huang et al. 35. Those studies leverage a graph attention network (GAT)70 to enhance expressive power and interpretability of their models. GAT computes representations of each node by attending to its one-hop neighboring nodes, which captures local dependencies but fails to model long-range interactions. In contrast, GNN-Transformer hybrids such as SubFormer[57] allocate short-range interactions to the MP-GNN and use the self-attention mechanism for long-range interactions. This approach not only supports multimodal features (e.g., a global token) but also enables distant nodes to attend to each other, regardless of graph distance. This difference is evident in the attention map patterns: GAT attention maps[33, 35] show predominantly diagonal patterns, reflecting a focus on local neighborhoods, while SubFormer-GVP attention maps reveal vertical, blockwise, and global patterns, reflecting instead a focus on specific amino acids and their long-range interactions.

Computational requirements

Equivariant geometric GNNs use both invariant and equivariant features to capture the three-dimensional structure of molecules. For typical numbers of features, the memory and time requirements are expensive even for small graphs. To illustrate, we show the memory and time requirements for inference using a TorchMD-ET GNN with a small batch size with varying numbers of hidden channels (numbers of features) for trp-cage (144 non-hydrogen atoms) and villin (272 non-hydrogen atoms) in Figure S14. Even this already requires tens of gigabytes of memory and several seconds; a more complex architecture like ViSNet is expected to increase the memory and time requirements by roughly 50%. The memory and time scale linearly with both the number of atoms in the graph and the batch size. We use a batch size of 5000 for VAMP (except for GVP variants, for which we use 1000) and 1000 for SPIB, making training a GNN on the fly prohibitive, as we discuss in further detail below.

Comparison of time and GPU memory usage of TorchMD-ET in training and inference modes

Figure S14. Comparison of time and GPU memory usage of TorchMD-ET in training and inference modes on the trp-cage (left) and villin (right) subsets, each consisting of 2000 frames with 100 frames per batch. The measurements are done without specific objective functions for usage benchmarking purposes. The model’s performance is evaluated across varying numbers of hidden channels (64, 128, 256, 384) and layers (1–6). The heatmaps on the left show GPU memory allocated in gigabytes (GB), while the heatmaps on the right depict time usage in seconds. The time represents one forward pass for inference mode and one forward plus one backward pass for training mode. Measurements were performed on an NVIDIA A100 GPU with 40G of memory. Missing values are due to out-of-memory errors for certain configurations.

Geom2vec decouples training the GNNs and the networks for the downstream tasks. This allows us to use a small batch size for pretraining the GNNs (which need be done only once), and the networks for the downstream tasks take as inputs the tokens, which are fewer in number than the number of graph nodes. For example, here, the number of graph nodes is the number of non-hydrogen atoms, while the number of tokens is the number of amino acids, which is an order of magnitude smaller.

The computational costs for training VAMPnets with different token mixers are shown in Figure 8. The simplest GNN using pooling is not much more computationally costly than an MLP that takes distances between CαC_\alpha atoms as inputs. The GNNs with token mixers are about an order of magnitude more computationally costly but still manageable (hundreds of seconds) even without advanced acceleration techniques such as flash-attention or compilation. We expect the memory and computational requirements to scale with token number quadratically for SubFormer and subquadratically for SubMixer (depending on the expansion dimension in the token-mixing blocks); these requirements should scale linearly with respect to embedding dimension and network depth.

Figure 8. Computational time for training a single VAMPNet. We employ early stopping and stop training when the training VAMP score does not increase for 1000 batches or the validation VAMP score does not increase for 10 batches. Times reported are averages over three training runs, with validation performed at each step using the second half of each trajectory, as depicted in Figure S2. All times are for training on a single NVIDIA A40 GPU.

To estimate the memory usage and training time for a VAMPnet based on a GNN without pretraining, we consider a TorchMD-ET model with specific settings: a batch size of 1000, a hidden dimension of 64, and 6 layers; we assume 50 epochs are required to converge. For trp-cage, a batch size of 100 required 10.64 GB of memory. Since memory usage and training time should scale linearly with batch size, we estimate that increasing the batch size to 1000 would raise the memory usage to approximately 10.64 GB×10=106.4 GB10.64\ \mathrm{GB} \times10 = 106.4\ \mathrm{GB}. Similarly, we estimate the training time at this larger batch size to be around 1.8 hours (excluding validation). The corresponding numbers for villin, which is about twice the size, are proportionally larger (219.0 GB and 4.0 hours for training). As noted above, employing a more complex architecture like ViSNet would further increase both time and memory requirements by roughly 50%. In the present study, we employ a batch size of 5000 for VAMP (except for GVP variants, as noted above); the batch size for SPIB is 1000, but it generally requires more iterations to converge. While rough, these estimates show that without pretraining equivariant GNNs for analyzing molecular dynamics are beyond the resources available to most researchers. By contrast, with pretraining, they are well within reach (Figure 8).

Conclusions

In this paper, we use pretrained GNNs to convert molecular conformations into rich vector representations that can then be used for diverse downstream tasks. Decoupling the training of the GNNs and the networks for the downstream tasks dramatically decreases the memory and computational time requirements. Here, we focused on downstream tasks concerned with analyzing dynamics in molecular simulations, specifically VAMP and SPIB. For these tasks, we were able to use equivariant GNNs that take all non-hydrogen atoms of small proteins as inputs for the first time. The results for folding and unfolding of small proteins show that the GNNs use information beyond distances between Cα_\alpha atoms, which are commonly used as input features.

For pretraining, we used a simple denoising task with a dataset of structures of diverse molecules. Given that this dataset was not specific to proteins and/or VAMP and SPIB, we expect the present models to generalize to other classes of molecules and tasks, but quantitative tests on a wider variety of systems and tasks remain to be done. It would be interesting to investigate whether more complex pretraining strategies[47] [71] together with datasets specific to the class of molecules of interest (e.g., those specific to proteins[48] [72, 73]) can improve performance. We also explored a variety of token mixers and observed that more complex architectures were able to yield better results, which suggests there is scope for further engineering in this regard; these should be informed by ablation studies.

In our tests, we took care to split the dataset in a way that minimized the correlation between the training and validation datasets, and we believe that this should be standard practice. Because the data consisted of long, unbiased trajectories62, there were relatively few events of interest (here, folding and unfolding). Adapting our approach to methods that take short trajectories[51] [74], which allow for greater control of sampling74,75, is an important area of study for the future.

Acknowledgments

We thank D. E. Shaw Research for making the molecular dynamics trajectories available to us. This work was supported by National Institutes of Health award R35 GM136381 and National Science Foundation award DMS-2054306. S.C.G. acknowledges support from the National Science Foundation Graduate Research Fellowship under Grant No. 2140001. Z.P. was supported with funding from the University of Chicago Data Science Institute’s AI+Science Research Initiative. This work was completed with computational resources administered by the University of Chicago Research Computing Center, including Beagle-3, a shared GPU cluster for biomolecular sciences supported by the NIH under the High-End Instrumentation (HEI) grant program award 1S10OD028655-0.

Data availability

Data sharing is not applicable to this article as no new data were created or analyzed in this study. Code for our implementation and examples are available at https://github.com/dinner-group/geom2vec.

References

GNN architectures

Equivariant GNNs

Assuming R=(r1,r2,…,rN)∈R3N\mathbf{R} = (\mathbf{r}_1,\mathbf{r}_2,\ldots,\mathbf{r}_N) \in\mathbb{R}^{3N} defines the molecular conformation, where ri∈R3\mathbf{r}_i \in\mathbb{R}^3 represents the three-dimensional (3D) coordinates of the ii-th atom, two of the most important symmetries for a GNN to obey are translation and rigid-body rotation invariances. These invariances require that the output of the GNN, f(R)f(\mathbf{R}), remains unchanged when the molecule is translated by a vector t∈R3\mathbf{t} \in\mathbb{R}^3, i.e., f(R)=f(R+t)f(\mathbf{R}) = f(\mathbf{R} + \mathbf{t}), or when the molecule undergoes a rigid body rotation represented by a rotation matrix P∈SO(3)P \in SO(3), i.e., f(R)=f(PR)f(\mathbf{R}) = f(P\mathbf{R}). Translation invariance can be easily realized by using the relative displacements or internal coordinates of atoms. On the other hand, rotational invariance and equivariance can be realized in two ways:

  1. representing the relative atom positions by internal coordinates (e.g., bond distances, bond angles, and dihedral angles) as in conventional classical force fields (“scalar-based network”)[53, 54] [76, 77];

  2. encoding the relative displacement vectors with spherical harmonics, which are then propagated with tensor products (“group-equivariant network”)[20, 21, 22].

The two approaches are theoretically and practically equivalent[53, 54] [78], but group-equivariant networks need to store the features for each irreducible representation and are more computationally costly owing to the tensor products. Therefore, we chose to employ scalar-based GNNs.

TorchMD-ET architecture

The TorchMD-ET architecture[54] is structured around three components that process and encode molecular geometric information effectively.

  1. Embedding layer: encodes atomic types and interatomic distances using exponential radial basis functions (eRBFs). Each distance dijd_{ij} within a cutoff rcutr_{\mathrm{cut}} is transformed by

eRBF⁡k(dij)=ϕ(dij)exp⁡(−βk(exp⁡(−dij)−μk)2),(12)\operatorname{eRBF}_k(d_{ij}) = \phi(d_{ij}) \exp\left(-\beta_k\left(\exp(-d_{ij})-\mu_k\right)^2\right), \tag*{(12)}

where ϕ(dij)\phi(d_{ij}) is a cosine cutoff function ensuring smooth transitions to zero beyond rcutr_{\mathrm{cut}}.

  1. Modified attention mechanism: incorporates edge information through an extended dot-product attention mechanism that integrates interatomic distances with distance kernels DKDK derived from eRBFs (⊙\odot denotes element-wise multiplication):

A=SiLU⁡(∑k=1FQk⊙Kk⊙DKk)⋅ϕ(dij).(13)A = \operatorname{SiLU}\left(\sum_{k=1}^{F} Q_k \odot K_k \odot DK_k\right)\cdot\phi(d_{ij}). \tag*{(13)}

The attention mechanism weights are then used to compute scalar features and filters qαi,sαij∈RFq_{\alpha i},s_{\alpha ij} \in\mathbb{R}^{F} (α=1,2,3\alpha= 1,2,3) for the update layer:

s1ij,s2ij,s3ij=Vj⊙DV(14)s_{1ij},s_{2ij},s_{3ij} = V_j \odot DV \tag*{(14)}
q1i,q2i,q3i=W(∑j∈N(i)∖iAij⋅s3ij)(15)q_{1i},q_{2i},q_{3i} = W\left(\sum_{j\in\mathcal{N}(i)\setminus i} A_{ij}\cdot s_{3ij}\right) \tag*{(15)}

where VjV_j and DVDV are attention value and distance projections analogous to QQ, KK, and DKDK.

  1. Update layer: updates both scalar (Δxi\Delta x_i) and vector (Δvi\Delta v_i) features using qαiq_{\alpha i} and sαijs_{\alpha ij}:

Δxi=q1i+q2i⊙(U1vi⋅U2vi)(16)\Delta x_i = q_{1i}+q_{2i}\odot(U_1v_i\cdot U_2v_i) \tag*{(16)}
Δvi=q3i⋅U3vi+∑j(s1ij⊙vj+s2ij⊙ri−rj∥ri−rj∥).(17)\Delta v_i = q_{3i}\cdot U_3v_i+\sum_j\left(s_{1ij}\odot v_j+s_{2ij}\odot\frac{\mathbf{r}_i-\mathbf{r}_j}{\lVert\mathbf{r}_i-\mathbf{r}_j\rVert}\right). \tag*{(17)}

This combines scalar and directional vector information describing the local atomic environment.

ViSNet

ViSNet[53] builds on TorchMD-ET by incorporating additional scalar features into its architecture. These additional features include information about angles, dihedral angles, and improper dihedral angles. This information is computed in an efficient fashion by first associating with each atom information about its neighbors:

uij=rij∥rij∥,(18)\mathbf{u}_{ij} = \frac{\mathbf{r}_{ij}}{\lVert\mathbf{r}_{ij} \rVert}, \tag*{(18)}
vi=∑j∈N(i)∖iuij,(19)\mathbf{v}_{i} = \sum_{j \in\mathcal{N}(i) \setminus i} \mathbf{u}_{ij}, \tag*{(19)}
wij=vi−(vi⋅uij)uij,(20)\mathbf{w}_{ij} = \mathbf{v}_{i} - (\mathbf{v}_{i} \cdot\mathbf{u}_{ij})\mathbf{u}_{ij}, \tag*{(20)}

where uij\mathbf{u}_{ij} is the unit vector pointing from atom ii to neighboring atom jj, and vi\mathbf{v}_{i} is the sum of all such unit vectors around atom ii. Then the inner product

wij⋅wji=∑m∈N(i)∖i∑n∈N(j)∖jcos⁡(φmijn)(21)\mathbf{w}_{ij} \cdot\mathbf{w}_{ji} = \sum_{m \in\mathcal{N}(i) \setminus i} \sum_{n \in\mathcal{N}(j) \setminus j} \cos(\varphi_{mijn}) \tag*{(21)}

represents the dihedral angle (φmijn)(\varphi_{mijn}) information around inner atoms ii and jj, with neighbors indexed by mm and nn, respectively.

Gated equivariant block

The output layer of the networks consists of gated equivariant blocks79,80, each of which combines the scalar (xi)(x_i) and vector (vi)(v_i) features from graph node ii of the previous layer (Algorithm 1).

Algorithm 1 Gated equivariant block (GEB)

Require: xi∈Rdix_i \in\mathbb{R}^{d_i}, vi∈R3div_i \in\mathbb{R}^{3d_i}

  1. vi1,vi2←∥Wvi∥,Wviv_i^1, v_i^2 \leftarrow\lVert Wv_i \rVert, Wv_i ▷ vi1∈Rdiv_i^1 \in\mathbb{R}^{d_i}, vi2∈R3dov_i^2 \in\mathbb{R}^{3d_o}

  2. xi←Concat⁡(xi,vi1)x_i \leftarrow\operatorname{Concat}(x_i, v_i^1) ▷ xi∈R2dix_i \in\mathbb{R}^{2d_i}

  3. xi←W(SiLU⁡((Wxi+b)))+bx_i \leftarrow W(\operatorname{SiLU}((W x_i + b))) + b ▷ xi∈R2dox_i \in\mathbb{R}^{2d_o}

  4. [xi,gi]←xi[x_i, g_i] \leftarrow x_i ▷ split xix_i into xi,gi∈Rdox_i, g_i \in\mathbb{R}^{d_o}

  5. xi←SiLU⁡(xi)x_i \leftarrow\operatorname{SiLU}(x_i)

  6. vi←gi⊙vi2v_i \leftarrow g_i \odot v_i^2 ▷ scales vector features, maintains equivariance

  7. return xi,vix_i, v_i ▷ xi∈Rdox_i \in\mathbb{R}^{d_o}, vi∈R3dov_i \in\mathbb{R}^{3d_o}

Geometric vector perceptron

Geometric vector perceptron (GVP) was one of the first architectures to incorporate vector features for protein modeling[60]. Here, we use the GVP architecture as an equivariant token mixer to combine scalar and vector features (Algorithm 2).

Algorithm 2 Geometric vector perceptron (GVP)

Require: xi∈Rdix_i \in\mathbb{R}^{d_i}, vi∈R3div_i \in\mathbb{R}^{3d_i}

  1. vh←Whviv_h \leftarrow W_hv_i ▷ project vector features, vi∈R3dhv_i \in\mathbb{R}^{3d_h}

  2. xh←∥vh∥2x_h \leftarrow\lVert v_h \rVert_2 ▷ compute the (row-wise) norm of projected vectors, xh∈Rdhx_h \in\mathbb{R}^{d_h}

  3. xh+di←Concat⁡(xh,xi)x_{h+d_i} \leftarrow\operatorname{Concat}(x_h,x_i) ▷ xh+di∈Rdh+dix_{h+d_i} \in\mathbb{R}^{d_h+d_i}

  4. xm←Wmxh+di+bmx_m \leftarrow W_m x_{h+d_i} + b_m ▷ xm∈Rdox_m \in\mathbb{R}^{d_o}

  5. xi′←σ(xm)x'_i \leftarrow\sigma(x_m) ▷ xi′∈Rdox'_i \in\mathbb{R}^{d_o}

  6. vo←Wovhv_o \leftarrow W_o v_h ▷ project vector features to R3do\mathbb{R}^{3d_o}

  7. vi′←σ(∥vo∥2)⊙vov'_i \leftarrow\sigma(\lVert v_o \rVert_2) \odot v_o ▷ column-wise multiplication with vector gate (on row-wise norm), vi′∈R3dov'_i \in\mathbb{R}^{3d_o}

  8. return xi′,vi′x'_i, v'_i

Pseudo-code of overall architecture

Algorithm 3 geom2vec

Require: Atomic numbers ziz_i, positions ri∈R3\mathbf{r}_i \in\mathbb{R}^3 (i=1,…,Ni = 1,\ldots,N), coarse-grained mapping S={S1,…,SM}\mathcal{S} = \{S_1,\ldots,S_M\}, mixer method {None,SubFormer,SubMixer,SubGVP}\{\mathrm{None},\mathrm{SubFormer},\mathrm{SubMixer},\mathrm{SubGVP}\}, global features G∈Rdg\mathbf{G} \in\mathbb{R}^{d_g}

  1. xiMP,viMP←GNN(zi,ri)x_i^{\mathrm{MP}}, \mathbf{v}_i^{\mathrm{MP}} \leftarrow\mathrm{GNN}(z_i,\mathbf{r}_i) ▷ atomistic featurization, xiMP∈Rdk,viMP∈R3dkx_i^{\mathrm{MP}} \in\mathbb{R}^{d_k}, \mathbf{v}_i^{\mathrm{MP}} \in\mathbb{R}^{3d_k}

  2. xmMP,vmMP←∑i∈SmxiMP,∑i∈SmviMPx_m^{\mathrm{MP}}, \mathbf{v}_m^{\mathrm{MP}} \leftarrow\sum_{i\in S_m} x_i^{\mathrm{MP}}, \sum_{i\in S_m} \mathbf{v}_i^{\mathrm{MP}} ▷ coarse-graining with mapping S\mathcal{S}, xmMP∈Rdk,vmMP∈R3dkx_m^{\mathrm{MP}} \in\mathbb{R}^{d_k}, \mathbf{v}_m^{\mathrm{MP}} \in\mathbb{R}^{3d_k}

  3. if None then

  4. xMP,vMP←∑mxmMP,∑mvmMPx^{\mathrm{MP}}, \mathbf{v}^{\mathrm{MP}} \leftarrow\sum_m x_m^{\mathrm{MP}}, \sum_m \mathbf{v}_m^{\mathrm{MP}} ▷ sum over indices mm for direct pooling, xMP∈Rdk,vMP∈R3dkx^{\mathrm{MP}} \in\mathbb{R}^{d_k}, \mathbf{v}^{\mathrm{MP}} \in\mathbb{R}^{3d_k}

  5. x,v←GEB(xMP,vMP)x,\mathbf{v} \leftarrow\mathrm{GEB}(x^{\mathrm{MP}},\mathbf{v}^{\mathrm{MP}}) ▷ Algorithm 1

  6. else if SubFormer then

  7. xm,vm←GEB(xmMP,vmMP)x_m,\mathbf{v}_m \leftarrow\mathrm{GEB}(x_m^{\mathrm{MP}},\mathbf{v}_m^{\mathrm{MP}}) ▷ apply GEB to input features

  8. if global token then

  9. g←MLP(G)g \leftarrow\mathrm{MLP}(\mathbf{G}) ▷ project global features G\mathbf{G} to Rdk\mathbb{R}^{d_k}

  10. x←TransformerEncoder([xm,g])x \leftarrow\mathrm{TransformerEncoder}([x_m,g]) ▷ pass regular and global tokens through the transformer encoder

  11. else

  12. x←TransformerEncoder(xm)x \leftarrow\mathrm{TransformerEncoder}(x_m) ▷ pass only the regular tokens through the transformer encoder

  13. end if

  14. else if SubMixer then

  15. xm,vm←GEB(xmMP,vmMP)x_m,\mathbf{v}_m \leftarrow\mathrm{GEB}(x_m^{\mathrm{MP}},\mathbf{v}_m^{\mathrm{MP}}) ▷ apply GEB to input features

  16. if global token then

  17. g←MLP(G)g \leftarrow\mathrm{MLP}(\mathbf{G}) ▷ project global features G\mathbf{G} to Rdk\mathbb{R}^{d_k}

  18. x←MLPMixer([xm,g])x \leftarrow\mathrm{MLPMixer}([x_m,g]) ▷ pass regular and global tokens through the MLP-Mixer

  19. else

  20. x←MLPMixer(xm)x \leftarrow\mathrm{MLPMixer}(x_m) ▷ pass only the regular tokens through the MLP-Mixer

  21. end if

  22. else if SubGVP then

  23. xmGVP,vmGVP←GVP(xmMP,vmMP)x_m^{\mathrm{GVP}},\mathbf{v}_m^{\mathrm{GVP}} \leftarrow\mathrm{GVP}(x_m^{\mathrm{MP}},\mathbf{v}_m^{\mathrm{MP}}) ▷ Algorithm 2

  24. if subformer or submixer then

  25. x←subformer or submixer(xmGVP)x \leftarrow\mathrm{subformer\ or\ submixer}(x_m^{\mathrm{GVP}}) ▷ apply subformer or submixer block to GVP features

  26. end if

  27. end if

  28. x←MLP(x)x \leftarrow\mathrm{MLP}(x)

  29. return xx ▷ x∈Rdox \in\mathbb{R}^{d_o}

Training details and hyperparameters

For pre-training, we use the Orbnet Denali dataset from Ref. 44, which consists of 2.3 million conformations of small to medium size molecules. These conformations are obtained with semi-empirical methods and contain nonequilibrium geometries, alternative tautomers, and non-bonded interactions. For pretraining, we randomly take 10,000 configurations as the validation set and the remainder as the training set.

HyperparameterTorchMD-ET and ViSNet
dd64, 128, 256, 384
# of MP layers6
# of attention heads8
Batch size100
Epochs10
# of RBFs64
rcutr_{\mathrm{cut}} (Å)5, 7.5
Learning rate0.0005
OptimizerAdamW (AMSGrad)
Noise level (Å)0.2

Table S1. Hyperparameters for pretrained networks

Supplementary Figures

Figure S12. Log-scaled attention weight heatmaps for villin SPIB states 4 to 7 from three layers of SubFormer-GVP. Each subplot displays attention weights with color-coded tick labels based on normalized sums. Colorbars indicate log-scaled attention values.

Figure S13. Log-scaled attention weight heatmaps for villin SPIB states 8 to 10 from three layers of SubFormer-GVP. Each subplot displays attention weights with color-coded tick labels based on normalized sums. Colorbars indicate log-scaled attention values.

References

  1. [1]B. E. Husic, R. T. McGibbon, M. M. Sultan, and V. S. Pande, “Optimized parameter selection reveals trends in Markov state models for protein folding,” Journal of Chemical Physics 145, 194103 (2016).pubmed.ncbi.nlm.nih.gov/27875868
  2. [2]M. K. Scherer, B. E. Husic, M. Hoffmann, F. Paul, H. Wu, and F. Noé, “Variational selection of features for molecular kinetics,” Journal of Chemical Physics 150, 194108 (2019).DOI
  3. [3]D. Nagel, S. Sartore, and G. Stock, “Selecting features for Markov modeling: A case study on HP35,” Journal of Physical Chemistry Letters 14, 6956–6967 (2023).DOI
  4. [4]R. E. Arbon, Y. Zhu, and A. S. J. S. Mey, “Markov state models: To optimize or not to optimize,” Journal of Chemical Theory and Computation 20, 977–988 (2024).DOI
  5. [5]G. A. Tribello, M. Ceriotti, and M. Parrinello, “Using sketch-map coordinates to analyze and bias molecular dynamics simulations,” Proceedings of the National Academy of Sciences 109, 5196–5201 (2012).DOI
  6. [6]M. A. Rohrdanz, W. Zheng, M. Maggioni, and C. Clementi, “Determination of reaction coordinates via locally scaled diffusion map,” Journal of Chemical Physics 134, 124116 (2011).DOI
  7. [7]L. Boninsegna, G. Gobbo, F. Noé, and C. Clementi, “Investigating molecular kinetics by variationally optimized diffusion maps,” Journal of Chemical Theory and Computation 11, 5947–5960 (2015).DOI
  8. [8]A. Mardt, L. Pasquali, H. Wu, and F. Noé, “VAMPnets for deep learning of molecular kinetics,” Nature Communications 9, 5 (2018).
  9. [9]F. Noé and F. Nüske, “A variational approach to modeling slow processes in stochastic dynamical systems,” Multiscale Modeling & Simulation 11, 635–655 (2013).
  10. [10]W. Chen, H. Sidky, and A. L. Ferguson, “Nonlinear discovery of slow molecular modes using state-free reversible VAMPnets,” Journal of Chemical Physics 150, 214114 (2019).DOI
  11. [11]H. Wu and F. Noé, “Variational approach for learning Markov processes from time series data,” Journal of Nonlinear Science 30, 23–66 (2020).arxiv.org/abs/1707.04659
  12. [12]C. Lorpaiboon, E. H. Thiede, R. J. Webber, J. Weare, and A. R. Dinner, “Integrated variational approach to conformational dynamics: A robust strategy for identifying eigenfunctions of dynamical operators,” Journal of Physical Chemistry B 124, 9354–9364 (2020).DOI
  13. [13]Z. Trstanova, B. Leimkuhler, and T. Lelièvre, “Local and global perspectives on diffusion maps in the analysis of molecular systems,” Proceedings of the Royal Society A 476, 20190036 (2020).DOI
  14. [14]J. Strahan, A. Antoszewski, C. Lorpaiboon, B. P. Vani, J. Weare, and A. R. Dinner, “Long-time-scale predictions from short-trajectory data: A benchmark analysis of the trp-cage miniprotein,” Journal of Chemical Theory and Computation 17, 2948–2963 (2021).DOI
  15. [15]E. H. Thiede, D. Giannakis, A. R. Dinner, and J. Weare, “Galerkin approximation of dynamical quantities using trajectory data,” Journal of Chemical Physics 150, 241111 (2019).DOI
  16. [16]P. G. Bolhuis, C. Dellago, and D. Chandler, “Reaction coordinates of biomolecular isomerization,” Proceedings of the National Academy of Sciences 97, 5877–5882 (2000).pubmed.ncbi.nlm.nih.gov/10801977
  17. [17]A. Ma and A. R. Dinner, “Automatic method for identifying reaction coordinates in complex systems,” Journal of Physical Chemistry B 109, 6769–6779 (2005).
  18. [18]S. C. Guo, R. Shen, B. Roux, and A. R. Dinner, “Dynamics of activation in the voltage-sensing domain of Ciona intestinalis phosphatase Ci-VSP,” Nature Communications 15, 1408 (2024).DOI
  19. [19]N. S. Herringer, S. Dasetty, D. Gandhi, J. Lee, and A. L. Ferguson, “Permutationally invariant networks for enhanced sampling (PINES): Discovery of multimolecular and solvent-inclusive collective variables,” Journal of Chemical Theory and Computation 20, 178–198 (2023).DOI
  20. [20]N. Thomas, T. Smidt, S. Kearnes, L. Yang, L. Li, K. Kohlhoff, and P. Riley, “Tensor field networks: Rotation- and translation-equivariant neural networks for 3D point clouds,” arXiv preprint arXiv:1802.08219 (2018).arxiv.org/abs/1802.08219
  21. [21]B. Anderson, T.-S. Hy, and R. Kondor, “Cormorant: Covariant molecular neural networks,” in Proceedings of the 33rd International Conference on Neural Information Processing Systems (Curran Associates Inc., Red Hook, NY, USA, 2019) pp. 14537–14546.arxiv.org/abs/1906.04015
  22. [22]S. Batzner, A. Musaelian, L. Sun, M. Geiger, J. P. Mailoa, M. Kornbluth, N. Molinari, T. E. Smidt, and B. Kozinsky, “E(3)-equivariant graph neural networks for data-efficient and accurate interatomic potentials,” Nature Communications 13, 2453 (2022).DOI
  23. [23]J. Han, J. Cen, L. Wu, Z. Li, X. Kong, R. Jiao, Z. Yu, T. Xu, F. Wu, Z. Wang, et al., “A survey of geometric graph neural networks: Data structures, models and applications,” arXiv preprint arXiv:2403.00485 (2024).arxiv.org/abs/2403.00485
  24. [24]J. Gilmer, S. S. Schoenholz, P. F. Riley, O. Vinyals, and G. E. Dahl, “Neural message passing for quantum chemistry,” in International Conference on Machine Learning (PMLR, 2017) pp. 1263–1272.arxiv.org/abs/1704.01212
  25. [25]P. W. Battaglia, J. B. Hamrick, V. Bapst, A. Sanchez-Gonzalez, V. Zambaldi, M. Malinowski, A. Tacchetti, D. Raposo, A. Santoro, R. Faulkner, C. Gulcehre, F. Song, A. Ballard, J. Gilmer, G. Dahl, A. Vaswani, K. Allen, C. Nash, V. Langston, C. Dyer, N. Heess, D. Wierstra, P. Kohli, M. Botvinick, O. Vinyals, Y. Li, and R. Pascanu, “Relational inductive biases, deep learning, and graph networks,” arXiv preprint arXiv:1806.01261 (2018).arxiv.org/abs/1806.01261
  26. [26]B. E. Husic, N. E. Charron, D. Lemmon, J. Wang, A. Pérez, M. Majewski, A. Krämer, Y. Chen, S. Olsson, G. de Fabritiis, et al., “Coarse graining molecular dynamics with graph neural networks,” Journal of Chemical Physics 153, 194101 (2020).
  27. [27]A. R. Jamasb, A. Morehead, Z. Zhang, C. K. Joshi, K. Didi, S. V. Mathis, C. Harris, J. Tang, J. Cheng, P. Lio, and T. L. Blundell, “Evaluating representation learning on the protein structure universe,” in The Twelfth International Conference on Learning Representations (2023).
  28. [28]P. Hermosilla and T. Ropinski, “Contrastive representation learning for 3D protein structures,” arXiv preprint arXiv:2205.15675 (2022).arxiv.org/abs/2205.15675
  29. [29]L. Wang, H. Liu, Y. Liu, J. Kurtin, and S. Ji, “Learning hierarchical protein representations via complete 3D graph networks,” arXiv preprint arXiv:2207.12600 (2023).arxiv.org/abs/2207.12600
  30. [30]Z. Zhang, M. Xu, A. Jamasb, V. Chenthamarakshan, A. Lozano, P. Das, and J. Tang, “Protein representation learning by geometric structure pretraining,” arXiv preprint arXiv:2203.06125 (2023).arxiv.org/abs/2203.06125
  31. [31]T. Xie, A. France-Lanord, Y. Wang, Y. Shao-Horn, and J. C. Grossman, “Graph dynamical networks for unsupervised learning of atomic scale dynamics in materials,” Nature Communications 10, 2667 (2019).DOI
  32. [32]S. Soltani, C. W. Sinclair, and J. Rottler, “Exploring glassy dynamics with Markov state models from graph dynamical neural networks,” Physical Review E 106, 025308 (2022).DOI
  33. [33]M. Ghorbani, S. Prasad, J. B. Klauda, and B. R. Brooks, “Graph-VAMPNet, using graph neural networks and a variational approach to Markov processes for dynamical modeling of biomolecules,” Journal of Chemical Physics 156, 184103 (2022).DOI
  34. [34]B. Liu, M. Xue, Y. Qiu, K. A. Konovalov, M. S. O’Connor, and X. Huang, “GraphVAMPnets for uncovering slow collective variables of self-assembly dynamics,” Journal of Chemical Physics 159, 094901 (2023).DOI
  35. [35]Y. Huang, H. Zhang, Z. Lin, Y. Wei, and W. Xi, “RevGraph-VAMP: A protein molecular simulation analysis model combining graph convolutional neural networks and physical constraints,” Methods 229, 163–174 (2024).
  36. [36]A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” in Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS’17 (Curran Associates Inc., Red Hook, NY, USA, 2017) pp. 6000–6010.arxiv.org/abs/1706.03762
  37. [37]A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al., “An image is worth 16x16 words: Transformers for image recognition at scale,” arXiv preprint arXiv:2010.11929 (2020).arxiv.org/abs/2010.11929
  38. [38]T. Mikolov, I. Sutskever, K. Chen, G. Corrado, and J. Dean, “Distributed representations of words and phrases and their compositionality,” in Proceedings of the 26th International Conference on Neural Information Processing Systems - Volume 2, NIPS’13 (Curran Associates Inc., Red Hook, NY, USA, 2013) pp. 3111–3119.arxiv.org/abs/1310.4546
  39. [39]Y. Guo, J. Wu, H. Ma, and J. Huang, “Self-supervised pre-training for protein embeddings using tertiary structures,” in Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 36 (2022) pp. 6801–6809.DOI
  40. [40]S. Zaidi, M.-G. Schimdt, J. Martens, H. Kim, Y. W. Teh, A. Sanchez-Gonzalez, P. Battaglia, R. Pascanu, and J. Godwin, “Pre-training via denoising for molecular property prediction,” arXiv preprint arXiv:2206.00133 (2022).arxiv.org/abs/2206.00133
  41. [41]C. Chen, J. Zhou, F. Wang, X. Liu, and D. Dou, “Structure-aware protein self-supervised learning,” Bioinformatics 39, btad189 (2023).DOI
  42. [42]J. Sohl-Dickstein, E. A. Weiss, N. Maheswaranathan, and S. Ganguli, “Deep unsupervised learning using nonequilibrium thermodynamics,” in Proceedings of the 32nd International Conference on International Conference on Machine Learning - Volume 37, ICML’15 (JMLR.org, Lille, France, 2015) pp. 2256–2265.
  43. [43]J. Ho, A. Jain, and P. Abbeel, “Denoising diffusion probabilistic models,” in Advances in Neural Information Processing Systems, Vol. 33 (Curran Associates, Inc., 2020) pp. 6840–6851.arxiv.org/abs/2006.11239
  44. [44]A. S. Christensen, S. K. Sirumalla, Z. Qiao, M. B. O’Connor, D. G. Smith, F. Ding, P. J. Bygrave, A. Anandkumar, M. Welborn, F. R. Manby, et al., “OrbNet Denali: A machine learning potential for biological and organic chemistry with semi-empirical cost and DFT accuracy,” Journal of Chemical Physics 155, 204103 (2021).DOI
  45. [45]D. Wang and P. Tiwary, “State predictive information bottleneck,” Journal of Chemical Physics 154, 134111 (2021).DOI
  46. [46]D. Wang, Y. Qiu, E. R. Beyerle, X. Huang, and P. Tiwary, “Information bottleneck approach for Markov model construction,” Journal of Chemical Theory and Computation 20, 5352–5367 (2024).DOI
  47. [47]Y. Ni, S. Feng, X. Hong, Y. Sun, W.-Y. Ma, Z.-M. Ma, Q. Ye, and Y. Lan, “Pre-training with fractional denoising to enhance molecular property prediction,” Nature Machine Intelligence 6, 1169–1178 (2024).arxiv.org/abs/2407.11086
  48. [48]Y. Vander Meersche, G. Cretin, A. Gheeraert, J.-C. Gelly, and T. Galochkina, “ATLAS: protein flexibility description from atomistic molecular dynamics simulations,” Nucleic Acids Research 52, D384–D392 (2024).DOI
  49. [49]C. X. Hernández, H. K. Wayment-Steele, M. M. Sultan, B. E. Husic, and V. S. Pande, “Variational encoding of complex dynamics,” Physical Review E 97, 062412 (2018).DOI
  50. [50]H. Chen, B. Roux, and C. Chipot, “Discovering reaction pathways, slow variables, and committor probabilities with machine learning,” Journal of Chemical Theory and Computation 19, 4414–4426 (2023).DOI
  51. [51]J. Strahan, S. C. Guo, C. Lorpaiboon, A. R. Dinner, and J. Weare, “Inexact iterative numerical linear algebra for neural network-based spectral estimation and rare-event prediction,” Journal of Chemical Physics 159, 014110 (2023).DOI
  52. [52]H. Jung, R. Covino, A. Arjun, C. Leitold, C. Dellago, P. G. Bolhuis, and G. Hummer, “Machine-guided path sampling to discover mechanisms of molecular self-organization,” Nature Computational Science 3, 334–345 (2023).DOI
  53. [53]Y. Wang, T. Wang, S. Li, X. He, M. Li, Z. Wang, N. Zheng, B. Shao, and T.-Y. Liu, “Enhancing geometric representations for molecules with equivariant vector-scalar interactive message passing,” Nature Communications 15, 2313 (2024).DOI
  54. [54]P. Thölke and G. De Fabritiis, “TorchMD-NET: Equivariant transformers for neural network based molecular potentials,” arXiv preprint arXiv:2202.02541 (2022).arxiv.org/abs/2202.02541
  55. [56]N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko, “End-to-end object detection with transformers,” in Computer Vision – ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part I (Springer-Verlag, Berlin, Heidelberg, 2020) pp. 213–229.
  56. [57]Z. Pengmei, Z. Li, C. Chan Tien, R. Kondor, and A. R. Dinner, “Transformers are efficient hierarchical chemical graph learners,” arXiv preprint arXiv:2310.01704 (2023).arxiv.org/abs/2310.01704
  57. [58]O. Tolstikhin, N. Houlsby, A. Kolesnikov, L. Beyer, X. Zhai, T. Unterthiner, J. Young, A. Steiner, D. Keyers, J. Uszkoreit, M. Lucic, and A. Dosovitskiy, “MLP-Mixer: An all-MLP architecture for vision,” in Advances in Neural Information Processing Systems, Vol. 34 (Curran Associates, Inc., 2021) pp. 24261–24272.arxiv.org/abs/2105.01601
  58. [59]Z. Pengmei and Z. Li, “Technical report: The graph spectral token – enhancing graph transformers with spectral information,” arXiv preprint arXiv:2404.05604 (2024).arxiv.org/abs/2404.05604
  59. [60]B. Jing, S. Eismann, P. Suriana, R. J. L. Townshend, and R. Dror, “Learning from protein structure with geometric vector perceptrons,” in International Conference on Learning Representations (2020).arxiv.org/abs/2009.01411

Paper details

vlnc:W_yt7vkytg
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 10

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents