betaFeedback

WorksComputer Science

HamFormer: A Hamiltonian Flow-based SE(3)-Transformer for Conformation Optimization

Abstract

The Transformer architecture plays a central role in modern deep learning, and its SE(3)-equivariant variants have been widely applied to molecular conformation modeling and optimization. Given its importance, many methods attempt to interpret and improve the Transformer through the lens of dynamics. However, they mainly focus on a single attention head rather than the full Transformer architecture and seldom consider SE(3)-equivariance in their proposed models. In this study, we propose an improved SE(3)-Transformer architecture, called HamFormer, which revisits the design of SE(3)-Transformer through the lens of Hamiltonian flow. In particular, we define a parametrized Hamiltonian flow in the phase space of position and velocity, corresponding to optimizing an energy functional that combines an optimal transport-based potential with a damped momentum. The Lie-Trotter splitting framework of the Hamiltonian flow leads to the proposed HamFormer architecture, whose feedforward computation achieves the optimization step of the proposed energy functional. Furthermore, we combine HamFormer with Equivariant Graph Neural Network (EGNN), build a new model for molecular conformation optimization. Extensive experiments on molecular ground-state conformation prediction show that HamFormer provides a strong and physically interpretable surrogate for classic SE(3)-Transformer and generally outperforms state-of-the-art competitors.

Introduction

Transformers (Vaswani et al., [27]) have become the dominant architecture for modeling interacting elements, from tokens in language to atoms in molecules and materials. For the elements in three-dimensional space, whose underlying physics is invariant under rotations and translations, SE(3)SE(3)-Transformers (Thomas et al., [26]; Fuchs et al., [8]) build this symmetry into attention itself, coupling invariant scalar features with equivariant coordinates so that predictions transform consistently with their inputs. This geometric inductive bias underpins a series of practical successes across the molecular sciences, including protein structure prediction in AlphaFold (Jumper et al., [18]; Abramson et al., [1]), molecular representation learning in Uni-Mol (Zhou et al., [33]), equivariant force fields and materials modeling (Satorras et al., 2021; Liao and Smidt, [19]), and ground-state conformation prediction (Wang et al., [28]). As the Transformer and its SE(3)SE(3)-equivariant variants are entrusted with physically consequential predictions, understanding and improving what their layers compute has become a question of practical rather than merely theoretical significance.

Considerable efforts have gone into understanding the mechanism behind the Transformer, such as interpreting its self-attention layer through interacting particle dynamics (Geshkovski et al., [11]; Sander et al., [23]; Wang et al., [28]) and revisiting its residual architecture as ODE flows (Chen et al., [3]). Building on these interpretations, some recent methods improve the Transformer architecture through the lens of dynamics. For example, Sinkformer (Sander et al., [22]) applies doubly stochastic attention maps based on the Sinkhorn-scaling algorithm (Cuturi, [5]) and, accordingly, interprets the improved Transformer as Wasserstein gradient flows. Recent YurriFormer (Zimin et al., [34]) and SympFormer (Stein et al., [25]) introduce high-order dynamical information into the Transformer and design their models as the Nesterov-accelerated gradient descent algorithm. However, extending these dynamical interpretations to molecular refinement requires considering both feature evolution and molecular geometry. Beyond the attention operator, the architecture combines multiple attention heads, tokenwise FFNs, and pair relations that inform equivariant coordinate updates. The challenge is to connect these components within a common dynamical framework that guides their interaction while preserving SE(3)SE(3)-equivariance.

In this paper, we propose HamFormer, which revisits the SE(3)SE(3)-Transformer from the viewpoint of parametric Hamiltonian flows and improves the architecture accordingly. In principle, HamFormer is built based on coupled interacting particle dynamics, which capture the evolution of a distribution in the phase space of position and momentum. To model the dynamics, we define a Hamiltonian flow in phase space generated by an energy functional consisting of an optimal transport-based potential and a learned kinetic term. We parameterize the energy functional and discretize the resulting dynamics in the Lie–Trotter splitting framework (Hairer et al., [15]), with momentum retention applied before the interaction update. Given a set of particles (e.g., atomic embeddings of a molecule) and their initial momentum and pairwise relations, the discrete implementation of splitting leads to the proposed HamFormer architecture, whose feedforward computation updates the positions, momentum, and relations of the particles following the reference kick–drift structure.

Notably, HamFormer augments the atom features and pair relations used in SE(3)SE(3)-equivariant molecular architectures ([8, 33, 28]) with an explicit latent momentum state, allowing interactions from earlier layers to influence subsequent feature updates. In addition, compared with existing accelerated and momentum-based Transformer architectures ([34, 25, 10]), HamFormer combines an optimal transport-based potential with multi-head feature updates and SE(3)SE(3)-equivariant coordinate refinement.

As shown in Figure 1, combining HamFormer with Equivariant Graph Neural Network (EGNN) leads to a new model for molecular conformation optimization. We evaluate the proposed model on the task of ground-state conformation prediction. Experimental results demonstrate that HamFormer provides a strong and physically interpretable surrogate for classic SE(3)SE(3)-Transformer, whose feature updates follow a Hamiltonian reference structure and coordinate refinement preserves SE(3)SE(3)-equivariance. Accordingly, the HamFormer-based model outperforms state-of-the-art ground-state conformation predictors in both accuracy and efficiency.

An illustration of the proposed molecule model based on HamFormer

Figure 1. An illustration of the proposed molecule model based on HamFormer.

Related Work and Preliminaries

Transformer and Its SE(3)SE(3)-Equivariant Version. The Transformer uses attention to aggregate information across a sequence or a set of samples. Typically, given an input X∈RN×dX \in\mathbb{R}^{N \times d}, where NN indicates the sequence length or sample-set size and dd is the input feature dimension, an attention head first obtains the value, query, and key matrices by linear maps to a head dimension dhd_h, i.e., V=XWVV = XW_V, Q=XWQQ = XW_Q, and K=XWKK = XW_K, with WV,WQ,WK∈Rd×dhW_V,W_Q,W_K \in\mathbb{R}^{d \times d_h}. It then aggregates VV using weights determined by QQ and KK as Att⁡(V,Q,K):=A(Q,K)V\operatorname{Att}(V,Q,K) := A(Q,K)V, where A(Q,K)∈RN×NA(Q,K) \in\mathbb{R}^{N \times N} is the attention map parametrized by QQ and KK. The vanilla Transformer (Vaswani et al. [27]) implements this map based on the softmax operation, i.e., A(Q,K)=Softmax⁡(QK⊤/dh)A(Q,K) = \operatorname{Softmax}(QK^{\top}/\sqrt{d_h}), where Softmax⁡(⋅)\operatorname{Softmax}(\cdot) is applied to each row, giving nonnegative attention weights that sum to one. The multi-head attention layer applies a group of linear maps, i.e., θatt={WV,h,WQ,h,WK,h}h=1H\theta_{\mathrm{att}} = \{W_{V,h},W_{Q,h},W_{K,h}\}_{h=1}^{H}, to construct HH attention heads with a common head dimension dhd_h and concatenates their outputs, with the output projection omitted, so that MHA⁡θatt(X):=Concat⁡col{Att⁡(Vh,Qh,Kh)}h=1H\operatorname{MHA}_{\theta_{\mathrm{att}}}(X) := \operatorname{Concat}_{\mathrm{col}}\{\operatorname{Att}(V_h,Q_h,K_h)\}_{h=1}^{H}, where MHA⁡θatt(X)∈RN×Hdh\operatorname{MHA}_{\theta_{\mathrm{att}}}(X) \in\mathbb{R}^{N \times Hd_h}, Concat⁡col\operatorname{Concat}_{\mathrm{col}} denotes column-wise concatenation, and Vh=XWV,hV_h = XW_{V,h}, Qh=XWQ,hQ_h = XW_{Q,h}, and Kh=XWK,hK_h = XW_{K,h} for h=1,…,Hh = 1,\ldots,H. A complete Transformer block combines multi-head attention with a token-wise feedforward network (FFN), residual connections, and layer normalization.

Beyond sequence modeling, attention architectures support geometric prediction. For molecular coordinates C=[c1,…,cN]⊤∈RN×3C = [c_1,\ldots,c_N]^{\top} \in\mathbb{R}^{N \times3}, a coordinate predictor ff is SE(3)-equivariant if f(CR⊤+1t⊤)=f(C)R⊤+1t⊤f(CR^{\top} + \mathbf{1}t^{\top}) = f(C)R^{\top} + \mathbf{1}t^{\top} for every rotation R∈SO⁡(3)R \in\operatorname{SO}(3) and translation t∈R3t \in\mathbb{R}^{3}, with invariant atom and pair attributes held fixed. SE(3)-Transformer and Equiformer combine attention with equivariant representations for three-dimensional data and atomistic learning ([8, 19]).

Improved Transformer Architectures. Transformer layers can be interpreted as time discretizations of differential equations or iterations of optimization algorithms. The multi-particle formulation of Lu et al. [20] associates attention and FFNs with interacting flows, while optimization-based analyses view layers as iterative updates ([32]). Broader accounts connect Transformer dynamics to interacting-particle systems, gradient flows, and mean-field limits ([12]).

Sinkformers replace softmax attention with Sinkhorn normalization, which approximately enforces both row and column sums ([23]). Entropy-regularized optimal transport provides the associated optimization problem ([5]). Under suitable assumptions, the resulting continuous-depth dynamics form a Wasserstein gradient flow. WGFormer applies transport-normalized attention to molecular conformation refinement and evolves pair relations across layers ([28]). Their gradient-flow analyses use a single attention head, although the implemented architectures can contain multiple heads.

MomentumNets introduce persistent momentum-like states in residual networks ([22]), and Hamiltonian neural networks learn energies governing coupled state–momentum dynamics ([13]). Energy Transformer derives attention and tokenwise updates from a designed energy ([16]). More recently, YuriiFormer interprets attention and MLP updates through Lie–Trotter splitting of a composite objective ([34]), while SympFormer derives accelerated attention from inertial dynamics on density spaces ([25]). Connecting these dynamical descriptions to molecular refinement requires accounting for both invariant feature processing and equivariant coordinate updates.

Proposed HamFormer Architecture

In HamFormer, a parametrized Hamiltonian reference flow couples invariant atom features with latent momentum. The potential force updates momentum, and the kinetic gradient advances the features. Its Lie–Trotter splitting determines the feedforward update structure for molecular conformation optimization. Evolving multi-head relations condition these interactions and provide invariant edge information to EGNN for equivariant coordinate refinement.

Hamiltonian Flow

Hamiltonian dynamics are specified by an energy and a canonical state–momentum pairing (Evans [7]). On a periodic domain Ω\Omega of dimension dd, let x∈Ωx \in\Omega denote position, ρ\rho a smooth probability density, and SS its conjugate potential, with velocity field ∇S\nabla S. Let F\mathcal{F} be a potential-energy functional with first variation F′\mathcal{F}'. For NN particles, collect positions xix_i and momentum pip_i as rows of X,P∈RN×dX,P \in\mathbb{R}^{N \times d}, and let ρN:=N−1∑i=1Nδxi\rho_N := N^{-1} \sum_{i=1}^{N} \delta_{x_i} be the empirical approximation of ρ\rho, where δx\delta_x denotes the Dirac measure at xx. The Wasserstein Hamiltonian [4] and its particle counterpart are

HW(ρ,S)=12∫Ω∥∇S(x)∥2ρ(x) dx+F(ρ)andHN(X,P)=12∑i=1N∥pi∥2+NF(X),(1)\mathcal{H}_W(\rho,S) = \frac{1}{2}\int_{\Omega} \lVert\nabla S(x)\rVert^2 \rho(x)\,\mathrm{d}x + \mathcal{F}(\rho) \quad\text{and}\quad\mathcal{H}_N(X,P) = \frac{1}{2}\sum_{i=1}^{N}\lVert p_i\rVert^2 + N\mathcal{F}(X), \tag*{(1)}

where the two terms of HN\mathcal{H}_N define the kinetic energy KN(P)K_N(P) and potential energy VN(X)V_N(X), respectively. The particle expression uses total-energy normalization and assumes F(X)\mathcal{F}(X) is well defined and smooth in XX.

Writing ρt,St\rho_t,S_t for the state at time tt, define the momentum along a characteristic trajectory x(t)x(t) by p(t)=∇St(x(t))p(t)=\nabla S_t(x(t)) and the phase-space Hamiltonian by ht(x,p)=12∥p∥2+F′(ρt)(x)h_t(x,p)=\frac{1}{2}\lVert p\rVert^2+\mathcal{F}'(\rho_t)(x). With the time-dependent additive constant in StS_t chosen appropriately, the density and potential satisfy the continuity and Hamilton–Jacobi equations, whose characteristic trajectories obey Hamilton’s equations, i.e.,

∂tρt+∇⋅(ρt∇St)=0(continuity)∂tSt+12∥∇St∥2+F′(ρt)=0(HJE)\begin{aligned} \partial_t\rho_t+\nabla\cdot(\rho_t\nabla S_t)&=0 &&\text{(continuity)}\\ \partial_tS_t+\frac{1}{2}\lVert\nabla S_t\rVert^2+\mathcal{F}'(\rho_t)&=0 &&\text{(HJE)} \end{aligned}
x˙=∇pht=pp˙=−∇xht=−∇F′(ρt)(x)(Hamilton),\begin{aligned} \dot{x}&=\nabla_p h_t=p\\ \dot{p}&=-\nabla_x h_t=-\nabla\mathcal{F}'(\rho_t)(x) \end{aligned} \qquad\text{(Hamilton)},

where the right-hand system is evaluated along (x(t),p(t))(x(t),p(t)), with probability mass preserved for smooth periodic solutions and HW\mathcal{H}_W conserved when F\mathcal{F} is time-independent (see also, Appendix § 6.§ 1 for more details.).

More generally, for smooth, time-independent kinetic and potential energies KN,VNK_N,V_N, the potential subflow changes momentum at fixed positions, while the kinetic subflow advances positions at fixed momentum. Their Lie–Trotter composition gives first-order symplectic Euler (see also, Wu et al., [29], § 4). For a step size η>0\eta>0, let Xk,PkX^k,P^k approximate the states at tk=kηt_k=k\eta. The particle dynamics and their discretization are

P˙=−∇XVN(X)X˙=∇PKN(P)(continuous)\begin{aligned} \dot{P}&=-\nabla_XV_N(X)\\ \dot{X}&=\nabla_PK_N(P) \end{aligned} \qquad\text{(continuous)}
\begin{aligned} P^{k+1}&=P^k-\eta\nabla_XV_N(X^k)\\ X^{k+1}&=X^k+\eta\nabla_PK_N(P^{k+1}) \end{aligned} \qquad\text{(discretized)} \tag*{(2)}\end{aligned}

with the second-order Störmer–Verlet scheme obtained by symmetrizing this kick–drift splitting (Hairer et al., [15]).

Generalizing SE(3)-Transformer to Parametrized Hamiltonian Flows

A Parametric Instantiation of Hamiltonian. With θ\theta denoting the model parameters, we retain the canonical structure of (1), replacing the quadratic kinetic energy by a learned density κθ\kappa_\theta and the potential by a transport value. The rows of X,PX,P now represent invariant atom features and latent momentum. Let μ\mu be a probability measure over (ξ,x,p)∈[0,1]×Rd×Rd(\xi,x,p)\in[0,1]\times\mathbb{R}^{d}\times\mathbb{R}^{d} with finite second moments, and let ν\nu be its marginal on (ξ,x)(\xi,x). The fixed auxiliary label ξ\xi preserves pair identity as features evolve and is used only in this distributional representation. For each head hh, a bounded nonnegative measurable kernel th(ξ,ξ′)t_h(\xi,\xi') specifies the relations, with t=(th)h=1Ht=(t_h)_{h=1}^{H} fixed during the reference dynamics.

The potential is the optimal value of a coupling problem, whose optimizer supplies the interaction weights. Let zhz_h project features to head dimension dhd_h. With Π(ν,ν)\Pi(\nu,\nu) denoting couplings with both marginals ν\nu and KL denoting relative entropy, define

aθ,h(ξ,x,ξ′,x′∣th)=[zh(x)⊤zh(x′)]dh−log⁡(th(ξ,ξ′)+εnum),a_{\theta,h}(\xi,x,\xi',x'\mid t_h)=\frac{[z_h(x)^\top z_h(x')]}{\sqrt{d_h}}-\log(t_h(\xi,\xi')+\varepsilon_{\mathrm{num}}),
VEOT,h[ν∣th]=sup⁡πh∈Π(ν,ν){∫aθ,h dπh−εOTKL⁡(πh∥ν⊗ν)}(3)\mathcal{V}_{\mathrm{EOT},h}[\nu\mid t_h]=\sup_{\pi_h\in\Pi(\nu,\nu)}\left\{\int a_{\theta,h}\,\mathrm{d}\pi_h-\varepsilon_{\mathrm{OT}}\operatorname{KL}(\pi_h\Vert\nu\otimes\nu)\right\} \tag*{(3)}

where zh(x)=Wh⊤x+bhz_h(x)=W_h^\top x+b_h for Wh∈Rd×dhW_h\in\mathbb{R}^{d\times d_h} and bh∈Rdhb_h\in\mathbb{R}^{d_h}. The constants εnum>0\varepsilon_{\mathrm{num}}>0 and εOT>0\varepsilon_{\mathrm{OT}}>0 stabilize the logarithm and control entropic regularization, respectively. This is entropy-regularized optimal transport with cost −aθ,h-a_{\theta,h}, see also Cuturi, [5].

Extending the empirical representation above, let μN=N−1∑i=1Nδ(ξi,xi,pi)\mu_N=N^{-1}\sum_{i=1}^{N}\delta_{(\xi_i,x_i,p_i)} for distinct labels ξi\xi_i, with labeled feature marginal νN\nu_N. Set (Th)ij=th(ξi,ξj)(\mathbf{T}_h)_{ij}=t_h(\xi_i,\xi_j) and T=(Th)h=1H\mathbf{T}=(\mathbf{T}_h)_{h=1}^{H}. Couplings in Π(νN,νN)\Pi(\nu_N,\nu_N) have masses πij/N\pi_{ij}/N, where the transport matrix π=(πij)\pi=(\pi_{ij}) belongs to Π:={π∈R+N×N:π1=πT1=1}\Pi:=\{\pi\in\mathbb{R}_{+}^{N\times N}:\pi\mathbf{1}=\pi^{\mathsf{T}}\mathbf{1}=\mathbf{1}\}. For a score matrix S∈RN×N\mathbf{S}\in\mathbb{R}^{N\times N}, the finite transport value is

ΦεOT(S)=max⁡π∈Π∑i,jπij[Sij−εOT(log⁡πij−1)].(4)\Phi_{\varepsilon_{\mathrm{OT}}}(\mathbf{S})=\max_{\pi\in\Pi}\sum_{i,j}\pi_{ij}\left[S_{ij}-\varepsilon_{\mathrm{OT}}(\log\pi_{ij}-1)\right]. \tag*{(4)}

Writing Zh\mathbf{Z}_h for the projected feature matrix, the distributional and particle Hamiltonians are

Hθ[μ∣t]=∫κθ(p) dμ+12∑h=1HdhVεOT,h[ν∣th]=:Kθ[μ]+Vθ[ν∣t],(5)\mathcal{H}_{\theta}[\mu\mid\mathbf{t}]=\int\kappa_{\theta}(p)\,\mathrm{d}\mu+\frac{1}{2}\sum_{h=1}^{H}\sqrt{d_h}\mathcal{V}_{\varepsilon_{\mathrm{OT}},h}[\nu\mid t_h]=:\mathcal{K}_{\theta}[\mu]+\mathcal{V}_{\theta}[\nu\mid\mathbf{t}], \tag*{(5)}
Hθ,N(X,P∣T)=∑i=1Nκθ(pi)+12∑h=1HdhΦεOT([ZhZhT]/dh−log⁡(Th+εnum)).(6)\mathcal{H}_{\theta,N}(\mathbf{X},\mathbf{P}\mid\mathbf{T})=\sum_{i=1}^{N}\kappa_{\theta}(p_i)+\frac{1}{2}\sum_{h=1}^{H}\sqrt{d_h}\Phi_{\varepsilon_{\mathrm{OT}}}\left([\mathbf{Z}_h\mathbf{Z}_h^{\mathsf{T}}]/\sqrt{d_h}-\log(\mathbf{T}_h+\varepsilon_{\mathrm{num}})\right). \tag*{(6)}

where Kθ,Vθ\mathcal{K}_{\theta},\mathcal{V}_{\theta} denote the kinetic and potential energy functionals, and the two terms of the particle Hamiltonian define Kθ(P)\mathcal{K}_{\theta}(\mathbf{P}) and Vθ(X∣T)\mathcal{V}_{\theta}(\mathbf{X}\mid\mathbf{T}). The particle Hamiltonian equals NHθ[μN∣t]N\mathcal{H}_{\theta}[\mu_N\mid\mathbf{t}] up to a constant independent of X,P\mathbf{X},\mathbf{P}, as in the total-energy normalization of (1), with the exact relation derived in Appendix § 6.§ 2. The projected features are Zh=XWh+1bhT\mathbf{Z}_h=\mathbf{X}\mathbf{W}_h+\mathbf{1}b_h^{\mathsf{T}}, with matrix logarithms and scalar offsets applied entrywise. The kinetic density is κθ(p)=12∥0.1(BTp+b)∥2\kappa_{\theta}(p)=\frac{1}{2}\lVert\sqrt{0.1}(\mathbf{B}^{\mathsf{T}}p+b)\rVert^2 for learned B∈Rd×d\mathbf{B}\in\mathbb{R}^{d\times d} and b∈Rdb\in\mathbb{R}^{d}, where σa\sigma_a denotes entrywise leaky-ReLU with negative slope aa. The slope 0.1\sqrt{0.1} ensures that ∇pκθ(p)=B σ0.1(BTp+b)\nabla_p\kappa_{\theta}(p)=\mathbf{B}\,\sigma_{0.1}(\mathbf{B}^{\mathsf{T}}p+b), matching the kinetic drift. We take ϕ0.1(u)=∫0uσ0.1(s) ds\phi_{0.1}(u)=\int_0^u\sigma_{0.1}(s)\,\mathrm{d}s, where σ0.1(u)=max⁡{u,0.1u}\sigma_{0.1}(u)=\max\{u,0.1u\} is leaky-ReLU, applied entrywise.

Lie–Trotter realization. Fix θ\theta and T\mathbf{T}, and let πh⋆\pi_h^\star be the unique optimizer of (4) for head hh. The envelope identity ∇SΦεOT(S)=π⋆\nabla_{\mathbf{S}}\Phi_{\varepsilon_{\mathrm{OT}}}(\mathbf{S})=\pi^\star, obtained from Danskin’s theorem [6], together with ϕ0.1′=σ0.1\phi'_{0.1}=\sigma_{0.1}, gives

∇PKθ(P)=σ0.1(PB+1bT)BTand∇XVθ(X∣T)=12∑h=1Hdh(πh⋆+(πh⋆)T)ZhWhT.\nabla_{\mathbf{P}}\mathcal{K}_{\theta}(\mathbf{P})=\sigma_{0.1}(\mathbf{P}\mathbf{B}+\mathbf{1}b^{\mathsf{T}})\mathbf{B}^{\mathsf{T}} \quad\text{and}\quad \nabla_{\mathbf{X}}\mathcal{V}_{\theta}(\mathbf{X}\mid\mathbf{T})=\frac{1}{2}\sum_{h=1}^{H}\sqrt{d_h}\left(\pi_h^\star+(\pi_h^\star)^{\mathsf{T}}\right)\mathbf{Z}_h\mathbf{W}_h^{\mathsf{T}}.

where the prefactor dh/2\sqrt{d_h}/2 in the potential accounts for score normalization and the two feature arguments. Symmetric relations imply symmetric optimal plans, reducing the force to −∑hπh⋆ZhWhT-\sum_h\pi_h^\star\mathbf{Z}_h\mathbf{W}_h^{\mathsf{T}}. Substitution into (2) gives the reference kick–drift step, with momentum retention incorporated by applying P←e−γηP\mathbf{P}\leftarrow e^{-\gamma\eta}\mathbf{P} before the kick, where γ≥0\gamma\geq0 and γ=0\gamma=0 recovers the conservative splitting. Appendices § 6.§ 3 and § 6.§ 4 derive the gradients and splitting, respectively.

Implementation. HamFormer uses finite Sinkhorn iterations for transport, normalized learned maps for the kick and drift, and a tokenwise residual feed-forward network (FFN). Its stages alternate with EGNN stages, which use the final relations as invariant edge information for SE(3)-equivariant coordinate refinement, as shown in Figure 2.

Figure 2. HamFormer architecture.

Let ciin∈R3c_i^{\mathrm{in}}\in\mathbb{R}^{3} be the input coordinates and Dijin=∥ciin−cjin∥D_{ij}^{\mathrm{in}}=\lVert c_i^{\mathrm{in}}-c_j^{\mathrm{in}}\rVert. For each head, let gh(eij)g_h(e_{ij}) embed the unordered atom-pair type eije_{ij} into a scalar. Write Sinkhorn⁡εOT,I(S)\operatorname{Sinkhorn}_{\varepsilon_{\mathrm{OT}},I}(\mathbf{S}) for II alternating row and column scaling iterations on exp⁡(S/εOT)\exp(\mathbf{S}/\varepsilon_{\mathrm{OT}}), targeting marginals 1/N1/N. Assuming max⁡u,vDuvin>0\max_{u,v}D_{uv}^{\mathrm{in}}>0, initialize

Th(0)=NSinkhorn⁡εOT,I([gh(eij)Dijin]ijmax⁡u,vDuvin)(7)\mathbf{T}_h^{(0)}=N\operatorname{Sinkhorn}_{\varepsilon_{\mathrm{OT}},I}\left(\frac{\left[g_h(e_{ij})D_{ij}^{\mathrm{in}}\right]_{ij}}{\max_{u,v}D_{uv}^{\mathrm{in}}}\right) \tag*{(7)}

that NN rescales the target row and column sums to one, and we use I=3I = 3 iterations to approximate these constraints. At layer ℓ\ell, use layer-indexed parameters and let LNin,ℓ\mathrm{LN}_{\mathrm{in},\ell}, LNP,ℓ\mathrm{LN}_{P,\ell}, and LNX,ℓ\mathrm{LN}_{X,\ell} denote LayerNorm on the input features, interaction message, and drift. The relation update is

Zh(ℓ)=LNin,ℓ(X(ℓ))Wh(ℓ)+1(bh(ℓ))T,Z_{h}^{(\ell)}=\mathrm{LN}_{\mathrm{in},\ell}\left(X^{(\ell)}\right)W_{h}^{(\ell)}+\mathbf{1}\left(b_{h}^{(\ell)}\right)^{\mathsf{T}},
Th(ℓ+1)=NSinkhorn⁡εOT,I(Zh(ℓ)(Zh(ℓ))Tdh−log⁡(Th(ℓ)+εnum)),T_{h}^{(\ell+1)}=N\operatorname{Sinkhorn}_{\varepsilon_{\mathrm{OT}},I}\left(\frac{Z_{h}^{(\ell)}\left(Z_{h}^{(\ell)}\right)^{\mathsf{T}}}{\sqrt{d_{h}}}-\log\left(T_{h}^{(\ell)}+\varepsilon_{\mathrm{num}}\right)\right),

with the updated relations supplying the current interaction weights and conditioning the next layer. At εOT=1\varepsilon_{\mathrm{OT}}=1, the kernel divides the exponentiated affinity entrywise by Th(ℓ)+εnumT_{h}^{(\ell)}+\varepsilon_{\mathrm{num}}. The initialization and updates depend only on invariant attributes, distances, and features, so the relations remain invariant under rigid motions.

With learned retention αℓ∈R\alpha_{\ell}\in\mathbb{R} and post-drift features UU, the state update is

P(ℓ+1)=−tanh⁡(αℓ)P(ℓ)+LNP,ℓ(−∑h=1HTh(ℓ+1)Zh(ℓ)(Wh(ℓ))T),X(ℓ+1)=U+FFN⁡ℓ(U),(8)P^{(\ell+1)}=-\tanh(\alpha_{\ell})P^{(\ell)}+\mathrm{LN}_{P,\ell}\left(-\sum_{h=1}^{H}T_{h}^{(\ell+1)}Z_{h}^{(\ell)}\left(W_{h}^{(\ell)}\right)^{\mathsf{T}}\right),\qquad X^{(\ell+1)}=U+\operatorname{FFN}_{\ell}(U), \tag*{(8)}

where U=X(ℓ)+LNX,ℓ[σ0.1(P(ℓ+1)Bℓ+1bℓT)BℓT]U=X^{(\ell)}+\mathrm{LN}_{X,\ell}\left[\sigma_{0.1}\left(P^{(\ell+1)}B_{\ell}+\mathbf{1}b_{\ell}^{\mathsf{T}}\right)B_{\ell}^{\mathsf{T}}\right]. Only the interaction branch of the momentum update is normalized. The coefficient −tanh⁡(αℓ)∈(−1,1)-\tanh(\alpha_{\ell})\in(-1,1) is shared within each layer and allows signed retention, with αℓ=0\alpha_{\ell}=0 at initialization. Also, it has no effect in the first layer because the initial momentum is P(0)=0P^{(0)}=0. Learned maps and normalizations replace explicit step sizes.

For hidden width dFFd_{\mathrm{FF}}, let W1,ℓ∈Rd×dFFW_{1,\ell}\in\mathbb{R}^{d\times d_{\mathrm{FF}}} and W2,ℓ∈RdFF×dW_{2,\ell}\in\mathbb{R}^{d_{\mathrm{FF}}\times d} have compatible biases b1,ℓ,b2,ℓb_{1,\ell},b_{2,\ell}. With input LayerNorm LNF,ℓ\mathrm{LN}_{F,\ell}, activation and output dropout Drop⁡a,ℓ,Drop⁡o,ℓ\operatorname{Drop}_{a,\ell},\operatorname{Drop}_{o,\ell}, and hidden activation GG, define FFN⁡ℓ(U)=Drop⁡o,ℓ(GW2,ℓ+1b2,ℓT)\operatorname{FFN}_{\ell}(U)=\operatorname{Drop}_{o,\ell}(GW_{2,\ell}+\mathbf{1}b_{2,\ell}^{\mathsf{T}}), where G=Drop⁡a,ℓ[GELU⁡(LNF,ℓ(U)W1,ℓ+1b1,ℓT)]G=\operatorname{Drop}_{a,\ell}[\operatorname{GELU}(\mathrm{LN}_{F,\ell}(U)W_{1,\ell}+\mathbf{1}b_{1,\ell}^{\mathsf{T}})].

Before output normalization, the tied drift equals the reference kinetic gradient. With fixed relations and exact symmetric transport plans, the interaction message equals the reference force with respect to normalized features. Training differentiates through the relation updates, normalization, and Sinkhorn iterations using the supervised loss (see also, Appendix § 6.6 for more details).

The combined gradient of the multi-head OT potential plays the interaction role of multi-head attention in an SE(3)-Transformer: it supplies the force that updates latent momentum. The tied tokenwise feedforward map realizes the kinetic gradient and advances features using the updated momentum. Together, these operations form the reference kick–drift structure. For fixed parameters and relations, quadratic kinetic energy gives X¨=−∇XVθ\ddot{X}=-\nabla_{X}V_{\theta}, introducing second-order feature dynamics in contrast to the first-order flows in the single-head analyses of Sinkformers and WGFormer ([23, 28]). The learned kinetic energy generalizes the momentum-to-velocity relation. The residual FFN and EGNN extend this reference through learned feature corrections and equivariant coordinate updates. This provides a physics-aware organization of the complete architecture, connecting attention, momentum, feedforward processing, and molecular geometry.

Equivariant coordinate refinement. The initial relations depend only on atom-pair types and distances, and HamFormer preserves invariance of the features, momentum, and relations under rigid motions. Each EGNN stage ((Satorras et al., 2021)) uses these invariant quantities to construct coordinate displacements. At EGNN layer qq, learned pair messages produce a residual feature update using atom features, current and stage-entry squared distances, and final HamFormer relations. A separate MLP uses the updated features and the same distances and relations to produce invariant scalar weights wij(q)w_{ij}^{(q)}. The stage-entry distances and relations remain fixed throughout the stage. Coordinates are updated by ci(q+1)=ci(q)+∑jwij(q)ci(q)−cj(q)∥ci(q)−cj(q)∥+εdistc_{i}^{(q+1)}=c_{i}^{(q)}+\sum_{j}w_{ij}^{(q)}\frac{c_{i}^{(q)}-c_{j}^{(q)}}{\lVert c_{i}^{(q)}-c_{j}^{(q)}\rVert+\varepsilon_{\mathrm{dist}}}, where εdist>0\varepsilon_{\mathrm{dist}}>0 stabilizes the denominator and scalar normalization factors are absorbed into wij(q)w_{ij}^{(q)}. The scalar coefficients are invariant, while relative vectors rotate and are unchanged by translation, giving an SE(3)-equivariant update. Appendix § 6.§ 5 proves equivariance of the complete pipeline.

EGNN leaves momentum unchanged and passes its feature output directly to the next HamFormer stage. Figure 1 shows the complete architecture.

Physical interpretation. Compared with the first-order transport-flow interpretations of Sinkformers and WGFormer [23, 28], HamFormer introduces an independent momentum state between attention and feature motion. The combined gradient of the multi-head transport potential supplies the momentum kick, while the tied feedforward map supplies the kinetic drift. For fixed parameters and relations, quadratic kinetic energy gives X¨=−∇XVθ\ddot{X}=-\nabla_X V_\theta, so the interaction force determines acceleration. The learned kinetic energy generalizes the momentum-to-velocity relation. For ground-state conformation prediction, momentum carries earlier interaction effects across refinement steps, while evolving relations retain atom-pair information. Transport couples the atom features across the molecule, and EGNN uses the resulting features and relations to produce equivariant coordinate displacements (Satorras et al., 2021). Table 1 summarizes these roles. The mechanical interpretation concerns latent features and momentum, with molecular coordinates updated by EGNN.

ComponentFormPhysicsStateMolecular Role
OT Attention−∇XVθ-\nabla_X V_\thetaForcePPAtom Interactions
Kinetic Map∇PKθ\nabla_P K_\thetaVelocityXXFeature Motion
Retention−tanh⁡(αℓ)P-\tanh(\alpha_\ell)PSigned CarryPPRefinement Memory
RelationsTh(ℓ)↦Th(ℓ+1)T_h^{(\ell)}\mapsto T_h^{(\ell+1)}CouplingTTPair Information
EGNNci−cjc_i-c_jDirectionCCCoordinate Refinement

Table 1. Physical roles of HamFormer components.

ModelReference dynamicsInteractionMomentum parametrizationPair memorySE(3)
Sinkformers (2022)Wasserstein PDE: ∂tρ=∇⋅(ρ∇F′(ρ))\partial_t\rho=\nabla\cdot(\rho\nabla F'(\rho))OT–××
WGFormer (2025)Wasserstein PDE: ∂tρ=∇⋅(ρ∇F′(ρ))\partial_t\rho=\nabla\cdot(\rho\nabla F'(\rho))OT–✓✓
YuriiFormer (2026)Nesterov: ΔXk+1=βkΔXk−ηk∇E(Xk+αkΔXk)\Delta X^{k+1}=\beta_k\Delta X^k-\eta_k\nabla E(X^k+\alpha_k\Delta X^k)SoftmaxLearned coefficients××
SympFormer (2026)Hamiltonian PDE: ∂tρ=HS′,∂tS=−Hρ′−γ(t)S\partial_t\rho=\mathcal{H}'_S,\quad\partial_t S=-\mathcal{H}'_\rho-\gamma(t)SSoftmaxMetric, damping××
HamFormer (ours)Hamiltonian PDE: ∂tμ+{μ,hμ}=0\partial_t\mu+\{\mu,h_\mu\}=0OTKinetic map, signed retention✓✓

Table 2. Reference dynamics under the respective theoretical assumptions and features of the reported architectures. OT denotes entropic optimal transport, and pair memory denotes relations retained across layers. The SympFormer row reports its softmax variant.

Table 2 compares the reference dynamics and architectural features. YuriiFormer introduces momentum through a Nesterov template, while SympFormer uses an attention-dependent kinetic metric. HamFormer combines a separable reference Hamiltonian at fixed relations with entropic transport interactions, persistent pair relations, and SE(3)-equivariant coordinate refinement.

In the Nesterov template, ΔXk=Xk−Xk−1\Delta X^k=X^k-X^{k-1}, and αk,βk,ηk\alpha_k,\beta_k,\eta_k control lookahead, retention, and step size for the reference objective EE. In the Hamiltonian rows, primes denote first variations, and hμ=Hμ′[μ∣t]h_\mu=\mathcal{H}'_\mu[\mu\mid t] is evaluated at fixed parameters and relations. The canonical Poisson bracket {f,g}=∇xf⋅∇pg−∇pf⋅∇xg\{f,g\}=\nabla_xf\cdot\nabla_pg-\nabla_pf\cdot\nabla_xg describes transport in feature–momentum space.

With feature dimensions, head count, and Sinkhorn iterations fixed, each dense HamFormer layer requires O(N2)O(N^2) time and storage, as detailed in Remark A.4. In practice, we use three Sinkhorn iterations and reuse each head’s projected features for both score construction and transport aggregation. The practical layer retains the reference kick–drift order, with finite Sinkhorn iterations, normalized learned maps, signed retention, and a residual FFN. Appendix A.6 gives the complete pipeline, both algorithms, and the supervised training objective.

Experiments

To validate the effectiveness and physical interpretability of HamFormer, we evaluate our model on molecular ground-state conformation prediction following the established experimental protocol of WGFormer Wang et al. [28]. In addition to quantitative comparisons with existing 2D graph models and 3D equivariant architectures, we provide an empirical analysis of the latent dynamics learned by HamFormer, focusing on the layerwise evolution of phase-space energy and the role of the latent momentum state.

Experimental Setup

Datasets and splitting strategy. We evaluate our model on two standard molecular conformation benchmarks: QM9 and the larger-scale Molecule3D [31]. QM9 comprises approximately 134k small organic molecules with up to nine heavy atoms, with ground-state geometries calculated via density functional theory (DFT). Molecule3D contains substantially larger and more flexible drug-like molecules. Following the benchmark protocol in Wang et al. [28], we adopt the standard training, validation, and test splits for QM9, and evaluate on the standard random split for Molecule3D.

Baselines. Following Wang et al. [28], we compare HamFormer with two categories of baseline methods: (1) 2D molecular graph models, including GINE [17], GATv2 [2], GPS [21], and GTMGC [30], which generate 3D coordinates directly from 2D molecular graphs; and (2) 3D conformation refinement methods, including ConfOpt with TwoAtom and ThreeAtom energy functions [14], ConfOpt variants using SE(3)-Transformer [8] and EGNN (Satorras et al., 2021) backbones, and WGFormer [28].

Evaluation metrics. Following the established evaluation protocol in [31, 28], we evaluate predicted conformations using three complementary geometric metrics: (1) D-MAE (Distance Mean Absolute Error), (2) D-RMSE (Distance Root Mean Square Error), which measure the accuracy of internal molecular geometry via pairwise interatomic distances, and (3) C-RMSD (Coordinate Root Mean Square Deviation), which assesses global 3D structural alignment by computing the coordinate error against the ground truth after optimal rotation and translation. All metrics are reported in Ångströms (Å), where lower values indicate better predictive quality.

Main Results

Table 3 reports the quantitative comparison on Molecule3D and QM9. Compared to WGFormer [28], HamFormer achieves lower errors across all metrics on QM9 (e.g., test C-RMSD of 0.198 Å vs. 0.206 Å), while delivering improved D-MAE and C-RMSD on Molecule3D. HamFormer attains these results with 16.3M parameters, an approximate 79% reduction compared to WGFormer (78.9M). Compared to the first-order Wasserstein gradient flow in WGFormer, incorporating second-order phase-space dynamics provides a more parameter-efficient inductive bias for molecular conformation refinement.

DatasetModelValidationTest
D-MAE ↓D-RMSE ↓C-RMSD ↓D-MAE ↓D-RMSE ↓C-RMSD ↓
Molecule3D (Random)GINE(2D)0.5901.0141.1160.5921.0181.116
GATv2(2D)0.5630.9831.0820.5640.9861.083
GPS(2D)0.5280.9091.0360.5290.9111.038
GTMGC(2D)0.4320.7190.7120.4330.7210.713
SE(3)-Transformer0.4660.7120.8000.4670.7740.802
EGNN0.4610.7040.7980.4620.7660.799
ConfOpt-TwoAtom0.4380.6680.7480.4380.6700.749
ConfOpt-ThreeAtom0.4290.6590.7340.4300.6610.736
WGFormer<u>0.391</u>0.6490.662<u>0.392</u>0.6520.664
HamFormer0.386<u>0.653</u>0.6520.387<u>0.656</u>0.654
QM9GINE(2D)0.3570.6730.6850.3570.6690.693
GATv2(2D)0.3390.6630.6610.3390.6590.666
GPS(2D)0.3260.6440.6620.3260.6400.666
GTMGC(2D)0.2620.4680.3620.2640.4700.367
SE(3)-Transformer0.2540.4510.2960.2560.4550.303
EGNN0.2480.4420.2570.2510.4490.265
ConfOpt-TwoAtom0.2450.4390.2450.2480.4440.254
ConfOpt-ThreeAtom0.2410.4330.2370.2440.4380.246
WGFormer<u>0.223</u><u>0.416</u><u>0.198</u><u>0.227</u><u>0.422</u><u>0.206</u>
HamFormer0.2150.4140.1910.2180.4190.198

Table 3. Table 3. Performance comparison on Molecule3D and QM9 datasets. ↓ indicates that lower values are better. Bold indicates the best performance, and <u>underline</u> indicates the second best. Baseline results follow Wang et al. [28].

Ablation Studies and Further Analysis

Momentum Evolution over Layers. Figure 3 examines the six HamFormer layers separated into two alternating stages (H1–H3 for Stage 1 and H4–H6 for Stage 2). As expected in a dissipative physical system, the latent dynamics exhibit a clear two-stage relaxation behavior: 1) across the network as a whole, both the potential energy and total Hamiltonian follow an overall downward trajectory, achieving net energy dissipation toward a stable ground-state equilibrium; 2) the latent kinetic energy remains relatively suppressed in Stage 1, but experiences a prominent transient surge at the onset of Stage 2 (layer H4, marked by the dashed line) as stored potential energy is rapidly unleashed, followed by smooth dissipation across the remaining layers.

Figure 3. Layer-wise evolution of the latent kinetic energy, potential energy, and total Hamiltonian evaluated on 1,000 random molecules from the Molecule3D test set. Solid curves denote the mean, and shaded regions represent one standard deviation across molecules. The dashed line marks the transition between Stage 1 (layers H1–H3) and Stage 2 (layers H4–H6).

Architectural and Momentum Ablations. Figure 4 (left) systematically investigates the necessity of individual components on the Molecule3D random test set: (1) Inference-time momentum retention: Zeroing out the learned momentum at inference (ρl=0\rho_l = 0) induces pronounced performance degradation across all metrics, with test C-RMSD rising from 0.654 Å to 0.714 Å and D-MAE increasing from 0.387 Å to 0.420 Å. This demonstrates that preserving momentum across layers acts as an indispensable dynamical memory, effectively preventing error accumulation during iterative refinement. (2) Hamiltonian block ablation: Removing HamFormer and using pure EGNN, which eliminates phase-space feature updates and relies solely

Figure 4. Qualitative conformation visualizations and quantitative ablation studies on Molecule3D. Left: Quantitative ablation on the test set comparing the full HamFormer with zeroed inference momentum (ρl=0\rho_l = 0) and the EGNN baseline. Right: Visual comparison of predicted 3D molecular geometries against DFT references for RDKit, WGFormer, and HamFormer.

on local message-passing—causes notable performance drops (e.g., test C-RMSD rises to 0.697 Å). This confirms that our phase-space Hamiltonian flow provides an essential non-local inductive bias for capturing global molecular geometry.

Conformation Visualizations. Figure 4 (right) presents visual comparisons of predicted 3D molecular conformations aligned against ground-truth reference structures for representative molecules. Compared WGFormer, the conformations generated by HamFormer consistently align more closely with the true reference geometries, exhibiting fewer steric clashes and more accurate bond lengths and angles.

Conclusion and Future Work

We presented HamFormer, which brings Hamiltonian-inspired latent dynamics into SE(3)-equivariant conformation refinement through the joint evolution of features, momentum, and transport-based relations. Its integration with EGNN yields improved refinement on QM9 and Molecule3D, suggesting that structured latent dynamics can complement geometric message passing. These findings motivate further study of the balance between representational capacity and computational efficiency, particularly as molecular size and structural complexity increase. Extending this framework beyond small molecules remains a promising direction for assessing its scalability and broader applicability.

Theoretical Details

Section A.1 derives the density equations, their characteristics, and conservation properties. Sections A.2 and A.3 establish the particle representation of the learned Hamiltonian and compute its force and drift. Section A.4 derives the Lie–Trotter updates and examines momentum retention, and Section A.5 proves SE(3)-equivariance of the complete pipeline. Section A.6 presents the molecular refinement algorithms and supervised objective. Finally, Section A.7 details Sinkhorn scaling, rotational alignment, and computational complexity. Continuum calculations assume sufficient regularity to justify differentiation and integration by parts.

Hamiltonian Dynamics on Density Space

Consider the Wasserstein Hamiltonian in (1) on a periodic domain Ω\Omega,

HW(ρ,S)=12∫Ω∥∇S(x)∥2ρ(x) dx+F(ρ).\mathcal{H}_{\mathrm{W}}(\rho,S) = \frac{1}{2} \int_{\Omega} \lVert\nabla S(x) \rVert^2 \rho(x)\,\mathrm{d}x + \mathcal{F}(\rho).

Let rr be a smooth density perturbation with ∫Ωr dx=0\int_{\Omega} r\,\mathrm{d}x = 0, and let ψ\psi be a smooth perturbation of SS. Writing DD for a directional derivative, the two variations are

DρHW(ρ,S)[r]=∫Ω(12∥∇S∥2+F′(ρ))r dx,D_{\rho}\mathcal{H}_{\mathrm{W}}(\rho,S)[r] = \int_{\Omega} \left(\frac{1}{2}\lVert\nabla S\rVert^2 + \mathcal{F}'(\rho)\right)r\,\mathrm{d}x,
DSHW(ρ,S)[ψ]=∫Ωρ∇S⋅∇ψ dx=−∫Ω∇⋅(ρ∇S)ψ dx,D_{S}\mathcal{H}_{\mathrm{W}}(\rho,S)[\psi] = \int_{\Omega} \rho\nabla S \cdot\nabla\psi\,\mathrm{d}x = -\int_{\Omega} \nabla\cdot(\rho\nabla S)\psi\,\mathrm{d}x,

where periodicity removes the boundary term. Since density perturbations have zero integral, the first variation with respect to ρ\rho is determined up to a spatial constant.

The canonical pairing identifies the density derivative with the variation in SS, and the potential derivative with the negative variation in ρ\rho [4]. Choosing the time-dependent additive constant in StS_t gives

{∂tρt=−∇⋅(ρt∇St),∂tSt=−12∥∇St∥2−F′(ρt),\begin{cases} \partial_t\rho_t = -\nabla\cdot(\rho_t\nabla S_t),\\ \partial_t S_t = -\frac{1}{2}\lVert\nabla S_t\rVert^2 - \mathcal{F}'(\rho_t), \end{cases}

where the first equation transports probability with velocity ∇St\nabla S_t, while the second determines the evolution of the conjugate potential.

Let x(t)x(t) satisfy x˙(t)=∇St(x(t))\dot{x}(t) = \nabla S_t(x(t)), and define p(t)=∇St(x(t))p(t) = \nabla S_t(x(t)). The chain rule along this trajectory yields

p˙(t)=∇∂tSt(x(t))+∇2St(x(t))x˙(t)=∇(∂tSt+12∥∇St∥2)(x(t))=−∇F′(ρt)(x(t)).\begin{aligned} \dot{p}(t) = \nabla\partial_t S_t(x(t)) + \nabla^2S_t(x(t))\dot{x}(t) \\ &= \nabla\left(\partial_t S_t + \frac{1}{2}\lVert\nabla S_t\rVert^2\right)(x(t)) \\ &= -\nabla\mathcal{F}'(\rho_t)(x(t)). \end{aligned}

Thus, with ht(x,p)=12∥p∥2+F′(ρt)(x)h_t(x,p) = \frac{1}{2}\lVert p\rVert^2 + \mathcal{F}'(\rho_t)(x), the characteristic equations are

x˙=∇pht=p,p˙=−∇xht=−∇F′(ρt)(x),\dot{x} = \nabla_p h_t = p,\qquad\dot{p} = -\nabla_x h_t = -\nabla\mathcal{F}'(\rho_t)(x),

which recover Hamilton’s equations through the classical Hamilton–Jacobi construction [7 Sec. 3.3.1]. Integrating the continuity equation over Ω\Omega gives

ddt∫Ωρt dx=−∫Ω∇⋅(ρt∇St) dx=0.\frac{\mathrm{d}}{\mathrm{d}t}\int_{\Omega}\rho_t\,\mathrm{d}x = -\int_{\Omega}\nabla\cdot(\rho_t\nabla S_t)\,\mathrm{d}x = 0.

If F\mathcal{F} has no explicit time dependence, differentiating the Hamiltonian and integrating by parts also gives

ddtHW(ρt,St)=∫Ω(12∥∇St∥2+F′(ρt))∂tρt dx+∫Ωρt∇St⋅∇∂tSt dx=∫Ω(−∂tSt ∂tρt+∂tρt ∂tSt) dx=0.\frac{\mathrm{d}}{\mathrm{d}t}\mathcal{H}_{W}(\rho_t,S_t)=\int_{\Omega}\left(\frac{1}{2}\lVert\nabla S_t\rVert^2+\mathcal{F}'(\rho_t)\right)\partial_t\rho_t\,\mathrm{d}x+\int_{\Omega}\rho_t\nabla S_t\cdot\nabla\partial_tS_t\,\mathrm{d}x =\int_{\Omega}\left(-\partial_tS_t\,\partial_t\rho_t+\partial_t\rho_t\,\partial_tS_t\right)\,\mathrm{d}x=0.

The conserved quantity is the total functional HW\mathcal{H}_{W}. The characteristic Hamiltonian hth_t can still depend on time through ρt\rho_t.

Entropic Interaction and Particle Representation

The learned Hamiltonian in (5) combines a parametrized kinetic energy density κθ\kappa_\theta with an entropic transport potential. We first verify that this energy is finite and then derive its particle representation.

For the interaction term, the inequality

∣zh(x)⊤zh(x′)∣≤12∥zh(x)∥2+12∥zh(x′)∥2\left|z_h(\boldsymbol{x})^{\top}z_h(\boldsymbol{x}')\right|\leq\frac{1}{2}\lVert z_h(\boldsymbol{x})\rVert^2+\frac{1}{2}\lVert z_h(\boldsymbol{x}')\rVert^2

and the finite second moments of the fixed marginals bound the affinity integral uniformly over admissible couplings. The logarithmic relation term is bounded because tht_h is bounded and ϵnum>0\epsilon_{\mathrm{num}}>0. Relative entropy is nonnegative, and the product coupling ν⊗ν\nu\otimes\nu is admissible with zero relative entropy. These observations give finite upper and lower bounds for the interaction value. The kinetic density has at most quadratic growth, so its integral is finite as well.

For distinct fixed atom identifiers ξi\xi_i, consider the empirical measures

μN=1N∑i=1Nδ(ξi,xi,pi)andνN=1N∑i=1Nδ(ξi,xi).\mu_N=\frac{1}{N}\sum_{i=1}^{N}\delta_{(\xi_i,x_i,p_i)} \qquad\text{and}\qquad \nu_N=\frac{1}{N}\sum_{i=1}^{N}\delta_{(\xi_i,x_i)}.

The identifiers distinguish atoms with identical features. For a coupling of νN\nu_N with itself, let πij/N\pi_{ij}/N denote the mass assigned to the ordered atom pair (i,j)(i,j), where

π∈Π:={π∈R+N×N:π1=π⊤1=1},\pi\in\Pi:=\left\{\pi\in\mathbb{R}_{+}^{N\times N}:\pi\mathbf{1}=\pi^{\top}\mathbf{1}=\mathbf{1}\right\},

so that π\pi has unit row and column sums, whereas π/N\pi/N is a probability coupling. With (Th)ij=th(ξi,ξj)(T_h)_{ij}=t_h(\xi_i,\xi_j) and Zh=XWh+1bh⊤Z_h=XW_h+\mathbf{1}b_h^{\top}, the sampled affinity is

Sh=ZhZh⊤dh−log⁡(Th+ϵnum),S_h=\frac{Z_hZ_h^{\top}}{\sqrt{d_h}}-\log(T_h+\epsilon_{\mathrm{num}}),

with finite transport value ΦϵOT(Sh)\Phi_{\epsilon_{\mathrm{OT}}}(S_h) from (4). The following identity accounts for the difference between the distributional and matrix entropy conventions.

Lemma A.1 (Particle normalization). For μN\mu_N and its corresponding relation tensor,

Hθ,N=NHθ[μN∣t]+ϵOTN(1+log⁡N)∑h=1Hdh2,H_{\theta,N}=NH_{\theta}[\mu_N\mid t]+\epsilon_{\mathrm{OT}}N(1+\log N)\sum_{h=1}^{H}\frac{\sqrt{d_h}}{2},

where the additive correction is independent of XX and PP.

Proof. The product measure νN⊗νN\nu_N\otimes\nu_N assigns mass 1/N21/N^2 to each ordered atom pair. Using 0log⁡0=00\log0=0 and ∑ijπij=N\sum_{ij}\pi_{ij}=N, a coupling with masses πij/N\pi_{ij}/N satisfies

KL⁡(πh∥νN⊗νN)=∑i,jπijNlog⁡(πij/N1/N2)=1N∑i,jπijlog⁡πij+log⁡N.(9)\operatorname{KL}(\pi_h\Vert\nu_N\otimes\nu_N)=\sum_{i,j}\frac{\pi_{ij}}{N}\log\left(\frac{\pi_{ij}/N}{1/N^2}\right)=\frac{1}{N}\sum_{i,j}\pi_{ij}\log\pi_{ij}+\log N. \tag*{(9)}

Consequently,

NVεOT,h[νN∣th]=max⁡π∈Π{∑i,jπij(Sh)ij−εOT∑i,jπijlog⁡πij}−εOTNlog⁡N=ΦεOT(Sh)−εOTN(1+log⁡N).\begin{aligned} N\mathcal{V}_{\varepsilon_{\mathrm{OT}},h}[\nu_N \mid t_h] = \max_{\pi\in\Pi}\left\{\sum_{i,j}\pi_{ij}(S_h)_{ij} - \varepsilon_{\mathrm{OT}}\sum_{i,j}\pi_{ij}\log\pi_{ij}\right\} - \varepsilon_{\mathrm{OT}}N\log N \\ &= \Phi_{\varepsilon_{\mathrm{OT}}}(S_h) - \varepsilon_{\mathrm{OT}}N(1+\log N). \end{aligned}

The kinetic term satisfies

N∫κθ(p) dμN=∑i=1Nκθ(pi)=Kθ(P),N\int\kappa_\theta(\mathbf{p})\,\mathrm{d}\mu_N = \sum_{i=1}^{N}\kappa_\theta(\mathbf{p}_i) = K_\theta(\mathbf{P}),

and summing the interaction identity over heads with weights dh/2\sqrt{d_h}/2 proves the result.

This is an exact evaluation on empirical measures and does not require a limit as N→∞N \to\infty. After total-energy normalization, the additive constant has no effect on the feature or momentum gradients.

Gradients of the Reference Hamiltonian

For matrices of the same size, write ⟨A,B⟩=tr⁡(ATB)\langle A,B\rangle= \operatorname{tr}(A^{\mathsf{T}}B). The entropy term makes the finite coupling problem strictly concave in its optimization variable.

Lemma A.2 (Transport gradient). For finite SS and εOT>0\varepsilon_{\mathrm{OT}} > 0, the problem in (4) has a unique positive optimizer π⋆\pi^\star, and

∇SΦεOT(S)=π⋆.\nabla_S\Phi_{\varepsilon_{\mathrm{OT}}}(S) = \pi^\star.

If SS is symmetric, then π⋆\pi^\star is symmetric.

Proof. The admissible set Π\Pi is nonempty and compact, and the objective is continuous under 0log⁡0=00\log0 = 0. A maximizer therefore exists, and strict concavity of the entropy term gives uniqueness.

If an optimizer has a zero entry, interpolate it with 11T/N\mathbf{1}\mathbf{1}^{\mathsf{T}}/N. For a small interpolation weight τ>0\tau> 0, the entropy gain at a zero entry has order −τlog⁡τ-\tau\log\tau, while the remaining changes have order τ\tau. The objective therefore increases for sufficiently small τ\tau, contradicting optimality.

At a fixed coupling, the derivative of the objective with respect to SS is π\pi. Danskin’s theorem and uniqueness give the stated gradient [6]. For symmetric SS, transposition preserves feasibility and the objective. Uniqueness then implies π⋆=(π⋆)T\pi^\star= (\pi^\star)^{\mathsf{T}}.

Fix θ\theta and T\mathbf{T}. For head hh, the score differential is

dSh=(dZh)ZhT+Zh(dZh)Tdh.\mathrm{d}S_h = \frac{(\mathrm{d}Z_h)Z_h^{\mathsf{T}} + Z_h(\mathrm{d}Z_h)^{\mathsf{T}}}{\sqrt{d_h}}.

where dZh=(dX)Wh\mathrm{d}Z_h = (\mathrm{d}X)W_h. Applying Lemma A.2 gives

d[dh2ΦεOT(Sh)]=12⟨πh⋆,(dZh)ZhT+Zh(dZh)T⟩=12⟨πh⋆+(πh⋆)T,Zh dZh⟩=12⟨(πh⋆+(πh⋆)T)ZhWhT,dX⟩.\begin{aligned} \mathrm{d}\left[\frac{\sqrt{d_h}}{2}\Phi_{\varepsilon_{\mathrm{OT}}}(S_h)\right] = \frac{1}{2}\left\langle\pi_h^\star,(\mathrm{d}Z_h)Z_h^{\mathsf{T}} + Z_h(\mathrm{d}Z_h)^{\mathsf{T}}\right\rangle = \frac{1}{2}\left\langle\pi_h^\star+ (\pi_h^\star)^{\mathsf{T}},Z_h\,\mathrm{d}Z_h\right\rangle\\ &= \frac{1}{2}\left\langle\left(\pi_h^\star+ (\pi_h^\star)^{\mathsf{T}}\right)Z_hW_h^{\mathsf{T}},\mathrm{d}X\right\rangle. \end{aligned}

Summing over heads yields

∇XVθ(X∣T)=12∑h=1H(πh⋆+(πh⋆)T)ZhWhT.\nabla_X V_\theta(X\mid\mathbf{T}) = \frac{1}{2}\sum_{h=1}^{H}\left(\pi_h^\star+ (\pi_h^\star)^{\mathsf{T}}\right)Z_hW_h^{\mathsf{T}}.

The factor dh/2\sqrt{d_h}/2 cancels the score normalization and averages the contributions from the two feature arguments. If each ThT_h is symmetric, then ShS_h and its optimal plan are symmetric, so the force reduces to

−∇XVθ(X∣T)=−∑h=1Hπh⋆ZhWh⊤.-\nabla_X V_\theta(X \mid\mathbf{T})=-\sum_{h=1}^{H}\pi_h^{\star}Z_hW_h^{\top}.

These derivatives use the exact optimizer and hold relations fixed.

For the kinetic term, the primitive of leaky-ReLU is

ϕ0.1(u)=∫0uσ0.1(s) ds={12u2,u≥0,0.12u2,u<0,\phi_{0.1}(u)=\int_0^u \sigma_{0.1}(s)\,\mathrm{d}s= \begin{cases} \frac{1}{2}u^2, & u\ge0,\\ \frac{0.1}{2}u^2, & u<0, \end{cases}

so ϕ0.1′(u)=σ0.1(u)\phi'_{0.1}(u)=\sigma_{0.1}(u). Under the row convention for P\mathbf{P},

Kθ(P)=∑i,rϕ0.1((PB+1b⊤)ir).K_\theta(\mathbf{P})=\sum_{i,r}\phi_{0.1}\left((\mathbf{P}\mathbf{B}+\mathbf{1}\mathbf{b}^{\top})_{ir}\right).

Its differential is

dKθ=⟨σ0.1(PB+1b⊤),(dP)B⟩=⟨σ0.1(PB+1b⊤)B⊤,dP⟩,\mathrm{d}K_\theta=\left\langle\sigma_{0.1}(\mathbf{P}\mathbf{B}+\mathbf{1}\mathbf{b}^{\top}),(\mathrm{d}\mathbf{P})\mathbf{B}\right\rangle =\left\langle\sigma_{0.1}(\mathbf{P}\mathbf{B}+\mathbf{1}\mathbf{b}^{\top})\mathbf{B}^{\top},\mathrm{d}\mathbf{P}\right\rangle,

and hence

∇PKθ(P)=σ0.1(PB+1b⊤)B⊤.\nabla_{\mathbf{P}}K_\theta(\mathbf{P})=\sigma_{0.1}(\mathbf{P}\mathbf{B}+\mathbf{1}\mathbf{b}^{\top})\mathbf{B}^{\top}.

The transpose appears because both linear maps arise from differentiating the same scalar energy. This gradient converts momentum into feature velocity. Momentum and velocity coincide for Kθ(P)=12∥P∥F2K_\theta(\mathbf{P})=\frac{1}{2}\lVert\mathbf{P}\rVert_F^2, but need not coincide for the learned energy.

The primitive ϕ0.1\phi_{0.1} is continuously differentiable with a Lipschitz derivative, but is only piecewise twice differentiable. Calculations requiring a classical Hessian therefore apply within regions where no kinetic preactivation crosses zero.

Lie–Trotter Splitting and the Implemented Layer

For fixed parameters and relations, write Hθ,N=Kθ+VθH_{\theta,N}=K_\theta+V_\theta. Hamilton’s equations are

X˙=∇PKθ(P),P˙=−∇XVθ(X∣T).\begin{aligned} \dot{\mathbf{X}}&=\nabla_{\mathbf{P}}K_\theta(\mathbf{P}),\\ \dot{\mathbf{P}}&=-\nabla_{\mathbf{X}}V_\theta(\mathbf{X}\mid\mathbf{T}). \end{aligned}

Along a differentiable solution,

ddtHθ,N=⟨∇XVθ,∇PKθ⟩−⟨∇PKθ,∇XVθ⟩=0,\frac{\mathrm{d}}{\mathrm{d}t}H_{\theta,N}=\left\langle\nabla_{\mathbf{X}}V_\theta,\nabla_{\mathbf{P}}K_\theta\right\rangle-\left\langle\nabla_{\mathbf{P}}K_\theta,\nabla_{\mathbf{X}}V_\theta\right\rangle=0,

so the autonomous reference flow conserves its Hamiltonian.

The potential subflow changes momentum at fixed features, while the kinetic subflow advances features at fixed momentum. Their exact maps over a step η>0\eta>0 are

φVη(X,P)=(X,P−η∇XVθ(X∣T)),\varphi_V^\eta(\mathbf{X},\mathbf{P})=\left(\mathbf{X},\mathbf{P}-\eta\nabla_{\mathbf{X}}V_\theta(\mathbf{X}\mid\mathbf{T})\right),
φKη(X,P)=(X+η∇PKθ(P),P).\varphi_K^\eta(\mathbf{X},\mathbf{P})=\left(\mathbf{X}+\eta\nabla_{\mathbf{P}}K_\theta(\mathbf{P}),\mathbf{P}\right).

Applying φVη\varphi_V^\eta first and φKη\varphi_K^\eta second gives (Xk+1,Pk+1)=φKη∘φVη(Xk,Pk)(\mathbf{X}^{k+1},\mathbf{P}^{k+1})=\varphi_K^\eta\circ\varphi_V^\eta(\mathbf{X}^k,\mathbf{P}^k), which expands to

Pk+1=Pk−η∇XVθ(Xk∣T),Xk+1=Xk+η∇PKθ(Pk+1).\begin{aligned} \mathbf{P}^{k+1}&=\mathbf{P}^k-\eta\nabla_{\mathbf{X}}V_\theta(\mathbf{X}^k\mid\mathbf{T}),\\ \mathbf{X}^{k+1}&=\mathbf{X}^k+\eta\nabla_{\mathbf{P}}K_\theta(\mathbf{P}^{k+1}). \end{aligned}

This is the momentum-first symplectic Euler step in (2). For sufficiently smooth energies, Lie–Trotter composition has first-order accuracy, while the symmetric composition φVη/2∘φKη∘φVη/2\varphi_V^{\eta/2} \circ\varphi_K^\eta\circ\varphi_V^{\eta/2} gives the second-order Störmer–Verlet method ([15]).

Symplecticity follows from the subflow Jacobians. After vectorizing XX and PP and suppressing evaluation points, these have the block forms

DφVη=(I0−η∇2VθI),DφKη=(Iη∇2Kθ0I).D\varphi_V^\eta= \begin{pmatrix} I & 0 \\ -\eta\nabla^2 V_\theta& I \end{pmatrix}, \qquad D\varphi_K^\eta= \begin{pmatrix} I & \eta\nabla^2 K_\theta\\ 0 & I \end{pmatrix}.

Symmetry of the Hessians gives (Dφ)⊤JDφ=J(D\varphi)^\top J D\varphi= J for J=(0I−I0)J = \begin{pmatrix} 0 & I \\ -I & 0 \end{pmatrix}, and composition preserves this identity.2 The resulting numerical maps need not conserve the original Hamiltonian exactly.

Momentum retention can be introduced through the linear decay subflow P˙=−γP\dot{P} = -\gamma P, with γ≥0\gamma\ge0. Its exact solution is P←e−γηPP \leftarrow e^{-\gamma\eta}P. Applying decay before the kick gives

{Pk+1=e−γηPk−η∇XVθ(Xk∣T),Xk+1=Xk+η∇PKθ(Pk+1),\begin{cases} P^{k+1} = e^{-\gamma\eta}P^k - \eta\nabla_X V_\theta(X^k \mid T),\\ X^{k+1} = X^k + \eta\nabla_P K_\theta(P^{k+1}), \end{cases}

with γ=0\gamma= 0 recovering the conservative splitting. Applying decay after the kick would also scale the newly added force.

For the corresponding continuous damped system, the energy balance is

ddtHθ,N=−γ⟨P,∇PKθ(P)⟩.\frac{\mathrm{d}}{\mathrm{d}t}H_{\theta,N} = -\gamma\langle P,\nabla_P K_\theta(P)\rangle.

For Kθ(P)=12∥P∥F2K_\theta(P) = \frac{1}{2}\lVert P\rVert_F^2, this becomes −γ∥P∥F2-\gamma\lVert P\rVert_F^2, and the feature dynamics satisfy

X¨+γX˙+∇XVθ(X∣T)=0.\ddot{X} + \gamma\dot{X} + \nabla_X V_\theta(X \mid T) = 0.

For a learned kinetic energy with a bias, the sign of ⟨P,∇PKθ(P)⟩\langle P,\nabla_P K_\theta(P)\rangle is not automatic. Linear momentum decay therefore requires an additional condition to imply energy dissipation.

These calculations concern the reference system with fixed parameters and relations. The learned update in (8) follows the same momentum-first order, with the implementation differences described in the main text. Algorithm 1 gives the complete layer computation.

SE(3)-Equivariance of the Complete Pipeline

The Hamiltonian states XX and PP are invariant latent features, while molecular coordinates are updated separately. We hold atom attributes, pair types, and validity masks fixed under rigid motions. During training, we compare transformations using the same feature dropout masks.

Lemma A.3 (SE(3)-equivariance). The complete coordinate predictor ff, including centering and centroid restoration, satisfies

f(CR⊤+1t⊤)=f(C)R⊤+1t⊤f(CR^\top+ 1t^\top) = f(C)R^\top+ 1t^\top

on valid atoms for every R∈SO(3)R \in\mathrm{SO}(3) and t∈R3t \in\mathbb{R}^3.

Proof. Under ci′=Rci+tc_i' = Rc_i + t, pairwise distances satisfy

∥ci′−cj′∥=∥R(ci−cj)∥=∥ci−cj∥.\lVert c_i' - c_j'\rVert= \lVert R(c_i-c_j)\rVert= \lVert c_i-c_j\rVert.

Relation initialization uses only pair types and these distances, so the initialized relations are unchanged. Atom embeddings and zero initial momentum are unchanged as well. Every HamFormer operation acts on

invariant quantities, so induction over its layers gives identical features, momentum, and relations for the original and transformed inputs. This argument holds for any fixed number of Sinkhorn iterations.

At EGNN layer qq, the current and stage-entry distances remain invariant. The atom features and relation vectors are also invariant, so the pair messages, updated features, and scalar coordinate weights wij(q)w_{ij}^{(q)} are unchanged. Writing ci′(q)=Rci(q)+tc_i^{\prime(q)} = Rc_i^{(q)} + t, the coordinate update gives

ci′(q+1)=Rci(q)+t+∑jwij(q)R(ci(q)−cj(q))∥ci(q)−cj(q)∥+εdist=Rci(q+1)+t.\begin{aligned} c_i^{\prime(q+1)} = Rc_i^{(q)} + t + \sum_j w_{ij}^{(q)} \frac{R(c_i^{(q)} - c_j^{(q)})}{\lVert c_i^{(q)} - c_j^{(q)} \rVert+ \varepsilon_{\mathrm{dist}}} \\ &= Rc_i^{(q+1)} + t. \end{aligned}

Thus the coordinate update is equivariant, as in the EGNN construction (Satorras et al., 2021). Its feature output remains invariant and can enter the next HamFormer stage. Passing momentum through EGNN unchanged preserves its invariance.

For mask mm and N=1⊤mN = \mathbf{1}^{\top}m, the input centroid is cctr=(Cin)⊤m/Nc_{\mathrm{ctr}} = (C^{\mathrm{in}})^{\top}m/N. It transforms as cctr′=Rcctr+tc_{\mathrm{ctr}}^{\prime} = Rc_{\mathrm{ctr}} + t and Cc′=CcR⊤C_c^{\prime} = C_cR^{\top}. The centered network output therefore rotates by RR. Restoring the input centroid gives C^′=C^R⊤+1t⊤\widehat{C}^{\prime} = \widehat{C}R^{\top} + \mathbf{1}t^{\top}, which proves the claim. □

Predicted pairwise distances are consequently invariant. This argument depends on the transformation properties of the network operations and does not require an exact Hamiltonian integrator or exact transport marginals.

HamFormer for Molecular Conformation Optimization

Given atom types a\mathbf{a}, pair types E=(eij)E = (e_{ij}), and input coordinates CinC^{\mathrm{in}}, the atom embedding serves as the encoder and the final EGNN as the SE(3)-equivariant coordinate decoder. The network operates on centered coordinates and restores the input centroid to obtain C^\widehat{C}, with pairwise distances D^\widehat{D}. Algorithms 1 and 2 specify the layer update and pipeline.

Both algorithms update states in place. For a valid-atom mask m∈{0,1}Npadm \in\{0,1\}^{N_{\mathrm{pad}}}, let N=1⊤mN = \mathbf{1}^{\top}m. Masked Sinkhorn targets marginals m/Nm/N, and geometric statistics use only valid atoms. Let LEL_E denote the number of HamFormer–EGNN stage pairs. For l=1,…,LEl = 1,\ldots,L_E, the stage maps HamFormer⁡l\operatorname{HamFormer}_l and EGNN⁡l\operatorname{EGNN}_l apply their layers in order. Each EGNN stage records its incoming coordinates to compute the fixed stage-entry distances. The map InitRelation implements (7).

Algorithm 1 HamFormer layer

▷ Require: XX, PP, TT, mm, parameters θℓ\theta_\ell, iterations II

  1. N←1⊤mN \leftarrow\mathbf{1}^{\top}m

  2. for h=1,…,Hh = 1,\ldots,H do

    1. Zh←LN⁡in,ℓ(X)Wh(ℓ)+1(bh(ℓ))⊤Z_h \leftarrow\operatorname{LN}_{\mathrm{in},\ell}(X)W_h^{(\ell)} + \mathbf{1}(b_h^{(\ell)})^{\top}

    2. S←ZhZh⊤/dh−log⁡(Th+εnum)S \leftarrow Z_hZ_h^{\top}/\sqrt{d_h} - \log(T_h+\varepsilon_{\mathrm{num}})

    3. Th←NSinkhorn⁡εOT,I(S,m)T_h \leftarrow N\operatorname{Sinkhorn}_{\varepsilon_{\mathrm{OT}},I}(S,m)

    4. end for

  3. P←−tanh⁡(αℓ)P+LN⁡P,ℓ(−∑h=1HThZh(Wh(ℓ))⊤)P \leftarrow-\tanh(\alpha_\ell)P + \operatorname{LN}_{P,\ell}\left(-\sum_{h=1}^{H}T_hZ_h(W_h^{(\ell)})^{\top}\right)

  4. X←X+LN⁡X,ℓ[σ0.1(PBℓ+1bℓ⊤)Bℓ⊤]X \leftarrow X + \operatorname{LN}_{X,\ell}\left[\sigma_{0.1}(PB_\ell+\mathbf{1}b_\ell^{\top})B_\ell^{\top}\right]

  5. X←X+FFN⁡ℓ(X)X \leftarrow X + \operatorname{FFN}_\ell(X)

  6. ▷ return XX, PP, TT

Algorithm 2 Conformation refinement

▷ Require: a\mathbf{a}, EE, CinC^{\mathrm{in}}, mm

  1. N←1⊤mN \leftarrow\mathbf{1}^{\top}m, X←Embed⁡(a)X \leftarrow\operatorname{Embed}(\mathbf{a}), P←0P \leftarrow0

  2. T←InitRelation⁡(E,Cin,m)T \leftarrow\operatorname{InitRelation}(E,C^{\mathrm{in}},m)

  3. cctr←(Cin)⊤m/Nc_{\mathrm{ctr}} \leftarrow(C^{\mathrm{in}})^{\top}m/N

  4. C←Cin−1cctr⊤C \leftarrow C^{\mathrm{in}}-\mathbf{1}c_{\mathrm{ctr}}^{\top}

  5. for l=1,…,LEl = 1,\ldots,L_E do

    1. (X,P,T)←HamFormer⁡l(X,P,T,m)(X,P,T) \leftarrow\operatorname{HamFormer}_l(X,P,T,m)

    2. (X,C)←EGNN⁡l(X,C,T,m)(X,C) \leftarrow\operatorname{EGNN}_l(X,C,T,m)

    3. end for

  6. C^←C+1cctr⊤\widehat{C} \leftarrow C+\mathbf{1}c_{\mathrm{ctr}}^{\top} and D^←[∥C^i−C^j∥]ij\widehat{D} \leftarrow[\lVert\widehat{C}_i-\widehat{C}_j\rVert]_{ij}

  7. ▷ return C^\widehat{C}, D^\widehat{D}

For ground-state conformation prediction [28], we minimize distance and aligned coordinate errors relative to target distances DgtD^{\mathrm{gt}} and coordinates CgtC^{\mathrm{gt}}:

Ltrain(θ)=1N2∑(i,j)∈Ppair∣D^ij−Dijgt∣+λNmin⁡R∈SO(3)∥C^c−CcgtR⊤∥F2(10)\mathcal{L}_{\mathrm{train}}(\theta) = \frac{1}{N^2}\sum_{(i,j)\in\mathcal{P}_{\mathrm{pair}}}\left|\widehat{D}_{ij}-D_{ij}^{\mathrm{gt}}\right|+\frac{\lambda}{N}\min_{R\in\mathrm{SO}(3)}\left\|\widehat{C}_c-C_c^{\mathrm{gt}}R^{\top}\right\|_F^2 \tag*{(10)}

where the subscript cc denotes centering over the NN valid atoms, Ipair\mathcal{I}_{\mathrm{pair}} contains all ordered valid atom pairs, and λ≥0\lambda\ge0 weights the coordinate loss. The first term measures absolute distance errors, while the second measures squared coordinate errors after optimal rotation. During training, this rotation is computed from detached predictions and held fixed during backpropagation. Section A.7 details rotational alignment and masked Sinkhorn scaling. All network parameters are trained jointly, with gradients propagated through the relation updates, normalization, and Sinkhorn iterations.

Transport Computation, Training, and Complexity

For the positive optimizer of (4), introduce row and column multipliers ζi\zeta_i and βj\beta_j. Stationarity gives

Sij−εOTlog⁡πij−ζi−βj=0S_{ij} - \varepsilon_{\mathrm{OT}} \log\pi_{ij} - \zeta_i - \beta_j = 0

so the optimizer has the form πij=uiexp⁡(Sij/εOT)vj\pi_{ij} = u_i \exp(S_{ij}/\varepsilon_{\mathrm{OT}})v_j, where ui=e−ζi/εOTu_i = e^{-\zeta_i/\varepsilon_{\mathrm{OT}}} and vj=e−βj/εOTv_j = e^{-\beta_j/\varepsilon_{\mathrm{OT}}}. Sinkhorn scaling alternately adjusts uu and vv to satisfy the row and column constraints [5, 9].

The implementation targets probability marginals m/N\mathbf{m}/N. Write πprob\pi^{\mathrm{prob}} for the resulting matrix, with invalid rows and columns excluded from normalization. On the valid block, the scaling updates are

ui←1/N∑jexp⁡(Sij/εOT)vj,vj←1/N∑iuiexp⁡(Sij/εOT)u_i \leftarrow\frac{1/N}{\sum_j \exp(S_{ij}/\varepsilon_{\mathrm{OT}})v_j}, \qquad v_j \leftarrow\frac{1/N}{\sum_i u_i\exp(S_{ij}/\varepsilon_{\mathrm{OT}})}

The relation update returns NπprobN\pi^{\mathrm{prob}}, targeting unit row and column sums on valid atoms. With finitely many iterations, both sets of constraints need not hold exactly. Alternating row and column normalization also need not preserve symmetry at intermediate iterations.

For the relation-conditioned score in (6),

exp⁡(Sh/εOT)=exp⁡(ZhZh⊤εOTdh)⊙(Th+εnum)−1/εOT,\exp(S_h/\varepsilon_{\mathrm{OT}}) = \exp\left(\frac{Z_h Z_h^{\top}}{\varepsilon_{\mathrm{OT}}\sqrt{d_h}}\right) \odot(T_h+\varepsilon_{\mathrm{num}})^{-1/\varepsilon_{\mathrm{OT}}},

where ⊙\odot and the power act entrywise. At εOT=1\varepsilon_{\mathrm{OT}}=1, the exponentiated feature affinity is divided entrywise by Th+εnumT_h+\varepsilon_{\mathrm{num}} before normalization.

For the coordinate term in (10), both coordinate matrices are restricted to valid atoms and centered before alignment. Expanding the coordinate error gives

∥C^c−CcgtR⊤∥F2=∥C^c∥F2+∥Ccgt∥F2−2tr⁡(R⊤C^c⊤Ccgt).\left\|\widehat{C}_c-C_c^{\mathrm{gt}}R^{\top}\right\|_{F}^{2}=\left\|\widehat{C}_c\right\|_{F}^{2}+\left\|C_c^{\mathrm{gt}}\right\|_{F}^{2}-2\operatorname{tr}\left(R^{\top}\widehat{C}_c^{\top}C_c^{\mathrm{gt}}\right).

The first two terms are independent of RR. Let C^c⊤Ccgt=UsvdΣVsvd⊤\widehat{C}_c^{\top}C_c^{\mathrm{gt}}=U_{\mathrm{svd}}\Sigma V_{\mathrm{svd}}^{\top} be a singular value decomposition with singular values in decreasing order. A minimizing proper rotation is

R⋆=Usvddiag⁡(1,1,det⁡(UsvdVsvd⊤))Vsvd⊤,R^{\star}=U_{\mathrm{svd}}\operatorname{diag}\left(1,1,\det\left(U_{\mathrm{svd}}V_{\mathrm{svd}}^{\top}\right)\right)V_{\mathrm{svd}}^{\top},

where the determinant correction enforces R⋆∈SO(3)R^{\star}\in\mathrm{SO}(3). At points where the minimized coordinate loss is differentiable, for example when the minimizing rotation is unique, the envelope identity gives

∇C^c[λNmin⁡R∈SO(3)∥C^c−CcgtR⊤∥F2]=2λN(C^c−Ccgt(R⋆)⊤).\nabla_{\widehat{C}_c}\left[\frac{\lambda}{N}\min_{R\in\mathrm{SO}(3)}\left\|\widehat{C}_c-C_c^{\mathrm{gt}}R^{\top}\right\|_{F}^{2}\right]=\frac{2\lambda}{N}\left(\widehat{C}_c-C_c^{\mathrm{gt}}(R^{\star})^{\top}\right).

Computing the exact optimal rotation from detached predictions and treating it as fixed therefore yields the gradient of the minimized value at such points. Degenerate configurations can have multiple minimizing rotations, so differentiability is not asserted everywhere.

Remark A.4 (Computational complexity). Consider one dense HamFormer layer with NN valid atoms, feature width dd, head widths dhd_h, FFN width dFFd_{\mathrm{FF}}, and II Sinkhorn iterations. Head projections cost O(Nd∑hdh)O(Nd\sum_h d_h). Forming scores and aggregating transport messages cost O(N2∑hdh)O(N^2\sum_h d_h), and Sinkhorn scaling costs O(IHN2)O(IHN^2). The kinetic map and FFN cost O(Nd2)O(Nd^2) and O(Nd dFF)O(Nd\,d_{\mathrm{FF}}), respectively. The total time complexity is therefore O(Nd∑hdh+N2∑hdh+IHN2+Nd2+Nd dFF)O(Nd\sum_h d_h + N^2\sum_h d_h + IHN^2 + Nd^2 + Nd\,d_{\mathrm{FF}}), which reduces to O(Nd2+N2d+IHN2+Nd dFF)O(Nd^2 + N^2d + IHN^2 + Nd\,d_{\mathrm{FF}}) when ∑hdh=d\sum_h d_h=d.

The relation tensor requires O(HN2)O(HN^2) storage. Including features, projected heads, and FFN activations gives forward activation storage O(HN2+Nd+N∑hdh+NdFF)O(HN^2 + Nd + N\sum_h d_h + Nd_{\mathrm{FF}}). Training requires additional saved intermediates depending on the differentiation and recomputation strategy. For dense padded tensors, replace NN by NpadN_{\mathrm{pad}} unless padded entries are excluded from computation. EGNN cost scales with the number of evaluated atom pairs at fixed MLP widths, and the complete network cost is obtained by summing over its layers.

Experimental Details

Datasets

We use the QM9 splits and the Molecule3D random split datasets, following the benchmark protocol used by WGFormer. The processed QM9 validation and test sets contain 9,900 and 10,697 molecules, respectively. The corresponding Molecule3D sets contain 779,918 and 779,915 molecules.

Evaluation

Following WGFormer, we use the following metrics to evaluate the quality of the predicted molecular structures.

  • D-MAE and D-RMSE: The mean absolute error and root mean square error between the predicted pairwise distances D^ij\hat{D}_{ij} and ground-truth distances DijgtD^{\mathrm{gt}}_{ij}, both computed from the corresponding Cartesian coordinates:

D-MAE=1N2∑(i,j)∈Tpair∣D^ij−Dijgt∣\mathrm{D\text{-}MAE} = \frac{1}{N^2}\sum_{(i,j)\in\mathcal{T}_{\mathrm{pair}}}\left|\hat{D}_{ij}-D^{\mathrm{gt}}_{ij}\right|
D-RMSE=1N2∑(i,j)∈Tpair(D^ij−Dijgt)2\mathrm{D\text{-}RMSE} = \sqrt{\frac{1}{N^2}\sum_{(i,j)\in\mathcal{T}_{\mathrm{pair}}}\left(\hat{D}_{ij}-D^{\mathrm{gt}}_{ij}\right)^2}
  • C-RMSD: The coordinate root mean square deviation between the predicted coordinates C^\hat{C} and ground-truth coordinates CgtC^{\mathrm{gt}} after optimal rigid alignment:

C-RMSD=1Nmin⁡R∈SO(3), t∈R3∥C^−(CgtR⊤+1Nt⊤)∥F2\mathrm{C\text{-}RMSD} = \sqrt{\frac{1}{N}\min_{R\in\mathrm{SO}(3),\,t\in\mathbb{R}^3}\left\lVert\hat{C}-\left(C^{\mathrm{gt}}R^{\top}+1_Nt^{\top}\right)\right\rVert_F^2}

For this metric, hydrogen atoms are removed, NN counts the retained atoms, and CgtC^{\mathrm{gt}} is understood to use the symmetry-equivalent atom ordering that minimizes the aligned RMSD.

Hyperparameters

Table 4 summarizes the full-model training settings used on the QM9 and Molecule3D random datasets, including the peak learning rate, scheduled epochs, warmup fraction, evaluation checkpoint epoch, number of training GPUs, batch size per GPU, global batch size, gradient accumulation steps, and random seed.

SettingQM9Molecule3D random
Peak learning rate8×10−48 \times10^{-4}4×10−44 \times10^{-4}
Scheduled epochs50040
Warmup fraction0.060.05
Evaluated checkpoint epoch29440
Training GPUs82
Batch size per GPU3264
Global batch size256128
Gradient accumulation11
Random seed11

Table 4. Full-model training settings.

Ablation Study on Model Components

To further validate the effectiveness of our proposed design, we perform module ablation studies on the validation and test sets of both QM9 and Molecule3D random split. As shown in Table 5, removing the HamFormer feature block and the P&T branch leads to a prominent performance degradation across all metrics. This significant drop clearly demonstrates that the introduced components are essential for maintaining the model’s high representation capacity and prediction accuracy.

VariantFeature blockP&T branchQM9 D-MAEQM9 D-RMSEQM9 C-RMSDMolecule3D random D-MAEMolecule3D random D-RMSEMolecule3D random C-RMSD
Full modelHamFormer✓0.2180.4190.1980.3870.6560.654
EGNN-only××0.2330.4270.2390.3990.6660.697

Table 5. Module ablation on the test sets. “Feature block” denotes the module applied before each EGNN stage.

Inference-Time Momentum-Retention Ablation

To further validate the importance of the inference-time momentum-retention mechanism, we investigate the effect of the momentum factor on the Molecule3D (random split) dataset. As shown in Table 6, setting the retention factor to zero (ρl=0\rho_l = 0) results in a substantial performance drop across both validation and test sets compared to using the learned momentum parameter (Learned ρl\rho_l). This prominent degradation strongly confirms the crucial role and importance of the momentum-retention mechanism during inference.

Inference settingValidation D-MAEValidation D-RMSEValidation C-RMSDTest D-MAETest D-RMSETest C-RMSD
Learned ρl\rho_l0.3860.6530.6520.3870.6560.654
ρl=0\rho_l = 00.4180.6840.7120.4200.7160.714

Table 6. Inference-time momentum-retention ablation on Molecule3D random.

Layer-wise Energy Evolution on the QM9 Dataset

Figure 5 presents the layer-wise evolution of the latent kinetic energy, potential energy, and total Hamiltonian on the QM9 test set. The energy profiles exhibit clear changes across HamFormer layers, particularly around the transition from Stage 1 to Stage 2. This observation is consistent with the corresponding results on Molecule3D and further suggests that the Hamiltonian dynamics are progressively transformed throughout the network.

Figure 5. Layer-wise evolution of the latent kinetic energy, potential energy, and total Hamiltonian evaluated on 1,000 randomly selected molecules from the QM9 test set.

Additional Visualizations

Visualizations of ground-state conformations predicted by RDKit, WGFormer, and HamFormer

Figure 6. The visualizations for the ground-state conformations predicted by RDKit, WGFormer, and HamFormer.

References

  1. [1]Josh Abramson, Jonas Adler, Jack Dunger, Richard Evans, Tim Green, Alexander Pritzel, Olaf Ronneberger, Lindsay Willmore, Andrew J. Ballard, Joshua Bambrick, et al. Accurate structure prediction of biomolecular interactions with AlphaFold 3. Nature, 630(8016):493–500, 2024.DOI
  2. [2]Shaked Brody, Uri Alon, and Eran Yahav. How Attentive are Graph Attention Networks? In International Conference on Learning Representations, 2022.arxiv.org/abs/2105.14491
  3. [3]Ricky T.Q. Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural ordinary differential equations. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pages 6572–6583, 2018.arxiv.org/abs/1806.07366
  4. [4]Shui-Nee Chow, Wuchen Li, and Haomin Zhou. Wasserstein Hamiltonian flows. Journal of Differential Equations, 268 (3):1205–1219, 2020.arxiv.org/abs/1903.01088
  5. [5]Marco Cuturi. Sinkhorn distances: lightspeed computation of optimal transport. In Proceedings of the 27th International Conference on Neural Information Processing Systems-Volume 2, pages 2292–2300, 2013.
  6. [6]John M. Danskin. The Theory of Max-Min, with Applications. SIAM Journal on Applied Mathematics, 14(4):641–664, 1966.DOI
  7. [7]Lawrence C. Evans. Partial differential equations, volume 19. American Mathematical Society, 2022.
  8. [8]Fabian B. Fuchs, Daniel E. Worrall, Volker Fischer, and Max Welling. SE(3)-Transformers: 3D Roto-Translation Equivariant Attention Networks. In Advances in Neural Information Processing Systems, volume 33, pages 1970–1981, 2020.arxiv.org/abs/2006.10503
  9. [9]Peyré Gabriel and Cuturi Marco. Computational optimal transport: With applications to data science. Foundations and trends® in machine learning, 11(5-6):355–607, 2019.DOI
  10. [10]Jingchu Gai, Nai-Chieh Huang, and Jiayun Wu. Momentum Streams for Optimizer-Inspired Transformers. arXiv preprint arXiv:2605.24425, 2026.arxiv.org/abs/2605.24425
  11. [11]Borjan Geshkovski, Cyril Letrouit, Yury Polyanskiy, and Philippe Rigollet. The emergence of clusters in self-attention dynamics. Advances in Neural Information Processing Systems, 36:57026–57037, 2023.DOI
  12. [12]Borjan Geshkovski, Cyril Letrouit, Yury Polyanskiy, and Philippe Rigollet. A mathematical perspective on Transformers. Bulletin (New Series) of the American Mathematical Society, 62(3):427, 2025.arxiv.org/abs/2312.10794
  13. [13]Samuel Greydanus, Misko Dzamba, and Jason Yosinski. Hamiltonian Neural Networks. In Advances in Neural Information Processing Systems, volume 32, 2019.arxiv.org/abs/1906.01563
  14. [14]Jiaqi Guan, Wesley Wei Qian, Qiang Liu, Wei-Ying Ma, Jianzhu Ma, and Jian Peng. Energy-inspired molecular conformation optimization. In International Conference on Learning Representations, 2022.
  15. [15]Ernst Hairer, Marlis Hochbruck, Arieh Iserles, and Christian Lubich. Geometric numerical integration. Oberwolfach Reports, 3(1):805–882, 2006.
  16. [16]Benjamin Hoover, Yuchen Liang, Bao Pham, Rameswar Panda, Hendrik Strobel, Duen Horng Chau, Mohammed Zaki, and Dmitry Krotov. Energy Transformer. volume 36, pages 27532–27559, 2023.arxiv.org/abs/2302.07253
  17. [17]Weihua Hu, Bowen Liu, Joseph Gomes, Marinka Zitnik, Percy Liang, Vijay Pande, and Jure Leskovec. Strategies for Pre-training Graph Neural Networks. 2019.arxiv.org/abs/1905.12265
  18. [18]John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasu-vunakool, Russ Bates, Augustin Žídek, Anna Potapenko, et al. Highly accurate protein structure prediction with AlphaFold. Nature, 596(7873):583–589, 2021.DOI
  19. [19]Yi-Lun Liao and Tess Smidt. Equiformer: Equivariant Graph Attention Transformer for 3D Atomistic Graphs. In International Conference on Learning Representations, 2023.arxiv.org/abs/2206.11990
  20. [20]Yiping Lu, Zhuohan Li, Di He, Zhiqing Sun, Bin Dong, Tao Qin, Liwei Wang, and Tie-Yan Liu. Understanding and improving transformer from a multi-particle dynamic system point of view. arXiv preprint arXiv:1906.02762, 2019.arxiv.org/abs/1906.02762
  21. [21]Ladislav Rampášek, Mikhail Galkin, Vijay Prakash Dwivedi, Anh Tuan Luu, Guy Wolf, and Dominique Beaini. Recipe for a general, powerful, scalable graph transformer. In Proceedings of the 36th International Conference on Neural Information Processing Systems, pages 14501–14515, 2022.arxiv.org/abs/2205.12454
  22. [22]Michael E. Sander, Pierre Ablin, Mathieu Blondel, and Gabriel Peyré. Momentum residual neural networks. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 9276–9287, 2021.arxiv.org/abs/2102.07870
  23. [23]Michael E. Sander, Pierre Ablin, Mathieu Blondel, and Gabriel Peyré. Sinkformers: Transformers with doubly stochastic attention. In International Conference on Artificial Intelligence and Statistics, pages 3515–3530. PMLR, 2022.arxiv.org/abs/2110.11773
  24. [25]Viktor Stein, Wuchen Li, and Gabriele Steidl. SympFormer: Accelerated attention blocks via inertial dynamics on density manifolds. arXiv preprint arXiv:2603.16535, 2026.arxiv.org/abs/2603.16535
  25. [26]Nathaniel Thomas, Tess Smidt, Steven Kearnes, Lusann Yang, Li Li, Kai Kohlhoff, and Patrick Riley. Tensor field networks: Rotation-and translation-equivariant neural networks for 3D point clouds. arXiv preprint arXiv:1802.08219, 2018.arxiv.org/abs/1802.08219
  26. [27]Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is All you Need. In Advances in Neural Information Processing Systems, volume 30, 2017.arxiv.org/abs/1706.03762
  27. [28]Fanmeng Wang, Minjie Cheng, and Hongteng Xu. WGFormer: An SE(3)-transformer driven by Wasserstein gradient flows for molecular ground-state conformation prediction. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pages 62652–62671. PMLR, 13–19 Jul 2025.arxiv.org/abs/2410.09795
  28. [29]Hao Wu, Shu Liu, Xiaojing Ye, and Haomin Zhou. Parameterized Wasserstein Hamiltonian flow. SIAM Journal on Numerical Analysis, 63(1):360–395, 2025.arxiv.org/abs/2306.00191
  29. [30]Guikun Xu, Yongquan Jiang, PengChuan Lei, Yan Yang, and Jim Chen. GTMGC: Using graph Transformer to predict molecule’s ground-state conformation. In International Conference on Learning Representations, volume 2024, pages 47712–47731, 2024.
  30. [31]Zhao Xu, Youzhi Luo, Xuan Zhang, Xinyi Xu, Yaochen Xie, Meng Liu, Kaleb Dickerson, Cheng Deng, Maho Nakata, and Shuiwang Ji. Molecule3D: A Benchmark for Predicting 3D Geometries from Molecular Graphs. arXiv preprint arXiv:2110.01717, 2021.arxiv.org/abs/2110.01717
  31. [32]Yongyi Yang and David P. Wipf. Transformers from an Optimization Perspective. volume 35, pages 36958–36971, 2022.arxiv.org/abs/2205.13891
  32. [33]Gengmo Zhou, Zhifeng Gao, Qiankun Ding, Hang Zheng, Hongteng Xu, Zhewei Wei, Linfeng Zhang, and Guolin Ke. Uni-Mol: A Universal 3D Molecular Representation Learning Framework. In The Eleventh International Conference on Learning Representations, 2023.DOI
  33. [34]Aleksandr Zimin, Yury Polyanskiy, and Philippe Rigollet. YuriiFormer: A suite of Nesterov-Accelerated Transformers. arXiv preprint arXiv:2601.23236, 2026.arxiv.org/abs/2601.23236

Paper details

vlnc:W_cudnzqq5
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents