betaFeedback

WorksComputer Science

ADT: All-Atom Diffusion Tokenizer for 3D Molecular Structures

Abstract

Discrete structural tokenization provides a promising route to autoregressive 3D molecular generation. Yet, existing tokenizers face a trade-off between compact representation and all-atom reconstruction: atom-level tokens preserve atomic details but produce long sequences, while compact residue-level protein tokens typically omit side-chain geometry. To overcome this limitation, we propose All-atom Diffusion Tokenizer (ADT), which decouples token granularity from reconstruction resolution. ADT uses one structural token per atom for small molecules and one token per protein residue that jointly represents backbone and side-chain geometry. To mitigate geometric distortion introduced by quantization, we leverage the generative capacity of diffusion models to recover fine-grained atomic geometry beyond deterministic decoding from discrete tokens. Extensive experiments across small molecules and proteins demonstrate that ADT achieves strong reconstruction from compact discrete representations while enabling high-quality autoregressive generation of both small molecules and all-atom protein structures. These results establish compact structural tokens as an effective discrete interface for all-atom molecular reconstruction and generation.

Introduction

Generating three-dimensional molecular structures is a central task in drug discovery and protein engineering [43, 34, 36, 20]. Molecular interactions depend critically on atomic geometry, from small-molecule conformations to protein backbones and side chains [23, 21]. Many existing approaches directly model molecular structures in continuous coordinate space [19, 49, 6, 38]. While effective, continuous coordinate representations do not naturally fit the autoregressive framework of large language models. Structural tokenization instead maps 3D geometry into discrete representations, enabling molecular structures to be generated autoregressively as token sequences and decoded into atomic coordinates. This formulation allows molecular structure generation to leverage scalable language-modeling paradigms demonstrated across language and biomolecular modeling [25, 18, 1].

Recent molecular tokenizers encode geometry at different granularities, as shown in Figure 1. Atom-level tokenizers [35, 14] assign structural codes to individual atoms, preserving all-atom coverage with sequence lengths that scale with atom count. For proteins, this yields substantially longer sequences than residue-level representations and requires more autoregressive prediction steps. Backbone-based protein tokenizers [47, 16, 9] use compact residue-level codes but do not explicitly encode side-chain conformations, limiting faithful reconstruction of the original all-atom structure. Recent approaches further incorporate generative decoders to improve backbone reconstruction [5] and enable adaptive compression of CαC\alpha structures [4].

Figure 1. Illustration of tokenizer granularity. (a) Atom-level tokenization assigns one token to each atom. (b) Backbone-only tokenization omits side-chain geometry. (c) ADT jointly encodes backbone and side-chain geometry into one token per residue and reconstructs the all-atom structure.

These limitations raise two challenges for compact all-atom tokenization. First, compact all-atom encoding: side-chain conformations are essential for describing molecular interactions, yet encoding every atom separately produces long token sequences for proteins. A useful representation must retain these atomic details while controlling sequence length. Second, accurate reconstruction under quantization error: mapping continuous conformations to discrete codes loses geometric information, making it difficult to recover precise local geometry and overall molecular structure.

We address these challenges by introducing All-atom Diffusion Tokenizer (ADT), which decouples token granularity from reconstruction resolution. For proteins, we jointly encode backbone and side-chain geometry into one structural token per residue. To mitigate geometric distortion introduced by quantization, we formulate reconstruction as conditional generative modeling, exploiting the generative capacity of a diffusion decoder to recover fine-grained atomic geometry from discrete structural tokens and chemical identities.

Specifically, ADT encodes all-atom geometry into one structural token per chemical unit: an atom for small molecules and a residue for proteins. To make these tokens better suited for generative modeling, we introduce a bitwise mutual-information regularizer that promotes effective code utilization. Conditioned on these tokens and their chemical identities, a diffusion decoder reconstructs atomic geometry from the discrete bottleneck. For proteins, the decoder first models residue-level context and then lifts each residue representation to atom-level features, enabling all-atom reconstruction from a single token per residue. Beyond reconstruction, we use the learned tokens for autoregressive molecular generation, where the chemical identity sequence is generated first, followed by the corresponding structural tokens conditioned on the complete chemical sequence. Together, these designs enable compact molecular tokens to support all-atom reconstruction and generation without tying token granularity to atomic resolution.

We comprehensively evaluate our framework on reconstruction and generation across small molecules and proteins. Our experiments show that ADT achieves accurate reconstruction from compact discrete representations, including all-atom protein reconstruction from a single structural token per residue. Beyond reconstruction, our experiments demonstrate that the learned tokens enable high-quality autoregressive generation of both small molecules and all-atom protein structures. Our main contributions are summarized as follows: 1) We introduce a unified discrete representation for small molecules and proteins that decouples token granularity from reconstruction resolution, using one structural token per atom for small molecules and per residue for proteins. 2) We formulate reconstruction as conditional generation and introduce a coarse-to-fine diffusion decoder that recovers fine-grained atomic geometry from discrete structural tokens. 3) Our experiments demonstrate that compact structural tokens support reconstruction and autoregressive generation of both small molecules and all-atom protein structures.

Related Work

Molecular Tokenization

Molecular geometry can be serialized through coordinate strings or invariant geometric sequences [7, 52, 32], while learned tokenizers compress continuous geometry into discrete structural codes. Such representations have been explored for small molecules [8, 14] and protein structures, including VQ-based backbone tokenization [10], ESM3 [16], DPLM-2 [47], and ProToken [33]. Recent methods further extend structural coverage beyond protein backbones: Bio2Token uses atom-level quantized representations across biomolecular systems [35], while BiHiTo introduces hierarchical structural codes [51]. At residue granularity, SLAE learns compact local all-atom representations [2], and related backmapping methods reconstruct atomistic structures from compressed representations [15]. We instead formulate molecular tokenization over chemical units, pairing each atom or residue identity with a discrete structural code and decoupling token granularity from all-atom reconstruction resolution, thereby providing a common discrete interface across molecular systems.

3D Molecular Generation

3D molecular generation has been explored in both continuous coordinate space and discrete token space. Early autoregressive approaches construct molecular structures sequentially by predicting atom types and coordinates [11, 12], while diffusion and flow models directly generate molecular geometry for small molecules [19, 39, 6] and protein structures [50, 48, 3, 13, 31]. More recently, discrete structural representations bridge 3D geometry and language modeling by converting molecular geometry into token sequences for autoregressive generation. This paradigm has been explored for small molecules [8] and proteins [10, 16, 47, 5 a]. However, discrete protein generation largely operates at Cα or backbone resolution, omitting side-chain and local atomic details needed to represent interaction interfaces and functional sites [3]. This limits its applicability to structure-based design tasks that require atomic-level detail. We address this limitation by combining compact autoregressive structural tokens with generative all-atom decoding.

Preliminaries

3D Molecular Representation. A 3D molecular structure is represented as a sequence of LL structural units, M=(U1,…,UL)\mathcal{M} = (U_1, \ldots, U_L), where each structural unit UiU_i contains a set of atoms indexed by Ai\mathcal{A}_i, with Cartesian coordinates xi,j∈R3\mathbf{x}_{i,j} \in\mathbb{R}^3 for j∈Aij \in\mathcal{A}_i. For small molecules, each structural unit corresponds to a single atom, whereas for proteins, each structural unit corresponds to a residue containing multiple atoms. We denote the chemical identities of the structural units by C=(c1,…,cL)C = (c_1, \ldots, c_L), corresponding to atom types for small molecules and amino-acid types for proteins. The atomic geometry is represented as X∈RL×A×3\mathbf{X} \in\mathbb{R}^{L \times A \times3}, where AA denotes the number of atomic identities per structural unit.

Discrete Molecular Tokenization. Given a molecular structure M\mathcal{M}, a discrete molecular tokenizer maps it into continuous latent representations through an encoder fθf_\theta, followed by a quantization function QQ that converts the latent representations into a sequence of discrete tokens:

H=fθ(M),S=Q(H)=(s1,…,sL),si∈{0,…,K−1},(1)\mathbf{H} = f_\theta(\mathcal{M}), \qquad\mathbf{S} = Q(\mathbf{H}) = (s_1, \ldots, s_L), \qquad s_i \in\{0, \ldots, K-1\}, \tag*{(1)}

where KK denotes the size of the discrete vocabulary. A decoder gϕg_\phi reconstructs the atomic geometry from the discrete structural tokens S\mathbf{S}, conditioned on the structural-unit identities CC.

Methodology

As shown in Figure 2, we introduce All-atom Diffusion Tokenizer (ADT), which decouples token granularity from reconstruction resolution. Each small-molecule atom and protein residue is represented by a single structural token, while a diffusion decoder mitigates quantization-induced distortion to recover fine-grained all-atom geometry. This design provides a common discrete interface across molecular modalities while retaining all-atom reconstruction.

Figure 2. Overview of ADT. (a) All-atom structures are represented at chemical-unit granularity. (b) Each unit is mapped to a discrete structural token via FSQ. (c) Conditional diffusion reconstructs all-atom geometry, with residue-to-atom expansion for proteins. (d) ADT enables autoregressive generation of all-atom molecular structures.

Unified Molecular Structure Tokenization

Unified Structure Representation. Small molecules and proteins exhibit different structural granularities, with atom-level units for small molecules and residue-level units for proteins. Following the structural-unit representation defined above, we jointly embed the chemical identity and atomic geometry of each structural unit into a common sequence space:

H0=MLP⁡([X;M])+E(C),H=TransformerEnc⁡(H0)∈RL×d.(2)\mathbf{H}^{0} = \operatorname{MLP}([\mathbf{X};\mathbf{M}]) + \mathbf{E}(C), \qquad\mathbf{H} = \operatorname{TransformerEnc}(\mathbf{H}^{0}) \in\mathbb{R}^{L \times d}. \tag*{(2)}

Here, M\mathbf{M} is a binary mask indicating valid atoms, and E(C)\mathbf{E}(\mathbf{C}) denotes the structural-unit identity embeddings, corresponding to atom types for small molecules and amino-acid types for proteins. This formulation yields one contextual representation per structural unit while preserving its underlying atomic geometry.

Discrete Molecular Tokenization. We further map the unified continuous representations into a common discrete space using finite scalar quantization (FSQ) [37]. Specifically, given the contextual representation H\mathbf{H}, we project each structural unit into an mm-dimensional quantization space, independently quantize each dimension into two symmetric levels, and map the resulting binary code to a discrete token:

Z=MLP⁡(H)∈RL×m,z^i,j=Q(zi,j)∈{−1,+1},si=∑j=1mz^i,j+122j−1.(3)\mathbf{Z} = \operatorname{MLP}(\mathbf{H}) \in\mathbb{R}^{L \times m}, \qquad\hat{z}_{i,j} = Q(z_{i,j}) \in\{-1,+1\}, \qquad s_i = \sum_{j=1}^{m} \frac{\hat{z}_{i,j}+1}{2}2^{j-1}. \tag*{(3)}

This yields a structural token sequence S=(s1,…,sL)\mathbf{S}=(s_1,\ldots,s_L) over the same vocabulary of size K=2mK=2^m, providing a common discrete representation for structural units across molecular modalities.

Molecular Structure Reconstruction

Coarse-to-Fine Diffusion Reconstruction. We reconstruct all-atom geometry from discrete structural tokens through coarse-to-fine diffusion decoding. The decoder progressively lifts token-level representations to atom-level representations, decoupling token granularity from reconstruction resolution. Given the discrete structural tokens S\mathbf{S}, we recover their quantized representations Z^\hat{\mathbf{Z}} and combine them with the corresponding chemical identity embeddings to form the conditioning representations:

D=MLP⁡(Z^)+Edec(C)∈RL×d,Pt=D(Xt;D,t)∈RL×d,(4)\mathbf{D} = \operatorname{MLP}(\hat{\mathbf{Z}}) + \mathbf{E}_{\mathrm{dec}}(\mathbf{C}) \in\mathbb{R}^{L \times d}, \qquad\mathbf{P}_t = \mathcal{D}(\mathbf{X}_t;\mathbf{D},t) \in\mathbb{R}^{L \times d}, \tag*{(4)}

where Edec\mathbf{E}_{\mathrm{dec}} is a learnable decoder-side embedding of the chemical identities C\mathbf{C}, and D\mathbf{D} denotes the resulting conditioning representations. Xt\mathbf{X}_t denotes the noisy molecular coordinates at diffusion time tt, which are projected into structural-unit representations and processed by the diffusion Transformer D\mathcal{D}. The resulting Pt\mathbf{P}_t contains one contextual representation for each structural unit.

Since each protein token represents an entire residue, directly decoding all atoms from a single residue-level representation creates a bottleneck for fine-grained atomic reconstruction. We therefore adopt a coarse-to-fine strategy that expands each intermediate residue representation into AA atom-wise representations, conditioned on learnable atom embeddings and noisy coordinates:

Ut,i,a0=WPt,i+ea+MLP⁡1(xt,i,a),\mathbf{U}^{0}_{t,i,a} = \mathbf{W}\mathbf{P}_{t,i} + \mathbf{e}_a + \operatorname{MLP}_1(\mathbf{x}_{t,i,a}),
Ut,i=Transformer⁡(Ut,i0),v^t,i,a=MLP⁡2(Ut,i,a)∈R3.(5)\mathbf{U}_{t,i} = \operatorname{Transformer}(\mathbf{U}^{0}_{t,i}), \qquad\hat{\mathbf{v}}_{t,i,a} = \operatorname{MLP}_2(\mathbf{U}_{t,i,a}) \in\mathbb{R}^{3}. \tag*{(5)}

Here, ea\mathbf{e}_a is a learnable embedding that distinguishes individual atoms within each residue. The remaining decoder layers refine the expanded atom-wise representations, which are finally projected to atom-wise denoising velocities. This transition from RL×d\mathbb{R}^{L \times d} to RL×A×d\mathbb{R}^{L \times A \times d} decouples residue-level tokenization from all-atom reconstruction resolution.

Reconstruction Objective. We train each diffusion decoder using a velocity-prediction objective. Conditioned on the discrete structural representation, the diffusion decoder predicts the velocity of each atom from its noisy coordinates. For a molecule with NN atoms, the objective is:

Lrecon=Et,ϵ[1N∑i=1N∥v^t,i−vt,i∥22],(6)\mathcal{L}_{\mathrm{recon}} = \mathbb{E}_{t,\epsilon}\left[\frac{1}{N}\sum_{i=1}^{N}\left\lVert\hat{\mathbf{v}}_{t,i}-\mathbf{v}_{t,i}\right\rVert_{2}^{2}\right], \tag*{(6)}

where v^t,i\hat{\mathbf{v}}_{t,i} and vt,i\mathbf{v}_{t,i} denote the predicted and target velocities of the ii-th atom, respectively. This objective propagates atom-level reconstruction supervision through the discrete bottleneck, encouraging each token to retain the structural information required for recovering molecular geometry.

Mutual-Information Regularization. High reconstruction fidelity alone does not guarantee effective generative modeling in the discrete space. We therefore introduce a bitwise mutual-information regularizer to encourage diverse code utilization across structural units while maintaining confident individual assignments. For the jj-th pre-quantization scalar zi,jz_{i,j} of structural unit UiU_i, we define the soft binary assignment:

pi,j=q(Bj=1∣hi)=exp⁡(zi,j/τ)exp⁡(zi,j/τ)+exp⁡(−zi,j/τ),(7)p_{i,j} = q(B_j = 1 \mid\mathbf{h}_i) = \frac{\exp(z_{i,j}/\tau)}{\exp(z_{i,j}/\tau) + \exp(-z_{i,j}/\tau)}, \tag*{(7)}

where BjB_j denotes the jj-th binary quantization variable and τ\tau is the temperature. Using the binary Shannon entropy h(p)=−plog⁡p−(1−p)log⁡(1−p)h(p) = -p\log p - (1-p)\log(1-p), we estimate the normalized bitwise mutual information over all valid structural units in each mini-batch as:

LMI=−1mlog⁡2∑j=1m[h(Ei[pi,j])−Ei[h(pi,j)]].(8)\mathcal{L}_{\mathrm{MI}} = -\frac{1}{m\log2}\sum_{j=1}^{m}\left[h\left(\mathbb{E}_i[p_{i,j}]\right)-\mathbb{E}_i[h(p_{i,j})]\right]. \tag*{(8)}

The first term encourages diverse bit utilization across structural units, while the second encourages confident assignments for individual units. The resulting LMI\mathcal{L}_{\mathrm{MI}} is used as a regularization term during tokenizer training.

Protein Geometry Regularization. For proteins, we complement the reconstruction objective with a backbone loss Lbb\mathcal{L}_{\mathrm{bb}} and auxiliary geometric losses on bond lengths, bond angles, steric clashes, and Cα\alpha pairwise distances. The resulting training objective is:

Lprot=Lrecon+λMILMI+ωT[Lbb,Lbond,Langle,Lclash,Ldist]T.(9)\mathcal{L}_{\mathrm{prot}} = \mathcal{L}_{\mathrm{recon}} + \lambda_{\mathrm{MI}}\mathcal{L}_{\mathrm{MI}} + \boldsymbol{\omega}^{\mathsf{T}}\left[\mathcal{L}_{\mathrm{bb}},\mathcal{L}_{\mathrm{bond}},\mathcal{L}_{\mathrm{angle}},\mathcal{L}_{\mathrm{clash}},\mathcal{L}_{\mathrm{dist}}\right]^{\mathsf{T}}. \tag*{(9)}

where ω∈R5\boldsymbol{\omega} \in\mathbb{R}^{5} balances the five auxiliary terms. For small-molecule tokenizer, the training objective is Lrecon+λMILMI\mathcal{L}_{\mathrm{recon}} + \lambda_{\mathrm{MI}}\mathcal{L}_{\mathrm{MI}}. Detailed definitions are provided in Appendix § 10.§ 2.

Autoregressive Molecule Generation

Discrete structural tokens allow both chemical identity and 3D geometry to be generated within the same autoregressive sequence-modeling paradigm. Building on this representation, we factorize molecular generation into chemical sequence generation followed by structural token generation conditioned on the complete chemical sequence. Let C=(c1,…,cL,EOS)\mathbf{C} = (c_1,\ldots,c_L,\mathrm{EOS}) and S=(s1,…,sL)\mathbf{S} = (s_1,\ldots,s_L) denote the chemical and structural sequences, respectively. We factorize their joint distribution as:

pθ(C,S)=pθ(C)pθ(S∣C)=[∏i=1L+1pθ(ci∣C<i)][∏i=1Lpθ(si∣C,S<i)].(10)p_{\theta}(\mathbf{C},\mathbf{S}) = p_{\theta}(\mathbf{C})p_{\theta}(\mathbf{S}\mid\mathbf{C}) = \left[\prod_{i=1}^{L+1}p_{\theta}(c_i\mid\mathbf{C}_{<i})\right]\left[\prod_{i=1}^{L}p_{\theta}(s_i\mid\mathbf{C},\mathbf{S}_{<i})\right]. \tag*{(10)}

Thus, each structural prediction has access to chemical information from all molecular positions. Both autoregressive processes are trained jointly with teacher forcing:

LAR=−∑i=1L+1log⁡pθ(ci∣C<i)−β∑i=1Llog⁡pθ(si∣C,S<i).(11)\mathcal{L}_{\mathrm{AR}} = - \sum_{i=1}^{L+1} \log p_{\theta}(c_i \mid\mathbf{C}_{<i}) - \beta\sum_{i=1}^{L} \log p_{\theta}(s_i \mid\mathbf{C}, \mathbf{S}_{<i}). \tag*{(11)}

where β\beta balances the chemical and structural autoregressive objectives.

At inference, chemical tokens are sampled until EOS, determining the sequence length LL. The model then generates one structural token for each chemical position, and the molecular decoder converts (S,C)(\mathbf{S}, \mathbf{C}) into 3D coordinates X\mathbf{X}. For structure generation from a given chemical sequence, C\mathbf{C} is fixed and only the structural autoregression is performed.

Experiments

We evaluate ADT on both small molecules and proteins, covering 3D structure reconstruction and autoregressive generation. We report reconstruction and generation results, followed by ablation studies on the key design choices of ADT.

Experiment Setup

Datasets. We conduct experiments on benchmarks spanning small molecules and proteins. For small molecules, we use QM9 (Ramakrishnan et al. [40]), containing approximately 130K organic molecules with up to nine heavy atoms, and ∇2\nabla^2DFT (Khrabrov et al. [26]), using 1.9M molecules for training and the three test splits for evaluation on unseen molecules with 8–27 heavy atoms. For proteins, we use AFDB SwissProt (Varadi et al. [46]), with CASP14 (Kryshtafovych et al. [27])/CASP15 (Kryshtafovych et al. [28]) held out for protein reconstruction evaluation. We exclude all evaluation targets from the tokenizer training set and remove training proteins with at least 80% sequence identity to evaluation target, as measured over Foldseek (Van Kempen et al. [45]). Further details are provided in Appendix § 10.

Evaluation Metrics. Small-molecule reconstruction is evaluated using structure match, validity, and RMSD. For protein reconstruction, we report Cα\alpha-lDDT, backbone and all-atom RMSD, all-atom lDDT, TM-score, GDT-TS, and GDT-HA. Small-molecule generation is evaluated using validity, uniqueness, diversity, and PoseBusters validity, while protein generation metrics are specified in the corresponding experiments. Detailed definitions and evaluation protocols are provided in Appendix § 10.§ 4.

Training and Evaluation. We use a codebook size of 8,192 and Transformer-based encoders with diffusion decoders for both molecular modalities. The tokenizers are optimized with AdamW with learning rate 3×10−43 \times10^{-4} on a single NVIDIA H200 GPU. Detailed model architectures, optimization settings, loss weights, and evaluation protocols are provided in Appendix § 10.§ 1.

Reconstruction on small molecules and all-atom proteins

Reconstruction on small molecules. We evaluate molecular reconstruction on QM9 and the three ∇2\nabla^2DFT test splits, covering both in-distribution molecules and generalization to unseen structures and scaffolds. Tables 1 and 2 reveal three main results: 1) Our method consistently outperforms existing discrete tokenizers. Compared with CoordToken, it reduces reconstruction RMSD by 28.2% on QM9 and 23.8–25.6% across the three ∇2\nabla^2DFT test splits, while achieving a higher match rate on QM9 and maintaining 100% validity on ∇2\nabla^2DFT. 2) Discrete tokenization does not preclude high reconstruction accuracy. On QM9, our method surpasses the continuous GeoLDM and ADiT baselines in both match rate and RMSD. 3) The advantage extends beyond QM9 to larger and more diverse molecules. Across the ∇2\nabla^2DFT test splits, including unseen structures and out-of-distribution

TypeMethodMatch↑RMSD↓
ContinuousGeoLDM (Xu et al. [49])87.20.1820
ContinuousADiT (Joshi et al. [24])94.60.0890
ContinuousSinAE (Ren et al. [41])100.00.0007
DiscreteMol-StrucTok (Gao et al. [8])90.70.1280
DiscreteCoordToken Geikyan et al. [14]99.40.0478
DiscreteADT (Ours)99.70.0343

Table 1. Reconstruction results on QM9.

scaffolds, ADT reduces RMSD over existing discrete tokenizers by 23.8–25.6% while maintaining complete validity.

MethodConformer RMSD ↓Conformer Valid ↑Structure RMSD ↓Structure Valid ↑Scaffolds RMSD ↓Scaffolds Valid ↑
mol2token (Liu et al. [35])0.201047.10.201047.10.201047.1
CoordToken Geikyan et al. [14]0.0671100.00.0673100.00.0671100.0
ADT (Ours)0.0499100.00.0512100.00.0511100.0

Table 2. Reconstruction performance on the ∇2\nabla^{2}DFT benchmark.

All-atom protein reconstruction. Table 3 shows that residue-level tokenization can enable accurate all-atom reconstruction while preserving backbone fidelity. Since DPLM-2 reconstructs only protein backbones, we recover all atoms from the reconstructed backbone using PDBFixer, cg2all (Heo & Feig, [17]), and Rosetta (Leaver-Fay et al., [30]). We additionally evaluate cg2all with the input backbone fixed, denoted as cg2all (fixed). Compared with DPLM-2 followed by all-atom completion, our method substantially reduces all-atom and backbone RMSD on CASP14 and CASP15, while maintaining strong backbone lDDT. Notably, each residue is represented by only a single structural token yet decoded into its full atomic geometry. These results demonstrate that compact token granularity need not constrain reconstruction resolution, supporting our design of decoupling residue-level encoding from all-atom decoding.

MethodCαC_{\alpha}-lDDT ↑BB RMSD ↓AA lDDT ↑AA RMSD ↓TM-score ↑GDT-TS ↑GDT-HA ↑
CASP14
DPLM-2 (Wang et al. [47])0.88202.8576––0.88600.77180.5724
+ Rosetta fixbb0.88202.85760.75323.49180.88590.77170.5723
+ PDBFixer0.88202.85760.66323.96570.88590.77170.5723
+ cg2all (nofix)0.88202.80930.77103.41200.88590.77180.5722
+ cg2all (fixed)0.88192.85760.76113.43580.88590.77170.5723
ADT (Ours)0.89700.96280.82231.31570.96950.93980.7686
CASP15
DPLM-2 (Wang et al. [47])0.89334.0223––0.86440.70320.5104
+ Rosetta fixbb0.89334.02230.76714.59310.86440.70320.5104
+ PDBFixer0.89334.02230.67675.02130.86440.70320.5104
+ cg2all (nofix)0.89333.97960.78794.50370.86440.70290.5101
+ cg2all (fixed)0.89334.02240.77724.52880.86440.70320.5104
ADT (Ours)0.89721.51910.82581.82610.94700.90010.7285

Table 3. Protein reconstruction performance on CASP14 and CASP15. DPLM-2 variants recover all-atom structures using different backbone-to-all-atom reconstruction methods.

Generation on small molecules and all-atom proteins

Generation on small molecules. We evaluate unconditional molecular generation on QM9. As shown in Table 4, ADT outperforms the evaluated discrete baselines across all four metrics, improving Valid and PB Valid over Mol-StrucTok from 0.97/0.88 to 1.00/0.94. ADT is also competitive with continuous generative models, matching the best PB Valid while maintaining high uniqueness and diversity. Notably, SinAE achieves substantially lower reconstruction error but comparable generation performance, indicating that reconstruction fidelity alone does not determine generative quality. Overall, these results demonstrate that ADT provides a strong discrete representation for autoregressive 3D molecular generation.

TypeMethodValid ↑Unique ↑Div. ↑PB Valid ↑
ContinuousEQGAT-diff [29]0.991.000.890.94
SemlaFlow [22]0.990.950.890.94
FlowMol [6]0.970.970.920.94
ADiT [24]0.970.970.900.92
SinAE [41]0.970.970.910.94
DiscreteMol-StrucTok [8]0.970.850.870.88
CoordToken Geikyan et al. [14]0.950.930.900.86
ADT (Ours)1.000.960.910.94

Table 4. Unconditional molecular generation results on QM9.

Sequence-Conditioned Protein Generation. We train Qwen3.5-0.8B (Team, [44]) on tokenized AFDB structures for sequence-conditioned protein structure generation, with training details provided in Appendix § 10.§ 3. To isolate the effect of structural tokenization, we use the same Qwen3.5-0.8B backbone with either the DPLM-2 tokenizer or ADT, while keeping all other components fixed. Given an amino-acid sequence, the model autoregressively predicts one structural token per residue. ADT directly decodes these tokens into all-atom coordinates using the frozen diffusion decoder, whereas DPLM-2 outputs are further converted to all-atom structures using the side-chain reconstruction methods described in § 5.2.

We evaluate on 183 CAMEO22 targets using sequence-length matching, Cα-lDDT, backbone lDDT, TM-score, and all-atom lDDT. As shown in Table 5, ADT achieves an all-atom lDDT of 0.543, outperforming all post-processed DPLM-2 variants, including cg2all (nofix) at 0.503. ADT also improves Cα\alpha-lDDT from 0.5607 to 0.5740 and TM-score from 0.633 to 0.665, while maintaining exact sequence-length matching. These results show that a single residue-level structural token can jointly represent backbone and side-chain geometry, enabling direct all-atom generation without a separate side-chain reconstruction stage.

MethodLength MatchCα\alpha-lDDTBB-lDDTTM-scoreAA-lDDT
DPLM-21.00.56070.56830.633–
+ Rosetta fixbb1.00.56070.56830.6330.482
+ PDBFixer1.00.56070.56830.6330.372
+ cg2all nofix1.00.56070.56830.6330.503
+ cg2all fixed1.00.56070.56830.6330.496
ADT (Ours)1.00.57400.58010.6650.543

Table 5. Sequence-conditioned protein structure generation on CAMEO22.

Effect of All-Atom Joint Encoding

We ablate how backbone and side-chain geometry should be represented by comparing three encoding strategies under the same residue-level tokenization framework: encoding backbone geometry only, encoding backbone and side chains separately (Separate BB+SC), and jointly encoding all-atom geometry into a single structural token per residue. Separate BB+SC independently encodes backbone and side-chain geometry into two residue-level structural tokens, whereas our joint variant represents both with a single token. All variants use the same model architecture and training protocol, differing only in the structural information encoded by the discrete representation. Table 6 reveals three key observations: 1) Geometric Complementarity: Joint all-atom encoding consistently outperforms backbone-only encoding across all metrics, while achieving comparable side-chain reconstruction to separate encoding and stronger overall backbone fidelity, revealing complementary geometric information between backbone and side chains. 2) Improved Structural Fidelity: Compared with separate encoding, joint encoding better preserves global structural fidelity, as reflected by consistently higher Cα\alpha-lDDT, TM-score, GDT-TS, and GDT-HA. 3) Compact Representation: A single residue-level discrete token enables high-fidelity all-atom reconstruction, decoupling token granularity from reconstruction resolution.

MethodCα\alpha-lDDT ↑\uparrowBB RMSD ↓\downarrowAA lDDT ↑\uparrowAA RMSD ↓\downarrowSC RMSD ↓\downarrowTM-score ↑\uparrowGDT-TS ↑\uparrowGDT-HA ↑\uparrow
CASP14
Backbone only0.88021.28040.76991.92692.38580.95550.90380.7241
Separate BB+SC0.87621.31050.82621.50751.67630.95490.91060.7238
Joint All-atom (ADT)0.89700.96280.82231.31571.59380.96950.93980.7686
CASP15
Backbone only0.88641.65170.78662.23862.66720.93650.88750.7150
Separate BB+SC0.88271.48840.83061.77112.03410.94160.88960.7065
Joint All-atom (ADT)0.89721.51910.82581.82612.07790.94700.90010.7285

Table 6. Ablation of protein structural encoding strategies on CASP14 and CASP15.

Analysis and Ablation Studies

Effect of Bitwise Regularization. We investigate the effect of bitwise regularization on reconstruction and downstream generation. As shown in Figure 3, mild regularization improves PB Valid, suggesting that shaping the discrete representation with the bitwise mutual-information objective facilitates downstream autoregressive modeling. Both PB Valid and reconstruction RMSD remain relatively stable across a broad range of λ\lambda, indicating limited sensitivity to the exact regularization strength. At larger λ\lambda, reconstruction RMSD gradually increases without further improvements in PB Valid, motivating the use of moderate regularization.

Effect of MI regularization on reconstruction and generation

Figure 3. Effect of MI regularization on reconstruction and generation.

Ablating Diffusion Reconstruction. To isolate the contribution of generative decoding, we replace the conditional diffusion decoder with a Transformer that directly predicts atomic coordinates, while keeping the remaining architectural and training settings unchanged. As shown in Table 7, diffusion decoding consistently outperforms the direct Transformer decoder across different depths, with the improvement increasing as the number of decoder layers grows. This trend further supports the effectiveness of diffusion decoding in mitigating quantization-induced geometric distortion.

DecoderLayersMatch ↑\uparrowRMSD ↓\downarrowPB Valid ↑\uparrowRMSD Gain ↑\uparrow
Transformer497.50.06950.88
Diffusion498.30.05060.8927.19%
Transformer898.40.06360.90
Diffusion899.10.04250.9133.18%
Transformer1298.90.05840.92
Diffusion1299.70.03430.9341.27%

Table 7. Effect of decoder depth on QM9 reconstruction.

We further stratify the QM9 test set by the number of heavy atoms and rotatable bonds to examine how this benefit scales with molecular complexity, as illustrated in Figure 4. Diffusion decoding yields larger absolute reductions in reconstruction error as complexity increases, while maintaining broadly comparable relative improvements. The growing absolute advantage on more challenging molecules suggests that generative decoding becomes increasingly valuable as the geometric information discarded by discrete quantization becomes harder to recover.

Reconstruction gains from generative decoding across molecular complexity

Figure 4. Reconstruction gains from generative decoding across molecular complexity.

Conclusion

We introduced All-atom Diffusion Tokenizer (ADT), a discrete tokenizer for 3D molecular structures that decouples token granularity from reconstruction resolution. ADT represents small molecules with one structural token per atom and proteins with one token per residue, while retaining all-atom reconstruction through conditional diffusion decoding. By leveraging generative decoding to mitigate geometric information loss introduced by quantization, ADT achieves accurate reconstruction from compact discrete representations. The resulting tokens further support autoregressive generation of both small molecules and all-atom protein structures. Together, our results demonstrate that compact structural tokens can serve as an effective discrete interface for all-atom molecular modeling, bridging efficient token-based generation with fine-grained atomic geometry.

Appendix

The appendix is organized as follows:

  • Appendix A provides further experimental results, including fine-grained reconstruction analyses, comparisons across token granularities and discrete protein tokenizers, and qualitative results.

  • Appendix B presents additional ablation studies on structural information in discrete tokens, coarse-to-fine decoding, discrete capacity, and autoregressive factorization.

  • Appendix C outlines additional implementation details, including model architectures, training settings, and evaluation metrics.

  • Appendix D contains the limitations of our work.

  • Appendix E describes the use of large language models during manuscript preparation.

Further Experimental Results

Comparison Across Token Granularities

Atom-level tokenization provides direct coverage of all-atom protein geometry, but requires one structural token per atom, resulting in substantially longer sequences than residue-level tokenization. On CASP14 and CASP15, proteins contain on average 2,141.3 and 2,598.5 atoms but only 268.6 and 336.5 residues, respectively, requiring approximately 8.0× and 7.7× as many structural tokens as residue-level tokenization. Table 8 compares all-atom reconstruction across these token granularities. Atom-level tokenizers achieve lower reconstruction error, whereas ADT directly reconstructs complete atomic geometry from a single structural token per residue and substantially improves over residue-level backbone tokenization followed by external all-atom completion.

ModelCASP14CASP15
AA RMSD ↓\downarrowTM-score ↑\uparrowAA RMSD ↓\downarrowTM-score ↑\uparrow
Atom-level
Bio2Token ([35])0.6440.9830.6930.981
BiHiTo ([51])0.5150.9870.5850.986
Residue-level
DPLM-2 ([47])–0.8860–0.8644
+ cg2all (no fix)3.41200.88594.50370.8644
+ PDBFixer3.96570.88595.02130.8644
+ cg2all (fixed)3.43580.88594.52880.8644
+ Rosetta fixbb3.49180.88594.59310.8644
ADT (ours)1.31570.96951.82610.9470

Table 8. All-atom protein reconstruction across token granularities on CASP14 and CASP15.

Comparison with Discrete Protein Tokenizers

For completeness, we additionally compare three discrete protein tokenizers, ESM3 (Hayes et al., [16]), DPLM-2 (Wang et al., [47]), and ADT, on protein structure reconstruction. ESM3 is pretrained at substantially larger scale, covering 2.78 billion protein sequences, whereas our tokenizer is trained on only 330K structures from AFDB SwissProt. Nevertheless, ESM3 reports all-atom reconstruction RMSDs of approximately 1.30 and 1.80 A˚\text{\AA} on CASP14 and CASP15, respectively, compared with 1.32 and 1.83 A˚\text{\AA} for our method, indicating comparable all-atom reconstruction accuracy. Since the publicly released ESM3 structure decoder exposes only backbone coordinates, we restrict the directly evaluated comparison in Table 9 to backbone reconstruction.

MethodCASP14CASP15
Cα\alpha-lDDT ↑\uparrowBB RMSD ↓\downarrowTM-score ↑\uparrowGDT-TS ↑\uparrowGDT-HA ↑\uparrowCα\alpha-lDDT ↑\uparrowBB RMSD ↓\downarrowTM-score ↑\uparrowGDT-TS ↑\uparrowGDT-HA ↑\uparrow
ESM30.96531.99340.95520.90410.80680.96722.06850.94550.86930.7470
DPLM-20.88202.85760.88600.77180.57240.89334.02230.86440.70320.5104
ADT (Ours)0.89700.96280.96950.93980.76860.89721.51910.94700.90010.7285

Table 9. Protein structure reconstruction performance on CASP14 and CASP15.

More importantly, under direct evaluation, our method achieves lower backbone RMSD and higher TM-score and GDT-TS than ESM3 on both CASP14 and CASP15, while ESM3 achieves higher Cα\alpha-lDDT. This contrast indicates that ESM3 better preserves local backbone distances, whereas our method better preserves global structural similarity. All metrics are averaged over individual proteins.

Unconditional Protein Generation

To further evaluate the generative capability of ADT, we perform unconditional protein structure generation using a multimodal discrete diffusion model over amino-acid and ADT structural tokens, and compare with discrete protein generative models ESM3 and DPLM-2. Following prior work (Ren et al., [41]), we generate 1,000 protein structures and evaluate designability (scRMSD ≤2.0A˚\le2.0\text{\AA}), diversity, DPT, and novelty. As shown in Table 10, ADT achieves the highest designability and diversity while maintaining competitive DPT and novelty. These results demonstrate that ADT supports diverse and designable protein generation across different discrete generative paradigms.

MethodDes. ↑\uparrowDiv. ↑\uparrowDPT ↓\downarrowNov. ↑\uparrow
ESM30.611270.370.84
DPLM-20.631300.370.72
ADT (Ours)0.681390.350.76

Table 10. Unconditional protein structure generation.

Qualitative Results

Figure 5 presents samples autoregressively generated by ADT across different molecular domains. On QM9, ADT generates diverse molecular topologies with multiple rings and complex bonding patterns, including samples with 3 rings and 11 bonds. On proteins, ADT scales to substantially larger structures and generates distinct global folds with nontrivial three-dimensional conformations. These results qualitatively demonstrate the capability of ADT to autoregressively generate complex structured objects across different structural scales and domains.

Qualitative results of samples generated by ADT for small molecules from QM9 and proteins

Figure 5. Qualitative results of samples generated by ADT for small molecules from QM9 and proteins.

Additional Ablation Studies

Structural Information in Discrete Tokens

To assess whether ADT tokens encode all-atom conformational information, we keep the amino-acid identities unchanged while randomly permuting the structural tokens within each protein, denoted as Shuffled tokens. As shown in Table 11, shuffling the structural tokens substantially degrades reconstruction across all metrics, indicating that the decoder relies on residue-specific geometric information carried by the structural tokens.

MethodCα\alpha lDDT ↑\uparrowBB RMSD ↓\downarrowAA lDDT ↑\uparrowAA RMSD ↓\downarrowSC RMSD ↓\downarrowTM-score ↑\uparrowGDT-TS ↑\uparrowGDT-HA ↑\uparrow
CASP14 Shuffled0.310021.1660.297821.70922.2780.22530.03730.0058
CASP14 ADT0.89700.96280.82231.31571.59380.96950.93980.7686
CASP15 Shuffled0.323322.2450.310322.73323.2740.22500.03540.0057
CASP15 ADT0.89721.51910.82581.82612.07790.94700.90010.7285

Table 11. Ablation of structural tokens on protein structure reconstruction.

Fine-grained Geometric Reconstruction on QM9

We further evaluate fine-grained geometric fidelity in Table 12. Diffusion decoding reduces RMSD from 0.0584 to 0.0343 while also improving structure match, PoseBusters validity, and errors in bond lengths, bond angles, and dihedral angles. These results show that generative decoding more effectively reconstructs fine-grained geometry from the discrete representation than direct coordinate prediction.

DecoderMatch ↑\uparrowRMSD ↓\downarrowPB Valid ↑\uparrowBond-length ↓\downarrowBond-angle ↓\downarrowDihedral ↓\downarrow
Transformer98.90.05840.920.022191.49714.5234
Diffusion99.70.03430.930.015571.09293.7622

Table 12. Fine-grained Geometric Reconstruction on QM9.

Effect of Coarse-to-Fine Decoding

To understand how residue-level representations can be progressively lifted to all-atom geometry, we investigate how decoder layers should be allocated between residue-level modeling and atom-level refinement. Table 13 presents an ablation of our coarse-to-fine decoding design, where Coarse:Fine denotes the number of residue-level and atom-level decoder layers, respectively. The results yield two key observations: 1) Benefit of coarse-to-fine decoding. Coarse-to-fine configurations out-perform both residue-only (12:0) and atom-only (0:12), showing that residue-level modeling and atom-level refinement provide complementary benefits for protein reconstruction. 2) Lightweight atomic refinement is sufficient. The 10:2 configuration consistently outperforms 8:4 and achieves the best performance across all reported metrics. This suggests that most structural modeling can be performed at the residue level, with only a lightweight atom-level stage needed for fine-grained refinement.

Coarse:FineCαC_\alpha lDDT ↑\uparrowBB RMSD ↓\downarrowAA lDDT ↑\uparrowAA RMSD ↓\downarrowTM-score ↑\uparrowGDT-TS ↑\uparrowGDT-HA ↑\uparrow
12:00.85681.11960.79341.55590.95280.90270.7089
8:40.88920.98760.81611.35740.96680.93240.7571
10:20.89700.96280.82231.31570.96950.93980.7686
0:120.86241.07380.80671.51260.95870.90680.7115
12:00.85741.76680.79672.15940.93070.86300.6688
8:40.88951.55830.81961.88400.94440.89300.7170
10:20.89721.51910.82581.82610.94700.90010.7285
0:120.86291.69450.81012.09950.93650.86710.6714

Table 13. Ablation of coarse-to-fine diffusion decoding on protein structure reconstruction.

Effect of Discrete Capacity

To examine the effect of discrete capacity on residue-level structural tokens, we vary the codebook size while keeping one structural token per residue. As shown in Table 14, reconstruction consistently improves as the codebook grows from 2048 to 16384 across both CASP14 and CASP15. This trend suggests that increasing the representational capacity of each structural token better preserves geometric information through the discrete bottleneck. A codebook of 8192 already provides strong reconstruction performance, although increasing the vocabulary to 16384 yields further improvements. We adopt 8192 as the default to balance reconstruction fidelity with vocabulary size, given the larger prediction space that an expanded vocabulary introduces for autoregressive modeling.

CodebookCαC_\alpha lDDT ↑\uparrowBB RMSD ↓\downarrowAA lDDT ↑\uparrowAA RMSD ↓\downarrowTM-score ↑\uparrowGDT-TS ↑\uparrowGDT-HA ↑\uparrow
20480.86341.03420.78641.47340.95630.91720.7243
40960.87210.98180.80131.38690.96340.91950.7431
81920.89700.96280.82231.31570.96950.93980.7686
163840.90560.94670.83151.27680.97110.95840.7869
20480.86281.63240.78992.04130.93380.88120.6853
40960.87061.55870.80491.92840.94090.88760.7041
81920.89721.51910.82581.82610.94700.90010.7285
163840.90391.46360.83881.73820.94910.91730.7467

Table 14. Effect of discrete codebook size on protein structure reconstruction.

Ablation of Autoregressive Factorization

We examine the effect of autoregressive factorization on QM9 by comparing three generation strategies: interleaved autoregression, geometry-to-chemical autoregression (G2C-AR), and our Chemistry-to-Geometry autoregression (C2G-AR). All variants differ only in autoregressive factorization, with all other settings kept fixed. The interleaved AR follows (s1,c1,…,sL,cL)(s_1, c_1, \ldots, s_L, c_L); G2C-AR generates the complete structural sequence before the chemical sequence; and C2G-AR generates the complete chemical sequence before the structural sequence.

Table 15 highlights two findings. First, C2G-AR consistently performs best, improving all metrics over interleaved AR and directly validating our Chemistry-to-Geometry factorization. Second, the gain is not a generic benefit of two-stage generation: reversing the order with G2C-AR reduces validity to 0.92 and PB validity to 0.85, below the interleaved baseline. These results isolate the benefit of generating structural tokens after the complete chemical sequence and directly support the C2G factorization used in our model.

StrategyValid ↑\uparrowUnique ↑\uparrowDiv. ↑\uparrowPB Valid ↑\uparrow
Interleaved AR0.940.930.900.88
G2C-AR0.920.890.910.85
C2G-AR1.000.960.910.94

Table 15. Effect of autoregressive factorization on QM9 generation.

More Implementation Details

Tokenizer Model Architecture

Both the small-molecule and protein tokenizers use pre-norm Transformer encoders, binary FSQ, and Diffusion Transformer (DiT) decoders, with 12 layers in both the encoder and decoder. The small-molecule tokenizer uses dmodel=1536d_{\mathrm{model}} = 1536, while the protein tokenizer uses dmodel=256d_{\mathrm{model}} = 256. For proteins, the 12-layer DiT decoder consists of 10 residue-level layers followed by 2 intra-residue atom-level layers. Both tokenizers use 13-bit binary FSQ, corresponding to a codebook size of 8192.

Training Details

Both the small-molecule and protein tokenizers are optimized using AdamW with a learning rate of 3×10−43 \times10^{-4}. The mutual-information regularizer is applied to both tokenizers with λMI=10−3\lambda_{\mathrm{MI}} = 10^{-3}. The temperature of mutual information τ\tau is set to 1.0.

Small-molecule tokenizer. The small-molecule tokenizer is trained with the diffusion reconstruction objective and the mutual-information regularizer.

Protein tokenize. For protein tokenizer training, we construct a high-confidence subset of AFDB Swiss-Prot by retaining monomeric proteins with 60–1,024 modeled residues and mean pLDDT above 85, resulting in 330,154 protein structures In addition to the diffusion reconstruction objective and mutual-information regularizer, the protein tokenizer uses auxiliary geometric losses. Specifically, we define

Lbond=1∣B∣∑(i,j)∈B(d^ij−dij)2,Langle=1∣T∣∑(i,j,k)∈T(cos⁡θ^ijk−cos⁡θijk)2,\mathcal{L}_{\mathrm{bond}} = \frac{1}{|\mathcal{B}|} \sum_{(i,j)\in\mathcal{B}} \left(\hat{d}_{ij} - d_{ij}\right)^2,\qquad\mathcal{L}_{\mathrm{angle}} = \frac{1}{|\mathcal{T}|} \sum_{(i,j,k)\in\mathcal{T}} \left(\cos\hat{\theta}_{ijk} - \cos\theta_{ijk}\right)^2,
Lclash=1∣N∣∑(i,j)∈N[max⁡(0,δij−d^ij)]2,Ldist=1∣P∣∑(i,j)∈P(d^ijCα−dijCα)2.\mathcal{L}_{\mathrm{clash}} = \frac{1}{|\mathcal{N}|} \sum_{(i,j)\in\mathcal{N}} \left[\max(0,\delta_{ij} - \hat{d}_{ij})\right]^2,\qquad\mathcal{L}_{\mathrm{dist}} = \frac{1}{|\mathcal{P}|} \sum_{(i,j)\in\mathcal{P}} \left(\hat{d}^{\mathrm{C}_{\alpha}}_{ij} - d^{\mathrm{C}_{\alpha}}_{ij}\right)^2.

Here, B\mathcal{B}, T\mathcal{T}, N\mathcal{N}, and P\mathcal{P} denote bonded atom pairs, bond-angle triplets, non-bonded atom pairs, and Cα_{\alpha} pairs, respectively. The weights for the backbone, bond-length, bond-angle, clash and distance are set to 1.0, 0.1, 0.1, 0.1 and 0.01, respectively.

Autoregressive Generation Details

We train an unconditional autoregressive model on QM9 to evaluate the generative capability of the learned discrete structural representation. Each molecule is represented by a sequence of atom identities C=(c1,…,cL)C = (c_1, \ldots, c_L) and the corresponding structural tokens S=(s1,…,sL)S = (s_1, \ldots, s_L) obtained from the pretrained tokenizer. Each atom is associated with one structural token. The tokenizer uses 13-bit binary FSQ, corresponding to a structural vocabulary of 213=81922^{13} = 8192 discrete codes. We set β=1.0\beta= 1.0.

Small-Molecule Autoregressive Generation. We use a 12-layer Transformer with a hidden dimension of 768, 12 attention heads, and a dropout rate of 0.1, trained from scratch. The autoregressive model is trained on the QM9 training split using AdamW with a batch size of 64 and a learning rate of 10−410^{-4}. Teacher-forced prediction performance is evaluated on the validation set containing 12,981 molecules.

Unconditional Protein Generation. For unconditional protein generation, we train a multimodal discrete diffusion model over amino-acid and ADT structural tokens, following the generation paradigm of prior discrete protein models to enable a controlled comparison. The model consists of 33 bidirectional Transformer layers with a hidden dimension of 1280 and 20 attention heads. During training, amino-acid and structural tokens are corrupted according to an absorbing discrete diffusion process, and the model learns to recover the clean tokens from their corrupted counterparts. At inference, generation starts from masked amino-acid and structural tokens and iteratively denoises both modalities to jointly generate a protein sequence and its corresponding structural-token sequence. The generated structural tokens are decoded into all-atom coordinates by the frozen ADT diffusion decoder. The model is trained with a learning rate of 1×10−41 \times10^{-4}.

Sequence-Conditioned All-atom Protein Autoregressive Generation. The training set is constructed from the first 80 shards of AFDB v6 with pLDDT ≥90\ge90, comprising approximately 32.2M structures and 10.1B residues after preprocessing. We remove training structures with sequence identity greater than 40% to any CAMEO22 target, requiring at least 80% alignment coverage of the evaluation target. Each training example consists of an amino-acid sequence paired with structural tokens obtained from the pretrained ADT tokenizer. We evaluate sequence-conditioned structure generation on 183 CAMEO22 targets [42].

We use a Qwen3.5-0.8B model as the backbone with a hidden dimension of 1024. During training, the amino-acid sequence is provided as context and the cross-entropy loss is applied only to the target structural-token sequence. We optimize the model using AdamW with β1=0.9\beta_1 = 0.9, β2=0.95\beta_2 = 0.95, ϵ=10−8\epsilon= 10^{-8}, and weight decay 0.1. We use a global batch size of 2048, a peak learning rate of 10−410^{-4}, and cosine learning-rate decay to 10−510^{-5}, with a 5% warmup period. Gradients are clipped to a maximum norm of 1.0. At inference, given an amino-acid sequence, the model autoregressively generates structural tokens, which are then decoded into all-atom coordinates by the frozen ADT diffusion decoder. We report greedy decoding in Table 5 for direct comparison with DPLM-2.

Evaluation Metrics

Reconstruction Metrics

RMSD. For a structure with NN atoms, ground-truth coordinates X∈RN×3\mathbf{X} \in\mathbb{R}^{N \times3} and reconstructed coordinates X^∈RN×3\hat{\mathbf{X}} \in\mathbb{R}^{N \times3}, we report the Kabsch-aligned RMSD:

RMSD⁡(X,X^)=min⁡R∈SO(3), t∈R31N∥X−(X^R⊤+1t⊤)∥F.\operatorname{RMSD}(\mathbf{X},\hat{\mathbf{X}})=\min_{\mathbf{R}\in SO(3),\,\mathbf{t}\in\mathbb{R}^{3}}\frac{1}{\sqrt{N}}\left\|\mathbf{X}-\left(\hat{\mathbf{X}}\mathbf{R}^{\top}+\mathbf{1}\mathbf{t}^{\top}\right)\right\|_{F}.

For proteins, RMSD is computed over the corresponding atom subset, including backbone (BB RMSD), all atoms (AA RMSD), or side-chain atoms (SC RMSD).

Structure Match Rate. For molecular reconstruction on QM9, a reconstruction is considered successfully matched if a valid chemical and atomic correspondence can be established with the reference molecule using a matching tolerance of 0.01 Å. Match rate is the fraction of successfully matched molecules, while RMSD is computed after optimal alignment and averaged over the matched subset.

lDDT. The local Distance Difference Test (lDDT) measures the preservation of local interatomic distances between the reconstructed and reference structures without requiring global alignment. We report Cα-lDDT over Cα atoms, BB-lDDT over backbone atoms, and AA-lDDT over all atoms.

TM-score. TM-score measures the global structural similarity between the reconstructed and reference protein structures, with higher values indicating greater agreement in the overall fold.

GDT. The Global Distance Test measures the fraction of residues that can be superimposed within predefined distance thresholds. We report GDT-TS and GDT-HA, with GDT-HA using stricter distance thresholds to assess higher-accuracy structural agreement.

Generation Metrics

Valid. Fraction of generated molecules that satisfy chemical validity, including valid valence assignments and a consistent molecular graph, as determined by RDKit sanitization.

Unique. Fraction of valid generated molecules that are chemically distinct, measured by unique canonical SMILES.

Novelty. Fraction of valid generated molecules that are not present in the training set, determined by canonical SMILES.

Diversity. Mean pairwise molecular dissimilarity among valid generated molecules, measured using Tanimoto similarity of Morgan fingerprints:

Diversity=1−2N(N−1)∑i<jTanimoto⁡(fi,fj)\mathrm{Diversity}=1-\frac{2}{N(N-1)}\sum_{i<j}\operatorname{Tanimoto}(f_i,f_j)

where fif_i denotes the Morgan fingerprint (radius 2, 2048 bits) of molecule ii.

PB Valid. Fraction of generated molecules that pass the PoseBusters validity checks, providing a more stringent assessment of physicochemical plausibility through chemical consistency, intra-molecular clashes, bond lengths, and bond angles.

Self-Consistency. For each generated protein structure, we design eight sequences using Protein-MPNN and predict their structures with ESMFold. We report scRMSD as the minimum Kabsch-aligned Cα RMSD and scTM as the maximum TM-score between the generated structure and the predicted structures of its designed sequences.

Designability. Fraction of generated protein structures with scRMSD ≤2.0\leq2.0 Å, indicating that at least one designed sequence folds consistently with the generated structure.

Protein Diversity. Number of distinct structural clusters among designable generated structures, obtained by clustering with pairwise TM-score at a cutoff of 0.5. Higher values indicate broader structural coverage.

Protein Novelty. Structural dissimilarity of designable generated structures to known protein structures, defined as

Novelty=1∣D∣∑X^∈D[1−max⁡P∈PDBTM⁡(X^,P)]\mathrm{Novelty}=\frac{1}{|\mathcal{D}|}\sum_{\hat{X}\in\mathcal{D}}\left[1-\max_{P\in\mathrm{PDB}}\operatorname{TM}(\hat{X},P)\right]

where D\mathcal{D} denotes the set of designable generated structures. Higher values indicate greater structural deviation from the closest known PDB structures.

Designable Pairwise TM-score (DPT). Designable Pairwise TM-score (DPT) measures the mean pairwise TM-score among designable samples (scRMSD ≤2.0\leq2.0 Å), where a lower value indicates greater structural diversity among valid generations.

Limitations

The focus of this work is to introduce and validate ADT as a compact discrete representation for all-atom molecular structures. To this end, we study two representative molecular modalities, small molecules and proteins, spanning atom-level and residue-level tokenization. A natural extension would be to expand ADT to other biomolecular modalities, such as nucleic acids and molecular complexes, toward a shared discrete representation across diverse molecular systems. Such a representation could enable joint autoregressive modeling of different molecular modalities and potentially benefit from structural knowledge transferred across them. While our experiments cover reconstruction, unconditional small-molecule generation, and sequence-conditioned protein structure generation, another promising direction is to incorporate functional or property conditioning for structure-based molecular design. We leave these directions for future investigation.

Usage Of LLMs

Large language models were employed as an auxiliary tool to support manuscript preparation, including grammar checking, sentence refinement, and clarification of technical descriptions. All AI-suggested text was carefully reviewed and revised by the authors to ensure accuracy, clarity, and scientific integrity.

References

  1. [1]Aadityot Bhatnagar, Sarthak Jain, Joel Beazer, Samuel Curran, Alexander Hoffnagle, Kyle Ching, Michael Martyn, Stephen Nayfach, Jeffrey Ruffolo, and Ali Madani. Scaling unlocks broader generation and deeper functional understanding of proteins. Advances in Neural Information Processing Systems, 38:46109–46145, 2026.DOI
  2. [2]Yilin Chen, Tianyu Lu, Cizhang Zhao, Hannah Wayment-Steele, and Po-Ssu Huang. SLAE: Strictly local all-atom environment for protein representation. In Forty-third International Conference on Machine Learning, 2026. URL https://openreview.net/forum?id=4HDRkXxqNs.DOI
  3. [3]Alexander E Chu, Jinho Kim, Lucy Cheng, Gina El Nesr, Minkai Xu, Richard W Shuai, and Po-Ssu Huang. An all-atom protein generative model. Proceedings of the National Academy of Sciences, 121(27):e2311500121, 2024.DOI
  4. [4]Rohit Dilip, Ayush Varshney, and David Van Valen. Adaptive protein tokenization. In Forty-third International Conference on Machine Learning, 2026a. URL https://openreview.net/forum?id=UGO6mAANLH.arxiv.org/abs/2602.06418
  5. [5]Rohit Dilip, Evan Zhang, Ayush Varshney, and David Van Valen. Flow autoencoders are effective protein tokenizers. In International Conference on Learning Representations, volume 2026, pp. 145901–145926, 2026b.arxiv.org/abs/2510.00351
  6. [6]Ian Dunn and David Ryan Koes. Mixed continuous and categorical flow matching for 3d de novo molecule generation. ArXiv, pp. arXiv–2404, 2024.arxiv.org/abs/2404.19739
  7. [7]Daniel Flam-Shepherd and Alán Aspuru-Guzik. Language models can generate molecules, materials, and protein binding sites directly in three dimensions as xyz, cif, and pdb files. arXiv preprint arXiv:2305.05708, 2023.
  8. [8]Kaiyuan Gao, Yusong Wang, Haoxiang Guan, Zun Wang, Qizhi Pei, John Hopcroft, Kun He, and Lijun Wu. Tokenizing 3d molecule structure with quantized spherical coordinates. In Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 1, pp. 291–301, 2026.arxiv.org/abs/2412.01564
  9. [9]Zhangyang Gao, Cheng Tan, Jue Wang, Yufei Huang, Lirong Wu, and Stan Z Li. Foldtoken: Learning protein language via vector quantization and beyond. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 219–227, 2025.
  10. [10]Benoit Gaujac, Jérémie Donà, Liviu Copoiu, Timothy Atkinson, Thomas Pierrot, and Thomas D Barrett. Learning the language of protein structure. arXiv preprint arXiv:2405.15840, 2024.arxiv.org/abs/2405.15840
  11. [11]Niklas Gebauer, Michael Gastegger, and Kristof Schütt. Symmetry-adapted generation of 3d point sets for the targeted discovery of molecules. Advances in neural information processing systems, 32, 2019.arxiv.org/abs/1906.00957
  12. [12]Niklas WA Gebauer, Michael Gastegger, Stefaan SP Hessmann, Klaus-Robert Müller, and Kristof T Schütt. Inverse design of 3d molecular structures with conditional generative neural networks. Nature communications, 13(1):973, 2022.DOI
  13. [13]Tomas Geffner, Kieran Didi, Zhonglin Cao, Danny Reidenbach, Zuobai Zhang, Christian Dallago, Emine Kucukbenli, Karsten Kreis, and Arash Vahdat. La-proteina: Atomistic protein generation via partially latent flow matching. In International Conference on Learning Representations, volume 2026, pp. 67350–67395, 2026.arxiv.org/abs/2507.09466
  14. [14]Filya Geikyan and Hrant Khachatrian. Accurate tokenization of 3d small organic molecules. In ICLR 2026 Workshop on Foundation Models for Science: Real-World Impact and Science-First Design, 2026.
  15. [15]Xu Han, Yuancheng Sun, Kai Chen, Yuxuan Ren, Kang Liu, and Qiwei Ye. Constraint decoupled latent diffusion for protein backmapping. Journal of Chemical Theory and Computation, 22(1):624–637, 2026.DOI
  16. [16]Thomas Hayes, Roshan Rao, Halil Akin, Nicholas J Sofroniew, Deniz Oktay, Zeming Lin, Robert Verkuil, Vincent Q Tran, Jonathan Deaton, Marius Wiggert, et al. Simulating 500 million years of evolution with a language model. Science, 387(6736):850–858, 2025.DOI
  17. [17]Lim Heo and Michael Feig. One bead per residue can describe all-atom protein structures. Structure, 32(1):97–111, 2024.
  18. [18]Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556, 2022.arxiv.org/abs/2203.15556
  19. [19]Emiel Hoogeboom, Victor Garcia Satorras, Clément Vignac, and Max Welling. Equivariant diffusion for molecule generation in 3d. In International conference on machine learning, pp. 8867–8887. PMLR, 2022.arxiv.org/abs/2203.17003
  20. [20]Han Huang, Leilei Sun, Bowen Du, and Weifeng Lv. Learning joint 2-d and 3-d graph diffusion models for complete molecule generation. IEEE transactions on neural networks and learning systems, 35(9):11857–11871, 2024.DOI
  21. [21]Ross Irwin, Spyridon Dimitriadis, Jiazhen He, and Esben Jannik Bjerrum. Chemformer: a pretrained transformer for computational chemistry. Machine Learning: Science and Technology, 3 (1):015022, 2022.DOI
  22. [22]Ross Irwin, Alessandro Tibo, Jon Paul Janet, and Simon Olsson. Efficient 3d molecular generation with flow matching and scale optimal transport. In ICML 2024 AI for Science Workshop, 2024.
  23. [23]Wengong Jin, Regina Barzilay, and Tommi Jaakkola. Junction tree variational autoencoder for molecular graph generation. In International conference on machine learning, pp. 2323–2332. PMLR, 2018.arxiv.org/abs/1802.04364
  24. [24]Chaitanya K. Joshi, Xiang Fu, Yi-Lun Liao, Vahe Gharakhanyan, Benjamin Kurt Miller, Anuroop Sriram, and Zachary Ward Ulissi. All-atom diffusion transformers: Unified generative modelling of molecules and materials. In Forty-second International Conference on Machine Learning, 2025. URL https://openreview.net/forum?id=8Q9QmZjThv.arxiv.org/abs/2503.03965
  25. [25]Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020.arxiv.org/abs/2001.08361
  26. [26]Kuzma Khrabrov, Anton Bren, Artem Tsypin, Konstantin Ushenin, Egor Rumiantsev, Alexander Telepov, Dmitry Protasov, Ilya Shenbin, Anton Alekseev, Mikhail Shirokikh, Sergey Nikolenko, Elena Tutubalina, and Artur Kadurin. ▽²dft: A universal quantum chemistry dataset of drug-like molecules and a benchmark for neural network potentials. Advances in Neural Information Processing Systems, 37:36869–36889, 2024.
  27. [27]Andriy Kryshtafovych, Torsten Schwede, Maya Topf, Krzysztof Fidelis, and John Moult. Critical assessment of methods of protein structure prediction (casp)—round xiv. Proteins: Structure, Function, and Bioinformatics, 89(12):1607–1617, 2021.DOI
  28. [28]Andriy Kryshtafovych, Torsten Schwede, Maya Topf, Krzysztof Fidelis, and John Moult. Critical assessment of methods of protein structure prediction (casp)—round xv. Proteins: Structure, Function, and Bioinformatics, 91(12):1539–1549, 2023.
  29. [29]Tuan Le, Julian Cremer, Frank Noe, Djork-Arné Clevert, and Kristof T Schütt. Navigating the design space of equivariant diffusion-based generative models for de novo 3d molecule generation. In International Conference on Learning Representations, volume 2024, pp. 26244–26266, 2024.arxiv.org/abs/2309.17296
  30. [30]Andrew Leaver-Fay, Michael Tyka, Steven M Lewis, Oliver F Lange, James Thompson, Ron Jacak, Kristian W Kaufman, P Douglas Renfrew, Colin A Smith, Will Sheffler, et al. Rosetta3: an object-oriented suite for the simulation and design of macromolecules. In Methods in enzymology, volume 487, pp. 545–574. Elsevier, 2011.
  31. [31]Shaoning Li, Le Zhuo, Yusong Wang, Mingyu Li, Xinheng He, Fandi Wu, Hongsheng Li, and Pheng-Ann Heng. Proteinae: Protein diffusion autoencoders for structure encoding. In International Conference on Learning Representations, volume 2026, pp. 69939–69960, 2026.arxiv.org/abs/2510.10634
  32. [32]Xiner Li, Limei Wang, Youzhi Luo, Carl Edwards, Shurui Gui, Yuchao Lin, Heng Ji, and Shuiwang Ji. Geometry informed tokenization of molecules for language model generation. In Forty-second International Conference on Machine Learning, 2025. URL https://openreview.net/forum?id=4umRQdvuW5.arxiv.org/abs/2408.10120
  33. [33]Xiaohan Lin, Zhenyu Chen, Yanheng Li, Zicheng Ma, Chuanliu Fan, Ziqiang Cao, Shihao Feng, Jun Zhang, and Yi Qin Gao. Unifying sequence-structure coding for advanced protein engineering via a multimodal diffusion transformer. Chemical Science, 16(24):11087–11102, 2025a.
  34. [34]Xiaohan Lin, Yijie Xia, Yanheng Li, Yu-Peng Huang, Shuo Liu, Jun Zhang, and Yi Qin Gao. Unifying sequence-structure coding for advanced protein engineering via a multimodal editing through physics-informed and preference-aligned generative foundation models. Nature Communications, 16(1):6043, 2025b.
  35. [35]Andrew Liu, Axel Elaldi, Nathan Russell, and Olivia Viessmann. Bio2token: All-atom tokenization of any biomolecular structure with mamba. In ICLR 2025 Workshop on Generative and Experimental Perspectives for Biomolecular Design, 2025. URL https://openreview.net/forum?id=EkV2sPRMy5.
  36. [36]Yanchen Luo, Zhiyuan Liu, Yi Zhao, Sihang Li, Hengxing Cai, Kenji Kawaguchi, Tat-Seng Chua, Yang Zhang, and Xiang Wang. Towards unified and lossless latent space for 3d molecular latent diffusion modeling. Advances in Neural Information Processing Systems, 38:30718–30748, 2025.
  37. [37]Fabian Mentzer, David Minnen, Eirikur Agustsson, and Michael Tschannen. Finite scalar quantization: Vq-vae made simple. In International Conference on Learning Representations, volume 2024, pp. 51772–51783, 2024.arxiv.org/abs/2309.15505
  38. [38]Alex Morehead and Jianlin Cheng. Geometry-complete diffusion for 3d molecule generation and optimization. Communications Chemistry, 7(1):150, 2024.
  39. [39]Xingang Peng, Jiaqi Guan, Qiang Liu, and Jianzhu Ma. Moldiff: Addressing the atom-bond inconsistency problem in 3d molecule diffusion generation. ICML, 2023.
  40. [40]Raghunathan Ramakrishnan, Pavlo O Dral, Matthias Rupp, and O Anatole Von Lilienfeld. Quantum chemistry structures and properties of 134 kilo molecules. Scientific data, 1(1):140022, 2014.
  41. [41]Yuxuan Ren, Fan Yang, Jianhua Yao, and Yatao Bian. Sinae: A single-architecture flow-matching autoencoder for cross-domain atomic systems. arXiv preprint arXiv:2607.12380, 2026.
  42. [42]Xavier Robin, Juergen Haas, Rafal Gumienny, Anna Smolinski, Gerardo TaurielIo, and Torsten Schwede. Continuous automated model evaluation (cameo)—perspectives on the future of fully automated evaluation of structure prediction methods. Proteins: Structure, Function, and Bioinformatics, 89(12):1977–1986, 2021.
  43. [43]Qun Su, Qiaolin Gou, Hui Zhang, Jike Wang, Huiyong Sun, Renling Hu, Rui Qin, Huanxiang Liu, Tingjun Hou, and Yu Kang. Lamgen: Llm-based 3d molecular generation for multi-target drug design. Nature communications, 2026.
  44. [44]Qwen Team. Qwen3. 5-omni technical report. arXiv preprint arXiv:2604.15804, 2026.arxiv.org/abs/2604.15804
  45. [45]Michel Van Kempen, Stephanie S Kim, Charlotte Tumescheit, Milot Mirdita, Jeongjae Lee, Cameron LM Gilchrist, Johannes Söding, and Martin Steinegger. Fast and accurate protein structure search with foldseek. Nature biotechnology, 42(2):243–246, 2024.
  46. [46]Mihaly Varadi, Stephen Anyango, Mandar Deshpande, Sreenath Nair, Cindy Natassia, Galabina Yordanova, David Yuan, Oana Stroe, Gemma Wood, Agata Laydon, et al. AlphaFold protein structure database: massively expanding the structural coverage of protein-sequence space with high-accuracy models. Nucleic acids research, 50(D1):D439–D444, 2022.
  47. [47]Xinyou Wang, Zaixiang Zheng, Fei Ye, Dongyu Xue, Shujian Huang, and Quanquan Gu. Dplm-2: A multimodal diffusion protein language model. In International Conference on Learning Representations, volume 2025, pp. 35463–35490, 2025.
  48. [48]Joseph L Watson, David Juergens, Nathaniel R Bennett, Brian L Trippe, Jason Yim, Helen E Eisenach, Woody Ahern, Andrew J Borst, Robert J Ragotte, Lukas F Milles, et al. De novo design of protein structure and function with rfdiffusion. Nature, 620(7976):1089–1100, 2023.DOI
  49. [49]Minkai Xu, Alexander S Powers, Ron O Dror, Stefano Ermon, and Jure Leskovec. Geometric latent diffusion models for 3d molecule generation. In International Conference on Machine Learning, pp. 38592–38610. PMLR, 2023.arxiv.org/abs/2305.01140
  50. [50]Jason Yim, Brian L. Trippe, Valentin De Bortoli, Emile Mathieu, Arnaud Doucet, Regina Barzilay, and Tommi S. Jaakkola. Se(3) diffusion model with application to protein backbone generation. In ICML, pp. 40001–40039, 2023. URL https://proceedings.mlr.press/v202/yim23a.html.arxiv.org/abs/2302.02277
  51. [51]Ruochong Zheng, Yutian Liu, Yian Zhao, Zhiwei Nie, Xuehan Hou, Chang Liu, Siwei Ma, Youdong Mao, and Jie Chen. Bihito: Biomolecular hierarchy-inspired tokenization. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, pp. 28848–28856, 2026.
  52. [52]Artem Zholus, Maksim Kuznetsov, Roman Schutski, Rim Shayakhmetov, Daniil Polykovskiy, Sarath Chandar, and Alex Zhavoronkov. Bindgpt: A scalable framework for 3d molecular design via language modeling and reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 26083–26091, 2025.arxiv.org/abs/2406.03686

Paper details

vlnc:W_kzr55zdk
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 9

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Large language models

View statement
Looking for related papers…
Contents