betaFeedback

WorksPhysics and Astronomy

Multi-Task Fine-Tuning Enables Robust Out-of-Distribution Generalization in Atomistic Models

Abstract

Accurate de novo molecular and materials design requires structure-property models that generalize beyond known regimes. Although pretrained atomistic models achieve strong in-distribution accuracy after fine-tuning, their reliability under out-of-distribution (OOD) conditions remains unclear. We identify a critical failure mode in downstream adaptation: standard fine-tuning induces representation collapse, erasing pretrained chemical and structural priors and severely degrading OOD performance. To address this limitation, we propose multi-task fine-tuning (MFT), which jointly optimizes downstream property prediction with a physically grounded force-field objective inherited from pretraining. This approach preserves essential chemical priors while enabling task-specific adaptation. Across molecular and materials benchmarks, MFT consistently improves OOD generalization, approaching the theoretical limit set by in-distribution accuracy, while outperforming standard fine-tuning, training from scratch, and state-of-the-art task-specific models. These results establish safe adaptation as a central requirement for large atomistic models and position MFT as a practical and data-efficient pathway toward robust molecular and materials discovery.

Introduction

De novo design of molecules and materials with exceptional properties remains highly challenging [60]. A crucial component of this endeavor is accurate modeling of structure–property relationships. These relationships enable two complementary strategies: (i) high-throughput virtual screening of databases to identify candidates with target properties [17], and (ii) conditional generative modeling to produce novel structures that meet specified property constraints [10, 5]. Atomistic machine learning (ML) models that learn mappings from three-dimensional atomic structures (coordinates and species) to properties consistently achieve superior predictive accuracy compared with models based solely on chemical formulae, 1D string representations (e.g., SMILES), or 2D planar graph representations [85, 3]. Consequently, atomistic ML models for property prediction have become indispensable tools in molecular and materials design.

De novo design aims to discover molecules and materials with properties, chemical compositions, or configurational features beyond those documented to date, and therefore requires models that can reliably generalize to out-of-distribution (OOD) molecules and materials. However, recent benchmark studies have highlighted a substantial degradation in OOD generalization, manifested as a pronounced performance gap between in-distribution (ID; drawn from the training data distribution) and OOD tasks, for ML models in property prediction. In the domain of organic molecules, the BOOM benchmark [3] defines OOD splits based on molecules whose target property values lie significantly above or below the range covered by the training data, and reveals that even top-performing models exhibit average OOD errors up to 3 times higher than their ID performance. Shimakawa et al. [66] further demonstrate severe performance degradation of ML property-prediction models under both property-based and structure-based OOD settings, particularly for tasks with limited training data. For inorganic crystalline materials, OOD benchmarks, either identifying lowest-density regions in feature or property space [48] or applying leave-one-out protocols in composition or clustered feature space [48, 41], consistently demonstrate that state-of-the-art (SOTA) graph neural networks (GNNs) exhibit substantial accuracy degradation when evaluated outside their training domain. Collectively, these findings underscore a critical and largely unresolved challenge: severe degradation in OOD generalization is an almost universally observed phenomenon in atomistic property prediction, across both molecular and materials systems, and largely independent of the specific definition of the OOD regime.

Pretraining on large-scale datasets to encode prior chemical knowledge into the representation (feature) space, followed by fine-tuning for downstream property prediction tasks, has proven effective in improving model generalizability. Self-supervised pretraining strategies, such as coordinate denoising and atom-type masking, have been shown to learn robust representations that significantly enhance downstream property prediction performance [81, 85, 68]. Similarly, supervised pretraining on force-field tasks [7, 32, 56, 78, 74, 8], namely, training on datasets of molecular or materials configurations labeled by first-principles calculations such as density functional theory (DFT) [6, 11, 39], has been shown to encode chemically intuitive representations [82], leading to substantial accuracy gains upon fine-tuning for materials and molecular property prediction [67, 78, 34]. However, the performance gains achieved by leveraging such prior chemical knowledge are almost exclusively evaluated under the ID settings. The fundamental question of whether fine-tuning pretrained models for property prediction tasks can reliably improve OOD performance remains largely underexplored.

Pretrained vision and language models often exhibit strong zero-shot performance [13, 26], whereas task-specific fine-tuning can be detrimental to OOD generalization through representation collapse, the distortion of general-purpose prior knowledge during downstream adaptation [2, 54]. In contrast, preserving pretrained representations such as through linear probing, yields superior OOD robustness, a gap widening as the quality of the pretrained representations improves [37].

In this work, we demonstrate that standard fine-tuning (FT) of pretrained atomistic models (Figure 1(b)) leads to a pronounced degradation in OOD generalization: the OOD performance of the fine-tuned model is substantially worse than its ID performance (Figure 2(a)). We further show that this degradation is accompanied by a loss of the chemical prior knowledge learned during pretraining, indicative of representation collapse (Figure 2(e)(Figure 2h)). Although linear probing (LP) model (Figure 1(c)) yields a slightly better OOD performance than FT, it suffers a significant decrease in ID generalizability (Figure 2(b)). To address this issue, we propose multi-task fine-tuning (MFT), which jointly optimizes the downstream property-prediction task and the upstream force-field pre-training task (Figure 1(d)). We systematically evaluate MFT using a large atomistic model, DPA-3.1-3M [83], pretrained on a diverse collection of force-field datasets spanning both materials and molecular systems [1]. Under MFT, the relative improvement in OOD generalization (measured against training from scratch) matches the corresponding ID improvement and is comparable to the ID performance achieved by FT (Figure 2(c)). This improvement is accompanied by a clear mitigation of representation collapse (Figure 2(f)(Figure 2i)). In terms of absolute accuracy, the MFT DPA-3.1-3M model achieves the best OOD performance across both molecular (Table 3) and materials (Table 4) benchmarks, including those designed around property-prediction-oriented architectures, demonstrating the robustness and practical readiness of the proposed approach for real-world molecular and materials design applications.

Image not extracted

Figure 1. Schematic illustration of pretraining and downstream adaptation strategies. The left panel depicts the pretraining phase. The right panel compares four downstream adaptation strategies: (a) training from scratch (Scratch), in which both the backbone and the task-specific prediction head are randomly initialized and optimized end-to-end; (b) fine-tuning (FT), where the backbone is initialized from the pretrained model and all parameters are jointly updated; (c) linear probing (LP), which freezes the pretrained backbone and optimizes only a randomly initialized task-specific head; and (d) multi-task fine-tuning (MFT), which jointly optimizes the pretrained backbone together with two heads: a task-specific head for property prediction and an auxiliary force-field head inherited from the pretraining stage.

Image not extracted

Figure 2. Analysis of representation collapse across downstream adaptation strategies. (a–c) Relative performance improvements of fine-tuning (FT), linear probing (LP), and multi-task fine-tuning (MFT) compared with training from scratch (Scratch) baseline. (d–f) and (g–i) t-SNE visualizations of atomic (node) and edge representations, respectively, projected into two-dimensional spaces. The clarity and separation of atomic and edge representation clusters are quantified using the Davies–Bouldin (DB) index, normalized with respect to the pretrained model. Larger relative DB index values indicate more severe degradation in cluster cohesion and separability, providing quantitative evidence of collapse in the pretrained representation space.

Model#parametersSplitHOMOLUMOGAPZPVE⟨R2⟩\langle R^2\rangleα\alphaμ\muCvC_v
ChemBERTa83MID0.00700.00930.01040.0039051.71.2540.7130.489
OOD0.02440.02560.03090.02253302.66.3282.7232.765
Graphormer47.1MID0.00400.00420.00550.0002433.40.4310.6260.180
OOD0.02360.02560.02750.01990314.17.3562.2323.667
EGNN0.2MID0.00480.00520.00690.0010319.60.5660.4810.267
OOD0.02120.02360.03120.00583181.35.6592.4462.079
ET6.8MID0.00270.00310.00430.0005728.20.4900.3680.160
OOD0.02200.02360.02710.01710298.06.5682.2573.405
MoLFormer48MID0.00500.00520.00640.0010640.11.0470.6020.785
OOD0.02360.02560.02750.01990314.17.3562.2323.667
RT27MID0.00900.01020.01330.0028968.22.2641.1040.654
OOD539.74584.880.033927.690625458699682.71911435
Geoformer51MID0.00270.00280.00460.0003010.90.3260.8470.124
OOD0.01570.01860.02400.0055763.84.2012.5441.354
Chemprop0.2MID0.00410.00460.00580.0018835.80.8660.5450.340
OOD0.01890.01790.02690.01277233.74.8502.3042.118
MACE3.9MID0.01500.01350.01810.001809.80.3220.4300.134
OOD0.03390.02470.04090.0021068.31.5432.2280.229
ModernBERT111MID0.00640.00760.00950.0007340.10.8700.6980.407
OOD0.02160.02320.03240.00170228.72.4892.6570.611
GotenNet9.2MID0.00520.00390.00880.0004311.80.5530.3190.197
OOD0.01260.01180.02290.0005316.71.8252.1730.302
DPA3-Scratch3.3MID0.0030 (0.0000)0.0029 (0.0000)0.0045 (0.0000)0.00013 (0.00000)8.5 (0.3)0.365 (0.006)0.294 (0.003)0.116 (0.001)
OOD0.0155 (0.0004)0.0130 (0.0002)0.0244 (0.0003)0.00029 (0.00000)21.4 (0.8)1.907 (0.019)2.087 (0.003)0.200 (0.006)
DPA3-LP3.3MID0.0046 (0.0000)0.0054 (0.0000)0.0077 (0.0000)0.00016 (0.00000)8.8 (0.0)0.458 (0.000)0.309 (0.000)0.149 (0.000)
OOD0.0125 (0.0001)0.0128 (0.0002)0.0193 (0.0011)0.00029 (0.00000)14.2 (0.0)1.641 (0.010)1.747 (0.011)0.281 (0.002)
DPA3-FT3.3MID0.0022 (0.0000)0.0023 (0.0000)0.0034 (0.0000)0.00010 (0.00000)8.1 (0.0)0.297 (0.001)0.276 (0.001)0.097 (0.001)
OOD0.0156 (0.0001)0.0135 (0.0003)0.0201 (0.0002)0.00026 (0.00000)18.7 (2.1)1.639 (0.022)2.267 (0.018)0.188 (0.002)
DPA3-MFT3.3MID0.0021 (0.0000)0.0024 (0.0000)0.0034 (0.0000)0.00010 (0.00000)8.3 (0.0)0.296 (0.001)0.278 (0.001)0.097 (0.001)
OOD0.0134 (0.0002)0.0107 (0.0002)0.0171 (0.0002)0.00025 (0.00000)13.7 (0.1)1.536 (0.005)2.087 (0.008)0.181 (0.001)

Table 3. Root-mean-square error (RMSE) on eight molecular property prediction tasks from the QM9 BOOM benchmark. We compare our methods, DPA3-Scratch (training from scratch), DPA3-LP (linear probing), DPA3-FT (standard fine-tuning), and DPA3-MFT (multi-task fine-tuning), against ChemBERTa [15], Graphormer [80], EGNN [61], ET [70], MoLFormer [58], RT [9], Geoformer [72], Chemprop [27], MACE [36], ModernBERT [73], and GotenNet [4]. The best and second-best results are highlighted in bold and _underlined_, respectively. Black text denotes the best in-distribution (ID) performance, while blue text denotes the best out-of-distribution (OOD) performance. All DPA3 results are averaged over three independent training runs.

OOD SplitDatasetsCGCNNMEGNetSchNetDimeNet++ALIGNNDeeperGATGNNcoGNcoNGNDPA3-FTDPA3-MFT
LOCOdielectric0.51442.68302.70742.77200.85920.59110.49840.49830.53050.4690
LOCOelasticity0.05851.44681.40651.42420.09740.11730.10170.10190.0806_0.0774_
LOCOperovskites0.06511.46541.46441.46660.03860.03650.06310.0632_0.0350_0.0341
SparseXclusterdielectric0.60062.80442.83682.93780.69531.4056_0.5241_0.52420.60690.4806
SparseXclusterelasticity0.04991.41131.34551.35620.08340.11090.14160.14170.0754_0.0689_
SparseXclusterperovskites0.08391.44851.45091.45670.04680.04640.13060.13060.0384_0.0403_
SparseXsingledielectric0.98883.83843.97673.86291.51151.5755_0.5519_0.55181.53351.2463
SparseXsingleelasticity0.08951.30991.23631.3214_0.0853_0.11400.29190.2918_0.0783_0.0782
SparseXsingleperovskites0.06891.50051.51331.52480.04570.03730.21990.2200_0.0370_0.0358
SparseYclusterdielectric0.52542.59182.59882.59470.43590.39590.46730.4674_0.3354_0.2688
SparseYclusterelasticity0.07521.56591.55921.54540.06310.08580.08230.0823_0.0529_0.0527
SparseYclusterperovskites0.07401.46851.47361.47320.0341_0.0333_0.09250.09260.03220.0300
SparseYsingledielectric0.47772.57062.55662.58660.25130.27330.35890.3589_0.2008_0.1790
SparseYsingleelasticity0.08401.44911.48551.4828_0.0450_0.08070.18520.18530.04770.0435
SparseYsingleperovskites0.07681.50481.51731.5148_0.0243_0.02590.18840.18860.02350.0212
Randomdielectric0.59880.33910.32770.34000.34490.33550.30880.3142_0.3082_0.2639
Randomelasticity0.08950.08710.07960.07920.07150.0903_0.0689_0.06700.06270.0609
Randomperovskites0.04520.03520.03420.03760.02880.02880.02690.02900.0272_0.0270_

Table 4. Mean absolute error (MAE) of different GNN models on the dielectric, elasticity, and perovskite datasets from MatBench. Evaluation includes 50-fold cross-validation over five distinct out-of-distribution (OOD) settings introduced in Ref. [48] (with detailed descriptions of the OOD splitting strategies provided in Section C.2), as well as a 5-fold random split for in-distribution (ID) comparison. We benchmark DPA3-FT (standard fine-tuning) and DPA3-MFT (multi-task fine-tuning) against CGCNN [77], MEGNet [12], SchNet [62, 63], DimeNet++ [24], ALIGNN [16], DeeperGATGNN [49], coGN [59], and coNGN [59]. Best and second-best performances are highlighted in bold and <u>underlined</u>, respectively.

Results

Overview of MFT

We provide an overview of different downstream adaptation strategies used in this work in Figure 1. Each downstream adaptation setup is compared against training from scratch (Scratch) baseline, in which both the backbone and the task head are randomly initialized, as shown in Figure 1(a). During FT, the pretrained backbone is transferred and coupled with a randomly initialized, task-specific prediction head, with all model parameters updated end-to-end, as depicted in Figure 1(b). Furthermore, we investigate the performance of LP model, i.e. freezing the backbone of the pretrained model and fine-tuning only the task-specific head, as illustrated in Figure 1(c). This setting constrains the representation to remain fixed and therefore free of distributional drift.

In this work, we propose MFT, as graphically displayed in Figure 1(d). Similar to FT, the backbone of the pretrained model is transferred to the downstream task. However, under MFT the backbone is connected to two task heads: one randomly initialized for downstream property prediction and the other inherited from the force-field prediction head used during pretraining. Both task heads, together with the shared backbone, are jointly optimized on the downstream property objective and an auxiliary force-field objective. The auxiliary pretraining task can be selected to align closely with the target downstream domain. For example, when fine-tuning for molecular property prediction, the SPICE2 dataset [20], which is designed for training machine-learning interatomic potentials for organic molecules, can be used as the auxiliary force-field task. For materials property prediction, the OMat24 dataset [6], which is proposed for universal materials machine-learning potentials with a particular emphasis on structural and compositional diversity, can be chosen as the corresponding force-field task. This joint optimization strategy constrains the backbone to learn a representation space that is simultaneously optimal for the specific downstream property prediction task while preserving the broad chemical and configurational prior knowledge acquired during pretraining. Table 1 summarizes the comparison between these methods, while § 4 provides a comprehensive description of each approach.

MethodBackbone InitializationUpdate BackboneUpdate Task HeadAuxiliary Task
ScratchRandom✓\checkmark✓\checkmark×\times
FTPretrained✓\checkmark✓\checkmark×\times
LPPretrained×\times✓\checkmark×\times
MFT (Ours)Pretrained✓\checkmark✓\checkmark✓\checkmark

Table 1. Comparison of downstream adaptation strategies. The table highlights the differences between training from scratch (Scratch), fine-tuning (FT), linear probing (LP), and our proposed multi-task fine-tuning (MFT) in terms of backbone initialization, update parameters and the inclusion of auxiliary training tasks.

We investigate the fine-tuning behavior of the pretrained GNN model DPA-3.1-3M [83] on QM9 molecular property prediction tasks. The results summarized in Figure 2 show that the performance gap between ID and OOD generalization is closely associated with representation collapse in the learned representation space. A similar representation-collapse phenomenon is also observed when fine-tuning the pretrained JMP model [67], with full results reported in Supplementary Section B. These observations indicate that representation collapse is a broadly occurring failure mode in fine-tuning pretrained atomistic models, rather than an artifact of a specific model architecture or pretraining setup.

MFT closes the generalization gap

Data split. The QM9 dataset is partitioned into training, validation, ID test, and OOD test subsets. The training, validation, and ID test sets are drawn from the same distribution, whereas OOD samples are selected from the tails of the property distributions (the lowest 10% probability mass estimated via kernel density estimation), representing extreme or rare molecular properties. This splitting protocol follows established OOD benchmarks for molecular property prediction, namely BOOM [3].

Significant ID gains and OOD degradation under FT. Relative to training from scratch, FT yields an average improvement of 17.8% in ID accuracy, but only a 5.6% improvement in OOD accuracy, as reported in Figure 2(a). Across the eight property prediction tasks, the relative OOD improvement is consistently smaller than the corresponding ID improvement, with the sole exception of electronic spatial extent (⟨R2⟩\langle R^2 \rangle). Notably, for the dipole moment (μ\mu) and the lowest unoccupied molecular orbital (LUMO) energy, the fine-tuned model underperforms the from-scratch baseline in the OOD regime by 8.7% and 3.8%, respectively.

OOD enhancement and ID degradation of LP. As reported in Figure 2(b), the OOD accuracy of LP improves by 7.8% relative to a model trained from scratch, which is marginally higher than the improvement achieved by FT. In contrast, the ID generalizability decreases by 37% compared to the from-scratch baseline, indicating that the expressive capacity of the model is severely limited by freezing the backbone. This behavior is expected, as the target quantities predicted in downstream property prediction tasks differ substantially from those in force-field learning. Specifically, property prediction models aim to infer quantities such as electronic structure properties (HOMO, LUMO, GAP, and dipole moment) and thermodynamic properties (e.g. heat capacity), whereas force-field models are trained to predict ground-state energies and their derivatives with respect to atomic coordinates (forces). This fundamental shift in prediction paradigm necessitates adaptation of the representation space.

MFT closes the generalization gap. An effective fine-tuning strategy must allow representations to adapt to downstream task requirements while simultaneously preserving the chemical and configurational prior knowledge acquired during pretraining, and these requirements are fulfilled by the proposed MFT strategy. As presented in Figure 2(c), under MFT the relative improvement in OOD generalization (measured against training from scratch) reaches 17.1%, which closely matches the corresponding ID improvement (17.3%) and is comparable to the ID performance achieved by FT (17.8%). In the best-performing case (⟨R2⟩\langle R^{2}\rangle), OOD generalization improves by 36.1%, whereas in the worst case (μ\mu), OOD performance remains comparable to that of a model trained from scratch. In this sense, MFT exhibits no discernible generalization gap, and its OOD generalizability approaches the theoretical upper bound given by the ID generalizability.

MFT mitigates the collapse of representations

Pretrained atomic representations. Figure 2(d) visualizes the atomic (node) representations of the pretrained model. Atoms of different chemical species are indicated by distinct colors, while atoms of the same species are further differentiated by their number of covalent neighbors (see Supplementary Section D.1 for details). As shown in Figure 2(d), the atomic representations of the pretrained model are well aligned with chemical intuition: atoms of the same species and identical covalent environments form well-separated clusters, clearly distinguished from atoms of different species or coordination environments.

Collapse of atomic representations. After FT, however, substantial overlap emerges between representations of distinct species, as illustrated in Figure 2(e). For instance, the representations of nitrogen and oxygen atoms begin to intermingle. In addition, the resolution with respect to local covalent environments is markedly reduced. Carbon atoms with different numbers of covalent neighbors become mixed, oxygen atoms with two neighbors split into multiple clusters influenced by hydrogen and fluorine, and hydrogen atoms separate into clusters by single-bonded carbon cluster. The clarity and separation of the representation clusters can be quantitatively assessed using the Davies–Bouldin (DB) index [18], where lower values indicate more distinct and cohesive clusters. As shown in Figure 2(e), after FT, the DB index of the atomic representations increases by a factor of 6.2 relative to the pretrained representations, indicating a substantial blurring and loss of cluster separability. Together, these observations indicate a collapse of the chemically meaningful prior representations, both in chemical identity and local covalent environment, learned during pretraining after FT.

MFT mitigates the collapse of atomic representations. The OOD improvement of MFT is accompanied by a clear mitigation of the collapse of atomic representation. As illustrated in Figure 2(f), atomic representations corresponding to the same chemical species and covalent environments are clearly separated into distinct clusters. Only minor exceptions are observed: nitrogen atoms with three covalent neighbors are split into two clusters influenced by hydrogen. The DB index, which quantitatively measures cluster separation, increases by only 30%, in stark contrast to standard FT, where the DB index increases by 520% relative to that of the pretrained atomic representations.

Pretrained edge representations. As a GNN, DPA-3.1-3M employs edge representations to encode pairwise atomic relationships within a preset cutoff radius. Figure 2(g) visualizes the edge representations of carbon–hydrogen (C–H) and carbon–oxygen (C–O) pairs, to two prevalent edge types in the downstream tasks. These edges are further stratified into near, middle, and far categories according to their pairwise Euclidean distances: near corresponds to atom pairs forming covalent bonds, while middle and far denote nonbonded pairs whose features are updated with or without angular information, respectively. The pretrained model exhibits a highly organized edge representation, with edges distinctly clustered by both chemical identity and interatomic distance, indicating that the pretrained model encodes strong chemical and geometric priors.

Collapse of edge representations. In contrast, FT induces an evident structural degradation of this latent organization, as shown in Figure 2(h). Although near edges remain relatively cohesive due to strong covalent signals, the representations of middle and far edges increasingly overlap: C–O edge representations begin to mix with C–H representations, and the middle and far subclasses become entangled. This loss of configurational resolution is quantitatively reflected by a 1.3-fold increase in the DB index. Overall, the observed collapse indicates that the model’s ability to distinguish long-range spatial correlations is compromised, suggesting that FT prioritizes downstream label fitting at the expense of preserving chemically and geometrically meaningful representations.

MFT mitigates the collapse of edge representations. The proposed MFT method effectively mitigates the collapse of edge representation. As shown in Figure 2(i), edge representations corresponding to different pair types and interatomic distances remain well separated under MFT, with a DB index that is even 10% lower than that of the pretrained edge representations. This behavior contrasts sharply with FT, which leads to a 30% increase in the DB index.

MFT mitigates distribution shift between pretrained and fine-tuned representations

Significant representation drift under FT. We formally quantify the similarity, or equivalently the drift, between representation distributions of the pretrained and fine-tuned models using the 1-Wasserstein distance, denoted by WW. As shown in Table 2, the drift distances for the training and ID test distributions before and after FT are very close, which is expected since they are sampled from the same underlying distribution and therefore undergo similar modifications. In contrast, a significantly larger drift is observed for the OOD test distribution, indicating that FT induces substantially stronger representation changes for OOD samples than for

TrainIDOOD
WPretrain→FTW_{\mathrm{Pretrain}\to\mathrm{FT}}3.763.756.54
WPretrain→MFTW_{\mathrm{Pretrain}\to\mathrm{MFT}}1.511.491.97

Table 2. Quantification of distribution shift between pretrained and fine-tuned representations using 1-Wasserstein distance (WPretrain→FTW_{\mathrm{Pretrain}\to\mathrm{FT}}, WPretrain→MFTW_{\mathrm{Pretrain}\to\mathrm{MFT}}). Metrics are reported for the QM9 LUMO prediction task across training, ID, and OOD sets.

ID samples. This pronounced distributional shift provides clear evidence of overfitting induced by FT.

MFT mitigates representation drift. MFT significantly reduces the distribution shift between pretrained and fine-tuned representations. As reported in Table 2, under the MFT framework the representation distribution drifts for the training and ID test sets are 1.51 and 1.49, respectively, which are substantially smaller than those observed under FT (3.76 and 3.75, respectively). The OOD distribution drift is 1.97, which is only mildly larger than the corresponding training/ID drifts and remains significantly smaller than the OOD distribution drift (6.54) induced by FT.

However, eliminating representation drift entirely is counterproductive. As shown in § 2.2, LP which maintains zero representation drift by freezing the backbone, suffers from poor ID performance. This occurs because the model fails to bridge the paradigm shift between force-field pre-training and downstream property prediction. An optimal fine-tuning strategy must strike a balance: adapting representations to specific task requirements while preserving the foundational chemical priors learned during pretraining. Our results suggest that MFT successfully achieves this equilibrium.

MFT outperforms SOTA molecular property predictors

The proposed MFT strategy is benchmarked against state-of-the-art (SOTA) property prediction baselines on the BOOM QM9 benchmark, as summarized in Table 3. The MFT model (denoted as DPA3-MFT) is compared with standard fine-tuning (DPA3-FT) and linear probing (DPA3-LP) applied to the same pretrained backbone, namely DPA-3.1-3M; a schematic illustration of these settings is provided in Figure 1. For a fair comparison, the same model architecture is also trained from scratch (DPA3-Scratch). Both DPA3-FT and DPA3-MFT demonstrate a clear advantage in ID scenarios: they consistently achieve the best ID performance across all eight tasks, confirming that fine-tuning from a pretrained model outperforms SOTA task-specific training from scratch, and that the MFT strategy does not compromise ID accuracy. The OOD results, evaluated using property extrapolation splits following BOOM [3], highlight the primary strength of the MFT approach. DPA3-MFT achieves the best OOD performance in six out of the eight tasks and the second-best performance in one task. In contrast, DPA3-FT often exhibits a performance trade-off when generalizing to OOD data, likely due to the collapse of chemical priors encoded in the representations learned during pretraining, resulting in elevated OOD errors. By explicitly mitigating this effect, DPA3-MFT substantially improves robustness under distribution shift.

MFT outperforms SOTA material property predictors

To evaluate the robustness of the proposed MFT strategy on inorganic materials property prediction tasks, we conducted a comprehensive benchmark comprising an ID random split and five OOD splits constructed using structure- and property-based criteria [48]. These splitting strategies were applied to three datasets from MatBench [19]: refractive index (dielectric), shear modulus (elasticity), and perovskite formation energy (perovskites). A summary of the three datasets is provided in Table 7. The OOD split strategies leverage the Orbital-Field Matrix (OFM) [53] representation to encode electronic structure and orbital interactions, yielding a physically interpretable feature space for subsequent clustering and density estimation [31]. Specifically, the five OOD splitting methods include Leave-One-Cluster-Out (LOCO), which partitions the dataset into 50 folds via kk-means clustering in OFM feature space. In addition, four sparse-target strategies (SparseXSingIe, SparseX-cluster, SparseYSingIe, and SparseYcluster) identify samples with the lowest density either in feature space (X) or property space (Y) using Gaussian kernel density estimation. These low-density samples are further grouped into 50 clusters using kk-means clustering, and OOD test sets are constructed by selecting either a single representative point from each cluster (single) or the representative point together with its 10 nearest neighbors (cluster), resulting in 50 OOD folds. For each splitting method, the 50 OOD test sets are generated in a leave-one-out manner, where one fold is held out as the OOD test set while the remaining data are used for training and validation. Further details of the five OOD split strategies are provided in Supplementary Section C.2.

PropertySamplesUnitsDescription
dielectric4764UnitlessRefractive index
elasticity10987log10(GPa)Shear modulus
perovskites18928eV/Unit cellFormation energy

Table 7. Information of benchmarking properties of MatBench

As demonstrated in Table 4, DPA3-MFT achieves the best performance in 13 of the 18 evaluated tasks and ranks second in 4 additional tasks. In nearly all cases, DPA3-MFT outperforms the standard fine-tuning baseline (DPA3-FT), with an average improvement of 7.8%; the sole exception occurs for the perovskite SparseXcluster split. Compared with other SOTA models, DPA3-MFT maintains a consistent performance advantage, although certain competing methods benefit from task-specific architectural design. Notably, DPA3-MFT leads OOD performance across all property-outlier-based split strategies (SparseYsingle and SparseYcluster). Exceptions arise for structure-based OOD splits, including elasticity under LOCO and SparseXcluster, and dielectric under SparseXsingle. In particular, the relatively poor performance of DPA3-MFT on the dielectric SparseXsingle split is attributable to an exceptionally high MAE in one of the 50 folds as shown in Figure 6, which disproportionately inflates the average test error. The most pronounced gains from MFT are observed on the dielectric dataset, where DPA3-MFT improves upon DPA3-FT by 16.0% (compared with 4.1% and 3.1% improvements for elasticity and perovskites, respectively). This behavior likely arises because the perovskites and elasticity tasks predict perovskite formation energies and shear moduli (the latter being closely related to second derivatives of the energy), which are intrinsically aligned with the energy and force prediction objectives used during pretraining. In contrast, the dielectric (refractive index) task probes electronic structure properties, representing a substantially larger task-domain shift relative to force-field pretraining. As a result, successful adaptation requires more substantial representation updates, making the preservation of chemically meaningful priors acquired during pretraining particularly critical; MFT is therefore especially effective in this regime.

Image not extracted

Figure 6. Distribution of the MAEs for each fold of DPA3-FT and DPA3-MFT on the dielectric dataset across five different OOD splits.

MFT enhances data efficiency and few-shot OOD generalization

To evaluate the impact of downstream data size on model performance, we investigated the data efficiency of MFT using the QM9 band gap (GAP) prediction task and MatBench refractive index (dielectric) prediction task. As depicted in Figure 3(a-Figure 3b), MFT is compared against two baselines: training from scratch and FT. In these experiments, the size of the training set is varied by randomly down-sampling the original training data to as few as 114 data points for GAP and 14 data points for dielectric. The ID and OOD test root-mean-square errors (RMSEs) or mean absolute errors (MAEs) are plotted as functions of the training data size, which spans three orders of magnitude. The implementation details are provided in Supplementary Section D.4.

Image not extracted

Figure 3. Data efficiency and ablation study of multi-task fine-tuning (MFT). (a) Data efficiency on the QM9 band-gap (GAP) prediction task. (b) Data efficiency on the MatBench dielectric prediction task. In (a,b), MFT is compared with training from scratch (Scratch) and standard fine-tuning (FT); in-distribution (ID) and out-of-distribution (OOD) test RMSEs or MAEs are shown as functions of training set size. (c–e) Ablation study of MFT on the QM9 HOMO, LUMO, and GAP prediction tasks. (c) FT baseline. (d) Ablation of transferred pretrained components. (e) Ablation of auxiliary force-field task selection. “Avg. ↑\uparrowRMSE” denotes the average RMSE increase across the HOMO, LUMO, and GAP prediction tasks relative to the baseline MFT configuration.

Figure 3(a) illustrates the results of QM9 GAP prediction task. In terms of ID accuracy, both FT and MFT consistently outperform training from scratch across all data scales. MFT maintains a modest but consistent advantage over FT, although this gap narrows when the training data size reaches 50% of the full dataset. In contrast, for OOD accuracy, MFT significantly outperforms both the from-scratch and FT baselines across all training sample sizes, demonstrating a strong few-shot generalization capability. Notably, with only 1/128 of the training data, MFT achieves an OOD accuracy comparable to that of a from-scratch model trained on the full dataset. Moreover, MFT requires only 1/16 of the training data to match the OOD performance of an FT model trained using the full dataset.

The MatBench dielectric prediction results, as illustrated in Figure 3(b), further validate these findings. For ID accuracy, both MFT and standard FT consistently outperform training from scratch across all data regimes. Although MFT maintains a performance advantage over FT, this gap narrows in the extremely low–data regime. For OOD performance, MFT substantially outperforms both FT and training from scratch at all training set sizes. Notably, MFT exhibits exceptional data efficiency, achieving OOD accuracy comparable to from-scratch and FT models trained on the full dataset using only 1/128 and 1/16 of the data, respectively. These trends closely mirror those observed for the GAP task.

Given that labeled data for molecular and materials property prediction is typically sparse and computationally expensive to generate, for example, accurate band gaps often require costly hybrid exchange-correlation functionals, this robust performance across a wide range of data regimes highlights

the practical utility of MFT for real-world molecular and materials design applications.

Ablation Study

We conducted ablation experiments on the QM9 HOMO, LUMO, and GAP prediction tasks to disentangle the contributions of knowledge transfer and to assess the impact of auxiliary task selection. The baseline MFT configuration uses SPICE2 as the auxiliary force-field task and transfers both the pretrained backbone and the force-field task head. We further consider variants in which the pretrained task head is not transferred, as well as variants in which neither the pretrained backbone nor the pretrained task head is transferred. In addition, we examine the sensitivity of MFT performance to the choice of auxiliary force-field task by replacing SPICE2 with datasets from other research domains, including OMat24 (inorganic materials), OC20 (heterogeneous catalysis), and ODAC23 (metal–organic frameworks, MOFs).

As illustrated in Figure 3(d), transferring the force-field task head is not strictly necessary, resulting in only marginal increases in RMSE of 0.1% (ID) and 0.4% (OOD) when the task head is replaced with a randomly initialized one. By contrast, omitting transfer of both the pretrained backbone and the task head markedly degrades performance, increasing RMSE by 38.0% in ID and 10.6% in OOD relative to the baseline. Notably, even without explicit knowledge transfer, the OOD performance achieved by joint multi-task training remains superior to that of FT as shown in Figure 3(c), highlighting the intrinsic effectiveness of the multi-task framework in aligning downstream property prediction with the chemical and configurational priors encoded by force-field training tasks.

We further investigated the influence of domain alignment between the auxiliary force-field task and the downstream property prediction task. A clear negative transfer effect is observed when the auxiliary domain diverges substantially from the target task. The results are summarized in Figure 3(e). While the ID performance remains relatively stable, the OOD performance degrades markedly across all three tasks, with average RMSE increases of 20.3%, 17.8%, and 22.4% when using OMat24, OC20M, and ODAC23 as auxiliary tasks, respectively. Notably, the OOD performance obtained with OMat24 and ODAC23 is even inferior to that of FT. In contrast, OC20M yields a slight improvement in OOD performance relative to FT, which can likely be attributed to the partial structural overlap between catalytic adsorbates and the small organic molecules in QM9, suggesting that limited structural similarity can still benefit OOD generalization. These observations are consistent with prior findings in the literature: knowledge transfer is not universally beneficial, and indiscriminate transfer across mismatched domains can induce negative transfer [14], wherein unrelated auxiliary data degrades downstream model performance.

Discussion

Pretrained atomistic models offer a promising unified foundation for molecular and materials property prediction; however, their practical impact critically depends on reliable generalization beyond the training domain. In this work, we identify and systematically analyze a critical yet underexplored failure mode of fine-tuning pretrained atomistic models – representation collapse. This phenomenon involves the degradation of chemically and geometrically meaningful priors acquired during pretraining, leading to a substantial deterioration in out-of-distribution (OOD) generalization performance.

To address this challenge, we propose multi-task fine-tuning (MFT), which enables task-specific adaptation while preserving general-purpose chemical and configurational priors by jointly optimizing the downstream property-prediction objective together with an auxiliary force-field task inherited from pretraining. Across a wide range of OOD settings, MFT consistently mitigates representation collapse and achieves state-of-the-art OOD performance without compromising in-distribution accuracy. Moreover, the strong data efficiency of MFT highlights its practical relevance for real-world molecular and materials discovery, where labeled data are costly to obtain and extrapolation beyond known regimes is the norm rather than the exception.

Our analysis provides mechanistic insight into both the failure of standard fine-tuning and the success of MFT. We show that OOD performance is strongly correlated with the degree of representation drift from the pretrained model, and that effective adaptation requires a careful balance between plasticity (needed to fit downstream targets) and stability (needed to retain pretrained knowledge). Linear probing enforces stability but lacks expressiveness, whereas standard fine-tuning offers greater expressiveness at the cost of representation collapse. By contrast, MFT occupies a favorable middle ground, enabling controlled representation adaptation guided by physically meaningful auxiliary objectives.

We anticipate that these insights will inform the design of next-generation atomistic models for molecular and materials discovery, where reliable extrapolation to previously unexplored chemical regimes is essential. By enabling fine-tuning without forgetting, the proposed framework advances atomistic foundation models toward fulfilling their role as dependable engines for data-efficient and predictive scientific discovery.

Methods

This section describes the pretraining procedure and the fine-tuning strategies, both standard fine-tuning and multi-task fine-tuning, employed in this work.

Pretraining

In this work, the large atomistic model DPA-3.1-3M [83] is pretrained on the OpenLAM-v1 collection [1], which consists of datasets labeled with energies and atomic forces computed using density functional theory (DFT) [33] [28]. The OpenLAM-v1 collection spans a broad range of systems, including large-scale materials datasets such as OMat24 [6], molecular datasets such as SPICE2 [20], as well as other publicly available datasets contributed by the community. A complete list of datasets included in OpenLAM-v1 is provided in Ref. [52]. We denote the datasets in the collection by {Dm}m=1M\{D_m\}_{m=1}^{M}, where the OpenLAM-v1 collection comprises a total of M=31M = 31 datasets. Each dataset Dm={(xmk,Emk,Fmk)}k=1KmD_m = \{(x_{mk}, E_{mk}, F_{mk})\}_{k=1}^{K_m} contains KmK_m configurations, where each data point consists of a three-dimensional atomic structure xmkx_{mk}, its DFT-computed ground-state potential energy EmkE_{mk}, and the corresponding atomic forces FmkF_{mk}. Formally, each structure is represented as xmk=(Rmk,Zmk)x_{mk} = (R_{mk}, Z_{mk}), where Rmk∈RNmk×3R_{mk} \in\mathbb{R}^{N_{mk} \times3} denotes the Cartesian coordinates of the NmkN_{mk} atoms, and Zmk∈NNmkZ_{mk} \in\mathbb{N}^{N_{mk}} denotes their atomic numbers. For structures under periodic boundary conditions, the simulation cell tensor is included in the structural representation xmkx_{mk}, and the corresponding DFT-computed virial tensor is included in the labels. For simplicity of exposition, we do not explicitly discuss the cell tensor or the virial tensor in the remainder of this work.

The GNN backbone used to learn atomistic representations is denoted by fθf_\theta, with trainable parameters θ\theta, while the task-specific fitting network is denoted by gϕg_\phi, with trainable parameters ϕ\phi. The predicted potential energy is given by

E^mk=gϕ(fθ(xmk),cm),(1)\hat{E}_{mk} = g_\phi\left(f_\theta(x_{mk}), c_m\right), \tag*{(1)}

where cmc_m denotes the one-hot encoding of the dataset DmD_m. Atomic forces are obtained as the negative gradient of the predicted potential energy with respect to the atomic coordinates,

F^mk=−∇RmkE^mk.(2)\hat{F}_{mk} = -\nabla_{R_{mk}} \hat{E}_{mk}. \tag*{(2)}

The empirical pretraining objective is defined as

Lpre(θ,ϕ)=∑m∈Bc∑k∈BmλELE(E^mk,Emk)+λFLF(F^mk,Fmk),(3)\mathcal{L}_{\mathrm{pre}}(\theta, \phi) = \sum_{m \in\mathcal{B}_c} \sum_{k \in\mathcal{B}_m} \lambda_E \mathcal{L}_E(\hat{E}_{mk}, E_{mk}) + \lambda_F \mathcal{L}_F(\hat{F}_{mk}, F_{mk}), \tag*{(3)}

where Bc\mathcal{B}_c denotes a mini-batch of datasets randomly sampled from the dataset collection, and Bm\mathcal{B}_m denotes a mini-batch of configurations randomly sampled from dataset DmD_m. Here, LE\mathcal{L}_E and LF\mathcal{L}_F are error metrics that quantify the discrepancy between the predicted and DFT reference energies and forces, respectively, while λE\lambda_E and λF\lambda_F are weighting coefficients that balance the contributions of the two loss terms. The pretraining task is thus defined as minimizing the loss in Eq. (3):

(Pretraining) \quad θ0,ϕ0=arg min⁡θ,ϕ Lpre(θ,ϕ)\theta_0, \phi_0 = \underset{\theta,\phi}{\operatorname{arg\,min}}\ \mathcal{L}_{\mathrm{pre}}(\theta,\phi).

θ0,ϕ0=arg min⁡θ,ϕ Lpre(θ,ϕ).(4)\theta_0, \phi_0 = \underset{\theta,\phi}{\operatorname{arg\,min}}\ \mathcal{L}_{\mathrm{pre}}(\theta,\phi). \tag*{(4)}

The pretrained parameters of the backbone and the task head are denoted by θ0\theta_0 and ϕ0\phi_0, respectively.

Fine-tuning

For a downstream property prediction task, we consider a labeled dataset D={(xk,yk)}k=1K\mathcal{D} = \{(x_k, y_k)\}_{k=1}^{K}, where yk∈Rdy_k \in\mathbb{R}^{d} denotes a molecular or materials property of interest (e.g., bandgap or dielectric constant). The dimensionality of the target property is denoted by d≥1d \ge1, where d>1d > 1 allows for simultaneous prediction of multiple properties. In this work, we restrict our study to the case d=1d = 1, i.e., only a single property is predicted at a time. We denote the task-specific prediction head by hψh_\psi. The resulting property prediction model is defined as

y^k=hψ(fθ(xk)).(5)\hat{y}_k = h_\psi\left(f_\theta(x_k)\right). \tag*{(5)}

The supervised loss for property prediction is given by

Ltask(θ,ψ)=∑k∈BLP(y^k,yk),(6)\mathcal{L}_{\mathrm{task}}(\theta,\psi) = \sum_{k \in\mathcal{B}} \mathcal{L}_P(\hat{y}_k,y_k), \tag*{(6)}

where B\mathcal{B} denotes a mini-batch sampled from the dataset D\mathcal{D}, and LP\mathcal{L}_P is an error metric that quantifies the discrepancy between the predicted and reference property values. In the following, we provide the mathematical formulations of the four training strategies demonstrated schematically in Figure 1.

From scratch. As a baseline, all model parameters {θ,ψ}\{\theta,\psi\} are randomly initialized, and the property prediction model is trained by minimizing the loss in Eq. (6):

(Scratch) \quad min⁡θ,ψ Ltask(θ,ψ)\underset{\theta,\psi}{\operatorname{min}}\ \mathcal{L}_{\mathrm{task}}(\theta,\psi)

min⁡θ,ψ Ltask(θ,ψ).(7)\underset{\theta,\psi}{\operatorname{min}}\ \mathcal{L}_{\mathrm{task}}(\theta,\psi). \tag*{(7)}

This training strategy serves as a control experiment to quantify the actual performance gains provided by pretraining. The performance of models trained from scratch is often limited by the size of the downstream dataset D\mathcal{D}, as graph neural networks tend to overfit when optimized on small-scale labeled data without informative prior representations. In practice, early stopping based on validation performance is applied to mitigate overfitting.

Linear probing (LP). In LP, the pretrained backbone representation is treated as a frozen featurization module, and only the parameters of the task-specific prediction head are optimized:

(LP) \quad min⁡ψ Ltask(θ0,ψ),\underset{\psi}{\operatorname{min}}\ \mathcal{L}_{\mathrm{task}}(\theta_0,\psi),

min⁡ψ Ltask(θ0,ψ),(8)\underset{\psi}{\operatorname{min}}\ \mathcal{L}_{\mathrm{task}}(\theta_0,\psi), \tag*{(8)}

Initialization: ψ←rand.⁡\psi\leftarrow\operatorname{rand.},

where ψ←rand.⁡\psi\leftarrow\operatorname{rand.} indicates that the task-head parameters ψ\psi are randomly initialized before training. LP is computationally efficient and preserves the robust representations learned during pretraining. However, freezing the backbone also limits the model’s expressiveness and its ability to adapt representations to downstream tasks that exhibit a substantial shift in prediction objectives relative to the pretraining tasks.

Fine-tuning (FT). In standard FT, the backbone parameters are initialized from the pretrained weights θ0\theta_0, while the task-specific head parameters are randomly initialized. All parameters {θ,ψ}\{\theta,\psi\} are then updated end-to-end using the downstream task objective:

(FT) \quad min⁡θ,ψ Ltask(θ,ψ),\underset{\theta,\psi}{\operatorname{min}}\ \mathcal{L}_{\mathrm{task}}(\theta,\psi),

min⁡θ,ψ Ltask(θ,ψ),(9)\underset{\theta,\psi}{\operatorname{min}}\ \mathcal{L}_{\mathrm{task}}(\theta,\psi), \tag*{(9)}

Initialization: θ←θ0, ψ←rand.⁡\theta\leftarrow\theta_0,\ \psi\leftarrow\operatorname{rand.}

Although this optimization allows the representations to adapt to the downstream task and better fit the target labels yiy_i, it often induces representation collapse, as discussed in § 2.2.

Multi-task fine-tuning (MFT). MFT reintroduces the original pretraining task (energy and force prediction) as an auxiliary objective during fine-tuning. When no prior knowledge about the downstream dataset is available, all pretraining datasets can be included in the auxiliary objective. Alternatively, for molecular property prediction tasks, molecular pretraining datasets such as SPICE2 may be selected, while for materials property prediction tasks, materials pretraining datasets such as OMat24 can be used. We denote the selected pretraining dataset by Dm∗D_{m^*} and define the regularization loss as

Lreg(θ,ϕ)=∑k∈Bm∗λELE(E^m∗k,Em∗k)+λFLF(F^m∗k,Fm∗k),(10)\mathcal{L}_{\mathrm{reg}}(\theta,\phi)=\sum_{k\in B_{m^*}}\lambda_E L_E(\hat{E}_{m^*k},E_{m^*k})+\lambda_F L_F(\hat{F}_{m^*k},F_{m^*k}), \tag*{(10)}

where Bm∗B_{m^*} denotes a mini-batch of configurations sampled from dataset Dm∗D_{m^*}. The total MFT objective is then defined as

LMFT(θ,ψ,ϕ)=Ltask(θ,ψ)+λregLreg(θ,ϕ),(11)\mathcal{L}_{\mathrm{MFT}}(\theta,\psi,\phi)=\mathcal{L}_{\mathrm{task}}(\theta,\psi)+\lambda_{\mathrm{reg}}\mathcal{L}_{\mathrm{reg}}(\theta,\phi), \tag*{(11)}

where λreg\lambda_{\mathrm{reg}} controls the relative weight of the regularization loss with respect to the downstream task loss.

The optimization problem is given by

min⁡θ,ψ,ϕLMFT(θ,ψ,ϕ),Initialization:θ←θ0, ψ←rand.⁡, ϕ←ϕ0,(12)\begin{aligned} \min_{\theta,\psi,\phi}\quad&\mathcal{L}_{\mathrm{MFT}}(\theta,\psi,\phi),\\ \text{Initialization}:\quad&\theta\leftarrow\theta_0,\ \psi\leftarrow\operatorname{rand.},\ \phi\leftarrow\phi_0, \tag*{(12)} \end{aligned}

where all parameters {θ,ψ,ϕ}\{\theta,\psi,\phi\} are jointly optimized. The backbone parameters θ\theta and the force-field task head parameters ϕ\phi are initialized from their pretrained values, while the downstream task head parameters ψ\psi are randomly initialized. As demonstrated in § 2.2, this MFT scheme effectively mitigates representation collapse by preserving the chemical and configurational priors learned during pretraining.

Acknowledgments

The computational resources utilized in this work were provided by the AI for Science Institute. The work of Chengqian Zhang, Duo Zhang, Yuzhi Zhang, Guolin Ke, and Han Wang was supported by the National Key R&D Program of China (Grant No. 2022YFA1004300). Han Wang was additionally supported by the National Natural Science Foundation of China (Grant Nos. 12525113 and 12561160120). Tiejun Li acknowledges the support from National Key R&D Program of China under grant 2021YFA1003301, and National Science Foundation of China under grant 12288101.

Data availability

The training and inference codes of multi-task fine-tuning, standard fine-tuning and linear probing on DPA3 model are available in the DeePMD-kit repository (https://github.com/deepmodeling/deepmd-kit). The scripts used to reproduce the reported results and figures in this paper are available in https://github.com/Chengqian-Zhang/Multitask-finetuning.

Related Work

Pretraining and fine-tuning for property predictions. In the pretraining stage, atomistic representations are learned from large-scale datasets and subsequently transferred to downstream tasks. At present, two mainstream paradigms dominate pretraining for 3D atomistic systems. The first paradigm is self-supervised pretraining, which typically focuses on denoising-based objectives applied to equilibrium structures, where per-atom forces are close to zero [81, 85, 22]. This line of work has recently been extended to non-equilibrium configurations by explicitly incorporating atomic force information into the learning objective [43, 45]. The second paradigm is supervised pretraining, which directly trains models on large-scale molecular or materials datasets annotated with energies, forces, and, in some cases, stress tensors [78, 30, 67, 34].

In the fine-tuning stage, most of the works mentioned above adopt a standard fine-tuning paradigm, in which the backbone parameters are initialized from a pretrained model and jointly optimized with an additional task-specific prediction head. JMP [67] further employs layer-wise learning rate decay (LLRD) [29] to alleviate overfitting. Nevertheless, in this work we still observe pronounced representation collapse in JMP-L, underscoring the persistent challenge of safe adaptation in downstream tasks. RoFt-Mol [44] provides a systematic evaluation of existing fine-tuning strategies across a wide range of molecular property prediction tasks and demonstrates that the optimal fine-tuning mechanism strongly depends on both the pretraining objective and the downstream task type. In contrast to these studies, we focus specifically on improving OOD generalization of pretrained 3D atomistic models, an underexplored yet critical problem in realistic molecular and materials design scenarios.

Representation collapse in CV and NLP. In computer vision (CV) and natural language processing (NLP), foundation models have demonstrated remarkable zero-shot and transfer capabilities [13, 54, 26]. However, a growing body of work has shown that adapting these pretrained models to specific downstream tasks often induces representation collapse, accompanied by a pronounced degradation in OOD generalization [2, 37, 75]. To mitigate this issue, a variety of strategies have been proposed to preserve pretrained knowledge and reduce overfitting during fine-tuning. These include optimization-based constraints, such as using a smaller learning rate than that employed during pretraining [35, 40] or adopting layer-wise learning rate schedules [57, 65], as well as architectural constraints that freeze subsets of model parameters while fine-tuning only selected components [38, 21]. Other approaches seek to update the entire model by interpolating between pretrained and fine-tuned weights [75], or by explicitly penalizing deviations in the latent representation manifold [42]. In addition, incorporating auxiliary training objectives inherited from the pretraining stage has proven effective in improving fine-tuning robustness and generalization [25, 46, 50]. Our proposed MFT follows this paradigm, but grounds the auxiliary objective in physically meaningful force-field supervision, making it particularly well suited for molecular and materials property prediction.

OOD generalization in molecular and materials design. OOD generalization is a central yet insufficiently explored challenge in molecular and materials discovery. Recent large-scale benchmarks demonstrate that state-of-the-art molecular property predictors suffer substantial performance degradation under OOD settings relative to ID evaluation [3, 66], a trend that is likewise observed in inorganic materials prediction [48, 41]. Prior efforts to address this limitation have primarily focused on uncertainty quantification (UQ) to assess predictive reliability [69], and on leveraging UQ-guided active learning to generate new training data and improve model generalization [79]. Complementary approaches incorporate large-scale unlabeled data via semi-supervised learning [76] or adapt bilinear transduction frameworks to enhance OOD robustness in both molecular and materials domains [64, 47]. Despite these advances, OOD generalization has rarely been examined in the context of fine-tuning large pretrained atomistic models for downstream property prediction. In particular, the systematic degradation of OOD performance during fine-tuning, its connection to representation collapse, and principled mitigation strategies remain largely unexplored.

Representation collapse of JMP-L

In this section, we investigate the representation collapse when fine-tuning the pretrained JMP-L model [67]. We use the exactly same data split of QM9 dataset as DPA-3.1-3M, as described in § 2.2. Relative to training from scratch, fine-tuning yields an average improvement of 16.9% in ID accuracy, but a 9.7% improvement in OOD accuracy, as reported in Figure 4. Across the eight property prediction tasks, the relative OOD improvement is consistently smaller than the corresponding ID improvement, with the sole exception of electronic spatial extent (⟨R2⟩\langle R^2 \rangle), which aligns with the observation of DPA-3.1-3M as shown in Figure 2(b). Notably, for the isotropic polarizability (α\alpha) and the dipole moment (μ\mu), the fine-tuned model underperforms the from-scratch baseline in the OOD regime by 3.6% and 13.6%, respectively. Notably, the OOD improvement of JMP-L is

Image not extracted

Figure 4. Analysis of representation collapse of JMP-L. Relative performance improvements of fine-tuning (FT) over the training from Scratch baseline.

more than DPA-3.1-3M, it is reasonable because JMP-L utilize both Layer-wise Learning Rate Decay (LLRD) strategy and a smaller learning rate than pretraining to mitigate overfitting. However, we still observe representation collapse and OOD performance degradation phenomenon in JMP-L, showing the challenge of safe adaptation for downstream task.

Figure 5(a) visualizes the atomic(node) representations of the last backbone layer of the pretrained JMP-L model. The colors are following DPA-3.1-3M in Figure 2. As shown in Figure 5(a), atoms of the same species and identical covalent environments tend to form clusters, thought the clustering performance is less than DPA-3.1-3M as illustrated in Figure 2, such as some overlap between representations of oxygen and nitrogen. After fine-tuning, however, significant overlap emerges between no matter distinct species or different local covalent environments of one species as depicted in Figure 5(b). Almost all distinct species with different local environments mix together, with Davies-Bouldin(DB) index increasing by a factor of 3.9 relative to the pretrained representations, indicating a substantial blurring and loss of cluster separability.

Image not extracted

Figure 5. t-SNE visualizations of atomic (node) and edge representations representations of the last backbone layer of JMP-L on QM9 LUMO task, reduced to 2-dimension space. The Davies–Bouldin (DB) index values are normalized to the pretrained model baseline.

Figure 5(c) visualizes the edge representations of carbon-oxygen(C-O) pairs, which are further stratified to near, middle and far categories according to their pairwise Euclidean distances. As shown in Figure 5(c), we can distinct the three types of edges in the pretrained representation space, far edges cluster at left top and near edges cluster at right bottom, though there is some mixing between middle and far edges. However, fine-tuning induces an evident structural degradation of this latent organizations, as shown in Figure 5(d). The representations of middle and far edges overlap significantly. This loss of configurational resolution is quantitatively reflected by a 6.6-fold increase in the DB index.

Furthermore, we quantify the drift between representation distributions of the pretrained and fine-tuned JMP-L models using the 1-Wasserstein distance(WW). As shown in Table 5, like DPA-3.1-3M, the drift distances for the training and ID test distributions are very close. However, the drift of ID test distribution(7.12) is much larger than DPA-3.1-3M(3.75), which implies a significant representation modifications after fine-tuning. Besides, the OOD test distribution has a larger drift than training and ID test distribution, causing overfitting.

TrainIDOOD
WPretrain→FTW_{\mathrm{Pretrain}\to\mathrm{FT}}7.547.128.58

Table 5. Quantification of distribution shift between JMP-L pretrained and fine-tuned representations using 1-Wasserstein distance (WPretrain→FTW_{\mathrm{Pretrain}\to\mathrm{FT}}, WPretrain→MFTW_{\mathrm{Pretrain}\to\mathrm{MFT}}). Metrics are reported for the QM9 LUMO dataset across training, ID, and OOD sets.

Benchmark details

OOD data split details of QM9

For the model performance evaluation for small molecular dataset QM9 [55], we use the same data split as BOOM [3]. Training, ID test and OOD test splits are generated based on the underlying property distributions of the datasets. For each of the eight molecular properties in Table 6, OOD splits are constructed by fitting a Kernel Density Estimator (KDE) with a Gaussian kernel to the property values to determine the probability density of each molecule. Molecules with the lowest 10% of probability scores are assigned to the OOD split for the QM9 dataset, which effectively isolates samples at the tails of the distribution for typical molecular property distributions. Unlike traditional partitioning via fixed cut-off values, this density-based approach allows for the capture of low-probability samples within general distributions that may be non-unimodal or complex. The ID test sets are then generated by randomly sampling 10% sample from the remaining molecules, while all residual data is reserved for training.

PropertySamplesUnitsDescription
α\alpha130ka03a_0^3isotropic polarizability
CvC_v130kcalmol K\frac{cal}{mol\ K}heat capacity
HOMO130kHartreehighest occupied molecular orbital energy
LUMO130kHartreelowest unoccupied molecular orbital energy
GAP130kHartreeHOMO-LUMO gap
μ\mu130kDebyedipole moment
⟨R2⟩\langle R^2\rangle130ka02a_0^2electronic spatial extent
ZPVE130kHartreezero point vibrational energy

Table 6. Information of benchmarking properties of QM9

OOD data split details of MatBench

The following data splitting strategies on MatBench [19], originally developed by Ref. [48], were employed to evaluate model performance on inorganic materials across different OOD regions in the feature and property space. These methods rely on the Orbital-Field Matrix (OFM) [53], which represents materials by encoding electronic structures and orbital interactions to provide a physically interpretable feature space.

Leave-one-cluster-out (LOCO). In this approach, the dataset is partitioned into 50 clusters using the kk-means algorithm based on OFM features. Model evaluation is conducted iteratively, where each cluster serves as a test set while the remaining clusters are used for training and validation. This ensures that the model is tested on groups of materials that are structurally distinct from the training data.

Single-point targets with the lowest structure density (SparseXsingle). This method identifies structurally unique materials by projecting the 1024-dimensional OFM features into a 2D space using t-distributed Stochastic Neighbor Embedding (t-SNE). A Gaussian Kernel Density Estimation (KDE) is applied to this 2D manifold to identify the 500 samples with the lowest density. These samples are grouped into 50 clusters via kk-means, and one representative sample is extracted from each cluster to form a 50-point test set.

Single-point targets with the lowest property density (SparseYsingle). Focusing on outliers in the property distribution, this method estimates the density of the target property values (yy-values) using Gaussian Kernel Density Estimation. The 500 samples exhibiting the lowest property density are selected and partitioned into 50 clusters through kk-means. One sample is then selected from each cluster, yielding a test set that represents rare or extreme property regimes.

Cluster targets with the lowest structure density (SparseXcluster). This method extends the SparseXsingle logic by forming target clusters rather than individual points. After the 50 representative sparse samples are identified, the NN nearest neighbors for each sample are included in the test set. These neighbors are determined by the Euclidean distance in the OFM feature space, with a constraint ensuring that no sample is assigned to multiple clusters.

Cluster targets with the lowest property density (SparseYcluster). Similarly, this method builds upon the SparseYsingle approach. Once the 50 samples from low-density property regions are identified, the test set is expanded to include their NN nearest neighbors in the OFM feature space. This allows for the evaluation of model performance on localized regions of the material space associated with rare property values.

Furthermore, the 5-fold random split used in Table 4 are sourced from Ref. [19].

Detail results of MatBench

The fold-wise MAE plots for DPA3-FT and DPA3-MFT on the dielectric dataset, elasticity dataset, and perovskites dataset are presented in Figure 6, Figure 7 and Figure 8, respectively. The distribution of the MAE for 50 folds show that only a small subset of challenging folds are responsible for the overall MAE for both DPA3-FT and DPA3-MFT, which aligns with previously reported observations for baseline models in Ref. [48]. Despite these shared distributional characteristics, DPA3-MFT consistently outperforms both DPA3-FT and baseline models as shown in Table 4.

Image not extracted

Figure 7. Distribution of the MAEs for each fold of DPA3-FT and DPA3-MFT on the elasticity dataset across five different OOD splits.

Image not extracted

Figure 8. Distribution of the MAEs for each fold of DPA3-FT and DPA3-MFT on the perovskites dataset across five different OOD splits.

Implementation details

Representation analysis details

In Figure 2(d-Figure 2f) visualize the atomic(node) representations of the pretrained and fine-tuned model. Atoms of different chemical species are indicated by distinct colors, while atoms of the same species are further differentiated by their number of covalent neighbors. Here, we consider two atoms with a distance less than 1.15 times the sum of covalent radii as neighbors to each other. We randomly sample 2000 atomic representations for each type of local covalent environment of each atom species in OOD test set for next-step analysis. Subsequently, we apply the t-distributed stochastic neighbor embedding (t-SNE) [71] for dimension reduction converting the 128-dimension atomic representations to 2-dimension for visualization via scikit-learn package [51]. We calculate Davies–Bouldin (DB) [18] index in 2-dimension representation space to quantitatively assess the clarity and separation of the representation clusters via scikit-learn package. The definition of DB index is

DB index=1N∑i=1Nmax⁡i≠j(Si+SjMij)(13)\mathrm{DB\ index} = \frac{1}{N}\sum_{i=1}^{N}\max_{i\ne j}\left(\frac{S_i+S_j}{M_{ij}}\right) \tag*{(13)}

where MijM_{ij} is the distance between vectors which are chosen as characteristic of clusters ii and jj, and SiS_i and SjS_j are the dispersions of clusters ii and jj, respectively. Specifically, SiS_i and MijM_{ij} are defined as

Si={1Ti∑j=1Ti∣Xj−Ai∣q}1q,Mij={∑k=1n∣aki−akj∣p}1p(14)S_i = \left\{\frac{1}{T_i}\sum_{j=1}^{T_i}\lvert X_j-A_i\rvert^q\right\}^{\frac{1}{q}},\qquad M_{ij}=\left\{\sum_{k=1}^{n}\lvert a_{ki}-a_{kj}\rvert^p\right\}^{\frac{1}{p}} \tag*{(14)}

where TiT_i is the number of vectors in cluster ii, AiA_i is the centroid of cluster ii, akia_{ki} is the kkth component of the nn-dimension vector aia_i, which is the centroid of cluster ii. Here we define p=2p=2 and q=1q=1, which means MijM_{ij} is the Euclidean distance between centroids, SiS_i becomes the average Euclidean distance of vectors in cluster ii to the centroid of cluster ii. Lower DB index values indicate more distinct and cohesive clusters.

Both DPA-3.1-3M and JMP-L employ edge representations to encode pairwise atomic relationships within a preset cutoff radius. We visualize the edge representations of carbon-hydrogen (C-H) and carbon-oxygen (C-O) pairs, two prevalent edge types in the downstream tasks for DPA-3.1-3M, and visualize C-O edge representations for JMP-L. These edges are further stratified into near, middle, and far categories according to their pairwise Euclidean distances. Here, we consider the edge where the distance between atomic pairs is less than 1.15 times the sum of the covalent radii of two atoms to be the near edge, i.e. the atom pairs form covalent bonds. We consider the edge where the distance between atomic pairs between 3 angstroms and 4 angstroms is the middle edge. The atom pairs in middle edges are nonbonded but their features are updated with angular information with angular cutoff in DPA-3.1-3M being 4 angstroms. We consider the edge where the distance between 5 angstroms and 6 angstroms is the far edge, as the cutoff of DPA-3.1-3M is 6 angstroms. Compared to middle edges, far edges are also nonbonded but the atom pairs’ features are updated without angular information. Similar to the analysis to atomic representation, we randomly sample 2000 edge representations for each type of edge in OOD test set, then use t-SNE to convert high-dimension edge representations to 2-dimension space for visualization and use DB index to assess the clarity and separation of edge representation clusters.

We formally quantify the similarity, or equivalently the drift, between atomic representation distributions of the pretrained and fine-tuned models using the 1-Wasserstein distance (W1\mathcal{W}_1), briefly denoted by W\mathcal{W} in this paper. In practice, we sample 5000 atomic representations of the pretrained and fine-tuned models respectively. The discrete optimal transport problem is defined as:

W=min⁡γ∑i,jγi,jCi,j(15)\mathcal{W}=\min_{\gamma}\sum_{i,j}\gamma_{i,j}C_{i,j} \tag*{(15)}

where γ∈R5000×5000\gamma\in\mathbb{R}^{5000\times5000} is the optimal transport matrix, C∈R5000×5000C\in\mathbb{R}^{5000\times5000} is cost matrix with Ci,jC_{i,j} defined as the Euclidean distance between the ii-th pretrained representation and the jj-th fine-tuned representation in the origin 128-dimension space. The results are computed via the POT package [23].

Model architecture details

Following the notation in the main text, we denote the GNN backbone used to learn atomistic representations as fθf_\theta, with trainable parameters θ\theta, while denote the force-field task head and downstream property prediction head as gϕg_\phi and hψh_\psi, with trainable parameters ϕ\phi and ψ\psi, respectively. The architecture of fθf_\theta and gϕg_\phi is exactly the same as DPA-3.1-3M [83] and we will not elaborate details in this article.

For a downstream property prediction task, we consider a labeled training dataset D={(xk,yk)}k=1K\mathcal{D}=\{(x_k,y_k)\}_{k=1}^{K}, where each data point consists of a three-dimensional atomic structure xkx_k, and yk∈Rdy_k\in\mathbb{R}^{d} denotes a molecular or materials property of interest (e.g., bandgap or dielectric constant). Formally, each structure is represented as xk=(Rk,Zk)x_k = (R_k, Z_k), where Rk∈RNk×3R_k \in\mathbb{R}^{N_k \times3} denotes the Cartesian coordinates of the NkN_k atoms, and Zk∈NNkZ_k \in\mathbb{N}^{N_k} denotes their atomic numbers. The dimensionality of the target property is denoted by d≥1d \ge1, where d>1d > 1 allows for simultaneous prediction of multiple properties. In this work, we restrict our study to the case d=1d = 1, i.e., only a single property is predicted at a time. Generally, the formulation of hψh_\psi is

y^k=hψ(fθ(xk))={∑i=1Nk[MLP⁡(fθ(xk)i)+p(Zki)],if yk is extensive,1Nk∑i=1Nk[MLP⁡(fθ(xk)i)+p(Zki)],if yk is intensive.(16)\hat{y}_k = h_\psi(f_\theta(x_k)) = \begin{cases} \sum_{i=1}^{N_k} [\operatorname{MLP}(f_\theta(x_k)_i) + p(Z_{ki})], & \text{if } y_k \text{ is extensive},\\ \frac{1}{N_k}\sum_{i=1}^{N_k} [\operatorname{MLP}(f_\theta(x_k)_i) + p(Z_{ki})], & \text{if } y_k \text{ is intensive}. \end{cases} \tag*{(16)}

where MLP refers to Multilayer Perceptron, which is employed in this study with a specific configuration of 3 layers and 240 neurons per layer. Furthermore, p(Zki)p(Z_{ki}) is a constant bias to the atomic property contribution according to the atomic number ZkiZ_{ki}. This property bias are determined by a least-square fitting of the properties in the training data. More precisely, we have KK training data frames, and within the kk-th frame, we have NkzN_{kz} atoms with atom number zz, and the labeled property of the frame is denoted by yky_k. Then a linear system is solved in the least-square sense.

∑zNkzp(z)={yk,if yk is extensive,Nkyk,if yk is intensivek=1,…,K(17)\sum_z N_{kz}p(z) = \begin{cases} y_k, & \text{if } y_k \text{ is extensive},\\ N_k y_k, & \text{if } y_k \text{ is intensive} \end{cases} \qquad k = 1,\ldots,K \tag*{(17)}

We regard HOMO, LUMO, GAP in QM9 and dielectric, elasticity, perovskites in MatBench as intensive properties, while regard QM9 α\alpha, ZPVE and CvC_v as extensive properties. There are three special cases: dielectric, ⟨R2⟩\langle R^2\rangle and μ\mu. As to dielectric, we do not use the property bias term p(Zki)p(Z_{ki}) in Equation (16), because there are element species in the test set that did not appear in the training set. As to QM9 ⟨R2⟩\langle R^2\rangle and μ\mu, we use customized task head architecture to better utilize priors. Specifically, we use the same prediction head formulation as previous work [70, 67, 4] for ⟨R2⟩\langle R^2\rangle prediction and the norm of dipole moment vectors [84] for predictions of dipole moment scalars μ\mu.

Model hyperparameters

The hyperparameters of training from scratch, fine-tuning (FT) and multi-task fine-tuning (MFT) utilized for DPA3 model are illustrated in Table 8. DPA3 uses dynamic batch size based on the number of atoms NN in each specific system. Formally, Auto:X=⌈XN⌉X = \left\lceil\frac{X}{N} \right\rceil where NN denotes the number of systems in each system and ⌈⋅⌉\lceil\cdot\rceil denotes the ceiling function which rounds the number up to the nearest integer. Notation “→\rightarrow” appears in energy, force and virial loss prefactors indicate that the prefactors are changed in synchronization with the learning rate. Besides, DPA3 employs a customized activation function termed SiLUT(SiLU threshold with Tanh) to improve numerical stability during training, which is defined as:

Hyper-parametersQM9MatBench
GeneralOptimizerAdamAdam
Learning rate schedulingExpExp
Maximum learning rate1e-31e-3
Minimum learning rate1e-51e-5
Scratch/FTNumber of training steps0.1M0.1M
Loss functionMSESmooth MAE
Batch size per GPUAuto:512Auto:512
Number of GPUs11
Activation functionTanhTanh
MFTNumber of training steps0.1M0.2M
Force-field task headSPICE2OMat24
Energy loss prefactor0.2→200.2 \to200.2→200.2 \to20
Force loss prefactor100→60100 \to60100→60100 \to60
Virial loss prefactor-0.02→10.02 \to1
Property loss prefactor11
Batch size per GPU (property task)Auto:512Auto:512
Batch size per GPU (force-field task)Auto:128Auto:128
Activation function (property head)TanhTanh
Activation function (force-field head)SiLUT:3.0SiLUT:3.0
Number of GPUs21

Table 8. Hyper-parameters for DPA3 models trained on QM9 and MatBench.

SiLUT⁡:t(x)={SiLU⁡(x),if x≤t,Tanh⁡(a⋅(x−t)+b),if x>t.(18)\operatorname{SiLUT}:t(x) = \begin{cases} \operatorname{SiLU}(x), & \text{if } x \le t,\\ \operatorname{Tanh}(a \cdot(x-t) + b), & \text{if } x > t. \end{cases} \tag*{(18)}

where tt represents the threshold, indicating when to transition from SiLU to Tanh. Constants aa and bb are determined based on tt to ensure the first and second continuity of the activation function at the threshold.

Regarding the JMP model hyperparameters, we employed a batch size of 28 and conducted training over 24 epochs. All remaining hyperparameters were maintained in accordance with Ref. [67] for both training from scratch and fine-tuning.

Details of data efficiency experiments

The data efficiency of MFT is evaluated through a series of experiments on the QM9 band gap (GAP) and MatBench refractive index (dielectric) prediction tasks, as depicted in Figure 3(a) and Figure 3(b). In these experiments, the training set size is varied by randomly down-sampling the original data across three orders of magnitude. This approach tests model performance in extreme low-data regimes, reaching as few as 114 data points for the QM9 GAP task and 14 data points for the MatBench dielectric task.

For the QM9 GAP prediction task, we generate 3 independent training subsets for every data scale using different random seeds and 3 independent runs are performed to ensure statistical reliability. The results are reported using the root-mean-square errors (RMSEs) for both ID and OOD test sets, plotted as a function of the number of training data.

In the MatBench dielectric prediction task, we utilize random splits to evaluate ID performance and leave-one-cluster-out (LOCO) splits for OOD evaluation across varying training data scales. Unlike QM9 GAP which has one fold, MatBench dielectric employs multiple folds, specifically 5 folds for random splits and 50 folds for LOCO. In practice, we selected the specific fold whose mean absolute error (MAE) of test sets most closely approximated the mean MAE across all folds during full-dataset FT. This representative fold is used for subsequent data efficiency experiments.

For the random split experiments on the dielectric task, we maintain the same sampling strategy used for QM9 GAP, utilizing 3 independent training subsets and 3 runs for every data scale. However, to mitigate the higher variance inherent in the LOCO split for the dielectric task, we implement a more rigorous sampling strategy. Specifically, for data scales ranging from 1 down to 1/16, we conducted 6 independent runs using 6 different random subsets. For the smaller data scales, including 1/32, 1/64, 1/128, and 1/256, the number of trials is increased to 10 independent runs with 10 distinct random subsets. The resulting MAEs are plotted against the training data size to demonstrate the model’s data efficiency.

References

  1. [1]The openlam-v1 dataset, 2025. https://www.aissquare.com/datasets/detail?name=OpenLAM-TrainingSet-v1&id=308&pageType=datasets. 2, 10
  2. [2]Armen Aghajanyan, Akshat Shrivastava, Anchit Gupta, Naman Goyal, Luke Zettlemoyer, and Sonal Gupta. Better fine-tuning by reducing representational collapse. In International Conference on Learning Representations, 2021. 2, 17arxiv.org/abs/2008.03156
  3. [3]Evan R Antoniuk, Shehtab Zaman, Tal Ben-Nun, Peggy Li, James Diffenderfer, Busra Sahin, Obadiah Hersh Slomenski, Everett Grethel, Tim Hsu, Anna Hizspan-ski, Kenneth Chiu, Bhavya Kailkhura, and Brian Van Essen. BOOM: Benchmarking out-of-distribution molecular property predictions of machine learning models. In The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2025. 1, 2, 4, 6, 17, 19arxiv.org/abs/2505.01912
  4. [4]Sarp Aykent and Tian Xia. Gotennet: Rethinking efficient 3d equivariant graph neural networks. In The Thirteenth International Conference on Learning Representations, 2025. 7, 22
  5. [5]Fan Bao, Min Zhao, Zhongkai Hao, Peiyao Li, Chongxuan Li, and Jun Zhu. Equivariant energy-guided SDE for inverse molecular design. In The Eleventh International Conference on Learning Representations, 2023. 1arxiv.org/abs/2209.15408
  6. [6]Luis Barroso-Luque, Muhammed Shuaibi, Xiang Fu, Brandon M Wood, Misko Dzamba, Meng Gao, Ammar Rizvi, C Lawrence Zitnick, and Zachary W Ulissi. Open materials 2024 (omat24) inorganic materials datasets and models. arXiv preprint arXiv:2410.12771, 2024. 2, 3, 10
  7. [7]Ilyes Batatia, Philipp Benner, Yuan Chiang, Alin M Elena, Dávid P Kovács, Janosh Riebesell, Xavier R Advincula, Mark Asta, Matthew Avaylon, William J Baldwin, et al. A foundation model for atomistic materials chemistry. The Journal of chemical physics, 163(18), 2025.
  8. [8]Anton Bochkarev, Yury Losgorsky, and Ralf Drautz. Graph atomic cluster expansion for semilocal interactions beyond equivariant message passing. Physical Review X, 14(2):021036, 2024.
  9. [9]Jannis Born and Matteo Manica. Regression transformer enables concurrent sequence regression and generation for molecular language modelling. Nature Machine Intelligence, 5(4):432–444, 2023.DOI
  10. [10]Zhendong Cao, Xiaoshan Luo, Jian Lv, and Lei Wang. Space group informed transformer for crystalline materials generation. Science Bulletin, 2025.DOI
  11. [11]Lowik Channusott, Abhishek Das, Siddharth Goyal, Thibaut Lavril, Muhammed Shuaibi, Morgane Riviere, Kevin Tran, Javier Heras-Domingo, Caleb Ho, Weihua Hu, et al. Open catalyst 2020 (oc20) dataset and community challenges. ACS Catalysis, 11(10):6059–6072, 2021.
  12. [12]Chi Chen, Weike Ye, Yunxing Zuo, Chen Zheng, and Shyue Ping Ong. Graph networks as a universal machine learning framework for molecules and crystals. Chemistry of Materials, 31:3564–3572, 2019.
  13. [13]Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning, 2020.
  14. [14]Xinyang Chen, Sinan Wang, Bo Fu, Mingsheng Long, and Jianmin Wang. Catastrophic forgetting meets negative transfer: Batch spectral shrinkage for safe transfer learning. Advances in neural information processing systems, 32, 2019.
  15. [15]Seyone Chithrananda, Gabriel Grand, and Bharath Ramsundar. Chemberta: Large-scale self-supervised pretraining for molecular property prediction, 2020.arxiv.org/abs/2010.09885
  16. [16]Kamal Choudhary and Brian DeCost. Atomistic line graph neural network for improved materials property predictions. npj Computational Materials, 7(1):185, 2021.
  17. [17]Stefano Curtarolo, Gus LW Hart, Marco Buongiorno Nardelli, Natalio Mingo, Stefano Sanvito, and Ohad Levy. The high-throughput highway to computational materials design. Nature materials, 12(3):191–201, 2013.DOI
  18. [18]David L. Davies and Donald W. Bouldin. A cluster separation measure. IEEE Transactions on Pattern Analysis and Machine Intelligence, PAMI-1(2):224–227, 1979.
  19. [19]Alexander Dunn, Qi Wang, Alex Ganose, Daniel Dopp, and Anubhav Jain. Benchmarking materials property prediction methods: The matbench test set and au-tomatminer reference algorithm. npj Computational Materials, 6:138, 2020.DOI
  20. [20]Peter Eastman, Benjamin P Pritchard, John D Chodera, and Thomas E Markland. Nutmeg and spice: models and data for biomolecular machine learning. Journal of chemical theory and computation, 20(19):8583–8593, 2024.
  21. [21]Utku Evci, Vincent Dumoulin, Hugo Larochelle, and Michael C Mozer. Head2Toe: Utilizing intermediate representations for better transfer learning. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 6009–6033. PMLR, 17–23 Jul 2022.arxiv.org/abs/2201.03529
  22. [22]Shikun Feng, Yuyan Ni, Yanyan Lan, Zhi-Ming Ma, and Wei-Ying Ma. Fractional denoising for 3D molecular pre-training. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 9938–9961. PMLR, 23–29 Jul 2023.arxiv.org/abs/2307.10683
  23. [23]Rémi Flamary, Nicolas Courty, Alexandre Gramfort, Mokhtar Z. Alaya, Aurélie Boisbunon, Stanislas Chambon, Laetitia Chapel, Adrien Corenflos, Kilian Fatras, Nemo Fournier, Léo Gautheron, Nathalie T.H. Gayraud, Hicham Janati, Alain Rakotomamonjy, Ievgen Redko, Antoine Rolet, Antony Schutz, Vivien Seguy, Danica J. Sutherland, Romain Tavenard, Alexander Tong, and Titouan Vayer. Pot: Python optimal transport. Journal of Machine Learning Research, 22(78):1–8, 2021.
  24. [24]Johannes Gasteiger, Shankari Giri, Johannes T. Margraf, and Stephan Günnemann. Fast and uncertainty-aware directional message passing for non-equilibrium molecules. In Machine Learning for Molecules Workshop, NeurIPS, 2020.arxiv.org/abs/2011.14115
  25. [25]Weifeng Ge and Yizhou Yu. Borrowing treasures from the wealthy: Deep transfer learning through selective joint fine-tuning. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.arxiv.org/abs/1702.08690
  26. [26]Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al. Deepseek-r1 incentivizes reasoning in llms through reinforcement learning. Nature, 645(8081):633–638, 2025.DOI
  27. [27]Esther Heid, Kevin P. Greenman, Yunsie Chung, Shih-Cheng Li, David E. Graff, Florence H. Vermeire, [28] Pierre Hohenberg and Walter Kohn. Inhomogeneous electron gas. Physical review, 136(3B):B864, 1964.
  28. [29]Jeremy Howard and Sebastian Ruder. Universal language model fine-tuning for text classification, 2018.arxiv.org/abs/1801.06146
  29. [30]Shuyi Jia, Akaash R. Parthasarathy, Rui Feng, Guojing Cong, Chao Zhang, and Victor Fung. Derivative-based pre-training of graph neural networks for materials property predictions. Digital Discovery, 3:586–593, 2024.
  30. [31]Mohammadreza Karamad, Rishikesh Magar, Yuting Shi, Samira Siahrostami, Ian D Gates, and Amir Barati Farimani. Orbital graph convolutional neural network for material property prediction. Physical Review Materials, 4(9):093801, 2020.
  31. [32]Jaesun Kim, Jisu Kim, Jaehoon Kim, Jiho Lee, Yutack Park, Youngho Kang, and Seungwu Han. Data-efficient multifidelity training for high-fidelity machine learning interatomic potentials. Journal of the American Chemical Society, 147(1):1042–1054, 2024.DOI
  32. [33]Walter Kohn and Lu Jeu Sham. Self-consistent equations including exchange and correlation effects. Physical review, 140(4A):A1133, 1965.
  33. [34]Lingyu Kong, Nima Shoghi, Guoxiang Hu, Pan Li, and Victor Fung. Mattertune: An integrated, user-friendly platform for fine-tuning atomistic foundation models to accelerate materials simulation and discovery. Digital Discovery, 4(4):2253–2262, 2025.arxiv.org/abs/2504.10655
  34. [35]Simon Kornblith, Jonathan Shlens, and Quoc V. Le. Do better imagenet models transfer better? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
  35. [36]Dávid Péter Kovács, Ilyes Batatia, Eszter Sára Arany, and Gábor Csányi. Evaluation of the mace force field architecture: From medicinal chemistry to materials science. The Journal of Chemical Physics, 159(4):044118, 07 2023.DOI
  36. [37]Ananya Kumar, Aditi Raghunathan, Robbie Jones, Tengyu Ma, and Percy Liang. Fine-tuning can distort pretrained features and underperform out-of-distribution. In The Tenth International Conference on Learning Representations, 2022.arxiv.org/abs/2202.10054
  37. [38]Yoonho Lee, Annie S Chen, Fahim Tajwar, Ananya Kumar, Huaxiu Yao, Percy Liang, and Chelsea Finn. Surgical fine-tuning improves adaptation to distribution shifts. In The Eleventh International Conference on Learning Representations, 2023.arxiv.org/abs/2210.11466
  38. [39]Daniel S Levine, Muhammed Shuaibi, Evan Walter Clark Spotte-Smith, Michael G Taylor, Muhammad R Hasyim, Kyle Michel, Ilyes Batatia, Gábor Csányi, Misko Dzamba, Peter Eastman, et al. The open molecules 2025 (omol25) dataset, evaluations, and models. arXiv preprint arXiv:2505.08762, 2025.arxiv.org/abs/2505.08762
  39. [40]Hao Li, Pratik Chaudhari, Hao Yang, Michael Lam, Avinash Ravichandran, Rahul Bhotika, and Stefano Soatto. Rethinking the hyperparameters for fine-tuning. In International Conference on Learning Representations, 2020.arxiv.org/abs/2002.11770
  40. [41]Kangming Li, Andre Niyongabo Rubungo, Xiangyun Lei, Daniel Persaud, Kamal Choudhary, Brian DeCost, Adji Bousso Dieng, and Jason Hattrick-Simpers. Probing out-of-distribution generalization in machine learning for materials. Communications Materials, 6(1):9, 2025.DOI
  41. [42]Xingjian Li, Haoyi Xiong, Hanchao Wang, Yuxuan Rao, Liping Liu, and Jun Huan. Delta: Deep learning transfer using feature map with attention for convolutional networks. In International Conference on Learning Representations, 2019.arxiv.org/abs/1901.09229
  42. [43]Yi-Lun Liao, Tess Smidt, Muhammed Shuaibi, and Abhishek Das. Generalizing denoising to non-equilibrium structures improves equivariant force fields. Transactions on Machine Learning Research, 2024.arxiv.org/abs/2403.09549
  43. [44]Shikun Liu, Deyu Zou, Nima Shoghi, Victor Fung, Kai Liu, and Pan Li. Roft-mol: Benchmarking robust fine-tuning with molecular graph foundation models. arXiv preprint arXiv:2509.00614, 2025.arxiv.org/abs/2509.00614
  44. [45]Xixian Liu, Rui Jiao, Zhiyuan Liu, Yurou Liu, Yang Liu, Ziheng Lu, Wenbing Huang, Yang Zhang, and Yixin Cao. Learning 3d anisotropic noise distributions improves molecular force fields. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025.arxiv.org/abs/2510.22123
  45. [46]Ziquan Liu, Yi Xu, Yuanhong Xu, Qi Qian, Hao Li, Xiangyang Ji, Antoni Chan, and Rong Jin. Improved fine-tuning by better leveraging pre-training data. Advances in Neural Information Processing Systems, 35:32568–32581, 2022.
  46. [47]Aviv Netanyahu, Abhishek Gupta, Max Simchowitz, Kaiqing Zhang, and Pulkit Agrawal. Learning to extrapolate: A transductive approach. In The Eleventh International Conference on Learning Representations, 2023.
  47. [48]Sadman Sadeed Omee, Nihang Fu, Rongzhi Dong, Ming Hu, and Jianjun Hu. Structure-based out-of-distribution (ood) materials property prediction: a benchmark study. npj Computational Materials, 10:144, 2024.arxiv.org/abs/2401.08032
  48. [49]Sadman Sadeed Omeе, Steph-Yves Louis, Nihang Fu, Lai Wei, Sourin Dey, Rongzhi Dong, Qinyang Li, and Jianjun Hu. Scalable deeper graph neural networks for high-performance materials property prediction. Patterns, 3(5):100491, 2022. 8
  49. [50]Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744, 2022. 17arxiv.org/abs/2203.02155
  50. [51]F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825–2830, 2011. 21
  51. [52]Anyang Peng, Chun Cai, Mingyu Guo, Duo Zhang, Chengqian Zhang, Antoine Loew, Linfeng Zhang, and Han Wang. LAMBench: A benchmark for large atomic models. arXiv preprint arXiv:2504.19578, 2025. 10arxiv.org/abs/2504.19578
  52. [53]Tien Lam Pham, Hiori Kino, Kiyoyuki Terakura, Takashi Miyake, Koji Tsuda, Ichigaku Takigawa, and Hieu Chi Dam. Machine learning reveals orbital interaction in materials. Science and technology of advanced materials, 18(1):756, 2017. 7, 20
  53. [54]Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 8748–8763. PMLR, 18–24 Jul 2021. 2, 17arxiv.org/abs/2103.00020
  54. [55]Raghunathan Ramakrishnan, Pavlo O Dral, Matthias Rupp, and O Anatole von Lilienfeld. Quantum chemistry structures and properties of 134 kilo molecules. Scientific Data, 1, 2014. 19DOI
  55. [56]Benjamin Rhodes, Sander Vandenhaute, Vaidotas Šimkus, James Gin, Jonathan Godwin, Tim Duignan, and Mark Neumann. Orb-v3: atomistic simulation at scale. arXiv preprint arXiv:2504.06231, 2025. 2arxiv.org/abs/2504.06231
  56. [57]Youngmin Ro and Jin Young Choi. Autorl: Layer-wise pruning and auto-tuning of learning rates in fine-tuning of deep networks. Proceedings of the AAAI Conference on Artificial Intelligence, 35(3):2486–2494, May 2021. 17
  57. [58]Jerret Ross, Brian Belgodore, Vijil Chenthamarakshan, Inkit Padhi, Youssef Mroueh, and Payel Das. Large-scale chemical language representations capture molecular structure and properties. Nature Machine Intelligence, 4(12):1256–1264, 2022. 7
  58. [59]Robin Ruff, Patrick Reiser, Jan Stühmer, and Pascal Friederich. Connectivity optimized nested line graph networks for crystal structures. Digital Discovery, 3(3):594–601, 2024. 8
  59. [60]Benjamin Sanchez-Lengeling and Alán Aspuru-Guzik. Inverse molecular design using machine learning: Generative models for matter engineering. Science, 361(6400):360–365, 2018. 1
  60. [61]Victor Garcia Satorras, Emiel Hoogeboom, and Max Welling. E(n) equivariant graph neural networks. In Proceedings of the 38th International Conference on Machine Learning, Proceedings of Machine Learning Research, page 9323–9332. PMLR, 2021. 7
  61. [62]Kristof Schütt, Pieter-Jan Kindermans, Huziel Enoc Sauceda Felix, Stefan Chmiela, Alexandre Tkatchenko, and Klaus-Robert Müller. Schnet: A continuous-filter convolutional neural network for modeling quantum interactions. Advances in neural information processing systems, 30, 2017. 8arxiv.org/abs/1706.08566
  62. [63]Kristof T Schütt, Farhad Arbabzadah, Stefan Chmiela, Klaus R Müller, and Alexandre Tkatchenko. Quantum-chemical insights from deep tensor neural networks. Nature communications, 8(1):1–8, 2017. 8DOI
  63. [64]Nofit Segal, Aviy Netyanahu, Kevin P Greenman, Pulkit Agrawal, and Rafael Gómez-Bombarelli. Known unknowns: Out-of-distribution property prediction in materials and molecules. npj Computational Materials, 11(1):345, 2025. 17arxiv.org/abs/2502.05970
  64. [65]Zhiqiang Shen, Zechun Liu, Jie Qin, Marios Savvides, and Kwang-Ting Cheng. Partial is better than all: Revisiting fine-tuning strategy for few-shot learning. Proceedings of the AAAI Conference on Artificial Intelligence, 35(11):9594–9602, May 2021. 17arxiv.org/abs/2102.03983
  65. [66]Hajime Shimakawa, Akiko Kumada, and Masahiro Sato. Extrapolative prediction of small-data molecular property using quantum mechanics-assisted machine learning. npj Computational Materials, 10(1):11, 2024. 2, 17DOI
  66. [67]Nima Shoghi, Adeesh Kolluru, John R. Kitchin, Zachary Ward Ulissi, C. Lawrence Zitnick, and Brandon M Wood. From molecules to materials: Pre-training large generalizable models for atomic property prediction. In The Twelfth International Conference on Learning Representations, 2024. 2, 3, 17, 22arxiv.org/abs/2310.16802
  67. [68]Yang Shu, Xingzhuo Guo, Jialong Wu, Ximei Wang, Jianmin Wang, and Mingsheng Long. CLIPood: Generalizing CLIP to out-of-distributions. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 31716–31731. PMLR, 23–29 Jul 2023.
  68. [69]Ava P Soleimany, Alexander Amini, Samuel Goldman, Daniela Rus, Sangeeta N Bhatia, and Connor W Coley. Evidential deep learning for guided molecular property prediction and discovery. ACS central science, 7(8):1356–1367, 2021.DOI
  69. [70]Philipp Thölke and Gianni De Fabritiis. Equivariant transformers for neural network based molecular potentials. In International Conference on Learning Representations, 2022.
  70. [71]Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Machine Learning Research, 9(86):2579–2605, 2008.
  71. [72]Yusong Wang, Shaoning Li, Tong Wang, Bin Shao, Nanming Zheng, and Tie-Yan Liu. Geometric transformer with interatomic positional encoding. In Advances in Neural Information Processing Systems, 2023.
  72. [73]Benjamin Warner, Antoine Chaffin, Benjamin Clavié, Orion Weller, Oskar Hallström, Said Taghadouini, Alexis Gallagher, Raja Biswas, Faisal Ladhak, Tom Aarsen, Nathan Cooper, Griffin Adams, Jeremy Howard, and Iacopo Poli. Smarter, better, faster, longer: A modern bidirectional encoder for fast, memory efficient, and long context finetuning and inference, 2024.arxiv.org/abs/2412.13663
  73. [74]Brandon M Wood, Misko Dzamba, Xiang Fu, Meng Gao, Muhammed Shuaibi, Luis Barroso-Luque, Kareem Abdelmaqsoud, Vahe Gharakhanyan, John R. Kitchin, Daniel S. Levine, Kyle Michel, Anuroop Sri-ram, Taco Cohen, Abhishek Das, Sushree Jagriti Sahoo, Ammar Rizvi, Zachary Ward Ulissi, and C. Lawrence Zitnick. UMA: A family of universal models for atoms. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025.arxiv.org/abs/2506.23971
  74. [75]Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo Lopes, Hannah Hajishirzi, Ali Farhadi, Hongseok Namkoong, and Ludwig Schmidt. Robust fine-tuning of zero-shot models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 7959–7971, June 2022.
  75. [76]Fang Wu, Shuting Jin, Siyuan Li, and Stan Z. Li. Instructor-inspired machine learning for robust molecular property prediction. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
  76. [77]Tian Xie and Jeffrey C. Grossman. Crystal graph convolutional neural networks for an accurate and interpretable prediction of material properties. Phys. Rev. Lett., 120:145301, Apr 2018.
  77. [78]Han Yang, Chenxi Hu, Yichi Zhou, Xixian Liu, Yu Shi, Jielan Li, Guanzhi Li, Zekun Chen, Shuizhou Chen, Claudio Zeni, et al. Mattersim: A deep learning atomistic model across elements, temperatures and pressures. arXiv preprint arXiv:2405.04967, 2024.arxiv.org/abs/2405.04967
  78. [79]Tianz hixi Yin, Gihan Panapitiya, Elizabeth D Coda, and Emily G Saldanha. Evaluating uncertainty-based active learning for accelerating the generalization of molecular property prediction. Journal of Cheminformatics, 15(1):105, 2023.
  79. [80]Chengxuan Ying, Tianle Cai, Shengjie Luo, Shuxin Zheng, Guolin Ke, Di He, Yanming Shen, and Tie-Yan Liu. Do transformers really perform bad for graph representation? In Advances in Neural Information Processing Systems, 2021.arxiv.org/abs/2106.05234
  80. [81]Sheheryar Zaidi, Michael Schaarschmidt, James Martens, Hyunjik Kim, Yee Whye Teh, Alvaro Sanchez-Gonzalez, Peter Battaglia, Razvan Pascanu, and Jonathan Godwin. Pre-training via denoising for molecular property prediction. In International Conference on Learning Representations, 2023.
  81. [82]Duo Zhang, Xinzijian Liu, Xiangyu Zhang, Chengqian Zhang, Chun Cai, Hangrui Bi, Yiming Du, Xuejian Qin, Anyang Peng, Jiameng Huang, et al. Dpa-2: a large atomic model as a multi-task learner. npj Computational Materials, 10(1):293, 2024.DOI
  82. [83]Duo Zhang, Anyang Peng, Chun Cai, Wentao Li, Yuanchang Zhou, Jinzhe Zeng, Mingyu Guo, Chengqian Zhang, Bowen Li, Hong Jiang, Tong Zhu, Weile Jia, Linfeng Zhang, and Han Wang. A graph neural network for the era of large atomistic models. arXiv preprint arXiv:2506.01686, 2025.arxiv.org/abs/2506.01686
  83. [84]Linfeng Zhang, Mohan Chen, Xifan Wu, Han Wang, Weinan E, and Roberto Car. Deep neural network for the dielectric response of insulators. Phys. Rev. B, 102(4):041121, 2020.DOI
  84. [85]Gengmo Zhou, Zhifeng Gao, Qian kun Ding, Hang Zheng, Hongteng Xu, Zhewei Ye, Linfeng Zhang, and Guolin Ke. Uni-mol: A universal 3d molecular representation learning framework. In The Eleventh International Conference on Learning Representations, 2023.DOI

Paper details

vlnc:W_46gxk4dk
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Physics and Astronomy

Not recorded for this paper

View statement
Looking for related papers…
Contents