betaFeedback

WorksComputer Science

A quantitative analysis of knowledge-learning preferences in large language models in molecular science

Abstract

Deep learning has significantly advanced molecular modeling and design, enabling efficient understanding and discovery of novel molecules. In particular, large language models (LLMs) introduce a fresh research paradigm to tackle scientific problems from a natural language processing (NLP) perspective. LLMs significantly enhance our understanding and generation of molecules, often surpassing existing methods with their capabilities to decode and synthesize complex molecular patterns. However, two key issues remain: how to quantify the match between model and data modalities and how to identify the knowledge-learning preferences of models. To address these challenges, we propose a multi-modal benchmark, named ChEBI-20-MM, and perform 1263 experiments to assess the model's compatibility with data modalities and knowledge acquisition. Through the modal transition probability matrix, we provide insights into the most suitable modalities for tasks. Furthermore, we introduce a statistically interpretable approach to discover context-specific knowledge mapping by localized feature filtering. Our analysis offers an exploration of the learning mechanism and paves the way for advancing LLMs in molecular science.

Molecular modeling and design are pivotal in the discovery and development of new molecules, serving a wide range of applications from drug discovery to materials science. These fields are fundamentally reliant on the ability to not only discover novel molecules but also to understand their structures and properties comprehensively. Such capabilities are crucial for developing targeted therapies in medicine, creating advanced materials with specific functionalities, and enhancing our fundamental understanding of chemical and biological processes. By unlocking the potential of molecular insights, we pave the way for breakthroughs that can transform technologies and improve human health. Nevertheless, traditional approaches to discovering new molecules or refining existing ones are often arduous, costly, and fraught with a high risk of failure. In contrast, modern computational approaches significantly improve the creation and modification of molecules. Yet, they require considerable computational resources.

Artificial Intelligence (AI) methods have made significant strides in this domain. They offer faster computations and the ability to handle large datasets. In recent years, AI-driven approaches such as Generative Adversarial Networks (GANs) [1] and diffusion models [2] have been designed for molecule generation. These techniques simulate the creative process of designing new molecular structures, thereby expanding the potential for groundbreaking discoveries in molecular science. The task of understanding molecular structures and properties, particularly through molecular description and embedding, has been significantly advanced by machine learning techniques [3] and graph models [4][5][6]. These approaches have enhanced our ability to accurately represent and interpret the intricate details of

molecules. However, most AI models often face challenges in generalizability and flexibility, requiring extensive data for training to ensure accuracy and require specific adjustments for each molecular task.

Transformers [7] offer an advantage with their robust text encoding and generation capabilities. These models can be fine-tuned with minimal task-specific adjustments, making them more versatile and efficient in molecular modeling and design. As depicted in Figure 1.a, we organize six pivotal molecular tasks into three objectives, each addressing specific scientific needs. In the description category, tasks such as captioning and IUPAC name recognition are pivotal for understanding complex molecular structures. Molecular captioning renders intricate chemical details more comprehensible. IUPAC name recognition is crucial for ensuring precise chemical identity across various platforms, despite the complexities of nomenclature rules and the scarcity of robust automated tools capable of handling such challenges. In the embedding category, we focus on property prediction and retrieval. These tasks are vital for extracting meaningful insights from molecular data and facilitating the prediction of molecular behaviors under various conditions. The generation category encompasses text-based molecule generation and optical recognition, which aim to create or recreate accurate molecular representations. Text-based generation enables the creation of novel molecules through predictive models, while optical recognition is essential for digitizing historical chemical data, ensuring that legacy research continues to inform and enhance contemporary scientific endeavors.

Moreover, since the advent of ChatGPT [8] and GPT-4 [9], LLMs have emerged as a groundbreaking trend, especially in molecular science. LLMs, with their advanced capabilities in processing and generating human-like text, present a novel paradigm for understanding and designing molecular structures. This paradigm is known as scientific language modeling (SLM). Their ability to assimilate and analyze vast amounts of textual data can provide unprecedented insights, overcoming some of the limitations of traditional AI methods. This new capability combines accuracy and novelty for improved outcomes, termed chemical knowledge. Its effectiveness depends on factors like input data, model architecture, and training strategies. However, current assessments of this capability remain incomplete.

Existing related work in molecular science, like the molecule-generation survey [10], lacks comprehensive model comparisons and has a limited task scope. The knowledge-driven review [11] categorizes molecular learning but misses detailed method comparisons and dataset discussions. And recent benchmarks, such as one testing ChatGPT [12], cover eight chemical tasks, each providing unique chemical insights. Mol-Instructions [13] offers a dataset for fine-tuning with various molecule and protein instructions, boosting biomolecular understanding in LLMs. Multi-modal data and methodologies are crucial in the molecular domain, as they naturally integrate diverse data types, providing complementary insights that enhance the understanding of molecular behaviors. However, these papers lack multi-modal content and do not sufficiently explore the models’ chemical knowledge.

In this study, we regard different representations of molecules as distinct modalities, transcending conventional formats like 1D text, 2D graphs, or images. We introduce ChEBI-20-MM, a comprehensive multi-modal benchmark encompassing 32,998 molecules. These molecules can be characterized by 1D textual chemical descriptors, including SMILES (Simplified Molecular Input Line Entry System) [14], InChI (International Chemical Identifier) [15], and SELFIES (Self-Referencing Embedded Strings) [16], along with their corresponding 2D graphs generated by cheminformatics tools such as RDKit [17]. These representations are classified as the molecules’ internal information because they encapsulate the molecular essence. Furthermore, these modalities can be interconverted using cheminformatics tools, although the processes of their conversion are not the focal point of our investigation. Conversely, modalities such as molecular captions, IUPAC (International Union of Pure and Applied Chemistry) [18] names, and images, which enhance human comprehension, are categorized as external information. The intrinsic connections between different modalities, such as images, 2D graphs, and text, are profound. For instance, while graphs can be directly transformed into images with minimal information loss, the reverse process from images to graphs tends to incur significant data loss. Similarly, converting molecular internal information into captions or IUPAC names typically results in a reduction of informational content.

To address these losses, models are essential for bridging these gaps. We construct a modal transition probability matrix to analyze the efficiency of modal conversion. As depicted in Figure 1.b, the classification is reflected in the evaluation framework of the six tasks (Task 1-6) and their respective modalities. Our benchmark not only evaluates the performance variances across modalities but also can assess the multi-modal synergistic performance. It includes the translation from various types of internal information to external information and generating captions based on IUPAC names, as represented by Task 7, which exemplifies typical multi-modal tasks. It should be noted that our benchmark encompasses seven modalities. Furthermore, to facilitate the evaluation of embedding ability, we integrate the MoleculeNet dataset [19] and classify molecular properties as external information, treating the property prediction as a form of translation.

In summary, this study analyzes the LLMs in molecular modeling and design. We categorize six common molecular tasks into three distinct objectives: description, embedding, and generation, as vividly depicted in Figure 1. Moreover, we establish a unified multi-modal benchmark ChEBI-20-MM and conduct experiments to assess the compatibility of data modalities, model architectures, and diverse task types, examining their impact on task performance. Additionally, our end-to-end visual method showcases the discovery of modeling insights embedded with chemical knowledge. Overall, our main contributions are the following:

  • This work quantitatively analyzes LLMs in molecule modeling, categorizes existing models, and presents a multi-modal benchmark (ChEBI-20-MM) for performance evaluation, backed by 1263 experiments.

  • We explore the modal transition probability matrix and identify the best match within typical data modalities and model architectures.

  • We propose a statistically interpretable approach to showcase the knowledge acquisition through localized feature filtering.

Results

To obtain comprehensive insights into the factors of model performance, it is essential to conduct a systematic and unbiased comparison across various LLMs. We introduce ChEBI-20-MM, a multi-modal benchmark developed from the ChEBI-20 dataset that integrates various data, including SMILES, InChI, IUPAC names, SELFIES, captions, and images. This benchmark is designed to assess the chemical knowledge of models across tasks such as molecule generation, image and IUPAC recognition, molecular captioning, and retrieval, featuring in-depth analysis of the compatibility between modalities and models. We explore the models’ tendencies for knowledge acquisition through two typical tasks, employing a localized feature filtering method. These processes demonstrate the feasibility of cross-domain knowledge-learning analysis. Case studies further validate the utility of this analysis process.

Performance evaluation of tasks

For the analysis of modality adaptation to different tasks, we select representative metrics indicative of successful task completion, such as the METEOR score for molecule captioning and the area under the receiver operating characteristic curve (ROC_AUC) for property classification tasks. We fill out the modal transition probability matrix, which provides insights into the most suitable data modalities for different task types, as depicted in Figure 2.a. Regarding model analysis for different tasks, Figure 2.b and Figure 2.c demonstrate the frequency of appearances of encoders and decoders in nine text-to-text tasks, as well as eight encoders and two pooling mechanisms in embedding tasks. Detailed experimental results are shown in Supplementary Section C. Table 3-Table 9. We also incorporate typical traditional machine learning methods, graph models, and transformer-based models for property prediction task analysis in Table 6 and Table 7.

Supplementary Table 6: Comparative performance of traditional machine learning (ML), GNN, and Transformer models on property prediction classification tasks. The evaluation metric is AUC_ROC. The numbers listed under each dataset represent the count of molecules and the number of classification tasks. The (G) denotes graph-based inputs, while (S) denotes SMILES-based inputs as the input modalities.

MethodModelBBBPTox21ToxCastSIDERClinToxBACEAvg
203978318575142714781513
1126172721
Traditional ML ModelsLogistic Regression [19]0.6990.6640.6000.5960.7820.7810.687
KernelSVM [83]0.7290.6790.6520.6200.8100.8620.725
RF [84]0.7140.6530.6340.6330.7650.8670.711
GNN-basedGIN [4]0.6400.7010.6130.5500.5800.7000.631
GraphCL [85]0.6750.7500.6280.6010.7890.6870.688
GROVER [86]0.7000.7430.6540.6480.8120.8260.731
GraphMVP [87]0.7240.7440.6310.6390.7750.8120.721
MoMu-SciBERT(G) [33]0.7050.7560.6340.6050.7990.7670.711
Transformer-basedMolT5 [21]0.6380.7360.5560.5260.9090.7080.679
KV-PLM [39]0.6690.6470.5860.5530.8430.7190.670
MoleculeSTM(S) [47]0.7080.7570.6520.6370.8660.8200.740

Table 6. Comparative performance of traditional machine learning (ML), GNN, and Transformer models on property prediction classification tasks. The evaluation metric is AUC_ROC. The numbers listed under each dataset represent the count of molecules and the number of classification tasks. The (G) denotes graph-based inputs, while (S) denotes SMILES-based inputs as the input modalities.

Supplementary Table 7: Comparative performance of traditional machine learning, GNN, and Transformer models on property prediction regression tasks. The evaluation metric is RMSE. The numbers provided under each dataset indicate the count of molecules. The (G) denotes graph-based inputs, while (S) denotes SMILES-based inputs as the input modalities.

MethodModelESOLFreesolvLipophilicity
11286424200
Traditional ML ModelsLogistic Regression [19]0.7320.9731.003
KernelSVM [83]0.7521.0860.718
RF [84]0.8171.3360.791
GNN-basedGIN [4]1.6423.7810.821
GROVER [86]0.9922.4700.810
GraphMVP [87]1.3412.9720.769
Uni-mol [88]0.7881.4800.603
Transformer-basedSMILES-TSFM [89]1.3393.0971.215
MolT5 [21]1.2523.1390.940

Table 7. Comparative performance of traditional machine learning, GNN, and Transformer models on property prediction regression tasks. The evaluation metric is RMSE. The numbers provided under each dataset indicate the count of molecules. The (G) denotes graph-based inputs, while (S) denotes SMILES-based inputs as the input modalities.

TaskData SplitingDatasetInputEncoderPoolingmse ↓rmse ↓mae ↓
property prediction (regression)scaffoldesolSMILESt5avg1.573 ± 0.1391.253 ± 0.0560.972 ± 0.027
property prediction (regression)scaffoldesolSMILESmolt5avg1.593 ± 0.4081.252 ± 0.1580.974 ± 0.102
property prediction (regression)scaffoldesolInChImolt5avg1.739 ± 0.0891.318 ± 0.0340.965 ± 0.038
property prediction (regression)scaffoldesolgraphgatavg1.750 ± 0.1731.321 ± 0.0660.983 ± 0.057
property prediction (regression)scaffoldesolSMILESt511avg1.910 ± 0.5331.369 ± 0.1921.039 ± 0.125
property prediction (regression)scaffoldfreesolvgraphgcnmax9.701 ± 1.8923.101 ± 0.2952.254 ± 0.225
property prediction (regression)scaffoldfreesolvSELFIESmolt5avg9.858 ± 0.2943.139 ± 0.0472.391 ± 0.007
property prediction (regression)scaffoldfreesolvgraphgatavg10.429 ± 2.1843.212 ± 0.3322.608 ± 0.267
property prediction (regression)scaffoldfreesolvSELFIESt5avg11.689 ± 0.9423.416 ± 0.1402.451 ± 0.113
property prediction (regression)scaffoldfreesolvSELFIESt511avg11.740 ± 3.5913.389 ± 0.5042.637 ± 0.411
property prediction (regression)scaffoldlipophilicitygraphginavg0.676 ± 0.0670.821 ± 0.0400.656 ± 0.043
property prediction (regression)scaffoldlipophilicitygraphgcnavg0.735 ± 0.0060.857 ± 0.0030.684 ± 0.007
property prediction (regression)scaffoldlipophilicitygraphgcnmax0.770 ± 0.0930.876 ± 0.0520.699 ± 0.046
property prediction (regression)scaffoldlipophilicitygraphgatavg0.855 ± 0.0670.924 ± 0.0360.736 ± 0.025
property prediction (regression)scaffoldlipophilicitySMILESmolt5max0.890 ± 0.1470.940 ± 0.0770.762 ± 0.064
property prediction (regression)randomesolInChIt511avg0.967 ± 0.0710.983 ± 0.0360.718 ± 0.053
property prediction (regression)randomesolSMILESmolt5avg1.035 ± 0.2921.007 ± 0.1420.779 ± 0.130
property prediction (regression)randomesolSMILESt511avg1.201 ± 0.3061.087 ± 0.1380.831 ± 0.126
property prediction (regression)randomesolInChImolt5avg1.234 ± 0.1571.109 ± 0.0700.794 ± 0.037
property prediction (regression)randomesolgraphgcnmax1.273 ± 0.3161.120 ± 0.1360.841 ± 0.103
property prediction (regression)randomfreesolvgraphgcnmax5.049 ± 0.4932.244 ± 0.1121.733 ± 0.031
property prediction (regression)randomfreesolvgraphgatmax5.175 ± 1.1192.261 ± 0.2511.818 ± 0.226
property prediction (regression)randomfreesolvgraphgatavg5.536 ± 0.2752.352 ± 0.0581.706 ± 0.059
property prediction (regression)randomfreesolvgraphgcnavg6.064 ± 1.4852.443 ± 0.3061.829 ± 0.218
property prediction (regression)randomfreesolvInChIt5avg7.343 ± 1.2552.700 ± 0.2321.920 ± 0.179
property prediction (regression)randomlipophilicitygraphginavg0.690 ± 0.0990.828 ± 0.0610.650 ± 0.043
property prediction (regression)randomlipophilicitygraphgatavg0.717 ± 0.0190.846 ± 0.0110.676 ± 0.008
property prediction (regression)randomlipophilicitygraphgcnavg0.785 ± 0.0770.885 ± 0.0430.693 ± 0.045
property prediction (regression)randomlipophilicitygraphgcnmax0.795 ± 0.1320.888 ± 0.0750.704 ± 0.080
property prediction (regression)randomlipophilicitySMILESmolt5avg0.887 ± 0.2210.934 ± 0.1210.740 ± 0.100

Table 9. Experimental results for regression tasks

Modal transition probability matrix. In our study, we construct a modal transition probability matrix based on outcomes from various text generation tasks, such as any-to-text, encompassing 13 distinct tasks derived from the ChEBI-20-MM dataset and experimental settings. Additionally, results for molecular property prediction tasks are obtained from MoleculeNet and mapped to the paradigms illustrated in Figure 1.b. This matrix features the inputs along the vertical axis and the outputs along the horizontal axis, with the transition probabilities within the same modality set to 1. For transitions involving molecular internal information, which can be directly converted using specialized tools, we set the transition success rate to 1. Molecular generation tasks are specifically addressed using SMILES representations, and the BLEU score serves as the metric for conversion effectiveness. This standard is uniformly applied to all entries pertaining to internal information generation within the same row. In cases where images can be converted from graphs using tools, the transition metrics for images are aligned with those for internal information. For captioning and IUPAC naming tasks, the METEOR score is utilized as the conversion metric. However, the molecular property prediction regression tasks can not be presented as probability. We select classification tasks and use the mean ROC_AUC as the conversion metric. Finally, tasks involving text generation based on molecular properties are not within the scope of our research and are assigned a transition probability of zero. This matrix provides a comprehensive view of modal adaptability across different task configurations, laying a foundational basis for future research.

Comparing model architectures. Our research demonstrates the novel performance advantages of T5-based LLMs in molecular science, surpassing the capabilities of BERT and GPT variants. We evaluate the performance of these models on molecular text-to-text and retrieval tasks. We select the top five results for each task, providing a comparative analysis of the corresponding encoders and decoders. As depicted in Figure 2.b, we present the frequency of various encoders and the proportions of encoder-decoder combinations utilized. T5-based models demonstrate a distinct advantage as encoders, while BERT models also show commendable performance. Notably, the BioGPT [20], a variant of the GPT model, excels in IUPAC name recognition tasks using captions and SELFIES, which is why it is prominently featured twice in our analysis. In terms of decoding, T5-based models not only excel as encoder-decoder models, but they also perform robustly when deployed as decoders. To assess the retrieval capabilities across different modalities, we consider all encoders and select the top five results for each task, as illustrated in Figure 2.c. We compare the embedding capabilities of different models and pooling strategies. Additionally, we demonstrate the suitability of these modalities for cross-modal retrieval, using R@1 as the comparing metric.

Results of benchmark

Figure 2. Results of benchmark. a. Modal transition probability matrix. This matrix presents the performance in text generation and property prediction tasks. The vertical axis represents input modalities, while the horizontal axis denotes output modalities. b. Encoders and decoders in nine text-to-text tasks. This illustration highlights the frequency of various models appearing in the top 5 rankings. The T5-based models exhibit a dominant presence. c. Encoders, pooling mechanisms, and retrieval performance in embedding tasks. Alongside model rankings, the figure indicates that average pooling is a preferred choice for the pooling layer.

Modality and architecture insights. From the results, we present the following modality insights:

  • IUPAC names for generation and captioning: IUPAC names are conducive to text-based molecule generation and molecule captioning.

  • SMILES for IUPAC recognition: SMILES representations are particularly well-suited for IUPAC recognition compared to other molecular internal information.

  • Preferred modalities for captioning: Since some captions include the IUPAC name of the molecule, IUPAC is the most appropriate modality for molecule captioning, followed by SMILES.

  • Choosing modalities for retrieval: For molecule retrieval tasks targeting captions, IUPAC is the optimal modality, followed by SMILES. Conversely, when the target is IUPAC names, SMILES becomes the most favorable modality.

  • Graph vs. textual modalities in property prediction: In molecular property prediction tasks, among the selected top five rank combinations from 90 groups, graph modalities appear 40 times and SMILES 25 times, indicating a clear advantage for graph modalities and the relative superiority of SMILES over other textual representations.

As for the model performance insights:

  • T5 series excellence: The T5 series models show excellent performance across these tasks.

  • Graph models vs. T5 series: Only the significantly larger T5 series models surpass GNNs, affirming the superiority of smaller-scale graph models for molecular embedding.

  • Efficacy of average pooling: The average pooling mechanism tends to achieve superior performance more easily.

Case studies on model knowledge-learning preferences

Our analysis aims to discern model preferences for knowledge acquisition. To enhance interpretability, we specifically focus on the mapping processes from IUPAC names to captions and from SELFIES to captions, which bridge natural language tasks and chemical cross-domain tasks. We analyze the text-inference processes of LLMs and compile two token mapping matrices. To mitigate the impact of overwhelmingly frequent mappings on our analysis, we introduce a localized feature filtering method. This approach allows us to analyze specific high-frequency token mappings that indicate knowledge-learning preferences. By adjusting the threshold TT, we ensure that the selected token mappings are statistically significant. Finally, we validate and analyze the effectiveness of our methods through molecular case studies, confirming their utility in real-world applications.

Tokens mapping matrix. Initially, we extract typical tokens from the model’s tokenizer, removing meaningless symbols and filtering the top 20 high-frequency chemical tokens to construct the mapping matrix AA. Figure 3 displays the token mapping matrices after normalization and rearrangement based on the total counts of rows and columns. The common high-frequency mappings, such as ‘oxy’ and ‘methyl,’ are prevalent across various tokens, indicating their widespread presence in chemical structures. Traditional filtering methods, while capturing many of these frequent mappings, may inadvertently exclude tokens with specific contextual significance. Our localized feature filtering method aims to refine this process by prioritizing tokens that provide greater contextual relevance, thereby enhancing the interpretation of our analysis.

Knowledge patterns and insights: tokens mapping matrices, threshold analyses, and case studies of knowledge-learning preferences

Figure 3. Knowledge patterns and insights. a. Tokens mapping matrix and threshold TT analysis. The two matrices represent the high-frequency tokens mapping patterns generated by the processes from IUPAC names and SELFIES to molecular captions. On the right of the figure, as the threshold TT increases, the selection criteria for identifying specific high-frequency token pairs become more stringent, consequently reducing their number and impacting the significance levels. b. Case studies of knowledge-learning preferences. These cases are selected from model inference results, where the mapping of tokens exemplifies the model’s preferences for knowledge learning.

Thresholds TT analysis. To identify particular high-frequency mapping pairs, we varied the threshold TT to obtain different quantities of token mappings and their corresponding confidence levels. We carefully balance the number of token mappings against the confidence levels to select an optimal TT, ensuring neither an overly small set of mappings nor insufficiently low confidence levels. Taking the IUPAC name to caption process as an example, as shown in Figure 3, at T=3.515T = 3.515, the Z-test attains a value of 2.758, corresponding to a confidence level of approximately 99.71%. This analysis identifies eleven specific token mapping pairs. After excluding pairs with identical row and column names, we select seven unique pairs. Similarly, for the SELFIES to caption process, at T=3.434T = 3.434, the Z-test reached a value of 2.476, translating to a confidence level of approximately 99.34%. From this, we identify five unique pairs. By consolidating mappings with the same leading or trailing tokens, we categorize the mappings into the following 12 groups:

  • IUPAC name to caption:

    • ‘benz’: ‘group’

    • ‘ace’: ‘amino’

    • ‘lan’: ‘phosphat’

    • ‘ent’: ‘acid’

    • ‘phen’: ‘one’

    • ‘acid’: { ‘box’, ‘lic’ }

  • ‘min’: ‘ine’

  • SELFIES to caption:

    • ‘[C]’: {‘acy’, ‘acid’, ‘conjug’}

    • ‘[O-1]’: {‘conjug’, ‘phosphat’}

    • ‘[P]’: {‘acy’, ‘phosphat’}

    • ‘[N]’: ‘ine’

    • ‘[=C]’: ‘acid’

Our analysis elucidates the diversity and specificity of chemical token relationships, as demonstrated by the mapping patterns outlined. These mappings not only facilitate a deeper understanding of chemical nomenclature across different representation methods but also reveal underlying scientific insights about molecular structure and function.

Case studies of knowledge-learning preferences. Following these mapping patterns, we randomly select corresponding molecular IUPAC names, SELFIES, and their captions, as shown in Figure 3.b. The token ‘min’ in the IUPAC name is directly mapped to the ‘amino’ group. It reflects the NH2NH_2 group, which is common in biomolecules and crucial for biological functions. The ‘[N]’ in SELFIES denotes a nitrogen atom, key to the amino group’s bonding and interactions. Similarly, the suffix ‘ine’ in captions often signifies nitrogen-based compounds like amines and amino acids such as methionine, highlighting the amino group’s presence. In another case study, the token ‘ent’ often indicates pentane derivatives, such as ‘pentanoic’ or ‘pentanoyl,’ signaling a five-carbon chain integral to the molecule’s structure. For instance, in a dipeptide composed of L-aspartic acid and L-arginine, the presence of ‘pentanoic’ denotes a five-carbon fatty acid essential for the molecule’s stability and functional integrity. Concurrently, the SELFIES token ‘[=C]’ underscores the significance of carbon double bonds, which are vital for maintaining molecular rigidity and facilitating biochemical interactions. Furthermore, the term ‘acid’ in the caption specifically refers to the carboxylic acid groups in both amino acids, which are indispensable for forming peptide bonds that define the dipeptide’s functionality. For more case studies, please refer to the Supplementary Figure S1.

Tokens mapping matrix and threshold T analysis of IUPAC name recognition and molecule generation

Figure S1. Tokens mapping matrix and threshold T analysis of IUPAC name recognition and molecule generation. Each subplot for the tasks displays a tokens co-occurrence matrix, illustrating the degree of mapping between key tokens across different modalities. The analysis is complemented by the T threshold derived from the localized feature filtering method, along with corresponding Z-test results, confidence, and insights into particular high-frequency token mappings.

The model exhibits a robust understanding of molecular structure and functional groups, effectively mapping specific tokens to key chemical components such as carbon chains and double bonds. It demonstrates the model’s ability to discern critical elements that define molecular stability and reactivity, essential for predicting molecular behavior and interactions in biochemical contexts. Through a series of model knowledge-learning insights, we demonstrate that the observed mapping patterns are not merely coincidental but reflective of the model’s inherent ability to process and interpret complex chemical information.

Discussion

Exploring the effects of multi-modal. To identify how data modalities influence model performance in molecular science tasks, we develop a multi-modal benchmark and carry out thousands of experiments. Furthermore, we perform experiment replication and gather experimental data from other research to analyze the effects of factors related to multi-modal fusion. Figure 1 illustrates that molecular multi-modal fusion tasks include molecular retrieval, molecule captioning, and property prediction, with the latter two commonly used as benchmarks, as they integrate information from multiple modalities to improve molecular representation learning. We use well-established benchmarks like MoleculeNet [19] for property prediction, and ChEBI-20 [21] and PubChem32k [22] for molecule captioning. As shown in Figure 4, our focus is on molecular property classification and molecule captioning, using SMILES, SELFIES, and graph representations processed by a graph isomorphism network (GIN) [4]. For more information about the results of multi-modal fusion, please refer to Supplementary Tables 10 and 11. The modal fusion techniques at the embedding layer include additive blending with weights, concatenation, and concatenation followed by self-attention. In contrast, the encoding layer predominantly utilizes contrastive learning and cross-attention methods. The comparative experiments based on pooling methods consistently utilize average pooling.

Bar charts showing multi-modal fusion performance for molecular property predictions and molecule captioning

Figure 4. Performance of multi-modal fusion. a. Multi-modal fusion performance for molecular property predictions. This figure displays the AUC-ROC results for various molecular property prediction classification tasks. It compares the performance of the SciBERT and MolT5 models as encoders using SMILES (S) as input text and BioT5 using SELFIES as input text, with the graph model (GIN) utilizing graph data (G). In each subplot, the final results contributed by the vectors obtained after encoding and pooling from the foundation models are shown. "add" represents vector addition, "weight_add" represents adaptive weighted vector addition, "concat" represents concatenated encoding followed by pooling, and "attention" represents concatenated encoding processed by the attention mechanism before pooling. Different colors represent different tasks. b. Multi-modal fusion performance for molecule captioning. This panel shows the performance of six textual similarity metrics across two common datasets for molecule captioning tasks. The x-axis represents the models and input modalities, while the y-axis represents the metric values. Each color corresponds to a different metric.

Our findings indicate that among the four fusion techniques at the embedding layer, only weighted additive blending consistently outperforms the baseline models, regardless of the underlying model and output modality. Moreover, contrastive learning strategies also enhance model performance and are applicable to both graph and text models. However, the use of cross-attention strategies tends to decrease performance in single-modality setups due to cross-modal interference while improving outcomes of modal fusion. In the molecule captioning task, both contrastive learning and cross-attention mechanisms emerge as critical factors, with the choice of pretraining datasets also influencing outcomes. Although our study predominantly focuses on prevalent data modalities, potentially overlooking some fingerprint modalities, our findings provide comparative insights and highlight areas for deeper investigation in future work.

Scientific insights discovery. To explore the knowledge-learning preferences, we provide an interpretable analysis, revealing the foundational mechanisms in molecular SLM. This analysis confirms the models’ proficiency in encoding chemical knowledge, not only from the NLP perspective but also from a cross-domain viewpoint. However, there remains substantial room for exploration. In Figure 3, we select knowledge patterns with a confidence level exceeding 99%. By lowering the threshold TT, although confidence may decrease, it provides an opportunity to uncover more novel insights, such as exploring unique properties of certain molecules or amino acids. As models increase in complexity

and the corpus of training data grows, the capacity of these systems to delve into and illuminate scientific phenomena also expands. This enlargement not only enhances the depth and accuracy of the models’ predictions but also allows for the exploration of previously unanticipated aspects of molecular behavior. To more clearly showcase the models’ capabilities in scientific understanding, future research should incorporate a broader spectrum of biochemical and physical chemistry knowledge.

Evaluating molecular language modeling. A benchmark must exhibit a balanced distribution, preventing biases toward any specific molecular representation. Analyzing the distribution of text lengths and tokenizer outputs helps confirm that a language model’s tokenizer captures molecular information effectively. Moreover, the diversity in scaffold distribution is crucial for determining the benchmark’s generalization across chemical spaces, ensuring the language model can learn and predict a wide array of chemical entities. Finally, the distribution and correlation of molecular descriptors reveal the complexity within the chemical space. Overall, the visual analyses (Extended Data Figure S1) of text length, scaffold count, and descriptor statistics provide insights into the dataset’s characteristics, affirming the benchmark’s robustness for tasks in molecular science. While we strive to include a broad range of metrics for various tasks, our approach to molecular generation tasks primarily relies on similarity metrics in comparison to target texts.

In future research, we will focus on a sophisticated exploration of multi-modal data fusion to understand how various data modalities enhance model performance. By incorporating scientific perspectives from biochemical and pharmacological fields, we expect to achieve a richer comprehension of the models’ capabilities and potentially uncover more scientific insights. We also plan to develop tailored evaluation metrics for various scientific language modeling tasks, moving beyond simple comparisons to target texts.

Methods

In this section, we present essential background information. We start by defining different molecular representation forms and vital LLMs, highlighting their significance in this field. We then provide a detailed introduction to our benchmark dataset and the setup of our evaluation framework. Finally, we elaborate on our statistically interpretable localized feature filtering method, which facilitates end-to-end analysis of model knowledge-learning preferences.

Molecular representation

Molecular structures can be represented in multiple ways for computational analyses, each with advantages and limitations. These include graph structures, images, as well as one-dimensional notations. Figure 1.a showcases examples of various data representation forms, while Figure 1.b categorizes these representations into molecular internal and external information. Additionally, it illustrates the relationships between these forms of representation and their corresponding tasks.

Molecular internal information. The SMILES offers a widely-used one-dimensional notation for describing molecular structures. InChI provides a unique identifier for chemical substances, facilitating the standard encoding of molecular information. Further advancing molecular representations, SELFIES offers machine learning-oriented approaches, addressing the limitations of SMILES. Moreover, graph structures naturally represent molecules, with atoms as nodes and bonds as edges, effectively capturing both local and global topological traits.

Molecular external information. Images offer a visual 2D or 3D representation of molecules. The IUPAC names provide a standardized naming system for organic compounds. Additionally, molecular properties serve as a form of representation aiding human comprehension. Furthermore, the molecular captions offer rich contextual information, encompassing textual descriptions of functions, properties, and inter-molecular relationships.

Large language models in molecular science

Transformers, known for their self-attention mechanisms, enable the model to weigh the importance of different parts of the input data, which is crucial for understanding complex molecular structures of text modality. LLMs are Transformer-based models scaled to a large extent. Among the notable foundation models in this domain are GPT [23] and BERT [24]. Especially in the fine-tuning stage, these models exhibit remarkable versatility and efficiency, requiring minimal task-specific adjustments for applications in molecular modeling. We present the classifications and architectures, as illustrated in Figure 5. The details of the pivotal developments in transformer-based models for molecular modeling and design are shown in Supplementary Figure 2 and Table 2. To organize the models under a unified framework, we categorize them into three groups based on their architectures.

An overview of model tasks and architectures

Figure 5. An overview of model tasks and architectures. a. Tasks and models. It clarifies the relationship between six downstream tasks and model architectures. b. Encoder-decoder model Architectures. It delineates three main frameworks: (1) text-text is primarily focused on text translation tasks; (2) graph-text is predominantly used in contrastive learning frameworks and serves as an encoder for downstream tasks; (3) image-text is chiefly applied in molecular image recognition tasks.

ArchitectureBackboneModelDateMCINRMPPMRMIRMGOT
Encoder-onlyBERTKV-PLM [39]2022/02•••
CLAMP [60]2023/05•
RoBERTaChemBERTa [27]2020/10•
DMP [61]2021/10•
TransformerMolformer [62]2022/04•
MM-Deacon [63]2022/04••
NoiseMol [64]2023/01•
TransFoxMol [65]2023/01•
Graphormer [66]CAFE-MPP [67]2023/01•
Decoder-onlyGPTMolGPT [68]2021/10◦
Galactica [69]2022/12••
MolXPT [28]2023/05•••
TransformerTaiga [70]2023/05◦
ChatGPTMolReGPT [29]2023/06••
LLM4Mol [71]2023/07•
Text-TextBART [31]Chemformer [72]2022/02◦
MolGen [73]2023/05◦
T5C5T5 [74]2021/08◦
T5Chem [75]2022/03•
MolT5 [21]2022/11••
Text+Chem T5[76]2023/05•••
ChatMol [77]2023/06••
TransAntivirus [38]2023/06•◦
BioT5 [46]2023/10•••
nach0 [78]2023/11••••
TransformerStruct2IUPAC [37]2020/11•
Transformer-R [79]2021/04◦
ChemCrow[80]2023/04◦•
Image-TextSwin Transformer [45]SwinOCSR [44]2022/07•
MolScribe [81]2023/03•
TransformerDECIMER1.0 [41]2021/08•
Image2SMILES[42]2022/01•
MICER [43]2022/08•
Graph-TextSci-BERTText2Mol [40]2021/09•
MoMu [33]2022/09••••
MegaMolBART [72]MoleculeSTM [47]2022/12••◦
BERTMolFM [34]2023/07•••
Q-Former [36]MolCA [22]2023/11••••
KV-PLMMolLM [82]2023/12••••◦
Image-Graph-TextQ-FormerGIT-Mol [35]2023/08••••

Table 2. Overview of models in molecular modeling design categorized by their architecture and backbone. The abbreviations MC, INR, MPP, MR, MIR, MG, and OT correspond to the tasks investigated: molecule captioning, IUPAC name recognition, molecular property prediction, molecular retrieval, molecular image recognition, molecule generation, and other tasks, respectively. Each • symbol in the table denotes the demonstrated task capabilities of the models as reported in their publications. Notably, the ◦ symbol specifically signifies the specialized task of molecular optimization within the broader context of molecule generation.

  • Encoder-based models. BERT and its derivatives like RoBERTa [25] and domain-specific variants such as SciBERT [26] and ChemBERTa [27] have been effectively employed for molecular property prediction.

The typical task can be mathematically represented as: Encoder(X)=HEncoder(X) = H, where XX is the input data and HH denotes the encoded output.

  • Decoder-based models. GPT-2 [23] and ChatGPT [8] are superior in text generation capabilities. Specific-domain models like MolXPT [28], based on GPT-2, and MolReGPT [29], based on ChatGPT, showcase strong performance in generating molecular sequences and descriptions. The typical decoding task can be formulated as: Yt=Decoder(Y<t,C)Y_t = Decoder(Y_{<t}, C), where YtY_t is the output at time tt, Y<tY_{<t} represents all preceding outputs, and CC is the contextual input.

  • Encoder-decoder models. The Transformer architecture itself exemplifies an end-to-end model, with T5 [30] and BART [31] models further enhancing robust end-to-end text generation capabilities, advancing domain-specific applications. Beyond these text-based models, Encoder-decoder frameworks also extend to multi-modal models, accommodating a broader range of data inputs and analytical tasks. The encoder-decoder framework can be captured by the equation: Y=Decoder(Encoder(X))Y = Decoder(Encoder(X)).

Molecule captioning. It aims to develop textual descriptions based on given molecular representations. It involves not only recognizing the molecular structure but also interpreting and conveying its functional properties, potential uses, and any relevant scientific context. MolT5 [21] is a groundbreaking model that uses the T5 architecture for end-to-end text tasks, achieving exceptional performance in single-modality models. Following this, MolXPT [28], built on GPT-2, utilizes a large corpus of PubMed titles and abstracts along with SMILES sequences from PubChem [32]. It performs exceptionally well in few-shot learning and generative pre-training. As for multi-modality models, MoMu [33] and MolFM [34] employ cross-modal contrastive learning techniques. MolCA [22] and GIT-Mol [35] combine Q-Former [36] with contrastive learning to integrate SMILES strings and graphs, leveraging LLMs’ generative capabilities for captioning.

IUPAC name recognition. Unlike free-form textual descriptions, IUPAC names possess the advantage of uniformity and specificity, making them highly structured and predictable. Furthermore, the task of IUPAC name recognition involves not just the decoding of these structured names but also understanding the underlying molecular structure they represent. Struct2IUPAC [37] employs a Transformer-based architecture, which is complemented by a specialized tokenizer adept at processing both SMILES strings and IUPAC nomenclature. Moreover, the TransAntivirus [38] goes a step further by adopting the versatile T5 model architecture.

Molecular property prediction. Molecular representation is critical in molecular modeling, providing ways to describe and encode molecular structures for computational use. Within this domain, molecular property prediction is essential, assessing attributes like solubility, toxicity, and protein binding affinity. Based on BERT, KV-PLM [39] employs a dual tokenizer to better recognize specific atoms and functional groups within SMILES strings. As for multi-modal methods, MoMu [33] applies contrastive learning techniques to assimilate modality-specific and complementary information from text and graph data. Post-encoding, embeddings are transformed into fixed-length vectors via a pooling layer, employing methods like average pooling and max pooling. Average pooling calculates the mean of all sequence vectors for a general context representation. Max pooling selects each feature’s maximum value across the sequence, highlighting key molecular features.

Molecular retrieval. Molecular retrieval aims for efficient, accurate identification of molecules in large datasets. Its effectiveness relies on the type of molecular data used, like structural formulas or SMILES strings. In cross-modal retrieval tasks, Text2Mol [40] excels by combining natural language with molecular data into a unified semantic space for effective retrieval. Similarly, MoMu [33] and MolFM [34] use advanced contrastive learning to merge text and molecular graphics, enhancing the feature space for robust retrieval tasks. As for the pooling layers, similar to those detailed in molecular property prediction.

Molecular image recognition. In our research, molecular image recognition focuses on converting visual representations of molecular structures into standardized, textual formats like SMILES. This task is particularly crucial for automating the interpretation of chemical literature, where molecular images are frequently used to describe compounds. DECIMER 1.0 [41], Image2SMILES [42] and MICER [43] employ Transformer-based architectures and image augmentation, handling large datasets effectively. SwinOCSR [44] utilizes Swin Transformers [45] for global feature extraction and tackles token imbalance with focal loss.

Molecule generation. Our research focuses on developing one-dimensional textual representations, particularly in SMILES format. This includes text-based de novo molecule generation and molecular optimization due to their methodological similarities. In molecular generation, models adept at molecule captioning are equally capable of molecule generation. BioT5 [46] further expands this to include protein-related texts, focusing on SELFIES and extending to protein property and drug-target interaction prediction. Molecular optimization aims to refine molecules for improved properties, aligning with generative goals. MoleculeSTM [47] employs contrastive learning on graphs and text, optimizing target attributes like binding affinity and drug-relatedness with precision.

ChEBI-20-MM and experimental setup

As interest in AI for molecular research continues to grow, an increasing array of datasets is coming into focus. We provide a detailed overview of common molecular datasets in Supplementary Table 1. Meanwhile, molecular representations, particularly molecular text descriptions which are typically scarce, are now seeing increased utilization with MolT5’s ChEBI-20 dataset in applications such as molecule generation and captioning. To facilitate multi-modal data evaluation, we expand our data modality variety, introducing ChEBI-20-MM. This benchmark effectively evaluates the versatility of data types and model combinations in describing, generating, and embedding molecular data. Figure 6 showcases our evaluation of prevalent foundation models and experiments. For comparing models’ embedding capabilities, our experiments include datasets from MoleculeNet, enabling thorough assessments in biochemical and physicochemical contexts.

Benchmark experiments overview

Figure 6. Benchmark experiments overview. Our study encompasses tests across eight primary model architectures, each featuring 3-4 common foundation models or composite models within its category. In total, 1263 experimental setups were conducted, demonstrating the adaptability of various model architectures to different task types. The figure annotates the parameter size of foundation models and labels with items (1-8) to indicate architecture patterns suited for specific tasks.

TypeData SourceContentNo. of EntriesDescription
DatabasePubChem [32]compounds and substances116Mmolecule database
DatabaseZINC [51]compounds and substances230Mvirtual screening database
DatabaseChEMBL [52]compounds with text descriptions2.4Mbioactive molecule database
DatabaseChEBI [53]small chemical compounds61,032molecular entity dictionary
DatabaseDrugBank [54]small molecule drugs12,695drug/target database
DatabaseUSPTOpatent filings and data——scientific corpus
DatasetC4 [30]Clean English text——Pretrain
DatasetMoMu dataset [33]Graph-document pairs15,613Pretrain
DatasetMOSES [55]Molecular structures1.9MGeneration
DatasetPCdes [39]SMILES-text descriptions1.5KMolecular retrieval
DatasetChEBI-20 [40]SMILES-text descriptions33,010Generation and captioning
DatasetPubChemSTM [47]SMILES-text descriptions281KGeneration
DatasetMol-Instructions [13]SMILES-text descriptions333KGeneration and captioning
DatasetMoleculeNet [19]Molecular properties.700KProperty prediction
DatasetUSPTO(OSRA) [56]Images and ground truth5,719Image recognition
DatasetCLEF [57]Images and ground truth865Image recognition
DatasetJPO [56]Images and ground truth450Image recognition
DatasetUOB [58]Images and ground truth5,740Image recognition
DatasetUSPTO-50k [59]Chemical reactions50KReactions prediction

Table 1. Overview of data sources for molecular science

Dataset construction. The ChEBI-20 dataset includes molecular SMILES and captions. Building on this, we use tools such as RDKit to supplement the dataset with InChI, graphs, SELFIES, and other internal information based on the SMILES. Additionally, IUPAC names and images are sourced from PubChem. During the dataset curation process, we maintain the original ChEBI-20 split of 8:1:1 (train: validation: test) and remove molecules that can not be converted to SELFIES to ensure the validity of all molecular data. In total, the refined dataset comprises 32,998 molecules, which are suitable for the text generation tasks depicted in Figure 1, including text generation based on multi-modal fusion.

Image not extracted

Figure 1. The paradigm of the analysis. a. Molecular modeling and design tasks, showcasing six task types with their standard modeling methods and data examples. b. The paradigms of tasks, we divide common molecular data into two categories: internal and external information. Internal information, integral to molecular representation, can be converted through various tools. External information is more accessible to human understanding. Additionally, this part highlights the research scope of our analysis, detailing the input and output for each task.

Data distribution. We analyze the suitability of data sources for language models and chemical space coverage. As shown in Extended Data fig:extended-data-1, we use different visualization methods to analyze the distribution of text lengths and the number of tokens generated by each model’s tokenizer for various text data types. This approach allows us to evaluate the adaptability of language models to the textual characteristics of our dataset. We also focus on the top 10 scaffolds within the dataset, counting the number of molecules for each scaffold. Here, semi-transparent bars represent the total count, while solid bars indicate the quantity in the training set. On the other hand, for the analysis of chemical space coverage, we choose molecular weight (MW), LogP, the number of aromatic rings, and the Topological Polar Surface Area (TPSA) as descriptors. We examine the distribution and correlation of these descriptors within the dataset, providing insights into the chemical diversity and complexity present in our data.

Experimental setup. We conduct 1263 experiments, encompassing 291 generative and retrieval tasks from the ChEBI-20-MM benchmark and 972 property prediction experiments. For nine text-to-text tasks, our models include 11 architectural variations (items 3, 4, 5 in Figure 6), combining three decoder-only models, four encoder-decoder models, and four composite models with MolT5 as the decoder paired with various encoders. We conduct experiments based on the setup, performing each experiment three times to calculate the mean and standard deviation of the results. For nine text-to-text translation tasks, our models encompass 11 architectural variations (as illustrated in Figure 6, items 4, 5, 6), which include three decoder-only models, four encoder-decoder models, and four composite models featuring MolT5 as the decoder paired with various encoders. For two graph-to-text tasks, we deploy 12 distinct architectures (item 7), and for two image-to-text tasks, we utilize 16 different architectures (item 8). Additionally, for two retrieval and nine property prediction tasks, we implement 11 encoder architectures (items 1, 2, 3, 5) alongside native encoder-decoder configurations serving as molecular encoders.

Metrics. We use a combination of BLEU-2, BLEU-4, ROUGE-1, ROUGE-2, ROUGE-L, and METEOR scores for IUPAC name recognition and molecule captioning. Moreover, for image recognition and molecule generation, we assess models using BLEU for textual similarity, Exact Match for precision, Valid for chemical validity of generated molecules, Levenshtein distance for structural similarity, and fingerprint-based metrics such as MACCS, RDKit, and Morgan for capturing chemical feature representation. Molecular retrieval is evaluated using mean reciprocal rank (MRR) and Recall at K (R@1, R@5, R@10). For the property prediction metrics, we utilize widely recognized classification and regression tasks from MoleculeNet. For classification, we measure the ROC-AUC, the area under the precision-recall curve (PR_AUC), and the mean F1 score (F1_score). For regression tasks, we use mean squared error (MSE_mean), root mean squared error (RMSE_mean), and mean absolute error (MAE_mean) to assess the predictive accuracy of the models quantitatively.

Analysis of model knowledge-learning preferences

The interpretability methods have become more important in LLMs, particularly in molecular science. As research increasingly validates the effectiveness of these LLMs, a significant challenge remains their ‘black box’ nature. This opacity limits our understanding of what knowledge the models have actually learned. Furthermore, it restricts the design and application of these models. While existing interpretability methods, such as case study-based attention and embedding visualizations, offer some insights, they often lack generalizability and provide limited understanding to researchers. Therefore, there is a pressing need for a unified analysis of batch data that can extract and distill insights into the models’ knowledge-learning preferences and present them in an interpretable textual format. To explore the knowledge-learning preferences of models in the process of molecular SLM, this section introduces a language-driven, statistically significant, interpretable method. This tool constructs a token mapping matrix between input data and inference outcomes, revealing the model’s mechanism.

Particular high-frequency token mappings. We decompose the input data and inference outcomes of LLMs into corresponding tokens and select high-frequency tokens to construct a mapping matrix, denoted as AA. Additionally, we apply special processing to these tokens, such as filtering out tokens related to chemical vocabulary in captions and removing irrelevant words. We typically view high-frequency token mappings as indicative of specialized knowledge. Our approach identifies two types of high-frequency token mappings: general pairs, which are widespread and lack specific context, and particular pairs, essential for certain contexts but not always the most frequent. The challenge arises when standard analysis methods prioritize these general high-frequency mappings, consequently overlooking the specific ones that often carry more valuable contextual insights.

Localized feature filtering method. We utilize a localized feature filtering method for high-frequency selection to eliminate common high-frequency mapping pairs, thus more accurately filtering the model’s knowledge mappings. Subsequently, a statistical significance method is applied to determine the filtering threshold, identifying high-frequency mapping pairs that reflect the model’s chemical knowledge.

Proof Point 1: Sorting the matrix AA in descending order by row and column totals enhances the similarity among adjacent values. Assuming each AijA_{ij} follows an independent and identical distribution, such as normal distribution, the row sums RiR_i and column sums CjC_j mimic this distribution. Consequently, in the sorted matrix A′A', elements next to each other are more likely to come from rows or columns with similar totals.

Considering Akl′A'_{kl} and its neighbors Ak±1,l±1′A'_{k\pm1,l\pm1} in A′A', which were close in the sum in the original matrix AA, the probability they are similar is mathematically represented as:

P(Akl′≈Ak±1,l±1′∣Ri≈Ri±1,Cj≈Cj±1)>P(Aij≈Ai±1,j±1)(1)P(A'_{kl} \approx A'_{k\pm1,l\pm1} \mid R_i \approx R_{i\pm1}, C_j \approx C_{j\pm1}) > P(A_{ij} \approx A_{i\pm1,j\pm1}) \tag*{(1)}

This highlights that adjacent elements in A′A' are more likely to have close values compared to those in AA, due to the sorting based on similar row and column sums.

Proof Point 2: To uncover significant patterns within a matrix, we introduce a threshold TT, aiming to spotlight elements AijA_{ij} that significantly surpass the mean plus a certain number of standard deviations of their neighboring values. This process starts with assuming elements in matrix AA are drawn from a distribution, enabling the calculation of each element’s neighborhood mean μneighbor\mu_{neighbor} and standard deviation σneighbor\sigma_{neighbor}. This step evaluates whether an element exceeds the set threshold, the condition to filter elements that starkly differ from their surrounding context:

Aij>μneighbor+T⋅σneighbor(2)A_{ij} > \mu_{neighbor} + T \cdot\sigma_{neighbor} \tag*{(2)}

To assess the actual significance of these findings, we calculate the actual proportion PactualP_{actual} of elements exceeding this threshold across the matrix and compare it to the expected proportion PexpectedP_{expected} under a normal distribution. The actual and expected proportion is determined by the fraction of elements satisfying our condition:

Pactual=∑Iijn×m(3)P_{actual} = \frac{\sum I_{ij}}{n \times m} \tag*{(3)}
Pexpected=1−Φ(μneighbor+T⋅σneighbor−μσ)(4)P_{expected} = 1 - \Phi\left(\frac{\mu_{neighbor} + T \cdot\sigma_{neighbor} - \mu}{\sigma}\right) \tag*{(4)}

where IijI_{ij} is an indicator function that equals 1 when the condition is met and 0 otherwise, and n×mn \times m represents the matrix size. The Φ\Phi represents the cumulative distribution function of the standard normal distribution, and μ\mu and σ\sigma are the mean and standard deviation of the entire matrix’s elements.

Finally, a statistical test, such as the Z-test, evaluates the significance of the observed proportion against the expected, indicating whether the identified patterns are indeed statistically significant beyond random chance:

Z=Pactual−PexpectedPexpected(1−Pexpected)n×m(5)Z = \frac{P_{actual} - P_{expected}}{\sqrt{\frac{P_{expected}(1-P_{expected})}{n \times m}}} \tag*{(5)}

This succinct methodology not only clarifies the process of identifying statistically significant patterns but also underlines its effectiveness in distinguishing genuine high-frequency elements from those of a random distribution, offering valuable insights into the structure of the knowledge-learning preferences of models.

Comparison with other interpretability methods. Interpretability in machine learning, especially within molecular science, utilizes various methods to render the decision-making processes of models both accessible and comprehensible. As shown in Supplementary Table 12, researchers frequently employ techniques such as text-based attention map visualizations and molecular graph-based attention heatmaps. These methods typically rely on inductive reasoning from specific case studies, employing a bottom-up approach to derive insights. While these methods can provide valuable insights, their dependence on extensive case studies renders the process time-consuming and complex. Furthermore, the insights obtained through these traditional methods often require further verification to confirm their validity, thus challenging their reliability and general applicability. In contrast, our approach, termed the localized feature filtering method, initiates the process of insight derivation through statistical significance techniques. We can set thresholds to balance the volume of insights with their confidence levels, employing a top-down validation approach to substantiate the effectiveness of our selected insights. This method not only ensures the credibility of the insights but also significantly boosts the efficiency of insight extraction. By melding rigorous statistical analysis with a systematic validation framework, our approach enhances the interpretability of molecular models, offering a robust and efficient alternative to traditional, more heuristic methods.

Utility and potential application. The LLMs demonstrate significant superiority in molecular science. While their performance in representation tasks may align with that of traditional approaches and graph models, their unparalleled ability to generate coherent and contextually relevant text sets them apart. Our benchmark and modal transition matrix are designed to rapidly identify optimal molecular modalities, methods, and training strategies for specialized tasks within molecular science. For instance, the BioT5+ model [48], which utilizes molecular IUPAC names and SELFIES for T5-based caption generation, achieves state-of-the-art results. It illustrates a new use of IUPAC names, a previously unrecognized strategy in the field, which is also concluded by our benchmark analysis.

The localized feature filtering method facilitates the rapid extraction of a model’s knowledge-learning preferences. This capability is instrumental in guiding model training, for example, by optimizing the tokenizer or refining model parameters to boost learning efficiency. Furthermore, it allows for the strategic adjustment of the threshold TT to unearth potential scientific insights. By exploring these insights, researchers can probe new dimensions of molecular behavior, paving the way for innovative breakthroughs in molecular design and synthesis. Through these applications and methodological advancements, our approach demonstrates its utility, providing a robust framework for enhancing both the interpretability and functionality of models in molecular science.

Acknowledgments

This work is supported by grants from the National Natural Science Foundation of China (61902446, 62172456, and 91937302), the Major Key Project of PCL PCL2021A13, and Peng Cheng Cloud-Brain.

Author Contributions Statement

Pengfei Liu: Conceptualization, Methodology, Data curation, Model training, Writing-original draft. Jun Tao: Funding acquisition, Writing review editing. Zhixiang Ren: Conceptualization, Formal analysis, Supervision, Funding acquisition, Writing review editing.

Competing Interests Statement

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Data availability

The dataset ChEBI-20-MM [49] can be accessed at https://huggingface.co/datasets/liupf/ChEBI-20-MM.

Code availability

The open-source project repository [50] is available in GitHub at https://github.com/AI-HPC-Research-Team/SLM4Mol. It is available for non-commercial use.

Visualization of data source suitability and chemical space diversity

Figure 1 (PDF p. 21). Visualization of data source suitability and chemical space diversity.

The databases and datasets for molecular science

In this section, we highlight vital data sources essential for molecular science research. As detailed in Supplementary Table 1, PubChem and ZINC databases provide extensive compounds for virtual screening. ChEMBL offers comprehensive data on bioactive molecules, while ChEBI focuses on cataloging small chemical compounds. DrugBank serves as a valuable resource for drug and target information. Lastly, the USPTO database, encompassing patent filings and research data, forms a vital part of the scientific literature.

Moreover, various vital datasets support a wide range of research, from pretraining to specific molecular science tasks. As shown in Supplementary Table 1, the Colossal Clean Crawled Corpus (C4) dataset offers a vast amount of clean English text scraped from the web for the pretraining of the T5 model. MOSES provides molecular structures from ZINC, while MoleculeNet encompasses a broad spectrum of molecular property predictions ranging from quantum mechanics to biophysics and physiology. Additionally, datasets such as USPTO, CLEF, JPO, and UOB offer molecular images with ground truth, supporting image recognition tasks. The PCdes, ChEBI-20, and PubChemSTM datasets, along with Mol-Instructions, provide essential SMILES-text pairs that facilitate cross-information retrieval, molecule generation, and molecular captioning tasks.

Supplementary case studies on the knowledge-learning preferences of models

This section presents additional application cases for our localized feature filtering method beyond the SELFIES to caption and IUPAC to caption tasks discussed in the manuscript. The method can support a variety of molecular modalities, all of which are text-generation tasks that can benefit from this analysis. As shown in Figure 1 of the manuscript, our method is applicable to tasks such as molecule captioning, IUPAC name recognition, and molecule generation. Due to the relatively low frequency of InChI usage, we present only a few modalities, such as SMILES, SELFIES, IUPAC names, and captions. As illustrated in Figure 3 of the manuscript, we have already demonstrated the SELFIES to caption and IUPAC to caption tasks. Furthermore, as illustrated in Supplementary Figure 2, we analyze the recognition of IUPAC names and the generation of molecules. We also present insights into the model’s knowledge-learning preferences, which include Z-test results, confidence levels, and specific high-frequency token mappings.

The details of models

We present a timeline showcasing the pivotal developments in transformer-based models for molecular modeling and design, alongside classifications and architectures, as illustrated in Supplementary Figure S1. An overview of model classifications and architectures is shown in Supplementary Table 2.

Timeline of the key developments of LLMs and Transformer-based models

Figure 2 (PDF p. 25). Timeline of the key developments of LLMs. The models based on the Transformer architecture are differentiated by various colors to denote that they are based on the nine foundation models.

Experimental results

SMILES Generation. Supplementary Table 3 illustrates the performance of SMILES generation across two methodologies: image recognition and text-based SMILES generation. The image recognition task is an essential task for chemical image understanding. The results show that models leveraging Swin Transformer (SwinOCSR) for encoding and t5 for decoding effectively generate SMILES from visual inputs.

Within the text-based SMILES generation, the combination of IUPAC nomenclature with the t5 model emerges as the most effective, highlighting the model’s ability to process and convert complex chemical language into accurate SMILES structures. Moreover, the molt5 model’s superior performance in generating molecules from captions suggests that its pre-training on SMILES expressions is advantageous for this specific task, emphasizing the value of domain-specific pre-training in enhancing model performance.

IUPAC Generation. Supplementary Table 4 emphasizes the tasks associated with IUPAC Generation, focusing on the recognition of IUPAC names from molecular images and internal information. Our benchmark does not include experiments for generating IUPAC names from captions because some captions already contain IUPAC names.

For IUPAC name recognition from the molecular images, the combination of swinocsr and t5 models yields the best performance, similar to the SMILES generation task. It suggests that the swinocsr, originally pre-trained on molecular SMILES and images, is also adept at transfer learning for IUPAC name recognition tasks. On the other hand, combining SMILES data with the t5 model achieves the best results for IUPAC recognition based on molecular internal information. Although graph-based models show only slightly lower performance in textual similarity metrics compared to language models, they exhibit a relative shortfall in achieving exact matches.

Molecule Captioning. The molecular captions encapsulate the essence of the molecule along with its biochemical and physicochemical properties. Molecule captioning involves the conversion of various representation forms into captions. As shown in Supplementary Table 4, unlike image recognition tasks for SMILES and IUPAC names, experiments on caption generation show that performance based on IUPAC and SMILES outperforms those based on image-derived captions. It suggests that image encoders excel at distilling key points from images. However, as the diversity of generated content increases and the length of the generated sequences extends, the performance of image encoder models tends to fall short of language models. Within the scope of language models, text description generation based on IUPAC is superior, partly because the target captions commonly include IUPAC names. Additionally, graph encoders exhibit weaker performance in text generation tasks, further emphasizing the strengths of language models in handling complex linguistic structures and content variability.

Molecular Retrieval. In molecular cross-modal retrieval tasks, the focus is primarily on the retrieval between molecular external information and either internal or external information. We categorize the retrieval tasks into two main types, targeting captions and IUPAC, further subdivided into nine specific retrieval tasks. As shown in Supplementary Table 5, the IUPAC-to-caption and SMILES-to-IUPAC tasks achieve the best results in their respective categories. Regarding the models, the t5 family demonstrates superiority over bart, encoder-only models, and graph encoders.

Property Prediction. We categorize the tasks into classification and regression, testing them across scaffold and random data splitting methodologies. The evaluation extends to nine distinct datasets, each encompassing three text modalities, eight text encoders, and three graph models [4][5][6], integrated with two different pooling strategies.

TaskInputEncoderDecoderBleu ↑Exact Match ↑Levenshtein ↓MACCS ↑RDK ↑Morgan ↑Validity ↑
image recognition (SMILES)imageswinocsrt50.8850.35612.7460.9510.8900.8680.908
image recognition (SMILES)imageswinocsrt5110.8330.17518.5890.8970.7850.7650.761
image recognition (SMILES)imagevitt50.8260.17919.8020.8890.7800.7340.873
image recognition (SMILES)imageswint50.7820.06724.7260.7940.6330.5810.891
image recognition (SMILES)imageswint5110.7790.06625.2030.8020.6480.5970.850
molecule generation (text-based)IUPACt5t50.8810.32714.2090.9090.8110.7730.905
molecule generation (text-based)IUPACt511t5110.8810.32314.1410.9110.8080.7710.893
molecule generation (text-based)IUPACmolt5molt50.8530.20617.6450.8810.7580.7230.773
molecule generation (text-based)IUPACscibertmolt50.6140.05451.0030.7720.5900.5260.680
molecule generation (text-based)IUPACbertt50.2960.00096.4840.4090.2680.1280.988
molecule generation (text-based)captionmolt5molt50.7540.09028.0340.8080.6730.5940.837
molecule generation (text-based)captiont5t50.6910.09734.8910.8140.6770.6000.858
molecule generation (text-based)captiont511t5110.6490.11040.5690.8190.6870.6080.857
molecule generation (text-based)captionscibertmolt50.5430.01257.1180.6700.4920.4310.693
molecule generation (text-based)captionchembertamolt50.4250.00273.0610.5610.4080.3250.740

Table Supplementary 3. Experimental results for SMILES generation

We have selected several commonly used traditional machine learning methods from MoleculeNet, as well as typical GNN-based and transformer-based models. The results in Supplementary Table 6 and Table 7 allow us to analyze the performance differences between traditional methods, graph models, and transformer-based models over recent years. Each type of model excels in specific tasks. However, both GNN-based and transformer-based models can enhance their performance through self-supervised pre-training strategies. Furthermore, transformer-based models, with their larger parameter count, possess a higher potential for capturing complex patterns. Consequently, each dataset is tested with 108 unique experimental setups. For each combination of input data and model, we select the top five performing models, as shown in Supplementary Table 8 and Table 9.

The scaffold data splitting method inherently increases the complexity of predictions. Unlike molecular retrieval tasks, where the focus is on embedding tasks with textual targets, leading to a better performance of language models over graph encoders. Here, graph encoders generally outperform language models in most cases.

TaskInputEncoderDecoderBleu-2 ↑Bleu-4 ↑Rouge-1 ↑Rouge-2 ↑Rouge-L ↑Meteor-score ↑Exact Match ↑
IUPAC recognition (image)imageswinocsrt50.8200.7580.7810.6210.7400.7990.220
IUPAC recognition (image)imageswinocsrmolt50.8030.7300.7510.5730.7070.7770.158
IUPAC recognition (image)imageswinocsrt5110.8000.7300.7570.5770.7110.7790.170
IUPAC recognition (image)imagevitt5110.7450.6550.6570.4450.6030.6770.088
IUPAC recognition (image)imagevitmolt50.7250.6320.6440.4310.5890.6670.069
IUPAC recognitionSMILESt5t50.8200.7550.7510.5730.7030.7670.210
IUPAC recognitionSMILESt511t5110.7940.7190.7110.5150.6600.7320.146
IUPAC recognitionSMILESchembertamolt50.7860.7100.7170.5240.6690.7390.134
IUPAC recognitionSMILESmolt5molt50.7520.6650.6690.4590.6150.6880.095
IUPAC recognitionSMILESbartbart0.5980.5000.6840.4630.6240.6760.170
IUPAC recognitionInChIt5t50.7440.6580.6480.4420.5910.6630.131
IUPAC recognitionInChIt511t5110.7280.6360.6220.4100.5650.6410.088
IUPAC recognitionInChImolt5molt50.7280.6350.6220.4130.5660.6420.093
IUPAC recognitionInChIchembertamolt50.6460.5410.5500.3260.4930.5740.037
IUPAC recognitionInChIrobertamolt50.6200.5050.4940.2490.4290.5090.012
IUPAC recognitionSELFIESmolt5molt50.7510.6660.6650.4580.6110.6870.111
IUPAC recognitionSELFIESt5t50.7440.6580.6680.4650.6150.6920.132
IUPAC recognitionSELFIESt511t5110.7200.6330.6550.4400.6010.6770.097
IUPAC recognitionSELFIESbiogptbiogpt0.6690.5850.5940.3910.5340.6020.005
IUPAC recognitionSELFIESbartbart0.5500.4330.5680.3110.5010.5690.063
IUPAC recognitiongraphgcnt5110.7190.6430.6400.4360.5910.6670.056
IUPAC recognitiongraphgatt50.7080.6300.6200.4200.5720.6520.046
IUPAC recognitiongraphgatt5110.6850.6030.5970.3930.5500.6280.037
IUPAC recognitiongraphgint50.6820.6000.6150.4030.5580.6530.052
IUPAC recognitiongraphginmolt50.6620.5800.6050.3950.5550.6430.029
IUPAC recognitioncaptiont511t5110.7360.6650.7070.5320.6600.7160.185
IUPAC recognitioncaptionbiogptbiogpt0.7180.6550.6500.4790.6050.6730.000
IUPAC recognitioncaptionmolt5molt50.6990.6150.6520.4600.6040.6670.129
IUPAC recognitioncaptiont5t50.6210.5450.6750.4950.6270.6790.183
IUPAC recognitioncaptionscibertmolt50.5960.5030.5310.3140.4780.5680.025
molecule captioningIUPACt5t50.5470.4570.6290.4760.5660.579/
molecule captioningIUPACt511t5110.5350.4460.6220.4690.5590.571/
molecule captioningIUPACmolt5molt50.5250.4330.6120.4550.5490.556/
molecule captioningIUPACbartbart0.4550.3520.5680.3910.4890.484/
molecule captioningIUPACchembertamolt50.4240.3330.5500.3860.4920.490/
molecule captioningSMILESmolt5molt50.5350.4360.6250.4660.5590.563/
molecule captioningSMILESt5t50.5080.4140.6050.4460.5450.543/
molecule captioningSMILESt511t5110.4880.3920.5860.4260.5280.522/
molecule captioningSMILESchembertamolt50.4880.3880.5780.4170.5220.520/
molecule captioningSMILESbartbart0.4060.3010.5380.3590.4650.443/
molecule captioningInChIt5t50.4770.3830.5770.4150.5180.511/
molecule captioningInChIt511t5110.4640.3690.5670.4040.5100.499/
molecule captioningInChImolt5molt50.4610.3670.5710.4060.5130.500/
molecule captioningInChIchembertamolt50.4510.3560.5570.3940.5020.490/
molecule captioningInChIbartbart0.3930.2840.5190.3350.4470.429/
molecule captioningSELFIESt5t50.4620.3600.5690.4000.5060.503/
molecule captioningSELFIESt511t5110.4460.3480.5590.3920.4990.490/
molecule captioningSELFIESchembertamolt50.4390.3400.5490.3810.4910.478/
molecule captioningSELFIESbartbart0.3770.2660.5100.3200.4340.414/
molecule captioningSELFIESbiogptbiogpt0.2770.2040.4250.2500.3550.345/
molecule captioninggraphgatmolt50.4190.3180.5210.3450.4600.449/
molecule captioninggraphginmolt50.4070.3050.5120.3320.4500.437/
molecule captioninggraphgcnmolt50.3920.2910.5120.3330.4500.430/
molecule captioninggraphgint5110.3620.2560.4790.2950.4180.401/
molecule captioninggraphgcnbart0.3580.2510.4890.2980.4130.391/
molecule captioningimageswinocsrmolt50.4950.3990.5900.4290.5320.527/
molecule captioningimageswinocsrt5110.4920.3940.5850.4220.5270.524/
molecule captioningimageswinocsrt50.4170.3170.5330.3610.4720.461/
molecule captioningimageswinmolt50.4090.3070.5170.3430.4600.442/
molecule captioningimageswint5110.4000.3010.5170.3440.4610.435/

Table 4. Experimental results for IUPAC and caption generation

TaskInputEncoderPoolingTarget ModalityR@1 ↑MRR ↑R@5 ↑R@10 ↑
molecular retrieval (caption)IUPACt5maxcaption0.637 ± 0.0240.741 ± 0.0220.880 ± 0.0190.936 ± 0.012
molecular retrieval (caption)IUPACscibertavgcaption0.635 ± 0.0840.736 ± 0.0760.870 ± 0.0630.935 ± 0.051
molecular retrieval (caption)IUPACscibertmaxcaption0.581 ± 0.0610.694 ± 0.0560.848 ± 0.0460.918 ± 0.036
molecular retrieval (caption)IUPACmolt5avgcaption0.578 ± 0.0260.701 ± 0.0260.869 ± 0.0270.931 ± 0.016
molecular retrieval (caption)IUPACt5avgcaption0.564 ± 0.0350.685 ± 0.0340.855 ± 0.0320.923 ± 0.022
molecular retrieval (caption)SMILESbartavgcaption0.440 ± 0.0430.589 ± 0.0430.797 ± 0.0440.895 ± 0.033
molecular retrieval (caption)SMILESt5maxcaption0.437 ± 0.0460.588 ± 0.0460.800 ± 0.0450.892 ± 0.028
molecular retrieval (caption)SMILESt5avgcaption0.427 ± 0.0560.583 ± 0.0560.797 ± 0.0560.897 ± 0.037
molecular retrieval (caption)SMILESrobertamaxcaption0.426 ± 0.0700.572 ± 0.0750.772 ± 0.0820.880 ± 0.066
molecular retrieval (caption)SMILESmolt5avgcaption0.402 ± 0.0540.557 ± 0.0550.772 ± 0.0590.875 ± 0.037
molecular retrieval (caption)InChIt5avgcaption0.339 ± 0.0330.504 ± 0.0360.733 ± 0.0400.858 ± 0.030
molecular retrieval (caption)InChIscibertavgcaption0.337 ± 0.0760.498 ± 0.0950.719 ± 0.1210.843 ± 0.111
molecular retrieval (caption)InChIt5¹¹avgcaption0.325 ± 0.0530.488 ± 0.0620.717 ± 0.0770.846 ± 0.062
molecular retrieval (caption)InChIbartavgcaption0.289 ± 0.0360.444 ± 0.0450.661 ± 0.0590.808 ± 0.051
molecular retrieval (caption)InChImolt5avgcaption0.280 ± 0.0550.433 ± 0.0640.646 ± 0.0790.790 ± 0.068
molecular retrieval (caption)SELFIESrobertamaxcaption0.436 ± 0.0790.579 ± 0.0810.778 ± 0.0860.875 ± 0.069
molecular retrieval (caption)SELFIESscibertmaxcaption0.406 ± 0.0530.553 ± 0.0570.757 ± 0.0630.875 ± 0.051
molecular retrieval (caption)SELFIESbertmaxcaption0.393 ± 0.0550.546 ± 0.0630.751 ± 0.0730.868 ± 0.062
molecular retrieval (caption)SELFIESbartavgcaption0.367 ± 0.0530.520 ± 0.0570.731 ± 0.0650.853 ± 0.055
molecular retrieval (caption)SELFIESmolt5maxcaption0.339 ± 0.0410.492 ± 0.0450.709 ± 0.0540.835 ± 0.045
molecular retrieval (caption)graphgatmaxcaption0.259 ± 0.0600.406 ± 0.0790.610 ± 0.1110.765 ± 0.111
molecular retrieval (caption)graphgcnmaxcaption0.242 ± 0.0480.383 ± 0.0590.576 ± 0.0770.724 ± 0.068
molecular retrieval (caption)graphginmaxcaption0.241 ± 0.0300.384 ± 0.0400.577 ± 0.0530.732 ± 0.053
molecular retrieval (caption)graphgatavgcaption0.189 ± 0.0460.319 ± 0.0680.501 ± 0.1020.648 ± 0.108
molecular retrieval (caption)graphginavgcaption0.185 ± 0.0380.309 ± 0.0580.476 ± 0.0860.630 ± 0.099
molecular retrieval (caption)imageswinocsrmaxcaption0.411 ± 0.1240.552 ± 0.1460.748 ± 0.1810.844 ± 0.164
molecular retrieval (caption)imageswinocsravgcaption0.384 ± 0.0940.534 ± 0.1060.747 ± 0.1270.856 ± 0.107
molecular retrieval (caption)imageresnetavgcaption0.233 ± 0.0410.385 ± 0.0550.587 ± 0.0750.753 ± 0.073
molecular retrieval (caption)imagevitmaxcaption0.164 ± 0.0450.285 ± 0.0680.443 ± 0.0990.607 ± 0.116
molecular retrieval (caption)imagevitavgcaption0.121 ± 0.0190.239 ± 0.0340.399 ± 0.0560.574 ± 0.070
molecular retrieval (IUPAC)SMILESt5avgIUPAC0.709 ± 0.0690.817 ± 0.0540.958 ± 0.0330.986 ± 0.015
molecular retrieval (IUPAC)SMILESt5¹¹maxIUPAC0.698 ± 0.1160.802 ± 0.0960.941 ± 0.0650.975 ± 0.032
molecular retrieval (IUPAC)SMILESscibertavgIUPAC0.681 ± 0.1470.789 ± 0.1270.930 ± 0.0970.967 ± 0.060
molecular retrieval (IUPAC)SMILESt5¹¹avgIUPAC0.655 ± 0.0990.775 ± 0.0820.930 ± 0.0550.970 ± 0.028
molecular retrieval (IUPAC)SMILESbartmaxIUPAC0.629 ± 0.0900.759 ± 0.0750.935 ± 0.0510.978 ± 0.025
molecular retrieval (IUPAC)InChIt5avgIUPAC0.545 ± 0.0640.687 ± 0.0590.877 ± 0.0540.945 ± 0.028
molecular retrieval (IUPAC)InChIt5maxIUPAC0.519 ± 0.0740.661 ± 0.0650.850 ± 0.0510.931 ± 0.031
molecular retrieval (IUPAC)InChIt5¹¹avgIUPAC0.518 ± 0.1140.665 ± 0.1110.867 ± 0.1060.936 ± 0.073
molecular retrieval (IUPAC)InChIbartavgIUPAC0.502 ± 0.0850.662 ± 0.0840.882 ± 0.0850.950 ± 0.049
molecular retrieval (IUPAC)InChIt5¹¹maxIUPAC0.499 ± 0.0990.654 ± 0.0940.868 ± 0.0900.945 ± 0.046
molecular retrieval (IUPAC)SELFIESt5¹¹maxIUPAC0.626 ± 0.0680.753 ± 0.0570.920 ± 0.0380.972 ± 0.021
molecular retrieval (IUPAC)SELFIESbartavgIUPAC0.603 ± 0.0940.731 ± 0.0830.906 ± 0.0670.962 ± 0.038
molecular retrieval (IUPAC)SELFIESt5avgIUPAC0.570 ± 0.0760.706 ± 0.0690.887 ± 0.0590.950 ± 0.035
molecular retrieval (IUPAC)SELFIESbartmaxIUPAC0.508 ± 0.0950.662 ± 0.0940.876 ± 0.0970.947 ± 0.063
molecular retrieval (IUPAC)SELFIESmolt5avgIUPAC0.506 ± 0.1120.643 ± 0.1120.828 ± 0.1140.912 ± 0.080
molecular retrieval (IUPAC)graphgcnmaxIUPAC0.279 ± 0.0670.421 ± 0.0840.621 ± 0.1130.756 ± 0.107
molecular retrieval (IUPAC)graphginmaxIUPAC0.272 ± 0.0680.423 ± 0.0880.631 ± 0.1180.773 ± 0.114
molecular retrieval (IUPAC)graphginavgIUPAC0.216 ± 0.0710.352 ± 0.1040.536 ± 0.1520.684 ± 0.169
molecular retrieval (IUPAC)graphgatavgIUPAC0.204 ± 0.0740.336 ± 0.1120.518 ± 0.1700.669 ± 0.191
molecular retrieval (IUPAC)graphgcnavgIUPAC0.186 ± 0.0300.321 ± 0.0460.500 ± 0.0680.661 ± 0.080
molecular retrieval (IUPAC)imageswinmaxIUPAC0.696 ± 0.0830.803 ± 0.0690.945 ± 0.0500.979 ± 0.030
molecular retrieval (IUPAC)imageswinavgIUPAC0.462 ± 0.1040.608 ± 0.1050.808 ± 0.1040.900 ± 0.076
molecular retrieval (IUPAC)imageresnetavgIUPAC0.289 ± 0.0570.450 ± 0.0700.674 ± 0.0910.834 ± 0.080
molecular retrieval (IUPAC)imageresnetmaxIUPAC0.215 ± 0.0700.376 ± 0.1100.598 ± 0.1700.761 ± 0.188
molecular retrieval (IUPAC)imagevitmaxIUPAC0.215 ± 0.0800.365 ± 0.1170.575 ± 0.1720.740 ± 0.178

Table 5. Supplementary Table 5: Experimental results for retrieval

TaskData SplittingDatasetInputEncoderPoolingroc_auc ↑pr_auc ↑f1_score ↑
property prediction (classification)scaffoldbaceSELFIESrobertamax0.742 ± 0.0360.750 ± 0.0130.544 ± 0.126
property prediction (classification)scaffoldbaceSELFIESmolt5max0.708 ± 0.0650.678 ± 0.0430.391 ± 0.033
property prediction (classification)scaffoldbaceInChIbartavg0.704 ± 0.0580.699 ± 0.0630.516 ± 0.112
property prediction (classification)scaffoldbaceInChIt5max0.703 ± 0.0300.699 ± 0.0430.646 ± 0.032
property prediction (classification)scaffoldbaceInChImolt5max0.678 ± 0.0390.699 ± 0.0360.565 ± 0.038
property prediction (classification)scaffoldbbbpgraphginavg0.652 ± 0.0090.684 ± 0.0240.528 ± 0.026
property prediction (classification)scaffoldbbbpgraphgatavg0.651 ± 0.0130.691 ± 0.0130.470 ± 0.057
property prediction (classification)scaffoldbbbpSMILESmolt5avg0.651 ± 0.0100.692 ± 0.0100.602 ± 0.034
property prediction (classification)scaffoldbbbpgraphgcnmax0.645 ± 0.0110.693 ± 0.0130.492 ± 0.118
property prediction (classification)scaffoldbbbpgraphgcnavg0.641 ± 0.0120.675 ± 0.0140.536 ± 0.057
property prediction (classification)scaffoldclintoxSELFIESt5llavg0.923 ± 0.0030.722 ± 0.0470.483 ± 0.000
property prediction (classification)scaffoldclintoxInChIt5avg0.908 ± 0.0110.700 ± 0.0220.483 ± 0.000
property prediction (classification)scaffoldclintoxSELFIESt5avg0.899 ± 0.0640.749 ± 0.0960.483 ± 0.000
property prediction (classification)scaffoldclintoxSELFIESt5max0.874 ± 0.0200.721 ± 0.0340.381 ± 0.144
property prediction (classification)scaffoldclintoxInChImolt5avg0.874 ± 0.0390.667 ± 0.0390.483 ± 0.000
property prediction (classification)scaffoldsiderSMILESt5llavg0.563 ± 0.0290.606 ± 0.0060.426 ± 0.026
property prediction (classification)scaffoldsidergraphgatavg0.557 ± 0.0070.605 ± 0.0030.442 ± 0.014
property prediction (classification)scaffoldsidergraphgatmax0.551 ± 0.0210.598 ± 0.0140.455 ± 0.010
property prediction (classification)scaffoldsiderInChIt5avg0.550 ± 0.0160.601 ± 0.0140.459 ± 0.011
property prediction (classification)scaffoldsidergraphgcnmax0.545 ± 0.0260.607 ± 0.0160.443 ± 0.003
property prediction (classification)scaffoldtox21graphginavg0.693 ± 0.0190.248 ± 0.0370.473 ± 0.000
property prediction (classification)scaffoldtox21graphgcnavg0.679 ± 0.0070.231 ± 0.0070.473 ± 0.000
property prediction (classification)scaffoldtox21SMILESmolt5avg0.664 ± 0.0310.200 ± 0.0580.488 ± 0.021
property prediction (classification)scaffoldtox21graphgatavg0.664 ± 0.0140.212 ± 0.0030.473 ± 0.000
property prediction (classification)scaffoldtox21SMILESt5llavg0.657 ± 0.0430.212 ± 0.0500.486 ± 0.018
property prediction (classification)scaffoldtoxcastgraphgatavg0.606 ± 0.0050.305 ± 0.0020.449 ± 0.003
property prediction (classification)scaffoldtoxcastgraphgcnavg0.604 ± 0.0060.306 ± 0.0020.452 ± 0.003
property prediction (classification)scaffoldtoxcastgraphginavg0.599 ± 0.0080.294 ± 0.0160.446 ± 0.001
property prediction (classification)scaffoldtoxcastSMILESt5llavg0.582 ± 0.0080.302 ± 0.0060.451 ± 0.007
property prediction (classification)scaffoldtoxcastSELFIESt5llavg0.579 ± 0.0020.293 ± 0.0020.453 ± 0.002
property prediction (classification)randombacegraphgcnavg0.824 ± 0.0340.752 ± 0.0230.683 ± 0.142
property prediction (classification)randombacegraphgatavg0.782 ± 0.0220.715 ± 0.0370.398 ± 0.039
property prediction (classification)randombacegraphginavg0.751 ± 0.0340.684 ± 0.0610.399 ± 0.042
property prediction (classification)randombaceSMILESt5avg0.712 ± 0.0430.609 ± 0.0560.429 ± 0.089
property prediction (classification)randombaceInChIt5llavg0.710 ± 0.0360.656 ± 0.0350.459 ± 0.065
property prediction (classification)randombbbpgraphgcnavg0.872 ± 0.0140.935 ± 0.0170.769 ± 0.031
property prediction (classification)randombbbpSMILESmolt5avg0.849 ± 0.0620.928 ± 0.0360.757 ± 0.071
property prediction (classification)randombbbpgraphgatavg0.848 ± 0.0200.932 ± 0.0130.768 ± 0.012
property prediction (classification)randombbbpSMILESt5llavg0.848 ± 0.0150.940 ± 0.0020.742 ± 0.016
property prediction (classification)randombbbpSMILESt5avg0.847 ± 0.0270.932 ± 0.0170.756 ± 0.003
property prediction (classification)randomclintoxSMILESt5avg0.861 ± 0.0410.654 ± 0.0290.485 ± 0.000
property prediction (classification)randomclintoxSMILESmolt5avg0.852 ± 0.0230.643 ± 0.0260.478 ± 0.004
property prediction (classification)randomclintoxSELFIESt5avg0.850 ± 0.0040.611 ± 0.0270.484 ± 0.003
property prediction (classification)randomclintoxSELFIESt5llavg0.845 ± 0.0630.652 ± 0.0880.482 ± 0.009
property prediction (classification)randomclintoxSELFIESt5llavg0.839 ± 0.0220.607 ± 0.0090.483 ± 0.005
property prediction (classification)randomsiderSMILESt5avg0.581 ± 0.0310.629 ± 0.0150.475 ± 0.017
property prediction (classification)randomsidergraphgatmax0.580 ± 0.0180.600 ± 0.0020.439 ± 0.013
property prediction (classification)randomsiderInChIt5avg0.568 ± 0.0380.618 ± 0.0170.461 ± 0.015
property prediction (classification)randomsiderSMILESmolt5avg0.562 ± 0.0080.600 ± 0.0080.461 ± 0.012
property prediction (classification)randomsidergraphginmax0.561 ± 0.0180.618 ± 0.0110.451 ± 0.008
property prediction (classification)randomtox21graphgcnavg0.795 ± 0.0230.283 ± 0.0210.481 ± 0.001
property prediction (classification)randomtox21graphginavg0.752 ± 0.0250.224 ± 0.0250.480 ± 0.001
property prediction (classification)randomtox21SMILESt5llavg0.745 ± 0.0080.246 ± 0.0410.494 ± 0.021
property prediction (classification)randomtox21SMILESmolt5avg0.738 ± 0.0110.268 ± 0.0110.492 ± 0.013
property prediction (classification)randomtox21graphginmax0.733 ± 0.0160.214 ± 0.0100.480 ± 0.001
property prediction (classification)randomtoxcastSMILESmolt5avg0.673 ± 0.0030.336 ± 0.0170.465 ± 0.007
property prediction (classification)randomtoxcastInChIt5avg0.673 ± 0.0180.322 ± 0.0270.471 ± 0.001
property prediction (classification)randomtoxcastSELFIESmolt5avg0.663 ± 0.0140.319 ± 0.0140.469 ± 0.004
property prediction (classification)randomtoxcastgraphgcnavg0.662 ± 0.0330.325 ± 0.0300.464 ± 0.007
property prediction (classification)randomtoxcastSMILESt5llavg0.657 ± 0.0380.318 ± 0.0290.464 ± 0.009

Table 8. Experimental results for classification tasks

MethodModelSMILESgraph2dBBBPTox21ToxCastSIDERClinToxBACEAvg
—SciBERT100.5530.5080.5240.5120.4450.4380.497
—GIN010.6400.7010.5930.5040.5040.6480.598
AddSciBERT, GIN110.6510.6960.6030.5810.5030.5590.599
Weight_addSciBERT, GIN110.6330.7170.6020.5220.5270.8020.634
ConcatSciBERT, GIN110.6420.7130.6030.5820.6270.6720.640
Concat_attentionSciBERT, GIN110.6200.5820.5570.4890.6760.6300.592
—MolT5100.6380.7360.5560.5260.9090.6860.675
AddMolT5, GIN110.6730.7240.6140.5590.8440.5410.659
Weight_addMolT5, GIN110.6340.7160.5770.5560.8860.7510.687
ConcatMolT5, GIN110.6430.7020.5780.5700.7310.6000.637
Concat_attentionMolT5, GIN110.6410.6800.5550.5200.8340.5360.628
—BioT5100.6810.7020.5980.5570.8900.6990.688
AddBioT5, GIN110.6930.7170.6030.5780.8700.7480.702
Weight_addBioT5, GIN110.7210.7130.5890.5830.8750.7010.697
ConcatBioT5, GIN110.7060.7050.5990.5570.8140.7200.683
Concat_attentionBioT5, GIN110.6440.6870.5980.5660.9030.5520.658
—KV-PLM100.7200.7000.5500.5980.8920.7850.708
Contrastive learningMoMu-SciBERT010.7050.7560.6340.6050.7990.7670.711
Contrastive learningMoMu-KV-PLM010.7010.7560.6300.6040.7740.7710.706
Contrastive learningMoleculeSTM100.7080.7570.6520.6370.8660.8200.740
Contrastive learningMoleculeSTM010.7000.7690.6510.6100.9250.8080.744
Contrastive learning, cross attentionMolFM010.7220.7660.6420.6320.7860.8260.729
Contrastive learning, cross attentionMolFM110.7290.7720.6440.6420.7970.8390.737
Contrastive learning, cross attentionMolCA100.7080.7600.5620.6110.8900.7930.721
Contrastive learning, cross attentionMolCA110.7000.7720.6450.6300.8950.7980.740
Contrastive learning, cross attentionGIT-Mol100.7190.7390.6210.6010.8350.6840.700
Contrastive learning, cross attentionGIT-Mol010.7110.7540.6530.5820.7890.6580.691
Contrastive learning, cross attentionGIT-Mol110.7390.7590.6680.6340.8830.8110.749

Table 10. Multi-modal fusion performance for molecular property predictions. This table displays the ROC_AUC results for various molecular property prediction classification tasks. “Add” denotes vector addition, “Weight_add” adaptive weighted addition, “Concat” concatenated encoding with pooling, and “Concat_attention” concatenated encoding with attention before pooling. The “SMILES” and “graph2d” columns indicate input modalities, with “1” for inclusion and “0” for exclusion. The columns BBBP, Tox21, ToxCast, SIDER, ClinTox, and BACE represent datasets.

BenchmarkMethodModelSMILESgraph2dBLEU-2BLEU-4ROUGE-1ROUGE-2ROUGE-LMeteor
ChEBI-20—RNN100.2510.1760.4500.2780.3940.363
ChEBI-20—Transformer100.0610.0270.2040.0870.1860.114
ChEBI-20—MolT5100.5400.4570.6340.4850.5780.569
ChEBI-20CLMoMu010.5490.4620.0000.0000.5750.576
ChEBI-20CLInstructMol010.4660.3650.5470.3650.4790.491
ChEBI-20CLInstructMol110.4750.3710.5660.3940.5020.509
ChEBI-20CL+CAMolFM110.5850.4980.6530.5080.5940.607
ChEBI-20CL+CAMolCA110.6200.5310.6810.5370.6180.651
PubChem324k—MolT5100.3010.2090.4030.2510.3380.356
PubChem324kCLMoMu010.3020.2150.4050.2510.3440.342
PubChem324kCL+CAMolCA100.3460.2690.4630.3230.4150.411
PubChem324kCL+CAMolCA010.3450.2620.4640.3160.4120.409
PubChem324kCL+CAMolCA110.3870.3030.5020.3590.4450.456
PubChem324kGIT-Mol100.2640.1760.4770.3740.4510.430
PubChem324kGIT-Mol010.2900.2100.5400.4450.5120.491
PubChem324kGIT-Mol110.3520.2630.5750.4850.5600.533

Table 11. Multi-modal fusion performance for molecule captioning tasks. This table presents the performance of six textual similarity metrics (BLEU-2, BLEU-4, ROUGE-1, ROUGE-2, ROUGE-L, and Meteor) across two common datasets, ChEBI-20 and PubChem324k, for molecule captioning tasks. The “Method" column includes “CL" for contrastive learning and “CA" for cross attention. The “SMILES" and “graph2d" columns indicate input modalities, with "1" for inclusion and "0" for exclusion.

Comparative Analysis of Interpretation Approaches

As indicated in Supplementary Table 12, we review commonly used interpretability methods in the domain of large molecular models. Most research utilizes approaches based on sample embedding visualization or inductive methods from case studies of text generation. However, embedding visualization primarily reflects changes in molecular representations, explaining how models understand and align data, but it does not necessarily offer interpretability to humans. Methods based on case studies require extensive time to extract insights, which is time-consuming and lacks statistical validation. Token frequency analysis is another common approach, yet selecting the appropriate tokens poses a significant challenge. Conventional methods like the Standard Deviation Method and the Quartile Method often result in the selection of ‘general high-frequency token mappings.’ In contrast, our Localized Feature Filtering Method is capable of identifying ‘particular high-frequency token mappings,’ providing more targeted and meaningful interpretative insights.

TaskApproachModeResearch
RepresentationEmbedding VisualizationSamples InductionSMPT [90], GIT-Mol [35], MV-Mol [91], KV-PLM [39], MoMu [33], Polymer_Walk [92]
RepresentationRule-based AnalysisInferencePolymer_Walk [92]
Text GenerationCase Study AnalysisCase Study InductionChatMol [77], GIT-Mol, Mol-ReGPT [29], MolFM [34], KV-PLM, Atomas [93]
Text GenerationAtom Attention VisualizationCase Study InductionMolFM
Molecule GenerationSubstructure Attention VisualizationCase Study InductionMOLGEN [94]
Text/Molecule GenerationTokens Frequency AnalysisStandard Deviation Method–
Text/Molecule GenerationTokens Frequency AnalysisQuartile Method–
Text/Molecule GenerationTokens Frequency AnalysisLocalized Feature Filtering MethodOurs

Table 12. Comparative analysis of interpretation approaches

References

  1. [1]A. Creswell, T. White, V. Dumoulin, K. Arulkumaran, B. Sengupta, and A. A. Bharath, “Generative adversarial networks: An overview,” IEEE signal processing magazine, vol. 35, no. 1, pp. 53–65, 2018.
  2. [2]M. Xu, L. Yu, Y. Song, C. Shi, S. Ermon, and J. Tang, “Geodiff: A geometric diffusion model for molecular conformation generation,” in International Conference on Learning Representations, 2022.arxiv.org/abs/2203.02923
  3. [3]K. Heikamp and J. Bajorath, “Support vector machines for drug discovery,” Expert opinion on drug discovery, vol. 9, no. 1, pp. 93–104, 2014.
  4. [4]K. Xu, W. Hu, J. Leskovec, and S. Jegelka, “How powerful are graph neural networks?,” in International Conference on Learning Representations, 2019.
  5. [6]P. Veličković, G. Cucurull, A. Casanova, A. Romero, P. Liò, and Y. Bengio, “Graph attention networks,” in International Conference on Learning Representations, 2018.arxiv.org/abs/1710.10903
  6. [7]A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” Advances in neural information processing systems, vol. 30, 2017.arxiv.org/abs/1706.03762
  7. [8]J. Schulman, B. Zoph, C. Kim, J. Hilton, J. Menick, J. Weng, F. C. Uribe, L. Fedus, L. Metz, M. Pokorny, et al., “Chatgpt: Optimizing language models for dialogue,” OpenAI blog, 2022.
  8. [9]OpenAI, “GPT-4 technical report,” CoRR, vol. abs/2303.08774, 2023.arxiv.org/abs/2303.08774
  9. [10]Y. Du, T. Fu, J. Sun, and S. Liu, “Molgensurvey: A systematic survey in machine learning models for molecule design,” arXiv preprint arXiv:2203.14500, 2022.arxiv.org/abs/2203.14500
  10. [11]Y. Fang, Z. Chen, X. Fan, N. Zhang, and H. Chen, “Knowledge-informed molecular learning: A survey on paradigm transfer,” in International Conference on Knowledge Science, Engineering and Management, pp. 86–98, Springer, 2024.
  11. [12]T. Guo, k. Guo, B. Nan, Z. Liang, Z. Guo, N. Chawla, O. Wiest, and X. Zhang, “What can large language models do in chemistry? a comprehensive benchmark on eight tasks,” in Advances in Neural Information Processing Systems, vol. 36, pp. 59662–59688, Curran Associates, Inc., 2023.
  12. [13]Y. Fang, X. Liang, N. Zhang, K. Liu, R. Huang, Z. Chen, X. Fan, and H. Chen, “Mol-instructions: A large-scale biomolecular instruction dataset for large language models,” in The Twelfth International Conference on Learning Representations, 2024.arxiv.org/abs/2306.08018
  13. [14]D. Weininger, “Smiles, a chemical language and information system. 1. introduction to methodology and encoding rules,” Journal of chemical information and computer sciences, vol. 28, no. 1, pp. 31–36, 1988.DOI
  14. [15]S. Heller, A. McNaught, S. Stein, D. Tchekhovskoi, and I. Pletnev, “Inchi-the worldwide chemical structure identifier standard,” Journal of cheminformatics, vol. 5, no. 1, pp. 1–9, 2013.DOI
  15. [16]M. Krenn, F. Häse, A. Nigam, P. Friederich, and A. Aspuru-Guzik, “Self-referencing embedded strings (selfies): A 100% robust molecular string representation,” Machine Learning: Science and Technology, vol. 1, no. 4, p. 045024, 2020.DOI
  16. [17]G. Landrum et al., “Rdkit: A software suite for cheminformatics, computational chemistry, and predictive modeling,” Greg Landrum, vol. 8, no. 31.10, p. 5281, 2013.
  17. [18]G. L. Long and J. D. Winefordner, “Limit of detection. a closer look at the iupac definition,” Analytical chemistry, vol. 55, no. 7, pp. 712A–724A, 1983.DOI
  18. [19]Z. Wu, B. Ramsundar, E. N. Feinberg, J. Gomes, C. Geniesse, A. S. Pappu, K. Leswing, and V. Pande, “Molecu-lenet: a benchmark for molecular machine learning,” Chemical science, vol. 9, no. 2, pp. 513–530, 2018.
  19. [20]R. Luo, L. Sun, Y. Xia, T. Qin, S. Zhang, H. Poon, and T.-Y. Liu, “Biogpt: generative pre-trained transformer for biomedical text generation and mining,” Briefings in bioinformatics, vol. 23, no. 6, p. bbac409, 2022.DOI
  20. [21]C. Edwards, T. Lai, K. Ros, G. Honke, K. Cho, and H. Ji, “Translation between molecules and natural language,” in Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 375–413, 2022.
  21. [22]Z. Liu, S. Li, Y. Luo, H. Fei, Y. Cao, K. Kawaguchi, X. Wang, and T.-S. Chua, “Molca: Molecular graph-language modeling with cross-modal projector and uni-modal adapter,” in Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 15623–15638, 2023.arxiv.org/abs/2310.12798
  22. [23]A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, et al., “Language models are unsupervised multitask learners,” OpenAI blog, 2019.
  23. [24]J. D. M.-W. C. Kenton and L. K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” in Proceedings of NAACL-HLT, pp. 4171–4186, 2019.
  24. [25]Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov, “Roberta: A robustly optimized bert pretraining approach,” arXiv preprint arXiv:1907.11692, 2019.arxiv.org/abs/1907.11692
  25. [26]I. Beltagy, K. Lo, and A. Cohan, “Scibert: A pretrained language model for scientific text,” in Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 3615–3620, 2019.arxiv.org/abs/1903.10676
  26. [27]S. Chithrananda, G. Grand, and B. Ramsundar, “Chemberta: large-scale self-supervised pretraining for molecular property prediction,” arXiv preprint arXiv:2010.09885, 2020.arxiv.org/abs/2010.09885
  27. [28]Z. Liu, W. Zhang, Y. Xia, L. Wu, S. Xie, T. Qin, M. Zhang, and T.-Y. Liu, “Molxpt: Wrapping molecules with text for generative pre-training,” in Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 1606–1616, 2023.arxiv.org/abs/2305.10688
  28. [29]J. Li, Y. Liu, W. Fan, X.-Y. Wei, H. Liu, J. Tang, and Q. Li, “Empowering molecule discovery for molecule-caption translation with large language models: A chatgpt perspective,” IEEE Transactions on Knowledge and Data Engineering, 2024.DOI
  29. [30]C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu, “Exploring the limits of transfer learning with a unified text-to-text transformer,” The Journal of Machine Learning Research, vol. 21, no. 1, pp. 5485–5551, 2020.arxiv.org/abs/1910.10683
  30. [31]M. Lewis, Y. Liu, N. Goyal, M. Ghazvininejad, A. Mohamed, O. Levy, V. Stoyanov, and L. Zettlemoyer, “Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension,” in Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7871–7880, 2020.
  31. [32]S. Kim, J. Chen, T. Cheng, A. Gindulyte, J. He, S. He, Q. Li, B. A. Shoemaker, P. A. Thiessen, B. Yu, et al., “Pubchem 2023 update,” Nucleic acids research, vol. 51, no. D1, pp. D1373–D1380, 2023.DOI
  32. [33]B. Su, D. Du, Z. Yang, Y. Zhou, J. Li, A. Rao, H. Sun, Z. Lu, and J.-R. Wen, “A molecular multimodal foundation model associating molecule graphs with natural language,” arXiv preprint arXiv:2210.05481, 2022.
  33. [34]Y. Luo, K. Yang, M. Hong, X. Liu, and Z. Nie, “Molfm: A multimodal molecular foundation model,” arXiv preprint arXiv:2307.09484, 2023.arxiv.org/abs/2307.09484
  34. [35]P. Liu, Y. Ren, J. Tao, and Z. Ren, “Git-mol: A multi-modal large language model for molecular science with graph, image, and text,” Computers in Biology and Medicine, vol. 171, p. 108073, 2024.DOI
  35. [36]J. Li, D. Li, S. Savarese, and S. Hoi, “Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models,” in International conference on machine learning, pp. 19730–19742, PMLR, 2023.arxiv.org/abs/2301.12597
  36. [37]L. Krasnov, I. Khokhlov, M. Fedorov, and S. Sosnin, “Struct2iupac–transformer-based artificial neural network for the conversion between chemical notations,” Scientific Reports, 2021.DOI
  37. [38]J. Mao, J. Wang, A. Zeb, K.-H. Cho, H. Jin, J. Kim, O. Lee, Y. Wang, and K. T. No, “Transformer-based molecular generative model for antiviral drug design,” Journal of Chemical Information and Modeling, 2023.DOI
  38. [39]Z. Zeng, Y. Yao, Z. Liu, and M. Sun, “A deep-learning system bridging molecule structure and biomedical text with comprehensive comparable to human professionals,” Nature communications, vol. 13, no. 1, p. 862, 2022.DOI
  39. [40]C. Edwards, C. Zhai, and H. Ji, “Text2mol: Cross-modal molecule retrieval with natural language queries,” in Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 595–607, 2021.
  40. [41]K. Rajan, A. Zielesny, and C. Steinbeck, “Decimer 1.0: deep learning for chemical image recognition using transformers,” Journal of Cheminformatics, vol. 13, no. 1, pp. 1–16, 2021.DOI
  41. [42]I. Khokhlov, L. Krasnov, M. V. Fedorov, and S. Sosnin, “Image2smiles: Transformer-based molecular optical recognition engine,” Chemistry-Methods, vol. 2, no. 1, p. e202100069, 2022.DOI
  42. [43]J. Yi, C. Wu, X. Zhang, X. Xiao, Y. Qiu, W. Zhao, T. Hou, and D. Cao, “Micer: a pre-trained encoder-decoder architecture for molecular image captioning,” Bioinformatics (Oxford, England), vol. 38, no. 19, pp. 4562–4572, 2022.DOI
  43. [44]Z. Xu, J. Li, Z. Yang, S. Li, and H. Li, “Swinocsr: end-to-end optical chemical structure recognition using a swin transformer,” Journal of Cheminformatics, vol. 14, no. 1, pp. 1–13, 2022.DOI
  44. [45]Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo, “Swin transformer: Hierarchical vision transformer using shifted windows,” in Proceedings of the IEEE/CVF international conference on computer vision, pp. 10012–10022, 2021.arxiv.org/abs/2103.14030
  45. [46]Q. Pei, W. Zhang, J. Zhu, K. Wu, K. Gao, L. Wu, Y. Xia, and R. Yan, “Biot5: Enriching cross-modal integration in biology with chemical knowledge and natural language associations,” in Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 1102–1123, 2023.arxiv.org/abs/2310.07276
  46. [47]S. Liu, W. Nie, C. Wang, J. Lu, Z. Qiao, L. Liu, J. Tang, C. Xiao, and A. Anandkumar, “Multi-modal molecule structure–text model for text-based retrieval and editing,” Nature Machine Intelligence, vol. 5, no. 12, pp. 1447–1457, 2023.
  47. [48]Q. Pei, L. Wu, K. Gao, X. Liang, Y. Fang, J. Zhu, S. Xie, T. Qin, and R. Yan, “Biot5+: Towards generalized biological understanding with iupac integration and multi-task tuning,” arXiv preprint arXiv:2402.17810, 2024.arxiv.org/abs/2402.17810
  48. [49]Pengfei Liu, “Chebi-20-mm (revision 7d44959),” 2024.
  49. [50]P. Liu and Z. Ren, “Ai-hpc-research-team/slm4mol: 1.0,” Dec. 2024.DOI
  50. [51]J. J. Irwin, K. G. Tang, J. Young, C. Dandarchuluun, B. R. Wong, M. Khurelbaatar, Y. S. Moroz, J. Mayfield, and R. A. Sayle, “Zinc20—a free ultralarge-scale chemical database for ligand discovery,” Journal of chemical information and modeling, vol. 60, no. 12, pp. 6065–6073, 2020.DOI
  51. [52]B. Zdrazil, E. Felix, F. Hunter, E. J. Manners, J. Blackshaw, S. Corbett, M. de Veij, H. Ioannidis, D. Mendez Lopez, J. F. Mosquera, et al., “The chembl database in 2023: a drug discovery platform spanning multiple bioactivity data types and time periods,” Nucleic Acids Research, p. gkad1004, 2023.DOI
  52. [53]J. Hastings, G. Owen, A. Dekker, M. Ennis, N. Kale, V. Muthukrishnan, S. Turner, N. Swainston, P. Mendes, and C. Steinbeck, “Chebi in 2016: Improved services and an expanding collection of metabolites,” Nucleic acids research, vol. 44, no. D1, pp. D1214–D1219, 2016.DOI
  53. [54]D. S. Wishart, Y. D. Feunang, A. C. Guo, E. J. Lo, A. Marcu, J. R. Grant, T. Sajed, D. Johnson, C. Li, Z. Sayeeda, et al., “Drugbank 5.0: a major update to the drugbank database for 2018,” Nucleic acids research, vol. 46, no. D1, pp. D1074–D1082, 2018.DOI
  54. [55]D. Polykovskiy, A. Zhebrak, B. Sanchez-Lengeling, S. Golovanov, O. Tatanov, S. Belyaev, R. Kurbanov, A. Artamonov, V. Aladinskiy, M. Veselov, et al., “Molecular sets (moses): a benchmarking platform for molecular generation models,” Frontiers in pharmacology, vol. 11, p. 565644, 2020.
  55. [56]“Osra validation datasets.” https://sourceforge.net/p/osra/wiki/Validation/, 2020. Accessed: 2020-06-24.
  56. [57]“CLEF-IP 2012 Structure Recognition Test Set.” https://www.ifs.tuwien.ac.at/~clef-ip/download/2012/qrels/clef-ip-2012-chem-recognition-qrels.tgz, 2012. Accessed: 2020-06-29.
  57. [58]“Molrecou benchmark dataset.” https://www.cs.bham.ac.uk/research/groupings/reasoning/sdag/chemical.php, 2020. Accessed: 2020-06-29.
  58. [59]N. Schneider, N. Stiefl, and G. A. Landrum, “What’s what: The (nearly) definitive guide to reaction role assignment,” Journal of chemical information and modeling, vol. 56, no. 12, pp. 2336–2346, 2016.DOI
  59. [60]P. Seidl, A. Vall, S. Hochreiter, and G. Klambauer, “Enhancing activity prediction models in drug discovery with the ability to understand human language,” in Proceedings of the 40th International Conference on Machine Learning, pp. 30458–30490, 2023.arxiv.org/abs/2303.03363
  60. [61]J. Zhu, Y. Xia, L. Wu, S. Xie, W. Zhou, T. Qin, H. Li, and T.-Y. Liu, “Dual-view molecular pre-training,” in Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 3615–3627, 2023.
  61. [62]J. Ross, B. Belgodere, V. Chenthamarakshan, I. Padhi, Y. Mroueh, and P. Das, “Large-scale chemical language representations capture molecular structure and properties,” Nature Machine Intelligence, vol. 4, no. 12, pp. 1256–1264, 2022.
  62. [63]Z. Guo, P. Sharma, A. Martinez, L. Du, and R. Abraham, “Multilingual molecular representation learning via contrastive pre-training,” in Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 3441–3453, 2022.
  63. [64]J. Jiang, R. Zhang, Y. Yuan, T. Li, G. Li, Z. Zhao, and Z. Yu, “Noisemol: A noise-robusted data augmentation via perturbing noise for molecular property prediction,” Journal of Molecular Graphics and Modelling, vol. 121, p. 108454, 2023.DOI
  64. [65]J. Gao, Z. Shen, Y. Xie, J. Lu, Y. Lu, S. Chen, Q. Bian, Y. Guo, L. Shen, J. Wu, et al., “Transfoxmol: predicting molecular property with focused attention,” Briefings in Bioinformatics, p. bad306, 2023.DOI
  65. [66]C. Ying, T. Cai, S. Luo, S. Zheng, G. Ke, D. He, Y. Shen, and T.-Y. Liu, “Do transformers really perform badly for graph representation?,” Advances in Neural Information Processing Systems, vol. 34, pp. 28877–28888, 2021.arxiv.org/abs/2106.05234
  66. [67]A. Xie, Z. Zhang, J. Guan, and S. Zhou, “Self-supervised learning with chemistry-aware fragmentation for effective molecular property prediction,” Briefings in Bioinformatics, p. bbad296, 2023.DOI
  67. [68]V. Bagal, R. Aggarwal, P. Vinod, and U. D. Priyakumar, “Molgpt: molecular generation using a transformer-decoder model,” Journal of Chemical Information and Modeling, vol. 62, no. 9, pp. 2064–2076, 2021.DOI
  68. [69]R. Taylor, M. Kardas, G. Cucurull, T. Scialom, A. Hartshorn, E. Saravia, A. Poulton, V. Kerkez, and R. Stojnic, “Galactica: A large language model for science,” arXiv preprint arXiv:2211.09085, 2022.arxiv.org/abs/2211.09085
  69. [70]E. Mazuz, G. Shtar, B. Shapira, and L. Rokach, “Molecule generation using transformers and policy gradient reinforcement learning,” Scientific Reports, vol. 13, no. 1, p. 8799, 2023.DOI
  70. [71]C. Qian, H. Tang, Z. Yang, H. Liang, and Y. Liu, “Can large language models empower molecular property prediction?,” arXiv preprint arXiv:2307.07443, 2023.arxiv.org/abs/2307.07443
  71. [72]R. Irwin, S. Dimitriadis, J. He, and E. J. Bjerrum, “Chemformer: a pre-trained transformer for computational chemistry,” Machine Learning: Science and Technology, vol. 3, no. 1, p. 015022, 2022.DOI
  72. [73]Y. Fang, N. Zhang, Z. Chen, L. Guo, X. Fan, and H. Chen, “Domain-agnostic molecular generation with chemical feedback,” in The Twelfth International Conference on Learning Representations, 2024.arxiv.org/abs/2301.11259
  73. [74]D. Rothchild, A. Tamkin, J. Yu, U. Misra, and J. Gonzalez, “C5t5: Controllable generation of organic molecules with transformers,” arXiv preprint arXiv:2108.10307, 2021.arxiv.org/abs/2108.10307
  74. [75]J. Lu and Y. Zhang, “Unified deep learning model for multitask reaction predictions with explanation,” Journal of Chemical Information and Modeling, p. 1376–1387, Mar 2022.DOI
  75. [76]D. Christofidellis, G. Giannone, J. Born, O. Winther, T. Laino, and M. Manica, “Unifying molecular and textual representations via multi-task language modelling,” in International Conference on Machine Learning, pp. 6140–6157, PMLR, 2023.arxiv.org/abs/2301.12586
  76. [77]Z. Zeng, B. Yin, S. Wang, J. Liu, C. Yang, H. Yao, X. Sun, M. Sun, G. Xie, and Z. Liu, “Interactive molecular discovery with natural language,” arXiv preprint arXiv:2306.11976, 2023.arxiv.org/abs/2306.11976
  77. [78]M. Livne, Z. Miftahutdinov, E. Tutubalina, M. Kuznetsov, D. Polykovskiy, A. Brundyn, A. JhunJhunwala, A. Costa, A. Aliper, A. Aspuru-Guzik, et al., “nacho: Multimodal natural and chemical languages foundation model,” Chemical Science, vol. 15, no. 22, pp. 8380–8389, 2024.
  78. [79]J. He, F. Mattsson, M. Forsberg, E. J. Bjerrum, O. Engkvist, C. Tyrchan, W. Czechitzky, et al., “Transformer neural network for structure constrained molecular optimization,” Journal of Cheminformatics, vol. 14, no. 1, p. 18, 2022.DOI
  79. [80]A. M. Bran, S. Cox, O. Schilter, C. Baldassari, A. D. White, and P. Schwaller, “Augmenting large language models with chemistry tools,” Nature Machine Intelligence, pp. 1–11, 2024.
  80. [81]Y. Qian, J. Guo, Z. Tu, Z. Li, C. W. Coley, and R. Barzilay, “Molscribe: Robust molecular structure recognition with image-to-graph generation,” Journal of Chemical Information and Modeling, vol. 63, no. 7, pp. 1925–1934, 2023.DOI
  81. [82]X. Tang, A. Tran, J. Tan, and M. Gerstein, “Mollm: Integrating 3d and 2d molecular representations with biomedical text via a unified pre-trained language model,” bioRxiv, pp. 2023–11, 2023.
  82. [83]C. Cortes and V. Vapnik, “Support-vector networks,” Machine learning, vol. 20, pp. 273–297, 1995.
  83. [84]L. Breiman, “Random forests,” Machine learning, vol. 45, pp. 5–32, 2001.
  84. [85]Y. You, T. Chen, Y. Sui, T. Chen, Z. Wang, and Y. Shen, “Graph contrastive learning with augmentations,” Advances in neural information processing systems, vol. 33, pp. 5812–5823, 2020.
  85. [86]Y. Rong, Y. Bian, T. Xu, W. Xie, Y. Wei, W. Huang, and J. Huang, “Self-supervised graph transformer on large-scale molecular data,” Advances in neural information processing systems, vol. 33, pp. 12559–12571, 2020.arxiv.org/abs/2007.02835
  86. [87]S. Liu, H. Wang, W. Liu, J. Lasenby, H. Guo, and J. Tang, “Pre-training molecular graph representation with 3d geometry,” in International Conference on Learning Representations, 2022.
  87. [88]G. Zhou, Z. Gao, Q. Ding, H. Zheng, H. Xu, Z. Wei, L. Zhang, and G. Ke, “Uni-mol: A universal 3d molecular representation learning framework,” in The Eleventh International Conference on Learning Representations, 2023.DOI
  88. [89]S. Honda, S. Shi, and H. R. Ueda, “Smiles transformer: Pre-trained molecular fingerprint for low data drug discovery,” arXiv preprint arXiv:1911.04738, 2019.arxiv.org/abs/1911.04738
  89. [90]Y. Li, W. Wang, J. Liu, and C. Wu, “Pre-training molecular representation model with spatial geometry for property prediction,” Computational Biology and Chemistry, vol. 109, p. 108023, 2024.DOI
  90. [91]Y. Luo, K. Yang, M. Hong, X. Y. Liu, Z. Nie, H. Zhou, and Z. Nie, “Learning multi-view molecular representations with structured and unstructured knowledge,” arXiv preprint arXiv:2406.09841, 2024.arxiv.org/abs/2406.09841
  91. [92]M. Sun, M. Guo, W. Yuan, V. Thost, C. E. Owens, A. F. Grosz, S. Selvan, K. Zhou, H. Mohiuddin, B. J. Pedretti, et al., “Representing molecules as random walks over interpretable grammars,” in Forty-first International Conference on Machine Learning.arxiv.org/abs/2403.08147
  92. [93]Y. Zhang, G. Ye, C. Yuan, B. Han, L.-K. Huang, J. Yao, W. Liu, and Y. Rong, “Atomas: Hierarchical alignment on molecule-text for unified molecule understanding and generation,” arXiv preprint arXiv:2404.16880, 2024.arxiv.org/abs/2404.16880
  93. [94]Y. Fang, N. Zhang, Z. Chen, L. Guo, X. Fan, and H. Chen, “Domain-agnostic molecular generation with chemical feedback,” in The Twelfth International Conference on Learning Representations, 2024.arxiv.org/abs/2301.11259

Paper details

vlnc:W_c5rmus4g
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 4

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents