WorksComputer Science
ProtChatGPT: Towards Understanding Proteins with Large Language Models
Abstract
Protein research is crucial in various fundamental disciplines, but understanding their intricate structure-function relationships remains challenging. Recent Large Language Models (LLMs) have made significant strides in comprehending task-specific knowledge, suggesting the potential for ChatGPT-like systems specialized in protein to facilitate basic research. In this work, we introduce ProtChatGPT, which aims at learning and understanding protein structures via natural languages. ProtChatGPT enables users to upload proteins, ask questions, and engage in interactive conversations to produce comprehensive answers. The system comprises protein encoders, a Protein-Language Pertaining Transformer (PLP-former), a projection adapter, and an LLM. The protein first undergoes protein encoders and PLP-former to produce protein embeddings, which are then projected by the adapter to conform with the LLM. The LLM finally combines user questions with projected embeddings to generate informative answers. Experiments show that ProtChatGPT can produce promising responses to proteins and their corresponding questions. We hope that ProtChatGPT could form the basis for further exploration and application in protein research. Code and our pre-trained model will be publicly available.
CCS Concepts • Information systems → Personalization.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Conference’17, July 2017, Washington, DC, USA
© 2025 Copyright held by the owner/author(s). Publication rights licensed to ACM. ACM ISBN 978-x-xxxx-xxxx-x/YY/MM https://doi.org/10.1145/nnnnnnn.nnnnnnn
Introduction
Proteins, as essential molecular entities for life, hold paramount significance in biological processes. The comprehensive understanding of protein structure and function is important for advancing research in the realms of biology and biomedicine. However, traditional protein research normally involves labor-intensive laboratory experiments and extensive literature reviews, which could be time-consuming and require specialized expertise in protein.
Recently, Large Language Models (LLMs), e.g., ChatGPT [47], have prevailed in Natural Language Processing (NLP) [8, 48, 53]. With superior language understanding and logical reasoning capabilities, these models can perform various intricate linguistic tasks such as question and answering (Q&A) [25, 71]. Considering that primary sequences can be regarded as protein’s own “natural language”, this intuitively motivates us to ride on LLMs’ coattails and customize them into protein research based on large-scale biological corpora, e.g., RCSB-PDB [14].
Empirically, with the capabilities of LLMs specialized in protein, researchers can potentially achieve (1) Protein Understanding and Analysis by simplifying the retrieval of crucial information (e.g., structures, functions, interactions, mutations, and disease associations) about specific proteins for research; (2) Customized Protein Design by characterizing the unique protein structures of patients to discover targeted drugs and further verify expected functions for healthcare.
In this paper, we propose an AI-based protein chat system, named ProtChatGPT, to implement ChatGPT-like functionalities for the protein research field. ProtChatGPT works in a similar way to natural language conversation systems. Users are allowed to upload protein 1D sequences or 3D structures (e.g., fasta or pdb files) and pose diverse related questions. ProtChatGPT produces comprehensive responses in an interactive manner based on the questions. In this way, researchers can intuitively acquire valuable insights and interactively delve into the complexities of diverse proteins. Specifically, ProtChatGPT consists of three stages: multi-level protein encoding, protein-language alignment, and instruction tuning of LLMs, as shown in Figure 1. First, we employ three pre-trained protein encoders to embed the primary sequence (i.e., by ESM-1b [51]), secondary structure (i.e., by NetSurfP [22]), and tertiary structure (i.e., by ESM-IF1 [15]), respectively. Next, to align the protein sequence and language modalities, we propose the PLP-former. PLP-former extracts features from the output of the protein encoder, and learns the protein representations that are most relevant to the text description. We also deploy a Protein Context Gating (PCG) module trained with contrastive loss further to align the sequence with other levels of protein embeddings. Third, we use an adapter as an information transmitter to convert protein embeddings into protein prompts that can be interpreted by the LLM. Finally, the LLM combines user questions (i.e., question prompts) with the transmitted protein prompts to produce corresponding answers. We conduct experiments on protein understanding and design. Experimental results show the effectiveness of the proposed method. In summary, our contributions are as follows:

Figure 1. Overview of the ProtChatGPT framework. Our pipeline consists of three stages: (1) multi-level protein encoding, (2) multi-level protein-language alignment, and (3) instruction tuning with an external web corpus and protein features. First, we utilize three pre-trained frozen large protein encoders to acquire high-quality multi-level embeddings. In the second stage, we first enforce the PLP Transformer, a lightweight transformer with learnable query tokens, to learn the protein representation most relevant to the text description. The PLP-former takes the sequence embedding , tokens, and protein descriptions as inputs and outputs the learned tokens as the selected embedding. The selected embedding is first aligned with the secondary structure embedding through protein context gating for aligned embedding , and then we adopt contrastive learning to enforce the tertiary structure embedding to further align with the joint representation . In the third stage, we perform protein-to-text generative learning by connecting the aligned and to an LLM decoder. Combined with the instruction pairs extracted from open protein-related literature, an adapter is further trained as the information bottleneck between protein embeddings and the LLM, such that its output can be interpreted by the language model. Finally, the LLM can produce descriptive answers given the question prompt and the multi-level protein prompt from the adapter.
We propose ProtChatGPT, an interactive ChatGPT-like system that engages Q&A for protein-related research, which significantly facilitates protein understanding and design.
We introduce a progressive protein-language learning strategy for dynamically aligning protein from multi-level structures with natural language, including a transformer-based PLP-former and a Protein Context Gating (PCG) module based on contrastive learning.
As a minor contribution, a protein-specific instruction tuning dataset is built to facilitate the LLMs to better correspond to the user questions.
We demonstrate ProtChatGPT’s versatility and range of applications by deploying it to tasks of a rather distinct nature, including protein understanding and design.
Related Work
Protein Representation Learning. Proteins are workhorses of the cell, which contain four distinct levels of structures carrying their fundamental functions. Previous protein representation works seek to learn protein representations based on different levels of structures. Considering protein sequences as language, several works [37, 42] encode amino acid tokens using transformer model [56] to extract pairwise relationships among amino acids, and autoregressively recover protein sequences based on extensive protein sequence databases. Alternatively, other sequence modeling methods [32, 39, 49, 51, 58] resort to using Masked Language Modeling (MLM) to develop attention patterns that correspond to the residue-residue contact map of the protein. Compared with sequence-based methods, structure-based methods [12, 43, 70] directly encode geometric information of proteins for topology-sensitive tasks such as molecule binding [18, 24], protein interface analysis [38, 50], and protein properties prediction [66]. In this paper, we aim to directly leverage these pre-trained Large Protein Models (LPMs) to acquire high-quality embeddings without fine-tuning their network parameters.
Large Language Models (LLMs). Recently, Natural Language Processing (NLP) has witnessed significant advances due to the development of LLMs [11] [47] trained on an extensive, diverse dataset. Consequently, many multi-modal variants [1, 19, 52, 54, 59] based on LLMs have gained significant attention for the understanding of information in other modalities beyond text. For example, BLIP-2[25] designs a Q-Former to align the visual features from the frozen visual encoder with large language models. FROMAGe [23] freezes the LLM and visual encoders, and fine-tunes linear mapping layers to achieve cross-modality interactions. For protein understanding, MedVQA [54] employs a Multi-Layer Perceptron (MLP) network that maps the extracted visual features from a frozen vision encoder to a set of learnable tokens, which develops an open-ended VQA for diagnoses and treatment decisions. Galactica [52] explicitly models the protein sequences and SMILES with scientific literature, and enables the model to explain the properties of the sequences. ProteinChat [14] further models the protein structure with its corresponding descriptions in Protein Data Bank (PDB) for the protein Q&A task. However, this method only considers the impact of protein structure on its function, while neglecting the complementarity between different levels of structure, such as amino acid sequences (primary structure) and geometric coordinates (tertiary structure). In this paper, we aim to adapt pre-trained general LLMs (e.g., Vicuna [7] and LLAMA3) for protein-specific ChatGPT-like tasks, which aligns multi-level protein features from LPMs with LLMs.
Proposed Method
While demonstrating excellent performance in natural language tasks, LLM cannot directly facilitate protein question-answering tasks due to the modality gap between protein structures and biomedical descriptions. As shown in Figure 1, in order to bridge this gap, we introduce a progressive protein-language learning framework with three stages: (1) multi-level protein encoding, (2) multi-level protein-language alignment, and (3) instruction tuning with protein features.
Multi-Level Protein Encoding
In this paper, we leverage three pre-trained Large Protein Models (LPMs) exclusively trained on different levels of protein structures, working synergistically to provide high-quality hybrid embeddings without fine-tuning their network parameters.
First, we use a pre-trained sequence encoder to extract protein sequence features. Specifically, given a protein sequence with amino acids, the encoder produces the corresponding sequence embedding , where is the number of embedding channels. In our implementation, we use ESM-1b [51] as the sequence encoder, where . Although ESM-1b is able to implicitly capture structural contact information, incorporating detailed structure (secondary and tertiary) information explicitly can be an effective way to model spatial interactions between residues. Therefore, we propose to enhance the ESM-1b with a supplementary protein secondary structure encoder NetSurfP [22]. Specifically, this encoder consists of two parallel convolution layers and an identity layer, whose outputs are concatenated in the feature dimension and fed to a two-layer bidirectional LSTM containing 1024 hidden units. The output is then projected to an 8-dimensional feature vector at each position as the secondary structure embedding . Moreover, we further introduce the ESM-IF1 [15] encoder for protein tertiary structure representation. Specifically, we select the feature from an intermediate layer as the tertiary structure embedding where the number of embedding channels , indicating the geometric protein knowledge.
By acquiring such multi-level hybrid property information, the quality of protein embeddings can be further improved, considering that the protein properties studied in ESM-1b and ESM-IF1 can correlate with each other. Note that all the employed encoders are frozen for efficient training.
Multi-Level Protein-Language Alignment
Protein-Language Pretraining (PLP).
As mentioned before, the key challenge of transferring LLMs to protein research lies in the modality gap between protein structures and biomedical texts. Despite the strong language generation and zero-shot transfer abilities of LLMs, directly retraining them end-to-end for protein specialization appears to be impractical due to the massive number of parameters and data requirements. Another alternative is fine-tuning the pre-trained parameters, but this often leads to catastrophic forgetting. Considering this trade-off, we propose a Protein-Language Pretraining Transformer (PLP-former) for efficient cross-modal alignment between protein and text, while remaining LLMs frozen during the training. Following existing Vision-Language Pretraining (VLP) works [9, 25, 71], we introduce the PLP-former to extract protein-related features from a frozen protein sequence encoder. More related works on VLP are also given in Appendix § 11.
As shown in Figure 2, PLP-former consists of three inputs: initial sequence embedding from ESM-1b, a set of learnable tokens , and the corresponding description. The learnable tokens first perform mutual interactions via self-attention layers in the protein transformer. Specifically, the tokens first acquire queries , keys , and values through three linear transformation matrices , , and . It can be formulated as:
Figure 2. Illustration of the PLP-former and protein-language representation learning. PLP-former consists of two transformer submodules with shared self-attention: (1) a text transformer that performs encoding and decoding of protein descriptions, and (2) a protein transformer that interacts with the frozen ESM-1b for sequence feature extraction. PLP-former is trained by jointly optimizing three pretraining objectives (dashed boxes) on sequence-description pairs.
Next, the attention map is computed through softmax where represents the dimensionality of the keys, softmax is the softmax activation function. The refined tokens after self-attention can be calculated as where is the projection matrix for output. Given the sequence features from ESM-1b as , tokens then interact with language embedding through cross-attention layers, which can be formulated as:
where represents the refined tokens after cross-attention, , , , and are a new set of learning transformation matrices. Additionally, benefiting from the text transformer, tokens can further interact with the textual descriptions through the same self-attention layers. Finally, the PLP-former produces the output after a linear feed-forward layer [56]. The size of selected () is much smaller than the size of frozen sequence features () as the length of protein usually numbers over hundreds. Depending on the pretraining task, we implement distinct attention masking strategies [25] within the self-attention block to regulate the token-text interaction. The training details of PLP-former are given in Appendix § 12. In this way, PLP-former can effectively select the most useful information for the LLM while removing irrelevant protein information. This reduces the burden of the LLM to learn protein-language alignment while mitigating the catastrophic forgetting problem, as the PLP-former establishes a fundamental correlation between the primary structure and its descriptive text before introducing more complex structural data.
Multi-Level Protein Alignment
After multi-level encoding, the next step involves aligning proteins with these multi-level features to harness the full potential of their hybrid structural representations. In this work, we introduce a progressive alignment strategy across different levels.
Protein Context Gating (PCG). Initially, we utilize a Protein Context Gating (PCG) module for the feature aggregation between the sequence embedding and secondary structure embedding . Our goal is to transform into dynamic gating maps, which effectively allows the PCG module to highlight or suppress based on the relevant secondary structural information. The formulation of PCG can be written as:
where represents the element-wise multiplication, denotes the sigmoid activation function that normalizes the gating values to the range . A convolution layer is first deployed on to project the dimension to . is a transformation matrix, is the bias to acquire the aligned embedding .
Contrastive Learning. Subsequently, we employ a contrastive loss to achieve alignment with the tertiary structure , which can be formulated as:
where is the positive sample corresponding to , represents the dot product, represents a temperature hyper-parameter modulating the learning sensitivity. We also added a convolutional layer on to match the dimensions with . By introducing the tertiary structural information, this contrastive mechanism allows for dynamic protein feature alignment and optimizes the representation for tasks requiring high discernment between similar protein sequences, such as homologous protein analysis.
Instruction Tuning with Protein Features
Projection Adapter
During the final instruction tuning stage, we further design a multi-level projection adapter to harvest the generative language capability of LLMs for protein-to-text generative learning. The adapter takes the pre-aligned embeddings and from ESM-IF1 as inputs, and acts as an information bottleneck to the LLM decoder, such that its output protein representation can be interpreted by the LLM. In practice, we use two individual Fully-Connected (FC) layers to linearly project the output protein embeddings into the same dimension as the question embedding of the LLM. They function as soft protein prompts that condition the LLM on protein representation from different levels.
Protein-Text Generation
Finally, the projected protein prompts are prepended to the question prompts (text embeddings of user questions) through concatenation. In implementation, we separately deploy two LLMs Vicuna-13B [7] and LLaMA3. We employ the LLM decoder to model the conditional generation probability in the language model, where is the generated tokens, represents the context vector of input tokens (i.e., protein embeddings along with user questions). To enhance the model training with protein-text pairs, we utilize a specialized token prompt as:
Q: < Protein >< ProteinPrompts >< /Protein >
< QuestionPrompts >
A: < Descriptions >
Here < ProteinPrompts > represents the soft prompts that symbolize the multi-level embeddings aligned after the projection adapter. < Protein > and < /Protein > respectively represent the start and end symbols for protein embeddings. < QuestionPrompts > represents the user questions that prompt the LLM to generate corresponding answers for the uploaded protein, such as “Describe the function of this protein”. To enable LLM to better understand user instructions and enhance the effectiveness and accuracy of protein analysis, we created a protein instruction tuning dataset in which each protein is associated with ten Q&A pairs (see § 4.1.2 for more details). < Descriptions > represents the generated answers, which have been substituted with publicly available protein descriptions during the second training stage. During the tuning stage, the generated tokens (< Descriptions >) are replaced with the descriptions from our instruction dataset. During the testing and inference stage, the generated tokens remain empty, expecting ProtChatGPT to generate descriptive answers for the given protein and corresponding questions.
Experiments
Dataset
Protein-Language Pretraining Dataset
In order to train our ProtChatGPT, dedicated protein-specific training datasets are indispensable for our proposed progressive training strategy. First, we adopt ProtDescribe dataset [62] to train the PLP-Transformer for protein-description representation learning, which contains 553,052 aligned pairs of protein sequences and textual descriptions such as protein names, functions, families, subcellular locations, etc.
Instruction Tuning Datasets
Nevertheless, despite covering the protein sequence positions, ProtDescribe de facto simply relies on textual descriptions to provide a rough indication of protein structural and functional similarity. It might be more straightforward to directly utilize structural information, especially considering that NetSurfP and ESM-IF are specifically designed for high-level protein structures. Considering this problem, we resort to the RCSB-PDB dataset [14] which comprises 143,508 tertiary structure-description aligned pairs of proteins. We further extract the protein sequence and expand this dataset with sequence and secondary structure labels. All secondary structure labels are constructed using DSSP (Dictionary of Protein Secondary Structure, see Appendix § 10).
Moreover, considering fine-tuning LLMs on other domains typically requires many task-specific examples, we further collect the corresponding literature from the RCSB PDB website based on the protein entry ID, and then extract the PubMed abstracts as an additional external web corpus. Given the protein-related document, we utilize a recent method Bonito [41] to convert unannotated text into corresponding Q&A-specific instruction pairs. In summary, for every protein taken into account, we compile its residue sequences, secondary structure label, tertiary atomic coordinates, along with corresponding 10 paired Q&A descriptions. We randomly select 1,000 pairs of protein for evaluation, and the rest part of the dataset is used for multi-level protein alignment (§ 3.2.2) and instruction tuning on LLMs (§ 3.3). More details and examples of our training datasets are given in Appendix § 13.
Implementation Details
In this paper, we use a progressive strategy to ensure that the pre-trained models retain their learned knowledge while fine-tuning them to align the protein embeddings with the LLMs. Firstly, we freeze all the multi-level protein encoders, solely focusing on training the PLP-former with the ProtDescribe dataset. Then the protein context gating module is trained with the RCSB-PDB dataset through contrastive learning. Finally, we freeze the LLM decoder and train the projection adapter with the generated instruction dataset. More implementation details are given in Appendix G.
Qualitative Comparisons
We first show some example conversations between the user and ProtChatGPT to indicate the promising results on both protein understanding and design tasks. For a fair comparison, we use the protein that does not appear in the training set of ProtChatGPT. Figure 3 shows several conversation examples of our ProtChatGPT related to protein understanding and design, as well as customized protein design tasks. It can be observed that our system comprehends the meaning of the question well, providing accurate and logically consistent responses that not only identify the types and characteristics of proteins, but also extend to exploring their related application areas and potential values. Additionally, when presented with a protein, ProtChatGPT can provide explanations regarding questions associated with protein understanding. This implies that researchers can swiftly survey the structures, functions, and other relevant properties through ProtChatGPT. Coupled with this comprehension capability, further judgments on mutations and disease association can be made about specific proteins, leading to the potential of targeted protein design in healthcare.
Figure 3. Several conversation examples. (a, b): ProtChatGPT can fully explore the intrinsic properties of proteins and accurately understand user queries, enabling protein understanding and analysis. (c): ProtChatGPT also has the potential to assist in drug development through pathogenicity analysis, diagnostic simulation, and protein design.
Case Study
Homologous Proteins
Homologous proteins are proteins that are derived from the same ancestral gene, which usually share similar amino acid sequences and structures. This study could help in understanding evolutionary relationships, predicting protein functions, and identifying potential therapeutic interventions. In this case, we choose Actin and Myosin proteins for our test. As shown in Figure 4 (a) and (Figure 4b), although these two proteins have similar amino acid sequences, our multi-level approach can still differentiate and make reasonable analyses due to their significant structural differences.
Figure 4. Challenging case studies. Benefitting from contrastive-based feature alignment and domain-specific instruction tuning, our proposed ProtChatGPT is capable of handling complex scenarios such as (a, b) homologous proteins and (c) mutually exclusive functions.
Mutually Exclusive Functions
Mutually exclusive protein functions describe situations where a certain protein in the same cell cannot simultaneously perform its roles, especially in signaling pathways, regulatory mechanisms, or cellular processes, where one activity inhibits or prevents another. Understanding these functions is important for grasping how cells make decisions and respond to their environment, particularly in complex processes like development, immune responses, and disease progression. In this case, we choose the p53 protein for our test. Figure 4 (c) indicates that our method can efficiently incorporate contextual semantics to provide varied analyses under different environment prompts.
Quantitative Comparisons
Protein Q&A
First, we provide comparisons for the protein Q&A task on the PDB-QA dataset [14]. We compared our LLAMA3-8B based method with ProteinChat [14] (Vicuna-13B based) and ProtT3 [34] (Galactica-1.3B based). We followed the same test setting as ProtT3 and selected the exact match as the evaluation metric. SP stands for structure/property, SI stands for supplementary information. We directly used the public checkpoints. As shown in Table 1, our method surpasses other baselines due to the combined encoding of all three levels of protein structures. In contrast, ProteinChat only uses tertiary structure while ProtT3 only uses protein sequence information, which ignores the cross-level complementarity and may lead to insufficient protein feature extraction.
| Model | String SP | String SI | Number SP | Number SI | Overall |
| ProteinChat | 7.2 | 10.3 | 28.9 | 19.2 | 15.5 |
| ProteinChat (pseudo label) | 7.8 | 9.5 | 27.4 | 17.5 | 14.2 |
| ProtT3 | 85.4 | 69.8 | 47.2 | 39.4 | 65.0 |
| Ours | 88.1 | 73.3 | 49.6 | 42.7 | 68.8 |
| Ours (pseudo label) | 86.7 | 72.8 | 49.2 | 41.9 | 67.7 |
Table 1. Protein Q&A task on PDB-QA dataset with GT and pseudo structures. Best performances are marked in bold.
Considering that users might only have the amino acid sequence, pseudo secondary and tertiary structures can be generated using structural prediction models such as AlphaFold2 [20]. To simulate
real-world application scenarios, we further provide more experiments on a large evaluation benchmark for the Protein Q&A task to verify the robustness under insufficient input. The original experiment was conducted on the PDB-QA dataset, in which the ground truth tertiary structures are directly obtained from RCSB-PDB [14].
With the same test set, here we use the protein sequence to predict the pseudo tertiary structure through AlphaFold2. We also use the pseudo-secondary structures obtained from NetSurfP [22] for experiments. It can be observed that even when using pseudo-structure labels as test inputs, our model still performs well. Additionally, we also conducted validation experiments on previous methods (ProtT3 is not applicable since it only utilizes sequence information) using the same pseudo inputs, which also resulted in good performance. This further demonstrates the feasibility of using the AlphaFold dataset [55] as pseudo labels for inference.
Protein-text Retrieval
We also provide experiments on the cross-modal protein-text retrieval task, as shown in Table 2. We use the ProteinKG25 [67] testset and select ProtST [62], Protein-CLAP [33] and ProtT3 [34] for comparisons. As the ProteinKG25 dataset only contains amino acid sequences as the protein query, this retrieval task essentially tests the performance of our PLP-former. For a fair comparison, we follow the same retrieval setting as ProtT3, which uses PTC to obtain the top- ranked candidates and PTM for re-ranking (see Appendix § 12 for PTC and PTM).
| Model | Protein to Text Acc | Protein to Text R@20 | Text to Protein Acc | Text to Protein R@20 |
| ProtST | 5.5 | 41.6 | 5.8 | 43.4 |
| ProteinCLAP | 39.0 | 89.4 | 39.3 | 89.7 |
| ProtT3 | 55.8 | 91.7 | 55.6 | 91.7 |
| Ours | 60.4 | 93.3 | 60.1 | 93.3 |
Table 2. Protein-text retrieval task on ProteinKG25 dataset.
Note that different from ProtT3 which only uses protein sequences as input, our ProtChatGPT introduces multi-level alignment from primary to tertiary structures, which leads to better protein representation (see Table 1), and further enables protein cross-level structure retrieval through subsequent contrastive learning (see Appendix § 15).
Ablation Study
To validate the effectiveness of our method, we further conduct several ablation studies. To fully showcase the capability of ProtChatGPT, we randomly select 1,000 protein protein-instruction pairs from our refined RCSB-PDB dataset to serve as the test set. Note that these testing protein pairs are not used during training for a fair comparison. We employ seven commonly used metrics in the image captioning and NLP domains to test the performance of ProtChatGPT. Detailed descriptions of these metrics can be found in Appendix § 16.
We first devise four variants to validate our contribution. (1) w/o tertiary structure: remove the branch of the tertiary structure encoding, relying solely on sequence and secondary structure embedding. (2) w/o secondary structure&PCG: remove the secondary structure encoder and the Protein Context Gating (PCG) module, directly use sequence and tertiary structure embeddings for contrastive learning. (3) w/o PLP-former: remove the PLP-former and directly use the sequence embedding from ESM-1b without the injection of protein descriptions. (4) w/o instruction tuning: using the original single descriptions from the RCSB-PDB dataset without generating multiple instruction pairs. It can be observed from Table 3 that all levels of protein structure information as well as the protein-language alignment play indispensable roles in supplementation and alignment, respectively. Notably, in contrast to common metrics like BLEU and METEOR, SPICE and BERTScore pay more attention to deeper semantic information, rather than just lexical and syntactic alignment. Particularly, we replace the original Bert encoder with PubMedBERT [13], a biomedical description-specific encoder pre-trained on large-scale datasets. To some extent, this indicator can reflect the scientific validity of the generated responses in the biomedicine domain.
| Method | BLEU-1 ↑ | BLEU-4 ↑ | ROUGE-L ↑ | METEOR ↑ | CIDEr ↑ | SPICE ↑ | PubMed BERTScore ↑ |
| w/o tertiary structure | 0.461 | 0.316 | 0.411 | 0.247 | 0.515 | 0.240 | 0.348 |
| w/o secondary structure & PCG | 0.509 | 0.332 | 0.445 | 0.250 | 0.544 | 0.265 | 0.411 |
| w/o PLP-former | 0.588 | 0.356 | 0.469 | 0.278 | 0.582 | 0.287 | 0.431 |
| w/o instruction tuning | 0.604 | 0.373 | 0.478 | 0.288 | 0.595 | 0.299 | 0.447 |
| ProtChatGPT (Vicuna-13B) | 0.618 | 0.399 | 0.493 | 0.299 | 0.640 | 0.321 | 0.465 |
| ProtChatGPT (LLAMA3-8B) | 0.643 | 0.417 | 0.515 | 0.326 | 0.687 | 0.346 | 0.487 |
Table 3. Quantitative comparisons on the proposed multi-level protein encoding, PCG module and PLP-former with frozen LLMs. ↑ indicates that a higher value corresponds to better performance.
Furthermore, we modified the training strategy of PLP and LLMs in an attempt to achieve better protein-specific dialogue capabilities. As shown in Figure 5, we further independently fine-tune the PLP-former and LLM decoders (using 16-bit LoRA [16] for Vicuna-13B and 4-bit QLoRA [10] for LLaMA3-8B) with our refined dataset with instruction pairs. The performance on two high-level semantic metrics SPICE and PubMed BERTScore indicate that further fine-tuning of both LLMs and PLP-former enhances the performance. More results under different LLMs and fine-tuning settings are given in Appendix § 15.
Figure 5. Comparison of fine-tuning of PLP-former and LLM decoders during the instruction tuning stage. We compute the SPICE and PubMed BERTScore for semantic evaluation.
Conclusion
In this paper, we introduce ProtChatGPT, an AI-based protein chat system, to implement ChatGPT-like functionalities for the protein research field. ProtChatGPT marks the initial effort at bootstrapping multi-level Protein-Language Pretraining (PLP) from pre-trained LLMs for various protein-language tasks. PLP sits at the intersection between protein and language, which effectively and efficiently enables ProtChatGPT to harvest the off-the-shelf large models from both protein and natural language communities. Experiments suggest that ProtChatGPT holds potential for application in protein understanding and design. We hope this work can facilitate protein research and further inspire other scientific disciplines.
References
Appendix
Appendix A: The limitation and discussion of ProtChatGPT.
Appendix B: The broader impact of ProtChatGPT.
Appendix C: We provide a brief introduction on proteins, including protein structures, homologous Proteins, mutually Exclusive Functions, and Dictionary of Protein Secondary Structure (DSSP).
Appendix D: Introduction and related work on Vision-Language Pretraining (VLP).
Appendix E: Details on the proposed Protein-Language Representation Learning (PLP).
Appendix F: Details and examples of the training Datasets.
Appendix G: More implementation details.
Appendix H: Additional experiments on LLM fine-tuning and protein retrieval.
Appendix I: Details on evaluation metrics used in this paper.
Limitation and Discussion
ProtChatGPT leverages the capabilities of LLMs for protein-specialized conversations. However, it inherits LLM’s potential language hallucination. It is an indispensable concern especially when it relates to protein research and healthcare. Although the two case studies in § 4.4 demonstrate that our ProtChatGPT can distinguish some complex cases (e.g., homologous proteins and mutually exclusive functions), these capabilities heavily depend on the presence of similar examples in our training dataset, especially the description part. Given an unknown protein, ProtChatGPT might produce certain descriptions that sound correct but lack proper scientific verification, possibly leading researchers astray. This issue might be alleviated by training the model with more high-quality, aligned protein-text pairs, or aligning with more advanced LLMs in the future. In this manner, rigorous data processing and selection strategies should be implemented to ensure the validity and reliability of the training data. Feedback from domain experts is also important to refine the model. Combined with reinforcement or continual learning techniques, ProtChatGPT could keep improving the quality of its responses. Furthermore, training only one projection adapter might not provide enough capacity to learn extensive protein-text alignment. This issue could be alleviated by designing a more powerful adapter to facilitate the interactions between sequence and structure embeddings or utilizing other powerful protein encoders such as GearNet [70] and ESM-2 [32]. In future work, addressing these issues and refining ProtChatGPT is essential. With ongoing improvements and regular expert feedback, ProtChatGPT has the potential to become a trusted assistant in protein research, offering more valuable insights for further investigations.
Broader Impact
ProtChatGPT extends across multiple dimensions of protein research and biotechnology. By facilitating dialogue-based interactions with a protein-specific model, ProtChatGPT could democratize access to complex protein analysis, enabling researchers, regardless of their computational expertise, to engage deeply with protein data. This could accelerate discovery processes in drug design, genetic research, and disease understanding by providing intuitive, conversational access to sophisticated protein analyses and hypothesis generation. Furthermore, ProtChatGPT could also serve educational purposes, aiding in the training of next-generation scientists by providing an interactive learning tool that offers immediate feedback and explanations about protein structures and functions. This might inspire innovative approaches in both academia and industry, potentially leading to novel therapeutic strategies and an enhanced understanding of biological mechanisms.
Brief Introduction on Proteins
Proteins are large molecules formed by amino acids linked through peptide bonds and are essential components of cells and organisms. They perform a range of critical functions in the body, from providing structural support to regulating biochemical processes. Proteins are made up of hundreds or thousands of smaller units called amino acids, which are attached to one another in long chains. There are 20 different types of amino acids that can be combined to make a protein. The sequence of amino acids determines the unique 3-dimensional structure and the specific function of each protein.
Protein structures
Proteins have four levels of structure:
Primary Structure: The sequence of amino acids in a polypeptide chain.
Secondary Structure: Localized areas stabilized by hydrogen bonds, such as alpha-helices and beta-sheets.
Tertiary Structure: The overall 3-dimensional structure of a single protein molecule; the spatial relationship of the secondary structures to one another.
Quaternary Structure: The structure formed by several protein molecules (polypeptide chains), usually called protein subunits in this context, which function as a single protein complex.
Homologous Proteins
Homologous proteins are proteins that share a common evolutionary origin, indicated by their similarity in sequence or structure across different species. These proteins may retain similar functions, suggesting their conservation through evolutionary processes. Homologous proteins can be categorized into two types: orthologs, which are proteins in different species that evolved from a common ancestral gene and perform similar functions; and paralogs, which are proteins within the same species that arose from gene duplication and have evolved new functions. The study of homologous proteins helps in understanding protein function, evolution, and the genetic relationships between species.
Mutually Exclusive Functions
Mutually exclusive functions of proteins refer to situations where different protein isoforms, arising from the same gene through alternative splicing, perform distinct and non-overlapping roles within a cell or organism. Each isoform might be involved in different pathways or cellular processes, ensuring that their functions do not overlap, thus providing a mechanism for regulating biological activities and responses to environmental changes. This concept is crucial for understanding how genetic information can give rise to multiple functional outcomes, allowing a single gene to contribute to various aspects of cellular function and organismal development
Dictionary of Protein Secondary Structure (DSSP)
DSSP [21] is a standard method used to describe the secondary structure of proteins, which has become one of the most widely used tools for analyzing protein structures. DSSP analyzes the atomic coordinates of proteins to determine the secondary structure type of each amino acid residue, such as -helix, -strand, turn, etc. It also provides additional information, such as hydrogen bonds between residues, the backbone torsion angles ( and ), and solvent accessibility.
The output format of DSSP typically consists of a text file, with each line corresponding to an amino acid residue in the protein structure. Each line contains a series of labels describing the secondary structure and its local environment for that residue. Generally, the DSSP output format includes the following key information:
Residue Number: Indicates the position number of each amino acid residue in the protein.
Residue Type: Specifies the type of amino acid residue, such as ALA, ARG, ASP, etc.
Chain Identifier: Indicates the identifier for different chains in the protein, usually a letter.
Secondary Structure: Indicates the secondary structure type for the residue, such as H (-helix), E (-sheet), T (turn), etc.
Hydrogen Bonds: Describes the hydrogen bonding interactions between the residue and its neighboring amino acid residues.
Phi () and Psi () angles: Represents the backbone and angles for the residue, used to describe the protein’s backbone conformation.
Solvent Accessibility: Describes the extent of exposure of the residue surface, i.e., the degree of contact between the residue and the aqueous solvent.
Each line in the DSSP output file contains a combination of these pieces of information to describe the features and environment of each residue in the protein structure.
Vision-Language Pretraining (VLP)
Data collected from different modalities generally offer distinct perspectives, frequently synergizing to yield a comprehensive understanding, enhancing the overall comprehension of the data. Vision-Language Pretraining (VLP) aims to learn multimodal foundation models, showing improved performance on various vision-and-language tasks [45]. Existing VLP methods can be roughly divided into representation learning-based and generative learning-based. Representation learning-based methods [17, 27, 29, 46, 63] usually consider the image-text pairs as multi-modal views of the same semantics, and perform contrastive or multi-view learning for the alignment between multiple modalities. Generative learning-based methods [4, 6, 28, 30, 36, 60, 65, 68] aim to reconstruct the corrupted text (image) with the assistance of visual (text) modality through
MLM-like objectives. For example, SimVLM [61] introduces a single prefix language modeling (PrefixLM) objective for exploiting large-scale weak supervision in VLP. CoCa [64] further verifies the representation ability of autoregressive language modeling in the vision-language domain. In this paper, we consider protein as a specialized biological language that encodes and communicates biological information through its amino acid sequences and interactions. Inspired by existing VLP methods [25, 71], we first propose a representation learning-based Protein-Language Pretraining (PLP) framework to understand protein sequences via natural language instructions, and then design a generative learning-based multi-level adapter to generate the question-related answers combined with extra structural embeddings.
Details on Protein-Language Representation Learning
Obtaining queries that can extract informative protein representation regarding text is significant for protein-language alignment. To achieve this, we connect our PLP-former with the ESM-1b [51] model during the representation learning phase and train with the protein-language pairs. Following [25, 26], we jointly train our model with three distinct pretraining tasks: Protein-Text Contrastive learning (PTC), Protein-grounded Text Generation (PTG), and Protein-Text Matching (PTM). Although these tasks utilize the same model structure and input format, they differ in the attention masking strategy applied between queries and text, thereby modulating their interaction.
Protein-Text Contrastive Learning (PTC). For Protein-Text Contrastive Learning, by maximizing the mutual information, we aim to ensure the latent representation of protein and text are well-aligned. Specifically, given the query representation from the protein transformer , we align it with the corresponding text embedding . This is achieved by maximizing the similarity of positive pairs against those negative pairs where we directly use the embedding of tokens from the text transformer as . Given that the output of the protein transformer comprises multiple embeddings, we calculate the pairwise similarity between each query output and . We then choose the highest value to represent the protein-text similarity. To prevent any information leakage, we utilize an uni-modal self-attention mask that restricts direct interaction between queries and text.
Protein-grounded Text Generation (PTG). The PTG task is designed to ensure that the learned queries can efficiently derive text-relevant information from the protein sequence. We train the PLP-former to produce descriptions matching the respective protein sequences. Since the PLP-former prevents direct interaction between the frozen ESM-1b and text tokens, the data needed for description generation must first be garnered by the queries, ensuring efficient information extraction. We use a multimodal causal self-attention mask, to manage the interaction between queries and text. While queries can interact with one another, they cannot engage with the text tokens. Conversely, each text token can refer to all queries as well as its preceding text tokens. Additionally, we substitute the token with a token at the beginning of the text sequence to indicate the decoding task. Protein-Text Matching (PTM). The protein-text Matching task is leveraged to align fine-grained protein-text representations. This task is designed as a binary classification task where the model needs to determine if a given image-text pair aligns (positive) or misaligns (negative). We employ a bi-directional self-attention mask, allowing all queries and texts to attend mutually. As a result, the obtained query embeddings, , encompass multimodal information. Each of these embeddings is then passed through a binary linear classifier to derive a logit, with the final matching score being the average of logits across all queries. For crafting informative negative pairs, we utilize the hard negative mining technique as described in [27].
Details and Examples of the Training Datasets
ProtDescribe Dataset. The ProtDescribe dataset [62] is designed to augment protein sequences with text descriptions of their functions and other important properties. This dataset contains 549,000 pairs of proteins, including the EntryName, ProteinName, Function (obtained from Uniport), SubcellularLocation, Similarity, and ProteinSequence. In our implementation, we only use the protein sequence and the corresponding protein functions for the training of PLP-former. The PLP-former takes both sequence and descriptions as inputs, after three combined training tasks (Appendix. § 12), then outputs 32 learned tokens indicating the sequence embeddings that are most relevant to the descriptions. Some examples of this dataset are given in Table. Table 4.
| EntryName | ProteinName | Function | SubcellularLocation | Similarity | Sequence |
| 14KL_BRUSI | Lectin-like protein BA14k | Has immunoglobulin-binding and hemagglutination properties, and can bind to mannose. Essential for virulence. May be involved in LPS biosynthesis or polysaccharide transport. | Cell membrane; Single-pass membrane protein | belongs to the BA14k family. | MNSFRKTCAGALALIFGATSIVPTVAAPMNMDRPAINQNVIQARAHYRPQNYNRGHRPGYWHGHRGYRHYRHGYRRHNDGWWYPLAAFGAGAIIGGAISQPRPVYRAPAGSPHVQWCYSRYKSYRASDNTFQPYNGPRKQCRSPYSR |
| 11013_ASFWA | Protein MGF 110-13L | Plays a role in virus cell tropism, and may be required for efficient virus replication in macrophages. | Host membrane; Multi-pass membrane protein | Belongs to the asfivirus MGF 110 family. | MGGGDYWPIIIRHCCFYLVFSIAFVGYIVFAYYKNLHLNTTMKLIALLCILIWLSQPGLNRPLSIFYMKQNLPRTYTPPIRELEYWCTYGKHCDFCWECRNGICKNKVWDDMPLIKQNDYISQCSIARYFDRCMYFIKPKTPYIHYMDCSQPTAYKGFSH |
Table 4. Examples of the original ProtDescribe dataset.
RCSB-PDB Dataset. The RCSB-PDB dataset [14] is originally sourced from the Research Collaboratory for Structural Bioinformatics Protein Data Bank1, which includes 204,826 experimentally determined 3D structures. ProteinChat [14] further selected 143,508 proteins that have a primary publication linked with a PubMed ID and can be extracted with a valid chain. Some examples of this dataset are given in Table. Table 5.
| Protein ID | Abstract of the Primary Publication |
| 1N21 | The x-ray crystal structure of dimeric (+)-bornyl diphosphate synthase, a metal-requiring monoterpene cyclase from Salvia officinalis, is reported at 2.0-A resolution. Each monomer contains two alpha-helical domains: the C-terminal domain catalyzes the cyclization of geranyl diphosphate, orienting and stabilizing multiple reactive carbocation intermediates; the N-terminal domain has no clearly defined function, although its N terminus caps the active site in the C-terminal domain during catalysis. Structures of complexes with aza analogs of substrate and carbocation intermediates, as well as complexes with pyrophosphate and bornyl diphosphate, provide "snapshots" of the terpene cyclization cascade. |
| 2J11 | The role of hydrophobic amino acids in the formation of hydrophobic cores as one of the major driving forces in protein folding has been extensively studied. However, the implication of neutral solvent-exposed amino acids is less clear and available information is scarce. We have used a combinatorial approach to study the structural relevance of three solvent-exposed residues (Tyr(327), Thr(329), and Gln(331)) located in the beta-sheet of the tetramerization domain of the tumor suppressor p53 (p53TD). A conformationally defined peptide library was designed where these three positions were randomized. The library was screened for tetramer stability. A set of p53TD mutants containing putative stabilizing or destabilizing residue combinations was synthesized for a thermodynamic characterization. Unfolding experiments showed a wide range of stabilities, with T(m) values between 27 and 83 degrees C. Wild type p53TD and some highly destabilized and stabilized mutants were further characterized. Thermodynamic and biophysical data indicated that these proteins were folded tetramers, with the same overall structure, in equilibrium with unfolded monomers. An NMR study confirmed that the main structural features of p53TD are conserved in all the mutants analyzed. The thermodynamic stability of the different p53TD mutants showed a strong correlation with parameters that favor the formation and stabilization of the beta-sheet. We propose that stabilization through hydrophobic interactions of key secondary structure elements might be the underlying mechanism for the strong influence of solvent-exposed residues in the stability of p53TD. |
| 3A08 | Two crystal structures of a collagen model peptide, (Pro-Pro-Gly)-Hyp-Hyp-Gly-(Pro-Pro-Gly) [where Hyp is (4R,2S)-L-hydroxyproline], showed very similar unit-cell parameters and belonged to the same space group P2(1). Both crystals exhibited pseudo-merohedral twinning. The main differences were in their molecular-packing arrangements. One modification showed pseudo-hexagonal packing, while the other showed pseudo-tetragonal packing. Despite their different packing arrangements, no significant differences were observed in the hydration states of these modifications. The peptide in the pseudo-tetragonal crystal showed a cyclic fluctuation of helical twists with a period of 20 A, while that in the pseudo-hexagonal crystal did not. In these modifications, the puckering conformations of four of the 12 Hyp residues at the X position of the Hyp(X)-Hyp(Y)-Gly sequence were in the opposite conformations to the previous hypothesis that Hyp(X) residue is involved in H(X)-Hyp(Y) and Hyp(X)-Pro(Y) stacking pairs prefer up-puckering and down-puckering conformations, respectively. Detailed investigation of the molecular interactions between Hyp(X) and adjacent molecules revealed that these opposite conformations appear because the puckering conformation, which follows the hypothesis, is subject to steric hindrance from the adjacent molecule. |
Table 5. Examples of the RCSB-PDB dataset with the protein entry ID from Protein Data Bank.
Instruction Tuning Dataset. Our refined instruction tuning dataset is based on the RCSB-PDB dataset. The RCSB-PDB dataset contains a protein entry ID from Protein Data Bank, a pdb file with atomic coordinates representing the molecular structure, along with the abstract of corresponding scientific literature. In this paper, we extract the amino acid chain as the primary sequence and further use the abstract to generate 10 Q&A pairs for instruction tuning. In summary, for every protein taken into account, we compile its residue sequences, secondary structure label, tertiary atomic coordinates, along with corresponding 10 paired Q&A descriptions. We use the pretrained Bonito [41], an open-source model2 for conditional task generation from specialized domains across three task types: yes-no question answering, extractive question answering, and natural language inference. Bonito takes the unannotated text and the task attribute as inputs, and the output consists of the instruction and the response. In this paper, we use Bonito to generate synthetic Q&A pairs based on the corresponding scientific literature (only abstract used for simplicity). In our implementation, the task attribute is set as “extractive question answering”. Some examples of our refined dataset are given in Table. Table 6 and Table. Table 7.
| Abstract | Instruction pairs |
| Assembly of polymerase chain reactions at room temperature can sometimes lead to low yields or unintentional products due to mispriming. Mutation of isoleucine 707 to leucine in DNA polymerase I from Thermus aquaticus substantially decreases its activity at room temperature without compromising its ability to amplify DNA. To understand why a conservative change to the enzyme over 20 Å from the active site can have a large impact on its activity at low temperature, we solved the X-ray crystal structure of the large (5’-to-3’ exonuclease-deleted) fragment of Taq DNA polymerase containing the cold-sensitive mutation in the ternary (E-DNA-ddNTP) and binary (E-DNA) complexes. The I707L KlenTaq1 ternary complex was identical to the wild-type in the closed conformation except for the mutation and a rotamer change in nearby phenylalanine 749, suggesting that the enzyme should remain active. However, soaking out of the nucleotide substrate at low temperature results in an altered binary complex made possible by the rotamer change at F749 near the tip of the polymerase O-helix. Surprisingly, two adenosines in the 5’-template overhang fill the vacated active site by stacking with the primer strand, thereby blocking the active site at low temperature. Replacement of the two overhanging adenosines with pyrimidines substantially increased activity at room temperature by keeping the template overhang out of the active site, confirming the importance of base stacking. These results explain the cold-sensitive phenotype of the I707L mutation in KlenTaq1 and serve as an example of a large conformational change affected by a conservative mutation. | Q: What is the effect of mutating isoleucine 707 to leucine in DNA polymerase I from Thermus aquaticus? A: The mutation substantially decreases the enzyme’s activity at room temperature without compromising its ability to amplify DNA. Q: Why is it significant that the mutation site is over 20 Å away from the active site? A: It is significant because it demonstrates that even conservative changes far from the active site can have a large impact on enzyme activity at low temperatures. Q: What method was used to understand the impact of the I707L mutation on enzyme activity? A: The X-ray crystal structure of the large fragment of Taq DNA polymerase containing the cold-sensitive mutation was solved in both ternary (E-DNA-ddNTP) and binary (E-DNA) complexes. Q: How did the I707L mutation affect the structure of the ternary complex? A: The I707L KlenTaq1 ternary complex was identical to the wild-type in the closed conformation, except for the mutation and a rotamer change in nearby phenylalanine 749. Q: How does the I707L mutation affect the enzyme’s activity at room temperature? A: The I707L mutation substantially decreases the enzyme’s activity at room temperature. Q: What happens to the I707L mutation’s nucleotide substrate at low temperatures? A: The nucleotide substrate soaks out at low temperatures, resulting in an altered binary complex. Q: What specific mutation was made to DNA polymerase I? A: Isoleucine 707 was mutated to leucine in DNA polymerase I. Q: What structural method was used to analyze the I707L mutation’s impact? A: X-ray crystallography was used to determine the structure of the mutated enzyme. Q: What was observed in the ternary complex structure of the I707L mutated enzyme? A: The ternary complex structure was identical to the wild-type except for the mutation and a rotamer change in phenylalanine 749. Q: What effect does the rotamer change at phenylalanine 749 have on the enzyme? A: The rotamer change enables two adenosines from the 5’-template overhang to fill the vacated active site, blocking it at low temperatures. |
Table 6. An example of our instruction tuning dataset. We only show the input abstract and the generated instruction pairs for simplicity.
| Abstract | Instruction pairs |
| The bacterial ubiD and ubiX or the homologous fungal fdc1 and pad1 genes have been implicated in the non-oxidative reversible decarboxylation of aromatic substrates, and play a pivotal role in bacterial ubiquinone (also known as coenzyme Q) biosynthesis or microbial biodegradation of aromatic compounds, respectively. Despite biochemical studies on individual gene products, the composition and cofactor requirement of the enzyme responsible for in vivo decarboxylase activity remained unclear. Here we show that Fdc1 is solely responsible for the reversible decarboxylase activity, and that it requires a new type of cofactor: a prenylated flavin synthesized by the associated UbiX/Pad1. Atomic resolution crystal structures reveal that two distinct isomers of the oxidized cofactor can be observed, an isoalloxazine N5-iminium adduct and a N5 secondary ketimine species with markedly altered ring structure, both having azomethine ylide character. Substrate binding positions the dipolarophile enoic acid group directly above the azomethine ylide group. The structure of a covalent inhibitor-cofactor adduct suggests that 1,3-dipolar cycloaddition chemistry supports reversible decarboxylation in these enzymes. Although 1,3-dipolar cycloaddition is commonly used in organic chemistry, we propose that this presents the first example, to our knowledge, of an enzymatic 1,3-dipolar cycloaddition reaction. Our model for Fdc1/UbiD catalysis offers new routes in alkene hydrocarbon production or aryl (de)carboxylation. | Q: What enzymatic activity is Fdc1 responsible for? A: Fdc1 is responsible for the reversible decarboxylation of aromatic substrates. Q: What type of cofactor does Fdc1 require for its activity? A: Fdc1 requires a prenylated flavin cofactor for its activity. Q: How is the cofactor for Fdc1 synthesized? A: The cofactor for Fdc1 is synthesized by UbiX/Pad1. Q: What are the two isomeric forms of the oxidized cofactor in Fdc1? A: The two isomeric forms are the isoalloxazine N5-iminium adduct and the N5 secondary ketimine species. Q:What characterizes the cofactor forms in Fdc1? A: Both cofactor forms in Fdc1 have azomethine ylide character. Q: How does substrate binding influence Fdc1’s catalytic mechanism? A: Substrate binding positions the enoic acid group directly above the azomethine ylide group, facilitating the catalytic process. Q: What does the covalent inhibitor-cofactor adduct structure in Fdc1 suggest about its mechanism? A: It suggests that 1,3-dipolar cycloaddition chemistry is a mechanism supporting reversible decarboxylation in Fdc1. Q: What makes Fdc1’s reaction significant in enzymatic chemistry? A: Fdc1’s reaction may represent the first enzymatic example of 1,3-dipolar cycloaddition, according to the text. Q: What potential industrial application does Fdc1’s enzymatic activity suggest? A: Fdc1’s enzymatic activity suggests potential applications in alkene hydrocarbon production and aryl (de)carboxylation processes. Q: What does the altered ring structure of the cofactor in Fdc1 indicate about its function? A: The altered ring structure indicates significant changes in electronic structure that likely influence the enzyme’s catalytic activity. |
Table 7. Another example of our instruction tuning dataset.
More Implementation Details
During training, the PLP-former is initialized with the pre-trained weights of PubMedBERT [13], whereas the cross-attention layers and learnable tokens are randomly initialized. Following the same setting as PubMedBERT, we use 32 learnable tokens with a dimension of 768, which is the same as the hidden dimension of the PLP-former. We pre-train the PLP-former on the ProtDescribe dataset for 20K epochs with a batch size of 64. We use the AdamW [35] optimizer with , , and a weight decay of 0.05. We use a cosine learning rate decay with a peak learning rate of 1e-4, a minimum learning rate of 8e-5, and a linear warm-up of 5K iterations. Note that we apply PLP only to the sequence embeddings since the reported protein structures are much less than sequences. For example, there are 182K experimentally determined structures in PDB [5] while 47M protein sequences in Pfam [40]. Thus we only use selected sequence-structure pairs from the RCSB-PDB dataset during the following stage. During contrastive learning of our Protein Context Gating (PCG) module, the temperature parameter is empirically set to 0.8, and is set to 128 as the number of both positive and negative samples. Finally, we freeze the PLP-former, PCG module and LLM decoder, and train the projection adapter for 1K epochs with a batch size of 128. The minimum learning rate for the second stage is set as 5e-5. We also conduct experiments on fine-tuning LLMs, evaluating both full parameter tuning and LoRA [16] tuning on Vicuna-13B. Additionally, we fine-tuned LLaMA3 with LoRA for comparison. To avoid out-of-memory issues, we randomly trim excessively long protein sequences and structures to 3,000 amino acids. All experiments are performed on 2 NVIDIA H-100 (80GB) GPUs. Our model with Vicuna-13B requires 3 days for the multi-level protein-language alignment stage and 2.5 days for the instruction tuning stage (full-parameter fine-tuning), while the LLaMA3-8B model takes approximately 1 day for the instruction tuning stage (LoRA fine-tuning). The pre-trained Vicuna-13B and LLaMA3-8B are obtained through the official repositories3.
Additional Experiments
More results on LLM fine-tuning
In our implementation, we use the frozen LLM encoder as default since the projection adapter works as an information bottleneck between hybrid protein features and the LLM decoder. We further improve the performance by deploying instruction tuning on the LLMs, as described in § 4.5. We perform 16-bit LoRA [16] on Vicuna-13B and quantized 4-bit QLoRA [10] on LLaMA3-8B. We also perform full-parameter funding on the Vicuna-13B model. Detailed results are shown in Table. Table 8.
| Method | BLEU-1 ↑ | BLEU-4 ↑ | ROUGE-L ↑ | METEOR ↑ | CIDEr ↑ | SPICE ↑ | PubMed BERTScore ↑ |
| ProtChatGPT (Vicuna-13B) | 0.618 | 0.399 | 0.493 | 0.299 | 0.640 | 0.321 | 0.465 |
| ProtChatGPT (LLaMA3-8B) | 0.643 | 0.417 | 0.515 | 0.326 | 0.687 | 0.346 | 0.487 |
| ProtChatGPT (Vicuna-13B-LoRA) | 0.670 | 0.435 | 0.547 | 0.344 | 0.695 | 0.377 | 0.505 |
| ProtChatGPT (LLaMA3-8B-LoRA) | 0.698 | 0.453 | 0.556 | 0.354 | 0.746 | 0.391 | 0.529 |
| ProtChatGPT (Vicuna-13B-FP) | 0.735 | 0.479 | 0.603 | 0.379 | 0.761 | 0.416 | 0.557 |
Table 8. Quantitative Comparisons on fine-tuned LLM decoders. ↑ indicates that a higher value corresponds to better performance. FP means full parameter tuning.
Experiments on Protein Retrieval
In this paper, we design a Protein Context Gating (PCG) module to align the protein sequence with secondary structure embedding, and further use contrastive learning to align the tertiary structures. To validate the effectiveness of our multi-level protein alignment, we conducted cross-level protein retrieval experiments. Specifically, we randomly selected sequences from the test set, processed them through the PCG module to obtain , and then identified the closest protein by calculating the cosine similarity between and the tertiary structure embedding of other proteins. The results are shown in Figure 6.
Figure 6. Results on cross-level protein retrieval task on the BLURB [13] (Biomedical Language Understanding and Reasoning Benchmark).
Details on Metrics
In our implementation, we use seven different metrics on the proposed ProtChatGPT to verify the performance. These metrics focus not only on low-level lexical and syntactic alignment but also on high-level semantic information.
BLEU [44] (BiLingual Evaluation Understudy) serves as a crucial metric for assessing the quality of the machine-generated text, particularly in machine translation contexts. It quantifies the similarity between the candidate and reference text, yielding a score within the range of 0 to 1. A higher BLEU score indicates a closer match between the candidate and reference texts. BLEU is mathematically defined as follows:
where BP (Brevity Penalty) helps penalize overly short translations and represents the precision of n-grams, ranging from 1 to a predefined maximum.
ROUGE-L [31] finds frequent use in the automatic evaluation of text summarization and machine translation. It calculates the longest common subsequence between the candidate and reference texts, prioritizing recall over precision. ROUGE-L can be formulated as:
where represents the predicted text with a length of . represents the ground truth text with a length of . is a hyperparameter used to adjust the emphasis on precision and recall. LCS calculates the length of the longest common subsequence, measures recall, and measures precision, respectively.
METEOR [3] provides a comprehensive evaluation of machine-generated text by considering not only exact word matches but also synonyms and stemming. It combines precision, recall, and alignment factors to offer a holistic assessment. METEOR is formulated as:
where Penalty penalizes excessive word mismatches, is a configurable parameter, and represents recall and precision respectively.
CIDEr [57] (Consensus-based Image Description Evaluation) primarily assesses the quality of image captions produced by automated systems. It places importance on consensus among multiple reference captions and emphasizes the inclusion of diverse descriptive words. The mathematical formulation of CIDEr is given by:
where represents the candidate text, denotes the set of reference texts, specifies the use of n-grams, represents the number of reference texts, and corresponds to the TF-IDF vector based on n-grams.
SPICE [2] offers a metric designed to evaluate the semantic content of image captions, with a focus on their precision within generated captions. The formulation of SPICE is as follows:
where the binary matching operator is the function that returns matching tuples in two scene graphs. represents the precision of semantic propositions and signifies the recall of semantic propositions, respectively.
BertScore [69] is a metric that leverages contextual embeddings from BERT models to assess the quality of machine-generated text. It measures the similarity between the candidate text and the reference text using contextual embeddings.
PubMed BERTScore comes from the classical BertScore [69], which is a metric that leverages contextual embeddings from BERT models to assess the quality of machine-generated text. It measures the similarity between the candidate text and the reference text using contextual embeddings. In our implementation, to better assess the quality of ProtChatGPT in generating protein-related descriptions, we further replaced the encoder with the encoder of PubMedBERT [13]. PubMedBERT is the latest BERT [11] model pre-trained on the biomedical corpus, which outperformed BioBERT
References
- [1]Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. 2022. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems 35 (2022), 23716–23736.arxiv.org/abs/2204.14198
- [2]Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. 2016. SPICE: Semantic Propositional Image Caption Evaluation. In ECCV.arxiv.org/abs/1607.08822
- [3]Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization. 65–72.
- [4]Hangbo Bao, Wenhu Wang, Li Dong, and Furu Wei. 2022. Vl-beit: Generative vision-language pretraining. arXiv preprint arXiv:2206.01127 (2022).arxiv.org/abs/2206.01127
- [5]Helen M Berman, John Westbrook, Zukang Feng, Gary Gilliland, Talapady N Bhat, Helge Weissig, Ilya N Shindyalov, and Philip E Bourne. 2000. The protein data bank. Nucleic acids research 28, 1 (2000), 235–242.
- [6]Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. 2020. UNITER: Universal image-text representation learning. In European Conference on Computer Vision (ECCV). https://arxiv.org/pdf/1909.11740
- [7]Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023. Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90%* ChatGPT Quality. https://lmsys.org/blog/2023-03-30-vicuna/
- [8]Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Bastian, et al. 2022. PaLM: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311 (2022).arxiv.org/abs/2204.02311
- [9]Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. 2023. InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning. arXiv:2305.06500 [cs.CV]arxiv.org/abs/2305.06500
- [10]Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2024. Qlora: Efficient finetuning of quantized llms. Advances in Neural Information Processing Systems 36 (2024).arxiv.org/abs/2305.14314
- [11]Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018).arxiv.org/abs/1810.04805
- [12]Vladimir Gligorijević, P Douglas Renfrew, Tomasz Kosciolek, Julia Koehler Leman, Daniel Berenberg, Tommi Vatanen, Chris Chandler, Bryn C Taylor, Ian M Fisk, Hera Vlamakis, et al. 2021. Structure-based protein function prediction using graph convolutional networks. Nature communications 12, 1 (2021), 3168.DOI
- [13]Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. 2021. Domain-specific language model pretraining for biomedical natural language processing. ACM Transactions on Computing for Healthcare (HEALTH) 3, 1 (2021), 1–23.DOI
- [14]Han Guo, Mingjia Huo, Ruiyi Zhang, and Pengtao Xie. 2023. Proteinchat: Towards achieving chatgpt-like functionalities on protein 3d structures. Authorea Preprints (2023).
- [15]Chloe Hsu, Robert Verkuil, Jason Liu, Zeming Lin, Brian Hie, Tom Sercu, Adam Lerer, and Alexander Rives. 2022. Learning inverse folding from millions of predicted structures. In International Conference on Machine Learning. 8946–8970.DOI
- [16]Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685 (2021).arxiv.org/abs/2106.09685
- [17]Chao Jia, Yinfeng Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V Le, Yunhsuan Sung, Zhen Li, and Tom Duerig. 2021. Scaling up visual and vision-language representation learning with noisy text supervision. arXiv preprint (2021).arxiv.org/abs/2102.05918
- [18]Wengong Jin, Jeremy Wohlwend, Regina Barzilay, and Tommi Jaakkola. 2021. Iterative refinement graph neural network for antibody sequence-structure co-design. arXiv preprint arXiv:2110.04624 (2021).arxiv.org/abs/2110.04624
- [19]Bowen Jing, Stephan Eismann, Patricia Suriana, Raphael JL Townshend, and Ron Dror. 2020. Learning from protein structure with geometric vector perceptrons. arXiv preprint arXiv:2009.01411 (2020).arxiv.org/abs/2009.01411
- [20]John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Žídek, Anna Potapenko, et al. 2021. Highly accurate protein structure prediction with AlphaFold. Nature 596, 7873 (2021), 583–589.DOI
- [21]Wolfgang Kabsch and Christian Sander. 1983. Dictionary of protein secondary structure: pattern recognition of hydrogen-bonded and geometrical features. Biopolymers: Original Research on Biomolecules 22, 12 (1983), 2577–2637.pubmed.ncbi.nlm.nih.gov/6667333
- [22]Michael Schantz Klausen, Martin Closter Jespersen, Henrik Nielsen, Kamilla Kjaergaard Jensen, Vanessa Isabell Juritz, Casper Kaae Soenderby, Morten Otto Alexander Sommer, Ole Winther, Morten Nielsen, Bent Petersen, et al. 2019. NetSurfP-2.0: Improved prediction of protein structural features by integrated deep learning. Proteins: Structure, Function, and Bioinformatics 87, 6 (2019), 520–527.DOI
- [23]Jing Yu Koh, Ruslan Salakhutdinov, and Daniel Fried. 2023. Grounding language models to images for multimodal generation. arXiv preprint arXiv:2301.13823 (2023).
- [24]Xiangzhe Kong, Wenbing Huang, and Yang Liu. 2022. Conditional antibody design as 3d equivariant graph translation. arXiv preprint arXiv:2208.06073 (2022).arxiv.org/abs/2208.06073
- [25]Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597 (2023).
- [26]Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. 2022. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International Conference on Machine Learning. 12888–12900.arxiv.org/abs/2201.12086
- [27]Junnan Li, Ramprasath R Selvaraju, Akhilesh Deepak Gotmare, Shafiq Joty, Caiming Xiong, and Steven Hoi. 2021. Align before Fuse: Vision and Language Representation Learning with Momentum Distillation. In Conference on Neural Information Processing Systems (NeurIPS).arxiv.org/abs/2107.07651
- [28]Liuan Liu Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. 2019. VisualBERT: A simple and performant baseline for vision and language. arXiv preprint (2019). https://arxiv.org/pdf/1908.03557
- [29]Liuan Liu Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, et al. 2022. Grounded language-image pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 10965–10975.arxiv.org/abs/2112.03857
- [30]Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, et al. 2020. Oscar: Object-semantics aligned pre-training for vision-language tasks. In European Conference on Computer Vision (ECCV). https://arxiv.org/pdf/2004.06165
- [31]Chin-Yew Lin and Edward Hovy. 2002. Manual and automatic evaluation of summaries. In Proceedings of the ACL-02 workshop on automatic summarization. 45–51.DOI
- [32]Zeming Lin, Halil Akin, Roshan Rao, Brian Hie, Zhongkai Zhu, Wenting Lu, Nikita Smetanin, Robert Verkuil, Ori Kabeli, Yaniv Shmueli, et al. 2023. Evolutionary-scale prediction of atomic-level protein structure with a language model. Science 379, 6637 (2023), 1123–1130.DOI
- [33]Shengchao Liu, Yuqiang Li, Yuanjing Li, Zhuoxinran Li, Anthony Gitter, Yutao Zhu, Jiarui Lu, Zhao Xu, Weili Nie, Arvind Ramanathan, Chaowei Xiao, et al. 2023. A text-guided protein design framework. arXiv preprint arXiv:2302.04611 (2023).DOI
- [34]Zhiyuan Liu, An Zhang, Hao Fei, Enzhi Zhang, Xiang Wang, Kenji Kawaguchi, and Tat-Seng Chua. 2024. ForT3: Protein-to-Text Generation for Text-based Protein Understanding. arXiv preprint arXiv:2405.12564 (2024).
- [35]Ilya Loshchilov and Frank Hutter. 2017. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 (2017).arxiv.org/abs/1711.05101
- [36]Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019. Wilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. In Conference on Neural Information Processing Systems (NeurIPS).arxiv.org/abs/1908.02265
- [37]Ali Madani, Ben Krause, Eric R Greene, Subu Subramanian, Benjamin P Mohr, James M Holton, Jose Luis Olmos Jr, Caiming Xiong, Zachary Z Sun, Richard Socher, et al. 2023. Large language models generate functional protein sequences across diverse families. Nature Biotechnology (2023), 1–8.
- [38]Sazan Mahbub and Md Shamsuzzoha Bayzid. 2022. EGRET: edge aggregated graph attention networks and transfer learning improve protein–protein interaction site prediction. Briefings in Bioinformatics 23, 2 (2022), bbab578.DOI
- [39]Joshua Meier, Roshan Rao, Robert Verkuil, Jason Liu, Tom Sercu, and Alex Rives. 2021. Language models enable zero-shot prediction of the effects of mutations on protein function. Advances in Neural Information Processing Systems 34 (2021), 29287–29303.DOI
- [40]Jaina Mistry, Sara Chuguransky, Lowri Williams, Matloob Qureshi, Gustavo A Salazar, Erik LL Sonnhammer, Silvio CE Tosatto, Lisanna Paladin, Shriya Raj, Lorna J Richardson, et al. 2021. Pfam: The protein families database in 2021. Nucleic acids research 49, D1 (2021), D412–D419.DOI
- [41]Nihal V Nayak, Yiyang Nan, Avi Trost, and Stephen H Bach. 2024. Learning to Generate Instruction Tuning Datasets for Zero-Shot Task Adaptation. arXiv preprint arXiv:2402.18334 (2024).arxiv.org/abs/2402.18334
- [42]Pascal Notin, Mafalda Dias, Jonathan Frazer, Javier Marchena Hurtado, Aidan N Gomez, Debora Marks, and Yarin Gal. 2022. Tranception: protein fitness prediction with autoregressive transformers and inference-time retrieval. In International Conference on Machine Learning. 16990–17017.arxiv.org/abs/2205.13760
- [43]OpenAI. 2023. GPT-4 Technical Report. arXiv preprint arXiv:2303.08774 (2023).arxiv.org/abs/2303.08774
- [44]Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics. 311–318.arxiv.org/abs/1606.05250
- [45]Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Gish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learning. PMLR, 8748–8763.arxiv.org/abs/2103.00020
- [46]Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Gish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. [47] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog 1, 8 (2019), 9.
- [48]Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research 21, 1 (2020), 5485–5551.arxiv.org/abs/1910.10683
- [49]Roshan Rao, Joshua Meier, Tom Sercu, Sergey Ovchinnikov, and Alexander Rives. 2020. Transformer protein language models are unsupervised structure learners. Biorxiv (2020), 20–12.DOI
- [50]Manon Réau, Nicolas Renaud, Li C Xue, and Alexandre MJJ Bonvin. 2023. DeepRank-GNN: a graph neural network framework to learn patterns in protein–protein interfaces. Bioinformatics 39, 1 (2023), btac759.DOI
- [51]Alexander Rives, Joshua Meier, Tom Sercu, Siddharth Goyal, Zeming Lin, Jason Liu, Demi Guo, Myle Ott, C Lawrence Zitnick, Jerry Ma, et al. 2021. Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences. Proceedings of the National Academy of Sciences 118, 15 (2021), e2016239118.DOI
- [52]Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic. 2022. Galactica: A large language model for science. arXiv preprint arXiv:2211.09085 (2022).arxiv.org/abs/2211.09085
- [53]Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 (2023).arxiv.org/abs/2302.13971
- [54]Tom van Sonsbeek, Mohammad Mahdi Derakhshani, Ivona Najdenkoska, Cees GM Snoek, and Marcel Worring. 2023. Open-ended medical visual question answering through prefix tuning of language models. arXiv preprint arXiv:2303.05977 (2023).arxiv.org/abs/2303.05977
- [55]Mihaly Varadi, Stephen Anyango, Mandar Deshpande, Sreenath Nair, Cindy Natassia, Galabina Yordanova, David Yuan, Oana Stroe, Gemma Wood, Agata Laydon, et al. 2022. AlphaFold Protein Structure Database: massively expanding the structural coverage of protein-sequence space with high-accuracy models. Nucleic acids research 50, D1 (2022), D439–D444.DOI
- [56]Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems 30 (2017).
- [57]Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. 2015. Cider: Consensus-based image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition. 4566–4575.arxiv.org/abs/1411.5726
- [58]Jesse Vig, Ali Madani, Lav R Varshney, Caiming Xiong, Richard Socher, and Nazneen Fatema Rajani. 2020. Bertology meets biology: Interpreting attention in protein language models. arXiv preprint arXiv:2006.15222 (2020).
- [59]Sheng Wang, Zih ao Zhao, Xi Ouyang, Qian Wang, and Dinggang Shen. 2023. Chatcad: Interactive computer-aided diagnosis on medical image using large language models. arXiv preprint arXiv:2302.07257 (2023).arxiv.org/abs/2302.07257
- [60]Teng Wang, Wenhao Jiang, Zhichao Lu, Feng Zheng, Ran Cheng, Chengguo Yin, and Ping Luo. 2022. VLMixer: Unpaired Vision-Language Pre-training via Cross-Modal CutMix. In International Conference on Machine Learning. PMLR, 22680–22690.arxiv.org/abs/2206.08919
- [61]Zirui Wang, Jiahui Yu, Adams Wei Yu, Zhihang Dai, Yulia Tsvetkov, and Yuan Cao. 2021. SimVLM: Simple Visual Language Model Pretraining with Weak Supervision. arXiv preprint (2021).arxiv.org/abs/2108.10904
- [62]Minghao Xu, Xinyu Yuan, Santiago Miret, and Jian Tang. 2023. Protst: Multi-modality learning of protein sequences and biomedical texts. arXiv preprint arXiv:2301.12040 (2023).arxiv.org/abs/2301.12040
- [63]Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunju Yang. 2021. Filip: Fine-grained interactive language-image pre-training. arXiv preprint arXiv:2111.07783 (2021).arxiv.org/abs/2111.07783
- [64]Jiahui Yu, Zirui Wang, Vijay Vasudevan, Yogesh K. Jang, Mojtaba Seyedhosseini, and Yonghui Wu. 2022. CoCa: Contrastive captions are image-text foundation models. arXiv preprint arXiv:2205.01917 (2022).arxiv.org/abs/2205.01917
- [65]Yan Zeng, Xinsong Zhang, and Hang Li. 2021. Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Concepts. arXiv preprint arXiv:2111.08276 (2021).arxiv.org/abs/2111.08276
- [66]Ningyu Zhang, Zhen Bi, Xiaozhuan Liang, Siyuan Cheng, Haosen Hong, Shumin Deng, Jiazhang Lian, Qiang Zhang, and Huajun Chen. 2022. Ontoprotein: Protein pretraining with gene ontology embedding. arXiv preprint arXiv:2201.11147 (2022).arxiv.org/abs/2201.11147
- [67]Ningyu Zhang, Zhen Bi, Xiaozhuan Liang, Siyuan Cheng, Haosen Hong, Shumin Deng, Jiazhang Lian, Qiang Zhang, and Huajun Chen. 2022. Ontoprotein: Protein pretraining with gene ontology embedding. In International Conference on Learning Representations (ICLR).arxiv.org/abs/2201.11147
- [68][68*] Pengchuan Zhang, Xiaojun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, Lijuan Wang, Yejin Choi, and Jianfeng Gao. 2021. VinVL: Revisiting visual representations in vision-language models. In Conference on Computer Vision and Pattern Recognitionarxiv.org/abs/2101.00529
- [69]Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675 (2019).arxiv.org/abs/1904.09675
- [70]Zuobai Zhang, Minghao Xu, Arian Jamshid, Vijil Chenthamarakshan, Aurelie Lozano, Payel Das, and Jian Tang. 2023. Protein representation learning by geometric structure pretraining. In International Conference on Learning Representations.arxiv.org/abs/2203.06125
- [71]Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. 2023. MiniGPT-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592 (2023).arxiv.org/abs/2304.10592




