betaFeedback

WorksComputer Science

Leveraging Partial SMILES Validation Scheme for Enhanced Drug Design in Reinforcement Learning Frameworks

Abstract

SMILES-based molecule generation has emerged as a powerful approach in drug discovery. Deep reinforcement learning (RL) using large language model (LLM) has been incorporated into the molecule generation process to achieve high matching score in term of likelihood of desired molecule candidates. However, a critical challenge in this approach is catastrophic forgetting during the RL phase, where knowledge such as molecule validity, which often exceeds 99% during pretraining, significantly deteriorates. Current RL algorithms applied in drug discovery, such as REINVENT, use prior models as anchors to retian pretraining knowledge, but these methods lack robust exploration mechanisms. To address these issues, we propose Partial SMILES Validation-PPO (PSV-PPO), a novel RL algorithm that incorporates real-time partial SMILES validation to prevent catastrophic forgetting while encouraging exploration. Unlike traditional RL approaches that validate molecule structures only after generating entire sequences, PSV-PPO performs stepwise validation at each auto-regressive step, evaluating not only the selected token candidate but also all potential branches stemming from the prior partial sequence. This enables early detection of invalid partial SMILES across all potential paths. As a result, PSV-PPO maintains high validity rates even during aggressive exploration of the vast chemical space. Our experiments on the PMO and GuacaMol benchmark datasets demonstrate that PSV-PPO significantly reduces the number of invalid generated structures while maintaining competitive exploration and optimization performance. While our work primarily focuses on maintaining validity, the framework of PSV-PPO can be extended in future research to incorporate additional forms of valuable domain knowledge, further enhancing reinforcement learning applications in drug discovery.

Proceedings of the 41st International Conference on Machine Learning, Vienna, Austria. PMLR 235, 2024. Copyright 2024 by the author(s).

Introduction

The Simplified Molecular Input Line Entry System (SMILES) (Weininger [24]) has been widely adopted as one of molecular representations for various molecular generation and prediction tasks (Segler et al., [22]; Bjerrum & Threlfall, 2017; Irwin et al., 2022; Bagal et al., 2021). Large language model (LLM) have demonstrated strong capabilities in SMILES-based molecule generation, and reinforcement learning (RL) is often employed to fine-tune these models to optimize desired molecular properties (Mokaya et al., [15]; Korshunova et al., [7]).

Despite their strong performance during pretraining, language models often experience catastrophic forgetting when fine-tuned with RL (Kirkpatrick et al., [6]). Reinforce Learning with exploration, while widely used across various applications, frequently overrides the model’s prior knowledge (Kemker et al., [5]), leading to instability, increased generation of invalid molecules, and inefficient learning. This issue is particularly concerning in molecular generation, where essential properties such as validity, which can exceed 99% during pretraining (Arús-Pous et al., 2020), deteriorate significantly once RL optimization begins.

To address pretraining knowledge degradation during RL, several methods have been proposed. Reinvent (Blaschke et al., 2020) uses a pre-trained prior model to maintain learned knowledge while guiding RL-based optimization. While this approach helps retain molecular validity, it limits the ability of the model to explore novel chemical structures, restricting diversity. Other methods, such as SELFIES (Krenn et al., [9]) and grammar-VAE (Kusner et al., [11]), enforce validity at the representation level by constraining molecular embeddings. Although these techniques effectively eliminate invalid structures at the encoding stage, studies have shown that SMILES-based methods often outperform them in terms of molecular property optimization and diversity (Ghugare et al., 2023). The rigid constraints imposed by these representations can hinder efficient chemical space exploration, reducing their effectiveness in real-world drug discovery applications.

Image not extracted

Figure 1. The training flow of the PSV-PPO algorithm. The process begins with the prior model generating molecular structures and their associated probabilities (1). Both the rewards and the Partial SMILES Validation (PSV) Table are computed concurrently, which minimizes the computational overhead (2). The experience replay memory pool is updated with the scored molecules (3). Samples are then drawn from the experience replay memory pool (4) and replayed through the current model (5). The PSV-PPO algorithm calculates multiple loss functions, including HD Loss, TPC Loss, Entropy Loss, GPS Loss, Value Loss, and Policy Loss (6), which are then accumulated (7) and used to perform a backward pass to update the model parameters (8). The PSV scheme ensures that the model learns to generate valid molecular structures by providing real-time feedback on token validity during the generation process.

Among RL algorithms, proximal policy optimization (PPO) has been widely adopted due to its strong performance in various domains, including natural language processing and control tasks (Wu et al., 2023). However, its application to SMILES-based molecular generation presents significant challenges. The entropy-driven exploration mechanism in PPO encourages diverse sampling but can lead to instability (Hao et al., 2023). Large policy updates introduce excessive variation between consecutive iterations, causing gradient explosion, which destabilizes training. On the other hand, reducing entropy to maintain stability leads to mode collapse, where the model consistently selects the same high-probability tokens [23]. This issue is particularly problematic in SMILES-based generation, where token dependencies are strict, and minor errors can render an entire molecule invalid. A collapsed policy may excessively favor frequently occurring tokens, disrupting structural integrity and reducing the validity of generated molecules. The challenge of balancing exploration and exploitation in this setting makes PPO-based molecular generation highly unstable.

Another fundamental issue in RL-based molecule generation is sparse and delayed rewards. Molecular validity and property scores are typically assessed only after a full SMILES string is generated, preventing the model from receiving real-time feedback on structural correctness. Without intermediate guidance, PPO fine-tuning may lead the model

toward structurally incorrect molecules, resulting in inefficient learning. Some methods embed validity constraints directly into the generation process, such as SELFIES and grammar-enforced models, but these approaches have been found to be less effective than elfSMILES-based techniques in optimizing molecular properties (Ghugare et al., 2023). Without an intermediate validation mechanism, PPO fine-tuning degrades molecular validity over time, leading to inefficient exploration and reduced performance in molecular optimization tasks.

To overcome these challenges, we propose Partial SMILES Validation - Proximal Policy Optimization (PSV-PPO), a reinforcement learning framework that integrates real-time molecular validity checks into PPO. Unlike conventional RL approaches that evaluate validity only after generating a full molecule, PSV-PPO introduces a partial SMILES validation mechanism that dynamically penalizes invalid token selections at each autoregressive step. This real-time intervention prevents errors from propagating, stabilizing training while improving overall molecular validity.

The core of PSV-PPO is the PSV truth table, which systematically evaluates candidate tokens before they are appended to a growing SMILES string. If a token is determined to lead to an invalid structure, the policy network is immediately penalized. This approach ensures:

  • Immediate feedback during generation, reducing wasted exploration.

  • Controlled exploration, allowing for diverse yet syntactically valid molecule generation.

  • Stable training by mitigating mode collapse and gradient explosion.

Our main contributions are as follows:

  1. Partial SMILES Validation Mechanism: We introduce the PSV truth table, a principled approach to verify token validity before appending it to a growing SMILES sequence. While PSV does not guarantee a completely valid final molecule, it ensures that invalid token sequences are detected and penalized during generation.

  1. PSV-PPO Algorithm: We seamlessly integrate PSV into PPO, enabling effective molecule generation while maintaining exploration and stability. This approach significantly improves molecule validity while preventing degenerative policies.

  1. Adaptive Loss Regularization: We introduce new loss terms that dynamically penalize tokens with excessively high probabilities, preventing mode collapse. The PSV table determines when to apply this regularization, ensuring effective exploration while maintaining token validity.

Related Works

Machine Learning for Drug Discovery

The integration of machine learning into drug discovery has significantly transformed the landscape of molecular design, enabling the generation of novel compounds through various molecular representations, including 1D, 2D, and 3D formats [19] (Wu et al. 2018; Lo et al. 2018). These approaches utilize advanced deep learning architectures to model and predict chemical properties, facilitate molecule generation, and optimize lead compounds.

SMILES Representation. SMILES is a text-based notation that encodes molecular structures as sequences of characters, where each character or set of characters corresponds to specific atoms, bonds, or topological features. This linear representation simplifies the complex task of molecule generation into a sequence prediction problem, similar to those in natural language processing (NLP). Leveraging this analogy, models such as recurrent neural networks (RNNs) and transformers, originally developed for NLP tasks, have been adapted to generate and analyze SMILES strings [2] (Gómez-Bombarelli et al. 2018).

2D and 3D Molecule Generation. While SMILES strings provide a straightforward and compact representation, 2D and 3D molecular representations offer more detailed insights into a molecule’s spatial and topological properties, which are crucial for understanding molecular interactions, predicting chemical properties, and designing drugs with specific biological activities. Machine learning models, particularly those based on graph neural networks (GNNs) and spatial convolutional networks, have been increasingly applied to generate and optimize these 2D and 3D structures. These models capture the complex geometry and electronic distribution of molecules, providing a more nuanced approach to drug discovery (Duvenaud et al., 2015; Gilmer et al., 2017).

Atom-wise Tokenization. Atom-wise tokenization offers an alternative to traditional SMILES-based representations by breaking down molecules into individual atoms and their associated bonds. This finer granularity enables models to better capture subtle variations in molecular structures that might be overlooked in more coarse-grained representations. By focusing on the atom-level details, this approach improves the accuracy and diversity of generated molecules, enhancing the model’s ability to predict and optimize molecular properties [4, 3].

Partial SMILES. The partialsmiles package [16] provide real-time syntax validation, kekulization, and valence checks, allowing for on-the-fly detection of errors in SMILES strings as they are generated. The Partial SMILES performs three key checks:

  1. Syntax Compliance: Guarantees that the SMILES string follows the established syntactical conventions.

  1. Aromaticity Handling: Checks if aromatic systems can be appropriately kekulized by making implicit bonds explicit where required.

  1. Valence Validation: Ensures that the valence of each atom falls within the acceptable chemical range.

Revisit of PPO

PPO is a widely adopted RL algorithm that enhances the stability of policy gradient methods through the use of a surrogate objective function [21]. PPO is commonly implemented using frameworks such as OpenAI Baselines (Dhariwal et al., 2017), which provide standard implementations of various reinforcement learning algorithms. This carefully balanced objective enables PPO to improve policies while maintaining stability, making it a robust and reliable choice for a wide range of reinforcement learning tasks, including complex robotic control (Andrychowicz et al., 2020). PPO builds on the principles of Trust Region Policy Optimization (TRPO) [20], carefully controlling the magnitude of policy updates, ensuring that the new policy remains within a constrained region. close to the old policy, thereby improving learning stability and efficiency (Henderson et al., 2018).

Image not extracted

Figure 2. Visualization of the Partial SMILES Validation (PSV) framework used in our study. (a) PSV Truth Table illustrating the validity assessment of each candidate token in a partially generated SMILES string. (b) Box plot showing the distribution of the number of valid candidate tokens based on the preceding token in the SMILES string. (c) Overall distribution of valid candidate tokens across all states derived from 10,000 sampled molecules.

At the core of PPO is the clipped surrogate objective, designed to maximize expected advantage while constraining policy changes to prevent overly large updates. Given a policy πθ\pi_\theta parameterized by θ\theta, the probability ratio rt(θ)r_t(\theta) between the new and old policies is defined as:

rt(θ)=πθ(at∣st)πθold(at∣st)(1)r_t(\theta) = \frac{\pi_\theta(a_t \mid s_t)}{\pi_{\theta_{\mathrm{old}}}(a_t \mid s_t)} \tag*{(1)}

The objective function for PPO is then expressed as:

LCLIP(θ)=Et[min⁡(rt(θ)A(st,at),clip⁡(rt(θ),1−ϵ,1+ϵ)A(st,at))](2)L^{\mathrm{CLIP}}(\theta) = \mathbb{E}_t \left[ \min\left( r_t(\theta) A(s_t, a_t), \operatorname{clip}(r_t(\theta), 1-\epsilon, 1+\epsilon) A(s_t, a_t) \right) \right] \tag*{(2)}

where A(st,at)A(s_t, a_t) represents the advantage estimate, and ϵ\epsilon is a hyperparameter controlling the clip range. This clipped objective mitigates the risk of excessive policy divergence, striking a balance between exploration and exploitation.

In addition to the policy loss, PPO optimizes a value function Vϕ(st)V_\phi(s_t) by minimizing the squared error between predicted values and actual returns, denoted as:

LVALUE(ϕ)=Et[(Vϕ(st)−Rt)2](3)L^{\mathrm{VALUE}}(\phi) = \mathbb{E}_t \left[ \left( V_\phi(s_t) - R_t \right)^2 \right] \tag*{(3)}

where RtR_t is the cumulative return at time step tt.

To encourage sufficient exploration, PPO incorporates entropy regularization, penalizing overly deterministic policies through an entropy loss:

LENTROPY(θ)=−βEt[∑aπθ(a∣st)log⁡πθ(a∣st)](4)L^{\mathrm{ENTROPY}}(\theta) = -\beta\mathbb{E}_t \left[ \sum_a \pi_\theta(a \mid s_t) \log\pi_\theta(a \mid s_t) \right] \tag*{(4)}

where β\beta is the entropy coefficient that controls the regularization strength.

Moreover, a Kullback-Leibler (KL) divergence [10] term can be optionally included to further restrict the policy update, ensuring the updated policy does not deviate significantly from the previous one:

LKL(θ)=Et[KL⁡[πθold(⋅∣st)∥πθ(⋅∣st)]](5)L^{\mathrm{KL}}(\theta) = \mathbb{E}_t \left[ \operatorname{KL} \left[ \pi_{\theta_{\mathrm{old}}}(\cdot\mid s_t) \mathbin{\|} \pi_\theta(\cdot\mid s_t) \right] \right] \tag*{(5)}

The overall objective function in PPO is a weighted combination of these components:

L(θ,ϕ)=LCLIP(θ)+c1LVALUE(ϕ)−c2LENTROPY(θ)+c3LKL(θ)(6)L(\theta, \phi) = L^{\mathrm{CLIP}}(\theta) + c_1 L^{\mathrm{VALUE}}(\phi) - c_2 L^{\mathrm{ENTROPY}}(\theta) + c_3 L^{\mathrm{KL}}(\theta) \tag*{(6)}

where c1c_1, c2c_2, and c3c_3 are coefficients that balance the contributions of the value loss, entropy loss, and KL divergence, respectively. This carefully balanced objective enables PPO to effectively improve policies while maintaining stability, making it a robust and reliable choice for a wide range of reinforcement learning tasks, including complex robotic control (Andrychowicz et al., 2020). Practical implementations of PPO, such as those provided by Kostrikov [8], are widely used in the reinforcement learning community for research and application.

Partial SMILES Validation PPO (PSV-PPO)

The overall architecture of PSV-PPO is depicted in Figure 1, which illustrates how PSV-PPO builds upon the principles of Proximal Policy Optimization (PPO) while incorporating the Partial SMILES Validation (PSV) table into the training loop. We first explain how we build a PSV table given a generated SMILES string, then

The core innovation of PSV-PPO lies in its ability to ensure chemically valid exploration of the molecular space. This is achieved through the integration of the PSV table, which plays a crucial role in guiding the model away from invalid token sequences during the training process. Additionally, PSV-PPO introduces several novel loss functions: PSV-Driven Entropy Loss, PSV-Driven Hellinger Distance Loss, Global Probability Stabilization Loss (GPS Loss), and Token Probability Control Loss (TPC Loss). These losses collectively prevent the model from overcommitting to specific molecular structures or tokens, thereby promoting diversity in the generated molecules.

PSV Truth Table

The PSV table is the foundational component of the PSV-PPO framework. The goal is to identify the validity of each candidate token at every step of the tokenized SMILES sequence generation. Figure 2 (a) represents an PSV table for the molecule ”ClCc1c[nH]cn1”. The tokenized sequence of this candidate tokens are shown as then row of the matrix table. Each column in the table is the binary vector for all candidate tokens from the vocabulary and its value indicates whether the current token will immediately cause the partially generated SMILES become invalid. The pseudo-code for generating the PSV table is provided in the Appendix.

Objectives of PSV-PPO

The PSV-PPO framework, illustrated in Figure 1, extends the standard PPO algorithm and incorporates experience replay domain-specific enhancements. At the beginning of one PSV-PPO learning epoch, molecular samples are initially generated by the prior policy, and these samples, along with their associated rewards and PSV tables, are stored in an experience replay memory pool. To optimize computational efficiency, we the probability distribution of all actions from the prior model is also stored in the memory pool. This stored probability distribution contains not only the probability of action tokens, but also all other candidates tokens at each state.

The current policy then replays molecules sampled from the memory pool, generating a probability distribution for the actions with the same way as the prior policy. Concurrently, the critic network predicts values that contribute to the calculation of the value loss. While the clip loss and value loss are retain their original forms, the entropy loss and KL divergence loss are modified to integrate domain specific knowledge from the PSV table into the learning process. Additionally, two new loss functions are introduced to mitigate the risk of mode collapse, particularly when the model struggles to discover molecules with higher scores than those already present in the memory pool. The final loss is a composite of the six sub-losses mentioned above, and it is used to perform backpropagation, thereby completing one epoch of the PSV-PPO learning process.

PSV-Driven Entropy Loss

The PSV-Driven Entropy Loss leverages the insights provided by the Partial SMILES Validation Grid (PSV table) to refine the traditional entropy regularization term. Traditional entropy regularization encourages exploration by maximizing the entropy of the action distribution, but it does so indiscriminately, potentially guiding the model toward chemically invalid states. The PSV-Driven Entropy Loss restricts exploration to tokens that lead to valid SMILES substrings while preventing the model from favoring actions with larger valid token sets.

LPSVENTROPY(θ)=−βEt[∑a∈DPSVπθ(a∣st)log⁡πθ(a∣st)log⁡(len⁡(DPSV))](7)L_{PSV}^{\mathrm{ENTROPY}}(\theta)=-\beta\mathbb{E}_{t}\left[\sum_{a\in D_{PSV}}\frac{\pi_{\theta}(a\mid s_t)\log\pi_{\theta}(a\mid s_t)}{\log(\operatorname{len}(D_{PSV}))}\right] \tag*{(7)}

Here, the PSV-valid vocabulary DPSVD_{PSV} consists of tokens validated by the PSV table, given the partially generated SMILES string at state sts_t. The normalization factor, log⁡(len⁡(DPSV))\log(\operatorname{len}(D_{PSV})), ensures that the entropy loss remains independent of the size of the valid token set, preventing the model from being biased towards actions with larger valid token sets and maintaining balanced exploration. Our experiments show that without this normalization, the model tends to converge prematurely to sub-optimal states, particularly favoring tokens associated with larger valid action spaces, such as the non-aromatic carbon token ”C”. By normalizing the entropy loss, the exploration process is guided by chemical validity rather than token set size, fostering more robust and diverse molecule generation.

PSV-Driven Hellinger Distance Loss

While the PSV-Driven Entropy Loss encourages the selection of valid tokens, it does not explicitly penalize the model for considering invalid tokens. To address this, we adapt the Hellinger Distance (HD) loss (Hellinger, 1909; Arjovsky et al., 2017). Traditionally, KL divergence is used to measure the difference between the current policy πθ\pi_{\theta} and the prior policy πθold\pi_{\theta_{\mathrm{old}}}, regularizing the learning process. However, KL divergence is incompatible with πθold,PSV\pi_{\theta_{\mathrm{old,PSV}}} because it contains zero probabilities introduced by the PSV table.

In our approach, we replace KL divergence with the Hellinger Distance, which is better suited for cases where the prior policy’s output is filtered through the PSV table, excluding invalid tokens. The PSV-Driven Hellinger Distance Loss is defined as:

LPSVHD(θ)=Et[HD⁡(πθold,PSV(⋅∣st)∥πθ(⋅∣st))](8)L_{PSV}^{\mathrm{HD}}(\theta)=\mathbb{E}_{t}\left[\operatorname{HD}\left(\pi_{\theta_{\mathrm{old,PSV}}}(\cdot\mid s_t)\parallel\pi_{\theta}(\cdot\mid s_t)\right)\right] \tag*{(8)}

Here, πθold,PSV\pi_{\theta_{\mathrm{old,PSV}}} represents the prior policy’s distribution filtered by the PSV table, ensuring that only valid actions are considered. This targeted penalty encourages the model not only to favor valid tokens but also to learn from and correct potential errors, promoting the generation of valid SMILES strings while maintaining balanced exploration of the chemical space.

TasksSMILES GASMILES LSTMGraph GAReinventGEGLMolRL-MGPTPSV-PPO
C11_{11}H24_{24}0.8290.9930.9710.9991.0001.0001.000
C9_9H10_{10}N2_2O2_2P2_2Cl0.8890.8790.9820.8771.0000.9391.000
Median molecules 10.3340.4380.4060.4340.4550.4490.459
Median molecules 20.3800.4220.4320.3950.4370.4220.392
Osimertinib MPO0.8860.9070.9530.8891.0000.9770.951
Fexofenadine MPO0.9310.9590.9981.0001.0001.0001.000
Ranolazine MPO0.8810.8550.9200.8950.9330.9390.936
Perindopril MPO0.6610.8080.7920.7640.8330.8100.849
Amlodipine MPO0.7220.8940.8940.8880.9050.9060.908
Sitagliptin MPO0.6890.5450.8910.5390.7490.8230.804
Zaleplon MPO0.4130.6690.7540.5900.7630.7900.764
Valsartan SMARTS0.5520.9780.9900.0951.0000.9970.999

Table 1. Scores of PSV-PPO and other baseline models on the GuacaMol benchmark.

Final Loss Function

The final loss function integrates these key components into the overall optimization process:

Loss=LCLIP(θ)+LValue(θ)+LPSVENTROPY(θ)+LPSVHD(θ)+LPSVTPC(θ)+LPSVGPS(θ)(9)\begin{aligned} Loss &= L^{\mathrm{CLIP}}(\theta) + L^{\mathrm{Value}}(\theta) + L^{\mathrm{ENTROPY}}_{\mathrm{PSV}}(\theta) \\ &\quad+ L^{\mathrm{HD}}_{\mathrm{PSV}}(\theta) + L^{\mathrm{TPC}}_{\mathrm{PSV}}(\theta) + L^{\mathrm{GPS}}_{\mathrm{PSV}}(\theta) \tag*{(9)} \end{aligned}

where LPSVTPC(θ)L^{\mathrm{TPC}}_{\mathrm{PSV}}(\theta) and LPSVGPS(θ)L^{\mathrm{GPS}}_{\mathrm{PSV}}(\theta) are additional regularization terms designed to prevent mode collapse. Detailed explanations can be found in the Appendix.

Experimental Results

PMO Benchmark

The PMO benchmark (Gao et al., 2022) evaluates molecular generation models using the area under the curve (AUC) of the top-KK average property value versus the number of oracle calls. Following previous studies, we report results for K=10K = 10. To ensure fair evaluation, we limit the number of oracle calls to 10, 000 and allow early stopping for HC and PPO-based methods to minimize the total number of oracle calls required to achieve the best scores. We implemented PSV-PPO on the pretrained LSTM model provided by the benchmark, named LSTM PSV-PPO. The pretrained LSTM model consists of 6 layers with 512 hidden dimension, and a dropout rate of 0.2. The cross entropy loss achieved by the pretrained model is 0.407.

We compare our method with Reinvent, LSTM Hill Climbing (LSTM HC), LSTM PPO and SMILES PSV-PPO. LSTM HC is an iterative learning method that fine-tunes a generative LSTM model by incorporating high-scoring molecules into the training set at each iteration. As a variant of the cross-entropy method and REINFORCE with specific reward shaping, LSTM HC adaptively refines the molecular distribution represented in SMILES strings to improve molecule generation.

Table 2 presents the AUC-Top10 results across multiple PMO tasks. We observe that LSTM PSV-PPO consistently

TaskREINVENTLSTM HCLSTM PPOLSTM PSV-PPO
albuterol_similarity0.882±0.0060.719±0.0180.527±0.0140.761±0.007
amlodipine_mpo0.635±0.0350.593±0.0160.587±0.0080.647±0.007
celecoxib_rediscovery0.713±0.0670.539±0.0180.532±0.0410.612±0.021
deco_hop0.666±0.0440.826±0.0170.761±0.0080.802±0.001
drd20.945±0.0070.919±0.0150.883±0.0120.959±0.013
fexofenadine_mpo0.784±0.0060.725±0.0030.695±0.0030.698±0.001
gsk3b0.865±0.0430.839±0.0150.794±0.0230.869±0.089
isomers_c9h10n2o2pf2cl0.642±0.0540.342±0.0270.608±0.0010.652±0.018

Table 2. Performance Comparison of Different Methods on PMO Tasks (AUC-Top10)

outperforms LSTM PPO across all reported tasks, demonstrating that incorporating Partial SMILES Validation significantly improves molecular generation. Notably, the improvement from LSTM PPO to LSTM PSV-PPO is particularly meaningful, as the performance of PPO-based models is strongly influenced by the quality of the pre-trained model. These findings emphasize the effectiveness of PSV-PPO in stabilizing training and enhancing exploration, enabling it to generate higher-quality molecules while maintaining strong validity constraints. For completeness, the full table of results is provided in the appendix.

Docking

Molecular docking provides an essential metric for evaluating the effectiveness of generative models in producing biologically relevant molecules. We evaluate our approach on docking tasks for a well-studied protein target: fa7Yang et al., [28]. The docking performance is assessed using multiple metrics, including AUC-1, AUC-10, AUC-100, synthetic accessibility (SA), and diversity (Div).

To perform docking, we utilize the PyScreener framework with QVina2 as the docking backend, which enables efficient and accurate binding affinity predictions. As shown in Figure 4, PSV-PPO consistently achieves higher scores than PPO and HC across all evaluated metrics. This result demonstrates the effectiveness of PSV-PPO in optimizing molecular docking performance.

Image not extracted

Figure 4. Docking Performance: Comparison of molecular docking performance on the fa7 target across different reinforcement learning methods. The plot presents scores for Top 1, Top 10, Top 100, and Diversity (Div.), evaluating HC (green), PPO (blue), and PSV-PPO (orange). PSV-PPO achieves competitive performance, demonstrating its effectiveness in generating diverse and high-quality molecules.

GuacaMol Benchmark

Pretraining

We pre-trained our model on the ChEMBL dataset (Gaulton et al., 2012), available through the Guacamol benchmark (Brown et al., 2019). For this, we fine-tuned a pre-trained GPT-2 model (Lagler et al., [12]) from Hugging Face (Wolf et al., [25]) for five epochs. A linear warmup scheduler was utilized with a maximum learning rate of 0.0001. The atomwise tokenizer is provided by the SPE package (Li

[13]. To meet the requirement of atomwise tokenizer, the final fully connected layer of the model was resized to align with the 48-token vocabulary of our atom-wise tokenizer, which includes tokens for beginning-of-sequence (BOS), end-of-sequence (EOS), and padding (PAD) tokens. The GPT-2 model used contains 1.5 billion trainable parameters across 12 transformer layers, each with an embedding size of 768.

We set the maximum SMILES sequences length to 100 tokens, though this can be adjusted per task to optimize GPU memory usage. Pretraining was conducted on a single NVIDIA A100 GPU for 8 hours. Upon completion, the pre-trained model achieved a validation rate of 0.9982 and a uniqueness rate of 0.998 across 30,000 generated samples. This metrics align with typical outcomes for pretrained models [18].

Benchmark Tasks

The GuacaMol benchmark is a comprehensive, open-source evaluation framework designed to assess the performance of de novo molecular design algorithms. It features 20 goal-directed tasks that closely simulate real-world drug discovery challenges. These tasks are categorized into various objectives commonly pursued in drug design, such as drug rediscovery, structural similarity, physicochemical property optimization, etc. The tasks assess the models’ ability to generate molecules with specific structural and physicochemical properties or to optimize multiple objectives simultaneously. The description of baseline models are explained in Appendix. The performance of our models

Image not extracted

Figure 5. Illustration of the top-scoring molecules generated by PSV-PPO for tasks 11-20 on the GuacaMol benchmark.

was evaluated using the standard scoring functions provided by the GuacaMol benchmark. For each task, the models was trained for up to 500 epochs, with an episode size of 512 and a batch size of 256. Notably, for the Valsartan SMARTS task, we observed that the initial molecule scores was zero, which could lead to aimless exploration by the model. To mitigate this, we implemented a warm-up sampling strategy. In this phase, the pretrained model was employed for generation until at least one molecule with a score greater than 0.01 was successfully discovered and added to the experience replay memory pool. Furthermore, we set a maximum tokenized SMILES sequence length of 100 tokens. Any sequences that did not produce an EOS token before reaching this length limit were discarded during generation.

Benchmark Results

The benchmark results in Table 4 follow the results reported in MoRL-MGPT (Hu et al., 2024), allowing for direct comparison between our method and the baseline models, which are: (1) SMILES GA (Yoshikawa et al., [29]), (2) SMILES LSTM (Segler et al., [22]), (3) Graph GA (Jensen, [1]), (4) Reinvent (Olivecrona et al., [17]), (5) GEGL (Ahn et al., 2020), and (6) MoRL-MGPT. We observe that PSV-PPO outperforms existing baselines and achieves 14 out of 20 highest scores among 20 tasks, including 11 tasks that our method achieves 1.00. This demonstrates the effectiveness of our methods. In Figure C.1, we present the molecule with highest score from task 11 to task 20 with corresponding scores for each task. See Appendix for full illustrations of top 20 molecules and the diversity of generated molecules.

TasksSMILES GASMILES LSTMGraph GAReinventGEGLMolRL-MGPTPSV-PPO
1. Celecoxib rediscovery0.7321.0001.0001.0001.0001.0001.000
2. Troglitazone rediscovery0.5151.0001.0001.0000.5521.0001.000
3. Thiothixene rediscovery0.5981.0001.0001.0001.0001.0001.000
4. Aripiprazole similarity0.8341.0001.0001.0001.0001.0001.000
5. Albuterol similarity0.9071.0001.0001.0001.0001.0001.000
6. Mestranol similarity0.7901.0001.0001.0001.0001.0001.000
7. C11H24C_{11}H_{24}0.8290.9930.9710.9991.0001.0001.000
8. C9H10N2O2P2ClC_{9}H_{10}N_{2}O_{2}P_{2}Cl0.8890.8790.9820.8771.0000.9391.000
9. Median molecules 10.3340.4380.4060.4340.4550.4490.459
10. Median molecules 20.3800.4220.4320.3950.4370.4220.392
11. Osimertinib MPO0.8860.9070.9530.8891.0000.9770.951
12. Fexofenadine MPO0.9310.9590.9981.0001.0001.0001.000
13. Ranolazine MPO0.8810.8550.9200.8950.9330.9390.936
14. Perindopril MPO0.6610.8080.7920.7640.8330.8100.849
15. Amlodipine MPO0.7220.8940.8940.8880.9050.9060.908
16. Sitagliptin MPO0.6890.5450.8910.5390.7490.8230.804
17. Zaleplon MPO0.4130.6690.7540.5900.7630.7900.764
18. Valsartan SMARTS0.5520.9780.9900.0951.0000.9970.999
19. Deco hop0.9700.9961.0000.9941.0001.0001.000
20. Scaffold hop0.8850.9981.0000.9901.0001.0001.000
Total14.39617.34017.98316.35017.62718.05218.061

Table 4. Scores of PSV-PPO and other baseline models on the GuacaMol benchmark.

Ablation Study

To better understand the contribution of different components in PSV-PPO, we conducted ablation studies focusing on the use of PSV table, and the GPS and TPC losses. The results of these studies for the Fexofenadine MPO and Osimertinib MPO tasks are illustrated in Figure 3.

Image not extracted

Figure 3. Ablation study results for the Fexofenadine MPO and Osimertinib MPO tasks, illustrating the impact of various components within the PSV-PPO framework. The plots compare PSV-PPO with two ablation models: PSV-PPO_WO_PSV, which uses the standard PPO entropy and KL divergence losses without integrating the PSV table, and PSV-PPO_WO_PL, which excludes the GPS and TPC losses. The score plots demonstrate the optimization effectiveness of each model across epochs. The valid rate plots reveal how the absence of the PSV table significantly reduces the validity of generated molecules. The duplication plots in both the experience replay memory and during generation highlight the role of GPS and TPC losses in maintaining diversity and preventing mode collapse in the generated molecules.

Contribution from the PSV Table

The inclusion of the PSV table is critical for maintaining the validity rate of the generated molecules. As shown in the second column of Figure 3, without the PSV table, the valid rate drastically drops below 10% after a certain number of epochs, which indicates that the model frequently generates invalid SMILES strings. Moreover, we also observes that the policy loss starts to show high peaks and eventually end up with gradient explosion and caused all parameters become ”nan”.

Effect of Token Probability Control

The GPS and TPC losses help prevent mode collapse by penalizing overconfident token selections. The Duplicated in Experience and Duplicated in Generation plots (third and fourth columns in Figure 3) demonstrate that models without this control mechanism tend to generate more duplicates. This indicates a lack of diversity in the generated molecules, which directly impacts the effectiveness of the exploration process. In summary, the ablation studies confirm that each component of PSV-PPO contributes significantly to the overall performance. The PSV table ensures high validity, length normalization controls sequence length, and token probability control enhances diversity, all of which are crucial for effective and efficient molecular generation.

Conclusion

In this study, we introduced PSV-PPO, an innovative variant of the Proximal Policy Optimization (PPO) algorithm, specifically designed to address the challenges of reinforcement learning in SMILES-based deep generative models. By incorporating a partial SMILES validation scheme directly into the reinforcement learning process, PSV-PPO achieves outstanding performance on the GuacaMol benchmark, and shows improvement from PPO in all tasks on the PMO benchmark. Our ablation studies underscore the importance of each component within the framework, highlighting the critical role of the PSV-PPO in enhancing exploration while maintaining stability and validity.

The Partial SMILES Validation scheme is versatile and independent of the specific task at hand, making it compatible with any sequential model that utilizes the SMILES-based auto-regressive generation design. This adaptability suggests that PSV-PPO could have broad applications across various areas of de novo drug discovery, offering a significant advancement in the development of more reliable and effective generative models.

Technical and Implementation Details

Algorithm 1 Partial Validity Table Generator from SMILES Actions

Input: Action Sample AA, Vocabulary Decoder DD, Index of First EOS idxeosidx_{eos}, EOS Token tokeostok_{eos}, Token ID idtokid_{tok}

Output: PSV Map PSVPSV

  1. Initialize PSVPSV as a zero matrix of size ∣A∣×∣D∣|A| \times|D|.

  2. Set smicurrent=∅smi_{current} = \emptyset.

  3. Set indexinvalid=−1index_{invalid} = -1.

  4. for each i,tokenactioni, token_{action} in enumerate A[:idxeos+1]A[: idx_{eos} + 1] do

  5.     for each j,tokenvocabj, token_{vocab} in enumerate DD do

  6.         mol=Nonemol = None

  7.         if tokenvocab==tokeostoken_{vocab} == tok_{eos} then

  8.             if i>10i > 10 then

  9.                 Continue

  10.             end if

  11.             mol=CheckValidCompleted(smicurrent)mol = CheckValidCompleted(smi_{current})

  12.         else

  13.             mol=CheckValidOnTheFly(smicurrent+tokenvocab)mol = CheckValidOnTheFly(smi_{current} + token_{vocab})

  14.         end if

  15.         if mol≠Nonemol \ne None then

  16.             Set PSV[i,j]=1PSV[i,j] = 1

  17.         else

  18.             Set PSV[i,j]=0PSV[i,j] = 0

  19.             if tokenaction==tokenvocabtoken_{action} == token_{vocab} then

  20.                 Set indexinvalid=iindex_{invalid} = i

  21.             end if

  22.         end if

  23.     end for

  24.     if indexinvalid≠−1index_{invalid} \ne-1 then

  25.         if ∑PSV[i,:]==0\sum PSV[i,:] == 0 then

  26.             Set PSV[i,idtok]=1PSV[i,id_{tok}] = 1

  27.             return PSV,indexinvalidPSV,index_{invalid}

  28.         end if

  29.     end if

  30.     if tokenaction==tokeostoken_{action} == tok_{eos} then

  31.         return PSV,indexinvalidPSV,index_{invalid}

  32.     end if

  33.     Append tokenactiontoken_{action} to smicurrentsmi_{current}

  34. end for

  35. return PSV,indexinvalidPSV,index_{invalid}

PSV Table Generator Algorithm

The algorithm presented in Algorithm 1 outlines the process for generating a PSV table. The algorithm takes as input an action sample AA, a vocabulary decoder DD, the index of the first EOS token idxeosidx_{eos}, the EOS token itself tokeostok_{eos}, and a token ID idtokid_{tok}. The output is the PSV table, denoted as PSVPSV, which is a matrix that captures the validity of each potential action at every step of the SMILES generation process.

The algorithm starts by initializing the PSV table as a zero matrix. A string variable smicurrentsmi_{current} holds the constructed SMILES string, and indexinvalidindex_{invalid} tracks the first invalid index, initialized to -1. For each token in AA up to the EOS index, the algorithm checks the validity of appending each token from the vocabulary to smicurrentsmi_{current}. If the token is valid, the PSV entry is set to 1; otherwise, it is set to 0, and indexinvalidindex_{invalid} is updated if the invalid token matches the action token. If an entire row in the PSV table is invalid, the entry corresponding to idtokid_{tok} is forced to 1, and the algorithm returns the PSV table and the invalid index.

If the resulting molecular structure is valid, the corresponding entry in the PSV table is set to 1. If it is invalid, the entry is set to 0, and the index of the invalid action is recorded if the current action token matches the invalid vocabulary token. The algorithm then checks if all entries in the current row of the PSV table are zero. If so, it forces a valid action by setting the entry corresponding to idtokid_{tok} to 1 and returns the PSV table along with the invalid index. The loop continues until the EOS token is reached, at which point the PSV table and invalid index are returned.

This algorithm ensures that at each step of the SMILES generation process, the validity of the generated string is maintained by providing immediate feedback on the feasibility of each token addition, thereby guiding the model toward generating valid molecular structures.

Adjusted Reward for Invalid SMILES Samples

In reinforcement learning, delayed rewards can hinder the model’s ability to associate specific actions with their outcomes. In the context of SMILES generation, assigning a negative reward only at the EOS token can obscure the source of invalidity, making it difficult for the model to learn which specific action caused the issue. To address this, we adjust the reward mechanism by penalizing the first invalid action as soon as it occurs, rather than waiting until the EOS token.

As detailed in Algorithm 1, if indexinvalid≠−1index_{invalid} \ne-1, a score of −1-1 is assigned immediately at this position, providing clear feedback to the model about the exact point of failure. This approach contrasts with traditional methods that apply penalties at the EOS token, thus allowing the model to quickly learn and avoid invalid token sequences.

The implementation of this adjusted reward system involves:

  1. Identifying the Invalid Index: After generating the PSV Table, if indexinvalidindex_{invalid} is not -1, it indicates where the SMILES string became invalid.

  1. Assigning Immediate Penalty: A score of -1 is placed at indexinvalidindex_{invalid}, rather than at the EOS token, ensuring that the model is penalized at the exact moment of invalidity.

  1. Standard Reward Handling: If no invalid index is found (indexinvalid=−1index_{invalid} = -1), reward assignment follows the usual protocol based on the task-specific reward function.

This method provides immediate, clear feedback, helping the model better understand and correct errors in SMILES generation. By correcting invalid steps as soon as they occur, the model improves in generating valid molecular structures, making the learning process more effective overall.

Loss Term Explanation

Global Probability Stabilization Loss (GPS Loss)

During training, we observe that the model tends to assign increasingly high probabilities to specific molecular structures that yield high rewards. This behavior can lead to the generation of repetitive molecules, reducing diversity and diminishing the utility of the generated samples across epochs. To mitigate this issue, we introduce the Global Probability Stabilization Loss (GPS Loss). This loss penalizes the model when the cumulative probability of generating a particular sequence exceeds a predefined threshold, preventing the model from becoming overly biased toward a narrow subset of molecules.

The loss is computed as follows:

LPSVGPS(θ)=12(max⁡(∏t=1Tπθ(at∣st),τgps)−τgps)2(10)L^{GPS}_{PSV}(\theta) = \frac{1}{\sqrt{2}}\left(\sqrt{\max\left(\prod_{t=1}^{T}\pi_{\theta}(a_t|s_t),\tau_{gps}\right)}-\sqrt{\tau_{gps}}\right)^2 \tag*{(10)}

where:

  • ∏t=1Tπθ(at∣st)\prod_{t=1}^{T}\pi_{\theta}(a_t|s_t) represents the cumulative probability of the sequence generated by the model.

  • τgps\tau_{gps} is the cumulative probability threshold (e.g., 1×10−51 \times10^{-5}).

  • max⁡(⋅,τgps)\max(\cdot,\tau_{gps}) ensures that the cumulative probability is not penalized below the threshold.

This loss term encourages the model to maintain balanced exploration of the chemical space by discouraging excessive focus on any single molecular structure. By doing so, we aim to enhance the diversity of generated molecules and improve the overall quality of samples produced during training.

Token Probability Control Loss (TPC Loss)

To ensure the diversity of generated molecules and prevent the model from overcommitting to specific tokens, we introduce the Token Probability Control Loss (TPC Loss). This loss penalizes action token probabilities that exceed a predefined threshold τtoken\tau_{token}, thereby discouraging the model from assigning disproportionately high probabilities to certain molecular fragments. The loss is computed as follows:

LPSVTPC(θ)=12∑t=1T(∑a∈At[πθ(a∣st)−τtoken]2⋅1∣PPSV(st)∣⋅I(πθ(a∣st)>τtoken))(11)L_{PSV}^{\mathrm{TPC}}(\theta)=\frac{1}{\sqrt{2}}\sum_{t=1}^{T}\left(\sum_{a\in A_t}\left[\sqrt{\pi_\theta(a|s_t)}-\sqrt{\tau_{token}}\right]^2\cdot\frac{1}{|P_{PSV}(s_t)|}\cdot\mathbb{I}\left(\pi_\theta(a|s_t)>\tau_{token}\right)\right) \tag*{(11)}

where:

  • πθ(a∣st)\pi_\theta(a|s_t) is the probability assigned to action aa given state sts_t.

  • τtoken\tau_{token} is the predefined threshold for token probabilities.

  • ∣PPSV(st)∣|P_{PSV}(s_t)| represents the number of valid candidate tokens at state sts_t.

  • I(⋅)\mathbb{I}(\cdot) is an indicator function that activates the penalty only when πθ(a∣st)>τtoken\pi_\theta(a|s_t)>\tau_{token}.

This formulation accounts for the size of the valid token set directly, ensuring that the penalty is appropriately scaled according to the number of choices available at each state.

Length Normalization

During our experiments, we observed that when the model stabilizes at a certain score level for an extended period, it tends to generate progressively longer sequences in an effort to explore uncharted chemical space. While this exploration is generally beneficial, excessively long sequences are often less relevant and can detract from the quality of the generated molecules. To mitigate this issue, we implement a length normalization term on the entropy loss and the token probability loss:

LTLN=LT×exp⁡(−γLN×T)(12)L_T^{\mathrm{LN}}=L_T\times\exp(-\gamma_{LN}\times T) \tag*{(12)}

This adjustment reduces the influence of excessively long sequences, guiding the model towards generating more plausible and relevant molecular structures.

GuacaMol Results

Top-Scoring Molecules

We present visualizations of the top-scoring molecules generated by our model for osimertinib MPO and Amlodipine MPO. Notably, even when the top-20 molecules have very close scores, our results demonstrate that the generated molecules exhibit distinct structural differences. This contrasts with models that tend to produce molecules with similar structures when scores are close. Our model’s ability to generate structurally diverse molecules, even with minimal score variation, highlights its effectiveness in exploring different chemical spaces while maintaining high-quality outputs.

Full Benchmark Tables

Osimertinib MPO

Image not extracted

Figure 1 (PDF p. 15). Osimertinib MPO. Scores by row: 0.948, 0.948, 0.948, 0.948, 0.948; 0.949, 0.949, 0.949, 0.949, 0.950; 0.950, 0.950, 0.950, 0.950, 0.950; 0.950, 0.950, 0.951, 0.951, 0.951.

Amlodipine MPO

Image not extracted

Figure 2 (PDF p. 15). Amlodipine MPO. Scores by row: 0.906, 0.906, 0.906, 0.906, 0.906; 0.906, 0.906, 0.906, 0.906, 0.906; 0.906, 0.906, 0.906, 0.906, 0.906; 0.906, 0.906, 0.906, 0.906, 0.906, 0.908.

Task IDscoreSMILES
90.406CC1CCC(C(C)C2CCC(CC2=O)C3(C)C(C)C(O)C1
100.400CCOc1cc(S(=O)(=O)N2CCN(CC2)cc1C1Cc2c([nH]c3ccccc23)C2CCC3c(C2)OCO3)N1C
110.951COc1cc(NC(C)(NS(=O)(=O)N(S(=O)=O)n2oc(-c3ccnc4cccc34)nc(=O)c2=O)cc1N1C
121.000O=C(O)(C(O)(C)C(C)CCN2CCC(C(O)(c3ccccc3)C3CC(C(O)C2)cc1
130.936Cc1ccc(OCCCCCCOC(C)C2cco2)C1NC(=O)CN1CCN(C(C)(C)C2=O)C2CCS
140.849CCOC(=O)C(CCn1[nH]n2oc(C)c12)NC(C)C(=O)N1C(C(=O)O)CC2CCCCC21
150.908CCOC(=O)C1=C(COCCN)NC(C)=C(C(=O)OC)C1C1=NOC1c1ccccc1Cl
160.804FC#CCS(=O)(c1cnc[nH]1)N(CCc1ccnnc1)C1CC(F)(F)C1(F)F
170.764CCC#Cc1cnn2c(-c3cccc(C(O)C(C)=O)c3)ccnc12
180.999C([H])N(Cc1ccc(-c2ccccc2)cc1Cl)C(=O)CN1CCCNNN=CC1S(=O)
191.000C#CCCCC=C=C(CBr)c1ncsc1-c1ccccc1Nc1ncnc2cc3c(cc12)CC(C)(C)CO3
201.000CCC(C)=CCCOC(=O)c1cc2c(Nc3ccc4ncsc4c3)ncnc2cc1Br

Table 3. SMILES Data with Scores

TaskREINVENTLSTM HCLSTM PPOLSTM PSV-PPO
albuterol_similarity0.882±0.0060.882\pm0.0060.719±0.0180.719\pm0.0180.527±0.0140.527\pm0.0140.761±0.0070.761\pm0.007
amlodipine_mpo0.635±0.0350.635\pm0.0350.593±0.0160.593\pm0.0160.587±0.0080.587\pm0.0080.647±0.0070.647\pm0.007
celecoxib_rediscovery0.713±0.0670.713\pm0.0670.539±0.0180.539\pm0.0180.532±0.0410.532\pm0.0410.612±0.0210.612\pm0.021
deco_hop0.666±0.0440.666\pm0.0440.826±0.0170.826\pm0.0170761±0.0080761\pm0.0080.802±0.0010.802\pm0.001
drd20.945±0.0070.945\pm0.0070.919±0.0150.919\pm0.0150.883±0.0120.883\pm0.0120.959±0.0130.959\pm0.013
fexofenadine_mpo0.784±0.0060.784\pm0.0060.725±0.0030.725\pm0.0030.695±0.0030.695\pm0.0030.698±0.0010.698\pm0.001
gsk3b0.865±0.0430.865\pm0.0430.839±0.0150.839\pm0.0150.794±0.0230.794\pm0.0230.869±0.0890.869\pm0.089
isomers_c7h8n2o20.852±0.0360.852\pm0.0360.485±0.0450.485\pm0.0450.582±0.1340.582\pm0.1340.650±0.0310.650\pm0.031
isomers_c9h10n2o2pf2cl0.642±0.0540.642\pm0.0540.342±0.0270.342\pm0.0270.608±0.0010.608\pm0.0010.652±0.0180.652\pm0.018
jnk30.783±0.0230.783\pm0.0230.661±0.0390.661\pm0.0390.567±0.0200.567\pm0.0200.681±0.0110.681\pm0.011
median10.356±0.0090.356\pm0.0090.255±0.0100.255\pm0.0100.204±0.0040.204\pm0.0040.193±0.0030.193\pm0.003
median20.276±0.0080.276\pm0.0080.248±0.0080.248\pm0.0080.226±0.0100.226\pm0.0100.239±0.0370.239\pm0.037
mestranol_similarity0.618±0.0480.618\pm0.0480.526±0.0320.526\pm0.0320.440±0.0120.440\pm0.0120.560±0.0130.560\pm0.013
osimertinib_mpo0.837±0.0090.837\pm0.0090.796±0.0020.796\pm0.0020.777±0.1530.777\pm0.1530.844±0.0080.844\pm0.008
perindopril_mpo0.537±0.0160.537\pm0.0160.489±0.0070.489\pm0.0070.461±0.0190.461\pm0.0190.452±0.0020.452\pm0.002
qed0.941±0.0000.941\pm0.0000.939±0.0000.939\pm0.0000.938±0.0030.938\pm0.0030.938±0.0510.938\pm0.051

Table 5. Performance Comparison of Different Methods on PMO Tasks (AUC-Top10)

References

  1. [1]Jensen, J. H. A graph-based genetic algorithm and generative model/monte carlo tree search for the exploration of chemical space. Chemical science, 10(12):3567–3572, 2019.DOI
  2. [2]Jin, W., Barzilay, R., and Jaakkola, T. Junction tree variational autoencoder for molecular graph generation. In International conference on machine learning, pp. 2323–2332. PMLR, 2018.arxiv.org/abs/1802.04364
  3. [3]Jin, W., Barzilay, R., and Jaakkola, T. Hierarchical generation of molecular graphs using structural motifs. In International conference on machine learning, pp. 4839–4848. PMLR, 2020.arxiv.org/abs/2002.03230
  4. [4]Kearnes, S., McCloskey, K., Berndl, M., Pande, V., and Riley, P. Molecular graph convolutions: moving beyond fingerprints. Journal of computer-aided molecular design, 30:595–608, 2016.DOI
  5. [5]Kemker, R., McClure, M., Abitino, A., Hayes, T., and Kanan, C. Measuring catastrophic forgetting in neural networks. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.arxiv.org/abs/1708.02072
  6. [6]Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Desjardins, G., Rusu, A. A., Milan, K., Quan, J., Ramalho, T., Grabska-Barwinska, A., et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):3521–3526, 2017.DOI
  7. [7]Korshunova, M., Huang, N., Capuzzi, S., Radchenko, D. S., Savvich, O., Moroz, Y. S., Wells, C. I., Willson, T. M., Tropsha, A., and Isayev, O. Generative and reinforcement learning approaches for the automated de novo design of bioactive compounds. Communications Chemistry, 5(1):129, 2022.DOI
  8. [8]Kostrikov, I. Pytorch implementations of reinforcement learning algorithms, 2018.
  9. [9]Krenn, M., Ai, Q., Barthel, S., Carson, N., Frei, A., Frey, N. C., Friederich, P., Gaudin, T., Gayle, A. A., Jablonka, K. M., et al. Selfies and the future of molecular string representations. Patterns, 3(10), 2022.
  10. [10]Kullback, S. and Leibler, R. A. On information and sufficiency. The annals of mathematical statistics, 22(1):79–86, 1951.DOI
  11. [11]Kusner, M. J., Paige, B., and Hernández-Lobato, J. M. Grammar variational autoencoder. In International conference on machine learning, pp. 1945–1954. PMLR, 2017.arxiv.org/abs/1703.01925
  12. [12]Lagler, K., Schindlegger, M., Böhm, J., Krásná, H., and Nilsson, T. Gpt2: Empirical slant delay model for radio space geodetic techniques. Geophysical research letters, 40(6):1069–1073, 2013.pubmed.ncbi.nlm.nih.gov/25821263
  13. [13]Li, X. and Fourches, D. Smiles pair encoding: a data-driven substructure tokenization algorithm for deep learning. Journal of chemical information and modeling, 61(4):1560–1569, 2021.DOI
  14. [15]Mokaya, M., Imrie, F., van Hoorn, W. P., Kalisz, A., Bradley, A. R., and Deane, C. M. Testing the limits of smiles-based de novo molecular generation with curriculum and deep reinforcement learning. Nature Machine Intelligence, 5(4):386–394, 2023.DOI
  15. [16]O’Boyle, N. partialsmiles, version 1.0. https://github.com/baoilleach/partialsmiles, 2024.
  16. [17]Olivecrona, M., Blaschke, T., Engkvist, O., and Chen, H. Molecular de-novo design through deep reinforcement learning. Journal of cheminformatics, 9:1–14, 2017.
  17. [18]Polykovskiy, D., Zhebrak, A., Sanchez-Lengeling, B., Golovanov, S., Tatanov, O., Belyaev, S., Kurbanov, R., Artamonov, A., Aladinskiy, V., Veselov, M., et al. Molecular sets (moses): a benchmarking platform for molecular generation models. Frontiers in pharmacology, 11:565644, 2020.DOI
  18. [19]Ramsundar, B., Eastman, P., Walters, P., Pande, V., Leswing, K., and Wu, Z. Deep Learning for the Life Sciences. O’Reilly Media, 2019. https://www.amazon.com/Deep-Learning-Life-Sciences-Microscopy/dp/1492039837.
  19. [20]Schulman, J., Levine, S., Abbeel, P., Jordan, M., and Moritz, P. Trust region policy optimization. In International conference on machine learning, pp. 1889–1897. PMLR, 2015.arxiv.org/abs/1502.05477
  20. [21]Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.arxiv.org/abs/1707.06347
  21. [22]Segler, M. H., Kogej, T., Tyrchan, C., and Waller, M. P. Generating focused molecule libraries for drug discovery with recurrent neural networks. ACS central science, 4(1):120–131, 2018.DOI
  22. [23]Shi, Z., Chen, X., Qiu, X., and Huang, X. Toward diverse text generation with inverse reinforcement learning. arXiv preprint arXiv:1804.11258, 2018.arxiv.org/abs/1804.11258
  23. [24]Weininger, D. Smiles, a chemical language and information system. 1. introduction to methodology and encoding rules. Journal of chemical information and computer sciences, 28(1):31–36, 1988.
  24. [25]Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Funtowicz, M., et al. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations, pp. 38–45, 2020.DOI
  25. [28]Yang, S., Hwang, D., Lee, S., Ryu, S., and Hwang, S. J. Hit and lead discovery with exploratory rl and fragment-based molecule generation. Advances in Neural Information Processing Systems, 34:7924–7936, 2021.arxiv.org/abs/2110.01219
  26. [29]Yoshikawa, N., Terayama, K., Sumita, M., Homma, T., Oono, K., and Tsuda, K. Population-based de novo molecule generation, using grammatical evolution. Chemistry Letters, 47(11):1431–1434, 2018.DOI

Paper details

vlnc:W_rx4rtvye
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 6

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents