Goal-directed molecule generation with fine-tuning by policy gradient

Abstract

Graph-structured drug molecule representations often struggle to generate molecules by particular intentions, which results in generated molecules without pharmacological properties. To address the problem, we propose a de novo molecular generation method that utilizes the policy gradient algorithm of reinforcement learning to fine-tune the molecular graph generation model. The training process of the method is divided into the pre-training stage and the fine-tuning stage. During the pre-training stage, it uses graph neural networks and multilayer perceptrons to train a molecule graph generation model. During the fine-tuning stage, scoring functions are devised for multiple goal-directed generation tasks, and subsequently, the policy loss function is formulated based on the reward shaping mechanism. A value network is designed to calculate the value of taking an action based on the current graph state during agent sampling to guide policy updates. To mitigate the issue of molecular uniqueness decline during the learning process, we dynamically adjust the weights of the two learning processes in the policy loss function, aiming to generate desirable molecules with high probability and reduce the descent of uniqueness. The experiments showed that after fine-tuning, the generative model exhibits a higher probability of generating molecules with desired properties compared to other models. Furthermore, our method effectively mitigates the issue of molecular uniqueness declining during the learning process when compared to alternative fine-tuning methods.