Molecular Assays Simulator to Unravel Predictors Hacking in Goal-Directed Molecular Generations.

Abstract

Generative models are being increasingly used in drug discovery, very often coupled with absorption, distribution, metabolism, and excretion (ADME) bioassays or quantitative structure-activity relationship (QSAR) models to optimize a given set of properties. The molecules proposed by these algorithms are often revealed to be false positives; that is, they are predicted to be active and turn out to be inactive after synthesis and testing, mostly due to overoptimization of the predicted scores, which leads to an actual decrease or stagnation of the real scores. This behavior is also known as the "hacking" of the predictive models by the generative model during the optimization step. This issue is reminiscent of adversarial examples in machine learning and it can be seen as enunciated by Goodhart's law: "when a measure becomes a target, it ceases to be a good measure." This issue is even more apparent in a multiparameter optimization (MPO) case, where the models need to extrapolate outside the training set distribution because there are no known molecules satisfying all the objectives simultaneously in the initial training set. Experimental evaluation of this problem is a hard and expensive task since it requires synthesis and testing of the generated molecules. Thus, efforts have been made to develop in silico "oracles"─real-valued functions used as proxies for molecular properties─to help with the evaluation of these generative-model-based pipelines. However, these oracles have had a limited value so far because they are often too easy to model in comparison with biological assays and are usually limited to mono-objective cases. In this work, we introduce a simulator of multitarget assays using a smartly initialized neural network (NN) that returns continuous values for any input molecule. We use this oracle to replicate a real-world prospective lead optimization (LO) scenario. First, we trained predictive models on an initial small sample of molecules aimed at predicting their oracle values. Afterward, we generated new optimized molecules using the open-source GuacaMol package coupled with the previously built predictive models. Finally, we selected compounds matching the candidate drug target profile (CDTP) according to the predicted values and evaluated them by computing the true oracle values. We observed that even when the predictive models had excellent estimated performance metrics, the final selection still contained multiple false positives according to the NN-based oracle. Then, we evaluated the optimization behavior in mono- and bi-objective scenarios using either a logistic regression or a random forest predictive model. We also propose and evaluate several methods to help mitigate the hacking issue.