Message Passing Neural Networks Research Articles

Message Passing Neural Networks Improve Prediction of Metabolite Authenticity.

Cytochrome P450 enzymes aid in the elimination of a preponderance of small molecule drugs, but can generate reactive metabolites that may adversely react with protein and DNA and prompt drug candidate attrition or market withdrawal. Previously developed models help understand how these enzymes modify molecule structure by predicting sites of metabolism or characterizing formation of metabolite-biomolecule adducts. However, the majority of reactive metabolites are formed by multiple metabolic steps, and understanding the progenitor molecule's network-level behavior necessitates an integrative approach that blends multiple site of metabolism and structure inference models. Our previously developed tool, XenoNet 1.0, generates metabolic networks, where nodes are molecules and weighted edges are metabolic transformations. We extend XenoNet with a bidirectional message passing neural network that integrates edge feature information and local network structure using edge-conditioned graph convolutions and jumping knowledge to predict the authenticity of inferred Phase I metabolite structures. Our model significantly outperformed prior work and algorithmic baselines on a data set of 311 networks and 6606 intermediates annotated using a chemically diverse set of 20 736 individual in vitro and in vivo reaction records accounting for 92.3% of all human Phase I metabolism in the Accelrys Metabolite Database. Cross-validated predictions resulted in area under the receiver operating characteristic curves of 88.5% and 87.6% for separating experimentally observed and unobserved metabolites at global and network levels, respectively. Further analysis verified robustness to networks of varying depth and breadth, accurate detection of metabolites, such as d,l-methamphetamine, that are experimentally observed or unobserved in different network contexts, extraction of important metabolic subnetworks, and identification of known bioactivation pathways, such as for nimesulide and terbinafine. By exploiting network structures, our approach accurately suggests unreported metabolites for experimental study and may rationalize modifications for avoiding deleterious pathways antecedent to reactive metabolite formation.

ABT-MPNN: an atom-bond transformer-based message-passing neural network for molecular property prediction

Graph convolutional neural networks (GCNs) have been repeatedly shown to have robust capacities for modeling graph data such as small molecules. Message-passing neural networks (MPNNs), a group of GCN variants that can learn and aggregate local information of molecules through iterative message-passing iterations, have exhibited advancements in molecular modeling and property prediction. Moreover, given the merits of Transformers in multiple artificial intelligence domains, it is desirable to combine the self-attention mechanism with MPNNs for better molecular representation. We propose an atom-bond transformer-based message-passing neural network (ABT-MPNN), to improve the molecular representation embedding process for molecular property predictions. By designing corresponding attention mechanisms in the message-passing and readout phases of the MPNN, our method provides a novel architecture that integrates molecular representations at the bond, atom and molecule levels in an end-to-end way. The experimental results across nine datasets show that the proposed ABT-MPNN outperforms or is comparable to the state-of-the-art baseline models in quantitative structure–property relationship tasks. We provide case examples of Mycobacterium tuberculosis growth inhibitors and demonstrate that our model's visualization modality of attention at the atomic level could be an insightful way to investigate molecular atoms or functional groups associated with desired biological properties. The new model provides an innovative way to investigate the effect of self-attention on chemical substructures and functional groups in molecular representation learning, which increases the interpretability of the traditional MPNN and can serve as a valuable way to investigate the mechanism of action of drugs.

Learning local equivariant representations for large-scale atomistic dynamics

A simultaneously accurate and computationally efficient parametrization of the potential energy surface of molecules and materials is a long-standing goal in the natural sciences. While atom-centered message passing neural networks (MPNNs) have shown remarkable accuracy, their information propagation has limited the accessible length-scales. Local methods, conversely, scale to large simulations but have suffered from inferior accuracy. This work introduces Allegro, a strictly local equivariant deep neural network interatomic potential architecture that simultaneously exhibits excellent accuracy and scalability. Allegro represents a many-body potential using iterated tensor products of learned equivariant representations without atom-centered message passing. Allegro obtains improvements over state-of-the-art methods on QM9 and revMD17. A single tensor product layer outperforms existing deep MPNNs and transformers on QM9. Furthermore, Allegro displays remarkable generalization to out-of-distribution data. Molecular simulations using Allegro recover structural and kinetic properties of an amorphous electrolyte in excellent agreement with ab-initio simulations. Finally, we demonstrate parallelization with a simulation of 100 million atoms.

Cooperative Lidar Sensing for Pedestrian Detection: Data Association Based on Message Passing Neural Networks

Cooperative Lidar Sensing for Pedestrian Detection: Data Association Based on Message Passing Neural Networks

Automatic identification of chemical moieties.

In recent years, the prediction of quantum mechanical observables with machine learning methods has become increasingly popular. Message-passing neural networks (MPNNs) solve this task by constructing atomic representations, from which the properties of interest are predicted. Here, we introduce a method to automatically identify chemical moieties (molecular building blocks) from such representations, enabling a variety of applications beyond property prediction, which otherwise rely on expert knowledge. The required representation can either be provided by a pretrained MPNN, or be learned from scratch using only structural information. Beyond the data-driven design of molecular fingerprints, the versatility of our approach is demonstrated by enabling the selection of representative entries in chemical databases, the automatic construction of coarse-grained force fields, as well as the identification of reaction coordinates.

Structure-based graph convolutional networks with frequency filter

• We design a latent space to correlate the underlying nodes for obtaining structured and meaningful messages. • We elaborate a structured adaptive frequency GNN by ingeniously combining the latent space nodes. • Comprehensively aggregating the vital components of neighbor nodes and underlying nodes. • We outperform the state-of-the-art methods significantly in terms of accuracy and computational efficiency Message-passing neural networks (MPNNs) have attracted a lot interesting in academia and industry and have been applied to various graph analytical problems of real-world applications and achieved prominent successes. For the most of current works on MPNNs, they mainly focus on two categories: spectral-based and spatial-based methods. The former seeks to distill useful information (e.g. low-pass and high-pass signals) and the later designs structural schemes. However, it is not enough to only utilize one of them. We tackle this drawback by proposing an efficient and elegant method of taking advantage of the structural graph neural network (GNN) for advanced spectral-based information filtering. Through the learnable frequency components and global virtual neighbor nodes, the proposed scheme is elaborated to obtain informative and structured messages of neighbor nodes and latent space correlational nodes. Experimental results show that our method outperforms the state-of-the-art performance in a series of datasets of graph tasks.

Deep Neural Message Passing With Hierarchical Layer Aggregation and Neighbor Normalization.

As a unified framework for graph neural networks, message passing-based neural network (MPNN) has attracted a lot of research interest and has been shown successfully in a number of domains in recent years. However, because of over-smoothing and vanishing gradients, deep MPNNs are still difficult to train. To alleviate these issues, we first introduce a deep hierarchical layer aggregation (DHLA) strategy, which utilizes a block-based layer aggregation to aggregate representations from different layers and transfers the output of the previous block to the subsequent block, so that deeper MPNNs can be easily trained. Additionally, to stabilize the training process, we also develop a novel normalization strategy, neighbor normalization (NeighborNorm), which normalizes the neighbor of each node to further address the training issue in deep MPNNs. Our analysis reveals that NeighborNorm can smooth the gradient of the loss function, i.e., adding NeighborNorm makes the optimization landscape much easier to navigate. Experimental results on two typical graph pattern-recognition tasks, including node classification and graph classification, demonstrate the necessity and effectiveness of the proposed strategies for graph message-passing neural networks.

Minimal data requirements for accurate compound activity prediction using machine learning methods of different complexity

Data sparseness is expected to negatively impact machine learning (ML) and especially deep learning (DL). In the era of DL, much emphasis is put on learning from large datasets. In cheminformatics, compound classification represents a suitable task for exploring characteristics of ML/DL. Here, we determine minimal data requirements for learning using activity-based compound classification as an exemplary application. Systematic predictions using message-passing neural networks, random forests, and simple controls are carried out on more than 100 activity classes using learning sets of increasing size. Contrary to expectations, predictive models are already obtained on the basis of “ultra-small” training sets containing only a few active compounds. These findings are rationalized by explainable ML, which reveals features that determine predictions. Accurate predictions often only depend on a few molecular features that might be detected even in ultra-small training sets. The findings revise current assumptions about data requirements for effective ML/DL.

Retention Time Prediction with Message-Passing Neural Networks

Retention time prediction, facilitated by advances in machine learning, has become a useful tool in untargeted LC-MS applications. State-of-the-art approaches include graph neural networks and 1D-convolutional neural networks that are trained on the METLIN small molecule retention time dataset (SMRT). These approaches demonstrate accurate predictions comparable with the experimental error for the training set. The weak point of retention time prediction approaches is the transfer of predictions to various systems. The accuracy of this step depends both on the method of mapping and on the accuracy of the general model trained on SMRT. Therefore, improvements to both parts of prediction workflows may lead to improved compound annotations. Here, we evaluate capabilities of message-passing neural networks (MPNN) that have demonstrated outstanding performance on many chemical tasks to accurately predict retention times. The model was initially trained on SMRT, providing mean and median absolute cross-validation errors of 32 and 16 s, respectively. The pretrained MPNN was further fine-tuned on five publicly available small reversed-phase retention sets in a transfer learning mode and demonstrated up to 30% improvement of prediction accuracy for these sets compared with the state-of-the-art methods. We demonstrated that filtering isomeric candidates by predicted retention with the thresholds obtained from ROC curves eliminates up to 50% of false identities.

Graph Convolutional Networks with Dual Message Passing for Subgraph Isomorphism Counting and Matching

Graph neural networks (GNNs) and message passing neural networks (MPNNs) have been proven to be expressive for subgraph structures in many applications. Some applications in heterogeneous graphs require explicit edge modeling, such as subgraph isomorphism counting and matching. However, existing message passing mechanisms are not designed well in theory. In this paper, we start from a particular edge-to-vertex transform and exploit the isomorphism property in the edge-to-vertex dual graphs. We prove that searching isomorphisms on the original graph is equivalent to searching on its dual graph. Based on this observation, we propose dual message passing neural networks (DMPNNs) to enhance the substructure representation learning in an asynchronous way for subgraph isomorphism counting and matching as well as unsupervised node classification. Extensive experiments demonstrate the robust performance of DMPNNs by combining both node and edge representation learning in synthetic and real heterogeneous graphs.

Constant Time Graph Neural Networks

The recent advancements in graph neural networks (GNNs) have led to state-of-the-art performances in various applications, including chemo-informatics, question-answering systems, and recommender systems. However, scaling up these methods to huge graphs, such as social networks and Web graphs, remains a challenge. In particular, the existing methods for accelerating GNNs either are not theoretically guaranteed in terms of the approximation error or incurred at least a linear time computation cost. In this study, we reveal the query complexity of the uniform node sampling scheme for Message Passing Neural Networks, including GraphSAGE, graph attention networks (GATs), and graph convolutional networks (GCNs). Surprisingly, our analysis reveals that the complexity of the node sampling method is completely independent of the number of the nodes, edges, and neighbors of the input and depends only on the error tolerance and confidence probability while providing a theoretical guarantee for the approximation error. To the best of our knowledge, this is the first article to provide a theoretical guarantee of approximation for GNNs within constant time. Through experiments with synthetic and real-world datasets, we investigated the speed and precision of the node sampling scheme and validated our theoretical results.

Graph neural networks for biochemistry that incorporate substructure

The success of neural networks has been considerable interest in applying neural networks to biological and chemical data. Often, this data is graph-like in nature: chemical structures, biological signalling networks, and cytoskeletons all take the form of graphs. Most attempts to apply neural networks to this data have used generic graph neural network architectures such as message-passing neural networks. However, these networks are strictly limited in their expressivity. Moreover, there is no obvious way to include our knowledge of important biochemical structures into the network.

Permute Me Softly: Learning Soft Permutations for Graph Representations.

Graph neural networks (GNNs) have recently emerged as a dominant paradigm for machine learning with graphs. Research on GNNs has mainly focused on the family of message passing neural networks (MPNNs). Similar to the Weisfeiler-Leman (WL) test of isomorphism, these models follow an iterative neighborhood aggregation procedure to update vertex representations, and they next compute graph representations by aggregating the representations of the vertices. Although very successful, MPNNs have been studied intensively in the past few years. Thus, there is a need for novel architectures which will allow research in the field to break away from MPNNs. In this paper, we propose a new graph neural network model, so-called π-GNN which learns a "soft" permutation (i. e., doubly stochastic) matrix for each graph, and thus projects all graphs into a common vector space. The learned matrices impose a "soft" ordering on the vertices of the input graphs, and based on this ordering, the adjacency matrices are mapped into vectors. These vectors can be fed into fully-connected or convolutional layers to deal with supervised learning tasks. In case of large graphs, to make the model more efficient in terms of running time and memory, we further relax the doubly stochastic matrices to row stochastic matrices. We empirically evaluate the model on graph classification and graph regression datasets and show that it achieves performance competitive with state-of-the-art models.

Δ-Quantum machine-learning for medicinal chemistry.

Many molecular design tasks benefit from fast and accurate calculations of quantum-mechanical (QM) properties. However, the computational cost of QM methods applied to drug-like molecules currently renders large-scale applications of quantum chemistry challenging. Aiming to mitigate this problem, we developed DelFTa, an open-source toolbox for the prediction of electronic properties of drug-like molecules at the density functional (DFT) level of theory, using Δ-machine-learning. Δ-Learning corrects the prediction error (Δ) of a fast but inaccurate property calculation. DelFTa employs state-of-the-art three-dimensional message-passing neural networks trained on a large dataset of QM properties. It provides access to a wide array of quantum observables on the molecular, atomic and bond levels by predicting approximations to DFT values from a low-cost semiempirical baseline. Δ-Learning outperformed its direct-learning counterpart for most of the considered QM endpoints. The results suggest that predictions for non-covalent intra- and intermolecular interactions can be extrapolated to larger biomolecular systems. The software is fully open-sourced and features documented command-line and Python APIs.

Explainable Molecular Sets: Using Information Theory to Generate Meaningful Descriptions of Groups of Molecules

Algorithmically identifying the meaningful similarities between an assortment of molecules is a critical chemical problem, and one which is only gaining in relevance as data-driven chemistry continues to progress. Effectively addressing this challenge can be achieved through a reformulation of the problem into information theory, cluster-based supervised classification, and the implementation of key concepts, particularly information entropy and mutual information. These concepts are combined with unsupervised learning atop learned chemical spaces to generate meaningful labels for arbitrary collections of molecules. An open-source and highly extensible codebase is provided to undertake these experiments, demonstrate the viability of the approach on known clusters, and glean insights into the learned representations of chemical space within message-passing neural networks, an architecture not readily permitting interpretability. This approach facilitates the interoperability between human chemical knowledge and the algorithmically derived insights, which will continue to become more prevalent in the coming years.

Remaining Useful Life Estimation for Engineered Systems Operating under Uncertainty with Causal GraphNets.

In this work, a novel approach, termed GNN-tCNN, is presented for the construction and training of Remaining Useful Life (RUL) models. The method exploits Graph Neural Networks (GNNs) and deals with the problem of efficiently learning from time series with non-equidistant observations, which may span multiple temporal scales. The efficacy of the method is demonstrated on a simulated stochastic degradation dataset and on a real-world accelerated life testing dataset for ball-bearings. The proposed method learns a model that describes the evolution of the system implicitly rather than at the raw observation level and is based on message-passing neural networks, which encode the irregularly sampled causal structure. The proposed approach is compared to a recurrent network with a temporal convolutional feature extractor head (LSTM-tCNN), which forms a viable alternative for the problem considered. Finally, by taking advantage of recent advances in the computation of reparametrization gradients for learning probability distributions, a simple, yet efficient, technique is employed for representing prediction uncertainty as a gamma distribution over RUL predictions.

Cartesian message passing neural networks for directional properties: Fast and transferable atomic multipoles.

The message passing neural network (MPNN) framework is a promising tool for modeling atomic properties but is, until recently, incompatible with directional properties, such as Cartesian tensors. We propose a modified Cartesian MPNN (CMPNN) suitable for predicting atom-centered multipoles, an essential component of ab initio force fields. The efficacy of this model is demonstrated on a newly developed dataset consisting of 46 623 chemical structures and corresponding high-quality atomic multipoles, which was deposited into the publicly available Molecular Sciences Software Institute QCArchive server. We show that the CMPNN accurately predicts atom-centered charges, dipoles, and quadrupoles and that errors in the predicted atomic multipoles have a negligible effect on multipole-multipole electrostatic energies. The CMPNN is accurate enough to model conformational dependencies of a molecule's electronic structure. This opens up the possibility of recomputing atomic multipoles on the fly throughout a simulation in which they might exhibit strong conformational dependence.

Predicting Infrared Spectra with Message Passing Neural Networks.

Infrared (IR) spectroscopy remains an important tool for chemical characterization and identification. Chemprop-IR has been developed as a software package for the prediction of IR spectra through the use of machine learning. This work serves the dual purpose of providing a trained general-purpose model for the prediction of IR spectra with ease and providing the Chemprop-IR software framework for the training of new models. In Chemprop-IR, molecules are encoded using a directed message passing neural network, allowing for molecule latent representations to be learned and optimized for the task of spectral predictions. Model training incorporates spectra metrics and normalization techniques that offer better performance with spectral predictions than standard practice in regression models. The model makes use of pretraining using quantum chemistry calculations and ensembling of multiple submodels to improve generalizability and performance. The spectral predictions that result are of high quality, showing capability to capture the extreme diversity of spectral forms over chemical space and represent complex peak structures.

Efficient lipophilicity prediction of molecules employing deep-learning models

Lipophilicity, expressed as logP, is a significant physiochemical property and is an indicator of absorption, distribution, metabolism and elimination characteristics of drugs used in medication. It is one of the major deciding factors of the fate of a molecule to be a successful drug. Mol2vec is a convenient and unsupervised machine learning technique which produces high-dimensional vector representations of molecules and its molecular substructures. The work described here aims to simplify prediction of logP values with high-degree of accuracy by using Deep Learning (DL) models paired with Mol2vec.The work described in this paper empirically demonstrates that by using the described DL models paired with Mol2vec, one can achieve results which are much better than the conventional ML techniques as well as more complex and recent algorithms like Message-passing Neural Networks (MPNN), Graph Convolution (GC) and Spatial Graph embedding (C-SGEN). Our RMSE (Root Mean Square Error) scores from the ensemble model is one of the best reported so far in literature. The methods elaborated in this paper are simple, yet effective in predicting logP values to a great degree of accuracy due to the use of Mol2vec and standard TensorFlow operators. The models employed here can be coded and maintained with much more ease compared to the techniques of MPNN, C-SGEN or GC.

Propagation Enhanced Neural Message Passing for Graph Representation Learning

Graph Neural Network (GNN) is capable of applying deep neural networks to graph domains. Recently, Message Passing Neural Networks (MPNNs) have been proposed to generalize several existing graph neural networks into a unified framework. For graph representation learning, MPNNs firstly generate discriminative node representations using the message passing function and then read from the node representation space to generate a graph representation using the readout function. In this paper, we analyze the representation capacity of the MPNNs for aggregating graph information and observe that the existing approaches ignore the self-loop for graph representation learning, leading to limited representation capacity. To alleviate this issue, we introduce a simple yet effective propagation enhanced extension, Self-Connected Neural Message Passing (SC-NMP), which aggregates the node representations of the current step and the graph representation of the previous step. To further improve the information flow, we also propose a Densely Self-Connected Neural Message Passing that connects each layer to every other layer in a feed-forward fashion. Extensive experiments on various benchmark datasets strongly demonstrate the effectiveness, leading to superior performance for graph classification and regression tasks.