Graph-based Deep Learning Research Articles

Machine learning for predicting the viscosity of binary liquid mixtures

Viscosity is an important parameter in process engineering and is a key design objective for application areas including the coatings, lubricants, personal care, and pharmaceutical industries. The lack of reliable and general methods for predicting the viscosities of mixtures creates a barrier for modern process engineering and product design. In this work, we developed a graph-based neural network architecture and applied it to the problem of predicting the viscosity of binary liquid mixtures as a function of composition and temperature. To obtain a high-quality training dataset, we also developed an automated curation pipeline and applied it to a large dataset collected from the literature by the National Institute of Standards and Technology (NIST) to be used as training data. The resulting model predicts viscosity with an MAE of 0.043 and an RMSE of 0.080 in log cP units (base 10). To improve the dependability of the model, we developed a classifier that evaluated the reliability of a prediction based on the variance between an ensemble of models. Using this approach, the model had an effective MAE of 0.029 and RMSE of 0.047 for predictions that were assessed as reliable (80% of the test set). Overall, this work provides 1) a large set of curated viscosity data that can be used for future machine learning efforts, 2) a new, graph-based deep learning approach for predicting the viscosity of binary mixtures, and 3) an illustrative case study for how deep learning can be used for accurate and reliable property prediction.

Toward Interpretable Graph Tensor Convolution Neural Network for Code Semantics Embedding

Intelligent deep learning-based models have made significant progress for automated source code semantics embedding, and current research works mainly leverage natural language-based methods and graph-based methods. However, natural language-based methods do not capture the rich semantic structural information of source code, and graph-based methods do not utilize rich distant information of source code due to the high cost of message-passing steps. In this paper, we propose a novel interpretable model, called graph tensor convolution neural network (GTCN), to generate accurate code embedding, which is capable of comprehensively capturing the distant information of code sequences and rich code semantics structural information. Firstly, we propose to utilize a high-dimensional tensor to integrate various heterogeneous code graphs with node sequence features, such as control flow, data flow. Secondly, inspired by the current advantages of graph-based deep learning and efficient tensor computations, we propose a novel interpretable graph tensor convolution neural network for learning accurate code semantic embedding from the code graph tensor. Finally, we evaluate three popular applications on the GTCN model, variable misuse detection, source code prediction, and vulnerability detection. Compared with current state-of-the-art methods, our model achieves higher scores with respect to the top-1 accuracy while costing less training time.

Molecule generation toward target protein (SARS-CoV-2) using reinforcement learning-based graph neural network via knowledge graph.

AI-driven approaches are widely used in drug discovery, where candidate molecules are generated and tested on a target protein for binding affinity prediction. However, generating new compounds with desirable molecular properties such as Quantitative Estimate of Drug-likeness (QED) and Dopamine Receptor D2 activity (DRD2) while adhering to distinct chemical laws is challenging. To address these challenges, we proposed a graph-based deep learning framework to generate potential therapeutic drugs targeting the SARS-CoV-2 protein. Our proposed framework consists of two modules: a novel reinforcement learning (RL)-based graph generative module with knowledge graph (KG) and a graph early fusion approach (GEFA) for binding affinity prediction. The first module uses a gated graph neural network (GGNN) model under the RL environment for generating novel molecular compounds with desired properties and a custom-made KG for molecule screening. The second module uses GEFA to predict binding affinity scores between the generated compounds and target proteins. Experiments show how fine-tuning the GGNN model under the RL environment enhances the molecules with desired properties to generate valid and unique compounds using different scoring functions. Additionally, KG-based screening reduces the search space of generated candidate molecules by while retaining of promising binding molecules against SARS-CoV-2 protein, i.e., 3C-like protease (3CLpro). We achieved a binding affinity score of 8.185 from the top rank of generated compound. In addition, we compared top-ranked generated compounds to Indinavir on different parameters, including drug-likeness and medicinal chemistry, for qualitative analysis from a drug development perspective. The online version contains supplementary material available at 10.1007/s13721-023-00409-2.

GDockScore: a graph-based protein-protein docking scoring function.

Protein complexes play vital roles in a variety of biological processes, such as mediating biochemical reactions, the immune response and cell signalling, with 3D structure specifying function. Computational docking methods provide a means to determine the interface between two complexed polypeptide chains without using time-consuming experimental techniques. The docking process requires the optimal solution to be selected with a scoring function. Here, we propose a novel graph-based deep learning model that utilizes mathematical graph representations of proteins to learn a scoring function (GDockScore). GDockScore was pre-trained on docking outputs generated with the Protein Data Bank biounits and the RosettaDock protocol, and then fine-tuned on HADDOCK decoys generated on the ZDOCK Protein Docking Benchmark. GDockScore performs similarly to the Rosetta scoring function on docking decoys generated using the RosettaDock protocol. Furthermore, state-of-the-art is achieved on the CAPRI score set, a challenging dataset for developing docking scoring functions. The model implementation is available at https://gitlab.com/mcfeemat/gdockscore. Supplementary data are available at Bioinformatics Advances online.

IEdgeDTA: integrated edge information and 1D graph convolutional neural networks for binding affinity prediction.

Artificial intelligence has become more prevalent in broad fields, including drug discovery, in which the process is costly and time-consuming when conducted through wet experiments. As a result, drug repurposing, which tries to utilize approved and low-risk drugs for a new purpose, becomes more attractive. However, screening candidates from many drugs for specific protein targets is still expensive and tedious. This study aims to leverage computational resources to aid drug discovery by utilizing drug-protein interaction data and estimating their interaction strength, so-called binding affinity. Our estimation approach addresses multiple challenges encountered in the field. First, we employed a graph-based deep learning technique to overcome the limitations of drug compounds represented in string format by incorporating background knowledge of node and edge information as separate multi-dimensional features. Second, we tackled the complexities associated with extracting the representation and structure of proteins by utilizing a pre-trained model for feature extraction. Also, we employed graph operations over the 1D representation of a protein sequence to overcome the fixed-length problem typically encountered in language model tasks. In addition, we conducted a comparative analysis with a baseline model that creates a protein graph from a contact map prediction model, giving valuable insights into the performance and effectiveness of our proposed method. We evaluated the performance of our model using the same benchmark datasets with a variety of matrices as other previous work, and the results show that our model achieved the best prediction results while requiring no contact map information compared to other graph-based methods.

HiGNN: A Hierarchical Informative Graph Neural Network for Molecular Property Prediction Equipped with Feature-Wise Attention.

Elucidating and accurately predicting the druggability and bioactivities of molecules plays a pivotal role in drug design and discovery and remains an open challenge. Recently, graph neural networks (GNNs) have made remarkable advancements in graph-based molecular property prediction. However, current graph-based deep learning methods neglect the hierarchical information of molecules and the relationships between feature channels. In this study, we propose a well-designed hierarchical informative graph neural network (termed HiGNN) framework for predicting molecular property by utilizing a corepresentation learning of molecular graphs and chemically synthesizable breaking of retrosynthetically interesting chemical substructure (BRICS) fragments. Furthermore, a plug-and-play feature-wise attention block is first designed in HiGNN architecture to adaptively recalibrate atomic features after the message passing phase. Extensive experiments demonstrate that HiGNN achieves state-of-the-art predictive performance on many challenging drug discovery-associated benchmark data sets. In addition, we devise a molecule-fragment similarity mechanism to comprehensively investigate the interpretability of the HiGNN model at the subgraph level, indicating that HiGNN as a powerful deep learning tool can help chemists and pharmacists identify the key components of molecules for designing better molecules with desired properties or functions. The source code is publicly available at https://github.com/idruglab/hignn.

Structural Analysis and Prediction of Hematotoxicity Using Deep Learning Approaches.

Hematotoxicity has been becoming a serious but overlooked toxicity in drug discovery. However, only a few in silico models have been reported for the prediction of hematotoxicity. In this study, we constructed a high-quality dataset comprising 759 hematotoxic compounds and 1623 nonhematotoxic compounds and then established a series of classification models based on a combination of seven machine learning (ML) algorithms and nine molecular representations. The results based on two data partitioning strategies and applicability domain (AD) analysis illustrate that the best prediction model based on Attentive FP yielded a balanced accuracy (BA) of 72.6%, an area under the receiver operating characteristic curve (AUC) value of 76.8% for the validation set, and a BA of 69.2%, an AUC of 75.9% for the test set. In addition, compared with existing filtering rules and models, our model achieved the highest BA value of 67.5% for the external validation set. Additionally, the shapley additive explanation (SHAP) and atom heatmap approaches were utilized to discover the important features and structural fragments related to hematotoxicity, which could offer helpful tips to detect undesired positive substances. Furthermore, matched molecular pair analysis (MMPA) and representative substructure derivation technique were employed to further characterize and investigate the transformation principles and distinctive structural features of hematotoxic chemicals. We believe that the novel graph-based deep learning algorithms and insightful interpretation presented in this study can be used as a trustworthy and effective tool to assess hematotoxicity in the development of new drugs.

Trajectory Prediction Using Graph-Based Deep Learning for Longitudinal Control of Autonomous Vehicles: A Proactive Approach for Autonomous Driving in Urban Dynamic Traffic Environments

This article presents a method for the trajectory prediction of surrounding vehicles and proactive longitudinal control of autonomous vehicles (AVs) in an urban road environment. A long short-term memory (LSTM)-based deep learning model is designed for the surrounding vehicles’ trajectory prediction. In our model, the historical evolution of the relation between a target vehicle and lanes is considered to learn the driver’s behavior in a lane-aware manner. Interaction among adjacent vehicles is captured based on a graph convolutional network (GCN), which uses a self-attention mechanism. Compared to other approaches, our prediction model utilizes environment information that is acquirable in AVs with local sensors. A model predictive control (MPC) is designed to derive the control inputs of acceleration for AVs. The proposed control method utilizes the prediction results of the target vehicle to give action requests to AV in a proactive manner considering both safety and ride quality. The results of comparative studies indicate that the proposed prediction model achieves improved accuracy compared to baselines. The control results provided by automated driving tests show that the proposed control algorithm applied by the LSTM-based prediction model enables AVs to achieve safety with respect to surrounding vehicles and provide ride comfort to passengers.

Enabling smart curb management with spatiotemporal deep learning

Curb spaces are important assets to cities. They are often used by travelers to switch transportation means, visitors to access curbside properties, and municipalities to place roadside infrastructure. The promotion of multi-modal transportation and the emergence of new mobility services have complicated both curb environments and their management. Consequently, some cities have started to explore new strategies for curb management, but lacked the anticipation on how different curb regulations and built-environment features may collectively influence curb-use patterns across user groups. We make a step toward smart curb environment by proposing a graph-based deep learning approach, i.e., MultiGCN-LSTM, to predict diverse curb uses across time and space. We used two graph convolution layers and an LSTM layer to capture the spatial, temporal, and semantic dependencies between curb regulations, built-environment semantics, and diverse curb uses. Two place-specific models were developed separately for a medium-sized college town and a metropolitan in the U.S. The effectiveness of the proposed models was validated with ablation studies and demonstrated in three scenario experiments. The research contributes to smart curb management in the face of more diversified and intensified curb uses with new mobility services and emerging vehicular technologies.

Graph neural networks for efficient learning of mechanical properties of polycrystals

We present graph neural networks (GNNs) as an efficient and accurate machine learning approach to predict mechanical properties of polycrystalline materials. Here, a GNN was developed based on graph representation of polycrystals incorporating only fundamental features of grains including their crystallographic orientation, size, and grain neighbor connectivity information. We tested our method on modeling stiffness and yield strength of α-Ti microstructures, varying in their crystallographic texture. We find the GNN predicts both properties with high accuracy with mean relative errors of ∼1% for unseen microstructures from a given set of textures and <2% for microstructures of unseen texture, even when presented with limited training data. This accuracy is comparable to methods that require high-resolution three-dimensional (3D) microstructure data, such as 3D convolutional neural networks (3D-CNNs) and models that depend on the computation of spatial statistics. The present results show that graph-based deep learning is a promising framework for property prediction, especially considering the high cost associated with obtaining high-resolution 3D microstructure data and the general scarcity of experimental materials datasets.

Similarity learning with deep CRF for person re-identification

The core of person re-identification (Re-ID) lies in robustly estimating similarities for each probe-gallery image pair. A common practice in existing works is to calculate the similarity of each image pair independently, ignoring relations between different probe-gallery pairs. In this paper, we present a deep learning conditional random field (Deep-CRF) graph to model group-wise similarities within a batch of images, and regard the Re-ID task as a CRF node labeling problem. Unlike the existing deep CRF based approach where the CRF inference is only involved in the training stage, our method intends to fully exploit the potential of CRF model, exhibiting inference consistency in both training and testing. Specifically, we design unary potentials for computing each probe-gallery similarity separately. To efficiently encode relationships between different probe-gallery pairs, pairwise potentials are built on an arbitrary node pair whose learning is achieved by a joint matching strategy using bidirectional LSTM. We pose the CRF inference as a RNN learning process, where unary and pairwise potentials are jointly optimized in an end-to-end manner. Extensive experiments on three large-scale person Re-ID datasets demonstrate the effectiveness of the proposed method. Our Deep-CRF achieves the best results compared with the previous graph-based deep learning approaches and substantially exceeds the existing deep CRF framework by 8% in Rank1 accuracy on CUHK03 dataset. It also behaves competitive among the current state-of-the-art methods.

Adopting integration of multispectral time resolved analysis and graph-based deep learning method in partial discharge type identification

Partial discharge (PD) is treated as one of the major threats for gas insulated switchgear (GIS). By using the new generation multispectral detection sensor named as SiPM-based multispectral discharge sensor (SMDS), the time resolved partial discharge with multispectral information (named as MTRPD) for creeping discharge, suspension discharge and tip discharge, respectively. It indicates that the MTRPD for the specific discharge defect perform unique spectral fingerprints in discharge mode. Based on the graph characteristics of MTRPD, we introduced the convolution neural network (CNN) to implement PD type identification whose overall accuracy of Δqi-Δqi+1 and Δti-Δti+1 were 99.7% and 98.9%, respectively. This paper provides a new technique tool for fine diagnosis of PD independent of phase analysis.

Not from Scratch: Predicting Thermophysical Properties through Model-Based Transfer Learning Using Graph Convolutional Networks.

In this study, a framework for the prediction of thermophysical properties based on transfer learning from existing estimation models is explored. The predictive capabilities of conventional group-contribution methods and traditional machine-learning approaches rely heavily on the availability of experimental datasets and their uncertainty. Through the use of a pretraining scheme, which leverages the knowledge established by other estimation methods, improved prediction models for thermophysical properties can be obtained after fine-tuning networks with more accurate experimental data. As our experiments show, for the case of critical properties of compounds, this pipeline not only improves the performance of the models on commonly found organic structures but can also help these models generalize to less explored areas of chemical space, where experimental data is scarce, such as inorganics and heavier organic compounds. Transfer learning from estimation models data also allows for graph-based deep learning models to create more flexible molecular features over a bigger chemical space, which leads to improved predictive capabilities and can give insights into the relationship between molecular structures and thermophysical properties. The generated molecular features can discriminate behavior discrepancy between isomers without the need of additional parameters. Also, this approach shows better robustness to outliers in experimental datasets.

A graph neural network framework based on preference-aware graph diffusion for recommendation.

Transforming user check-in data into graph structure data is a popular and powerful way to analyze users' behaviors in the field of recommendation. Graph-based deep learning methods such as graph embeddings and graph neural networks have shown promising performance on the task of point-of-interest recommendation in recent years. Despite effectiveness, existing methods fail to capture deep graph structural information, leading the suboptimal representations. In addition, they lack the ability of learning the influences of both global preference and user preference on the check-in behavior. To address the aforementioned issues, we propose a general framework based on preference-aware graph diffusion, named PGD. We first construct two types of graphs to represent the global preference and user preference. Then, we apply a graph diffusion process to capture the structural information of the generated graphs, resulting in weighted adjacency matrices. Finally, graph neural network-based backbones are introduced to learn the representations of users and POIs on weighted adjacency matrices. A learnable aggregation module is developed to learn the final representations from global preference and user preference adaptively. Extensive experiments on four real-world datasets demonstrate the superiority of PGD on POI recommendation, compared with the mainstream graph-based deep learning methods.

Deep Learning-Based Increment Theory for Formation Enthalpy Predictions.

Machine learning predictions of molecular thermochemistry, such as formation enthalpy, have been limited for large and complicated species because of the lack of available training data. Such predictions would be important in the prediction of reaction thermodynamics and the construction of kinetic models. Herein, we introduce a graph-based deep learning approach that can separately learn the enthalpy contribution of each atom in its local environment with the effect of the overall molecular structure taken into account. Because this approach follows the additivity scheme of increment theory, it can be generalized to larger and more complicated species not present in the training data. By training the model on molecules with up to 11 heavy atoms, it can predict the formation enthalpy of testing molecules with up to 42 heavy atoms with a mean absolute error of 2 kcal/mol, which is less than half of the error of the conventional increment theory. We expect that this approach will also enable rapid prediction of other extensive properties of large molecules that are difficult to derive from experiments or ab initio calculation.

LIGHTHOUSE illuminates therapeutics for a variety of diseases including COVID-19

One of the bottlenecks in the application of basic research findings to patients is the enormous cost, time, and effort required for high-throughput screening of potential drugs for given therapeutic targets. Here we have developed LIGHTHOUSE, a graph-based deep learning approach for discovery of the hidden principles underlying the association of small-molecule compounds with target proteins. Without any 3D structural information for proteins or chemicals, LIGHTHOUSE estimates protein-compound scores that incorporate known evolutionary relations and available experimental data. It identified therapeutics for cancer, lifestyle-related disease, and bacterial infection. Moreover, LIGHTHOUSE predicted ethoxzolamide as a therapeutic for coronavirus disease 2019 (COVID-19), and this agent was indeed effective against alpha, beta, gamma, and delta variants of severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) that are rampant worldwide. We envision that LIGHTHOUSE will help accelerate drug discovery and fill the gap between bench side and bedside.

Deep Feature Aggregation Framework Driven by Graph Convolutional Network for Scene Classification in Remote Sensing.

Scene classification of high spatial resolution (HSR) images can provide data support for many practical applications, such as land planning and utilization, and it has been a crucial research topic in the remote sensing (RS) community. Recently, deep learning methods driven by massive data show the impressive ability of feature learning in the field of HSR scene classification, especially convolutional neural networks (CNNs). Although traditional CNNs achieve good classification results, it is difficult for them to effectively capture potential context relationships. The graphs have powerful capacity to represent the relevance of data, and graph-based deep learning methods can spontaneously learn intrinsic attributes contained in RS images. Inspired by the abovementioned facts, we develop a deep feature aggregation framework driven by graph convolutional network (DFAGCN) for the HSR scene classification. First, the off-the-shelf CNN pretrained on ImageNet is employed to obtain multilayer features. Second, a graph convolutional network-based model is introduced to effectively reveal patch-to-patch correlations of convolutional feature maps, and more refined features can be harvested. Finally, a weighted concatenation method is adopted to integrate multiple features (i.e., multilayer convolutional features and fully connected features) by introducing three weighting coefficients, and then a linear classifier is employed to predict semantic classes of query images. Experimental results performed on the UCM, AID, RSSCN7, and NWPU-RESISC45 data sets demonstrate that the proposed DFAGCN framework obtains more competitive performance than some state-of-the-art methods of scene classification in terms of OAs.

Traffic-GGNN: Predicting Traffic Flow via Attentional Spatial-Temporal Gated Graph Neural Networks

Recent spatial-temporal graph-based deep learning methods for Traffic Flow Prediction (TFP) problems have shown superior performance in modeling higher-level spatial interactions and temporal correlations. However, most of these methods suffer from post-fusion efficiency difficulty caused by separate explorations of the spatial communications and the temporal dependencies, which could result in delayed and biased predictions. To address that, we propose a Traffic Gated Graph Neural Networks (Traffic-GGNN) for real-time-fused spatial-temporal representation modeling. Firstly, we adopt bidirectional message passing to capture the location-wise spatial interactions. Secondly, we apply a GRU-based module to explore and aggregate the spatial interactions with the temporal correlations in a real-time fusion way. Lastly, we introduce a self-attention mechanism to reweight the location-based importance and produce the final prediction. Moreover, our proposed model allows end-to-end training thus it is easy to scale to diverse types of traffic datasets and yield better efficiency and effectiveness on three real-world datasets (SZ-taxi, Los-loop, and PEMS-BAY).

Identification of accident-injury type and bodypart factors from construction accident reports: A graph-based deep learning framework

Accident reports provide information to understand why and how events occur. Learning from past accident reports is critical for preventing accidents or injuries in construction safety management. However, there are two issues: (1) manual analysis of such accident reports is time-consuming and labor-intensive; and (2) previous research mainly focused on analyzing the causal factors of accidents. Not much research concentrates on the injury effect in an accident and the influential relationship between accident cause and injury effect. To tackle this problem, a graph-based deep learning framework is proposed to identify accident-injury type and bodypart factors automatically to enable managers to make timely and better-informed decisions to prevent accidents and injuries for on-site safety. In this framework, a graph-based deep learning approach (specifically, the Graph Convolutional Network) is developed to automatically classify accident reports labeled with accident_type and injury_type, whereas the traversal method is developed to identify the bodypart factors. To further intuitively visualize these safety risk factors (e.g., accident_type, injury_type, and bodypart factors), the co-occurrence networks are drawn to further intuitively reveal the interdependency in accident-injury and injury-bodypart types respectively. From the perspective of theoretical and practical contributions, the framework proposed in this study not only represents a substantial data-driven advancement in construction accident report classification and keyword extraction tasks, but also enables managers to get knowledge of construction safety performance (i.e., accident causes and injury effects) and further formulate corresponding strategies to prevent accidents and injuries in on-site safety management.

GNN-SubNet: disease subnetwork detection with explainable graph neural networks.

The tremendous success of graphical neural networks (GNNs) already had a major impact on systems biology research. For example, GNNs are currently being used for drug target recognition in protein-drug interaction networks, as well as for cancer gene discovery and more. Important aspects whose practical relevance is often underestimated are comprehensibility, interpretability and explainability. In this work, we present a novel graph-based deep learning framework for disease subnetwork detection via explainable GNNs. Each patient is represented by the topology of a protein-protein interaction (PPI) network, and the nodes are enriched with multi-omics features from gene expression and DNA methylation. In addition, we propose a modification of the GNNexplainer that provides model-wide explanations for improved disease subnetwork detection. The proposed methods and tools are implemented in the GNN-SubNet Python package, which we have made available on our GitHub for the international research community (https://github.com/pievos101/GNN-SubNet). Supplementary data are available at Bioinformatics online.