Attention-based Graph Neural Network Research Articles

Dual separated attention-based graph neural network

Graph neural networks (GNNs) offer a viable solution to model the inter-dependencies among labeled and unlabeled samples in a semi-supervised manner. However, their performances can degrade dramatically when the number of labels is extremely limited, which is due to the limitations of typical graph convolutional network design in most existing GNNs, including over-smoothing, difficulty in extending the propagation step, and failure to preserve the distinctiveness of each node. To address the issues, we propose a dual separated attention-based graph neural network (DSA-GNN) to deal with label scarcity in semi-supervised node classification. Firstly, DSA-GNN decouples feature propagation from representation transformation to alleviate the problems of over-smoothing and overfitting. Secondly, DSA-GNN separates self-representation learning from neighbor-representation learning by two feature extractors with different learnable parameters. As a result, the commonality between connected nodes and the distinctiveness of each node can be both preserved. Thirdly, DSA-GNN incorporates an attention-based label propagation mechanism to refine the label prediction of each node, by aggregating label prediction among the neighborhood based on adaptive edge coefficients. The extensive experimental results on the real-world datasets demonstrate the superiority of DSA-GNN for semi-supervised node classification, especially when the observed labels are extremely limited.

Multi-layered knowledge graph neural network reveals pathway-level agreement of three breast cancer multi-gene assays

Multi-gene assays have been widely used to predict the recurrence risk for hormone receptor (HR)-positive breast cancer patients. However, these assays lack explanatory power regarding the underlying mechanisms of the recurrence risk. To address this limitation, we proposed a novel multi-layered knowledge graph neural network for the multi-gene assays. Our model elucidated the regulatory pathways of assay genes and utilized an attention-based graph neural network to predict recurrence risk while interpreting transcriptional subpathways relevant to risk prediction. Evaluation on three multi-gene assays—Oncotype DX, Prosigna, and EndoPredict—using SCAN-B dataset demonstrated the efficacy of our method. Through interpretation of attention weights, we found that all three assays are mainly regulated by signaling pathways driving cancer proliferation especially RTK-ERK-ETS-mediated cell proliferation for breast cancer recurrence. In addition, our analysis highlighted that the important regulatory subpathways remain consistent across different knowledgebases used for constructing the multi-level knowledge graph. Furthermore, through attention analysis, we demonstrated the biological significance and clinical relevance of these subpathways in predicting patient outcomes. The source code is available at http://biohealth.snu.ac.kr/software/ExplainableMLKGNN.

ResMatch: Residual Attention Learning for Feature Matching

Attention-based graph neural networks have made great progress in feature matching. However, the literature lacks a comprehensive understanding of how the attention mechanism operates for feature matching. In this paper, we rethink cross- and self-attention from the viewpoint of traditional feature matching and filtering. To facilitate the learning of matching and filtering, we incorporate the similarity of descriptors into cross-attention and relative positions into self-attention. In this way, the attention can concentrate on learning residual matching and filtering functions with reference to the basic functions of measuring visual and spatial correlation. Moreover, we leverage descriptor similarity and relative positions to extract inter- and intra-neighbors. Then sparse attention for each point can be performed only within its neighborhoods to acquire higher computation efficiency. Extensive experiments, including feature matching, pose estimation and visual localization, confirm the superiority of the proposed method. Our codes are available at https://github.com/ACuOoOoO/ResMatch.

Predicting Cardiotoxicity of Molecules Using Attention-Based Graph Neural Networks.

In drug discovery, the search for new and effective medications is often hindered by concerns about toxicity. Numerous promising molecules fail to pass the later phases of drug development due to strict toxicity assessments. This challenge significantly increases the cost, time, and human effort needed to discover new therapeutic molecules. Additionally, a considerable number of drugs already on the market have been withdrawn or re-evaluated because of their unwanted side effects. Among the various types of toxicity, drug-induced heart damage is a severe adverse effect commonly associated with several medications, especially those used in cancer treatments. Although a number of computational approaches have been proposed to identify the cardiotoxicity of molecules, the performance and interpretability of the existing approaches are limited. In our study, we proposed a more effective computational framework to predict the cardiotoxicity of molecules using an attention-based graph neural network. Experimental results indicated that the proposed framework outperformed the other methods. The stability of the model was also confirmed by our experiments. To assist researchers in evaluating the cardiotoxicity of molecules, we have developed an easy-to-use online web server that incorporates our model.

PathoGraph: An Attention-Based Graph Neural Network Capable of Prognostication Based on CD276 Labelling of Malignant Glioma Cells.

Computerized methods have been developed that allow quantitative morphological analyses of whole slide images (WSIs), e.g., of immunohistochemical stains. The latter are attractive because they can provide high-resolution data on the distribution of proteins in tissue. However, many immunohistochemical results are complex because the protein of interest occurs in multiple locations (in different cells and also extracellularly). We have recently established an artificial intelligence framework, PathoFusion which utilises a bifocal convolutional neural network (BCNN) model for detecting and counting arbitrarily definable morphological structures. We have now complemented this model by adding an attention-based graph neural network (abGCN) for the advanced analysis and automated interpretation of such data. Classical convolutional neural network (CNN) models suffer from limitations when handling global information. In contrast, our abGCN is capable of creating a graph representation of cellular detail from entire WSIs. This abGCN method combines attention learning with visualisation techniques that pinpoint the location of informative cells and highlight cell-cell interactions. We have analysed cellular labelling for CD276, a protein of great interest in cancer immunology and a potential marker of malignant glioma cells/putative glioma stem cells (GSCs). We are especially interested in the relationship between CD276 expression and prognosis. The graphs permit predicting individual patient survival on the basis of GSC community features. Our experiments lay a foundation for the use of the BCNN-abGCN tool chain in automated diagnostic prognostication using immunohistochemically labelled histological slides, but the method is essentially generic and potentially a widely usable tool in medical research and AI based healthcare applications.

Infrared spectra prediction using attention-based graph neural networks

In this work, we present attention-based graph neural networks to predict infrared (IR) spectra from chemical structures.

FAGRec: Alleviating data sparsity in POI recommendations via the feature-aware graph learning

<abstract> <p>Point-of-interest (POI) recommendation has attracted great attention in the field of recommender systems over the past decade. Various techniques, such as those based on matrix factorization and deep neural networks, have demonstrated outstanding performance. However, these methods are susceptible to the impact of data sparsity. Data sparsity is a significant characteristic of POI recommendation, where some POIs have limited interaction records and, in extreme cases, become cold-start POIs with no interaction history. To alleviate the influence of data sparsity on model performance, this paper introduced FAGRec, a POI-recommendation model based on the feature-aware graph. The key idea was to construct an interaction graph between POIs and their initial features. This allows the transformation of POI features into a weighted aggregation of initial features. Different POIs can share the learned representations of initial features, thereby mitigating the issue of data sparsity. Furthermore, we proposed attention-based graph neural networks and a user preference estimation method based on delayed time factors for learning representations of POIs and users, contributing to the generation of recommendations. Experimental results on two real-world datasets demonstrate the effectiveness of FAGRec in the task of POI recommendation.</p> </abstract>

Towards deep probabilistic graph neural network for natural gas leak detection and localization without labeled anomaly data

Deep learning has been widely applied to automated leakage detection and location of natural gas pipe networks. Prevalent deep learning approaches do not consider the spatial dependency of sensors, which limits leakage detection performance. Graph deep learning is a promising alternative to prevailing approaches as it can model spatial dependency. However, the challenge of collecting real-world anomaly data for training limits the accuracy and robustness of currently used graph deep learning approaches. This study proposes a deep probabilistic graph neural network in which attention-based graph neural network is built to model spatial sensor dependency. Variational Bayesian inference is integrated to model the posterior distribution of sensor dependency so that the leakage can be localized. An urban natural gas pipe network experiment is employed to construct the benchmark dataset, in which normal time-series data is applied to develop our proposed model while anomaly leakage data is used for performance comparison between our model and other state-of-the-art models. The results demonstrate that our model exhibits competitive detection accuracy (AUC)=0.9484, while the additional uncertainty interval provides more comprehensive leakage detection information compared to state-of-the-art deep learning models. In addition, our model’s posterior distribution enhances the leakage localization with the accuracy of positioning (PAc)= 0.8, which is higher than that of other state-of-the-art graph deep learning models. This study provides a comprehensive and robust alternative for subsequent decision-making to mitigate natural gas leakage from pipe networks.

Similarity and Complementarity Attention-Based Graph Neural Networks for Mashup-Oriented Cloud API Recommendation

Mashups, which combine various web application programming interfaces (APIs) to implement some complex requirements, have grown to be a popular technique for developing service-oriented software. However, recommending suitable cloud APIs for mashup creation is challenging due to the rapidly increasing number of comparable APIs. Many existing mashup-oriented cloud API recommendations focus on functional similarity and ignore functional complementarity, which significantly impacts the accuracy of the recommendation results. Therefore, this paper proposed a feature representation and recommendation method for cloud APIs that fuses both similarity and complementarity. A heterogeneous information network of the cloud API ecosystem was constructed, and the neighbors, based on metapaths, were aggregated using a self-attention mechanism to generate the features of similarity and complementarity for the cloud APIs. Then, the mashup-related attention was utilized to fuse the two features, taking into consideration the varying preferences of different mashups towards similarity and complementarity features of cloud APIs. This fusion resulted in features that align cloud APIs with mashup requirements, which were employed to predict the probability of the mashup invoking a particular candidate cloud API. The proposed method was evaluated on a real dataset, and the results showed that it outperforms the baseline method and enhances the performance of mashup-oriented cloud API recommendations.

HNetGO: protein function prediction via heterogeneous network transformer.

Protein function annotation is one of the most important research topics for revealing the essence of life at molecular level in the post-genome era. Current research shows that integrating multisource data can effectively improve the performance of protein function prediction models. However, the heavy reliance on complex feature engineering and model integration methods limits the development of existing methods. Besides, models based on deep learning only use labeled data in a certain dataset to extract sequence features, thus ignoring a large amount of existing unlabeled sequence data. Here, we propose an end-to-end protein function annotation model named HNetGO, which innovatively uses heterogeneous network to integrate protein sequence similarity and protein-protein interaction network information and combines the pretraining model to extract the semantic features of the protein sequence. In addition, we design an attention-based graph neural network model, which can effectively extract node-level features from heterogeneous networks and predict protein function by measuring the similarity between protein nodes and gene ontology term nodes. Comparative experiments on the human dataset show that HNetGO achieves state-of-the-art performance on cellular component and molecular function branches.

An integrated fuzzy neural supervision and attention-based graph neural network for improving network clustering

An integrated fuzzy neural supervision and attention-based graph neural network for improving network clustering

Signed attention based graph neural network for graphs with heterophily

Graph Neural Networks (GNNs) are an effective deep learning methodology used to encode graph-structured data. Several popular GNNs have demonstrated excellent results in various downstream tasks, including node classification and visualization, especially when homophily occurs in graphs where connected nodes may share identical features or labels. Although recent models can extend GNNs beyond homophily, two fundamental weaknesses limit their expressive ability. First, most of the existing GNNs fail to adaptively aggregate information from different neighbors, especially negative information from different classes. Even worse, they fail to encode the positional information, leading to the inevitable conflation of information from different neighborhoods that should be specified with different importance. To overcome these limitations, we propose a novel graph neural network model based on the Signed Attention mechanism (SA), namely SAGNN. Specifically, the signed attention mechanism is presented to adaptively learn the complex relationship between exact neighboring nodes. Compared with the original non-negative attention mechanism, SA aggregates information from both positive and negative neighbors, especially on graphs with heterophily. Furthermore, we propose an Exactly Higher-Order Neighborhood Aggregation that implicitly encodes positional information to specify the importance of different neighborhoods. Extensive experiments conducted on eight benchmark datasets demonstrate that the proposed SAGNN model learns more robust and expressive representations, and achieves superior performance on node classification and visualization tasks.

Attention-based graph neural networks: a survey

Attention-based graph neural networks: a survey

Prediction of molecular field points using SE(3)-transformer model

Due to their computational efficiency, 2D fingerprints are typically used in similarity-based high-content screening. The interaction of a ligand with its target protein, however, relies on its physicochemical interactions in 3D space. Thus, ligands with different 2D scaffolds can bind to the same protein if these ligands share similar interaction patterns. Molecular fields can represent those interaction profiles. For efficiency, the extrema of those molecular fields, named field points, are used to quantify the ligand similarity in 3D. The calculation of field points involves the evaluation of the interaction energy between the ligand and a small probe shifted on a fine grid representing the molecular surface. These calculations are computationally prohibitive for large datasets of ligands, making field point representations of molecules intractable for high-content screening. Here, we overcome this roadblock by one-shot prediction of field points using generative neural networks based on the molecular structure alone. Field points are predicted by training an SE(3)-Transformer, an equivariant, attention-based graph neural network architecture, on a large set of ligands with field point data. Resulting data demonstrates the feasibility of this approach to precisely generate negative, positive and hydrophobic field points within 0.5 Å of the ground truth for a diverse set of drug-like molecules.

Crysformer: An attention-based graph neural network for properties prediction of crystals

We present a novel approach for the prediction of crystal material properties that is distinct from the computationally complex and expensive density functional theory (DFT)-based calculations. Instead, we utilize an attention-based graph neural network that yields high-accuracy predictions. Our approach employs two attention mechanisms that allow for message passing on the crystal graphs, which in turn enable the model to selectively attend to pertinent atoms and their local environments, thereby improving performance. We conduct comprehensive experiments to validate our approach, which demonstrates that our method surpasses existing methods in terms of predictive accuracy. Our results suggest that deep learning, particularly attention-based networks, holds significant promise for predicting crystal material properties, with implications for material discovery and the refined intelligent systems.

Attention-Based Graph Neural Network for Label Propagation in Single-Cell Omics

Single-cell data analysis has been at forefront of development in biology and medicine since sequencing data have been made available. An important challenge in single-cell data analysis is the identification of cell types. Several methods have been proposed for cell-type identification. However, these methods do not capture the higher-order topological relationship between different samples. In this work, we propose an attention-based graph neural network that captures the higher-order topological relationship between different samples and performs transductive learning for predicting cell types. The evaluation of our method on both simulation and publicly available datasets demonstrates the superiority of our method, scAGN, in terms of prediction accuracy. In addition, our method works best for highly sparse datasets in terms of F1 score, precision score, recall score, and Matthew's correlation coefficients as well. Further, our method's runtime complexity is consistently faster compared to other methods.

Predicting future production system bottlenecks with a graph neural network approach

Predicting throughput bottlenecks has received an increasing research attention in the recent years. Several statistical and machine learning models have been developed, with the objective of identifying future bottleneck locations for more efficient and cost-saving productions. However, bottleneck prediction remains a challenging task due to the complexity of manufacturing systems and the interactive effects from many impacting factors. This paper proposed the BSTAN (bottleneck spatial-temporal attention network) – an interpretable modeling framework to improve system bottlenecks prediction capability for complex manufacturing systems by: (1) characterizing line topological features using graph representations; (2) modeling spatial-temporal relationships and interactions between stations using graph attention network with gated recurrent units. In this proposed framework, the production line is first transformed into the representation of a weighted undirected graph to depict network structural information of a production line. An attention-based graph neural network with gated recurrent units is applied to capture both the temporal trends and station interactions to predict each station’s blockage and starvation. Future bottleneck stations are identified by analyzing the forecasted blockage and starvation distribution. To evaluate the model performance, an industrial case study is performed with data collected from a one-year production at an automotive powertrain assembly line. With the proper model construction, BSTAN has outperformed both statistical and machine learning benchmark models, achieving up to 15% lower root mean square error compared to the best benchmark method in predicting overall system blockage and starvation time.

Attention-Based Graph Neural Network for Molecular Solubility Prediction.

Drug discovery (DD) research is aimed at the discovery of new medications. Solubility is an important physicochemical property in drug development. Active pharmaceutical ingredients (APIs) are essential substances for high drug efficacy. During DD research, aqueous solubility (AS) is a key physicochemical attribute required for API characterization. High-precision in silico solubility prediction reduces the experimental cost and time of drug development. Several artificial tools have been employed for solubility prediction using machine learning and deep learning techniques. This study aims to create different deep learning models that can predict the solubility of a wide range of molecules using the largest currently available solubility data set. Simplified molecular-input line-entry system (SMILES) strings were used as molecular representation, models developed using simple graph convolution, graph isomorphism network, graph attention network, and AttentiveFP network. Based on the performance of the models, the AttentiveFP-based network model was finally selected. The model was trained and tested on 9943 compounds. The model outperformed on 62 anticancer compounds with metric Pearson correlation R 2 and root-mean-square error values of 0.52 and 0.61, respectively. AS can be improved by graph algorithm improvement or more molecular properties addition.

Deep Attention Aware Feature Learning for Data Association in Multiple Source Localization

The letter addresses the problem of data association for multiple source localization in a distributed sensor network. To increase the efficiency of localization, we present a learning methodology to generate highly distinctive measurement features for data association. We propose an attention-based graph neural network that exploits the relationship among the phases from the Short Time Fourier transform and the estimated measurements to generate highly descriptive features for each of the detected source signals. The methodology is free from any selective threshold and processes each of the measurements to detect and enrich the similar and distinguishable aspects of the measurements. These aspects are then aggregated in feature representations for each source measurement. The generated features can then be employed with any data association method to segregate the measurements belonging to the same source. It is observed in experimental evaluation that the proposed features employed with state-of-the-art association frameworks yield lower association and localization errors.

DTiGNN: Learning drug-target embedding from a heterogeneous biological network based on a two-level attention-based graph neural network.

In vitro experiment-based drug-target interaction (DTI) exploration demands more human, financial and data resources. In silico approaches have been recommended for predicting DTIs to reduce time and cost. During the drug development process, one can analyze the therapeutic effect of the drug for a particular disease by identifying how the drug binds to the target for treating that disease. Hence, DTI plays a major role in drug discovery. Many computational methods have been developed for DTI prediction. However, the existing methods have limitations in terms of capturing the interactions via multiple semantics between drug and target nodes in a heterogeneous biological network (HBN). In this paper, we propose a DTiGNN framework for identifying unknown drug-target pairs. The DTiGNN first calculates the similarity between the drug and target from multiple perspectives. Then, the features of drugs and targets from each perspective are learned separately by using a novel method termed an information entropy-based random walk. Next, all of the learned features from different perspectives are integrated into a single drug and target similarity network by using a multi-view convolutional neural network. Using the integrated similarity networks, drug interactions, drug-disease associations, protein interactions and protein-disease association, the HBN is constructed. Next, a novel embedding algorithm called a meta-graph guided graph neural network is used to learn the embedding of drugs and targets. Then, a convolutional neural network is employed to infer new DTIs after balancing the sample using oversampling techniques. The DTiGNN is applied to various datasets, and the result shows better performance in terms of the area under receiver operating characteristic curve (AUC) and area under precision-recall curve (AUPR), with scores of 0.98 and 0.99, respectively. There are 23,739 newly predicted DTI pairs in total.