Heterogeneous graph attention networks for passage retrieval

Emerging evidence has proved that circular RNAs (circRNAs) are implicated in pathogenic processes. They are regarded as promising biomarkers for diagnosis due to covalently closed loop structures. As opposed to traditional experiments, computational approaches can identify circRNA-disease associations at a lower cost. Aggregating multi-source pathogenesis data helps to alleviate data sparsity and infer potential associations at the system level. The majority of computational approaches construct a homologous network using multi-source data, but they lose the heterogeneity of the data. Effective methods that use the features of multi-source data are considered as a matter of urgency. In this paper, we propose a model (CDHGNN) based on edge-weighted graph attention and heterogeneous graph neural networks for potential circRNA-disease association prediction. The circRNA network, micro RNA network, disease network and heterogeneous network are constructed based on multi-source data. To reflect association probabilities between nodes, an edge-weighted graph attention network model is designed for node features. To assign attention weights to different types of edges and learn contextual meta-path, CDHGNN infers potential circRNA-disease association based on heterogeneous neural networks. CDHGNN outperforms state-of-the-art algorithms in terms of accuracy. Edge-weighted graph attention networks and heterogeneous graph networks have both improved performance significantly. Furthermore, case studies suggest that CDHGNN is capable of identifying specific molecular associations and investigating biomolecular regulatory relationships in pathogenesis. The code of CDHGNN is freely available at https://github.com/BioinformaticsCSU/CDHGNN.

Drug sensitivity is essential for identifying effective treatments. Meanwhile, circular RNA (circRNA) has potential in disease research and therapy. Uncovering the associations between circRNAs and cellular drug sensitivity is crucial for understanding drug response and resistance mechanisms. In this study, we proposed DeepHeteroCDA, a novel circRNA-drug sensitivity association prediction method based on multi-scale heterogeneous network and graph attention mechanism. We first constructed a heterogeneous graph based on drug-drug similarity, circRNA-circRNA similarity, and known circRNA-drug sensitivity associations. Then, we embedded the 2D structure of drugs into the circRNA-drug sensitivity heterogeneous graph and use graph convolutional networks (GCN) to extract fine-grained embeddings of drug. Finally, by simultaneously updating graph attention network for processing heterogeneous networks and GCN for processing drug structures, we constructed a multi-scale heterogeneous network and use a fully connected layer to predict the circRNA-drug sensitivity associations. Extensive experimental results highlight the superior of DeepHeteroCDA. The visualization experiment shows that DeepHeteroCDA can effectively extract the association information. The case studies demonstrated the effectiveness of our model in identifying potential circRNA-drug sensitivity associations. The source code and dataset are available at https://github.com/Hhhzj-7/DeepHeteroCDA.

To solve the problem of finding overlapping communities in heterogeneous networks, this paper proposes a model for finding overlapping communities in heterogeneous networks based on density peak clustering. The model improves the heterogeneous graph attention network, which fully mines for the important information of nodes and meta-paths in information representation from the node level and the semantic level attention mechanisms and performs hierarchical aggregation to obtain heterogeneous graph embedding vectors. It combines the heterogeneous graph attention network with density peak clustering and calculates node density and relative distance through the heterogeneous graph embedding vector to divide community center nodes. Then it also uses the weight information of the node-level attention mechanism to generate a community membership matrix. The experimental results on real datasets show that the model can utilize the diversity of heterogeneous network node information to discover overlapping communities with good stability and accuracy.

Lexicon have been proved to enhance character representation to help Chinese named entity recognition (NER) model distinguish entities. Although lexicon information includes both semantic and boundary information of words, existing studies usually use only part of them and has low utilization of lexicon information. To efficiently extract dictionary features and integrate character representation, we propose a Heterogeneous Graph and Dynamic Attention Network (HGDAN), aiming at fusing contextual information and capturing dynamic associations between characters and words, thus improving the performance of Chinese NER. PGDNA uses the boundary information of the dictionary to construct a heterogeneous graph and uses the graph attention method to extract semantic information, as well as suppressing lexical noise through the gating unit. In addition, we found the traditional attention model has a non-zero attention problem that will distract the attention of the model, and proposed a simple and effective method to solve it. Experiments on the performance and inference speed of HGDAN on four Chinese datasets have proved its superiority.

Link prediction is a very important research issue in social networks analysis, and it has a very wide range of applications. Real world social networks are usually heterogeneous networks which contain rich semantic information. Meta-paths are often used to characterize this semantic information in the analysis of heterogeneous social networks. Existing methods either use only topology information or use only a single meta path to extract semantic information in the network. In this paper, we propose a link prediction method based on SEmantic Subgraphs and Graph ATtention network (SESGAT). SESGAT not only makes full use of the different semantic information contained in different semantic subgraphs, but also uses the attention mechanism to learn the different importance of different semantic subgraphs for link prediction. Experiment results on real social networks show that our approach exhibits better predictive performance than other state-of-the-art methods.

Objectives The work is to investigate the trajectory prediction for multiple types of traffic participants in signalized intersection scenarios within intelligent connected environments based on Heterogeneous Spatio-Temporal Multi-Scale Attention Network (HST-MSAN), where participants include Connected and Automated Vehicles (CAVs), Human Vehicles (HVs), cyclists, and pedestrians. Methods A novel method of trajectory prediction that integrates spatio-temporal interaction features and multi-scale map features is proposed based on HST-MSAN. The interaction model is established based on Spatio-Temporal Graph Attention Network (STGAN). The trajectory prediction model is constructed based on STGAN and Multi-Scale Squeeze-and-Excitation Network (MS-SENet). First, an STGAN is developed to differentially encode the historical trajectory of each participant, model the complex interactions, and quantify the interaction intensity among participants. Second, an MS-SENet that integrates Multi-Scale Convolutional (MSC) and a Squeeze-and-Excitation (SE) module is proposed, where multiple parallel convolutional kernels are employed to extract both local and global map features. Results The proposed model is validated through the INTERACTION dataset. The results of three-second trajectory prediction show that the average displacement error (ADE) and final displacement error (FDE) can reach to 0.17 and 0.68 m, respectively. ADE is reduced by 26.1%, 22.7%, 10.5%, and 29.2%, respectively, and FDE is reduced by 10.5%, 12.8%, 8.1%, and 5.6%, respectively, compared with prediction methods of multiple participants of Heterogeneous Edge-enhanced graph attention network (HEAT), Heterogeneous Driving Graph Transformer (HDGT), Hybrid transformer trajectory network (HTTNet), and Flock-inspired network (FN). The ablation experiments show that ADE is reduced by 22.2% and 19.0%, respectively, and FDE is reduced by 10.0% and 5.6%, respectively, compared with the models without STGAN and without MS-SENet. Conclusions This model of trajectory prediction jointly models the temporal interaction features of the participants, the spatial interaction features with the surrounding participants, and the multi-scale map features that are most suitable for the current state of the participants. By accurately predicting the future movement trajectories of the surrounding participants, CAVs can identify potential conflict points in advance, optimize their trajectory planning, and reduce the risk of traffic accidents.

Quality of Service (QoS) prediction for Web services is a hot research topic in the field of services computing. Recently, representation learning of heterogeneous networks has attracted much attention, and specifically the relationship between users and services, as a typical heterogeneous network in which heterogeneity and rich semantic information provide a new perspective for QoS prediction. This paper proposes a novel QoS Prediction scheme based on a heterogeneous graph attention network. Our method first unitizes the user’s location information to construct an attributed user-service network. Then, considering the difference between nodes and links in the latter network, we model a heterogeneous graph neural network based on a hierarchical attention machine (HGN2HIA) that includes node- and semantic-level attentions. Specifically, node-level attention aims to learn the importance between a node and its meta-path-based neighbors, while semantic-level attention learns the importance of different meta-paths. Finally, user embedding will be generated by aggregating features from meta-path-based neighbors in a hierarchical manner, used for QoS prediction. Experimental results on the public WS-Dream dataset demonstrate the superior performance of the proposed model over the current state-of-the-art methods, with NMAE and RMSE metrics reduced by at least 2.56% and 1.3%, respectively. Furthermore, the experimental results highlight that node-level attention contributes more than semantic-level. Overall, we demonstrate that introducing these attention levels improves the QoS prediction performance.

Anchor Link Prediction (ALP) across heterogeneous networks plays a pivotal role in inter-network applications. The difficulty of anchor link prediction in heterogeneous networks lies in how to consider the factors affecting nodes alignment comprehensively. In recent years, predicting anchor links based on network embedding has become the main trend. For heterogeneous networks, previous anchor link prediction methods first integrate various types of nodes associated with a user node to obtain a fusion embedding vector from global perspective, and then predict anchor links based on the similarity between fusion vectors corresponding with different user nodes. However, the fusion vector ignores effects of the local type information on user nodes alignment. To address the challenge, we propose a novel type-aware anchor link prediction across heterogeneous networks (TALP), which models the effect of type information and fusion information on user nodes alignment from local and global perspective simultaneously. TALP can solve the network embedding and type-aware alignment under a unified optimization framework based on a two-layer graph attention architecture. Through extensive experiments on real heterogeneous network datasets, we demonstrate that TALP significantly outperforms the state-of-the-art methods.

HetReGAT-FC: Heterogeneous Residual Graph Attention Network via Feature Completion

Graph neural networks have demonstrated its success in many applications on graph-structured data. Many efforts have been devoted to elaborating new network architectures and learning algorithms over the past decade. The exploration of applying ensemble learning techniques to enhance existing graph algorithms have been overlooked. In this work, we propose a simple generic bagging-based ensemble learning strategy which is applicable to any backbone graph models. We then propose two ensemble graph neural network models – Ensemble-GAT and Ensemble-HetGAT by applying the ensemble strategy to the graph attention network (GAT), and a heterogeneous graph attention network (HetGAT). We demonstrate the effectiveness of the proposed ensemble strategy on GAT and HetGAT through comprehensive experiments with four real-world homogeneous graph datasets and three real-world heterogeneous graph datasets on node classification tasks. The proposed Ensemble-GAT and Ensemble-HetGAT outperform the state-of-the-art graph neural network and heterogeneous graph neural network models on most of the benchmark datasets. The proposed ensemble strategy also alleviates the over-smoothing problem in GAT and HetGAT.

Network embedding (graph embedding) has become the focus of studying graph structure in recent years. In addition to the research on homogeneous networks and heterogeneous networks, there are also some methods to attempt to solve the problem of dynamic network embedding. However, in dynamic networks, there is no research method specifically for heterogeneous networks. Therefore, this paper proposes a heterogeneous dynamic graph attention network (HDGAN), which attempts to use the attention mechanism to take the heterogeneity and dynamics of the network into account at the same time, so as to better learn network embedding. Our method is based on three levels of attention, namely structural-level attention, semantic-level attention and time-level attention. Structural-level attention pays attention to the network structure itself, and obtains the representation of structural-level nodes by learning the attention coefficients of neighbor nodes. Semantic-level attention integrates semantic information into the representation of nodes by learning the optimal weighted combination of different meta-paths. Time-level attention is based on the time decay effect, and the time feature is introduced into the node representation by neighborhood formation sequence. Through the above three levels of attention mechanism, the final network embedding can be obtained.Through experiments on two real-world heterogeneous dynamic networks, our models have the best results, proving the effectiveness of the HDGAN model.

Accurate identification of drug-target interactions (DTIs) plays a crucial role in drug discovery. Compared with traditional experimental methods that are labor-intensive and time-consuming, computational methods are more and more popular in recent years. Conventional computational methods almost simply view heterogeneous networks which integrate diverse drug-related and target-related dataset instead of fully exploring drug and target similarities. In this paper, we propose a new method, named DTIHNC, for $\mathbf{D}$rug-$\mathbf{T}$arget $\mathbf{I}$nteraction identification, which integrates $\mathbf{H}$eterogeneous $\mathbf{N}$etworks and $\mathbf{C}$ross-modal similarities calculated by relations between drugs, proteins, diseases and side effects. Firstly, the low-dimensional features of drugs, proteins, diseases and side effects are obtained from original features by a denoising autoencoder. Then, we construct a heterogeneous network across drug, protein, disease and side-effect nodes. In heterogeneous network, we exploit the heterogeneous graph attention operations to update the embedding of a node based on information in its 1-hop neighbors, and for multi-hop neighbor information, we propose random walk with restart aware graph attention to integrate more information through a larger neighborhood region. Next, we calculate cross-modal drug and protein similarities from cross-scale relations between drugs, proteins, diseases and side effects. Finally, a multiple-layer convolutional neural network deeply integrates similarity information of drugs and proteins with the embedding features obtained from heterogeneous graph attention network. Experiments have demonstrated its effectiveness and better performance than state-of-the-art methods. Datasets and a stand-alone package are provided on Github with website https://github.com/ningq669/DTIHNC.

Circular RNAs (circRNAs), a class of non-coding RNAs characterized by their covalently closed loop structures, play active roles in diverse physiological processes through interactions with biological macromolecules. Despite the growing discovery of circRNAs enabled by high-throughput technologies, their functional annotations remain largely unexplored. This highlights the need for automated batch annotation methods to unveil the functional roles of circRNAs. In this study, we present a novel approach for predicting Gene Ontology (GO) functions associated with circRNA by leveraging self-supervised pre-training on circRNA-protein heterogeneous network. First, we construct the heterogeneous network by combining circRNA co-expression data, circRNA-protein association data, and protein-protein interaction (PPI) data. Second, we initialize the features and pseudo-labels for nodes using three graph processing methods including walking, aggregation and clustering. The initialized node features and pseudo labels, combined with protein GO annotations, are employed for heterogeneous graph pre-training. During the pre-training, the node features are learned using a heterogeneous graph attention network and the pseudo-labels are updated using the label propagation algorithm (LPA) with an attention mechanism. Finally, the initial node features are combined with those learned during pre-training to predict circRNA GO terms. Evaluation results on the independent test set reveal the superior performance of our method compared with existing approaches. Furthermore, our analysis underscores the importance of network structure and initialization strategies, highlighting the potential benefits of incorporating additional heterogeneous information and association networks.

Representation learning using Attention Network and CNN for Heterogeneous networks

Besides data sparsity and cold start, recommender systems often face the problems of selection bias and exposure bias. These problems influence the accuracy of recommendations and easily lead to overrecommendations. This paper proposes a recommendation approach based on heterogeneous network and dynamic knowledge graph (HN-DKG). The main steps include (1) determining the implicit preferences of users according to user’s cross-domain and cross-platform behaviors to form multimodal nodes and then building a heterogeneous knowledge graph; (2) Applying an improved multihead attention mechanism of the graph attention network (GAT) to realize the relationship enhancement of multimodal nodes and constructing a dynamic knowledge graph; and (3) Leveraging RippleNet to discover user’s layered potential interests and rating candidate items. In which, some mechanisms, such as user seed clusters, propagation blocking, and random seed mechanisms, are designed to obtain more accurate and diverse recommendations. In this paper, the public datasets are used to evaluate the performance of algorithms, and the experimental results show that the proposed method has good performance in the effectiveness and diversity of recommendations. On the MovieLens-1M dataset, the proposed model is 18%, 9%, and 2% higher than KGAT on F1, NDCG@10, and AUC and 20%, 2%, and 0.9% higher than RippleNet, respectively. On the Amazon Book dataset, the proposed model is 12%, 3%, and 2.5% higher than NFM on F1, NDCG@10, and AUC and 0.8%, 2.3%, and 0.35% higher than RippleNet, respectively.