Graph Embedding Methods Research Articles

UMP-MG: A Uni-directed Message-Passing Multi-label Generation Model for Hierarchical Text Classification

Hierarchical Text Classification (HTC) is a formidable task which involves classifying textual descriptions into a taxonomic hierarchy. Existing methods, however, have difficulty in adequately modeling the hierarchical label structures, because they tend to focus on employing graph embedding methods to encode the hierarchical structure while disregarding the fact that the HTC labels are rooted in a tree structure. This is significant because, unlike a graph, the tree structure inherently has a directive that ordains information flow from one node to another—a critical factor when applying graph embedding to the HTC task. But in the graph structure, message-passing is undirected, which will lead to the imbalance of message transmission between nodes when applied to HTC. To this end, we propose a unidirectional message-passing multi-label generation model for HTC, referred to as UMP-MG. Instead of viewing HTC as a classification problem as previous methods have done, this novel approach conceptualizes it as a sequence generation task, introducing prior hierarchical information during the decoding process. This further enables the blocking of information flow in one direction to ensure that the graph embedding method is better suited for the HTC task and thus resulted in the enhanced tree structure representation. Results obtained through experimentation on both the public WOS dataset and an E-commerce user intent classification dataset demonstrate that our proposed model can achieve superlative results.

The Use of Attentive Knowledge Graph Perceptual Propagation for Improving Recommendations

Collaborative filtering (CF) usually suffers from data sparsity and cold starts. Knowledge graphs (KGs) are widely used to improve recommendation performance. To verify that knowledge graphs can further alleviate the above problems, this paper proposes an end-to-end framework that uses attentive knowledge graph perceptual propagation for recommendations (AKGP). This framework uses a knowledge graph as a source of auxiliary information to extract user–item interaction information and build a sub-knowledge base. The fusion of structural and contextual information is used to construct fine-grained knowledge graphs via knowledge graph embedding methods and to generate initial embedding representations. Through multi-layer propagation, the structured information and historical preference information are embedded into a unified vector space, and the potential user–item vector representation is expanded. This article used a knowledge perception attention module to achieve feature representation, and finally, the model was optimized using the stratified sampling joint learning method. Compared with the baseline model using MovieLens-1M, Last-FM, Book-Crossing and other data sets, the experimental results demonstrate that the model outperforms state-of-the-art KG-based recommendation methods, and the shortcomings of the existing model are improved. The model was applied to product design data and historical maintenance records provided by an automotive parts manufacturing company. The predictions of the recommended system are matched to the product requirements and possible failure records. This helped reduce costs and increase productivity, helping the company to quickly determine the cause of failures and reduce unplanned downtime.

Towards Robust Knowledge Graph Embedding via Multi-Task Reinforcement Learning

Nowadays, Knowledge graphs (KGs) have been playing a pivotal role in AI-related applications. Despite the large sizes, existing KGs are far from complete and comprehensive. In order to continuously enrich KGs, automatic knowledge construction and update mechanisms are usually utilized, which inevitably bring in plenty of noise. However, most existing knowledge graph embedding (KGE) methods assume that all the triple facts in KGs are correct, and project both entities and relations into a low-dimensional space without considering noise and knowledge conflicts. This will lead to low-quality and unreliable representations of KGs. To this end, in this paper, we propose a general multi-task reinforcement learning framework, which can greatly alleviate the noisy data problem. In our framework, we exploit reinforcement learning for choosing high-quality knowledge triples while filtering out the noisy ones. Also, in order to take full advantage of the correlations among semantically similar relations, the triple selection processes of similar relations are trained in a collective way with multi-task learning. Moreover, we extend popular KGE models TransE, DistMult, ConvE and RotatE with the proposed framework. Finally, the experimental validation shows that our approach is able to enhance existing KGE models and can provide more robust representations of KGs in noisy scenarios.

Unsupervised Graph Embedding via Adaptive Graph Learning.

Graph autoencoders (GAEs) are powerful tools in representation learning for graph embedding. However, the performance of GAEs is very dependent on the quality of the graph structure, i.e., of the adjacency matrix. In other words, GAEs would perform poorly when the adjacency matrix is incomplete or be disturbed. In this paper, two novel unsupervised graph embedding methods, unsupervised graph embedding via adaptive graph learning (BAGE) and unsupervised graph embedding via variational adaptive graph learning (VBAGE) are proposed. The proposed methods expand the application range of GAEs on graph embedding, i.e, on the general datasets without graph structure. Meanwhile, the adaptive learning mechanism can initialize the adjacency matrix without being affected by the parameter. Besides that, the latent representations are embedded with the Laplacian graph structure to preserve the topology structure of the graph in the vector space. Moreover, the adjacency matrix can be self-learned for better embedding performance when the original graph structure is incomplete. With adaptive learning, the proposed method is much more robust to the graph structure. Experimental studies on several datasets validate our design and demonstrate that our methods outperform baselines by a wide margin in node clustering, node classification, link prediction, and graph visualization tasks.

A multi-scale graph embedding method via multiple corpora

Graph embedding aims at learning continuous vector representations for graphs which is crucial for graph analytics. Natural Language Process (NLP)-based graph embedding methods build corpus for graph data by treating substructures as words and then use NLP models to learn graph embeddings. However, the size difference and data redundancy among substructures are less explored in the built corpora. To mitigate this problem, we propose an unsupervised multi-scale graph embedding method. To be a specific, we first build multiple graph corpora for a graph dataset, where each corpus only contains substructures of specific granularity. Then, we extend a document embedding model to each graph corpus to obtain graph embeddings of different scales. At last, we obtain the final multi-scale embedding of a graph by pooling its multiple embeddings. Comprehensive experiments on real graph datasets indicate that the proposed method obtains competitive results with state-of-the-arts, and is superior to some classic graph kernels and graph embedding methods on six out of ten benchmark datasets.

Relation-Aware Weighted Embedding for Heterogeneous Graphs

Heterogeneous graph embedding, aiming to learn the low-dimensional representations of nodes, is effective in many tasks, such as link prediction, node classification, and community detection. Most existing graph embedding methods conducted on heterogeneous graphs treat the heterogeneous neighbours equally. Although it is possible to get node weights through attention mechanisms mainly developed using expensive recursive message-passing, they are difficult to deal with large-scale networks. In this paper, we propose R-WHGE, a relation-aware weighted embedding model for heterogeneous graphs, to resolve this issue. R-WHGE comprehensively considers structural information, semantic information, meta-paths of nodes and meta-path-based node weights to learn effective node embeddings. More specifically, we first extract the feature importance of each node and then take the nodes’ importance as node weights. A weighted random walks-based embedding learning model is proposed to generate the initial weighted node embeddings according to each meta-path. Finally, we feed these embeddings to a relation-aware heterogeneous graph neural network to generate compact embeddings of nodes, which captures relation-aware characteristics. Extensive experiments on real-world datasets demonstrate that our model is competitive against various state-of-the-art methods.

Enhancing Drug-Drug Interaction Prediction Using Deep Attention Neural Networks.

Drug-drug interactions are one of the main concerns in drug discovery. Accurate prediction of drug-drug interactions plays a key role in increasing the efficiency of drug research and safety when multiple drugs are co-prescribed. With various data sources that describe the relationships and properties between drugs, the comprehensive approach that integrates multiple data sources would be considerably effective in making high-accuracy prediction. In this paper, we propose a Deep Attention Neural Network based Drug-Drug Interaction prediction framework, abbreviated as DANN-DDI, to predict unobserved drug-drug interactions. First, we construct multiple drug feature networks and learn drug representations from these networks using the graph embedding method; then, we concatenate the learned drug embeddings and design an attention neural network to learn representations of drug-drug pairs; finally, we adopt a deep neural network to accurately predict drug-drug interactions. The experimental results demonstrate that our model DANN-DDI has improved prediction performance compared with state-of-the-art methods. Moreover, the proposed model can predict novel drug-drug interactions and drug-drug interaction-associated events.

Distributed Graph Embedding with Information-Oriented Random Walks

Graph embedding maps graph nodes to low-dimensional vectors, and is widely adopted in machine learning tasks. The increasing availability of billion-edge graphs underscores the importance of learning efficient and effective embeddings on large graphs, such as link prediction on Twitter with over one billion edges. Most existing graph embedding methods fall short of reaching high data scalability. In this paper, we present a general-purpose, distributed, information-centric random walk-based graph embedding framework, DistGER, which can scale to embed billion-edge graphs. DistGER incrementally computes information-centric random walks. It further leverages a multi-proximity-aware, streaming, parallel graph partitioning strategy, simultaneously achieving high local partition quality and excellent workload balancing across machines. DistGER also improves the distributed Skip-Gram learning model to generate node embeddings by optimizing the access locality, CPU throughput, and synchronization efficiency. Experiments on real-world graphs demonstrate that compared to state-of-the-art distributed graph embedding frameworks, including KnightKing, DistDGL, and Pytorch-BigGraph, DistGER exhibits 2.33×--129× acceleration, 45% reduction in cross-machines communication, and >10% effectiveness improvement in downstream tasks.

Multi-view contrastive learning for multilayer network embedding

Graph embedding aims to learn low-dimension node representations which can be applied to various downstream tasks, such as node clustering and node classification. When it comes to heterogeneous graphs, things get more challenging as the graph contains richer semantic and structural information. Most existing heterogeneous graph embedding methods usually convert the graph into several homogenous graphs, and then use the manual label information to train the model. However, Graph neural networks (GNNs) mostly require task-dependent labels to learn representations, which is even more costly for heterogeneous graphs. Furthermore, these models treat each converted homogenous graph independently while ignoring the inter-layer informative relational clues among the multi-graphs. Inspired by recent advances of Deep Graph Infomax and multi-view learning in homogeneous graph embedding, we propose an unsupervised graph neural network called CL-MNE that adopts multi-view contrastive learning and multilayer information fusion for heterogeneous graph embedding. The proposed model learns rich local and global graph information by maximizing the local-global mutual information from adjacency matrix and diffusion matrix two different views. Moreover, a variant model CL-MNEreg, which provides embedding information of all graph layers, is constructed by addingg a consensus regularization to the CL-MNE. Experiments of node clustering and classification tasks on mainstream datasets demonstrate the effectiveness of the proposed two models.

On augmenting topological graph representations for attributed graphs

Graph representations based on embedding methods allow for easier analysis of the network structure and can be used for a variety of tasks, such as link prediction and node classification. These methods have been shown to be effective in a variety of settings and have become an important tool in the field of graph learning. These methods are easy to implement, and their predictions yield interpretable results. However, most graph embedding methods rely solely on graph structural information and do not consider node/edge attributes, limiting their applicability. In this paper, we propose graph-theoretic designs to incorporate node and edge attributes within the topology, enabling graph-embedding methods to seamlessly work on attributed graphs. To find ideal representation for a given attributed graph, we propose augmenting special subgraph structures within original network. We discuss the potential challenges of the proposed approach and prove some of its theoretical limitations. We test the efficacy of our approach by comparing state-of-the-art graph classification models on 15 standard bioinformatics datasets. We observe an encouraging improvement of up to 5% in classification accuracy on the augmented graphs compared to the results on the original graphs.

Hyper-node Relational Graph Attention Network for Multi-modal Knowledge Graph Completion

Knowledge graphs often suffer from incompleteness, and knowledge graph completion (KGC) aims at inferring the missing triplets through knowledge graph embedding from known factual triplets. However, most existing knowledge graph embedding methods only use the relational information of knowledge graph and treat the entities and relations as IDs with simple embedding layer, ignoring the multi-modal information among triplets, such as text descriptions, images, etc. In this work, we propose a novel network to incorporate different modal information with graph structure information for more precise representation of multi-modal knowledge graph, termed as hyper-node relational graph attention (HRGAT) network. In HRGAT, we use low-rank multi-modal fusion to model the intra-modality and inter-modality dynamics, which transforms the original knowledge graph to a hyper-node graph. Then, relational graph attention (RGAT) network is used, which contains relation-specific attention and entity-relation fusion operation to capture the graph structure information. Finally, we aggregate the updated multi-modal information and graph structure information to generate the final embeddings of knowledge graph to achieve KGC. By exploring multi-modal information and graph structure information, HRGAT embraces faster convergence speed and achieves the state-of-the-art for KGC on the standard datasets. Implementation code is available at https://github.com/broliang/HRGAT.

Detection of autism spectrum disorder using graph representation learning algorithms and deep neural network, based on fMRI signals.

Can we apply graph representation learning algorithms to identify autism spectrum disorder (ASD) patients within a large brain imaging dataset? ASD is mainly identified by brain functional connectivity patterns. Attempts to unveil the common neural patterns emerged in ASD are the essence of ASD classification. We claim that graph representation learning methods can appropriately extract the connectivity patterns of the brain, in such a way that the method can be generalized to every recording condition, and phenotypical information of subjects. These methods can capture the whole structure of the brain, both local and global properties. The investigation is done for the worldwide brain imaging multi-site database known as ABIDE I and II (Autism Brain Imaging Data Exchange). Among different graph representation techniques, we used AWE, Node2vec, Struct2vec, multi node2vec, and Graph2Img. The best approach was Graph2Img, in which after extracting the feature vectors representative of the brain nodes, the PCA algorithm is applied to the matrix of feature vectors. The classifier adapted to the features embedded in graphs is an LeNet deep neural network. Although we could not outperform the previous accuracy of 10-fold cross-validation in the identification of ASD versus control patients in this dataset, for leave-one-site-out cross-validation, we could obtain better results (our accuracy: 80%). The result is that graph embedding methods can prepare the connectivity matrix more suitable for applying to a deep network.

How to Realize Efficient and Scalable Graph Embeddings via an Entropy-Driven Mechanism

Graph embedding is becoming widely adopted as an efficient way to learn graph representations required to solve graph analytics problems. However, most existing graph embedding methods, owing to computation-efficiency challenges for large-scale graphs, generally employ a one-size-fits-all strategy to extract information, resulting in a large amount of redundant or inaccurate representations. In this work, we propose HuGE+, an efficient and scalable graph embedding method enabled by an entropy-driven mechanism. Specifically, HuGE+ leverages hybrid-property heuristic random walk to capture node features, which considers both information content of nodes and the number of common neighbors in each walking step. More importantly, to guarantee information effectiveness of sampling, HuGE+ adopts two heuristic methods to decide the random walk length and the number of walks per node, respectively. Extensive experiments on real-world graphs demonstrate that HuGE+ achieves both efficiency and performance advantages over recent popular graph embedding approaches. For three downstream graph tasks, our approach not only offers >10% average gains, but also exhibits 23×–127× speedup over existing sampling-based methods. In addition, HuGE+ significantly reduces memory footprint by an average of 68.9%, facilitating training for billion-node-scale graph embeddings.

Attention Knowledge Network Combining Explicit and Implicit Information

The existing knowledge graph embedding (KGE) method has achieved good performance in recommendation systems. However, the relevancy degree among entities reduces gradually along the spread in the knowledge graph. Focusing on the explicit and implicit relationships among entities, this paper proposes an attention knowledge network combining explicit and implicit information (AKNEI) to effectively capture and exactly describe the correlation between entities in the knowledge graph. First, we design an information-sharing layer (ISL) to realize information sharing between projects and entities through implicit interaction. We innovatively propose a cross-feature fusion module to extract high-order feature information in the model. At the same time, this paper uses the attention mechanism to solve the problem of the decline of information relevance in the process of knowledge graph propagation. Finally, the features of KGE and cross feature fusion module are integrated into the end-to-end learning framework, the item information in the recommendation task and the knowledge graph entity information are interacted implicitly and explicitly, and the characteristics between them are automatically learned. We performed extensive experiments on multiple public datasets that include movies, music, and books. According to the experimental results, our model has a great improvement in performance compared with the latest baseline.

PeerRemove: An adaptive node removal strategy for P2P botnet based on deep reinforcement learning

Botnets have become one of the major intrusion threats to cybersecurity. P2P botnets have high concealment and resilience because of their distributed structure, which are difficult to be completely dismantled and destroyed. Existing methods based on traffic statistics and vulnerability tracing cannot effectively solve the problem of P2P botnet disintegration. Although P2P networks emphasize that each node is peer-to-peer, the difference in processing power, resource distribution, and node bandwidth can lead to a certain heterogeneity. The critical nodes bridge the underlying bot nodes and the upper control server. Traditional methods for ranking the importance of nodes mainly relies on classical graph-theoretic feature statistics method, such as degree, betweenness, clustering coefficient, feature vector centrality, PageRank, etc. In this paper, botnet defense strategies are investigated from the perspective of complex network graph theory, and graph embedding and deep reinforcement learning combination optimization methods are adopted to handle the critical nodes identification problem of P2P botnets. Then, a novel adaptive node removal model called PeerRemove is proposed. The model uses Structure2vec graph embedding to characterize the network structure information as a low-dimensional embedding space, and it uses n-step Q-learning to train the model to learn complex topological patterns to find the critical nodes that effectively disintegrate the network. To evaluate the effectiveness of the proposed method, the Area Under the Curve (AUC) of the Largest Connected Component (LCC) size during node removal is used as an evaluation indicator, and six different types real or synthetic P2P botnets are selected, namely Sality, ZeroAccess, NSIS, Mozi, Gnutella, and Peer sampling service. Experiments are conducted on many real and model networks with node sizes reaching thousands and tens of thousands, and our method is compared with five classical static or dynamic node attack methods of HAD, PageRank, CI, BPD, and HPRA. The experimental results show that the overall AUC curve of the PeerRemove method is lower than that of the benchmark method, which can minimize botnet resiliency at a small cost. The proposed method is superior to the existing node removal methods and shows good robustness and feasibility. To demonstrate the generality of this method, it is tested on a centralized topological dataset and good experimental results are obtained.

CSGVD: A deep learning approach combining sequence and graph embedding for source code vulnerability detection

In order to secure software, it is critical to detect potential vulnerabilities. The performance of traditional static vulnerability detection methods is limited by predefined rules, which rely heavily on the expertise of developers. Existing deep learning-based vulnerability detection models usually use only a single sequence or graph embedding approach to extract vulnerability features. Sequence embedding-based models ignore the structured information inherent in the code, and graph embedding-based models lack effective node and graph embedding methods. As a result, we propose a novel deep learning-based approach, CSGVD (Combining Sequence and Graph embedding for Vulnerability Detection), which considers function-level vulnerability detection as a graph binary classification task. Firstly, we propose a PE-BL module, which inherits and enhances the knowledge from the pre-trained language model. It extracts the code’s local semantic features as node embedding in the control flow graph by using sequence embedding. Secondly, CSGVD uses graph neural networks to extract the structured information of the graph. Finally, we propose a mean biaffine attention pooling, M-BFA, to better aggregate node information as a graph’s feature representation. The experimental results show that CSGVD outperforms the existing state-of-the-art models and obtains 64.46% accuracy on the real-world benchmark dataset from CodeXGLUE for vulnerability detection.

Relational data embeddings for feature enrichment with background information

For many machine-learning tasks, augmenting the data table at hand with features built from external sources is key to improving performance. For instance, estimating housing prices benefits from background information on the location, such as the population density or the average income. However, this information must often be assembled across many tables, requiring time and expertise from the data scientist. Instead, we propose to replace human-crafted features by vectorial representations of entities (e.g. cities) that capture the corresponding information. We represent the relational data on the entities as a graph and adapt graph-embedding methods to create feature vectors for each entity. We show that two technical ingredients are crucial: modeling well the different relationships between entities, and capturing numerical attributes. We adapt knowledge graph embedding methods that were primarily designed for graph completion. Yet, they model only discrete entities, while creating good feature vectors from relational data also requires capturing numerical attributes. For this, we introduce KEN: Knowledge Embedding with Numbers. We thoroughly evaluate approaches to enrich features with background information on 7 prediction tasks. We show that a good embedding model coupled with KEN can perform better than manually handcrafted features, while requiring much less human effort. It is also competitive with combinatorial feature engineering methods, but much more scalable. Our approach can be applied to huge databases, creating general-purpose feature vectors reusable in various downstream tasks.

Road-Type Classification with Deep AutoEncoder.

Machine learning algorithms are among the driving forces towards the success of intelligent road network systems design. Such algorithms allow for the design of systems that provide safe road usage, efficient infrastructure, and traffic flow management. One such application of machine learning in intelligent road networks is classifying different road network types that provide useful traffic information to road users. We propose a deep autoencoder model for representation learning to classify road network types. Each road segment node is represented as a feature vector. Unlike existing graph embedding methods that perform road segment embedding using the neighbouring road segments, the proposed method performs embedding directly on the road segment vectors. The proposed method performs embedding directly on the road segment vectors. Comparison with state-of-the-art graph embedding methods show that the proposed method outperforms graph convolution networks, GraphSAGE-MEAN, graph attention networks, and graph isomorphism network methods, and it achieves similar performance to GraphSAGE-MAXPOOL.

Adaptive Reference-Related Graph Embedding for Hyperspectral Anomaly Detection

Graph embedding provides an effective way to reveal intrinsic feature of high-dimensional data on the foundation of preserving topological properties. Under the framework of graph embedding, the hyperspectral image can be represented by a weighted graph, where pixels and similarities among them are treated as vertices and edge weights, respectively. In this article, an adaptive reference-related graph embedding (ARGE) method is proposed to efficaciously obtain the low-dimensional feature and improve computational efficiency. The ARGE method is composed of two primary processes. The key to connecting these two processes is reference vertices set, which is the abstraction of graph topological features. Firstly, the reference vertices are adaptively selected through a three-step adaptive reference set selection (ARSS) algorithm. Secondly, the original high-dimensional graph is embedded as a low-dimensional graph through preserving the reference-related structure. Specifically, the pairwise similarities between vertices and reference vertices are preserved in embedding space. In addition, a new hybrid dissimilarity measure of Rao distance and spectral information divergence (RD-SID) is designed to depict the spectral difference between pixels. To evaluate the effectiveness of the proposed method, the obtained low-dimensional feature is fed into the anomaly detector to detect anomalous pixels. The experimental results on five real and one synthetic hyperspectral datasets demonstrate the superiority of the proposed ARGE method over the compared feature extraction methods.

A self-attentive model for tracing knowledge and engagement in parallel

• An effective model for tracing knowledge state and engagement in parallel is designed. • An adaptive heterogeneous graph embedding methods is proposed to extract the latent information of question-concept graph. • An engagement encoder is proposed model students’ engage state according to their learning behaviors. • State-of-the-art results on the benchmark datasets for knowledge tracing task. Knowledge tracing refers to the task of modeling students’ evolving knowledge state according to their historical learning trajectories. Although many research has devoted to exploiting the latent information of question-concept bipartite graph, most of them need to split the graph into question-question and concept-concept graph to obtain questions’ and concepts’ representation respectively. This operation will cause information loss of original bipartite graph. To this end, an adaptive heterogeneous graph embedding module is proposed to utilize the latent information of graph adequately. Besides, many works ignore that students’ engagement could also impact students’ learning performance. Therefore, a student engagement encoder and a student knowledge state encoder are designed to capture students’ engagement and knowledge respectively. To be specific, the question-concept bipartite graph is input to graph embedding module directly to obtain each question’s and concept’s representation, the two encoders use these representation and students’ learning history to encode their engagement and knowledge state.