Learn Node Representations Research Articles

Learning the Explainable Semantic Relations via Unified Graph Topic-Disentangled Neural Networks

Graph Neural Networks (GNNs) such as Graph Convolutional Networks (GCNs) can effectively learn node representations via aggregating neighbors based on the relation graph. However, despite a few exceptions, most of the previous work in this line does not consider the topical semantics underlying the edges, making the node representations less effective and the learned relation between nodes hard to explain. For instance, the current GNNs make us usually don’t know what is the reason for the connection of network nodes, such as the specific research topics cited in this article and the concerns among friends on social platforms. Some methods have begun to explore the extraction of relation semantics in recent related literature, but existing studies generally face two bottlenecks, i.e., either being unable to explain the mined latent relations to ensure their reasonableness and independence, or demanding the textual content of edges which is unavailable in most real-world datasets. Actually, these two issues are both crucial in practical use. In our work, we propose a novel Topic-Disentangled Graph Neural Network (TDG) to address the above two issues at the same time, which explores the relation topics from the perspective of node contents. We design an optimized graph topic module to handle node features to construct independent and explainable semantic subspaces, then the reasonable relation topics that correspond to these subspaces are assigned to each graph relation via a neighborhood routing mechanism. Our proposed model can be easily combined with related graph tasks to form an end-to-end model, to avoid the risk of deviation between node representation space and task space. To evaluate the efficiency of our model, sufficient node-related tasks are conducted on three public datasets in the experimental section. The results show the obvious superiority of TDG compared with the state-of-the-art models.

Generating post-hoc explanations for Skip-gram-based node embeddings by identifying important nodes with bridgeness

Node representation learning in a network is an important machine learning technique for encoding relational information in a continuous vector space while preserving the inherent properties and structures of the network. Recently, unsupervised node embedding methods such as DeepWalk (Perozzi et al., 2014), LINE (Tang et al., 2015), struc2vec (Ribeiro et al., 2017), PTE (Tang et al., 2015), UserItem2vec (Wu et al., 2020), and RWJBG (Li et al., 2021) have emerged from the Skip-gram model (Mikolov et al., 2013) and perform better performance in several downstream tasks such as node classification and link prediction than the existing relational models. However, providing post-hoc explanations of unsupervised embeddings remains a challenging problem because of the lack of explanation methods and theoretical studies applicable for embeddings. In this paper, we first show that global explanations to the Skip-gram-based embeddings can be found by computing bridgeness under a spectral cluster-aware local perturbation. Moreover, a novel gradient-based explanation method, which we call GRAPH-wGD, is proposed that allows the top-q global explanations about learned graph embedding vectors more efficiently. Experiments show that the ranking of nodes by scores using GRAPH-wGD is highly correlated with true bridgeness scores. We also observe that the top-q node-level explanations selected by GRAPH-wGD have higher importance scores and produce more changes in class label prediction when perturbed, compared with the nodes selected by recent alternatives, using five real-world graphs.

Temporal SIR-GN: Efficient and Effective Structural Representation Learning for Temporal Graphs

Node representation learning (NRL) generates numerical vectors (embeddings) for the nodes of a graph. Structural NRL specifically assigns similar node embeddings for those nodes that exhibit similar structural roles. This is in contrast with its proximity-based counterpart, wherein similarity between embeddings reflects spatial proximity among nodes. Structural NRL is useful for tasks such as node classification where nodes of the same class share structural roles, though there may exist a distant, or no path between them. Athough structural NRL has been well-studied in static graphs, it has received limited attention in the temporal setting. Here, the embeddings are required to represent the evolution of nodes' structural roles over time. The existing methods are limited in terms of efficiency and effectiveness: they scale poorly to even moderate number of timestamps, or capture structural role only tangentially. In this work, we present a novel unsupervised approach to structural representation learning for temporal graphs that overcomes these limitations. For each node, our approach clusters then aggregates the embedding of a node's neighbors for each timestamp, followed by a further temporal aggregation of all timestamps. This is repeated for (at most) d iterations, so as to acquire information from the d -hop neighborhood of a node. Our approach takes linear time in the number of overall temporal edges, and possesses important theoretical properties that formally demonstrate its effectiveness. Extensive experiments on synthetic and real datasets show superior performance in node classification and regression tasks, and superior scalability of our approach to large graphs.

Travel Time Distribution Estimation by Learning Representations Over Temporal Attributed Graphs

Travel time estimation is a crucial task in practical transportation applications, while providing the reliability of estimation is important in many working scenarios. Most existing studies do not consider the dynamics of traffic status for different road segments in real time, thus yielding unsatisfactory results. To address the problem, we propose to formulate the traffic network as a temporal attributed graph and perform node representation learning on it. The learned representation is capable of jointly exploiting the dynamic traffic conditions and the topology of the road network, which is then fed into a route-based spatio-temporal dependence learning module to estimate the travel time. By incorporating a distribution loss function, our proposed model is able to predict the distribution of travel time. In the meantime, we design an auxiliary local task of predicting the congestion status of each road segment, which further enhances the generalization performance of the representation learning. Extensive experiments on real-world large-scale datasets demonstrated the superiority of our method compared with the state-of-the-arts.

Community preserving adaptive graph convolutional networks for link prediction in attributed networks

Link prediction in attributed networks has attracted increasing attention recently due to its valuable real-world applications. Various related methods have been proposed, but most of them cannot effectively utilize community structure, neither can they well fuse attribute information and link information to improve the performance. Inspired by our empirical observations on how community structure affects the generation of links, we propose a novel Community Preserving Adaptive Graph Convolutional Networks (CPAGCN) method to tackle the link prediction task in attributed networks. Specifically, CPAGCN is composed of two core modules: network embedding and link prediction. Network embedding module utilizes AGCN to seamlessly fuse link information and attribute information to obtain node representations, which are simultaneously driven to preserve community structure via an appropriate community detection model. Taking these node representations as the input, link prediction module applies multilayer perception (MLP) to directly learn the prediction scores for potential links. Through combining the graph reconstruction loss with the prediction loss to train AGCN and MLP jointly, CPAGCN can learn node representations that are more beneficial to predicting links. To verify the effectiveness of CPAGCN, we conduct extensive experiments on six real-world attributed networks. The results demonstrate that CPAGCN performs better than several strong competitors in link prediction. The source code is available at https://github.com/GDM-SCNU/CPAGCN.

Dynamic Scale-free Graph Embedding via Self-attention

Graph neural networks (GNNs) have recently become increasingly popular due to their ability to learn node representations in complex graphs. Existing graph representation learning methods mainly target static graphs in Euclidean space, whereas many graphs in practical applications are dynamic and evolve continuously over time. Recent work has demonstrated that real-world graphs exhibit hierarchical properties. Unfortunately, many methods typically do not account for these latent hierarchical structures. In this work, we propose a dynamic network in hyperbolic space via self-attention, referred to as DynHAT, which leverages both the hyperbolic geometry and attention mechanism to learn node representations. More specifically, DynHAT captures hierarchical information by mapping the structural graph onto hyperbolic space, and time-varying dynamic evolution by flexibly weighting historical representations. Through extensive experiments on three real-world datasets, we show the superiority of our model in embedding dynamic graphs in hyperbolic space and competing methods in a link prediction task. In addition, our results show that embedding dynamic graphs in hyperbolic space has competitive performance when necessitating low dimensions.

Attention‐based network embedding with higher‐order weights and node attributes

AbstractNetwork embedding aspires to learn a low‐dimensional vector of each node in networks, which can apply to diverse data mining tasks. In real‐life, many networks include rich attributes and temporal information. However, most existing embedding approaches ignore either temporal information or network attributes. A self‐attention based architecture using higher‐order weights and node attributes for both static and temporal attributed network embedding is presented in this article. A random walk sampling algorithm based on higher‐order weights and node attributes to capture network topological features is presented. For static attributed networks, the algorithm incorporates first‐order to k‐order weights, and node attribute similarities into one weighted graph to preserve topological features of networks. For temporal attribute networks, the algorithm incorporates previous snapshots of networks containing first‐order to k‐order weights, and nodes attribute similarities into one weighted graph. In addition, the algorithm utilises a damping factor to ensure that the more recent snapshots allocate a greater weight. Attribute features are then incorporated into topological features. Next, the authors adopt the most advanced architecture, Self‐Attention Networks, to learn node representations. Experimental results on node classification of static attributed networks and link prediction of temporal attributed networks reveal that our proposed approach is competitive against diverse state‐of‐the‐art baseline approaches.

Self-Supervised Graph Representation Learning via Topology Transformations

We present the Topology Transformation Equivariant Representation learning, a general paradigm of self-supervised learning for node representations of graph data to enable the wide applicability of Graph Convolutional Neural Networks (GCNNs). We formalize the proposed model from an information-theoretic perspective, by maximizing the mutual information between topology transformations and node representations before and after the transformations. We derive that maximizing such mutual information can be relaxed to minimizing the cross entropy between the applied topology transformation and its estimation from node representations. In particular, we seek to sample a subset of node pairs from the original graph and flip the edge connectivity between each pair to transform the graph topology. Then, we self-train a representation encoder to learn node representations by reconstructing the topology transformations from the feature representations of the original and transformed graphs. In experiments, we apply the proposed model to the downstream node classification, graph classification and link prediction tasks, and results show that the proposed method outperforms the state-of-the-art unsupervised approaches.

Graph representation learning via redundancy reduction

In recent years, self-supervised learning methods based on mutual information maximization have achieved remarkable success on graph data tasks. However, most of them heavily rely on a large number of negative samples, which is computationally expensive. These methods also fail to extract the semantic cluster information of the data. To overcome these problems, we propose a novel self-supervised approach called Graph Representation Learing via Redundancy Reduction (GRRR) to learn node representations based on the redundancy-reduction principle. The proposed GRRR preserves as much topological information of the graph as possible, and minimizes the redundancy of representation in terms of node instance and semantic cluster information. Specifically, we first design three graph data augmentation strategies to construct two augmented views. Then, to filter the redundant information in each augmented view, we propose the self-redundancy reduction module which implements the structural reconstruction. Finally, we propose the joint redundancy reduction module to further filter undesirable information via a cross-view approach. It preserves the most essential instance features and semantic cluster information by maximizing the agreement across different views not only on node instance features but also on cluster assignments. Results on several benchmark datasets show that GRRR outperforms state-of-the-art methods in downstream node classification, link prediction, and node clustering tasks.

Are Graph Convolutional Networks With Random Weights Feasible?

Graph Convolutional Networks (GCNs), as a prominent example of graph neural networks, are receiving extensive attention for their powerful capability in learning node representations on graphs. There are various extensions, either in sampling and/or node feature aggregation, to further improve GCNs' performance, scalability and applicability in various domains. Still, there is room for further improvements on learning efficiency because performing batch gradient descent using the full dataset for every training iteration, as unavoidable for training (vanilla) GCNs, is not a viable option for large graphs. The good potential of random features in speeding up the training phase in large-scale problems motivates us to consider carefully whether GCNs with random weights are feasible. To investigate theoretically and empirically this issue, we propose a novel model termed Graph Convolutional Networks with Random Weights (GCN-RW) by revising the convolutional layer with random filters and simultaneously adjusting the learning objective with regularized least squares loss. Theoretical analyses on the model's approximation upper bound, structure complexity, stability and generalization, are provided with rigorous mathematical proofs. The effectiveness and efficiency of GCN-RW are verified on semi-supervised node classification task with several benchmark datasets. Experimental results demonstrate that, in comparison with some state-of-the-art approaches, GCN-RW can achieve better or matched accuracies with less training time cost.

GA-ENs: A novel drug–target interactions prediction method by incorporating prior Knowledge Graph into dual Wasserstein Generative Adversarial Network with gradient penalty

Bipartite graph-based drug–target interactions (DTIs) prediction methods are commonly limited by the sparse structure of the graph, resulting in acquiring suboptimal node feature representations. In addition, these defective node features will also interfere with the representation quality of corresponding edge features. To alleviate the sparsity of bipartite graph and get better node representation, according to the prior Knowledge Graph (KG), we developed a novel prediction model based on Variational Graph Auto-Encoder (VGAE) combined with our proposed dual Wasserstein Generative Adversarial Network with gradient penalty strategy (dual-WGAN-GP) for generating edge information and augmenting their representations. Specifically, GA-ENs first utilized dual-WGAN-GP to fill possible edges by a prior KG containing various molecular associations knowledge when constructing a bipartite graph of known DTIs. Moreover, we utilized the KG transfer probability matrix to redefine the drug–drug and target–target similarity matrix, thus constructing the final graph adjacent matrix. Combining graph adjacent matrix with node features, we learn node representations by VGAE and augment them by utilizing dual-WGAN-GP again, thus obtaining final edge representations. Finally, a fully connected network with three layers was designed to predict potential DTIs. Extensive experiment results show that GA-ENs has excellent performance for DTIs prediction and could be a supplement tool for practical DTIs biological screening.

Meta-path based graph contrastive learning for micro-video recommendation

Nowadays, micro-video sharing platforms have become popular tools for people creating and viewing micro-videos in daily life. The micro-video recommendation task has attracted significant attention from researchers, recently. The key to a high-quality micro-video recommendation system is learning meaningful user and micro-video representations. Recently, many graph neural networks (GNNs) are proposed to learn node representations in graphs, which could be useful for recommendation tasks. However, we argue that directly utilizing existing GNNs in micro-video recommendations is ill-posed. The reasons are: (1) most previous GNNs fail to capture the heterogeneity of heterogeneous graphs since they are designed for homogeneous graphs; (2) they overemphasize node proximity and may hurt the robustness of node embeddings. In contrast to previous work, we propose a meta-path-based graph contrastive learning network (MPGCL) that learns more meaningful user and video embeddings for recommendation. Specifically, we yield homogeneous graphs for user type and video type to better capture heterogeneity based on a well-designed meta-path-based random walk strategy. Furthermore, to learn more robust node embeddings that are less sensitive to noise, we propose a graph contrastive learning network on different types of homogeneous graphs, which maximizes the consistency between graph representations with different views. We do extensive experiments on three real-world datasets and the results show that our model can learn more meaningful embeddings for users and micro-videos and outperforms the strong baselines.

Graph-Augmentation-Free Self-Supervised Learning for Social Recommendation

Social recommendation systems can improve recommendation quality in cases of sparse user–item interaction data, which has attracted the industry’s attention. In reality, social recommendation systems mostly mine real user preferences from social networks. However, trust relationships in social networks are complex and it is difficult to extract valuable user preference information, which worsens recommendation performance. To address this problem, this paper proposes a social recommendation algorithm based on multi-graph contrastive learning. To ensure the reliability of user preferences, the algorithm builds multiple enhanced user relationship views of the user’s social network and encodes multi-view high-order relationship learning node representations using graph and hypergraph convolutional networks. Considering the effect of the long-tail phenomenon, graph-augmentation-free self-supervised learning is used as an auxiliary task to contrastively enhance node representations by adding uniform noise to each layer of encoder embeddings. Three open datasets were used to evaluate the algorithm, and it was compared to well-known recommendation systems. The experimental studies demonstrated the superiority of the algorithm.

Deep embedded clustering with distribution consistency preservation for attributed networks

Many complex systems in the real world can be characterized as attributed networks. To mine the potential information in these networks, deep embedded clustering, which obtains node representations and clusters simultaneously, has been given much attention in recent years. Under the assumption of consistency for data in different views, the cluster structure of network topology and that of node attributes should be consistent for an attributed network. However, many existing methods ignore this property, even though they separately encode node representations from network topology and node attributes and cluster nodes on representation vectors learned from one of the views. Therefore, in this study, we propose an end-to-end deep embedded clustering model for attributed networks. It utilizes graph autoencoder and node attribute autoencoder to learn node representations and cluster assignments. In addition, a distribution consistency constraint is introduced to maintain the latent consistency of cluster distributions in two views. Extensive experiments on several datasets demonstrate that the proposed model achieves significantly better or competitive performance compared with the state-of-the-art methods. The source code can be found at https://github.com/Zhengymm/DCP.

Geometric Interaction Graph Neural Network for Predicting Protein-Ligand Binding Affinities from 3D Structures (GIGN).

Predicting protein-ligand binding affinities (PLAs) is a core problem in drug discovery. Recent advances have shown great potential in applying machine learning (ML) for PLA prediction. However, most of them omit the 3D structures of complexes and physical interactions between proteins and ligands, which are considered essential to understanding the binding mechanism. This paper proposes a geometric interaction graph neural network (GIGN) that incorporates 3D structures and physical interactions for predicting protein-ligand binding affinities. Specifically, we design a heterogeneous interaction layer that unifies covalent and noncovalent interactions into the message passing phase to learn node representations more effectively. The heterogeneous interaction layer also follows fundamental biological laws, including invariance to translations and rotations of the complexes, thus avoiding expensive data augmentation strategies. GIGN achieves state-of-the-art performance on three external test sets. Moreover, by visualizing learned representations of protein-ligand complexes, we show that the predictions of GIGN are biologically meaningful.

Privacy-Preserving Federated Graph Neural Network Learning on Non-IID Graph Data

Since the concept of federated learning (FL) was proposed by Google in 2017, many applications have been combined with FL technology due to its outstanding performance in data integration, computing performance, privacy protection, etc. However, most traditional federated learning-based applications focus on image processing and natural language processing with few achievements in graph neural networks due to the graph’s nonindependent identically distributed (IID) nature. Representation learning on graph-structured data generates graph embedding, which helps machines understand graphs effectively. Meanwhile, privacy protection plays a more meaningful role in analyzing graph-structured data such as social networks. Hence, this paper proposes PPFL-GNN, a novel privacy-preserving federated graph neural network framework for node representation learning, which is a pioneer work for graph neural network-based federated learning. In PPFL-GNN, clients utilize a local graph dataset to generate graph embeddings and integrate information from other collaborative clients to utilize federated learning to produce more accurate representation results. More importantly, by integrating embedding alignment techniques in PPFL-GNN, we overcome the obstacles of federated learning on non-IID graph data and can further reduce privacy exposure by sharing preferred information.

A quantum‐like approach for text generation from knowledge graphs

AbstractRecent text generation methods frequently learn node representations from graph‐based data via global or local aggregation, such as knowledge graphs. Since all nodes are connected directly, node global representation encoding enables direct communication between two distant nodes while disregarding graph topology. Node local representation encoding, which captures the graph structure, considers the connections between nearby nodes but misses out onlong‐range relations. A quantum‐like approach to learning better‐contextualised node embeddings is proposed using a fusion model that combines both encoding strategies. Our methods significantly improve on two graph‐to‐text datasets compared to state‐of‐the‐art models in various experiments.

Learning Graph Representations With Maximal Cliques.

Non-Euclidean property of graph structures has faced interesting challenges when deep learning methods are applied. Graph convolutional networks (GCNs) can be regarded as one of the successful approaches to classification tasks on graph data, although the structure of this approach limits its performance. In this work, a novel representation learning approach is introduced based on spectral convolutions on graph-structured data in a semisupervised learning setting. Our proposed method, COnvOlving cLiques (COOL), is constructed as a neighborhood aggregation approach for learning node representations using established GCN architectures. This approach relies on aggregating local information by finding maximal cliques. Unlike the existing graph neural networks which follow a traditional neighborhood averaging scheme, COOL allows for aggregation of densely connected neighboring nodes of potentially differing locality. This leads to substantial improvements on multiple transductive node classification tasks.

An Approch for Representation of Node Using Graph Transformer Networks

Abstract: In representation learning on graphs, graph neural networks (GNNs) have been widely employed and have attained cutting-edge performance in tasks like node categorization and link prediction. However, the majority of GNNs now in use are made to learn node representations on homogenous and fixed graphs. The limits are particularly significant when learning representations on a network that has been incorrectly described or one that is heterogeneous, or made up of different kinds of nodes and edges. This study proposes Graph Transformer Networks (GTNs), which may generate new network structures by finding valuable connections between disconnected nodes in the original graph and learning efficient node representation on the new graphs end-to-end. A basic layer of GTNs called the Graph Transformer layer learns a soft selection of edge types and composite relations to produce meaningful multi-hop connections known as meta-paths. This research demonstrates that GTNs can learn new graph structures from data and tasks without any prior domain expertise and that they can then use convolution on the new graphs to provide effective node representation. GTNs outperformed state-of-the-art approaches that need predefined meta-paths from domain knowledge in all three benchmark node classification tasks without the use of domain-specific graph pre-processing.

AIC-GNN: Adversarial information completion for graph neural networks

Graph neural networks (GNNs) have attracted extensive attention due to their demonstrated powerful performance in various graph mining tasks. The implicit assumption of GNNs being able to work is that all nodes have adequate information for meaningful aggregation. However, this is not easy to satisfy because the degrees of a real-world graph commonly follow the power-law distribution, where most nodes belong to low-degree nodes with limited neighborhoods. In this paper, we argue that to make GNNs better handle the low-degree node representation learning is the key to solving the above problem and propose a pluggable framework named Adversarial Information Completion Graph Neural Networks (AIC-GNN). It introduces a novel Graph Information Generator to fit adaptively the node missing information distribution. Then, the Graph Embedding Discriminator distinguishes between the node embeddings with the ideal information and the node embeddings after information completion. The representational capacity of the model is enhanced by adversarial training between the Generator and Discriminator. Meanwhile, the dual node embedding alignment mechanisms are employed to guide the high quality of predicted information. Extensive experiments demonstrate that AIC-GNN outperforms state-of-the-art methods on four real-world graphs.