Learn Node Representations Research Articles

Inductive Subgraph Embedding for Link Prediction

AbstractLink prediction, which aims to infer missing edges or predict future edges based on currently observed graph connections, has emerged as a powerful technique for diverse applications such as recommendation, relation completion, etc. While there is rich literature on link prediction based on node representation learning, direct link embedding is relatively less studied and less understood. One common practice in previous work characterizes a link by manipulate the embeddings of its incident node pairs, which is not capable of capturing effective link features. Moreover, common link prediction methods such as random walks and graph auto-encoder usually rely on full-graph training, suffering from poor scalability and high resource consumption on large-scale graphs. In this paper, we propose Inductive Subgraph Embedding for Link Prediciton (SE4LP) — an end-to-end scalable representation learning framework for link prediction, which utilizes the strong correlation between central links and their neighborhood subgraphs to characterize links. We sample the “link-centric induced subgraphs” as input, with a subgraph-level contrastive discrimination as pretext task, to learn the intrinsic and structural link features via subgraph classification. Extensive experiments on five datasets demonstrate that SE4LP has significant superiority in link prediction in terms of performance and scalability, when compared with state-of-the-art methods. Moreover, further analysis demonstrate that introducing self-supervision in link prediction can significantly reduce the dependence on training data and improve the generalization and scalability of model.

A graph convolutional neural network model based on fused multi-subgraph as input and fused feature information as output

The graph convolution neural network (GCN)-based node classification model tackles the challenge of classifying nodes in graph data through learned feature representations. However, most existing graph neural networks primarily focus on the same type of edges, which might not accurately reflect the intricate real-world graph structure. This paper introduces a novel graph neural network model, MF-GCN, which integrates subgraphs with various edge types as input and combines feature information from each graph convolutional neural network layer to produce the final output. This model learns node feature representations by separately feeding subgraphs with different edge types into the graph convolutional layer. It then computes the weight vectors for fusing various edge type subgraphs based on the learned node features. Additionally, to efficiently extract feature information, the outputs of each graph convolution layer, without an activation function, are weighted and summed to obtain the final node features. This approach resolves the challenges of determining fusion weights and effectively extracting feature information during subgraph fusion. Experimental results show that the proposed model significantly improves performance on all three datasets, highlighting its effectiveness in node representation learning tasks.

Multi-agent Attacks for Black-box Social Recommendations

The rise of online social networks has facilitated the evolution of social recommender systems, which incorporate social relations to enhance users’ decision-making process. With the great success of Graph Neural Networks (GNNs) in learning node representations, GNN-based social recommendations have been widely studied to model user-item interactions and user-user social relations simultaneously. Despite their great successes, recent studies have shown that these advanced recommender systems are highly vulnerable to adversarial attacks, in which attackers can inject well-designed fake user profiles to disrupt recommendation performances. While most existing studies mainly focus on targeted attacks to promote target items on vanilla recommender systems, untargeted attacks to degrade the overall prediction performance are less explored on social recommendations under a black-box scenario. To perform untargeted attacks on social recommender systems, attackers can construct malicious social relationships for fake users to enhance the attack performance. However, the coordination of social relations and item profiles is challenging for attacking black-box social recommendations. To address this limitation, we first conduct several preliminary studies to demonstrate the effectiveness of cross-community connections and cold-start items in degrading recommendations performance. Specifically, we propose a novel framework MultiAttack based on multi-agent reinforcement learning to coordinate the generation of cold-start item profiles and cross-community social relations for conducting untargeted attacks on black-box social recommendations. Comprehensive experiments on various real-world datasets demonstrate the effectiveness of our proposed attacking framework under the black-box setting.

Disentangled contrastive learning for fair graph representations

Graph Neural Networks (GNNs) play a key role in efficiently learning node representations of graph-structured data through message passing, but their predictions are often correlated with sensitive attributes and thus lead to potential discrimination against some groups. Given the increasingly widespread applications of GNNs, solutions are required urgently to prevent algorithmic discrimination associated with GNNs, to protect the rights of vulnerable groups and to build trustworthy artificial intelligence. To learn the fair node representations of graphs, we propose a novel framework, the Fair Disentangled Graph Neural Network (FDGNN). With the proposed FDGNN framework, we enhance data diversity by generating instances that have identical sensitivity values but different adjacency matrices through data augmentation. Additionally, we design a counterfactual augmentation strategy for constructing instances with varying sensitive values while preserving the same adjacency matrices, thereby balancing the distribution of sensitive values across different groups. Subsequently, we employ a disentangled contrastive learning strategy to acquire disentangled representations of non-sensitive attributes such that sensitive information does not affect the prediction of node information. Finally, the learned fair representations of non-sensitive attributes are employed for building a fair predictive model. Extensive experiments on three real-world datasets demonstrate that FDGNN yields the best fairness predictions compared to the baseline methods. Additionally, the results demonstrate the potential of disentanglement in learning fair representations.

Contrastive learning for fair graph representations via counterfactual graph augmentation

Graph neural networks (GNNs) have exhibited excellent performance in graph representation learning. However, GNNs might inherit biases from the data, leading to discriminatory predictions. Existing study mainly concentrates on attaining fairness through counterfactuals related to node attributes, overlooking the causal impact of bias in the graph structure. Herein, we introduce a novel framework called fair contrastive learning based on counterfactual graph augmentation (FCLCA), aimed at learning counterfactual fairness by mitigating graph structure bias. FCLCA first generates two counterfactual graphs through structural augmentation. Next, we maximize the consistency between representations produced by nodes in these two counterfactual graphs using contrastive learning. In addition, FCLCA uses adversarial debiasing learning to further reduce the influence of sensitive attributes on the learned node representations. Finally, an optimized training strategy is used for contrastive learning to enhance the learning of counterfactual fairness. Comprehensive experiments conducted on four real-world datasets proved the effectiveness of FCLCA in balancing classification performance and fairness.

Adaptive denoising graph contrastive learning with memory graph attention for recommendation

Graph contrastive learning has emerged as a powerful technique for dealing with graph noise and mining latent information in networks, that has been widely applied in GNN-based collaborative filtering. Traditional graph contrastive learning methods commonly generate multiple augmented views, and then learn node representations by maximizing the consistency between these views. However, on one hand, manual view construction methods necessitate expert knowledge and a trial-and-error process. On the other hand, adaptive view construction methods require decoders which results in increased training costs. To address the aforementioned limitations, in this paper, we propose the Adaptive Denoising Graph Contrastive Learning with Memory Graph Attention for Recommendation (ADGA) framework. Firstly, we introduce the memory graph attention mechanism to capture node attention during multi-hop information aggregation. Then, unlike previous methods that required additional node representations to generate views, ADGA proposes, for the first time, directly using attention to adaptively generate structure-aware contrastive learning views. It reduces the training cost of the model and improves the cross-view consistency of node representations, that offers a new paradigm for adaptive graph contrastive learning. Experimental results on three real-world datasets demonstrate that ADGA achieves state-of-the-art performance in recommendation tasks. The code is available at https://github.com/Andrewsama/ADGA.

Graph pooling in graph neural networks: methods and their applications in omics studies

Graph neural networks (GNNs) process the graph-structured data using neural networks and have proven successful in various graph processing tasks. Currently, graph pooling operators have emerged as crucial components that bridge the gap between node representation learning and diverse graph-level tasks by transforming node representations into graph representations. Given the rapid growth and widespread adoption of graph pooling, this review aims to summarize the existing graph pooling operators for GNNs and their representative applications in omics. Specifically, we first present a comprehensive taxonomy of existing graph pooling algorithms, expanding the categorization for both global and hierarchical pooling operators, and for the first time reviewing the inverse operation of graph pooling, named unpooling. Next, we describe the general evaluation framework for graph pooling operators, encompassing three fundamental aspects: experimental setup, ablation analysis, and model interpretation. We also discuss open issues that significantly influence the design of graph pooling operators, including complexity, connectivity, adaptability, additional loss, and attention mechanisms. Finally, we summarize bioinformatics applications of graph pooling operators in omics, including graphs of gene interaction, medical images, and protein structures for drug discovery and disease diagnosis. Furthermore, we showcase the impact of graph pooling operators on research in specific real-world domains, with a focus on prediction performance and model interpretability. This review provides methodological insights in machine learning based graph modeling and related omics research, as well as an ongoing resource by gathering related papers and code in a dedicated GitHub repository (https://github.com/Hou-WJ/Graph-Pooling-Operators-and-Bioinformatics-Applications).

Demystifying and Mitigating Bias for Node Representation Learning.

Node representation learning has attracted increasing attention due to its efficacy for various applications on graphs. However, fairness is a largely under-explored territory within the field, although it is shown that the use of graph structure in learning amplifies bias. To this end, this work theoretically explains the sources of bias in node representations obtained via graph neural networks (GNNs). It is revealed that both nodal features and graph structure lead to bias in the obtained representations. Building upon the analysis, fairness-aware data augmentation frameworks are developed to reduce the intrinsic bias. Our theoretical analysis and proposed schemes can be readily employed in understanding and mitigating bias for various GNN-based learning mechanisms. Extensive experiments on node classification and link prediction over multiple real networks are carried out, and it is shown that the proposed augmentation strategies can improve fairness while providing comparable utility to state-of-the-art methods.

Label-aware aggregation on heterophilous graphs for node representation learning

Learning node representation on heterophilous graphs has been challenging due to nodes with diverse labels/attributes being connected. The main idea is to balance contributions between the center node and neighborhoods. However, existing methods failed to make full use of personalized contributions of different neighborhoods based on whether they own the same label as the center node, making it necessary to explore the distinctive contributions of similar/dissimilar neighborhoods. We reveal that both similar/dissimilar neighborhoods have positive impacts on feature aggregation under different homophily ratios. Especially, dissimilar neighborhoods play a significant role under low homophily ratios. Based on this, we propose LAAH, a label-aware aggregation approach for node representation learning on heterophilous graphs. LAAH separates each center node from its neighborhoods and generates their own node representations. Additionally, for each neighborhood, LAAH records its label information based on whether it belongs to the same class as the center node and then aggregates its effective feature in a weighted manner. Finally, a learnable parameter is used to balance the contributions of each center node and all its neighborhoods, leading to updated representations. Extensive experiments on 8 real-world heterophilous datasets and a synthetic dataset verify that LAAH can achieve competitive or superior accuracy in node classification with lower parameter scale and computational complexity compared with the SOTA methods. The code is released at GitHub: https://github.com/laah123graph/LAAH.

PathMLP: Smooth path towards high-order homophily

Real-world graphs exhibit increasing heterophily, where nodes no longer tend to be connected to nodes with the same label, challenging the homophily assumption of classical graph neural networks (GNNs) and impeding their performance. Intriguingly, from the observation of heterophilous data, we notice that certain high-order information exhibits higher homophily, which motivates us to involve high-order information in node representation learning. However, common practices in GNNs to acquire high-order information mainly through increasing model depth and altering message-passing mechanisms, which, albeit effective to a certain extent, suffer from three shortcomings: (1) over-smoothing due to excessive model depth and propagation times; (2) high-order information is not fully utilized; (3) low computational efficiency. In this regard, we design a similarity-based path sampling strategy to capture smooth paths containing high-order homophily. Then we propose a lightweight model based on multi-layer perceptrons (MLP), named PathMLP, which can encode messages carried by paths via simple transformation and concatenation operations, and effectively learn node representations in heterophilous graphs through adaptive path aggregation. Extensive experiments demonstrate that our method outperforms baselines on 16 out of 20 datasets, underlining its effectiveness and superiority in alleviating the heterophily problem. In addition, our method is immune to over-smoothing and has high computational efficiency. The source code will be available in https://github.com/Graph4Sec-Team/PathMLP.

Central-Smoothing Hypergraph Neural Networks for Predicting Drug-Drug Interactions.

Predicting drug-drug interactions (DDIs) is the problem of predicting side effects (unwanted outcomes) of a pair of drugs using drug information and known side effects of many pairs. This problem can be formulated as predicting labels (i.e., side effects) for each pair of nodes in a DDI graph, of which nodes are drugs and edges are interacting drugs with known labels. State-of-the-art methods for this problem are graph neural networks (GNNs), which leverage neighborhood information in the graph to learn node representations. For DDI, however, there are many labels with complicated relationships due to the nature of side effects. Usual GNNs often fix labels as one-hot vectors that do not reflect label relationships and potentially do not obtain the highest performance in the difficult cases of infrequent labels. In this brief, we formulate DDI as a hypergraph where each hyperedge is a triple: two nodes for drugs and one node for a label. We then present CentSmoothie, a hypergraph neural network (HGNN) that learns representations of nodes and labels altogether with a novel " central-smoothing " formulation. We empirically demonstrate the performance advantages of CentSmoothie in simulations as well as real datasets.

Attributed graph clustering under the contrastive mechanism with cluster-preserving augmentation

Attributed graph clustering is a fundamental task in complex network analysis. Many existing graph clustering methods utilize graph representation learning techniques to learn node representations, subsequently applying K-means for clustering. Despite the significant attention and promising outcomes of graph contrastive learning in graph representation learning, we find two essential problems that need to be solved. (1) How to achieve augmentation for contrast that preserves the cluster structure of a given graph? (2) How to design an effective contrastive learning mechanism that collaborates with clustering? Therefore, we propose an attributed graph clustering method under the contrastive mechanism with cluster-preserving augmentation, integrating node representation learning and clustering into a unified framework. Specifically, we construct a contrasting view based on a generated kNN graph and edge betweenness centrality to preserve the cluster structure in the original graph. Meanwhile, a multilevel contrast mechanism based on pseudo-label-guided negative sampling is proposed to maximize the agreement between node representations in multiple latent spaces. We jointly optimize a specific clustering objective during the contrastive process, leading to refined cluster distribution directly in training. Extensive experiments on several datasets demonstrate that our proposed model consistently outperforms existing state-of-the-art methods on clustering.

Explicitly Exploiting Implicit User and Item Relations in Graph Convolutional Network (GCN) for Recommendation

Most existing collaborative filtering-based recommender systems rely solely on available user–item interactions for user and item representation learning. Their performance often suffers significantly when interactions are sparse, as limited user and item interactions are insufficient for learning robust representations. To address this issue, recent research has explored additional information between users and items by leveraging the user–item bipartite graph. However, these methods have not fully exploited high-order neighborhood information, primarily using sampled interactions to enrich training data rather than integrating this information directly into representation learning. In this paper, we propose a novel model, EIR-GCN (Embedding Integration with Relational Graph Convolutional Network), which directly incorporates various types of collaborative relations, such as user–user and item–item interactions, into the embedding function for user preference modeling. Specifically, our model employs advanced graph convolutional network (GCN) techniques to integrate user–item, user–user, and item–item relations for comprehensive representation learning. EIR-GCN initially selects the most influential second-order neighbors from the user–item bipartite graph to form user–user and item–item connections. With these enriched connections, a message-passing method is adopted to learn node representations by aggregating messages from directly linked nodes, including first-order item neighbors and selected second-order user neighbors. Extensive experiments on several public datasets demonstrate that EIR-GCN outperforms strong baselines, including recent GCN-based models and those exploiting high-order information. Our results show that EIR-GCN achieves state-of-the-art performance and effectively addresses the sparsity issue, highlighting its robustness and efficacy in recommendation tasks.

Toward Graph Self-Supervised Learning With Contrastive Adjusted Zooming.

Graph representation learning (GRL) is critical for graph-structured data analysis. However, most of the existing graph neural networks (GNNs) heavily rely on labeling information, which is normally expensive to obtain in the real world. Although some existing works aim to effectively learn graph representations in an unsupervised manner, they suffer from certain limitations, such as the heavy reliance on monotone contrastiveness and limited scalability. To overcome the aforementioned problems, in light of the recent advancements in graph contrastive learning, we introduce a novel self-supervised GRL algorithm via graph contrastive adjusted zooming, namely, G-Zoom, to learn node representations by leveraging the proposed adjusted zooming scheme. Specifically, this mechanism enables G-Zoom to explore and extract self-supervision signals from a graph from multiple scales: micro (i.e., node level), meso (i.e., neighborhood level), and macro (i.e., subgraph level). First, we generate two augmented views of the input graph via two different graph augmentations. Then, we establish three different contrastiveness on the above three scales progressively, from node, neighboring, to subgraph level, where we maximize the agreement between graph representations across scales. While we can extract valuable clues from a given graph on the micro and macro perspectives, the neighboring-level contrastiveness offers G-Zoom the capability of a customizable option based on our adjusted zooming scheme to manually choose an optimal viewpoint that lies between the micro and macro perspectives to better understand the graph data. In addition, to make our model scalable to large graphs, we use a parallel graph diffusion approach to decouple model training from the graph size. We have conducted extensive experiments on real-world datasets, and the results demonstrate that our proposed model outperforms the state-of-the-art methods consistently.

A Systematic Review of Attention Models in Natural Language Processing

The role of attention models in neural networks has drawn much focus among scholars in recent years mainly because of its efficiency and versatility. Attention models are important for many Natural Language Processing applications like question answering, semantic analysis, sentiment analysis and machine translation performing significantly better than other usual approaches. These mechanisms improve neural networks capability in interpretation and tackle issues like performance decay, which affects the Recurrent Neural Networks by concentrating on significant data inputs and context. A stream of research has explored the integration of attention mechanisms to other fields, namely computer vision and graph analysis improving tasks like object detection, image captioning or node representation learning. However, for more detailed research of modern achievements and various domains and model architecture structures there is still the necessity to use the more profound survey-based investigation. This research article provides a review of previous work done on attention-based model in NLP. It also outlines the problems and provides relevant solutions using different methodologies. Future directions of the research are provided for the improvement of the attention models for NLP, discussing potential issues in terms of model performance and its capacity, as well as its interpretability. This review is intended to help further research to more universal attention-based solutions in the domain of NLP.

PGCNMDA: Learning node representations along paths with graph convolutional network for predicting miRNA-disease associations

Identifying miRNA-disease associations (MDAs) is crucial for improving the diagnosis and treatment of various diseases. However, biological experiments can be time-consuming and expensive. To overcome these challenges, computational approaches have been developed, with Graph Convolutional Network (GCN) showing promising results in MDA prediction. The success of GCN-based methods relies on learning a meaningful spatial operator to extract effective node feature representations. To enhance the inference of MDAs, we propose a novel method called PGCNMDA, which employs graph convolutional networks with a learning graph spatial operator from paths. This approach enables the generation of meaningful spatial convolutions from paths in GCN, leading to improved prediction performance. On HMDD v2.0, PGCNMDA obtains a mean AUC of 0.9229 and an AUPRC of 0.9206 under 5-fold cross-validation (5-CV), and a mean AUC of 0.9235 and an AUPRC of 0.9212 under 10-fold cross-validation (10-CV), respectively. Additionally, the AUC of PGCNMDA also reaches 0.9238 under global leave-one-out cross-validation (GLOOCV). On HMDD v3.2, PGCNMDA obtains a mean AUC of 0.9413 and an AUPRC of 0.9417 under 5-CV, and a mean AUC of 0.9419 and an AUPRC of 0.9425 under 10-CV, respectively. Furthermore, the AUC of PGCNMDA also reaches 0.9415 under GLOOCV. The results show that PGCNMDA is superior to other compared methods. In addition, the case studies on pancreatic neoplasms, thyroid neoplasms and leukemia show that 50, 50 and 48 of the top 50 predicted miRNAs linked to these diseases are confirmed, respectively. It further validates the effectiveness and feasibility of PGCNMDA in practical applications.

An integrated graph data privacy attack framework based on graph neural networks in IoT

SummaryKnowledge graphs contain a large amount of entity and relational data, and graph neural networks, as a class of efficient graph representation techniques based on deep learning, excel in knowledge graph modeling. However, previous neural network architectures for the most part only learn node representations and do not fully consider the heterogeneity of data. In this article, we innovatively propose a privacy attack framework based on IoT, PAFI, which is able to classify entities and relations, learn embedding representations in multi‐relational graphs, and can be applied to some existing neural network algorithms. Based on this, a fine‐grained privacy attack model, FPM, is proposed, which can perform attack operations on multiple targets, achieve selectivity of target tasks, and greatly improve the generalization ability of the attack model. In this article, the effectiveness of PAFI and FPM is demonstrated by real network datasets, and compared with previous attack methods, both of which achieve good results.

A deep learning framework for predicting disease-gene associations with functional modules and graph augmentation

BackgroundThe exploration of gene-disease associations is crucial for understanding the mechanisms underlying disease onset and progression, with significant implications for prevention and treatment strategies. Advances in high-throughput biotechnology have generated a wealth of data linking diseases to specific genes. While graph representation learning has recently introduced groundbreaking approaches for predicting novel associations, existing studies always overlooked the cumulative impact of functional modules such as protein complexes and the incompletion of some important data such as protein interactions, which limits the detection performance.ResultsAddressing these limitations, here we introduce a deep learning framework called ModulePred for predicting disease-gene associations. ModulePred performs graph augmentation on the protein interaction network using L3 link prediction algorithms. It builds a heterogeneous module network by integrating disease-gene associations, protein complexes and augmented protein interactions, and develops a novel graph embedding for the heterogeneous module network. Subsequently, a graph neural network is constructed to learn node representations by collectively aggregating information from topological structure, and gene prioritization is carried out by the disease and gene embeddings obtained from the graph neural network. Experimental results underscore the superiority of ModulePred, showcasing the effectiveness of incorporating functional modules and graph augmentation in predicting disease-gene associations. This research introduces innovative ideas and directions, enhancing the understanding and prediction of gene-disease relationships.

Information-enhanced deep graph clustering network

Graph clustering is a significant task in complex network research. Deep graph clustering aims to uncover the potential community structure in graph data using the powerful feature extraction capability of deep learning, garnering much attention in recent decades. However, existing graph clustering methods fall short in fully utilizing available information, particularly in effectively fusing structural and attribute information, as well as utilizing coarse-grained data. Consequently, learned node representations remain limited, leading to suboptimal clustering results. To address these challenges, we propose Information-Enhanced Deep Graph Clustering Network (IEDGCN) for unsupervised attribute graphs. IEDGCN introduces key components to enhance information utilization and improve clustering performance. Firstly, we design a new higher-order neighborhood-weighted attribute matrix, effectively integrating higher-order neighborhood information with attributes. Secondly, a graph generation model guides the learning of the structural feature space more effectively. Additionally, IEDGCN captures more coarse-grained information by utilizing community and higher-order neighborhood features to refine clustering results. Finally, the proposed method is uniformly guided through a jointly supervised strategy for representation learning and cluster assignment. Experimental results on different benchmark datasets demonstrate the effectiveness of IEDGCN compared to state-of-the-art methods, emphasizing the importance of information enhancement for graph clustering.

Self-Supervised Node Representation Learning via Node-to-Neighbourhood Alignment.

Self-supervised node representation learning aims to learn node representations from unlabelled graphs that rival the supervised counterparts. The key towards learning informative node representations lies in how to effectively gain contextual information from the graph structure. In this work, we present simple-yet-effective self-supervised node representation learning via aligning the hidden representations of nodes and their neighbourhood. Our first idea achieves such node-to-neighbourhood alignment by directly maximizing the mutual information between their representations, which, we prove theoretically, plays the role of graph smoothing. Our framework is optimized via a surrogate contrastive loss and a Topology-Aware Positive Sampling (TAPS) strategy is proposed to sample positives by considering the structural dependencies between nodes, which enables offline positive selection. Considering the excessive memory overheads of contrastive learning, we further propose a negative-free solution, where the main contribution is a Graph Signal Decorrelation (GSD) constraint to avoid representation collapse and over-smoothing. The GSD constraint unifies some of the existing constraints and can be used to derive new implementations to combat representation collapse. By applying our methods on top of simple MLP-based node representation encoders, we learn node representations that achieve promising node classification performance on a set of graph-structured datasets from small- to large-scale.