Graph-based Tasks Research Articles

Enhancing Graph Neural Networks via Memorized Global Information

Graph neural networks (GNNs) have gained significant attention for their impressive results on different graph-based tasks. The essential mechanism of GNNs is the message-passing framework, whereby node representations are aggregated from local neighborhoods. Recently, Transformer-based GNNs have been introduced to learn the long-range dependencies, enhancing performance. However, their quadratic computational complexity, due to the attention computation, has constrained their applicability on large-scale graphs. To address this issue, we propose MGIGNN ( M emorized G lobal I nformation G raph N eural N etwork), an innovative approach that leverages memorized global information to enhance existing GNNs in both transductive and inductive scenarios. Specifically, MGIGNN captures long-range dependencies by identifying and incorporating global similar nodes, which are defined as nodes exhibiting similar features, structural patterns and label information within a graph. To alleviate the computational overhead associated with computing embeddings for all nodes, we introduce an external memory module to facilitate the retrieval of embeddings and optimize performance on large graphs. To enhance the memory-efficiency, MGIGNN selectively retrieves global similar nodes from a small set of candidate nodes. These candidate nodes are selected from the training nodes based on a sparse node selection distribution with a Dirichlet prior. This selecting approach not only reduces the memory size required but also ensures efficient utilization of computational resources. Through comprehensive experiments conducted on ten widely-used and real-world datasets, including seven homogeneous datasets and three heterogeneous datasets, we demonstrate that our MGIGNN can generally improve the performance of existing GNNs on node classification tasks under both inductive and transductive settings.

Asymmetric augmented paradigm-based graph neural architecture search

In most scenarios of graph-based tasks, graph neural networks (GNNs) are trained end-to-end with labeled samples. Labeling graph samples, a time-consuming and expert-dependent process, leads to huge costs. Graph data augmentations can provide a promising method to expand labeled samples cheaply. However, graph data augmentations will damage the capacity of GNNs to distinguish non-isomorphic graphs during the supervised graph representation learning process. How to utilize graph data augmentations to expand labeled samples while preserving the capacity of GNNs to distinguish non-isomorphic graphs is a challenging research problem. To address the above problem, we abstract a novel asymmetric augmented paradigm in this paper and theoretically prove that it offers a principled approach. The asymmetric augmented paradigm can preserve the capacity of GNNs to distinguish non-isomorphic graphs while utilizing augmented labeled samples to improve the generalization capacity of GNNs. To be specific, the asymmetric augmented paradigm will utilize similar yet distinct asymmetric weights to classify the real sample and augmented sample, respectively. To systemically explore the benefits of asymmetric augmented paradigm under different GNN architectures, rather than studying individual asymmetric augmented GNN (A2GNN) instance, we then develop an auto-search engine called Asymmetric Augmented Graph Neural Architecture Search (A2GNAS) to save human efforts. We empirically validate our asymmetric augmented paradigm on multiple graph classification benchmarks, and demonstrate that representative A2GNN instances automatically discovered by our A2GNAS method achieve state-of-the-art performance compared with competitive baselines. Our codes are available at: https://github.com/csubigdata-Organization/A2GNAS.

FairGAT: Fairness-Aware Graph Attention Networks

Graphs can facilitate modeling various complex systems such as gene networks and power grids as well as analyzing the underlying relations within them. Learning over graphs has recently attracted increasing attention, particularly graph neural network (GNN)–based solutions, among which graph attention networks (GATs) have become one of the most widely utilized neural network structures for graph-based tasks. Although it is shown that the use of graph structures in learning results in the amplification of algorithmic bias, the influence of the attention design in GATs on algorithmic bias has not been investigated. Motivated by this, the present study first carries out a theoretical analysis in order to demonstrate the sources of algorithmic bias in GAT-based learning for node classification. Then, a novel algorithm, FairGAT, which leverages a fairness-aware attention design, is developed based on the theoretical findings. Experimental results on real-world networks demonstrate that FairGAT improves group fairness measures while also providing comparable utility to the fairness-aware baselines for node classification and link prediction.

MINES: Message Intercommunication for Inductive Relation Reasoning over Neighbor-Enhanced Subgraphs

GraIL and its variants have shown their promising capacities for inductive relation reasoning on knowledge graphs. However, the uni-directional message-passing mechanism hinders such models from exploiting hidden mutual relations between entities in directed graphs. Besides, the enclosing subgraph extraction in most GraIL-based models restricts the model from extracting enough discriminative information for reasoning. Consequently, the expressive ability of these models is limited. To address the problems, we propose a novel GraIL-based framework, termed MINES, by introducing a Message Intercommunication mechanism on the Neighbor-Enhanced Subgraph. Concretely, the message intercommunication mechanism is designed to capture the omitted hidden mutual information. It introduces bi-directed information interactions between connected entities by inserting an undirected/bi-directed GCN layer between uni-directed RGCN layers. Moreover, inspired by the success of involving more neighbors in other graph-based tasks, we extend the neighborhood area beyond the enclosing subgraph to enhance the information collection for inductive relation reasoning. Extensive experiments prove the promising capacity of the proposed MINES from various aspects, especially for the superiority, effectiveness, and transfer ability.

G-Adapter: Towards Structure-Aware Parameter-Efficient Transfer Learning for Graph Transformer Networks

It has become a popular paradigm to transfer the knowledge of large-scale pre-trained models to various downstream tasks via fine-tuning the entire model parameters. However, with the growth of model scale and the rising number of downstream tasks, this paradigm inevitably meets the challenges in terms of computation consumption and memory footprint issues. Recently, Parameter-Efficient Fine-Tuning (PEFT) (e.g., Adapter, LoRA, BitFit) shows a promising paradigm to alleviate these concerns by updating only a portion of parameters. Despite these PEFTs having demonstrated satisfactory performance in natural language processing, it remains under-explored for the question: whether these techniques could be transferred to graph-based tasks with Graph Transformer Networks (GTNs)? Therefore, in this paper, we fill this gap by providing extensive benchmarks with traditional PEFTs on a range of graph-based downstream tasks. Our empirical study shows that it is sub-optimal to directly transfer existing PEFTs to graph-based tasks due to the issue of feature distribution shift. To address this issue, we propose a novel structure-aware PEFT approach, named G-Adapter, which leverages graph convolution operation to introduce graph structure information (e.g., graph adjacency matrix) as an inductive bias to guide the updating process. Further, we propose Bregman proximal point optimization to alleviate feature distribution shift by preventing the model from aggressive update. Extensive experiments demonstrate that G-Adapter obtains state-of-the-art performance compared to counterparts on nine graph benchmark datasets based on diverse pre-trained GTNs, and delivers tremendous memory footprint efficiency compared to the conventional paradigm.

Correlation-aware Graph Data Augmentation with Implicit and Explicit Neighbors

In recent years, there has been a significant surge in commercial demand for citation graph-based tasks, such as patent analysis, social network analysis, and recommendation systems. Graph Neural Networks (GNNs) are widely used for these tasks due to their remarkable performance in capturing topological graph information. However, GNNs’ output results are highly dependent on the composition of local neighbors within the topological structure. To address this issue, we identify two types of neighbors in a citation graph: explicit neighbors based on the topological structure and implicit neighbors based on node features. Our primary motivation is to clearly define and visualize these neighbors, emphasizing their importance in enhancing graph neural network performance. We propose a Correlation-aware Network (CNet) to re-organize the citation graph and learn more valuable informative representations by leveraging these implicit and explicit neighbors. Our approach aims to improve graph data augmentation and classification performance, with the majority of our focus on stating the importance of using these neighbors, while also introducing a new graph data augmentation method. We compare CNet with state-of-the-art (SOTA) GNNs and other graph data augmentation approaches acting on GNNs. Extensive experiments demonstrate that CNet effectively extracts more valuable informative representations from the citation graph, significantly outperforming baselines. The code is available on public GitHub. 1

SSFG: Stochastically Scaling Features and Gradients for Regularizing Graph Convolutional Networks.

Graph convolutional networks (GCNs) have been successfully applied in various graph-based tasks. In a typical graph convolutional layer, node features are updated by aggregating neighborhood information. Repeatedly applying graph convolutions can cause the oversmoothing issue, i.e., node features at deep layers converge to similar values. Previous studies have suggested that oversmoothing is one of the major issues that restrict the performance of GCNs. In this article, we propose a stochastic regularization method to tackle the oversmoothing problem. In the proposed method, we stochastically scale features and gradients (SSFG) by a factor sampled from a probability distribution in the training procedure. By explicitly applying a scaling factor to break feature convergence, the oversmoothing issue is alleviated. We show that applying stochastic scaling at the gradient level is complementary to that applied at the feature level to improve the overall performance. Our method does not increase the number of trainable parameters. When used together with ReLU, our SSFG can be seen as a stochastic ReLU activation function. We experimentally validate our SSFG regularization method on three commonly used types of graph networks. Extensive experimental results on seven benchmark datasets for four graph-based tasks demonstrate that our SSFG regularization is effective in improving the overall performance of the baseline graph networks. The code is available at https://github.com/vailatuts/SSFG-regularization.

Accurate graph classification via two-staged contrastive curriculum learning.

Given a graph dataset, how can we generate meaningful graph representations that maximize classification accuracy? Learning representative graph embeddings is important for solving various real-world graph-based tasks. Graph contrastive learning aims to learn representations of graphs by capturing the relationship between the original graph and the augmented graph. However, previous contrastive learning methods neither capture semantic information within graphs nor consider both nodes and graphs while learning graph embeddings. We propose TAG (Two-staged contrAstive curriculum learning for Graphs), a two-staged contrastive learning method for graph classification. TAG learns graph representations in two levels: node-level and graph level, by exploiting six degree-based model-agnostic augmentation algorithms. Experiments show that TAG outperforms both unsupervised and supervised methods in classification accuracy, achieving up to 4.08% points and 4.76% points higher than the second-best unsupervised and supervised methods on average, respectively.

SH-GAT: Software-hardware co-design for accelerating graph attention networks on FPGA

<abstract><p>Graph convolution networks (GCN) have demonstrated success in learning graph structures; however, they are limited in inductive tasks. Graph attention networks (GAT) were proposed to address the limitations of GCN and have shown high performance in graph-based tasks. Despite this success, GAT faces challenges in hardware acceleration, including: 1) The GAT algorithm has difficulty adapting to hardware; 2) challenges in efficiently implementing Sparse matrix multiplication (SPMM); and 3) complex addressing and pipeline stall issues due to irregular memory accesses. To this end, this paper proposed SH-GAT, an FPGA-based GAT accelerator that achieves more efficient GAT inference. The proposed approach employed several optimizations to enhance GAT performance. First, this work optimized the GAT algorithm using split weights and softmax approximation to make it more hardware-friendly. Second, a load-balanced SPMM kernel was designed to fully leverage potential parallelism and improve data throughput. Lastly, data preprocessing was performed by pre-fetching the source node and its neighbor nodes, effectively addressing pipeline stall and complexly addressing issues arising from irregular memory access. SH-GAT was evaluated on the Xilinx FPGA Alveo U280 accelerator card with three popular datasets. Compared to existing CPU, GPU, and state-of-the-art (SOTA) FPGA-based accelerators, SH-GAT can achieve speedup by up to 3283$ \times $, 13$ \times $, and 2.3$ \times $.</p></abstract>

Self-Supervised Temporal Graph Learning With Temporal and Structural Intensity Alignment.

Temporal graph learning aims to generate high-quality representations for graph-based tasks with dynamic information, which has recently garnered increasing attention. In contrast to static graphs, temporal graphs are typically organized as node interaction sequences over continuous time rather than an adjacency matrix. Most temporal graph learning methods model current interactions by incorporating historical neighborhood. However, such methods only consider first-order temporal information while disregarding crucial high-order structural information, resulting in suboptimal performance. To address this issue, we propose a self-supervised method called S2T for temporal graph learning, which extracts both temporal and structural information to learn more informative node representations. Notably, the initial node representations combine first-order temporal and high-order structural information differently to calculate two conditional intensities. An alignment loss is then introduced to optimize the node representations, narrowing the gap between the two intensities and making them more informative. Concretely, in addition to modeling temporal information using historical neighbor sequences, we further consider structural knowledge at both local and global levels. At the local level, we generate structural intensity by aggregating features from high-order neighbor sequences. At the global level, a global representation is generated based on all nodes to adjust the structural intensity according to the active statuses on different nodes. Extensive experiments demonstrate that the proposed model S2T achieves at most 10.13% performance improvement compared with the state-of-the-art competitors on several datasets.

RAN-GNNs: Breaking the Capacity Limits of Graph Neural Networks.

Graph neural networks (GNNs) have become a staple in problems addressing learning and analysis of data defined over graphs. However, several results suggest an inherent difficulty in extracting better performance by increasing the number of layers. Recent works attribute this to a phenomenon peculiar to the extraction of node features in graph-based tasks, i.e., the need to consider multiple neighborhood sizes at the same time and adaptively tune them. In this article, we investigate the recently proposed randomly wired architectures in the context of GNNs. Instead of building deeper networks by stacking many layers, we prove that employing a randomly wired architecture can be a more effective way to increase the capacity of the network and obtain richer representations. We show that such architectures behave like an ensemble of paths, which are able to merge contributions from receptive fields of varied size. Moreover, these receptive fields can also be modulated to be wider or narrower through the trainable weights over the paths. We also provide extensive experimental evidence of the superior performance of randomly wired architectures over multiple tasks and five graph convolution definitions, using recent benchmarking frameworks that address the reliability of previous testing methodologies.

Learning to Count Isomorphisms with Graph Neural Networks

Subgraph isomorphism counting is an important problem on graphs, as many graph-based tasks exploit recurring subgraph patterns. Classical methods usually boil down to a backtracking framework that needs to navigate a huge search space with prohibitive computational cost. Some recent studies resort to graph neural networks (GNNs) to learn a low-dimensional representation for both the query and input graphs, in order to predict the number of subgraph isomorphisms on the input graph. However, typical GNNs employ a node-centric message passing scheme that receives and aggregates messages on nodes, which is inadequate in complex structure matching for isomorphism counting. Moreover, on an input graph, the space of possible query graphs is enormous, and different parts of the input graph will be triggered to match different queries. Thus, expecting a fixed representation of the input graph to match diversely structured query graphs is unrealistic. In this paper, we propose a novel GNN called Count-GNN for subgraph isomorphism counting, to deal with the above challenges. At the edge level, given that an edge is an atomic unit of encoding graph structures, we propose an edge-centric message passing scheme, where messages on edges are propagated and aggregated based on the edge adjacency to preserve fine-grained structural information. At the graph level, we modulate the input graph representation conditioned on the query, so that the input graph can be adapted to each query individually to improve their matching. Finally, we conduct extensive experiments on a number of benchmark datasets to demonstrate the superior performance of Count-GNN.

DPGNN: Dual-perception graph neural network for representation learning

Graph neural networks (GNNs) have drawn increasing attention in recent years and achieved remarkable performance in many graph-based tasks, especially in semi-supervised learning on graphs. However, most existing GNNs are based on the message-passing paradigm to iteratively aggregate neighborhood information in a single topology space. Despite their success, the expressive power of GNNs is limited by some drawbacks, such as inflexibility of message source expansion, negligence of node-level message output discrepancy, and restriction of single message space. To address these drawbacks, we present a novel message-passing paradigm, based on the properties of multi-step message source, node-specific message output, and multi-space message interaction. To verify its validity, we instantiate the new message-passing paradigm as a Dual-Perception Graph Neural Network (DPGNN), which applies a node-to-step attention mechanism to aggregate node-specific multi-step neighborhood information adaptively. Our proposed DPGNN can capture the structural neighborhood information and the feature-related information simultaneously for graph representation learning. Experimental results on six benchmark datasets with different topological structures demonstrate that our method outperforms the latest state-of-the-art models, which proves the superiority and versatility of our method. To our knowledge, we are the first to consider node-specific message passing in the GNNs.

Graph Learning for Anomaly Analytics: Algorithms, Applications, and Challenges

Anomaly analytics is a popular and vital task in various research contexts that has been studied for several decades. At the same time, deep learning has shown its capacity in solving many graph-based tasks, like node classification, link prediction, and graph classification. Recently, many studies are extending graph learning models for solving anomaly analytics problems, resulting in beneficial advances in graph-based anomaly analytics techniques. In this survey, we provide a comprehensive overview of graph learning methods for anomaly analytics tasks. We classify them into four categories based on their model architectures, namely graph convolutional network, graph attention network, graph autoencoder, and other graph learning models. The differences between these methods are also compared in a systematic manner. Furthermore, we outline several graph-based anomaly analytics applications across various domains in the real world. Finally, we discuss five potential future research directions in this rapidly growing field.

NC-GNN: Consistent Neighbors of Nodes Help More in Graph Neural Networks

Graph neural networks, as the promising methodology in data mining for graph data, currently attract much attention and are broadly applied in graph-based tasks. Existing GNN methods mostly follow the assumption of homophily, where the connected nodes are similar and share the same labels. Most graphs in the real world can satisfy the assumption. However, for the particular nodes, the situation is not always satisfied. The connections between different-labeled nodes will introduce noise in feature aggregation and result in node representation deviating in the wrong direction. In this paper, we focus on the different-labeled neighbors of labeled nodes in the graphs. By regarding aggregation among neighbors as the procedure of node feature reconstruction, we devise a novel metric neighbor consistency to measure the difference between nodes and their neighborhoods. In this way, we can evaluate the reliability of nodes after aggregation. Furthermore, we propose a novel method, Neighbor Consistent Graph Neural Networks (NC-GNN), to promote the training of graph neural networks by reweighting the influence of labeled nodes based on neighbor consistency scores. Systematic experiments are conducted on benchmark datasets, and the results demonstrate the effectiveness of our method.

Network representation learning method embedding linear and nonlinear network structures

With the rapid development of neural networks, much attention has been focused on network embedding for complex network data, which aims to learn low-dimensional embedding of nodes in the network and how to effectively apply learned network representations to various graph-based analytical tasks. Two typical models exist namely the shallow random walk network representation method and deep learning models such as graph convolution networks (GCNs). The former one can be used to capture the linear structure of the network using depth-first search (DFS) and width-first search (BFS), whereas Hierarchical GCN (HGCN) is an unsupervised graph embedding that can be used to describe the global nonlinear structure of the network via aggregating node information. However, the two existing kinds of models cannot simultaneously capture the nonlinear and linear structure information of nodes. Thus, the nodal characteristics of nonlinear and linear structures are explored in this paper, and an unsupervised representation method based on HGCN that joins learning of shallow and deep models is proposed. Experiments on node classification and dimension reduction visualization are carried out on citation, language, and traffic networks. The results show that, compared with the existing shallow network representation model and deep network model, the proposed model achieves better performances in terms of micro-F1, macro-F1 and accuracy scores.

Practical Near-Data-Processing Architecture for Large-Scale Distributed Graph Neural Network

Graph Neural Networks have drawn tremendous attention in the past few years due to their convincing performance and high interpretability in various graph-based tasks like link prediction and node classification. With the ever-growing graph size in the real world, especially for industrial graphs at a billion-level, the storage of graphs can easily consume Terabytes so that the process of GNNs has to be processed in a distributed manner. As a result, the execution could be inefficient due to the expensive cross-node communication and irregular memory access. Various GNN accelerators have been proposed for efficient GNN processing. They, however, mainly focused on small and medium-size graphs, which is not applicable to large-scale distributed graphs. In this paper, we present a practical Near-Data-Processing architecture based on a memory-pool system for large-scale distributed GNNs. We propose a customized memory fabric interface to construct the memory pool for low-latency and high throughput cross-node communication, which can provide flexible memory allocation and strong scalability. A practical Near-Data-Processing design is proposed for efficient work offloading and bandwidth utilization improvement. Moreover, we also introduce a partition and scheduling scheme to further improve performance and achieve workload balance. Comprehensive evaluations demonstrate that the proposed architecture can achieve up to <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$27\times $ </tex-math></inline-formula> and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$8\times $ </tex-math></inline-formula> higher training speed compared to two state-of-the-art distributed GNN frameworks: Deep Graph Library and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$P^{3}$ </tex-math></inline-formula> , respectively.

Curvature graph neural network

Graph neural networks (GNNs) have achieved great success in many graph-based tasks. Much work is dedicated to empowering GNNs with adaptive locality ability, which enables the measurement of the importance of neighboring nodes to the target node by a node-specific mechanism. However, the current node-specific mechanisms are deficient in distinguishing the importance of nodes in the topology structure. We believe that the structural importance of neighboring nodes is closely related to their importance in aggregation. In this paper, we introduce discrete graph curvature (the Ricci curvature) to quantify the strength of the structural connection of pairwise nodes. We propose a curvature graph neural network (CGNN), which effectively improves the adaptive locality ability of GNNs by leveraging the structural properties of graph curvature. To improve the adaptability of curvature on various datasets, we explicitly transform curvature into the weights of neighboring nodes by the necessary negative curvature processing module and curvature normalization module. Then, we conduct numerous experiments on various synthetic and real-world datasets. The experimental results on synthetic datasets show that CGNN effectively exploits the topology structure information and that the performance is significantly improved. CGNN outperforms the baselines on 5 dense node classification benchmark datasets. This study provides a deepened understanding of how to utilize advanced topology information and assign the importance of neighboring nodes from the perspective of graph curvature and encourages bridging the gap between graph theory and neural networks. The source code is available at https://github.com/GeoX-Lab/CGNN.

Understanding the Implication of Non-Volatile Memory for Large-Scale Graph Neural Network Training

Graph neural networks (GNNs) can extract features by learning both the representation of each objects (i.e., graph nodes) as well as the relationship across different objects (i.e., the edges that connect nodes), achieving state-of-the-art performance on a wide range of graph-based tasks. Despite its strengths, utilizing these algorithms in a production environment faces several key challenges as the number of graph nodes and edges amount to several billions to hundreds of billions scale, requiring substantial storage space for training. Unfortunately, existing ML frameworks based on the in-memory processing model significantly hamper the productivity of algorithm developers as it mandates the overall working set to fit within DRAM capacity constraints. In this work, we first study state-of-the-art, large-scale GNN training algorithms. We then conduct a detailed characterization on utilizing capacity-optimized non-volatile memory solutions for storing memory-hungry GNN data, exploring the feasibility of SSDs for large-scale GNN training.

Enhancing Graph Neural Networks by a High-quality Aggregation of Beneficial Information

Graph Neural Networks (GNNs), such as GCN, GraphSAGE, GAT, and SGC, have achieved state-of-the-art performance on a wide range of graph-based tasks. These models all use a technique called neighborhood aggregation, in which the embedding of each node is updated by aggregating the embeddings of its neighbors. However, not all information aggregated from neighbors is beneficial. In some cases, a portion of the neighbor information may be harmful to the downstream tasks. For the high-quality aggregation of beneficial information, we propose a flexible method EGAI (Enhancing Graph neural networks by a high-quality Aggregation of beneficial Information). The core concept of this method is to filter out the redundant and harmful information by removing specific edges during each training epoch. The practical and theoretical motivations, considerations, and strategies related to this method are discussed in detail. EGAI is a general method that can be combined with many backbone models (e.g., GCN, GraphSAGE, GAT, and SGC) to enhance their performance in the node classification task. In addition, EGAI reduces the convergence speed of over-smoothing that occurs when models are deepened. Extensive experiments on three real-world networks demonstrate that EGAI indeed improves the performance for both shallow and deep GNN models, and to some extent, mitigates over-smoothing. The code is available at https://github.com/liucoo/egai.