Graph Neural Architecture Research Articles

Asymmetric augmented paradigm-based graph neural architecture search

In most scenarios of graph-based tasks, graph neural networks (GNNs) are trained end-to-end with labeled samples. Labeling graph samples, a time-consuming and expert-dependent process, leads to huge costs. Graph data augmentations can provide a promising method to expand labeled samples cheaply. However, graph data augmentations will damage the capacity of GNNs to distinguish non-isomorphic graphs during the supervised graph representation learning process. How to utilize graph data augmentations to expand labeled samples while preserving the capacity of GNNs to distinguish non-isomorphic graphs is a challenging research problem. To address the above problem, we abstract a novel asymmetric augmented paradigm in this paper and theoretically prove that it offers a principled approach. The asymmetric augmented paradigm can preserve the capacity of GNNs to distinguish non-isomorphic graphs while utilizing augmented labeled samples to improve the generalization capacity of GNNs. To be specific, the asymmetric augmented paradigm will utilize similar yet distinct asymmetric weights to classify the real sample and augmented sample, respectively. To systemically explore the benefits of asymmetric augmented paradigm under different GNN architectures, rather than studying individual asymmetric augmented GNN (A2GNN) instance, we then develop an auto-search engine called Asymmetric Augmented Graph Neural Architecture Search (A2GNAS) to save human efforts. We empirically validate our asymmetric augmented paradigm on multiple graph classification benchmarks, and demonstrate that representative A2GNN instances automatically discovered by our A2GNAS method achieve state-of-the-art performance compared with competitive baselines. Our codes are available at: https://github.com/csubigdata-Organization/A2GNAS.

Optimizing graph neural network architectures for schizophrenia spectrum disorder prediction using evolutionary algorithms

Background and Objective:The accurate diagnosis of schizophrenia spectrum disorder plays an important role in improving patient outcomes, enabling timely interventions, and optimizing treatment plans. Functional connectivity analysis, utilizing functional magnetic resonance imaging data, has been demonstrated to offer invaluable biomarkers conducive to clinical diagnosis. However, previous studies mainly focus on traditional machine learning methods or hand-crafted neural networks, which may not fully capture the spatial topological relationship between brain regions. Methods:This paper proposes an evolutionary algorithm (EA) based graph neural architecture search (GNAS) method. EA-GNAS has the ability to search for high-performance graph neural networks for schizophrenia spectrum disorder prediction. Moreover, we adopt GNNExplainer to investigate the explainability of the acquired architectures, ensuring that the model’s predictions are both accurate and comprehensible. Results:The results suggest that the graph neural network model, derived using genetic algorithm search, outperforms under five-fold cross-validation, achieving a fitness of 0.1850. Relative to conventional machine learning and other deep learning approaches, the proposed method yields superior accuracy, F1 score, and AUC values of 0.8246, 0.8438, and 0.8258, respectively. Conclusion:Based on a multi-site dataset from schizophrenia spectrum disorder patients, the findings reveal an enhancement over prior methods, advancing our comprehension of brain function and potentially offering a biomarker for diagnosing schizophrenia spectrum disorder.

Contrastive meta-reinforcement learning for heterogeneous graph neural architecture search

Heterogeneous Graph Neural Networks (HGNNs) have demonstrated significant success in capturing complex interactions within heterogeneous graphs to learn graph representations. However, designing effective HGNN architecture is a challenging task that demands substantial domain knowledge and human effort. Fortunately, the advent of Heterogeneous Graph Architecture Search (HGNAS) has automated this process and yielded networks surpassing those designed by humans. Nonetheless, conventional HGNAS approaches are limited in their ability to generalize to unseen or diverse task types. To address these limitations, we propose an efficient method called Contrastive Meta-reinforcement learning-based Heterogeneous Graph Neural Architecture Search (CM-HGNAS). Our approach aims to produce high-performance networks tailored to different types of tasks. Firstly, we use gradient-based meta-learning to rapidly adapt to new tasks by leveraging the knowledge acquired from meta-training tasks. Secondly, since our meta-training tasks encompass various types of tasks, including node classification and link prediction, we employ contrastive learning to unify the evaluation metric across all tasks. This unification enhances the generalization capabilities of our model across diverse tasks. Experimental results demonstrate that our method successfully generalizes to unseen tasks and surpasses existing HGNN baselines and HGNAS techniques in terms of performance.

Depth-adaptive graph neural architecture search for graph classification

In recent years, graph neural networks (GNNs) based on neighborhood aggregation schemes have become a promising method in various graph-based applications. To solve the expert-dependent and time-consuming problem in human-designed GNN architectures, graph neural architecture search (GNAS) has been popular. However, as the mainstream GNAS methods automatically design GNN architectures with fixed GNN depth, they cannot mine the true potential of GNN architectures for graph classification. Although a few GNAS methods have explored the importance of adaptive GNN depth based on fixed GNN architectures, they have not designed a general search space for graph classification, which limits the discovery of excellent GNN architectures. In this paper, we propose Depth-Adaptive Graph Neural Architecture Search for Graph Classification (DAGC), which systemically constructs and explores the search space for graph classification, rather than studying individual designs. Through decoupling the graph classification process, DAGC proposes a complete and flexible search space, including GNN depth, aggregation function, and pooling operation components. To this end, DAGC adopts a learnable agent based on reinforcement learning to effectively guide the search for depth-adaptive GNN architectures. Extensive experiments on five real-world datasets demonstrate that DAGC outperforms the state-of-the-art human-designed GNN architectures and GNAS methods. The code is available at: https://github.com/Zhen-Peng-Wu/DAGC.

Component importance preference-based evolutionary graph neural architecture search

Recently, graph neural architecture search (GNAS) has become an increasingly hot research topic as a promising technique for automatically searching graph neural networks (GNNs) with no or little domain expertise. The search space and optimization strategy are the core factors of GNAS. However, the search space of existing GNAS methods is limited, and their optimization strategies treat components indiscriminately. In the current study, a more expressive search space is first designed. Subsequently, it is illustrated that components contribute different importance to data-specific tasks or datasets, and this study assumes component importance as a probability parameter. To this end, a component importance preference-based evolutionary GNAS method (called CIPE) is proposed. CIPE defines component importance and its preference selection and updating method. Subsequently, the designed importance preference-guided multipoint crossover and multistrategy mutation operators are applied to the evolutionary process. Finally, the effectiveness of CIPE is verified for transductive and inductive tasks. The experimental results demonstrate the validity of the component importance assumption and the superiority of CIPE compared with the state-of-the-art handcrafted GNNs and GNAS methods on all eight datasets. The mean accuracy obtained by CIPE on the datasets Cora, CiteSeer, PubMed, Cornell, Texas, Wisconsin, and Chameleon is 83.84%, 73.23%, 80.28%, 78.38%, 86.49%, 82.35%, and 73.59%, respectively. Specifically, the mean accuracy is improved by 2.71% and 5.28% on the datasets Texas and Chameleon, respectively. The mean F1-score obtained by CIPE on the dataset PPI is 99.37%, with an improvement of 0.24%. The code is available at https://github.com/chnyliu/CIPE.

AutoAMS: Automated attention-based multi-modal graph learning architecture search

Multi-modal attention mechanisms have been successfully used in multi-modal graph learning for various tasks. However, existing attention-based multi-modal graph learning (AMGL) architectures heavily rely on manual design, requiring huge effort and expert experience. Meanwhile, graph neural architecture search (GNAS) has made great progress toward automatically designing graph-based learning architectures. However, it is challenging to directly adopt existing GNAS methods to search for better AMGL architectures because of the search spaces that only focus on designing graph neural network architectures and the search objective that ignores multi-modal interactive information between modalities and long-term content dependencies within different modalities. To address these issues, we propose an automated attention-based multi-modal graph learning architecture search (AutoAMS) framework, which can automatically design the optimal AMGL architectures for different multi-modal tasks. Specifically, we design an effective attention-based multi-modal (AM) search space consisting of four sub-spaces, which can jointly support the automatic search of multi-modal attention representation and other components of multi-modal graph learning architecture. In addition, a novel search objective based on an unsupervised multi-modal reconstruction loss and task-specific loss is introduced to search and train AMGL architectures. The search objective can extract the global features and capture multi-modal interactions from multiple modalities. The experimental results on multi-modal tasks show strong evidence that AutoAMS is capable of designing high-performance AMGL architectures.

Adaptive multi-scale Graph Neural Architecture Search framework

Graph neural networks (GNNs) have gained significant attention for their ability to learn representations from graph-structured data, in which message passing and feature fusion strategies play an essential role. However, traditional Graph Neural Architecture Search (GNAS) mainly focuses on optimization with a static perceptive field to ease the search process. To efficiently utilize latent relationships between non-adjacent nodes as well as edge features, this work proposes a novel two-stage approach that is able to optimize GNN structures more effectively by adaptively aggregating neighborhood features in multiple scales. This adaptive multi-scale GNAS is able to assign optimal weights for different neighbors in different graphs and learning tasks. In addition, it takes latent relationships and edge features into message passing into account, and can incorporate different feature fusion strategies. Compared with traditional ones, our proposed approach can explore a much larger and more diversified search space efficiently. We also prove that traditional multi-hop GNNs are low-pass filters, which can lead to the removal of important low-frequency components of signals from remote neighbors in a graph, and they are not even expressive enough to distinguish some simple regular graphs, justifying the superiority of our approach. Experiments with seven datasets across three graph learning tasks, including graph regression, node classification, and graph classification, demonstrate that our method yields significant improvement compared with state-of-the-art GNAS approaches and human-designed GNN approaches. Specifically, for example, with our framework, the MAE of the 12-layer AM-GNAS was 0.102 for the ZINC dataset, yielding over 25% improvement.

Decoupled differentiable graph neural architecture search

The differentiable graph neural architecture search (GNAS) effectively designs graph neural networks (GNNs) efficiently and automatically with excellent performance based on different graph data distributions. Given a GNN search space containing multiple GNN component operation candidates, the differentiable GNAS method builds a mixed supernet using learnable architecture parameters multiplied by the GNN component operation candidates. When the mixed supernet completes optimization, the mixed supernet is pruned based on the best architecture parameters to efficiently identify the optimal GNN architecture in the GNN search space. However, the multiplicative relationship between the architecture parameters and the GNN component operation candidates introduces a coupled optimization bias into the weight optimization process of the mixed supernet GNN component operation candidates. This bias results in differentiable GNAS performance degradation. To solve the problem of coupled optimization bias in the previous differentiable GNAS method, we propose the Decoupled Differentiable Graph Neural Architecture Search (D2GNAS). It utilizes the Gumbel distribution as a bridge to decouple the weights optimization of supernet GNN component candidate operation and architecture parameters for constructing the decoupled differentiable GNN architecture sampler. The sampler is capable of selecting promising GNN architectures based on architecture parameters treated as sampling probabilities, and it is further optimized through the validation gradients derived from the sampled GNN architectures. Simultaneously, D2GNAS builds a single-path supernet with a pruning strategy to compress the supernet progressively to improve search efficiency further. We conduct extensive experiments on multiple benchmark graphs. The experimental findings demonstrate that D2GNAS outperforms all established baseline methods, both manual GNN and GNAS methods, in terms of performance. Additionally, D2GNAS has a lower time complexity than previous differentiable GNAS methods. Based on the fair GNN search space, it achieves an average 5x efficiency improvement. Codes are available at https://github.com/AutoMachine0/D2GNAS.

Graph neural architecture search with heterogeneous message-passing mechanisms

Graph neural architecture search with heterogeneous message-passing mechanisms

GANN: Graph Alignment Neural Network for semi-supervised learning

Graph neural networks (GNNs) have been widely investigated in the field of semi-supervised graph machine learning. Most methods fail to exploit adequate graph information when labeled data is limited, leading to the problem of oversmoothing. To overcome this issue, we propose the Graph Alignment Neural Network (GANN), a simple and effective graph neural architecture. A unique learning algorithm with three alignment rules is proposed to thoroughly explore hidden information for insufficient labels. Firstly, to better investigate attribute specifics, we suggest the feature alignment rule to align the inner product of both the attribute and embedding matrices. Secondly, to properly utilize the higher-order neighbor information, we propose the cluster center alignment rule, which involves aligning the inner product of the cluster center matrix with the unit matrix. Finally, to get reliable prediction results with few labels, we establish the minimum entropy alignment rule by lining up the prediction probability matrix with its sharpened result. Extensive studies on graph benchmark datasets demonstrate that GANN can achieve considerable benefits in semi-supervised node classification and outperform state-of-the-art competitors.

CommGNAS: Unsupervised Graph Neural Architecture Search for Community Detection

Graph neural architecture search (GNAS) has been successful in many supervised learning tasks, such as node classification, graph classification, and link prediction. GNAS uses a search algorithm to sample graph neural network (GNN) architectures from the search space and evaluates sampled GNN architectures based on estimation strategies to generate feedback for the search algorithm. In traditional GNAS, the typical estimation strategy requires using labeled graph data to generate feedback, which plays a fundamental and vital role in the search algorithm to sample a better GNN architecture during the search process. However, a large portion of real-world graph data is unlabeled. The estimation strategy in traditional GNAS cannot use unlabeled graph data to generate feedback for the search algorithm, so the traditional supervised GNAS fails to solve unsupervised problems, such as community detection tasks. To solve this challenge, this paper proposed CommGNAS, an effective node representation learning method with unsupervised graph neural architecture search for community detection. In CommGNAS, we design an unsupervised evaluation strategy with self-supervised and self-representation learning. It represents the first research work in literature to solve the problems of unsupervised graph neural architecture search for community detection. The experimental results show that CommGNAS can obtain the best performance in community detection tasks on real-world graphs against the state-of-the-art baseline methods.

Data-Augmented Curriculum Graph Neural Architecture Search under Distribution Shifts

Graph neural architecture search (NAS) has achieved great success in designing architectures for graph data processing.However, distribution shifts pose great challenges for graph NAS, since the optimal searched architectures for the training graph data may fail to generalize to the unseen test graph data. The sole prior work tackles this problem by customizing architectures for each graph instance through learning graph structural information, but failed to consider data augmentation during training, which has been proven by existing works to be able to improve generalization.In this paper, we propose Data-augmented Curriculum Graph Neural Architecture Search (DCGAS), which learns an architecture customizer with good generalizability to data under distribution shifts. Specifically, we design an embedding-guided data generator, which can generate sufficient graphs for training to help the model better capture graph structural information. In addition, we design a two-factor uncertainty-based curriculum weighting strategy, which can evaluate the importance of data in enabling the model to learn key information in real-world distribution and reweight them during training. Experimental results on synthetic datasets and real datasets with distribution shifts demonstrate that our proposed method learns generalizable mappings and outperforms existing methods.

Multimodal Graph Neural Architecture Search under Distribution Shifts

Multimodal graph neural architecture search (MGNAS) has shown great success for automatically designing the optimal multimodal graph neural network (MGNN) architecture by leveraging multimodal representation, crossmodal information and graph structure in one unified framework. However, existing MGNAS fails to handle distribution shifts that naturally exist in multimodal graph data, since the searched architectures inevitably capture spurious statistical correlations under distribution shifts. To solve this problem, we propose a novel Out-of-distribution Generalized Multimodal Graph Neural Architecture Search (OMG-NAS) method which optimizes the MGNN architecture with respect to its performance on decorrelated OOD data. Specifically, we propose a multimodal graph representation decorrelation strategy, which encourages the searched MGNN model to output representations that eliminate spurious correlations through iteratively optimizing the feature weights and controller. In addition, we propose a global sample weight estimator that facilitates the sharing of optimal sample weights learned from existing architectures. This design promotes the effective estimation of the sample weights for candidate MGNN architectures to generate decorrelated multimodal graph representations, concentrating more on the truly predictive relations between invariant features and ground-truth labels. Extensive experiments on real-world multimodal graph datasets demonstrate the superiority of our proposed method over SOTA baselines.

Trajectory-User Linking via Hierarchical Spatio-Temporal Attention Networks

Trajectory-User Linking (TUL) is crucial for human mobility modeling by linking different trajectories to users with the exploration of complex mobility patterns. Existing works mainly rely on the recurrent neural framework to encode the temporal dependencies in trajectories, have fall short in capturing spatial-temporal global context for TUL prediction. To fill this gap, this work presents a new hierarchical spatio-temporal attention neural network, called AttnTUL , to jointly encode the local trajectory transitional patterns and global spatial dependencies for TUL. Specifically, our first model component is built over the graph neural architecture to preserve the local and global context and enhance the representation paradigm of geographical regions and user trajectories. Additionally, a hierarchically structured attention network is designed to simultaneously encode the intra-trajectory and inter-trajectory dependencies, with the integration of the temporal attention mechanism and global elastic attentional encoder. Extensive experiments demonstrate the superiority of our AttnTUL method as compared to state-of-the-art baselines on various trajectory datasets. The source code of our model is available at https://github.com/Onedean/AttnTUL .

Uncertainty quantification for molecular property predictions with graph neural architecture search

AutoGNNUQ employs neural architecture search to enhance uncertainty quantification for molecular property prediction via graph neural networks.

Reinforced GNNs for Multiple Instance Learning.

Multiple instance learning (MIL) trains models from bags of instances, where each bag contains multiple instances, and only bag-level labels are available for supervision. The application of graph neural networks (GNNs) in capturing intrabag topology effectively improves MIL. Existing GNNs usually require filtering low-confidence edges among instances and adapting graph neural architectures to new bag structures. However, such asynchronous adjustments to structure and architecture are tedious and ignore their correlations. To tackle these issues, we propose a reinforced GNN framework for MIL (RGMIL), pioneering the exploitation of multiagent deep reinforcement learning (MADRL) in MIL tasks. MADRL enables the flexible definition or extension of factors that influence bag graphs or GNNs and provides synchronous control over them. Moreover, MADRL explores structure-to-architecture correlations while automating adjustments. Experimental results on multiple MIL datasets demonstrate that RGMIL achieves the best performance with excellent explainability. The code and data are available at https://github.com/RingBDStack/RGMIL.

A Comprehensive Survey on Deep Graph Representation Learning Methods

There has been a lot of activity in graph representation learning in recent years. Graph representation learning aims to produce graph representation vectors to represent the structure and characteristics of huge graphs precisely. This is crucial since the effectiveness of the graph representation vectors will influence how well they perform in subsequent tasks like anomaly detection, connection prediction, and node classification. Recently, there has been an increase in the use of other deep-learning breakthroughs for data-based graph problems. Graph-based learning environments have a taxonomy of approaches, and this study reviews all their learning settings. The learning problem is theoretically and empirically explored. This study briefly introduces and summarizes the Graph Neural Architecture Search (G-NAS), outlines several Graph Neural Networks’ drawbacks, and suggests some strategies to mitigate these challenges. Lastly, the study discusses several potential future study avenues yet to be explored.

MaGNAS: A Mapping-Aware Graph Neural Architecture Search Framework for Heterogeneous MPSoC Deployment

Graph Neural Networks (GNNs) are becoming increasingly popular for vision-based applications due to their intrinsic capacity in modeling structural and contextual relations between various parts of an image frame. On another front, the rising popularity of deep vision-based applications at the edge has been facilitated by the recent advancements in heterogeneous multi-processor Systems on Chips (MPSoCs) that enable inference under real-time, stringent execution requirements. By extension, GNNs employed for vision-based applications must adhere to the same execution requirements. Yet contrary to typical deep neural networks, the irregular flow of graph learning operations poses a challenge to running GNNs on such heterogeneous MPSoC platforms. In this paper, we propose a novel unified design-mapping approach for efficient processing of vision GNN workloads on heterogeneous MPSoC platforms. Particularly, we develop MaGNAS, a mapping-aware Graph Neural Architecture Search framework. MaGNAS proposes a GNN architectural design space coupled with prospective mapping options on a heterogeneous SoC to identify model architectures that maximize on-device resource efficiency. To achieve this, MaGNAS employs a two-tier evolutionary search to identify optimal GNNs and mapping pairings that yield the best performance trade-offs. Through designing a supernet derived from the recent Vision GNN (ViG) architecture, we conducted experiments on four (04) state-of-the-art vision datasets using both ( i ) a real hardware SoC platform (NVIDIA Xavier AGX) and ( ii ) a performance/cost model simulator for DNN accelerators. Our experimental results demonstrate that MaGNAS is able to provide 1.57 × latency speedup and is 3.38 × more energy-efficient for several vision datasets executed on the Xavier MPSoC vs. the GPU-only deployment while sustaining an average 0.11% accuracy reduction from the baseline.

HGNAS++: Efficient Architecture Search for Heterogeneous Graph Neural Networks

Heterogeneous graphs are commonly used to describe networked data with multiple types of nodes and edges. Heterogeneous Graph Neural Networks (HGNNs) are powerful tools for analyzing heterogeneous graphs. However, designing neural architectures of HGNNs requires extensive domain knowledge and time-consuming manual work. Recently, neural architecture search algorithms have become popular in automatically designing neural architectures for homogeneous graph neural networks. In this paper, we present a Heterogeneous Graph Neural Architecture Search algorithm (HGNAS for short) which allows the automatic design of heterogeneous graph neural architectures. Specifically, HGNAS first designs a new search space based on existing popular HGNNs. Then, HGNAS uses a policy network as the controller to sample and find the best neural architecture from the designed search space by maximizing the expected accuracy of the selected architectures on a given validation dataset. Moreover, we design a new method HGNAS++ to improve the efficiency of HGNAS by training the RNN controller within a generative adversarial learning framework. The basic idea of HGNAS++ is to embed a pairwise ranker into the reinforcement learning based architecture search algorithm. The pairwise ranker can be taken as a discriminator which selects more accurate architectures between pairs of candidate architectures. Then, the RNN controller can be updated more efficiently by only using a relatively small number of candidate architectures selected by the pairwise ranker. Experiments on real-world heterogeneous graph datasets show that HGNAS is capable of designing novel HGNNs that beat the best human-invented HGNNs. On the benchmark datasets, HGNAS++ improves HGNAS in terms of evaluation cost, with a reduction of 50% of the evaluated candidate architectures and a decrease of 24% in search time on average. As a byproduct, HGNAS++ can find sparse yet powerful neural architectures for HGNNs.

Graph neural architecture prediction

Graph neural architecture prediction