Manifold filter-combine networks

In recent years, several machine-learning (ML) solutions have been proposed to solve the problems of seizure detection, seizure phase classification, seizure prediction, and seizure onset zone (SOZ) localization, achieving excellent performance with accuracy levels above 95%. However, none of these solutions has been fully deployed in clinical settings. The primary reason has been a lack of trust from clinicians towards the complex decision-making operability of ML. More recently, research efforts have focused on systematized and generalizable frameworks of ML models that are clinician-friendly. In this paper, we propose a generalizable pipeline that leverages graph representation data structures as a flexible tool for graph neural networks. Moreover, we conducted an analysis of graph neural networks (GNN), a paradigm of artificial neural networks optimized to operate on graph-structured data, as a framework to classify seizure phases (preictal vs. ictal vs. postictal) from intracranial electroencephalographic (iEEG) data. We employed two multi-center international datasets, comprising 23 and 16 patients and 5 and 7 h of iEEG recordings. We evaluated four GNN models, with the highest performance achieving a seizure phase classification accuracy of 97%, demonstrating its potential for clinical application. Moreover, we show that by leveraging t-SNE, a statistical method for visualizing high-dimensional data, we can analyze how GNN’s influence the iEEG and graph representation embedding space. We also discuss the scientific implications of our findings and provide insights into future research directions for enhancing the generalizability of ML models in clinical practice.

Curvature graph neural network

Guest Editorial: Special issue on machine learning and deep learning algorithms for complex networks

Graph neural networks (GNNs) build on the success of deep learning models by extending them for use in graph spaces. Transfer learning has proven extremely successful for traditional deep learning problems, resulting in faster training and improved performance. Despite the increasing interest in GNNs and their use cases, there is little research on their transferability. This research demonstrates that transfer learning is effective with GNNs, and describes how source tasks and the choice of GNN impact the ability to learn generalisable knowledge. We perform experiments using real-world and synthetic data within the contexts of node classification and graph classification. To this end, we also provide a general methodology for transfer learning experimentation and present a novel algorithm for generating synthetic graph classification tasks. We compare the performance of GCN, GraphSAGE and GIN across both synthetic and real-world datasets. Our results demonstrate empirically that GNNs with inductive operations yield statistically significantly improved transfer. Further, we show that similarity in community structure between source and target tasks support statistically significant improvements in transfer over and above the use of only the node attributes.

Graph Neural Networks (GNNs) leverage the structural properties of the graph to inform the architecture of the neural network, thus achieving improved accuracy in graph learning tasks. However, like many neural network models, GNNs face a significant challenge with interpretability. To mitigate this issue, recent works have developed post-hoc instance-level explanation methods that focus on identifying minimal and sufficient subgraphs which strongly influence GNN predictions. Approaches that build on the graph information bottleneck principle (GIB) to quantify minimality and sufficiency have received particular attention, and have been used in several state-of-the-art explanation mechanisms. This work identifies several fundamental issues in such quantifications, particularly a signaling issue in the sufficiency, and a redundancy issue in the minimality quantifications. These may lead to explanations that do not accurately reflect the rationale behind GNN decisions. To overcome these challenges, we propose a new objective function and explainer architecture, dubbed the SculptEdgeX. The SculptEdgeX framework assesses the sufficiency of an input subgraph by generating an in-distribution supergraph and evaluating its prediction accuracy when processed by the GNN. This involves an initial densification process that adds edges to the input graph, followed by a selective edge removal step — called edge sculpting — to produce an in-distribution supergraph. To ensure the in-distribution property, we pre-train a calibrator network that parametrizes the underlying distribution of a given graph, hence enabling us to compare the distribution parameters with those of the original input distribution. We validate our method through extensive experiments on both synthetic and real-world datasets, demonstrating the effectiveness of SculptEdgeX in producing informative explanations.

Graph Neural Networks (GNNs) are proposed without considering the agnostic distribution shifts between training graphs and testing graphs, inducing the degeneration of the generalization ability of GNNs in Out-Of-Distribution (OOD) settings. The fundamental reason for such degeneration is that most GNNs are developed based on the I.I.D hypothesis. In such a setting, GNNs tend to exploit subtle statistical correlations existing in the training set for predictions, even though it is a spurious correlation. This learning mechanism inherits from the common characteristics of machine learning approaches. However, such spurious correlations may change in the wild testing environments, leading to the failure of GNNs. Therefore, eliminating the impact of spurious correlations is crucial for stable GNN models. To this end, in this paper, we argue that the spurious correlation exists among subgraph-level units and analyze the degeneration of GNN in causal view. Based on the causal view analysis, we propose a general causal representation framework for stable GNN, called StableGNN. The main idea of this framework is to extract high-level representations from raw graph data first and resort to the distinguishing ability of causal inference to help the model get rid of spurious correlations. Particularly, to extract meaningful high-level representations, we exploit a differentiable graph pooling layer to extract subgraph-based representations by an end-to-end manner. Furthermore, inspired by the confounder balancing techniques from causal inference, based on the learned high-level representations, we propose a causal variable distinguishing regularizer to correct the biased training distribution by learning a set of sample weights. Hence, GNNs would concentrate more on the true connection between discriminative substructures and labels. Extensive experiments are conducted on both synthetic datasets with various distribution shift degrees and eight real-world OOD graph datasets. The results well verify that the proposed model StableGNN not only outperforms the state-of-the-arts but also provides a flexible framework to enhance existing GNNs. In addition, the interpretability experiments validate that StableGNN could leverage causal structures for predictions.

Processing-in-memory (PIM) has been proposed as an alternative computing paradigm for training Deep Neural Networks, including Graph Neural Networks (GNNs). Despite these advancements, training GNN workloads on PIM devices necessitates off-chip memory access. This off-chip access is expensive in terms of performance and energy, thereby impacting the overall energy efficiency of the training process on PIM platforms. In this article, we propose a novel hybrid training framework called ERGo that automatically switches from a full-parameter phase to a parameter-efficient phase during the training process with negligible loss in predictive accuracy. The parameter-efficient phase employs low-rank representations of GNN weights, effectively reducing the number of trainable parameters. This helps in reducing the off-chip access during the end-to-end training process on PIM architectures, leading to notable improvements in both performance and energy efficiency. ERGo outperforms the conventional full-parameter GNN training on a PIM-based platform by up to 3.15x in speedup and enhances energy efficiency by 10.5x. Furthermore, ERGo offers additional advantages when utilized on heterogeneous architectures incorporating non-volatile memory (NVM) and static random-access memory (SRAM). Training on the heterogeneous NVM-SRAM architecture typically utilizes NVM for the forward-pass and SRAM for the backward-pass. However, NVM cells suffer from low device endurance and repeated write operations due to weight updates can lead to poor lifetime. ERGo improves the lifetime of NVM devices by freezing the weights and preventing weight updates in the parameter-efficient phase. Experimental results demonstrate that ERGo improves the lifetime of heterogeneous NVM-SRAM architecture by up to 33x in comparison to full-parameter implementation.

The Vapnik–Chervonenkis dimension of graph and recursive neural networks

Graph neural networks (GNNs) are emerging as a powerful technique for modeling graph structures. Due to the sparsity of real-world graph data, GNN performance is limited by extensive sparse matrix multiplication (SpMM) operations involved in computation. While the right sparse matrix storage format varies across input data, existing deep learning frameworks employ a single, static storage format, leaving much room for improvement. This paper investigates how the choice of sparse matrix storage formats affect the GNN performance. We observe that choosing a suitable sparse matrix storage format can significantly improve the GNN training performance, but the right format depends on the input workloads and can change as the GNN iterates over the input graph. We then develop a predictive model to dynamically choose a sparse matrix storage format to be used by a GNN layer based on the input matrices. Our model is first trained offline using training matrix samples, and the trained model can be applied to any input matrix and GNN kernels with SpMM computation. We implement our approach on top of PyTorch and apply it to 5 representative GNN models running on a multi-core CPU using real-life and synthetic datasets. Experimental results show that our approach gives an average speedup of 1.17x (up to 3x) for GNN running time.

Graph Neural Networks (GNNs) demonstrate their significance by effectively modeling complex interrelationships within graph-structured data. To enhance the credibility and robustness of GNNs, it becomes exceptionally crucial to bolster their ability to capture causal relationships. However, despite recent advancements that have indeed strengthened GNNs from a causal learning perspective, conducting an in-depth analysis specifically targeting the causal modeling prowess of GNNs remains an unresolved issue. In order to comprehensively analyze various GNN models from a causal learning perspective, we constructed an artificially synthesized dataset with known and controllable causal relationships between data and labels. The rationality of the generated data is further ensured through theoretical foundations. Drawing insights from analyses conducted using our dataset, we introduce a lightweight and highly adaptable GNN module designed to strengthen GNNs' causal learning capabilities across a diverse range of tasks. Through a series of experiments conducted on both synthetic datasets and other real-world datasets, we empirically validate the effectiveness of the proposed module. The codes are available at https://github.com/yaoyao-yaoyao-cell/CRCG.

Recent studies show that models trained on synthetic datasets are able to outperform models trained on real-world datasets for generalizable person re-identification (GPReID). On the other hand, due to the limitations of real-world person ReID datasets, it would also be important and interesting to use large-scale synthetic datasets as test sets to benchmark algorithms. Yet this raises a critical question: is synthetic dataset reliable for benchmarking GPReID? In the literature there is no evidence showing this. To address this, we design a method called Pair-wise Ranking Analysis (PRA) to quantitatively measure the ranking similarity and perform the statistical test of identical distributions. Specifically, we employ Kendall rank correlation coefficients to evaluate pairwise similarity values between algorithm rankings on different datasets. Then, a non-parametric two-sample Kolmogorov-Smirnov (KS) test is performed for the judgement of whether algorithm ranking correlations between synthetic and real-world datasets and those only between real-world datasets lie in identical distributions. We conduct comprehensive experiments, with ten representative algorithms, three popular real-world person ReID datasets, and three recently released large-scale synthetic datasets. Through the designed pairwise ranking analysis and comprehensive evaluations, we conclude that a recent large-scale synthetic dataset ClonedPerson can be reliably used to benchmark GPReID, statistically the same as real-world datasets. Therefore, this study guarantees the usage of synthetic datasets for both source training set and target testing set, with completely no privacy concerns from real-world surveillance data. Besides, the study in this paper might also inspire future designs of synthetic datasets.

Prediction of chemical yields is crucial for exploring untapped chemical reactions and optimizing synthetic pathways for targeted compounds. Recently, graph neural networks have proven successful in achieving high predictive accuracy. However, they remain intrinsically black-box models, offering limited interpretability. Understanding how each reaction component contributes to the yield of a chemical reaction can help identify critical factors driving the success or failure of reactions, thereby potentially revealing opportunities for yield optimization. In this study, we present a novel method for interpretable chemical reaction yield prediction, which represents the yield of a chemical reaction as a simple summation of component-wise contributions from individual reaction components. To build an interpretable prediction model, we introduce a graph neural additive network architecture, wherein shared neural networks process individual reaction components in an input reaction while leveraging a reaction-level embedding to derive their respective contributions. The predicted yield is obtained by summing these component-wise contributions. The model is trained using a learning objective designed to effectively quantify the contributions of individual components by amplifying the influence of significant components and suppressing that of less influential components. The experimental results on benchmark datasets demonstrated that the proposed method achieved both high predictive accuracy and interpretability, making it suitable for practical use in synthetic pathway design for real-world applications.

Graph Neural Networks (GNNs) are a powerful tool for processing graphs, that represent a natural way to collect information coming from several areas of science and engineering - e.g. data mining, computer vision, molecular chemistry, molecular biology, pattern recognition -, where data are intrinsically organized in entities and relationships among entities. Nevertheless, GNNs suffer, so as recurrent/recursive models, from the long-term dependency problem that makes the learning difficult in deep structures. In this paper, we present a new architecture, called Layered GNN (LGNN), realized by a cascade of GNNs: each layer is fed with the original data and with the state information calculated by the previous layer in the cascade. Intuitively, this allows each GNN to solve a subproblem, related only to those patterns that were misclassified by the previous GNNs. Some experimental results are reported, based on synthetic and real-world datasets, which assess a significant improvement in performances w.r.t. the standard GNN approach.

Hypergraph, an expressive structure with flexibility to model the higher-order correlations among entities, has recently attracted increasing attention from various research domains. Despite the success of Graph Neural Networks (GNNs) for graph representation learning, how to adapt the powerful GNN-variants directly into hypergraphs remains a challenging problem. In this paper, we propose UniGNN, a unified framework for interpreting the message passing process in graph and hypergraph neural networks, which can generalize general GNN models into hypergraphs. In this framework, meticulously-designed architectures aiming to deepen GNNs can also be incorporated into hypergraphs with the least effort. Extensive experiments have been conducted to demonstrate the effectiveness of UniGNN on multiple real-world datasets, which outperform the state-of-the-art approaches with a large margin. Especially for the DBLP dataset, we increase the accuracy from 77.4% to 88.8% in the semi-supervised hypernode classification task. We further prove that the proposed message-passing based UniGNN models are at most as powerful as the 1-dimensional Generalized Weisfeiler-Leman (1-GWL) algorithm in terms of distinguishing non-isomorphic hypergraphs. Our code is available at https://github.com/OneForward/UniGNN.

The recent work ``Combinatorial Optimization with Physics-Inspired Graph Neural Networks'' [Nat Mach Intell 4 (2022) 367] introduces a physics-inspired unsupervised Graph Neural Network (GNN) to solve combinatorial optimization problems on sparse graphs. To test the performances of these GNNs, the authors of the work show numerical results for two fundamental problems: maximum cut and maximum independent set (MIS). They conclude that "the graph neural network optimizer performs on par or outperforms existing solvers, with the ability to scale beyond the state of the art to problems with millions of variables." In this comment, we show that a simple greedy algorithm, running in almost linear time, can find solutions for the MIS problem of much better quality than the GNN. The greedy algorithm is faster by a factor of $10^4$ with respect to the GNN for problems with a million variables. We do not see any good reason for solving the MIS with these GNN, as well as for using a sledgehammer to crack nuts. In general, many claims of superiority of neural networks in solving combinatorial problems are at risk of being not solid enough, since we lack standard benchmarks based on really hard problems. We propose one of such hard benchmarks, and we hope to see future neural network optimizers tested on these problems before any claim of superiority is made.