Hierarchical Graph Convolution Network Research Articles

Online Fault Diagnosis of Harmonic Drives Using Semisupervised Contrastive Graph Generative Network via Multimodal Data

Harmonic drive is a core component of the industrial robot, its failure will directly affect the robot's performance. Moreover, as the harmonic drive often works with excessive speed and load, it may fail unpredictably. Therefore, online fault diagnosis is quite significant. In this paper, we propose an online intelligent fault diagnosis method for harmonic drives using a Semi-supervised Contrastive Graph Generative Network (SCGGN) via multimodal data. First, multimodal data (including motor current signals and encoder signals) of the harmonic drive are collected online. The Euclidean distance is used to analyze the similarity of the data in the frequency domain. Second, Multiple Graph Convolution Network (MGCN) and Hierarchical Graph Convolution Network (HGCN) are used to obtain complementary fault features from local and global views, respectively. Third, the Contrastive Learning Network is constructed to obtain high-level information through unsupervised learning and perform data clustering to obtain the multi-classification output. Finally, a combination of learnable loss functions is used to optimize the SCGGN. The presented method is tested on an industrial robot. The experimental results show that the method can achieve 86.15% accuracy with 8% of the labeled training data and 79.9% accuracy with only 0.5% of the labeled training data, which is superior to the existing methods.

Hierarchical Graph Convolutional Network Built by Multiscale Atlases for Brain Disorder Diagnosis Using Functional Connectivity.

Functional connectivity network (FCN) data from functional magnetic resonance imaging (fMRI) is increasingly used for the diagnosis of brain disorders. However, state-of-the-art studies used to build the FCN using a single brain parcellation atlas at a certain spatial scale, which largely neglected functional interactions across different spatial scales in hierarchical manners. In this study, we propose a novel framework to perform multiscale FCN analysis for brain disorder diagnosis. We first use a set of well-defined multiscale atlases to compute multiscale FCNs. Then, we utilize biologically meaningful brain hierarchical relationships among the regions in multiscale atlases to perform nodal pooling across multiple spatial scales, namely "Atlas-guided Pooling (AP)." Accordingly, we propose a multiscale-atlases-based hierarchical graph convolutional network (MAHGCN), built on the stacked layers of graph convolution and the AP, for a comprehensive extraction of diagnostic information from multiscale FCNs. Experiments on neuroimaging data from 1792 subjects demonstrate the effectiveness of our proposed method in the diagnoses of Alzheimer's disease (AD), the prodromal stage of AD i.e., mild cognitive impairment (MCI), as well as autism spectrum disorder (ASD), with the accuracy of 88.9%, 78.6%, and 72.7%, respectively. All results show significant advantages of our proposed method over other competing methods. This study not only demonstrates the feasibility of brain disorder diagnosis using resting-state fMRI empowered by deep learning but also highlights that the functional interactions in the multiscale brain hierarchy are worth being explored and integrated into deep learning network architectures for a better understanding of the neuropathology of brain disorders. The codes for MAHGCN are publicly available at "https://github.com/MianxinLiu/ MAHGCN-code."

Hierarchical Graph Convolutional Networks for Action Quality Assessment

Action quality assessment (AQA) automatically evaluates how well humans perform actions in a given video, a technique widely used in fields such as rehabilitation medicine, athletic competitions, and specific skills assessment. However, existing works that uniformly divide the video sequence into small clips of equal length suffer from intra-clip confusion and inter-clip incoherence, hindering the further development of AQA. To address this issue, we propose a hierarchical graph convolutional network (GCN). First, semantic information confusion is corrected through clip refinement, generating the ‘shot’ as the basic action unit. We then construct a scene graph by combining several consecutive shots into meaningful scenes to capture local dynamics. These scenes can be viewed as different procedures of a given action, providing valuable assessment cues. The video-level representation is finally extracted via sequential action aggregation among scenes to regress the predicted score distribution, enhancing discriminative features and improving assessment performance. Experiments on the AQA-7, MTL-AQA, and JIGSAWS datasets demonstrate the superiority of the proposed hierarchical GCN over state-of-the-art methods.

A joint hierarchical cross‐attention graph convolutional network for multi‐modal facial expression recognition

AbstractEmotional recognition in conversations (ERC) is increasingly being applied in various IoT devices. Deep learning‐based multimodal ERC has achieved great success by leveraging diverse and complementary modalities. Although most existing methods try to adopt attention mechanisms to fuse different information, these methods ignore the complementarity between modalities. To this end, the joint cross‐attention model is introduced to alleviate this issue. However, multi‐scale feature information on different modalities is not utilized. Moreover, the context relationship plays an important role in feature extraction in the expression recognition task. In this paper, we propose a novel joint hierarchical graph convolution network (JHGCN) which exploits different layer features and context relationships for facial expression recognition based on audio‐visual (A‐V) information. Specifically, we adopt different deep networks to extract features from different modalities individually. For V modality, we construct V graph data based on patch embeddings which are extracted from the transformer encoder. Moreover, we embed the graph convolution which can leverage the intra‐modality relationships with the transformer encoder. Then, the deep feature from different layers is fed to the hierarchical fusion module to enhance feature representation. At last, we use the joint cross‐attention mechanism to exploit the complementary inter‐modality relationships. To validate the proposed model, we have conducted various experiments on the AffWild2 and CMU‐MOSI datasets. All results confirm that our proposed model achieves highly promising performance compared to the joint cross‐attention model and other methods.

Hierarchical Graph Convolutional Networks for Structured Long Document Classification.

Long document classification (LDC) has been a focused interest in natural language processing (NLP) recently with the exponential increase of publications. Based on the pretrained language models, many LDC methods have been proposed and achieved considerable progression. However, most of the existing methods model long documents as sequences of text while omitting the document structure, thus limiting the capability of effectively representing long texts carrying structure information. To mitigate such limitation, we propose a novel hierarchical graph convolutional network (HGCN) for structured LDC in this article, in which a section graph network is proposed to model the macrostructure of a document and a word graph network with a decoupled graph convolutional block is designed to extract the fine-grained features of a document. In addition, an interaction strategy is proposed to integrate these two networks as a whole by propagating features between them. To verify the effectiveness of the proposed model, four structured long document datasets are constructed, and the extensive experiments conducted on these datasets and another unstructured dataset show that the proposed method outperforms the state-of-the-art related classification methods.

Self-Supervised Learning for Multilevel Skeleton-Based Forgery Detection via Temporal-Causal Consistency of Actions

Skeleton-based human action recognition and analysis have become increasingly attainable in many areas, such as security surveillance and anomaly detection. Given the prevalence of skeleton-based applications, tampering attacks on human skeletal features have emerged very recently. In particular, checking the temporal inconsistency and/or incoherence (TII) in the skeletal sequence of human action is a principle of forgery detection. To this end, we propose an approach to self-supervised learning of the temporal causality behind human action, which can effectively check TII in skeletal sequences. Especially, we design a multilevel skeleton-based forgery detection framework to recognize the forgery on frame level, clip level, and action level in terms of learning the corresponding temporal-causal skeleton representations for each level. Specifically, a hierarchical graph convolution network architecture is designed to learn low-level skeleton representations based on physical skeleton connections and high-level action representations based on temporal-causal dependencies for specific actions. Extensive experiments consistently show state-of-the-art results on multilevel forgery detection tasks and superior performance of our framework compared to current competing methods.

Graph dropout self-learning hierarchical graph convolution network for traffic prediction

Traffic prediction is a challenging topic in urban traffic construction and management due to its complex dynamic spatial–temporal correlations. Currently, graph neural network achieves good results in traffic prediction, but the existing methods usually do not take into account the multi-layer information of the graph and the redundant transmission of information. Moreover, they do not consider the over-smoothing of the graph. In this paper, we propose a novel graph dropout self-learning hierarchical graph convolution network (DHGCN). Firstly, we design a self-learning hierarchical graph convolution network, which captures spatial features at different layers through multiple self-learning dynamic graphs and avoids redundant information transmission. Secondly, a novel graph dropout structure is proposed to sparsify the graph and avoid the over-smoothing of the graph. Meanwhile, an encoder–decoder architecture with gated residuals is designed to capture dynamic temporal features. In addition, this paper addresses two important traffic forecasting tasks: traffic flow and traffic speed. Extensive experiments with six real-world datasets verify that our method achieves state-of-the-art performance in both traffic flow and traffic speed and consistently outperforms baselines.

An extensible hierarchical graph convolutional network for early Alzheimer’s disease identification

Background and Objective: For early identification of Alzheimer’s disease (AD) based on multi-modal magnetic resonance imaging (MRI) data, it is important to make comprehensive use of image features and non-image information to analyze the gray matter atrophy and the structural/functional connectivity abnormalities for different courses of AD.Methods: In this study, we propose an extensible hierarchical graph convolutional network (EH-GCN) for early AD identification. Based on the extracted image features from multi-modal MRI data using the presented multi-branch residual network (ResNet), the brain regions-of-interests (ROIs) based GCN is built to extract structural and functional connectivity features between different ROIs of the brain. In order to further improve the performance of AD identification, an optimized spatial GCN is proposed as convolution operator in the population-based GCN to avoid rebuilding the graph network and take advantage of relationships between subjects. Finally, the proposed EH-GCN is built by embedding the image features and internal brain connectivity features into the spatial population-based GCN, which provides an extensible way to improve early AD identification performance by adding imaging features and non-image information from multi-modal data.Results: Experiments are performed on two datasets, which illustrate the effectiveness of the extracted structural/functional connectivity features and the high computational efficiency of the proposed method. The classification accuracy of AD vs NC, AD vs MCI and MCI vs NC classification tasks reaches 88.71%, 82.71% and 79.68% respectively. The extracted connectivity features between ROIs indicate that functional abnormalities are earlier than gray matter atrophy and abnormalities of structural connections, which is consistent with the clinical manifestations. The proposed method allows for the addition of other modal image features and non-image information from multi-modal data to continuously improve the performance of clinical data analysis.Conclusions: The proposed method can help us comprehensively analyze the role of gray matter atrophy, the damage of white matter nerve fiber tracts and the degradation of functional connectivity for different courses of AD, which could be useful for further extraction of clinical biomarkers for early AD identification.

Spatio-Temporal-Spectral Hierarchical Graph Convolutional Network With Semisupervised Active Learning for Patient-Specific Seizure Prediction.

Graph theory analysis using electroencephalogram (EEG) signals is currently an advanced technique for seizure prediction. Recent deep learning approaches, which fail to fully explore both the characterizations in EEGs themselves and correlations among different electrodes simultaneously, generally neglect the spatial or temporal dependencies in an epileptic brain and, thus, produce suboptimal seizure prediction performance consequently. To tackle this issue, in this article, a patient-specific EEG seizure predictor is proposed by using a novel spatio-temporal-spectral hierarchical graph convolutional network with an active preictal interval learning scheme (STS-HGCN-AL). Specifically, since the epileptic activities in different brain regions may be of different frequencies, the proposed STS-HGCN-AL framework first infers a hierarchical graph to concurrently characterize an epileptic cortex under different rhythms, whose temporal dependencies and spatial couplings are extracted by a spectral-temporal convolutional neural network and a variant self-gating mechanism, respectively. Critical intrarhythm spatiotemporal properties are then captured and integrated jointly and further mapped to the final recognition results by using a hierarchical graph convolutional network. Particularly, since the preictal transition may be diverse from seconds to hours prior to a seizure onset among different patients, our STS-HGCN-AL scheme estimates an optimal preictal interval patient dependently via a semisupervised active learning strategy, which further enhances the robustness of the proposed patient-specific EEG seizure predictor. Competitive experimental results validate the efficacy of the proposed method in extracting critical preictal biomarkers, indicating its promising abilities in automatic seizure prediction.

Predicting compound-protein interaction using hierarchical graph convolutional networks.

MotivationConvolutional neural networks have enabled unprecedented breakthroughs in a variety of computer vision tasks. They have also drawn much attention from other domains, including drug discovery and drug development. In this study, we develop a computational method based on convolutional neural networks to tackle a fundamental question in drug discovery and development, i.e. the prediction of compound-protein interactions based on compound structure and protein sequence. We propose a hierarchical graph convolutional network (HGCN) to encode small molecules. The HGCN aggregates a molecule embedding from substructure embeddings, which are synthesized from atom embeddings. As small molecules usually share substructures, computing a molecule embedding from those common substructures allows us to learn better generic models. We then combined the HGCN with a one-dimensional convolutional network to construct a complete model for predicting compound-protein interactions. Furthermore we apply an explanation technique, Grad-CAM, to visualize the contribution of each amino acid into the prediction.ResultsExperiments using different datasets show the improvement of our model compared to other GCN-based methods and a sequence based method, DeepDTA, in predicting compound-protein interactions. Each prediction made by the model is also explainable and can be used to identify critical residues mediating the interaction.

Hierarchical CADNet: Learning from B-Reps for Machining Feature Recognition

Deep learning approaches have been shown to be capable of recognizing shape features (e.g. machining features) in Computer-Aided Design (CAD) models in certain circumstances, yet still have issues when the features intersect, and in exploiting the geometric and topological information which comprises the boundary representation (B-Rep) of the typical CAD model. This paper presents a novel hierarchical B-Rep graph shape representation which encodes information about the surface geometry and face topology of the B-Rep. To learn from this new shape representation, a novel hierarchical graph convolutional network called Hierarchical CADNet has been created, which has been shown to outperform other state-of-the-art neural architectures on feature identification, including machining features that intersect, with improvements in accuracy for some more complex CAD models. • A novel representation and deep learning framework for learning from B-Rep CAD models. • A complex CAD model dataset with labeled machining features is proposed. • Improvements over current state-of-the-art deep learning frameworks for machining feature recognition.

PH-GCN: Person Retrieval With Part-Based Hierarchical Graph Convolutional Network

Compact feature representation of person image is important for person re-identification (Re-ID) task. Recently, part-based representation models have been widely studied for extracting the more compact and robust feature representation for person image to improve person Re-ID results. However, existing part-based representation models mostly extract the features of different parts independently which ignore the spatial relationship information among different parts. To address this issue, in this paper we propose a novel deep learning framework, named Part-based Hierarchical Graph Convolutional Network (PH-GCN) for person Re-ID problem. Given a person image, PH-GCN first constructs a hierarchical graph to represent the spatial relationships among different parts. Then, both local and global feature learning is achieved by the feature information passing in PH-GCN, which takes the information of other parts into account for part feature representation. Finally, a perceptron layer is adopted for the final person part label prediction and re-identification. The proposed framework provides a general solution that integrates <i>local</i>, <i>global</i> and <i>structural</i> feature learning simultaneously in a unified end-to-end network representation and learning. Extensive experiments on several widely used benchmark datasets demonstrate the effectiveness and benefits of the proposed PH-GCN approach for person Re-ID task.

Partially Supervised Compatibility Modeling.

Fashion Compatibility Modeling (FCM), which aims to automatically evaluate whether a given set of fashion items makes a compatible outfit, has attracted increasing research attention. Recent studies have demonstrated the benefits of conducting the item representation disentanglement towards FCM. Although these efforts have achieved prominent progress, they still perform unsatisfactorily, as they mainly investigate the visual content of fashion items, while overlooking the semantic attributes of items (e.g., color and pattern), which could largely boost the model performance and interpretability. To address this issue, we propose to comprehensively explore the visual content and attributes of fashion items towards FCM. This problem is non-trivial considering the following challenges: a) how to utilize the irregular attribute labels of items to partially supervise the attribute-level representation learning of fashion items; b) how to ensure the intact disentanglement of attribute-level representations; and c) how to effectively sew the multiple granulairites (i.e, coarse-grained item-level and fine-grained attribute-level) information to enable performance improvement and interpretability. To address these challenges, in this work, we present a partially supervised outfit compatibility modeling scheme (PS-OCM). In particular, we first devise a partially supervised attribute-level embedding learning component to disentangle the fine-grained attribute embeddings from the entire visual feature of each item. We then introduce a disentangled completeness regularizer to prevent the information loss during disentanglement. Thereafter, we design a hierarchical graph convolutional network, which seamlessly integrates the attribute- and item-level compatibility modeling, and enables the explainable compatibility reasoning. Extensive experiments on the real-world dataset demonstrate that our PS-OCM significantly outperforms the state-of-the-art baselines. We have released our source codes and well-trained models to benefit other researchers (https://site2750.wixsite.com/ps-ocm).

A Hierarchical Graph Convolution Network for Representation Learning of Gene Expression Data

The curse of dimensionality, which is caused by high-dimensionality and low-sample-size, is a major challenge in gene expression data analysis. However, the real situation is even worse: labelling data is laborious and time-consuming, so only a small part of the limited samples will be labelled. Having such few labelled samples further increases the difficulty of training deep learning models. Interpretability is an important requirement in biomedicine. Many existing deep learning methods are trying to provide interpretability, but rarely apply to gene expression data. Recent semi-supervised graph convolution network methods try to address these problems by smoothing the label information over a graph. However, to the best of our knowledge, these methods only utilize graphs in either the feature space or sample space, which restrict their performance. We propose a transductive semi-supervised representation learning method called a hierarchical graph convolution network (HiGCN) to aggregate the information of gene expression data in both feature and sample spaces. HiGCN first utilizes external knowledge to construct a feature graph and a similarity kernel to construct a sample graph. Then, two spatial-based GCNs are used to aggregate information on these graphs. To validate the model's performance, synthetic and real datasets are provided to lend empirical support. Compared with two recent models and three traditional models, HiGCN learns better representations of gene expression data, and these representations improve the performance of downstream tasks, especially when the model is trained on a few labelled samples. Important features can be extracted from our model to provide reliable interpretability.

Hi-GCN: A hierarchical graph convolution network for graph embedding learning of brain network and brain disorders prediction

PurposeRecently, brain connectivity networks have been used for the classification of neurological disorder, such as Autism Spectrum Disorders (ASD) or Alzheimer's disease (AD). Network analysis provides a new way for exploring the association between brain functional deficits and the underlying structural disruption related to brain disorders. Network embedding learning that aims to automatically learn low-dimensional representations for brain networks has drawn increasing attention in recent years. MethodIn this work we build upon graph neural network in order to learn useful representations for graph classification in an end-to-end fashion. Specifically, we propose a hierarchical GCN framework (called hi-GCN) to learn the graph feature embedding while considering the network topology information and subject's association at the same time. ResultsTo demonstrate the effectiveness of our approach, we evaluate the performance of the proposed method on the Alzheimer's Disease Neuroimaging Initiative (ADNI) dataset and Autism Brain Imaging Data Exchange (ABIDE) dataset. Extensive experiments on ABIDE and ADNI datasets have demonstrated competitive performance of the hi-GCN model. Specifically, we obtain an average accuracy of 73.1%/78.5% as well as AUC of 82.3%/86.5% on ABIDE/ADNI. The comprehensive experiments demonstrate that our hi-GCN is effective for graph classification with brain disorders diagnosis. ConclusionThe proposed hi-GCN method performs the graph embedding learning from a hierarchical perspective while considering the structure in individual brain network and the subject's correlation in the global population network, which can capture the most essential embedding features to improve the classification performance of disease diagnosis. Moreover, the proposed jointly optimizing strategy also achieves faster training and easier convergence than both the hi-GCN with pre-training and two-step supervision.

Enhanced Unsupervised Graph Embedding via Hierarchical Graph Convolution Network

Graph embedding aims to learn the low-dimensional representation of nodes in the network, which has been paid more and more attention in many graph-based tasks recently. Graph Convolution Network (GCN) is a typical deep semisupervised graph embedding model, which can acquire node representation from the complex network. However, GCN usually needs to use a lot of labeled data and additional expressive features in the graph embedding learning process, so the model cannot be effectively applied to undirected graphs with only network structure information. In this paper, we propose a novel unsupervised graph embedding method via hierarchical graph convolution network (HGCN). Firstly, HGCN builds the initial node embedding and pseudo-labels for the undirected graphs, and then further uses GCNs to learn the node embedding and update labels, finally combines HGCN output representation with the initial embedding to get the graph embedding. Furthermore, we improve the model to match the different undirected networks according to the number of network node label types. Comprehensive experiments demonstrate that our proposed HGCN and HGCN∗ can significantly enhance the performance of the node classification task.