Graph CNN Research Articles

Random memristor-based dynamic graph CNN for efficient point cloud learning at the edge

The broad integration of 3D sensors into devices like smartphones and AR/VR headsets has led to a surge in 3D data, with point clouds becoming a mainstream representation method. Efficient real-time learning of point cloud data on edge devices is crucial for applications such as autonomous vehicles and embodied AI. Traditional machine learning models on digital processors face limitations, with software challenges like high training complexity, and hardware challenges such as large time and energy overheads due to von Neumann bottleneck. To address this, we propose a software-hardware co-designed random memristor-based dynamic graph CNN (RDGCNN). Software-wise, we transform point cloud into graph, and propose random EdgeConv for efficient hierarchical and topological features extraction. Hardware-wise, leveraging memristor’s intrinsic stochasticity and in-memory computing capability, we achieve significant reductions in training complexity and energy consumption. RDGCNN demonstrates high accuracy and efficiency across various point cloud tasks, paving the way for future edge 3D vision.

Dynamic graph CNN based semantic segmentation of concrete defects and as-inspected modeling

Obtaining accurate information of defective areas of infrastructures helps to perform repair actions more efficiently. Recently, LiDAR scanners have been used for the inspection of surface defects. Moreover, machine learning methods have attracted the attention of researchers for semantic segmentation and classification based on point cloud data. Although much work has been done for processing visual information with images, research on machine learning methods for semantic segmentation of raw point cloud data is still in its early stages. Moreover, LiDAR technology is commonly used to create as-is BIM models. Therefore, the BIM model needs to be integrated with the results of defect semantic segmentation after the LiDAR-based inspection. Addressing the above issues, this paper has the following objectives: (1) Developing a method for point cloud-based concrete surface defects semantic segmentation; and (2) Developing a semi-automated process for as-inspected modeling. The challenges related to the size of the dataset and imbalanced classes are studied. Sensitivity analysis is applied to capture the best combination of hyperparameters and investigate their effects on the network performance. The proposed method resulted in 98.56% and 96.50% recalls for semantic segmentation of cracks and spalls, respectively. Furthermore, post-processing of the results of the concrete surface defects semantic segmentation is done to semi-automate the process of as-inspected modeling. As-inspected BIM includes the updated information of the facilities at the time of data collection. This semi-automated process made it possible to manage and visualize the detected defects by extracting their dimensions and identifying the conditions on the 3D model.

Multi-branch Fusion: A Multi-branch Attention Framework by Combining Graph Convolutional Network and CNN for Hyperspectral Image Classification

Multi-branch Fusion: A Multi-branch Attention Framework by Combining Graph Convolutional Network and CNN for Hyperspectral Image Classification

Detection of Fittings Based on the Dynamic Graph CNN and U-Net Embedded with Bi-Level Routing Attention

Accurate detection of power fittings is crucial for identifying defects or faults in these components, which is essential for assessing the safety and stability of the power system. However, the accuracy of fittings detection is affected by a complex background, small target sizes, and overlapping fittings in the images. To address these challenges, a fittings detection method based on the dynamic graph convolutional neural network (DGCNN) and U-shaped network (U-Net) is proposed, which combines three-dimensional detection with two-dimensional object detection. Firstly, the bi-level routing attention mechanism is incorporated into the lightweight U-Net network to enhance feature extraction for detecting the fittings boundary. Secondly, pseudo-point cloud data are synthesized by transforming the depth map generated by the Lite-Mono algorithm and its corresponding RGB fittings image. The DGCNN algorithm is then employed to extract obscured fittings features, contributing to the final refinement of the results. This process helps alleviate the issue of occlusions among targets and further enhances the precision of fittings detection. Finally, the proposed method is evaluated using a custom dataset of fittings, and comparative studies are conducted. The experimental results illustrate the promising potential of the proposed approach in enhancing features and extracting information from fittings images.

SUPER-RESOLUTION OF DIFFUSION MRI VIA JOINT CONSIDERATION OF SPATIAL AND ANGULAR SPACES USING GRAPH CNN

Diffusion magnetic resonance imaging (dMRI) is an essential technique for studying brain white matter. However, its application suffers from the tradeoff between resolution, signal-to-noise ratio (SNR), and scanning time. Under a given setting, post-acquisition super-resolution (SR) has displayed the exciting performance of increasing the spatial or angular resolution of dMRI without prolonging the acquisition time. dMRI is acquired from a heterogeneous space composed of the spatial domain (x-space) and the diffusion wave vector domain (q-space). We introduced a graph convolution neural network (GCNN) to fit the mapping from low-resolution (LR) dMRI to high-resolution (HR) counterpart by jointly considering x-space and q-space to make an effective learning in this non-Euclidean space to increase the spatial resolution. Real brain dMRI data were downloaded from the Human Connectome Project (HCP). Comparisons were made between different spaces, including x-space, q-space, and xq-space. The results indicated that the best performance was obtained when two spaces were considered jointly using GCNN. Compared with the traditional methods, our network fully used spatio-angular information to improve the quality of dMRI SR. Furthermore, some feature metrics, including fractional anisotropic mapping, fiber orientation distribution (FOD), and fiber tractography, were presented to support the improvement. The cross-phantom test on synthetic data generated by the Phantomas tool was performed to further evaluate our model. Extended experiments revealed that GCNN might serve as a new way of enhancing dMRI.

Orthognathic surgical planning using graph CNN with dual embedding module: External validations with multi-hospital datasets

Background and objectiveDespite recent development of AI, prediction of the surgical movement in the maxilla and mandible by OGS might be more difficult than that of tooth movement by orthodontic treatment. To evaluate the prediction accuracy of the surgical movement using pairs of pre-(T0) and post-surgical (T1) lateral cephalograms (lat-ceph) of orthognathic surgery (OGS) patients and dual embedding module-graph convolution neural network (DEM-GCNN) model. Methods599 pairs from 3 institutions were used as training, internal validation, and internal test sets and 201 pairs from other 6 institutions were used as external test set. DEM-GCNN model (IEM, learning the lat-ceph images; LTEM, learning the landmarks) was developed to predict the amount and direction of surgical movement of ANS and PNS in the maxilla and B-point and Md1crown in the mandible. The distance between T1 landmark coordinates actually moved by OGS (ground truth) and predicted by DEM-GCNN model and pre-existed CNN-based Model-C (learning the lat-ceph images) was compared. ResultsIn both internal and external tests, DEM-GCNN did not exhibit significant difference from ground truth in all landmarks (ANS, PNS, B-point, Md1crown, all P > 0.05). When the accumulated successful detection rate for each landmark was compared, DEM-GCNN showed higher values than Model-C in both the internal and external tests. In violin plots exhibiting the error distribution of the prediction results, both internal and external tests showed that DEM-GCNN had significant performance improvement in PNS, ANS, B-point, Md1crown than Model-C. DEM-GCNN showed significantly lower prediction error values than Model-C (one-jaw surgery, B-point, Md1crown, all P < 0.005; two-jaw surgery, PNS, ANS, all P < 0.05; B point, Md1crown, all P < 0.005). ConclusionWe developed a robust OGS planning model with maximized generalizability despite diverse qualities of lat-cephs from 9 institutions.

Fusion of an RGB camera and LiDAR sensor through a Graph CNN for 3D object detection

Despite the recent development of RGB camera and LiDAR sensor fusion technology using deep learning, fusion without loss of information is a very difficult problem because the structural data characteristics of the two sensors are different. To solve this problem, we use a graph convolutional neural network (Graph CNN) to fuse RGB and LiDAR sensors. The proposed method creates a fusion feature by supplementing the geometric information of each feature in the process of fusing the features of two different sensors. Based on the experimental data, the proposed method has higher accuracy in detecting distant objects and complex situations than the existing method.

Modeling of Whale Optimization with Deep Learning based Brain Disorder Detection and Classification

Brain disorders are a significant source of economic strain and unfathomable suffering in modern society. Imaging techniques help diagnose, monitor and treat mental health, neurological, and developmental disorders. To aid in the Computer-Aided Diagnosis of brain diseases, deep learning (DL) was used for the analysis of neuroimages from modalities including Positron Emission Tomography (PET), Structural Magnetic Resonance Imaging (SMRI), and functional MRI. In this study, a Whale Optimization Algorithm is used with Deep Learning to analyse MRI scans for signs of neurological disease. WOADL-BDDC may detect and label abnormalities in the brain based on an MRI scan. It uses a two-step pre-processing procedure, first using guided filtering to get rid of background noise and then using U-Net segmentation to get rid of the top of the head. QuickNAT, along with RMSProp, is used to segment the brain. When analysing data, WOADL-BDDC uses radionics to collect information from every layer. When used in a convolutional recurrent neural network model, the Whale Optimization Algorithm may accurately categorize mental illness. WOADL-BDDC is put through its paces using ADNI 3D. Compared to state-of-the-art classification results from Vgg16, Graph CNN, Modified ResNet18, Non-linear SVM, ResNet50-SVM, ResNet50-RF, the suggested technique achieved the greatest accuracy. It was demonstrated that the suggested model is superior to other models for classification from MRI images. In simulations, the proposed approach is shown to be effective in optimizing hyperparameters with an accuracy of 94.38 % on TR set and 94.87% on TS set, Precision of 96.43% on TR set and 97.62% on TS set, and an F1-Score of 89.35 % and 92.10% on TR and TS set, respectively.

Mesh-Based DGCNN: Semantic Segmentation of Textured 3-D Urban Scenes

Textured 3D mesh is one of the final user products in photogrammetry and remote sensing. However, research on the semantic segmentation of complex urban scenes represented by textured 3D meshes is in its infancy. We present a mesh-based dynamic graph CNN (DGCNN) for the semantic segmentation of textured 3D meshes. To represent each mesh facet, composite input feature vectors are constructed by concatenating the face-inherent features, i.e., XYZ coordinates of the center of gravity (CoG), texture values, and normal vectors. A texture fusion module is embedded into the proposed mesh-based DGCNN to generate high-level semantic features of the high-resolution texture information, which is useful for semantic segmentation. We achieve competitive accuracies when the proposed method is applied to the SUM mesh datasets. The overall accuracy (OA), Kappa coefficient (Kap), mean precision (mP), mean recall (mR), mean F1 score (mF1), and mean intersection over union (mIoU) are 93.3%, 88.7%, 79.6%, 83.0%, 80.7%, and 69.6%, respectively. In particular, the OA, mean class accuracy (mAcc), mIoU, and mF1 increase by 0.3%, 12.4%, 3.4%, and 6.9%, respectively, compared to the state-of-the-art method.

Towards an Advanced Deep Learning for the Internet of Behaviors: Application to Connected Vehicles

In recent years, intensive research has been conducted to enable people to live more comfortably. Developments in the Internet of Things (IoT) , big data, and artificial intelligence have taken this type of research to a new level and led to the emergence of the Internet of Behaviors (IoB) , which analyzes behavioral patterns. However, current IoB technologies are not capable of handling heterogeneous data. While it is quite common to have different formats of sensor data for the same behavioral observation, the use of these different data formats can significantly help to obtain a more accurate classification of the observation. Another limitation is that existing IoB deep learning models rely on inefficient hyperparameter tuning strategies. In this paper, we present an Advanced Deep Learning framework for IoB (ADLIoB) applied to connected vehicles. Several deep learning architectures are employed in this framework: CNN, Graph CNN (GCNN), and LSTM are used to train sensor data of different formats. In addition, a branch-and-bound technique is used to intelligently select hyperparameters. To validate ADLIoB, experiments were conducted on four databases for connected vehicles. The results clearly show that ADLIoB is superior to the baseline solutions in terms of both accuracy and runtime.

Image Classification for Rice varieties using Deep Learning Models

Rice is the most rapidly growing crop in India, and as the population grows, demand for rice also increases. The majority of Asian countries grow rice and export it worldwide. The various rice varieties have been cultivated depending on the people's food culture. At the same time, food quality is a top priority, so we use computer vision techniques to extract rice qualitative features. The products are analyzed using image processing techniques for physical attributes such as Visual Geometry Graph (VGG16) and Vanilla CNN (also known as vanilla neural networks) to identify traits and textual features of rice grain images. VGG16 consists o f CNN architecture with 16 layers. It can train millions of datasets and achieve the highest accuracy rate. In addition, another model, namely vanilla neural networks, is an extension of the linear regression model. However, vanilla CNN has an additional hidden layer between inputs and outputs that help with extra computations. Image processing techniques are combined with neural networks to provide more accuracy in the training model rather than the manual process. Jasmine, Basmati, Arborio, Ipsala, and Karacadag rice varieties are the five types of rice image datasetsused for classification. Each of these varieties has 15000 images, for a total of 75000 images used in the training and testing process. The best image classifier is chosen based on the best accuracy score. The proposed model's outcome deliberates the better performance in determining the rice varieties. Keywords: Convolutional Neural Network, Vanilla Neural Network, VGG16, Rice varieties

VMV-GCN: Volumetric Multi-View Based Graph CNN for Event Stream Classification

Event cameras can perceive pixel-level brightness changes to output asynchronous event streams, and have notable advantages in high temporal resolution, high dynamic range and low power consumption for challenging vision tasks. To apply existing learning models on event data, many researchers integrate sparse events into dense frame-based representations which can work with convolutional neural networks directly. Although these works achieve high performance on event-based classification, their models need lots of parameters to process dense event frames which do not fit with the sparsity of event data. To utilize the sparse nature of events, we propose a voxel-wise graph learning model ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">VMV-GCN</i> ) for spatio-temporal feature learning on event streams. Specifically, we design the volumetric multi-view fusion module ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">VMVF</i> ) to extract spatial and temporal information from views of voxelized event data. Then we take representative event voxels as vertices and use a novel dual-graph construction strategy to connect them. By aggregating neighborhood information based on relationships of vertices, the proposed dynamic neighborhood feature learning module ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">DNFL</i> ) can capture discriminative spatio-temporal features on dynamically updated graphs. Experiments show that our method achieves state-of-the-art performance with low model complexity on event-based classification tasks, such as object classification and action recognition.

BubblEX: An Explainable Deep Learning Framework for Point-Cloud Classification

Point clouds data are nowadays one of the major data sources for describing our environment. Recently, deep architectures have been proposed as a key step in understanding and retrieving semantic information. Despite the great contribution of deep learning in this field, the explainability of these models for 3D data is still fairly unexplored. Explainability, identified as a potential weakness of Deep Neural Networks (DNNs), can help researchers against skepticism, considering that these models are far from being self-explanatory. Although literature provides many examples on the exploitation of Explainable Artificial Intelligence (XAI) approaches with 2D data, only a few studies have attempted to investigate it for 3D DNNs. To overcome these limitations, it is here proposed BubblEX, a novel multimodal fusion framework to learn the 3D point features. BubblEX framework comprises two stages: &#x201C;Visualization Module&#x201D; for the visualisation of features learned from the network in its hidden layers and &#x201C;Interpretability Module&#x201D;, which aims at describing how the neighbour points are involved in the features extraction. For our experiments, it has been used Dynamic Graph CNN (DGCNN), trained on Modelnet40 dataset. The developed framework extends a method for obtaining saliency maps from image data, to deal with 3D point cloud data, allowing the analysis, comparison and contrasting of multiple features. Besides, it permits the generation of visual explanations from any DNN-based network for 3D point cloud classification without requiring architectural changes or re-training. Our findings will be extremely useful for both scientists and non-experts in understanding and improving future AI-based models.

Multimodal Ensemble Deep Learning to Predict Disruptive Behavior Disorders in Children

Oppositional defiant disorder and conduct disorder, collectively referred to as disruptive behavior disorders (DBDs), are prevalent psychiatric disorders in children. Early diagnosis of DBDs is crucial because they can increase the risks of other mental health and substance use disorders without appropriate psychosocial interventions and treatment. However, diagnosing DBDs is challenging as they are often comorbid with other disorders, such as attention-deficit/hyperactivity disorder, anxiety, and depression. In this study, a multimodal ensemble three-dimensional convolutional neural network (3D CNN) deep learning model was used to classify children with DBDs and typically developing children. The study participants included 419 females and 681 males, aged 108–131 months who were enrolled in the Adolescent Brain Cognitive Development Study. Children were grouped based on the presence of DBDs (n = 550) and typically developing (n = 550); assessments were based on the scores from the Child Behavior Checklist and on the Schedule for Affective Disorders and Schizophrenia for School-age Children-Present and Lifetime version for DSM-5. The diffusion, structural, and resting-state functional magnetic resonance imaging (rs-fMRI) data were used as input data to the 3D CNN. The model achieved 72% accuracy in classifying children with DBDs with 70% sensitivity, 72% specificity, and an F1-score of 70. In addition, the discriminative power of the classifier was investigated by identifying the cortical and subcortical regions primarily involved in the prediction of DBDs using a gradient-weighted class activation mapping method. The classification results were compared with those obtained using the three neuroimaging modalities individually, and a connectome-based graph CNN and a multi-scale recurrent neural network using only the rs-fMRI data.

Towards point cloud completion: Point Rank Sampling and Cross-Cascade Graph CNN

The Point Fractal Network (PF-Net) is a seminal work with capability of completing the missing regions of point clouds. However, the multi-resolution structure of PF-Net neglects effective feature fusion between each resolution, which makes the resulting shape lack local geometric details. To tackle this problem, we design a novel shape completion network named PRSCN. We first present Point Rank Sampling to rate and sample feature points more objectively through local outline form. In this way, the sampled points can facilitate the downstream tasks. Subsequently, considering the correlation between features from different scales, we design a Cross-Cascade Module to combine features hierarchically. Moreover, we propose Leap-type EdgeConv to enlarge the receptive field while maintain the kernel size unchanged. These improvements together make our CD error 3% lower than that of state-of-the-art method on ShapeNet-part dataset.

A Multi-view Image Sets Classification Based on Graph Convolutional Neural Network

Multi-view images represent the target object from various perspectives. We propose a new spectral graph convolutional neural network (BSGCN) for multi-view image sets, which uses spectral graph convolution technology to process multi-view image sets. With the batch normalization techniques, the BSGCN speeds up the model convergence and improves the generalization capabilities of the model. We analyze the performance of two variants of BSGCN and its model by experiments on multi-view datasets. Experimental results show that the BSGCN is 2.84% and 7.26% higher than the classic graph CNN [17] on Modelnet10 and Modelnet40 multi-view datasets, respectively, and 0.34% higher than the classical CNN [6] on Modelnet40 multi-view dataset.

Point cloud classification by dynamic graph CNN with adaptive feature fusion

The deep neural network has made the most advanced breakthrough in almost all 2D image tasks, so we consider the application of deep learning in 3D images. Point cloud data, as the most basic and important form of representation of 3D images, can accurately and intuitively show the real world. The authors propose a new network based on feature fusion to improve the point cloud classification and segmentation tasks. Our network mainly consists of three parts: global feature extractor, local feature extractor and adaptive feature fusion module. A multi-scale transformation network is devised to guarantee the invariance of the transformation of the global feature, and a residual block is introduced to alleviate the problem of gradient disappearance to enhance the global feature extractor. Based on the edge convolution and multi-layer perceptron, a local feature extractor is constructed. Finally, an adaptive feature-fusion module is proposed to complete the fusion of global features and local features. Extensive experiments on point cloud classification and segmentation tasks are carried out to verify the effectiveness of the proposed method. The classification accuracy of the ModelNet40 is 93.6%, which is 4.4% higher than that of the PointNet. Similarly, the segmentation accuracy on the ShapeNet is 85.6%, which is higher than other methods.

Fast vertex-based graph convolutional neural network and its application to brain images

This paper proposes a vertex-based graph convolutional neural network (vertex-CNN) for analyzing structured data on graphs. We represent graphs using semi-regular triangulated meshes in which each vertex has 6 connected neighbors. We generalize classical CNN defined on equi-spaced grids to that defined on semi-regular triangulated meshes by introducing main building blocks of the CNN, including convolution, down-sampling, and pooling, on a vertex domain. By exploiting the regularity of semi-regular meshes in terms of vertex connections, the proposed vertex-CNN keeps the inherent properties of classical CNN in a Euclidean space, such as shift-invariance and down-sampling at a rate of 2, 4, etc. We employ brain images from Alzheimer’s Disease Neuroimaging Initiative (ADNI) (n = 6767) and extract cortical features (e.g., cortical thickness, surface area, curvature, Jacobian, sulcal depth, and volume) for the classification of healthy controls (CON), patients with mild cognitive impairment (MCI) and Alzheimer’s disease (AD). Based on cortical thickness, we show that the proposed vertex-CNN is near 3 times faster and performs significantly better in the classification performance of CON, MCI, and AD than an existing graph CNN defined on the graph spectral domain given in Defferrard (2016). Moreover, we examine the robustness of a multi-channel implementation of vertex-CNN on 6 cortical measures for the MCI and AD classification. Finally, we show a promising finding of the prediction accuracy from MCI to AD as a function of years before the onset of AD. Our experiments demonstrate the fast computation and promising classification performance of the vertex-CNN.

Learning Dense Correspondences for Non-Rigid Point Clouds With Two-Stage Regression.

We propose a novel deep learning method to predict dense correspondences for partial point clouds of non-rigidly deformable targets. Dense correspondences are learned in the form of vertex displacements of a template mesh towards the point clouds. A two-stage regression framework is proposed to estimate accurate displacement vectors, including the global and local regression networks. Specifically, the global regression network estimates global displacements from the global features of the template mesh and point clouds through a graph CNN based hierarchical encoder-decoder network. Based on the initial displacements, a mesh can be generated that fits to the point clouds roughly. In the local regression network, a local feature embedding layer fuses local features of point clouds with graph features on the generated mesh through an attention mechanism. Consequently, the embedded local features are employed to refine the correspondences in local regions of the targets by predicting the increments of vertex displacements. Our method is further generalized to correspondence estimation on unseen real data with a robust fine-tuning method. The experimental results on diverse datasets of various deformable subjects (e.g., human bodies, animals, and hands) demonstrate that the proposed approach can accurately and robustly estimate dense correspondences from non-rigid point clouds.

Attentive Spatial Temporal Graph CNN for Land Cover Mapping From Multi Temporal Remote Sensing Data

Satellite image time series (SITS) collected by modern Earth Observation (EO) systems represent a valuable source of information that supports several tasks related to the monitoring of the Earth surface dynamics over large areas. A main challenge is then to design methods able to leverage the complementarity between the temporal dynamics and the spatial patterns that characterize these data structures. Focusing on land cover classification (or mapping) tasks, the majority of approaches dealing with SITS data only considers the temporal dimension, while the integration of the spatial context is frequently neglected. In this work, we propose an attentive spatial temporal graph convolutional neural network that exploits both spatial and temporal dimensions in SITS. Despite the fact that this neural network model is well suited to deal with spatio-temporal information, this is the first work that considers it for the analysis of SITS data. Experiments are conducted on two study areas characterized by different land cover landscapes and real-world operational constraints (i.e., limited labeled data due to acquisition costs). The results show that our model consistently outperforms all the competing methods obtaining a performance gain, in terms of F-Measure, of at least 5 points with respect to the best competing approaches on both benchmarks.