Skeleton Graph Research Articles

Action recognition algorithm based on skeleton graph with multiple features and improved adjacency matrix

AbstractAlthough graph convolutional networks have achieved good performances in skeleton‐graph‐based action recognition, there are still some problems which include the incomplete utilization of skeleton graph features and the lacking of logical adjacency information between nodes in adjacency matrix. In this article, a human action recognition algorithm is proposed based on multiple features from the skeleton graph to solve these problems. More specifically, an improved adjacency matrix is constructed to make full use of the multiple skeleton graph features. These features include local differential features, multi‐scale edge features, features of the original skeleton graph, nodal features, and nodal motion features. Extensive results are conducted on four standard datasets (NTU RGB‐D 60, NTU RGB‐D 120, Kinetics, and Northwestern‐UCLA). The experimental results show that the proposed algorithm outperforms the SOTA action recognition algorithms.

Learning Heterogeneous Spatial-Temporal Context for Skeleton-Based Action Recognition.

Graph convolution networks (GCNs) have been widely used and achieved fruitful progress in the skeleton-based action recognition task. In GCNs, node interaction modeling dominates the context aggregation and, therefore, is crucial for a graph-based convolution kernel to extract representative features. In this article, we introduce a closer look at a powerful graph convolution formulation to capture rich movement patterns from these skeleton-based graphs. Specifically, we propose a novel heterogeneous graph convolution (HetGCN) that can be considered as the middle ground between the extremes of (2 + 1)-D and 3-D graph convolution. The core observation of HetGCN is that multiple information flows are jointly intertwined in a 3-D convolution kernel, including spatial, temporal, and spatial-temporal cues. Since spatial and temporal information flows characterize different cues for action recognition, HetGCN first dynamically analyzes pairwise interactions between each node and its cross-space-time neighbors and then encourages heterogeneous context aggregation among them. Considering the HetGCN as a generic convolution formulation, we further develop it into two specific instantiations (i.e., intra-scale and inter-scale HetGCN) that significantly facilitate cross-space-time and cross-scale learning on skeleton graphs. By integrating these modules, we propose a strong human action recognition system that outperforms state-of-the-art methods with the accuracy of 93.1% on NTU-60 cross-subject (X-Sub) benchmark, 88.9% on NTU-120 X-Sub benchmark, and 38.4% on kinetics skeleton.

Learning the structure of multivariate regression chain graphs by testing complete separators in prime blocks

This paper introduces an algorithm to construct a bidirectional causal graph using an augmented graph. The algorithm decomposes the augmented graph, significantly reducing the size of the variable set required for conditional independence testing. Simultaneously, it preserves the fundamental structure of the augmented graph after decomposition, saving time and cost in constructing a global skeleton graph. Through experiments on discrete and continuous datasets, the algorithm demonstrates a clear advantage in runtime compared to traditional methods. In large-scale sparse networks, the training time is only about one-tenth of traditional methods. Additionally, the algorithm shows improvement in terms of construction error. Since the input to the algorithm is an augmented graph, this paper also discusses the impact on construction error when using both real and generated augmented graphs as input. Furthermore, the concept of markov blanket is extended to multivariate regression chain graphs, providing a method for rapidly constructing augmented graphs given certain prior knowledge.

GKE-TUNet: Geometry-Knowledge Embedded TransUNet Model for Retinal Vessel Segmentation Considering Anatomical Topology.

Automated retinal vessel segmentation is crucial for computer-aided clinical diagnosis and retinopathy screening. However, deep learning faces challenges in extracting complex intertwined structures and subtle small vessels from densely vascularized regions. To address these issues, we propose a novel segmentation model, called Geometry-Knowledge Embedded TransUNet (GKE-TUNet), which incorporates explicit embedding of topological features of retinal vessel anatomy. In the proposed GKE-TUNet model, a skeleton extraction network is pre-trained to extract the anatomical topology of retinal vessels from refined segmentation labels. During vessel segmentation, the dense skeleton graph is sampled as a graph of key-points and connections and is incorporated into the skip connection layer of TransUNet. The graph vertices are used as node features and correspond to positions in the low-level feature maps. The graph attention network (GAT) is used as the graph convolution backbone network to capture the shape semantics of vessels and the interaction of key locations along the topological direction. Finally, the node features obtained by graph convolution are read out as a sparse feature map based on their corresponding spatial coordinates. To address the problem of sparse feature maps, we employ convolution operators to fuse sparse feature maps with low-level dense feature maps. This fusion is weighted and connected to deep feature maps. Experimental results on the DRIVE, CHASE-DB1, and STARE datasets demonstrate the competitiveness of our proposed method compared to existing ones.

Auto-Correlation and Channel Attention Enhanced Deep Graph Convolution Networks for Gait Phase Prediction Based on Multi-IMU System

Gait phase prediction is important in controlling assistive robotic devices such as exoskeletons, where the control unit must differentiate between gait phases to provide the necessary assistance when the user is wearing the exoskeleton. To achieve the objective of precisely identifying the gait phase of users for the accurate control of the exoskeleton, this study proposes Auto-Correlation and Channel Attention enhanced Deep Graph Convolutional Networks (ACCA-DGCN) for gait phase prediction, and a gait phase prediction model based on multiple inertial measurement units (IMUs) and skeleton graph was established, in order to fully utilize the dependency among joints, and enhance accuracy and reliability of gait phase prediction. First, a human lower limb gait data acquisition equipment was developed, and the gait data of human walking were collected. The skeleton graph of the human lower limb was constructed through the natural connection relationship of joints in the human skeleton. After that, the ACCA-DGCN-based gait phase prediction model was constructed by using the gait data of human walking. Auto-Correlation (AC) and Efficient Channel Attention (ECA) were introduced to effectively capture periodic features of gait data and focus on the channels with high contributions to gait phase prediction. Finally, the effect of the window size on the performance of the ACCA-DGCN model was explored, and the proposed algorithm was compared with the other five deep learning algorithms: CNN, RNN, TCN, LSTM, and DGCN. The experimental results show that the average accuracy of gait phase prediction model based on ACCA-DGCN reaches up to 92.26% and 97.21% in user-independent and user-dependent experiments, respectively, which is superior to the other five algorithms. This study provides a new method for gait phase prediction, which is useful for improving the control of exoskeleton robots.

Hand-aware graph convolution network for skeleton-based sign language recognition

Skeleton-based sign language recognition (SLR) is a challenging research area mainly due to the fast and complex hand movement. Currently, graph convolution networks (GCNs) have been employed in skeleton-based SLR and achieved remarkable performance. However, existing GCN-based SLR methods suffer from a lack of explicit attention to hand topology which plays an important role in the sign language representation. To address this issue, we propose a novel hand-aware graph convolution network (HA-GCN) to focus on hand topological relationships of skeleton graph. Specifically, a hand-aware graph convolution layer is designed to capture both global body and local hand information, in which two sub-graphs are defined and incorporated to represent hand topology information. In addition, in order to eliminate the over-fitting problem, an adaptive DropGraph is designed in construction of hand-aware graph convolution block to remove the spatial and temporal redundancy in the sign language representation. With the aim to further improve the performance, the joints information, bones, together with their motion information are simultaneously modeled in a multi-stream framework. Extensive experiments on the two open-source datasets, AUTSL and INCLUDE, demonstrate that our proposed algorithm outperforms the state-of-the-art with a significant margin. Our code is available at https://github.com/snorlaxse/HA-SLR-GCN.

Skeleton-based coordinate system construction method for non-cooperative targets

Due to the lack of structure and part information in non-cooperative targets, the task of pose estimation suffers from unstable and unreliable initial coordinate system. In this paper, we introduce a curve-skeleton-based method to construct the robust coordinate system automatically for non-cooperative targets. First, a nonmanifold-Laplacian-based method is proposed to extract the curve skeleton from point clouds. The proposed method solves the over-contraction problem while preserves the topology and structure of skeleton well during point clouds contraction. Additionally, we create the skeleton graph to detect the skeleton structure and key points automatically. Then, the detected skeleton feature is used to construct target coordinate system, and improve the stability and robustness. Specifically, a real-scan dataset is constructed in this paper, and the proposed point cloud contraction method outperforms previous methods in structural representation and visualization. The mean rotation error of proposed coordinate system construct method performs 0.1249° and 1.2135° in 0%–30% noise ratio of distortions and outliers, and performs 7.04° and 4.21° in real-scan aircraft and satellite datasets. The results in complete and incomplete datasets verify the robustness and reliability of our skeleton-based coordinate system.

A Spatio‐Temporal Enhanced Graph‐Transformer AutoEncoder embedded pose for anomaly detection

AbstractDue to the robustness of skeleton data to human scale, illumination changes, dynamic camera views, and complex backgrounds, great progress has been made in skeleton‐based video anomaly detection in recent years. The spatio‐temporal graph convolutional network has been proven to be effective in modelling the spatio‐temporal dependencies of non‐Euclidean data such as human skeleton graphs, and the autoencoder based on this basic unit is widely used to model sequence features. However, due to the limitations of the convolution kernel, the model cannot capture the correlation between non‐adjacent joints, and it is difficult to deal with long‐term sequences, resulting in an insufficient understanding of behaviour. To address this issue, this paper applies the Transformer to the human skeleton and proposes the Spatio‐Temporal Enhanced Graph‐Transformer AutoEncoder (STEGT‐AE) to improve the capability of modelling. In addition, the multi‐memory model with skip connections is employed to provide different levels of coding features, thereby enhancing the ability of the model to distinguish similar heterogeneous behaviours. Furthermore, the STEGT‐AE has a single encoder‐double decoder architecture, which can improve the detection performance by the combining reconstruction and prediction error. The experimental results show that performances of STEGT‐AE is significantly better than other advanced algorithms on four baseline datasets.

An adaptively multi-correlations aggregation network for skeleton-based motion recognition

Previous work based on Graph Convolutional Networks (GCNs) has shown promising performance in 3D skeleton-based motion recognition. We believe that the 3D skeleton-based motion recognition problem can be explained as a modeling task of dynamic skeleton-based graph construction. However, existing methods fail to model human poses with dynamic correlations between human joints, ignoring the information contained in the skeleton structure of the non-connected relationship during human motion modeling. In this paper, we propose an Adaptively Multi-correlations Aggregation Network(AMANet) to capture dynamic joint dependencies embedded in skeleton graphs, which includes three key modules: the Spatial Feature Extraction Module (SFEM), Temporal Feature Extraction Module (TFEM), and Spatio-Temporal Feature Extraction Module (STFEM). In addition, we deploy the relative coordinates of the joints of various parts of the human body via moving frames of Differential Geometry. On this basis, we design a Data Preprocessing Module (DP), enriching the characteristics of the original skeleton data. Extensive experiments are conducted on three public datasets(NTU-RGB+D 60, NTU-RGB+D 120, and Kinetics-Skeleton 400), demonstrating our proposed method’s effectiveness.

A robust approach to 3D neuron shape representation for quantification and classification

We consider the problem of finding an accurate representation of neuron shapes, extracting sub-cellular features, and classifying neurons based on neuron shapes. In neuroscience research, the skeleton representation is often used as a compact and abstract representation of neuron shapes. However, existing methods are limited to getting and analyzing “curve” skeletons which can only be applied for tubular shapes. This paper presents a 3D neuron morphology analysis method for more general and complex neuron shapes. First, we introduce the concept of skeleton mesh to represent general neuron shapes and propose a novel method for computing mesh representations from 3D surface point clouds. A skeleton graph is then obtained from skeleton mesh and is used to extract sub-cellular features. Finally, an unsupervised learning method is used to embed the skeleton graph for neuron classification. Extensive experiment results are provided and demonstrate the robustness of our method to analyze neuron morphology.

Angle based hand gesture recognition using graph convolutional network

AbstractHand gesture recognition has attracted huge interest in the areas of autonomous driving, human computer systems, gaming and many others. Skeleton based techniques along with graph convolutional networks (GCNs) are being popularly used in this field due to the easy estimation of joint coordinates and better representation capability of graphs. Simple hand skeleton graphs are unable to capture the finer details and complex spatial features of hand gestures. To address these challenges, this work proposes an “angle‐based hand gesture graph convolutional network” (AHG‐GCN). This model introduces two additional types of novel edges in the graph to connect the wrist with each fingertip and finger's base, explicitly capturing their relationship, which plays an important role in differentiating gestures. Besides, novel features for each skeleton joint are designed using the angles formed with fingertip/finger‐base joints and the distance among them to extract semantic correlation and tackle the overfitting problem. Thus, an enhanced set of 25 features for each joint is obtained using these novel techniques. The proposed model achieves 90% and 88% accuracy for 14 and 28 gesture configurations for the DHG 14/28 dataset and, 94.05% and 89.4% accuracy for 14 and 28 gesture configurations for the SHREC 2017 dataset, respectively.

Multi-Stream General and Graph-Based Deep Neural Networks for Skeleton-Based Sign Language Recognition

Sign language recognition (SLR) aims to bridge speech-impaired and general communities by recognizing signs from given videos. However, due to the complex background, light illumination, and subject structures in videos, researchers still face challenges in developing effective SLR systems. Many researchers have recently sought to develop skeleton-based sign language recognition systems to overcome the subject and background variation in hand gesture sign videos. However, skeleton-based SLR is still under exploration, mainly due to a lack of information and hand key point annotations. More recently, researchers have included body and face information along with hand gesture information for SLR; however, the obtained performance accuracy and generalizability properties remain unsatisfactory. In this paper, we propose a multi-stream graph-based deep neural network (SL-GDN) for a skeleton-based SLR system in order to overcome the above-mentioned problems. The main purpose of the proposed SL-GDN approach is to improve the generalizability and performance accuracy of the SLR system while maintaining a low computational cost based on the human body pose in the form of 2D landmark locations. We first construct a skeleton graph based on 27 whole-body key points selected among 67 key points to address the high computational cost problem. Then, we utilize the multi-stream SL-GDN to extract features from the whole-body skeleton graph considering four streams. Finally, we concatenate the four different features and apply a classification module to refine the features and recognize corresponding sign classes. Our data-driven graph construction method increases the system’s flexibility and brings high generalizability, allowing it to adapt to varied data. We use two large-scale benchmark SLR data sets to evaluate the proposed model: The Turkish Sign Language data set (AUTSL) and Chinese Sign Language (CSL). The reported performance accuracy results demonstrate the outstanding ability of the proposed model, and we believe that it will be considered a great innovation in the SLR domain.

Deep Learning for Human Activity Recognition on 3D Human Skeleton: Survey and Comparative Study.

Human activity recognition (HAR) is an important research problem in computer vision. This problem is widely applied to building applications in human-machine interactions, monitoring, etc. Especially, HAR based on the human skeleton creates intuitive applications. Therefore, determining the current results of these studies is very important in selecting solutions and developing commercial products. In this paper, we perform a full survey on using deep learning to recognize human activity based on three-dimensional (3D) human skeleton data as input. Our research is based on four types of deep learning networks for activity recognition based on extracted feature vectors: Recurrent Neural Network (RNN) using extracted activity sequence features; Convolutional Neural Network (CNN) uses feature vectors extracted based on the projection of the skeleton into the image space; Graph Convolution Network (GCN) uses features extracted from the skeleton graph and the temporal-spatial function of the skeleton; Hybrid Deep Neural Network (Hybrid-DNN) uses many other types of features in combination. Our survey research is fully implemented from models, databases, metrics, and results from 2019 to March 2023, and they are presented in ascending order of time. In particular, we also carried out a comparative study on HAR based on a 3D human skeleton on the KLHA3D 102 and KLYOGA3D datasets. At the same time, we performed analysis and discussed the obtained results when applying CNN-based, GCN-based, and Hybrid-DNN-based deep learning networks.

Abnormal sitting posture recognition based on multi-scale spatiotemporal features of skeleton graph

To fuse the spatial dimension, temporal dimension, and whole-body skeleton features of human abnormal sitting posture to improve the recognition accuracy, we propose an abnormal sitting posture recognition method based on the multi-scale spatiotemporal features of skeleton graph (ASPR) from the perspective of changes of human posture. Firstly, we build a human abnormal sitting posture dataset (HASP) with multidimensional features. Then, based on the structure of high-to-low resolution subnetworks, we use the spatiotemporal graph convolution module as the feature extraction unit A multiple scale spatiotemporal feature extraction model based on graph convolutional network (M2SGCN) is proposed. It is used to capture the spatiotemporal features. Next, a feature extraction model of local skeletal angles based on recurrent neural network is proposed to capture the change rule of skeletal angles of human sitting posture. Finally, we investigate the optimal parameters of ASPR by checking its performance with different human skeleton combination schemes and different features fusion coefficient. Experiment results show that ASPR achieves excellent performance compared with four state-of-the-art models and their combined models. ASPR shows an average recognition accuracy, a recall rate, an F1 score, and average time of 95.24%, 95.61%, 95.14%, and 7.094 ms, respectively.

Skeleton-based Human Action Recognition via Large-kernel Attention Graph Convolutional Network.

The skeleton-based human action recognition has broad application prospects in the field of virtual reality, as skeleton data is more resistant to data noise such as background interference and camera angle changes. Notably, recent works treat the human skeleton as a non-grid representation, e.g., skeleton graph, then learns the spatio-temporal pattern via graph convolution operators. Still, the stacked graph convolution plays a marginal role in modeling long-range dependences that may contain crucial action semantic cues. In this work, we introduce a skeleton large kernel attention operator (SLKA), which can enlarge the receptive field and improve channel adaptability without increasing too much computational burden. Then a spatiotemporal SLKA module (ST-SLKA) is integrated, which can aggregate long-range spatial features and learn long-distance temporal correlations. Further, we have designed a novel skeleton-based action recognition network architecture called the spatiotemporal large-kernel attention graph convolution network (LKA-GCN). In addition, large-movement frames may carry significant action information. This work proposes a joint movement modeling strategy (JMM) to focus on valuable temporal interactions. Ultimately, on the NTU-RGBD 60, NTU-RGBD 120 and Kinetics-Skeleton 400 action datasets, the performance of our LKA-GCN has achieved a state-of-the-art level.

Part-Wise Adaptive Topology Graph Convolutional Network for Skeleton-Based Action Recognition

Human action recognition is a computer vision challenge that involves identifying and classifying human movements and activities. The behavior of humans comprises movements of multiple body parts, and Graph Convolutional Networks (GCNs) have emerged as a promising approach for this task. However, most contemporary GCN methods perform graph convolution on the entire skeleton graph without considering that the human body consists of distinct body parts. To address these shortcomings, we propose a novel method that optimizes the representation of the skeleton graph by designing temporal and spatial convolutional blocks while introducing the Part-wise Adaptive Topology Graph Convolution (PAT-GC) technique. PAT-GC adaptively learns the segmentation of different body parts and dynamically integrates the spatial relevance between them. Furthermore, we utilize hierarchical modeling to divide the skeleton graph, capturing a more comprehensive representation of the human body. We evaluate our approach on three publicly available large datasets: NTU RGB + D 60, NTU RGB + D 120, and Kinetics Skeleton 400. Our experimental results demonstrate that our approach achieves state-of-the-art performance, thus validating the efficiency of our proposed technique for human action recognition.

Motion-Driven Spatial and Temporal Adaptive High-Resolution Graph Convolutional Networks for Skeleton-Based Action Recognition

Graph convolutional networks (GCN) have attracted increasing interest in action recognition in recent years. GCN models human skeleton sequences as spatio-temporal graphs. Also, attention mechanisms are often jointly used with GCNs to highlight important frames or body joints in a sequence. However, attention modules learn parameters offline and are fixed, so may not adapt well to unseen samples. In this paper, we propose a simple but effective motion-driven spatial and temporal adaptation strategy to dynamically strengthen the features of important frames and joints for skeleton-based action recognition. The rationale is that the joints and frames with dramatic motions are generally more informative and discriminative. We combine the spatial and temporal refinements by using a two-branch structure, in which the joint and frame-wise feature refinements perform in parallel. Such a structure can lead to learn more complementary feature representations. Moreover, we propose to use the fully connected graph convolution to learn the long-range spatial dependencies. Besides, we investigate two high-resolution skeleton graphs by creating virtual joints, aiming to improve the representation of skeleton features. By combining the above proposals, we develop a novel motion-driven spatial and temporal adaptive high-resolution GCN. Experimental results demonstrate that the proposed model achieves state-of-the-art (SOTA) results on the challenging large-scale Kinetics-Skeleton and UAV-Human datasets, and it is on par with the SOTA methods on the two NTU-RGB+D 60&120 datasets. Additionally, our motion-driven adaptation method shows encouraging performance when compared with the attention mechanisms.

JointContrast: Skeleton-Based Interaction Recognition with New Representation and Contrastive Learning

Skeleton-based action recognition depends on skeleton sequences to detect categories of human actions. In skeleton-based action recognition, the recognition of action scenes with more than one subject is named as interaction recognition. Different from the single-subject action recognition methods, interaction recognition requires an explicit representation of the interaction information between subjects. Recalling the success of skeletal graph representation and graph convolution in modeling the spatial structural information of skeletal data, we consider whether we can embed the inter-subject interaction information into the skeletal graph and use graph convolution for a unified feature representation. In this paper, we propose the interaction information embedding skeleton graph representation (IE-Graph) and use the graph convolution operation to represent the intra-subject spatial structure information and inter-subject interaction information in a uniform manner. Inspired by recent pre-training methods in 2D vision, we propose unsupervised pre-training methods for skeletal data as well as contrast loss. In SBU datasets, JointContrast achieves 98.2% recognition accuracy. in NTU60 datasets, JointContrast respectively achieves 94.1% and 96.8% recognition accuracy under Cross-Subject and Cross-View evaluation metrics.

GHOSM: Graph-based Hybrid Outline and Skeleton Modelling for Shape Recognition

An efficient and accurate shape detection model plays a major role in many research areas. With the emergence of more complex shapes in real-life applications, shape recognition models need to capture the structure with more effective features to achieve high accuracy rates for shape recognition. This article presents a new method for 2D/3D shape recognition based on graph spectral domain handcrafted features, which are formulated by exploiting both an outline and a skeleton shape through the global outline and internal details. A fully connected graph is generated over the shape outline to capture the global outline representation while a hierarchically clustered graph with adaptive connectivity is formed on the skeleton to capture the structural descriptions of the shape. We demonstrate the ability of the Fiedler vector to provide the graph partitioning of the skeleton graph. The performance evaluation demonstrates the efficiency of the proposed method compared to state-of-the-art studies with increments of 4.09%, 2.2%, and 14.02% for 2D static hand gestures, 2D shapes, and 3D shapes, respectively.

Representing Spatial Data with Graph Contrastive Learning

Large-scale geospatial data pave the way for geospatial machine learning algorithms, and a good representation is related to whether the machine learning model is effective. Hence, it is a critical task to learn effective feature representation for geospatial data. In this paper, we construct a spatial graph from the locations and propose a geospatial graph contrastive learning method to learn the location representations. Firstly, we propose a skeleton graph in order to preserve the primary structure of the geospatial graph to solve the positioning bias problem of remote sensing. Then, we define a novel mixed node centrality measure and propose four data augmentation methods based on the measure. Finally, we propose a heterogeneous graph attention network to aggregate information from both the structural neighborhood and semantic neighborhood separately. Extensive experiments on both geospatial datasets and non-geospatial datasets are conducted to illustrate that the proposed method outperforms state-of-the-art baselines.