Skeleton Data Research Articles

Enhancing skeleton-based human motion recognition with Lie algebra and memristor-augmented LSTM and CNN

<abstract><p>Lately, as a subset of human-centric studies, vision-oriented human action recognition has emerged as a pivotal research area, given its broad applicability in fields like healthcare, video surveillance, autonomous driving, sports, and education. This brief applies Lie algebra and standard bone length data to represent human skeleton data. A multi-layer long short-term memory (LSTM) recurrent neural network and convolutional neural network (CNN) are applied for human motion recognition. Finally, the trained network weights are converted into the crossbar-based memristor circuit, which can accelerate the network inference, reduce energy consumption, and obtain an excellent computing performance.</p></abstract>

Research on Artificial Intelligence-Assisted Teaching Methods in Chinese Classical Dance Movement Training

Abstract Taking movement training in the teaching of classical Chinese dance as an entry point, combined with the application of digital technology, improves and enriches the existing teaching methods of movement training of classical Chinese dance. Based on the Kinect skeleton data acquisition technology, the study pre-processes the collected data and proposes a dance movement data segmentation method based on movement rhythm to segment dance movement segments. The ST-GCN network structure is improved to recognize Chinese classical dance by AAST-GCN, a spatio-temporal graph convolutional network based on adaptive and attention mechanisms. Finally, the DTW algorithm is used to analyze the degree of similarity between the recognized action sequences and the standard action sequences, so as to provide targeted assisted teaching of classical Chinese dance. The results show that the intersection and concurrency ratio of the human dance posture detection method based on deep convolutional learning is always located in [0.5, 1.0], and the system can give training suggestions according to the comparison results, and the trainer can reasonably adjust the movements according to the software prompts, so as to achieve the purpose of assisted training, and then effectively improve the trainer’s dance level. After 12 weeks of AI-assisted training, the dancers obtained more obvious improvement than the control group in both rotational speed and the number of rotations, and the enhancement of these two abilities is precisely the necessary condition for realizing high-quality classical dance turn-like movements.

DMMG: Dual Min-Max Games for Self-Supervised Skeleton-Based Action Recognition.

In this work, we propose a new Dual Min-Max Games (DMMG) based self-supervised skeleton action recognition method by augmenting unlabeled data in a contrastive learning framework. Our DMMG consists of a viewpoint variation min-max game and an edge perturbation min-max game. These two min-max games adopt an adversarial paradigm to perform data augmentation on the skeleton sequences and graph-structured body joints, respectively. Our viewpoint variation min-max game focuses on constructing various hard contrastive pairs by generating skeleton sequences from various viewpoints. These hard contrastive pairs help our model learn representative action features, thus facilitating model transfer to downstream tasks. Moreover, our edge perturbation min-max game specializes in building diverse hard contrastive samples through perturbing connectivity strength among graph-based body joints. The connectivity-strength varying contrastive pairs enable the model to capture minimal sufficient information of different actions, such as representative gestures for an action while preventing the model from overfitting. By fully exploiting the proposed DMMG, we can generate sufficient challenging contrastive pairs and thus achieve discriminative action feature representations from unlabeled skeleton data in a self-supervised manner. Extensive experiments demonstrate that our method achieves superior results under various evaluation protocols on widely-used NTU-RGB+D, NTU120-RGB+D and PKU-MMD datasets.

Posture and stress detection system using open CV and media pipe

Posture and stress are two critical factors affecting a person's physical and mental well-being. Proper posture and stress management can help avoid a range of health issues. Poor posture may result in chronic discomfort, decreased mobility, and an increased risk of musculoskeletal problems. Similarly, stress can negatively impact physical and mental health, contributing to conditions such as depression, anxiety, and cardiovascular disease. Traditional methods for assessing posture and stress, including physical examinations or self-reporting, can be subjective and time-consuming. Recent advances in machine learning and computer vision techniques have enabled the development of models that automatically detect posture and stress levels from video data. In this study, a Django framework was built that incorporates models to assess posture by calculating angles between tracked distance vectors. Stress levels are evaluated based on facial features and expressions. The use of skeleton data for human posture recognition is a key research area in human-computer interaction. By employing the MSR 3D Action Dataset, 33 key skeletal points on the body are detected, aiding in posture determination. Additionally, analyzing facial features and emotions is essential for estimating stress levels. The approach relies on convolutional neural networks.

Lightweight deep learning framework for recognizing construction workers' activities based on simplified node combinations

Monitoring construction workers' activities is vital to effective construction project management. However, most existing studies on skeleton-based worker activity recognition use full-body skeleton data, which involve inconvenient movement and high computational demands. This research aimed to identify simplified skeleton node combinations at various scales and develop a framework that reduces computational demands without sacrificing accuracy for the combinations. To this end, this study selected five node combinations at different scales using five deep learning algorithms and developed a lightweight deep learning framework by reducing input features and sample frequencies and stacking temporal convolution network (TCN) blocks. The results demonstrate that this framework outperforms the original deep learning algorithm utilizing the entire skeleton by approximately 1.94%–6.75%. This research contributes to the field of automated construction workers' activity recognition by reducing inconvenient movements and computational demands. Further research needs to investigate the relationships between sensor locations and specific types of motions.

Action Recognition Based on Multi-Level Topological Channel Attention of Human Skeleton.

In action recognition, obtaining skeleton data from human poses is valuable. This process can help eliminate negative effects of environmental noise, including changes in background and lighting conditions. Although GCN can learn unique action features, it fails to fully utilize the prior knowledge of human body structure and the coordination relations between limbs. To address these issues, this paper proposes a Multi-level Topological Channel Attention Network algorithm: Firstly, the Multi-level Topology and Channel Attention Module incorporates prior knowledge of human body structure using a coarse-to-fine approach, effectively extracting action features. Secondly, the Coordination Module utilizes contralateral and ipsilateral coordinated movements in human kinematics. Lastly, the Multi-scale Global Spatio-temporal Attention Module captures spatiotemporal features of different granularities and incorporates a causal convolution block and masked temporal attention to prevent non-causal relationships. This method achieved accuracy rates of 91.9% (Xsub), 96.3% (Xview), 88.5% (Xsub), and 90.3% (Xset) on NTU-RGB+D 60 and NTU-RGB+D 120, respectively.

View-Normalized and Subject-Independent Skeleton Generation for Action Recognition

Skeleton-based action recognition has attracted great interest in computer vision. For this task, a challenging problem concerns the large intraclass variances of skeleton data, which are mainly caused by diverse viewpoints and subjects, and greatly increase the difficulty of modeling actions through a network. To address the above problem, we propose a variance reduction (VaRe) framework for skeleton-based action recognition, which consists of a view-normalization generative adversarial network (VN-GAN), a subject-independent network (SINet) and a classification network. First, the VN-GAN is responsible for reducing view-induced intraclass variances. Specifically, this network, comprising a generator and a discriminator, is aimed at learning a mapping from a diverse-view skeleton distribution to a unified-view skeleton distribution in an unsupervised manner, thereby generating a view-normalized skeleton. Second, taking the view-normalized skeleton as input, the SINet focuses on reducing the influences of the personal habits of subjects on action recognition. To generate SI skeleton data, the SINet automatically adjusts the human pose according to the human kinematic structure under a classification loss constraint. Finally, without the interference of view- and subject-induced variances, the classification network can concentrate more on learning discriminative action features to predict classes. Furthermore, by combining the joint and bone modalities, the proposed framework achieves competitive performance on three benchmarks: NTU RGB+D, NTU-120 RGB+D and Northwestern-UCLA Multiview Action 3D.

Motion classification using distributional features of 3D skeleton data

Motion classification using distributional features of 3D skeleton data

A Spatio‐Temporal Enhanced Graph‐Transformer AutoEncoder embedded pose for anomaly detection

AbstractDue to the robustness of skeleton data to human scale, illumination changes, dynamic camera views, and complex backgrounds, great progress has been made in skeleton‐based video anomaly detection in recent years. The spatio‐temporal graph convolutional network has been proven to be effective in modelling the spatio‐temporal dependencies of non‐Euclidean data such as human skeleton graphs, and the autoencoder based on this basic unit is widely used to model sequence features. However, due to the limitations of the convolution kernel, the model cannot capture the correlation between non‐adjacent joints, and it is difficult to deal with long‐term sequences, resulting in an insufficient understanding of behaviour. To address this issue, this paper applies the Transformer to the human skeleton and proposes the Spatio‐Temporal Enhanced Graph‐Transformer AutoEncoder (STEGT‐AE) to improve the capability of modelling. In addition, the multi‐memory model with skip connections is employed to provide different levels of coding features, thereby enhancing the ability of the model to distinguish similar heterogeneous behaviours. Furthermore, the STEGT‐AE has a single encoder‐double decoder architecture, which can improve the detection performance by the combining reconstruction and prediction error. The experimental results show that performances of STEGT‐AE is significantly better than other advanced algorithms on four baseline datasets.

TAEKWONDO POSE ESTIMATION WITH DEEP LEARNING ARCHITECTURES ON ONE-DIMENSIONAL AND TWO-DIMENSIONAL DATA

Practicing sports is an activity that helps people maintain and improve their health, enhance memory and concentration, reduce anxiety and stress, and train teamwork and leadership ability. With the development of science and technology, artificial intelligence in sports has become increasingly popular with the public and brings many benefits. In particular, many applications help people track and evaluate athletes' achievements in competitions. This study extracts images from Taekwondo videos and generates skeleton data from frames using the Fast Forward Moving Picture Experts Group (FFMPEG) technique using MoveNet. After that, we use deep learning architectures such as Long Short-Term Memory Networks, Convolutional Long Short-Term Memory, and Long-term Recurrent Convolutional Networks to perform the poses classification tasks in Taegeuk in Jang lessons. This work presents two approaches. The first approach uses a sequence skeleton extracted from the image by Movenet. Second, we use sequence images to train using video classification architecture. Finally, we recognize poses in sports lessons using skeleton data to remove noise in the image, such as background and extraneous objects behind the exerciser. As a result, our proposed method has achieved promising performance in pose classification tasks in an introductory Taekwondo lesson.

Extracting gait and balance pattern features from skeleton data to diagnose attention deficit/hyperactivity disorder in children

Extracting gait and balance pattern features from skeleton data to diagnose attention deficit/hyperactivity disorder in children

An adaptively multi-correlations aggregation network for skeleton-based motion recognition

Previous work based on Graph Convolutional Networks (GCNs) has shown promising performance in 3D skeleton-based motion recognition. We believe that the 3D skeleton-based motion recognition problem can be explained as a modeling task of dynamic skeleton-based graph construction. However, existing methods fail to model human poses with dynamic correlations between human joints, ignoring the information contained in the skeleton structure of the non-connected relationship during human motion modeling. In this paper, we propose an Adaptively Multi-correlations Aggregation Network(AMANet) to capture dynamic joint dependencies embedded in skeleton graphs, which includes three key modules: the Spatial Feature Extraction Module (SFEM), Temporal Feature Extraction Module (TFEM), and Spatio-Temporal Feature Extraction Module (STFEM). In addition, we deploy the relative coordinates of the joints of various parts of the human body via moving frames of Differential Geometry. On this basis, we design a Data Preprocessing Module (DP), enriching the characteristics of the original skeleton data. Extensive experiments are conducted on three public datasets(NTU-RGB+D 60, NTU-RGB+D 120, and Kinetics-Skeleton 400), demonstrating our proposed method’s effectiveness.

Volleyball Action Recognition based on Skeleton Data

This research explores the intricacies of volleyball action recognition using skeleton data through the lens of the Long Short-Term Memory (LSTM) model. With the objective of accurately identifying distinct volleyball actions—Serve, Spike, Block, Dig, and Set—the study implemented a structured LSTM network, achieving a commendable 95% accuracy rate consistently across all actions. The findings underscore the transformative potential of deep learning, particularly the LSTM network, in sports analytics, suggesting a paradigm shift in understanding and analyzing sports actions. The research serves as a foundation for future studies, offering insights into the blend of artificial intelligence in sports, with applications extending to coaching support and enhanced sports broadcasts.

Human-centric multimodal fusion network for robust action recognition

Skeleton-based methods have made remarkable strides in human action recognition (HAR). However, the performance of existing unimodal approaches is still limited by the lack of diverse visual features in skeleton data. Concretely, due to the absence of interaction information between individuals and objects, skeleton-based methods tend to confuse similar actions. Moreover, the view invariant property of unimodal models is susceptible to restrictions. In this work, we propose an innovative skeleton-guided multimodal data fusion methodology that transforms depth, RGB, and optical flow modalities into human-centric images (HCI) based on keypoint sequences. Building upon this foundation, we introduce a human-centric multimodal fusion network (HCMFN), which can comprehensively extract the action patterns of different modalities. Our model significantly enhances the performance of skeleton-based techniques, achieving remarkable results with rapid inference speed. Extensive experiments on two large-scale multimodal datasets, namely NTU RGB+D and NTU RGB+D 120, validate the capacity of HCMFN to bolster the robustness of skeleton-based methods in two challenging HAR tasks: (1) discriminating between actions with subtle inter-class differences, and (2) recognizing actions from varying viewpoints. Compared to state-of-the-art multimodal methods, our HCMFN achieves exciting results.

Action Recognition Based on GCN with Adjacency Matrix Generation Module and Time Domain Attention Mechanism

Different from other computer vision tasks, action recognition needs to process larger-scale video data. How to extract and analyze the effective parts from a huge amount of video information is the main difficulty of action recognition technology. In recent years, due to the outstanding performance of Graph Convolutional Networks (GCN) in many fields, a new solution to the action recognition algorithm has emerged. However, in current GCN models, the constant physical adjacency matrix makes it difficult to mine synergistic relationships between key points that are not directly connected in physical space. Additionally, a simple time connection of skeleton data from different frames makes each frame in the video contribute equally to the recognition results, which increases the difficulty of distinguishing action stages. In this paper, the information extraction ability of the model has been optimized in the space domain and time domain, respectively. In the space domain, an Adjacency Matrix Generation (AMG) module, which can pre-analyze node sets and generate an adaptive adjacency matrix, has been proposed. The adaptive adjacency matrix can help the graph convolution model to extract the synergistic information between the key points that are crucial for recognition. In the time domain, the Time Domain Attention (TDA) mechanism has been designed to calculate the time-domain weight vector through double pooling channels and complete the weights of key point sequences. Furthermore, performance of the improved TDA-AMG-GCN modules has been verified on the NTU-RGB+D dataset. Its detection accuracy at the CS and CV divisions reached 84.5% and 89.8%, respectively, with an average level higher than other commonly used detection methods at present.

A Comparative Study of Supervised and Unsupervised Approaches in Human Activity Analysis Based on Skeleton Data

A Comparative Study of Supervised and Unsupervised Approaches in Human Activity Analysis Based on Skeleton Data

Multi-view graph convolution network for the recognition of human action with spatial and temporal occlusion problems

Human action recognition holds great significance in the field of social security. However, the challenges posed by occlusion and changes in viewpoint create specific obstacles in achieving accurate recognition. In this study, we propose a multi-view graph convolution fusion method to effectively address this issue. Specifically, considering the limited availability of public human action datasets that include occlusion, we introduce an adaptive multi-view spatial–temporal occlusion generation method. It allows us to generate occlusion skeleton data from multiple viewpoints, ensuring a close resemblance to real-life scenarios with minimal modifications to existing public datasets. Additionally, we present a plug-and-play multi-view information fusion module, briefed as MGL, which aims to solve the occlusion problem. The MGL combines the capabilities of Graph Convolutional network (GCN) and Long Short-Term Memory network (LSTM), in which GCN is employed to reconstruct human skeleton information using multi-view spatial occlusion data, and LSTM captures the long-term dependencies within the temporal sequences of the multi-view data. Moreover, an attention mask mechanism is introduced to highlight key joint features. Experiment results illustrate the excellent performance of our method on the NTU RGB+D 60 and NTU RGB+D 120 datasets.

Reconstructed Prototype Network Combined with CDC-TAGCN for Few-Shot Action Recognition

Research on few-shot action recognition has received widespread attention recently. However, there are some blind spots in the current research: (1) The prevailing practice in many models is to assign uniform weights to all samples; nevertheless, such an approach may yield detrimental consequences for the model in the presence of high-noise samples. (2) Samples with similar features but different classes make it difficult for the model to be distinguished. (3) Skeleton data harbors rich temporal features, but most encoders face challenges in effectively extracting them. In response to these challenges, this study introduces a reconstructed prototype network (RC-PN) based on a prototype network framework and a novel spatiotemporal encoder. The RC-PN comprises two enhanced modules: Sample coefficient reconstruction (SCR) and a reconstruction loss function (LRC). SCR leverages cosine similarity between samples to reassign sample weights, thereby generating prototypes robust to noise interference and more adept at conveying conceptual essence. Simultaneously, the introduction of LRC enhances the feature similarity among samples of the same class while increasing feature distinctiveness between different classes. In the encoder aspect, this study introduces a novel spatiotemporal convolutional encoder called CDC-TAGCN. The temporal convolution operator is redefined in CDC-TAGCN. The vanilla temporal convolution operator can only capture the surface-level characteristics of action samples. Drawing inspiration from differential convolution (CDC), this research enhances TCN to CDC-TGCN. CDC-TGCN allows for the fusion of discrepant features from action samples into the features extracted by the vanilla convolutional operator. Extensive feasibility and ablation experiments are performed on the skeleton action dataset NTU-RGB + D 120 and Kinetics and compared with recent research.

Comprehensive receptive field adaptive graph convolutional networks for action recognition

Action recognition has shown a broad prospect for application in human–computer interaction and autonomous vehicles. According to skeleton data being lightweight and robust to environmental disturbance, skeleton-based methods are increasingly used in action recognition. Existing Graph Convolutional Networks (GCNs) have limitations such as Global Average Pooling (GAP) causing information oversmoothing, attention mechanism increasing computational complexity, and fixed spatial configuration partition strategy limiting flexibility. Our model overcomes these limitations by incorporating Temporal Covariance Pooling (TCP) for better feature utilization, Sample-Shared attention for weight sharing, and a Comprehensive Receptive Field strategy for optimal joint connections. Experiments show that the recognition accuracy of our model outperforms baseline 2s-AGCN by 1.1 % and 0.7 % under X-Sub and X-View for NTU RGB + D dataset, 1.8 % and 1.9 % in X-Sub and X-Set for NTU RGB + D 120 dataset, and 1.6 % on Kinetics-Skeleton dataset.

Skeleton-based action recognition based on multidimensional adaptive dynamic temporal graph convolutional network

Due to the superior capability to process the topology of graphs, graph convolutional networks are gaining popularity in the field of action recognition based on skeleton data. However, it remains difficult to effectively extract features with more distinguishing information for both spatial and temporal dimension. A novel multidimensional adaptive dynamic temporal graph convolutional network (MADT-GCN) model for skeleton-based action recognition is proposed in this work. It consists of two modules, one multidimensional adaptive graph convolutional network (MD-AGCN) module and one dynamic temporal convolutional network (DY-TCN) module. Firstly, MD-AGCN has the ability to adaptively change the graph topology in accordance with varieties of the layers and multidimensional information of spatial, temporal, and channel dimensions that are contained in various action samples to capture the complex connections of each couple of joints. Then, DY-TCN is proposed in order to boost the representation capability to capture expressive temporal features. Moreover, the information of both the joints and bones, together with their motion information, are simultaneously modeled in a multi-stream framework, which shows notable improvements in recognition accuracy. Finally, extensive experiments are conducted on two standard datasets, NTU-RGB+D and NTU-RGB+D 120. The experimental results demonstrate the effectiveness of the proposed method.