Skeleton Information Research Articles

Human Activity Recognition System Based on Continuous Learning with Human Skeleton Information

Human Activity Recognition System Based on Continuous Learning with Human Skeleton Information

Domain Transfer-Based Hypergraph Convolutional Network for Posture Anomaly Detection in Physical Education Teaching

Human skeleton-based posture anomaly detection has been widely applied in the field of physical education teaching. The existing spatio-temporal graph convolutional networks (ST-GCN) can fully utilize the local and global information of the human skeleton for action recognition, but the entire model requires a large amount of computation and the modeling of high-order relationships between joint points of the human skeleton is insufficient. To this end, this paper proposes a novel domain adaptive hypergraph convolutional network for basketball posture anomaly analysis by exploiting 2D skeleton information. First, we designed an effective hypergraph convolution feature extraction network to improve the high-order dependency modeling. To further improve the performance of the hypergraph convolutional network, we introduce domain adaptive learning technology to supervise the feature extraction learning of the target domain (2D skeleton) through the source domain (3D skeleton). At last, we construct a basketball action teaching analysis dataset for model evaluation. We conducted a large number of comparative experiments on the public dataset NTU RGB+D and our self-built dataset. All the results showed that our proposed hypergraph convolutional model effectively extracts features of 2D human skeletons, and by introducing domain adaptive learning, the performance of basketball anomaly detection is further improved.

An efficient feature reuse distillation network for lightweight image super-resolution

In recent research, single-image super-resolution (SISR) using deep Convolutional Neural Networks (CNN) has seen significant advancements. While previous methods excelled at learning complex mappings between low-resolution (LR) and high-resolution (HR) images, they often required substantial computational and memory resources. We propose the Efficient Feature Reuse Distillation Network (EFRDN) to alleviate these challenges. EFRDN primarily comprises Asymmetric Convolutional Distillation Modules (ACDM), incorporating the Multiple Self-Calibrating Convolution (MSCC) units for spatial and channel feature extraction. It includes an Asymmetric Convolution Residual Block (ACRB) to enhance the skeleton information of the square convolution kernel and a Feature Fusion Lattice Block (FFLB) to convert low-order input signals into higher-order representations. Introducing a Transformer module for global features, we enhance feature reuse and gradient flow, improving model performance and efficiency. Extensive experimental results demonstrate that EFRDN outperforms existing methods in performance while conserving computing and memory resources.

Skeleton-guided 3D convolutional neural network for tubular structure segmentation.

Accurate segmentation of tubular structures is crucial for clinical diagnosis and treatment but is challenging due to their complex branching structures and volume imbalance. The purpose of this study is to propose a 3D deep learning network that incorporates skeleton information to enhance segmentation accuracy in these tubular structures. Our approach employs a 3D convolutional network to extract 3D tubular structures from medical images such as CT volumetric images. We introduce a skeleton-guided module that operates on extracted features to capture and preserve the skeleton information in the segmentation results. Additionally, to effectively train our deep model in leveraging skeleton information, we propose a sigmoid-adaptive Tversky loss function which is specifically designed for skeleton segmentation. We conducted experiments on two distinct 3D medical image datasets. The first dataset consisted of 90 cases of chest CT volumetric images, while the second dataset comprised 35 cases of abdominal CT volumetric images. Comparative analysis with previous segmentation approaches demonstrated the superior performance of our method. For the airway segmentation task, our method achieved an average tree length rate of 93.0%, a branch detection rate of 91.5%, and a precision rate of 90.0%. In the case of abdominal artery segmentation, our method attained an average precision rate of 97.7%, a recall rate of 91.7%, and an F-measure of 94.6%. We present a skeleton-guided 3D convolutional network to segment tubular structures from 3D medical images. Our skeleton-guided 3D convolutional network could effectively segment small tubular structures, outperforming previous methods.

Towards Smartphone-based 3D Hand Pose Reconstruction Using Acoustic Signals

Accurately reconstructing 3D hand poses is a pivotal element for numerous Human-Computer Interaction applications. In this work, we propose SonicHand, the first Smartphone-based 3D Hand Pose Reconstruction system using purely inaudible acoustic signals. SonicHand incorporates signal processing techniques and a deep learning framework to address a series of challenges. Firstly, it encodes the topological information of the hand skeleton as prior knowledge and utilizes a deep learning model to realistically and smoothly reconstruct the hand poses. Secondly, the system employs adversarial training to enhance the generalization ability of our system to be deployed in a new environment or for a new user. Thirdly, we adopt a hand tracking method based on channel impulse response (CIR) estimation. It enables our system to handle the scenario where the hand performs gestures while moving arbitrarily as a whole. We conduct extensive experiments on a smartphone testbed to demonstrate the effectiveness and robustness of our system from various dimensions. The experiments involve 10 subjects performing up to 12 different hand gestures in 3 distinctive environments. When the phone is held in one of the user’s hand, the proposed system can track joints with an average error of 18.64 mm.

D-CyPre: a machine learning-based tool for accurate prediction of human CYP450 enzyme metabolic sites

The advancement of graph neural networks (GNNs) has made it possible to accurately predict metabolic sites. Despite the combination of GNNs with XGBOOST showing impressive performance, this technology has not yet been applied in the realm of metabolic site prediction. Previous metabolic site prediction tools focused on bonds and atoms, regardless of the overall molecular skeleton. This study introduces a novel tool, named D-CyPre, that amalgamates atom, bond, and molecular skeleton information via two directed message-passing neural networks (D-MPNN) to predict the metabolic sites of the nine cytochrome P450 enzymes using XGBOOST. In D-CyPre Precision Mode, the model produces fewer, but more accurate results (Jaccard score: 0.497, F1: 0.660, and precision: 0.737 in the test set). In D-CyPre Recall Mode, the model produces less accurate, but more comprehensive results (Jaccard score: 0.506, F1: 0.669, and recall: 0.720 in the test set). In the test set of 68 reactants, D-CyPre outperformed BioTransformer on all isoenzymes and CyProduct on most isoenzymes (5/9). For the subtypes where D-CyPre outperformed CyProducts, the Jaccard score and F1 scores increased by 24% and 16% in Precision Mode (4/9) and 19% and 12% in Recall Mode (5/9), respectively, relative to the second-best CyProduct. Overall, D-CyPre provides more accurate prediction results for human CYP450 enzyme metabolic sites.

Exploring High-Order Skeleton Correlations with Physical and Non-Physical Connection for Action Recognition

Hypergraphs have received widespread attention in modeling complex data correlations due to their superior performance. In recent years, some researchers have used hypergraph structures to characterize complex non-pairwise joints in the human skeleton and model higher-order correlations of the human skeleton. However, traditional methods of constructing hypergraphs based on physical connections ignore the dependencies among non-physically connected joints or bones, and it is difficult to model the correlation among joints or bones that are highly correlated in human action but are physically connected at long distances. To address these issues, we propose a skeleton-based action recognition method for hypergraph learning based on skeleton correlation, which explores the effects of physically and non-physically connected skeleton information on accurate action recognition. Specifically, in this paper, spatio-temporal correlation modeling is performed on the natural connections inherent in humans (physical connections) and the joints or bones that are more dependent but not directly connected (non-physical connection) during human actions. In order to better learn the hypergraph structure, we construct a spatio-temporal hypergraph neural network to extract the higher-order correlations of the human skeleton. In addition, we use an attentional mechanism to compute the attentional weights among different hypergraph features, and adaptively fuse the rich feature information in different hypergraphs. Extensive experiments are conducted on two datasets, NTU-RGB+D 60 and Kinetics-Skeleton, and the results show that compared with the state-of-the-art skeleton-based methods, our proposed method can achieve an optimal level of performance with significant advantages, providing a more accurate environmental perception and action analysis for the development of embodied intelligence.

Characterization of rutting damage based on two-dimensional image analysis of changes in mesoscopic aggregate properties of asphalt mixtures

Accurately predicting the rutting development law of asphalt pavement is of great significance for preventing and controlling pavement diseases. The aggregate skeleton of the asphalt mixture is the main load-bearing part and plays an important role in its anti-rutting performance. However, quantitative and mesoscopic evaluation indicators and standards for aggregate skeletons have not been established. This study aims to reveal the relationship between the mesostructured characteristics of asphalt mixtures and rutting damage. This paper uses two-dimensional image technology to analyze the dynamic change process of internal skeleton information of two graded asphalt mixtures under cyclic wheel load. According to the fractal theory, the fractal dimension and multifractal spectrum of the full-depth aggregate were calculated. The results indicate that the linear regression model between the mesoscopic skeleton information of asphalt mixture and the average rutting depth during the rutting test can be used to quantify the rutting performance. There is an obvious correlation between the fractal dimension of the section aggregate and the loading times, and there is a certain linear relationship between the multifractal spectrum indexes and the loading times. It indicates that the fractal theory index of aggregate can reflect the change of the macro performance of loading times.

A Systematic Study on the Application of Computer Big Data Analysis to Improve the Accuracy of Referee Decision in Sports Events

Due to the complex working environment, long communication distance and high reliability requirements of referees in weightlifting events, this project uses TI's MSP430F5299 as its core processing chip. RS485 for long-distance communication. This system realizes a variety of judgments, arbitration supervision, system function setting, LED large screen display, PC data real-time transmission and other functions. Then, Fourier pyramid filtering method is used to extract human skeleton information, and SVM is used to classify human behavior to complete the construction of human behavior recognition model. The findings from the study indicate that the architectural approach detailed within this manuscript yields a higher degree of precision in identification, thereby offering judges an enhanced instrument for assessment.

Separate first, then segment: An integrity segmentation network for salient object detection

Current methods aggregate multi-level features or introduce auxiliary information to get more refined saliency maps. However, little attention is paid to how to obtain complete salient objects in cluttered background. To address this problem, we propose an integrity segmentation network (ISNet) with a novel detection paradigm that first separates the targets completely and then segment them finely. Specifically, the ISNet architecture consists of a target separation (TS) branch and an object segmentation (OS) branch, trained using a hierarchical difference-aware (HDA) loss. The TS branch equipped with a fractal structure is utilized to produce saliency features with expanded boundary (SF w/ EB), which can enlarge the difference of border details to separate the target from background completely. Compared with the edge and skeleton information, the SF w/ EB contains a more complete structure, which can supplement the defect of salient objects. The OS branch is leveraged to generate complementary features, which gradually integrates the SF w/ EB and aggregated features to segment complete saliency maps. Moreover, we propose the HDA loss to further improve the structural integrity of prediction, which hierarchically assigns weight to pixels according to their differences. Hard pixels will be given more attention to discriminate the similar parts between foreground and background. Comprehensive experimental results on five datasets show that the proposed ISNet outperforms the state-of-the-art methods both quantitatively and qualitatively. Concretely, compared with three typical models, the average gain percentage reaches 2.6% in terms of Fβ, Sm and MAE on two large complex datasets. The improvements demonstrate that the proposed ISNet are beneficial for improving the integrity of prediction. Besides, the ISNet is efficient and runs at a real-time speed of 39.5 FPS when processing an image with size of 320 × 320. Furthermore, the proposed model has better generalization, which can also be applied to other vision tasks to handle complex scenes. Codes are available at https://github.com/lesonly/ISNet.

Gestures recognition based on multimodal fusion by using 3D CNNs

Gestures have long been recognized as an interaction technique that can provide a more natural, creative, and intuitive way to communicate with computers. However, some existing difficulties include the high probability that the same type of movement done at different speeds will be recognized as a different category of movement; cluttered, occluded, and low-resolution backgrounds; and the near-impossibility of fusing different types of features. To this end, we propose a novel framework for integrating different scales of RGB and motion skeletons to obtain higher recognition accuracy using multiple features. Specifically, we provide a network architecture that combines a three-dimensional convolutional neural network (3DCNN) and post-fusion to better embed different features. Also, we combine RGB and motion skeleton information at different scales to mitigate speed and background issues. Experiments on several gesture recognition public datasets show desirable results, validating the superiority of the proposed gesture recognition method. Finally, we do a human-computer interaction experiment to prove its practicality.

Detecting video anomalies by jointly utilizing appearance and skeleton information

A Video anomaly detection detects abnormal content that does not appear in the training set, and unsupervised methods such as generative methods are applied because the training set only offers normal content. Recently, skeleton-based features have been leveraged to alleviate the background distractions meanwhile outstanding human motion. However, we regard the appearance and skeleton information as complementary, and propose a generative method consisting of an appearance branch implemented by a 3D U-Net and a skeleton branch implemented by a novel Skeleton-Transformer. Moreover, a Multi-head Co-attention-based fusion module is proposed to fuse the intermediate features extracted from the appearance and skeleton branches, and then transfer the fusion information to each branch. This fusion module addresses the challenge of maintaining the feature structure of each branch during the fusion process, which is essential in the generative model. Experimental results show that the fusion module improves the performances of the appearance branch and skeleton branch, and their combination achieves state-of-the-art performance on HR-ShanghaiTech and Corridor datasets.

Beyond Pattern Variance: Unsupervised 3-D Action Representation Learning With Point Cloud Sequence.

This work pays the first research effort to address unsupervised 3-D action representation learning with point cloud sequence, which is different from existing unsupervised methods that rely on 3-D skeleton information. Our proposition is built on the state-of-the-art 3-D action descriptor 3-D dynamic voxel (3DV) with contrastive learning (CL). The 3DV can compress the point cloud sequence into a compact point cloud of 3-D motion information. Spatiotemporal data augmentations are conducted on it to drive CL. However, we find that existing CL methods (e.g., SimCLR or MoCo v2) often suffer from high pattern variance toward the augmented 3DV samples from the same action instance, that is, the augmented 3DV samples are still of high feature complementarity after CL, while the complementary discriminative clues within them have not been well exploited yet. To address this, a feature augmentation adapted CL (FACL) approach is proposed, which facilitates 3-D action representation via concerning the features from all augmented 3DV samples jointly, in spirit of feature augmentation. FACL runs in a global-local way: one branch learns global feature that involves the discriminative clues from the raw and augmented 3DV samples, and the other focuses on enhancing the discriminative power of local feature learned from each augmented 3DV sample. The global and local features are fused to characterize 3-D action jointly via concatenation. To fit FACL, a series of spatiotemporal data augmentation approaches is also studied on 3DV. Wide-range experiments verify the superiority of our unsupervised learning method for 3-D action feature learning. It outperforms the state-of-the-art skeleton-based counterparts by 6.4% and 3.6% with the cross-setup and cross-subject test settings on NTU RGB + D 120, respectively. The source code is available at https://github.com/tangent-T/FACL.

Disentangling Modality and Posture Factors: Memory-Attention and Orthogonal Decomposition for Visible-Infrared Person Re-Identification.

Striving to match the person identities between visible (VIS) and near-infrared (NIR) images, VIS-NIR reidentification (Re-ID) has attracted increasing attention due to its wide applications in low-light scenes. However, owing to the modality and pose discrepancies exhibited in heterogeneous images, the extracted representations inevitably comprise various modality and posture factors, impacting the matching of cross-modality person identity. To solve the problem, we propose a disentangling modality and posture factors (DMPFs) model to disentangle modality and posture factors by fusing the information of features memory and pedestrian skeleton. Specifically, the DMPF comprises three modules: three-stream features extraction network (TFENet), modality factor disentanglement (MFD), and posture factor disentanglement (PFD). First, aiming to provide memory and skeleton information for modality and posture factors disentanglement, the TFENet is designed as a three-stream network to extract VIS-NIR image features and skeleton features. Second, to eliminate modality discrepancy across different batches, we maintain memory queues of previous batch features through the momentum updating mechanism and propose MFD to integrate features in the whole training set by memory-attention layers. These layers explore intramodality and intermodality relationships between features from the current batch and memory queues under the optimization of the optimal transport (OT) method, which encourages the heterogeneous features with the same identity to present higher similarity. Third, to decouple the posture factors from representations, we introduce the PFD module to learn posture-unrelated features with the assistance of the skeleton features. Besides, we perform subspace orthogonal decomposition on both image and skeleton features to separate the posture-related and identity-related information. The posture-related features are adopted to disentangle the posture factors from representations by a designed posture-features consistency (PfC) loss, while the identity-related features are concatenated to obtain more discriminative identity representations. The effectiveness of DMPF is validated through comprehensive experiments on two VIS-NIR pedestrian Re-ID datasets.

Research on Artificial Intelligence Data Based on Image Texture Characterization and Aerobics Teaching Mode in Colleges and Universities

Abstract In this paper, we first studied the action recognition method based on image texture features, established a human body model, and realized the extraction of skeletal data and joint calibration. Based on the skeleton information, the aerobic movements are recognized, the difference between the aerobics movement sequence and the standard movement sequence is calculated, and the movements are scored. Then, the database for aerobic movements was established, and an AI-assistant aerobics teaching model was developed. Finally, the effect of the teaching method was analyzed through indicators of recognition effect, skill impact, and students’ learning interests. The results show that the accuracy of the system’s action recognition reaches 95%, and the overall response time is between 5.2-6.4s, with high real-time performance. And it has a significant effect on students’ movement participation, active interest, and independent learning behavior, with p-values of 0.013, 0.041, and 0.036, respectively, p<0.05. This study promotes the development and innovation of aerobics, which can scientifically adjust training countermeasures and enhance the skill level of aerobics athletes.

An Efficient Graph Convolution Network for Skeleton-Based Dynamic Hand Gesture Recognition

Dynamic hand gesture recognition has evolved as a prominent topic of computer vision research due to its vast applications in human-computer interaction, robotics, and other domains. Although there are numerous related recognition studies, the State-Of-The-Art (SOTA) methods are overparametrized. Specifically, the number of model parameters is quite large, which results in high computational costs. This work, referring to Songs ResGCN, designs an efficient and lightweight graph convolutional network, named ResGCNeXt. ResGCNeXt learns rich features from skeleton information and achieves high accuracy with less number of model parameters. First, three data preprocessing strategies according to motion analysis are designed to provide sufficient features for the recognition model. Then, an efficient graph convolutional network structure combining bottleneck and group convolution is designed to reduce the number of model parameters without loss of accuracy. Furthermore, an attention block called SENet-Part attention (SEPA) is added to improve channel and spatial feature learning. This study is validated on two benchmark datasets, and the experimental results show that ResGCNeXt provides competitive performance, especially in significantly reducing the number of model parameters. Compared to HAN-2S, which is one of the best SOTA methods, our method has half model parameters and a 0.3% higher recognition rate.

Edge and Skeleton Guidance Network for Salient Object Detection in Optical Remote Sensing Images

The salient object detection of optical remote sensing images (ORSI-SOD) is an important research direction in ORSI processing, which has achieved promising results in the last few years. Many recent works heavily rely on feature learning of regions for the improvement of the detection accuracy, while neglecting the concrete role of edge and skeleton information in the calculation. In this work, we propose a two-stage edge and skeleton guidance network (ESGNet) for ORSI-SOD in a coarse-to-fine way, and further demonstrate that the fused features of edge and skeleton are essential for ORSI-SOD. In the first stage, we construct the spatial graph attention (SGA) module for saliency features to generate an initial saliency map, and apply the spatial self-optimization (SSO) to enhance edge and skeleton features. The multi-level interactive fusion (MIF) module is used for the adequate integration of edge and skeleton features into saliency features. In the second stage, with the aim to accomplish better prediction of salient object localization and shape, the feature enhancement integration operation is introduced to recover object details from the learned edge and skeleton features. Extensive experiments on three public ORSI-SOD datasets demonstrate that our ESGNet achieves competitive performance with the state-of-the-art methods and also confirms the importance of edge and skeleton information for ORSI-SOD. Meanwhile, generalizability experiments on natural image datasets show that our method is competent for many types of SOD tasks. The code and results of our method are available at https://github.com/aoao0206/ESGNet.

A multimedia art interaction model for motion recognition based on data-driven model

Ergonomics is a relatively important part of user experience in multimedia art design. The study aims to improve human–computer interaction efficiency using a data-driven neural network model combined with video data and wearable devices to achieve high-precision human motion recognition. Firstly, the human motion skeleton information is extracted based on video information through the OpenPose framework, and the motion characteristics of each joint of the human body are calculated. Then, use the inertia data of the wearable bracelet to calculate the exercise intensity. At last, the two kinds of information are sent to recurrent neural networks (RNN) together to achieve high-precision human motion recognition. The innovative contribution of the article is to establish a multimodal fusion model for human activity recognition. The experimental results show that the recognition precision of the proposed method reaches 97.85%, much higher than the backpropagation neural network (BPNN) and K-nearest neighbor (KNN), whose precision is 94.35% and 90.12%, respectively. The method’s superior performance convinces us that the model can provide strong technical support for the interaction design of multimedia art.

Identifying Unsafe Behavior of Construction Workers: A Dynamic Approach Combining Skeleton Information and Spatiotemporal Features

Identifying Unsafe Behavior of Construction Workers: A Dynamic Approach Combining Skeleton Information and Spatiotemporal Features

Multi-view graph convolution network for the recognition of human action with spatial and temporal occlusion problems

Human action recognition holds great significance in the field of social security. However, the challenges posed by occlusion and changes in viewpoint create specific obstacles in achieving accurate recognition. In this study, we propose a multi-view graph convolution fusion method to effectively address this issue. Specifically, considering the limited availability of public human action datasets that include occlusion, we introduce an adaptive multi-view spatial–temporal occlusion generation method. It allows us to generate occlusion skeleton data from multiple viewpoints, ensuring a close resemblance to real-life scenarios with minimal modifications to existing public datasets. Additionally, we present a plug-and-play multi-view information fusion module, briefed as MGL, which aims to solve the occlusion problem. The MGL combines the capabilities of Graph Convolutional network (GCN) and Long Short-Term Memory network (LSTM), in which GCN is employed to reconstruct human skeleton information using multi-view spatial occlusion data, and LSTM captures the long-term dependencies within the temporal sequences of the multi-view data. Moreover, an attention mask mechanism is introduced to highlight key joint features. Experiment results illustrate the excellent performance of our method on the NTU RGB+D 60 and NTU RGB+D 120 datasets.