Skeleton Information Research Articles

Pose recognition in sports scenes based on deep learning skeleton sequence model

Human skeleton extraction is a basic problem in the field of computer vision. With the rapid progress of science and technology, it has become a hot issue in the field of target detection such as pedestrian recognition, behavior monitoring, and pedestrian gesture recognition. In recent years, due to the development of deep neural networks, modeling of human joints in acquired images has made progress in skeleton extraction. However, most models have low modeling accuracy, poor real-time performance, and poor model availability. problem. Aiming at the above-mentioned human target detection problem, this paper uses the deep learning skeleton sequence model gesture recognition method in sports scenes to study, aiming to provide a gesture recognition method with strong noise resistance, good real-time performance and accurate model. This article uses motion video frame images to train the VGG16 network. Using the network to extract skeleton information can strengthen the posture feature expression, and use HOG for feature extraction, and use the Adam algorithm to optimize the network to extract more posture features, thereby improving the posture of the network Recognition accuracy. Then adjust the hyperparameters and network structure of the basic network according to the training results, and obtain the key poses in the sports scene through the final classifier.

Spatio-Temporal Convolutional Networks and N-Ary Ontologies for Human Activity-Aware Robotic System

Endowing a companion robot with cognitive abilities to recognize human daily activities, in particular from body skeletons information, is a significant challenge, which needs complex and novel approaches. Recently, most of the proposed approaches exploit the hand-crafted features or the predefined traversal rules techniques to recognize human daily activities from skeleton information, which often lead to the deficit of robustness and generalization. In this work, a novel hybrid framework for human activity-aware robotic system is proposed. In the low-level, a novel Spatio-Temporal Joint based Convolutional Neural Network (STJ-CNN) is proposed to recognize human daily activities in the ambient environments. In the high-level, novel representation and inference services based on Narrative Knowledge Representation Language (NKRL) are proposed to represent and combine the detected human activities with the ambient events, and to infer the semantic context of the detected activity. Empirical experiments on real-world datasets have been conducted, besides an online demonstration created to validate the proposed approach. The final results show that the proposed approach outperforms the baseline models with a significant improvement up to 24% in terms of $F$-score on DAHLIA dataset.

Estimation of ground reaction forces during stair climbing in patients with ACL reconstruction using a depth sensor-driven musculoskeletal model

BackgroundAlthough stair ambulation should be included in the rehabilitation of the long-term effects of ACL injury on knee function, the assessment of kinetic parameter in the situation where stair gait can only be established using costly and cumbersome force platforms via conventional inverse dynamic analysis. Therefore, there is a need to develop a practical laboratory setup as an assessment tool of the stair gait abnormalities in lower extremity that arise from an ACL deficiency. Research questionCan the use of a single depth sensor-driven full-body musculoskeletal gait model be considered an accurate assessment tool of the ground reaction forces (GRFs) during stair climbing for patients following ACL reconstruction (ACLR) surgery? MethodsA total of 15 patients who underwent ACLR participated in this study. GRFs data during stair climbing was collected using a custom-built 3-step staircase with two embedded force platforms. A single depth sensor, commercially available and cost effective, was used to obtain participants’ depth map information to extract the full-body skeleton information. The AnyBody TM GaitFullBody model was utilized to estimate GRFs attained by 25 artificial muscle-like actuators placed under each foot. Mean differences between the measured and estimated GRFs were compared using paired samples t-tests. The ensemble curves of the GRFs were compared between both approaches during stance phase of the gait cycle. ResultsThe findings of this study showed that the estimation of the GRFs produced during staircase gait using a depth sensor-driven musculoskeletal model can produce acceptable results when compared to the traditional inverse dynamics modelling approach as an alternative tool in clinical settings for individuals who had undergone ACLR. SignificanceThe introduced approach of full-body musculoskeletal modelling driven by a single depth sensor has the potential to be a cost-effective stair gait analysis tool for patients with ACL injury.

Driver activity recognition using spatial‐temporal graph convolutional LSTM networks with attention mechanism

Driver activity engagement while driving plays a vital role that leads to negative outcomes of driving safety. To reduce traffic accidents and ensure driving safety, real-time driver activity recognition architecture is proposed in this study. Specifically, a total of eight kinds of common driving-related activities are identified, which include the normal driving, left or right checking, texting, answering the phone, using media, drinking, and picking up objects. Raw experiment videos are collected via onboard monocular cameras, which are used for the upper body skeleton information extraction of the driver. Then, the graph convolutional networks (GCN) are constructed for spatial structure feature reasoning in a single frame, which is consecutively followed by long short-term memory (LSTM) networks for temporal motion feature learning within the sequence. Moreover, the attention mechanism is further utilised to emphasise the keyframes to select discriminative sequential information. Finally, a large-scale driver activity dataset, consisting of both naturalistic driving data and simulative driving data, is collected for model training and evaluations. Experimental results show that the general recall ratio of those eight driving-related activities reaches up to 88.8% and the real-time recognition efficiency can reach up to 24 fps, which would satisfy the real-time requirements of engineering applications.

Human action recognition based on Kinect

With the emergence of Kinect, many research results have emerged in human action recognition based on skeleton information, which has promoted the development of human-computer interaction. In this paper, from the skeleton data obtained by Kinect, static features and dynamic features are extracted, and the two are merged; SVM classifier is used for action recognition. It is verified on the MSR Daily Activity 3D data set, and the experimental results show that the method in this paper improves the accuracy of action recognition.

The Recognition of Teacher Behavior Based on Multimodal Information Fusion

Teaching reflection based on videos is the main method in teacher education and professional development. However, it takes a long time to analyse videos, and teachers are easy to fall into the state of information overload. With the development of “AI + education,” automatic recognition of teacher behavior to support teaching reflection has become an important research topic. In this paper, taking online open classroom teaching video as the data source, we collected and constructed a teacher behavior dataset. Using this dataset, we explored the behavior recognition methods based on RGB video and skeleton information, and the information fusion between them is carried out to improve the recognition accuracy. The experimental results show that the fusion of RGB information and skeleton information can improve the recognition accuracy, and the early-fusion effect is better than the late-fusion effect. This study helps to solve the problems of time-consumption and information overload in teaching reflection and then helps teachers to optimize the teaching strategies and improve the teaching efficiency.

Prevention of Unintended Appearance in Photos Based on Human Behavior Analysis

Nowadays, with smartphones, people can easily take photos, post photos to any social networks, and use the photos for various purposes. This leads to a social problem that unintended appearance in photos may threaten the facial privacy of photographed people. Some solutions to protect facial privacy in photos have already been proposed. However, most of them rely on different techniques to de-identify photos which can be done only by photographers, giving no choice to photographed person. To deal with that, we propose an approach that allows a photographed person to proactively detect whether someone is intentionally/unintentionally trying to take pictures of him. Thereby, he can have appropriate reaction to protect the facial privacy. In this approach, we assume that the photographed person uses a wearable camera to record the surrounding environment in real-time. The skeleton information of likely photographers who are captured in the monitoring video is then extracted and put into the calculation of dynamic programming score which is eventually compared with a threshold for recognition of photo-taking behavior. Experimental results demonstrate that by using the proposed approach, the photo-taking behavior is precisely recognized with high accuracy of 92.5%.

Deep Learning-Based Action Recognition Using 3D Skeleton Joints Information

Human action recognition has turned into one of the most attractive and demanding fields of research in computer vision and pattern recognition for facilitating easy, smart, and comfortable ways of human-machine interaction. With the witnessing of massive improvements to research in recent years, several methods have been suggested for the discrimination of different types of human actions using color, depth, inertial, and skeleton information. Despite having several action identification methods using different modalities, classifying human actions using skeleton joints information in 3-dimensional space is still a challenging problem. In this paper, we conceive an efficacious method for action recognition using 3D skeleton data. First, large-scale 3D skeleton joints information was analyzed and accomplished some meaningful pre-processing. Then, a simple straight-forward deep convolutional neural network (DCNN) was designed for the classification of the desired actions in order to evaluate the effectiveness and embonpoint of the proposed system. We also conducted prior DCNN models such as ResNet18 and MobileNetV2, which outperform existing systems using human skeleton joints information.

Using a Multilearner to Fuse Multimodal Features for Human Action Recognition

The representation and selection of action features directly affect the recognition effect of human action recognition methods. Single feature is often affected by human appearance, environment, camera settings, and other factors. Aiming at the problem that the existing multimodal feature fusion methods cannot effectively measure the contribution of different features, this paper proposed a human action recognition method based on RGB-D image features, which makes full use of the multimodal information provided by RGB-D sensors to extract effective human action features. In this paper, three kinds of human action features with different modal information are proposed: RGB-HOG feature based on RGB image information, which has good geometric scale invariance; D-STIP feature based on depth image, which maintains the dynamic characteristics of human motion and has local invariance; and S-JRPF feature-based skeleton information, which has good ability to describe motion space structure. At the same time, multiple K-nearest neighbor classifiers with better generalization ability are used to integrate decision-making classification. The experimental results show that the algorithm achieves ideal recognition results on the public G3D and CAD60 datasets.

Using Graph Convolutional Networks Skeleton-Based Pedestrian Intention Estimation Models for Trajectory Prediction

In autonomous driving scenarios, pedestrian trajectory prediction is an important research direction. Based on the spatio-temporal graph convolutional neural network, we propose a new pedestrian trajectory prediction algorithm. The new algorithm constructs a series of new models around pedestrian intention estimation. The construction of the estimation algorithm considers the following aspects: the contextual information of pedestrians and the surrounding environment, the “pedestrian ego-vehicle” interaction combined with the vehicle speed estimation, the pedestrian’s own skeletal structure information and body language estimation, which includes head joints and the relative structural relationship of the torso joints, including whether it is out of the same plane, is rotated, and so on. Skeleton information feature extraction and construction adopts the method of graph convolutional neural network to structure pedestrians into joints in the form of graphs in non-Euclidean space, and further adopts spatial temporal graph convolutional network for feature extraction and learning. The new method is named a “head-torso”-based spatial temporal graph convolutional network (HT-STGCN). On the dataset PID, the novel method achieves substantial improvements over mainstream methods. Experimental results show that combining HT-STGCN with observed action can improve trajectory prediction.

Deep learning-based multi-modal approach using RGB and skeleton sequences for human activity recognition

The deep learning techniques have achieved great success in the application of human activity recognition (HAR). In this paper, we propose a technique for HAR that utilizes the RGB and skeleton information with the help of a convolutional neural network (Convnet) and long short-term memory (LSTM) as a recurrent neural network (RNN). The proposed method has two parts: first, motion representation images like motion history image (MHI) and motion energy image (MEI) have been created from the RGB videos. The convnet has been trained, using these images with feature-level fusion. Second, the skeleton data have been utilized with a proposed algorithm that develops skeleton intensity images, for three views (top, front and side). Each view is first analyzed by a convnet, that generates the set of feature maps, which are fused for further analysis. On top of convnet sub-networks, LSTM has been used to exploit the temporal dependency. The softmax scores from these two independent parts are later combined at the decision level. Apart from the given approach for HAR, this paper also presents a strategy that utilizes the concept of cyclic learning rate to develop a multi-modal neural network by training the model only once to make the system more efficient. The suggested approach privileges for the perfect utilization of RGB and skeleton data available from an RGB-D sensor. The proposed approach has been tested on three famous and challenging multimodal datasets which are UTD-MHAD, CAD-60 and NTU-RGB + D120. Results have shown that the stated method gives a satisfactory result as compared to the other state-of-the-art systems.

Kinematic skeleton graph augmented network for human parsing

Human parsing, which is a task of labeling pixels in human images into different fine-grained semantic parts, has achieved significant progress during the past decade. However, there are still several challenges in human parsing, due to occlusions, varying poses and similar appearance between the left/right parts. To tackle these problems, a Human Kinematic Skeleton Graph Layer (HKSGL) is proposed to augment regular neural networks with human kinematic skeleton information. The HKSGL has two major components: kinematic skeleton graph and interconnected modular neural layer. The kinematic skeleton graph is a user pre-defined skeleton graph, which models the interconnections between different semantic parts. Then the skeleton graph is passed to the interconnected modular neural layer which is composed of a set of modular blocks corresponding to different semantic parts. The HKSGL is a lightweight, low costs layer which can be easily attached to any existing neural networks. To demonstrate the power of the HKSGL, a new dataset on human parsing in occlusions is also collected, termed the RAP-Occ. Extensive experiments have been performed on four datasets on human parsing, including the LIP, the CIHP, the ATR and the RAP-Occ. And two popular baselines, i.e., the Deeplab V3+ and the CE2P, are agumented by the proposed HKSGL. Competitive performance of the augmented models has been achieved in comparison with state-of-the-art methods.

3D Pose Estimation Based on Reinforce Learning for 2D Image-Based 3D Model Retrieval

In this paper, we propose a novel characteristic view selection model (CVSM) to address the 2D image-based 3D object retrieval problem. This work includes two key contributions: 1) we propose a novel reinforcement learning model to estimate the 3D pose based on a 2D image; and 2) we render the pose-specific model to generate a representative angle view for retrieval applications. First, we define state, policy, action and reward functions to train an agent with the reinforcement learning framework, by which the agent can effectively reduce the computational cost of the characteristic view selection and directly obtain the 3D model pose. Second, to resolve the problem of computing similarity in the cross-domain between the virtual 3D model view and the real query image, we project them into the skeleton domain, and the skeleton information can effectively bridge the gap between the image and 3D model view for cross-media retrieval. To demonstrate the performance of our approach, we compare with some classic 3D pose estimation methods using the popular Pascal3D dataset. To demonstrate the performance of our approach in model retrieval, we collect a new dataset that includes pairs of 2D images and 3D objects, where 3D objects are based on the ModelNet40 dataset and 2D images are based on the ImageNet dataset, and we experiment with our method using the SHREC 2018 and SHREC 2019 databases. The experimental results demonstrate the superiority of our method.

Fall Detection Based on Key Points of Human-Skeleton Using OpenPose

According to statistics, falls are the primary cause of injury or death for the elderly over 65 years old. About 30% of the elderly over 65 years old fall every year. Along with the increase in the elderly fall accidents each year, it is urgent to find a fast and effective fall detection method to help the elderly fall.The reason for falling is that the center of gravity of the human body is not stable or symmetry breaking, and the body cannot keep balance. To solve the above problem, in this paper, we propose an approach for reorganization of accidental falls based on the symmetry principle. We extract the skeleton information of the human body by OpenPose and identify the fall through three critical parameters: speed of descent at the center of the hip joint, the human body centerline angle with the ground, and width-to-height ratio of the human body external rectangular. Unlike previous studies that have just investigated falling behavior, we consider the standing up of people after falls. This method has 97% success rate to recognize the fall down behavior.

Vision-Based Contactless Pose Estimation for Human Thermal Discomfort

Real-time and effective human thermal discomfort detection plays a critical role in achieving energy efficient control of human centered intelligent buildings because estimation results can provide effective feedback signals to heating, ventilation and air conditioning (HVAC) systems. How to detect occupant thermal discomfort is a challenge. Unfortunately, contact or semi-contact perception methods are inconvenient in practical application. From the contactless perspective, a kind of vision-based contactless human discomfort pose estimation method was proposed in this paper. Firstly, human pose data were captured from a vision-based sensor, and corresponding human skeleton information was extracted. Five thermal discomfort-related human poses were analyzed, and corresponding algorithms were constructed. To verify the effectiveness of the algorithms, 16 subjects were invited for physiological experiments. The validation results show that the proposed algorithms can recognize the five human poses of thermal discomfort.

Research on Human Action Recognition Algorithm Based on Sine Feature

Recent studies have shown that exploring features of the skeleton data is vital for human action recognition. Nevertheless, how to effectively extract discriminative features is still a challenging work. In this paper, we propose a novel method that extracts a sine feature for human action recognition from skeleton data. Kinect is used to extract human skeleton information (3D coordinates of each joint point) firstly, then two joint points are connected to define a skeleton vector according to the principle of human body structure, and the sine feature of each skeleton vector is calculated as a new pose description feature, SVM is used to classify the obtained features for human action classification. Experimental results on Cornell Activity Database are provided, and the results demonstrate the effectiveness of our approach.

Dynamic gesture recognition by using CNNs and star RGB: A temporal information condensation

Due to technological advances, machines are increasingly present in people’s daily lives. Thus, there has been more and more effort to develop interfaces that provide an intuitive way of interaction, such as dynamic gestures. Currently, the most common trend is to use multimodal data, as depth and skeleton information, to enable dynamic gesture recognition. However, it would be more interesting if only color information was used, since RGB cameras are usually available in almost every public place, and could be used for gesture recognition without the need of installing additional equipment. The main problem with such approach is the difficulty of representing spatio-temporal information using just color. With this in mind, we propose a technique capable of condensing a dynamic gesture, shown in a video, in just one RGB image. We call this technique star RGB. This image is then passed to a classifier formed by two Resnet CNNs, a soft-attention ensemble, and a fully connected layer, which indicates the class of the gesture present in the input video. Experiments were carried out using Montalbano, GRIT, and isoGD datasets. For Montalbano dataset, the proposed approach achieved an accuracy of 94.58%. Such result reaches the state-of-the-art when considering this dataset and only color information. For GRIT dataset, our proposal achieves more than 98% of accuracy, recall, precision, and F1-score, outperforming the authors’ approach by more than 6%. Regarding the large scale isoGD dataset, the proposal achieved 52.18% of accuracy. However, taking into account the complexity of the dataset (eight different gestures categories) and the amount of classes (249), we consider that our approach is competitive with previous ones, since we employed only color information to recognize gestures instead of all the multimodal data available, usually used by other methods.

Shape refinement and rigging of raw-scanned 3D volume by a user-specified skeleton

RGB-D camera-based scanning has increased in popularity; however, raw-scanned three-dimensional (3D) models contain several issues, such as fused arms and legs, that hinder animation. Here, we describe a semiautomatic method that allows generation of a rigged 3D mesh from a raw-scanned 3D volume with simple annotations. The method allows a user to annotate a skeleton structure on registered photographs captured in the scanning step, followed by automatic cutting of fused body parts using the skeleton information to beautify the cut surface by applying mesh smoothing. The system then generates a skinned 3D mesh based on the user-specified 3D skeleton. We tested our method using several raw-scanned 3D plush toy models and successfully generated plausible animations.

Development of a Low Cost and Path-free Autonomous Patrol System Based on Stereo Vision System and Checking Flags

Nowadays, security guard patrol services are becoming roboticized. However, high construction prices and complex systems make patrol robots difficult to be popularized. In this research, a simplified autonomous patrolling robot is proposed, which is fabricated by upgrading a wheeling household robot with stereo vision system (SVS), radio frequency identification (RFID) module, and laptop. The robot has four functions: independent patrolling without path planning, checking, intruder detection, and wireless backup. At first, depth information of the environment is analyzed through SVS to find a passable path for independent patrolling. Moreover, the checkpoints made with RFID tag and color pattern are placed in appropriate positions within a guard area. While a color pattern is detected by the SVS, the patrolling robot is guided to approach the pattern and check its RFID tag. For more, the human identification function of SVS is used to detect an intruder. While a skeleton information of the human is analyzed by SVS, the intruder detection function is triggered, then the robot follows the intruder and record the images of the intruder. The recorded images are transmitted to a server through Wi-Fi to realize the remote backup, and users can query the recorded images from the network. Finally, an experiment is made to test the functions of the autonomous patrolling robot successfully.

Human skeleton mutual learning for person re-identification

Person re-identification refers to matching people across non-overlapping camera views on different locations and at different times. In the case of changes in perspective, light, background, veil, and person's clothing, traditional method cannot achieve person recognition effectively and reliably. In this paper, we propose a novel biometric metric learning method named Human Skeleton Mutual Learning person re-identification (HSMLP-Reid). The purpose of HSML person re-identification method (HSMLP-Reid) largely aims to use the new pedestrian local segmentation method proposed in this paper combined with the global skeleton information to solve the influence of background and local posture change. Firstly, bottom-up method is used to estimate the pedestrian posture and skeleton and the joint points of the pedestrians will be marked in this process. A new local segmentation method proposed in this paper named joint segmentation is used to locally segment pedestrians and perform local block matching. Furthermore, we learn the global skeleton information by defined joint distances from the pedestrian 2D skeleton estimation by the bottom-up method and use global skeleton information to global skeleton matching. Finally, we use local match and global skeleton match for mutual learning. Local match based on pedestrian nodes and global skeleton match based on pedestrian skeleton are based on biometrics. We learn the classification loss and metric learning loss to train model. Metric loss includes global skeletal distance and local block metric distance. Extensive experimental results on the large-scale Market1501, CUHK03 and CUHK-SYSU data sets demonstrate that the proposed method achieves consistently superior performance and outperforms most of the state-of-the-art methods.