Human Pose Estimation Method Research Articles

Multi-modal Human pose estimation based on probability distribution perception on a depth convolution neural network

• A method of human key-point coordinate representation based on probability distribution perception. • Addressing issues :key-point representation quantization error , algorithm inconsistency. • This method gives a high level of precision in the experiment. Existing human pose estimation methods have some problems, such as key-point representation quantization error, algorithm inconsistency, and easy-to-introduce unequal quantization error. Addressing these issues, this paper proposes a method of human key-point coordinate representation based on probability distribution perception on a depth convolution neural network . In this method, the perceived probability of the key-point coordinate distribution is extracted by a convolutional neural network , and the cross entropy is formed with the prediction probability distribution as the loss function. Under the constraint of minimizing the Kullback-Leibler (KL) distance between the prediction and the ground truth, the convolution neural network is optimized, and the coordinates of the heat map are finally located. The experimental results show that compared with the traditional method based on a heat map, the proposed method can accelerate the reasoning speed of the model in the COCO training set, and at the same time, it can improve the map by 1.7

Smart-VPoseNet: 3D human pose estimation models and methods based on multi-view discriminant network

In the multi-view 3D human pose estimation, when there are some poor-quality views in multi-view data, these views will cause serious errors to multi-view 3D pose estimation. We also found many similar problems in the single-view model. For example, in a motion sequence of the current view, there are some frames in which the human body are occluded or the body joints exceed the camera’s field of view. Therefore, we hope to implement an intelligent method to make the model jump in multiple views during 3D pose modeling, so as to avoid the model using the views that have serious problems and pay more attention to higher quality views. We call this method Smart-VPoseNet. We defined three view selection rules to explain the quality of each view in the dataset, and designed a novel view discriminant network. According to the three rules, the data distribution and characteristics in the dataset are visualized for analyzing and designing the details of the model. In addition, we also introduced a hybrid view selection method based on these rules. In the experiment, we used a single-view pretrained model and Smart-VPoseNet to complete a series of comprehensive comparison experiments. These experiments illustrate that the work of this article has a novel and feasible optimization theory and show the application effect of our work in 3D pose estimation. We believe that our work has opened up a new optimization model based on multiple views.

Human Pose Estimation Based on Multi-resolution Feature Parallel Network for Public Security

Smoking detection is an essential part of safety production management. With the wide application of artificial intelligence technology in all kinds of behavior monitoring applications, the technology of real-time monitoring smoking behavior in production areas based on video is essential. In order to carry out smoking detection, it is necessary to analyze the position of key points and posture of the human body in the input image. Due to the diversity of human pose and the complex background in general scene, the accuracy of human pose estimation is not high. To predict accurate human posture information in complex backgrounds, a deep learning network is needed to obtain the feature information of different scales in the input image. The human pose estimation method based on multi-resolution feature parallel network has two parts. The first is to reduce the loss of semantic information by hole convolution and deconvolution in the part of multi-scale feature fusion. The second is to connect different resolution feature maps in the output part to generate the high-quality heat map. To solve the problem of feature loss of previous serial models, more accurate human pose estimation data can be obtained. Experiments show that the accuracy of the proposed method on the coco test set is significantly higher than that of other advanced methods. Accurate human posture estimation results can be better applied to the field of smoking detection, and the smoking behavior can be detected by artificial intelligence, and the alarm will be automatically triggered when the smoking behavior is found.

Deep Learning Methods for 3D Human Pose Estimation under Different Supervision Paradigms: A Survey

The rise of deep learning technology has broadly promoted the practical application of artificial intelligence in production and daily life. In computer vision, many human-centered applications, such as video surveillance, human-computer interaction, digital entertainment, etc., rely heavily on accurate and efficient human pose estimation techniques. Inspired by the remarkable achievements in learning-based 2D human pose estimation, numerous research studies are devoted to the topic of 3D human pose estimation via deep learning methods. Against this backdrop, this paper provides an extensive literature survey of recent literature about deep learning methods for 3D human pose estimation to display the development process of these research studies, track the latest research trends, and analyze the characteristics of devised types of methods. The literature is reviewed, along with the general pipeline of 3D human pose estimation, which consists of human body modeling, learning-based pose estimation, and regularization for refinement. Different from existing reviews of the same topic, this paper focus on deep learning-based methods. The learning-based pose estimation is discussed from two categories: single-person and multi-person. Each one is further categorized by data type to the image-based methods and the video-based methods. Moreover, due to the significance of data for learning-based methods, this paper surveys the 3D human pose estimation methods according to the taxonomy of supervision form. At last, this paper also enlists the current and widely used datasets and compares performances of reviewed methods. Based on this literature survey, it can be concluded that each branch of 3D human pose estimation starts with fully-supervised methods, and there is still much room for multi-person pose estimation based on other supervision methods from both image and video. Besides the significant development of 3D human pose estimation via deep learning, the inherent ambiguity and occlusion problems remain challenging issues that need to be better addressed.

A review of 3D human pose estimation algorithms for markerless motion capture

Human pose estimation is a very active research field, stimulated by its important applications in robotics, entertainment or health and sports sciences, among others. Advances in convolutional networks triggered noticeable improvements in 2D pose estimation, leading modern 3D markerless motion capture techniques to an average error per joint of 20 mm. However, with the proliferation of methods, it is becoming increasingly difficult to make an informed choice. Here, we review the leading human pose estimation methods of the past five years, focusing on metrics, benchmarks and method structures. We propose a taxonomy based on accuracy, speed and robustness that we use to classify de methods and derive directions for future research.

DARLENE – Improving situational awareness of European law enforcement agents through a combination of augmented reality and artificial intelligence solutions

Background: Augmented reality (AR) and artificial intelligence (AI) are highly disruptive technologies that have revolutionised practices in a wide range of domains. Their potential has not gone unnoticed in the security sector with several law enforcement agencies (LEAs) employing AI applications in their daily operations for forensics and surveillance. In this paper, we present the DARLENE ecosystem, which aims to bridge existing gaps in applying AR and AI technologies for rapid tactical decision-making in situ with minimal error margin, thus enhancing LEAs’ efficiency and Situational Awareness (SA). Methods: DARLENE incorporates novel AI techniques for computer vision tasks such as activity recognition and pose estimation, while also building an AR framework for visualization of the inferenced results via dynamic content adaptation according to each individual officer’s stress level and current context. The concept has been validated with end-users through co-creation workshops, while the decision-making mechanism for enhancing LEAs’ SA has been assessed with experts. Regarding computer vision components, preliminary tests of the instance segmentation method for humans’ and objects’ detection have been conducted on a subset of videos from the RWF-2000 dataset for violence detection, which have also been used to test a human pose estimation method that has so far exhibited impressive results and will constitute the basis of further developments in DARLENE. Results: Evaluation results highlight that target users are positive towards the adoption of the proposed solution in field operations, and that the SA decision-making mechanism produces highly acceptable outcomes. Evaluation of the computer vision components yielded promising results and identified opportunities for improvement. Conclusions: This work provides the context of the DARLENE ecosystem and presents the DARLENE architecture, analyses its individual technologies, and demonstrates preliminary results, which are positive both in terms of technological achievements and user acceptance of the proposed solution.

INFORMATION ACQUISITION ON PEDESTRIAN MOVEMENTS IN URBAN TRAFFIC WITH A MOBILE MULTI-SENSOR SYSTEM

Abstract. This paper presents an approach which combines LiDAR sensors and cameras of a mobile multi-sensor system to obtain information about pedestrians in the vicinity of the sensor platform. Such information can be used, for example, in the context of driver assistance systems. In the first step, our approach starts by using LiDAR sensor data to detect and track pedestrians, benefiting from LiDAR’s capability to directly provide accurate 3D data. After LiDAR-based detection, the approach leverages the typically higher data density provided by 2D cameras to determine the body pose of the detected pedestrians. The approach combines several state-of-the-art machine learning techniques: it uses a neural network and a subsequent voting process to detect pedestrians in LiDAR sensor data. Based on the known geometric constellation of the different sensors and the knowledge of the intrinsic parameters of the cameras, image sections are generated with the respective regions of interest showing only the detected pedestrians. These image sections are then processed with a method for image-based human pose estimation to determine keypoints for different body parts. These keypoints are finally projected from 2D image coordinates to 3D world coordinates using the assignment of the original LiDAR points to a particular pedestrian.

Ergonomic postural assessment using a new open-source human pose estimation technology (OpenPose)

Observational ergonomic postural assessment methods have been commonly used to evaluate the risks of musculoskeletal disorders. Researchers have proposed semi-automatic methods using Kinect, known for limitations with body occlusions and non-frontal tracking. Meanwhile, new human pose estimation methods have been actively developed, and a popular open-source technology is OpenPose. This study aims to propose the OpenPose-based system for computing joint angles and RULA/REBA scores and validate against the reference motion capture system, and compare its performance to the Kinect-based system. Recordings of 10 participants performing 12 experimental tasks under different conditions: with/without body occlusions and tracked from frontal/non-frontal views were analyzed. OpenPose showed good performance under all task conditions, whereas Kinect performed significantly worse than OpenPose especially at cases with body occlusions or non-frontal tracking. The findings suggested that OpenPose could be a promising technology to measure joint angles and conduct semi-automatic ergonomic postural assessments in the real workspace where the conditions are often non-ideal.

Context-Guided Adaptive Network for Efficient Human Pose Estimation

Although recent work has achieved great progress in human pose estimation (HPE), most methods show limitations in either inference speed or accuracy. In this paper, we propose a fast and accurate end-to-end HPE method, which is specifically designed to overcome the commonly encountered jitter box, defective box and ambiguous box problems of box-based methods, e.g. Mask R-CNN. Concretely, 1) we propose the ROIGuider to aggregate box instance features from all feature levels under the guidance of global context instance information. Further, 2) the proposed Center Line Branch is equipped with a Dichotomy Extended Area algorithm to adaptively expand each instance box area, and Ambiguity Alleviation strategy to eliminate duplicated keypoints. Finally, 3) to achieve efficient multi-scale feature fusion and real-time inference, we design a novel Trapezoidal Network (TNet) backbone. Experimenting on the COCO dataset, our method achieves 68.1 AP at 25.4 fps, and outperforms Mask-RCNN by 8.9 AP at a similar speed. The competitive performance on the HPE and person instance segmentation tasks over the state-of-the-art models show the promise of the proposed method. The source code will be made available at https://github.com/zlcnup/CGANet.

Body parts relevance learning via expectation–maximization for human pose estimation

Recently, most existing human pose estimation methods fuse multi-stage convolutional modules to learn a shared feature representation. In this paper, we propose a expectation–maximization (EM) mapping-based network to learn specific related body parts for human pose estimation, named EMposeNet. It maps specific feature of related parts from the original fully shared feature space. From the perspective of multi-task learning, we can regard the task of human pose estimation as a homogeneous multi-task learning. Sharing features among related tasks can result in a more compact model and better generalization ability. However, sharing features for those unrelated or weakly related tasks will deteriorate the estimation performance. Our proposed method aims at performing EM algorithm to learn the related body part, where the predicted keypoint heatmap is potentially more accurate and spatially more precise. We conduct extensive experiments on two benchmark datasets, including the MSCOCO keypoint detection dataset and the MPII human pose dataset, to empirically demonstrate the validity of the proposed method. The results on such two benchmark datasets show that the proposed approach achieves a competitive performance.

A novel dataset and deep learning-based approach for marker-less motion capture during gait

BackgroundThe deep learning-based human pose estimation methods, which can estimate joint centers position, have achieved promising results on the publicly available human pose datasets (e.g., Human3.6 M). However, these datasets may be less efficient for gait study, particularly for clinical applications, because of the limited number of subjects, their homogeneity (all asymptomatic adults), and the errors introduced by marker placement on subjects’ regular clothing. Research questionHow a new human pose dataset, adapted for gait study, could contribute to the advancement and evaluation of marker-less motion capture systems? MethodsA marker-less system, based on deep learning-based pose estimation methods, was proposed. A new dataset (ENSAM dataset) was collected. Twenty-two asymptomatic adults, one adult with scoliosis, one adult with spondylolisthesis, and seven children with bone disease performed ten walking trials, while being recorded both by the proposed marker-less system and a reference system – combining a marker-based motion capture system and a medical imaging system (EOS). The dataset was split into training and test sets. The pose estimation method, already trained on the Human3.6 M dataset, was evaluated on the ENSAM test set, then reevaluated after further training on the ENSAM training set. The joints coordinates were evaluated, using Bland-Altman bias and 95 % confidence interval, and joint position error (the Euclidean distance between the estimated joint centers and the corresponding reference values). ResultsThe Bland-Altman 95 % confidence intervals were substantially improved after finetuning the pose estimation method on the ENSAM training set (e.g., from 106.9 mm to 17.4 mm for the hip joint). With the new dataset and approach, the mean joint position error varied from 6.2 mm for ankles to 21.1 mm for shoulders. SignificanceThe proposed marker-less system achieved promising results in terms of joint position errors. Future studies are necessary to assess the system in terms of gait parameters.

3D Human Pose Estimation Based on a Fully Connected Neural Network With Adversarial Learning Prior Knowledge

3D human pose estimation is more and more widely used in the real world, such as sports guidance, limb rehabilitation training, augmented reality, and intelligent security. Most existing human pose estimation methods are designed based on an RGB image obtained by one optical sensor, such as a digital camera. There is some prior knowledge, such as bone proportion and angle limitation of joint hinge motion. However, the existing methods do not consider the correlation between different joints from multi-view images, and most of them adopt fixed spatial prior constraints, resulting in poor generalizations. Therefore, it is essential to build a multi-view image acquisition system using optical sensors and customized algorithms for a 3D reconstruction of the human pose in the image. Inspired by generative adversarial networks (GAN), we used a data-driven method to learn the implicit spatial prior information and classified joints according to the natural connection characteristics. To accelerate the proposed method, we proposed a fully connected network with skip connections and used the SMPL model to make the 3D human body reconstruction. Experimental results showed that compared with other state-of-the-art methods, the joints’ average error of the proposed method was the smallest, which indicated the best performance. Moreover, the running time of the proposed method was 1.3 seconds per frame, which may not meet real-time requirements, but is still much faster than most existing methods.

An adversarial human pose estimation network injected with graph structure

Because of the invisible human keypoints in images caused by illumination, occlusion and overlap, it is likely to produce unreasonable human pose prediction for most of the current human pose estimation methods. In this paper, we design a novel generative adversarial network (GAN) to improve the localization accuracy of visible joints when some joints are invisible. The network consists of two simple but efficient modules, i.e., Cascade Feature Network (CFN) and Graph Structure Network (GSN). First, the CFN utilizes the prediction maps from the previous stages to guide the prediction maps in the next stage to produce accurate human pose. Second, the GSN is designed to contribute to the localization of invisible joints by passing message among different joints. According to GAN, if the prediction pose produced by the generator G cannot be distinguished by the discriminator D, the generator network G has successfully obtained the underlying dependence of human joints. We conduct experiments on three widely used human pose estimation benchmark datasets, i.e., LSP, MPII and COCO, whose results show the effectiveness of our proposed framework.

Designing Compact Convolutional Filters for Lightweight Human Pose Estimation

Existing methods for human pose estimation usually use a large intermediate tensor, leading to a high computational load, which is detrimental to resource‐limited devices. To solve this problem, we propose a low computational cost pose estimation network, MobilePoseNet, which includes encoder, decoder, and parallel nonmaximum suppression operation. Specifically, we design a lightweight upsampling block instead of transposing the convolution as the decoder and use the lightweight network as our downsampling part. Then, we choose the high‐resolution features as the input for upsampling to reduce the number of model parameters. Finally, we propose a parallel OKS‐NMS, which significantly outperforms the conventional NMS in terms of accuracy and speed. Experimental results on the benchmark datasets show that MobilePoseNet obtains almost comparable results to state‐of‐the‐art methods with a low compilation load. Compared to SimpleBaseline, the parameter of MobilePoseNet is only 4%, while the estimation accuracy reaches 98%.

A Novel Approach of Intelligent Computing for Multiperson Pose Estimation with Deep High Spatial Resolution and Multiscale Features

Currently, human pose estimation (HPE) methods mainly rely on the design framework of Convolutional Neural Networks (CNNs). These CNNs typically consist of high‐to‐low‐resolution subnetworks (encoder) to learn semantic information and low‐to‐high subnetworks (decoder) to raise the resolution for keypoint localization. Because too low‐resolution feature maps in encoder will inevitably lose some spatial information, which cannot be recovered in the upsampling stages, keeping high spatial resolution features is critical for human pose estimation. On the other hand, due to scale variation of human body parts, multiscale features are also very important for human pose estimation. In this paper, a novel backbone network is proposed specifically for HPE, named High Spatial Resolution and Multiscale Networks (HSR‐MSNet), which maintain high spatial resolution features in deeper layers of the encoder and meanwhile construct multiscale features within one single residual block via subgroup splitting and fusion of feature maps. Experiments show that our approach outperforms other state‐of‐the‐art methods with more accurate keypoint locations on COCO dataset.

Fast and Lightweight Human Pose Estimation

Although achieving significant improvement on pose estimation, the major drawback is that most top-performing methods tend to adopt complex architecture and spend large computational cost to achieve higher performance. Due to the edge device’s limited resources, its top-performing methods are hard to maintain fast inference speed in practice. To address this issue, we proposed the fast and lightweight human pose estimation method to maintain high performance and bear the less computational cost. Especially, the proposed method consists of two parts, i.e., the fast and lightweight pose network (FLPN) for pose estimation and a novel lightweight bottleneck block for reducing computational cost, which can integrate the simple network and lightweight bottleneck into an efficient method for accurate pose estimation. In terms of lightweight bottleneck block, we introduce the structural similarity measurement (SSIM) to refine the appropriate ratio of intrinsic feature maps and reduce the model size. Furthermore, an attention mechanism is also adopted in our lightweight bottleneck block for modeling the contextual information. We demonstrate the performance of the proposed method with extensive experiments on the two standard benchmark datasets by comparing our method with state-of-the-art methods. On the COCO keypoint detection dataset, our proposed method attains a similar accuracy with these state-of-the-art methods, but the computational cost of these top-performing methods is more than 7 times that of ours.

Error Compensation Heatmap Decoding for Human Pose Estimation

As a fundamental component of heatmap-based human pose estimation methods, heatmap decoding is to transform heatmaps into joint coordinates. We found that previous heatmap decoding methods generally ignored the effect of systematic errors introduced by the resolution increaseing operations in the network decoder. This work fills the gap by taking the systematic errors in heatmap decoding into consideration. We proposed a fast method to reduces the systematic and random errors in one shot by error compensation. The proposed method outperforms the previous best method on the COCO and the MPII datasets while being over 2 times faster. Extensive experiments with different networks, resolutions, metrics and datasets have proved the rationality of the proposed idea.

Automatically Measure the Quality of Infants’ Spontaneous Movement via Videos to Predict the Risk of Cerebral Palsy

Objective: Early brain development assessment of infants is very important for infant growth and development. Timely diagnosis of cerebral palsy can reduce its negative effects. In this paper, a novel automatic measurement method was proposed for early prediction of infant cerebral palsy with videos towards Prechtl’s General Movements Assessment (GMA). Firstly, we adopted the-state-of-the-art human pose estimation method to obtain the 2D or 3D posture of the lying infant. After a series of interpolation and filtering, continuous infant’s limbs movements curves were obtained. Through appropriate feature extraction, the infant’s limbs movements complexity and the movements correlation between the limbs were integrated, and finally a comprehensive evaluation score for evaluating the complexity of the infant’s spontaneous movements was obtained. To verify our score, we did some experiments on a public dataset consists of 12 real infants’ movements. Our method achieved the state-of-the-art with sensitivity of 100%, specificity of 87.5%, and accuracy of 91.67% on the public dataset whether in 2D or 3D posture input. We also collected another 47 cases of infant movement color videos ourselves. And achieved an accuracy of 91.5%, sensitivity of 100%, and specificity of 87.8% on all 59 (12+47) subjects. The results show that our score method has a great prospect in assessing the infant’s early brain development. The proposed method can be widely used in the screening of babies in homes and hospitals.

DSPNet: A low computational-cost network for human pose estimation

Existing human pose estimation methods usually have a high computational load, which is very unfavorable for resource-limited equipment. To address this issue, we propose a low computational-cost deep supervision pyramid network called DSPNet. Firstly, we design a lightweight up-sampling unit instead of transposed convolution as a decoder for the network. In the case of decreased computation, it has brought an increase in prediction accuracy. Secondly, we present a novel deep supervision pyramid architecture to improve the multi-scale obtaining ability of MSRA SimpleBaseline while not bringing any increase in the number of parameters. The experimental results on both MPII and COCO pose estimation benchmarks illustrate that DSPNet achieves almost equivalent state-of-the-art results with a low computational load. Especially, the computational cost of DSPNet is 12.7% of SimpleBaseline and the estimation accuracy is improved by 0.9 points when both methods use the same backbone network (EfficientNet) on MPII validation set. The code of the proposed method is availabe at https://github.com/sumaliqinghua/DSPNet.

Pose estimation by extended Kalman filter using noise covariance matrices based on sensor output

This paper presents an extended Kalman filter for pose estimation using noise covariance matrices based on sensor output. Compact and lightweight nine-axis motion sensors are used for motion analysis in widely various fields such as medical welfare and sports. A nine-axis motion sensor includes a three-axis gyroscope, a three-axis accelerometer, and a three-axis magnetometer. Information obtained from the three sensors is useful for estimating joint angles using the Kalman filter. The extended Kalman filter is used widely for state estimation because it can estimate the status with a small computational load. However, determining the process and observation noise covariance matrices in the extended Kalman filter is complicated. The noise covariance matrices in the extended Kalman filter were found for this study based on the sensor output. Postural change appears in the gyroscope output because the rotational motion of the joints produces human movement. Therefore, the process noise covariance matrix was determined based on the gyroscope output. An observation noise covariance matrix was determined based on the accelerometer and magnetometer output because the two sensors’ outputs were used as observation values. During a laboratory experiment, the lower limb joint angles of three participants were measured using an optical 3D motion analysis system and nine-axis motion sensors while participants were walking. The lower limb joint angles estimated using the extended Kalman filter with noise covariance matrices based on sensor output were generally consistent with results obtained from the optical 3D motion analysis system. Furthermore, the lower limb joint angles were measured using nine-axis motion sensors while participants were running in place for about 100 s. The experiment results demonstrated the effectiveness of the proposed method for human pose estimation.