Human Pose Estimation Algorithm Research Articles

Clinical gait analysis using video-based pose estimation: Multiple perspectives, clinical populations, and measuring change.

Gait dysfunction is common in many clinical populations and often has a profound and deleterious impact on independence and quality of life. Gait analysis is a foundational component of rehabilitation because it is critical to identify and understand the specific deficits that should be targeted prior to the initiation of treatment. Unfortunately, current state-of-the-art approaches to gait analysis (e.g., marker-based motion capture systems, instrumented gait mats) are largely inaccessible due to prohibitive costs of time, money, and effort required to perform the assessments. Here, we demonstrate the ability to perform quantitative gait analyses in multiple clinical populations using only simple videos recorded using low-cost devices (tablets). We report four primary advances: 1) a novel, versatile workflow that leverages an open-source human pose estimation algorithm (OpenPose) to perform gait analyses using videos recorded from multiple different perspectives (e.g., frontal, sagittal), 2) validation of this workflow in three different populations of participants (adults without gait impairment, persons post-stroke, and persons with Parkinson's disease) via comparison to ground-truth three-dimensional motion capture, 3) demonstration of the ability to capture clinically relevant, condition-specific gait parameters, and 4) tracking of within-participant changes in gait, as is required to measure progress in rehabilitation and recovery. Importantly, our workflow has been made freely available and does not require prior gait analysis expertise. The ability to perform quantitative gait analyses in nearly any setting using only low-cost devices and computer vision offers significant potential for dramatic improvement in the accessibility of clinical gait analysis across different patient populations.

Comparison of Shoulder Range of Motion Quantified with Mobile Phone Video-Based Skeletal Tracking and 3D Motion Capture-Preliminary Study.

The accuracy of human pose tracking using smartphone camera (2D-pose) to quantify shoulder range of motion (RoM) is not determined. Twenty healthy individuals were recruited and performed shoulder abduction, adduction, flexion, or extension, captured simultaneously using a smartphone-based human pose estimation algorithm (Apple's vision framework) and using a skin marker-based 3D motion capture system. Validity was assessed by comparing the 2D-pose outcomes against a well-established 3D motion capture protocol. In addition, the impact of iPhone positioning was investigated using three smartphones in multiple vertical and horizontal positions. The relationship and validity were analysed using linear mixed models and Bland-Altman analysis. We found that 2D-pose-based shoulder RoM was consistent with 3D motion capture (linear mixed model: R2 > 0.93) but was somewhat overestimated by the smartphone. Differences were dependent on shoulder movement type and RoM amplitude, with adduction the worst performer among all tested movements. All motion types were described using linear equations. Correction methods are provided to correct potential out-of-plane shoulder movements. Shoulder RoM estimated using a smartphone camera is consistent with 3D motion-capture-derived RoM; however, differences between the systems were observed and are likely explained by differences in thoracic frame definitions.

Accuracy of Video-Based Gait Analysis Using Pose Estimation During Treadmill Walking Versus Overground Walking in Persons After Stroke.

Video-based pose estimation is an emerging technology that shows significant promise for improving clinical gait analysis by enabling quantitative movement analysis with little costs of money, time, or effort. The objective of this study is to determine the accuracy of pose estimation-based gait analysis when video recordings are constrained to 3 common clinical or in-home settings (ie, frontal and sagittal views of overground walking and sagittal views of treadmill walking). Simultaneous video and motion capture recordings were collected from 30 persons after stroke during overground and treadmill walking. Spatiotemporal and kinematic gait parameters were calculated from videos using an open-source human pose estimation algorithm and from motion capture data using traditional gait analysis. Repeated-measures analyses of variance were then used to assess the accuracy of the pose estimation-based gait analysis across the different settings, and the authors examined Pearson and intraclass correlations with ground-truth motion capture data. Sagittal videos of overground and treadmill walking led to more accurate measurements of spatiotemporal gait parameters versus frontal videos of overground walking. Sagittal videos of overground walking resulted in the strongest correlations between video-based and motion capture measurements of lower extremity joint kinematics. Video-based measurements of hip and knee kinematics showed stronger correlations with motion capture versus ankle kinematics for both overground and treadmill walking. Video-based gait analysis using pose estimation provides accurate measurements of step length, step time, and hip and knee kinematics during overground and treadmill walking in persons after stroke. Generally, sagittal videos of overground gait provide the most accurate results. Many clinicians lack access to expensive gait analysis tools that can help identify patient-specific gait deviations and guide therapy decisions. These findings show that video-based methods that require only common household devices provide accurate measurements of a variety of gait parameters in persons after stroke and could make quantitative gait analysis significantly more accessible.

Human Pose Estimation based on HRNet and Feature Pyramids

A popular area of research in the realm of computer vision is human pose estimation, which is the process of recovering human joint points from the given images or videos. Convolutional neural networks, which have great feature representation capabilities, have become a fundamental component of human pose estimation algorithms as a result of the deep learning field's quick development. To enhance feature quality and raise the precision of human pose estimate, in this paper, we propose a new model based on HRNet neural network and feature pyramid to more accurately capture each main part of the human body. The model uses HRNet as the backbone network, taking advantage of its ability to combine low resolution with high resolution, and afterwards adds characteristic pyramids to the HRNetnetwork in order to enhance the capability of detecting small objects by utilizing its ability to solve multi-scale problems. The model can estimate a human's pose accurately, according to experimental results, and on the MPII Human Pose dataset, the addition of the feature pyramid enhances the model's detection performance by 0.05. in order to achieve smoothing after adding feature mapping with different resolutions, the detection performance improves by 0.17 after further improvement by adding convolution. For the elbow, it improves by almost 0.5. The model's ability to increase the precision of human pose estimate is supported by all of the outcomes.

A review of human pose estimation algorithm based on convolutional neural network

A review of human pose estimation algorithm based on convolutional neural network

Improved lightweight human pose estimation algorithm

Improved lightweight human pose estimation algorithm

A Study on 3D Human Pose Estimation Using Through-Wall IR-UWB Radar and Transformer

In this paper, we propose a human pose estimation algorithm for an impulse radio ultra-wideband (IR-UWB) radar based on the transformer-based deep learning model. We have built an IR-UWB radar system with an 8-by-8 multiple-input multiple-output (MIMO) antenna array. The IR-UWB radar system in our paper is advantageous for the through-wall detection application since it operates on a very low frequency range (i.e., 0.45 to 3.55 GHz) compared to other existing works on RF-based human pose estimation. Moreover, the human pose estimation by an IR-UWB radar has not been studied very well in other existing works since all existing works have used a frequency-modulated continuous wave (FMCW) radar or a WiFi device. We propose a 3D-TransPOSE algorithm for the 3D human pose estimation from the IR-UWB radar signals. The proposed algorithm is designed based on the transformer architecture. While the transformer has actively been studied in the natural language processing (NLP) or vision domains, no prior work has applied the transformer model to the RF-based human pose estimation problem. The attention mechanism of the proposed algorithm is able to focus on the relevant time segments of the IR-UWB radar signals for detecting the human pose, eliminating the needs of converting radar signals to a voxelized 3D image. We have gathered a large dataset of IR-UWB radar signals labeled with 3D human skeletons, and shown that the proposed algorithm is able to detect human skeletons with a high accuracy.

Simultaneously-Collected Multimodal Lying Pose Dataset: Enabling In-Bed Human Pose Monitoring.

Computer vision field has achieved great success in interpreting semantic meanings from images, yet its algorithms can be brittle for tasks with adverse vision conditions and the ones suffering from data/label pair limitation. Among these tasks is in-bed human pose monitoring with significant value in many healthcare applications. In-bed pose monitoring in natural settings involves pose estimation in complete darkness or full occlusion. The lack of publicly available in-bed pose datasets hinders the applicability of many successful human pose estimation algorithms for this task. In this paper, we introduce our Simultaneously-collected multimodal Lying Pose (SLP) dataset, which includes in-bed pose images from 109 participants captured using multiple imaging modalities including RGB, long wave infrared (LWIR), depth, and pressure map. We also present a physical hyper parameter tuning strategy for ground truth pose label generation under adverse vision conditions. The SLP design is compatible with the mainstream human pose datasets; therefore, the state-of-the-art 2D pose estimation models can be trained effectively with the SLP data with promising performance as high as 95% at PCKh@0.5 on a single modality. The pose estimation performance of these models can be further improved by including additional modalities through the proposed collaborative scheme.

Identity-Preserved Human Posture Detection in Infrared Thermal Images: A Benchmark

Human pose estimation has a variety of real-life applications, including human action recognition, AI-powered personal trainers, robotics, motion capture and augmented reality, gaming, and video surveillance. However, most current human pose estimation systems are based on RGB images, which do not seriously take into account personal privacy. Although identity-preserved algorithms are very desirable when human pose estimation is applied to scenarios where personal privacy does matter, developing human pose estimation algorithms based on identity-preserved modalities, such as thermal images concerned here, is very challenging due to the limited amount of training data currently available and the fact that infrared thermal images, unlike RGB images, lack rich texture cues which makes annotating training data itself impractical. In this paper, we formulate a new task with privacy protection that lies between human detection and human pose estimation by introducing a benchmark for IPHPDT (i.e., Identity-Preserved Human Posture Detection in Thermal images). This task has a threefold novel purpose: the first is to establish an identity-preserved task with thermal images; the second is to achieve more information other than the location of persons as provided by human detection for more advanced computer vision applications; the third is to avoid difficulties in collecting well-annotated data for human pose estimation in thermal images. The presented IPHPDT dataset contains four types of human postures, consisting of 75,000 images well-annotated with axis-aligned bounding boxes and postures of the persons. Based on this well-annotated IPHPDT dataset and three state-of-the-art algorithms, i.e., YOLOF (short for You Only Look One-level Feature), YOLOX (short for Exceeding YOLO Series in 2021) and TOOD (short for Task-aligned One-stage Object Detection), we establish three baseline detectors, called IPH-YOLOF, IPH-YOLOX, and IPH-TOOD. In the experiments, three baseline detectors are used to recognize four infrared human postures, and the mean average precision can reach 70.4%. The results show that the three baseline detectors can effectively perform accurate posture detection on the IPHPDT dataset. By releasing IPHPDT, we expect to encourage more future studies into human posture detection in infrared thermal images and draw more attention to this challenging task.

Comparative Analysis of Skeleton-Based Human Pose Estimation

Human pose estimation (HPE) has become a prevalent research topic in computer vision. The technology can be applied in many areas, such as video surveillance, medical assistance, and sport motion analysis. Due to higher demand for HPE, many HPE libraries have been developed in the last 20 years. In the last 5 years, more and more skeleton-based HPE algorithms have been developed and packaged into libraries to provide ease of use for researchers. Hence, the performance of these libraries is important when researchers intend to integrate them into real-world applications for video surveillance, medical assistance, and sport motion analysis. However, a comprehensive performance comparison of these libraries has yet to be conducted. Therefore, this paper aims to investigate the strengths and weaknesses of four popular state-of-the-art skeleton-based HPE libraries for human pose detection, including OpenPose, PoseNet, MoveNet, and MediaPipe Pose. A comparative analysis of these libraries based on images and videos is presented in this paper. The percentage of detected joints (PDJ) was used as the evaluation metric in all comparative experiments to reveal the performance of the HPE libraries. MoveNet showed the best performance for detecting different human poses in static images and videos.

A 3D graph convolutional networks model for 2D skeleton‐based human action recognition

With the popularity of cameras, the application of action recognition is more and more extensive. After the emergence of RGB-D cameras and human pose estimation algorithms, human actions can be represented by a sequence of skeleton joints. Therefore, skeleton-based action recognition has been a research hotspot. In this paper, a novel 3D Graph Convolutional Network model (3D-GCN) with space-time attention mechanism for 2D skeleton data is proposed. Three-dimensional graph convolution is employed to extract spatiotemporal features of skeleton descriptor that is composed of joint coordinates, frame differences and angles. Meanwhile, different joints and different frames are given different attention to achieve action classification. A zebra crossing pedestrian dataset named ZCP is also provided, which simulates possible pedestrian actions on the zebra crossing in real scenes. Experimental evaluation is carried out on ZCP dataset and NTU RGB+D dataset. Experimental results show that our method is better than current 2D-based methods and is comparable with 3D methods.

"Stand-up straight!": human pose estimation to evaluate postural skills during orthopedic surgery simulations.

Surgery simulators can be used to learn technical and non-technical skills and, to analyse posture. Ergonomic skill can be automatically detected with a Human Pose Estimation algorithm to help improve the surgeon's work quality. The objective of this study was to analyse the postural behaviour of surgeons and identify expertise-dependent movements. Our hypothesis was that hesitation and the occurrence of surgical instruments interfering with movement (defined as interfering movements) decrease with expertise. Sixty surgeons with three expertise levels (novice, intermediate, and expert) were recruited. During a training session using an arthroscopic simulator, each participant's movements were video-recorded with an RGB camera. A modified OpenPose algorithm was used to detect the surgeon's joints. The detection frequency of each joint in a specific area was visualized with a heatmap-like approach and used to calculate a mobility score. This analysis allowed quantifying surgical movements. Overall, the mean mobility score was 0.823, 0.816, and 0.820 for novice, intermediate and expert surgeons, respectively. The mobility score alone was not enough to identify postural behaviour differences. A visual analysis of each participants' movements highlighted expertise-dependent interfering movements. Video-recording and analysis of surgeon's movements are a non-invasive approach to obtain quantitative and qualitative ergonomic information in order to provide feedback during training. Our findings suggest that the interfering movements do not decrease with expertise but differ in function of the surgeon's level.

Traditional African Dances Preservation Using Deep Learning Techniques

Human action recognition continues to evolve and improve through deep learning techniques. There have been studies with some success in the field of action recognition, but only a few of them have focused on traditional dance. This is because dance actions, especially in traditional African dance, are long and involve fast movements. This research proposes a novel framework that applies data science algorithms to the field of cultural preservation by applying various deep learning techniques to identify, classify, and model traditional African dances from videos. Traditional dances are an important part of African culture and heritage. Digital preservation of these dances in their multitude and form is a challenging problem. The dance dataset was constituted from freely available YouTube videos. Four traditional African dances were used for the dance classification process: Adowa, Swange, Bata, and Sinte dance. Five Convolutional Neural Network (CNN) models were used for the classification and achieved an accuracy between 93% and 98%. Additionally, human pose estimation algorithms were applied to Sinte dance. A model of Sinte dance that can be exported to other environments was obtained.

Lightweight human pose estimation algorithm based on polarized self-attention

In recent years, human pose estimation has been widely used in human-computer interaction, augmented reality, video surveillance, and many other fields, but the task of pose estimation still faces many challenges. To address the large number of parameters and complicated calculation in the current mainstream human pose estimation network, this paper proposes a lightweight pose estimation network (Lightweight Polarized Network, referred to as LPNet) based on a polarized self-attention mechanism. First, ghost convolution is used to reduce the number of parameters of the feature extraction network; second, by introducing the polarized self-attention module, the pixel-level regression task can be better solved, the lack of extracted features due to the decrease in the number of parameters can be reduced, and the accuracy of the regression of human keypoints can be improved; finally, a new coordinate decoding method is designed to reduce the error in the heatmap decoding process and improve the accuracy of keypoint regression. The method proposed in this paper was evaluated on the human keypoint detection datasets COCO and MPII, and compared with the current mainstream methods. The experimental results show that the proposed method greatly reduces the number of parameters of the model while ensuring a small loss in accuracy.

Organ-based estimation and minimization of clinician's X-ray dose.

Hybrid surgeries, allowing real-time visualization of patient inner anatomy, are possible through the use of intraoperative X-ray imaging. However, the intensive use of X-rays can have undesired consequences for the clinicians or the patient in the operating room (OR). In this paper, we provide a tool to visualize the X-rays and to optimally place protective shields in the hybrid operating room to reduce the clinician's dose according to their most sensitive body parts. We first acquire measurements in a hybrid operating room with dosimeters placed at different locations on a mannequin simulating a clinician. We demonstrate that a small displacement of a protective shield has significant consequences on the dose received by a clinician. Then, we reproduce the scene virtually and use Monte Carlo simulations to estimate the dose received by the clinician. Finally, we optimally place protective shields with a Nelder-Mead-based numerical optimization algorithm. The results show a high sensitivity of the clinician's dose to protective shield placement. Numerical optimization of the shields' placement can help to reduce the dose and show a decrease between 79 and 89% of the exposition when comparing no external shield protection and our optimal external shield position. Our work can help to raise awareness of the risks induced by X-rays during intraoperative surgery and reduce the dose received by the clinicians. In future work, our approach can be linked with human pose estimation algorithms to trace surgeons' moves, estimate dynamically the dose and summarize it in a surgical report, giving the dose for important organs.

Human joints estimation system of rescue robot with occlusion

The human body is often in a partially visible state during rescue at the disaster site. At the same time, when the human body is grasped, the camera located on the actuator at the end of the robotic arm cannot fully bring the human body into the field of view. The existing human pose estimation algorithms do not perform well in this situation, and a joint detection method that can estimate the complete human pose through partial human body information and can perform data refresh in real time is required. We use the occlusion part classification module to extract the weight vector, re-weight it and the joint features extracted from AlphaPose, and add human body geometric constraints when generating 3D human body joints to obtain more accurate 3D human body joint positions. Finally, the robot is successfully used to complete the human grasping experiment, which verifies the effectiveness of the algorithm.

Method for discovering spatial arm positions with depth sensor data at low-performance devices

A method of arm aiming direction estimation for low performance Internet of Things devices is proposed. It uses Human Pose Estimation (HPE) algorithms for retrieving human skeleton key points. Having these key points, arm aiming directions model is calculated. Two well-known HPE methods (PoseNet and OpenPose) are examined. These algorithms have been tested and compared by the average angle of error. The system includes a Raspberry Pi 4B single-board computer and an Intel RealSense D435i depth sensor. The developed approach may be utilized in “smart home” gesture control systems.

Design and Implementation of a Vision- and Grating-Sensor-Based Intelligent Unmanned Settlement System

In this paper, a new vision- and grating-sensor-based intelligent unmanned settlement (IUS) system is proposed for convenience stores to automatically recognize the shopping behavior of customers, record their identities, and generate invoices. First, we design a new IUS architecture, which includes a shelf module and exit module. To achieve automatic settlement for each customer, a shopping event detection method is proposed. In this method, a vision-based human pose estimation algorithm is used to detect a human form standing in front of a shelf. The hand actions of each customer are detected by a grating sensor, and an image recognition method based on a convolutional neural network (CNN) is applied to recognize the items in the hands of customers. To reduce the image annotation workload, we propose a semisupervised training method for the recognition network. Based on hand action detection and item recognition, a shopping event recognition method is designed for the system, and a facial image of the customer corresponding to each shopping behavior is captured. Finally, each detected shopping event is added to the invoice of the corresponding customer via a facial recognition method. To verify the effectiveness of the proposed IUS system, we have built a handheld item image dataset and a shopping event dataset for an unmanned convenience store. The experimental results show that the proposed system can accurately recognize shopping behaviors and generate invoices.

FastNet: Fast high-resolution network for human pose estimation

Aiming at the problem of developing efficient models for human pose estimation algorithms under computation-limited resources. In this paper, we proposed an efficient high-resolution network for human pose estimation, which named FastNet. First of all, The Lite-HRNet was constructed by using four parallel subnetworks. Each subnetwork contained multiple bottleneck block for feature extraction. The Lite-HRNet can effectively learn the high-resolution representation features of key points because of maintaining high-resolution. Secondly, instead of standard convolution, the asymmetric convolution was introduced to build an asymmetric bottleneck module. The asymmetric bottleneck module has different aspect ratios, and can be used to exact image features of keypoints with multi-scale characteristics and reduce the number of parameters. Thirdly, the waterfall module composed of multiple parallel convolutions were proposed to aggregates features with the same spatial size which can efficiently obtain delicate local representations. It retains rich spatial information and results in precise keypoint localization. Finally, the bottleneck blocks were replaced with asymmetric bottleneck modules in the third subnetwork of LiteHRNet. By which, The waterfall module is embedded into the structure of FastNet. Comprehensive experiments demonstrate that the proposed method achieves superior results on two benchmark datasets, MSCOCO and MPII. Moreover, FastNet demonstrates superior results on human pose estimation over popular lightweight networks.

Applications of Pose Estimation in Human Health and Performance across the Lifespan.

The emergence of pose estimation algorithms represents a potential paradigm shift in the study and assessment of human movement. Human pose estimation algorithms leverage advances in computer vision to track human movement automatically from simple videos recorded using common household devices with relatively low-cost cameras (e.g., smartphones, tablets, laptop computers). In our view, these technologies offer clear and exciting potential to make measurement of human movement substantially more accessible; for example, a clinician could perform a quantitative motor assessment directly in a patient’s home, a researcher without access to expensive motion capture equipment could analyze movement kinematics using a smartphone video, and a coach could evaluate player performance with video recordings directly from the field. In this review, we combine expertise and perspectives from physical therapy, speech-language pathology, movement science, and engineering to provide insight into applications of pose estimation in human health and performance. We focus specifically on applications in areas of human development, performance optimization, injury prevention, and motor assessment of persons with neurologic damage or disease. We review relevant literature, share interdisciplinary viewpoints on future applications of these technologies to improve human health and performance, and discuss perceived limitations.