Human Pose Estimation Research Articles

3D skeleton aware driver behavior recognition framework for autonomous driving system

The recognition of the driver’s behaviors inside an autonomous vehicle can effectively address emergency handling in autonomous driving and is crucial for ensuring the driver’s safety. Driver behavior recognition is a challenging task due to factors such as variations, diversities, complexities, and strong interferences in behaviors. In this paper, to realize the application in the autonomous driving scenes, a novel 3D skeleton aware behavior recognition framework is proposed to recognize various driver behaviors in autonomous driving systems. First, a 3D human pose estimation network (Pose-GTFNet) with temporal Transformer and spatial graph convolutional network (GCN) is designed to infer 3D human poses from 2D pose sequences. Second, based on the obtained 3D human pose sequences, a behavior recognition network (Beh-MSFNet) with multi-skeleton feature fusion is designed to recognize driver behaviors. In the experiments, the Pose-GTFNet and Beh-MSFNet can get the best performance compared with most state-of-the-art (SOTA) methods on the Human3.6M human pose dataset, JHMDB and SHREC action recognition dataset, respectively. In addition, the proposed driver behavior recognition framework can achieve SOTA performance on the Drive&Act and Driver-Skeleton driver behavior datasets.

BodyFlow: An Open-Source Library for Multimodal Human Activity Recognition

Human activity recognition is a critical task for various applications across healthcare, sports, security, gaming, and other fields. This paper presents BodyFlow, a comprehensive library that seamlessly integrates human pose estimation and multiple-person estimation and tracking, along with activity recognition modules. BodyFlow enables users to effortlessly identify common activities and 2D/3D body joints from input sources such as videos, image sets, or webcams. Additionally, the library can simultaneously process inertial sensor data, offering users the flexibility to choose their preferred input, thus facilitating multimodal human activity recognition. BodyFlow incorporates state-of-the-art algorithms for 2D and 3D pose estimation and three distinct models for human activity recognition.

DenseFusion-DA2: End-to-End Pose-Estimation Network Based on RGB-D Sensors and Multi-Channel Attention Mechanisms.

Notably, 6D pose estimation is a critical technology that enables robotics to perceive and interact with their operational environment. However, occlusion causes a loss of local features, which, in turn, restricts the estimation accuracy. To address these challenges, this paper proposes an end-to-end pose-estimation network based on a multi-channel attention mechanism, DA2Net. Firstly, a multi-channel attention mechanism, designated as "DA2Net", was devised using A2-Nets as its foundation. This mechanism is constructed in two steps. In the first step, the essential characteristics are extracted from the global feature space through the second-order attention pool. In the second step, a feature map is generated by the integration of position and channel attention. Subsequently, the extracted key features are assigned to each position of the feature map, enhancing both the feature representation capacity and the overall performance. Secondly, the designed attention mechanism is introduced into both the feature fusion and pose iterative refinement networks to enhance the network's capacity to acquire local features thus improving its overall performance. The experimental results demonstrated that the estimation accuracy of DenseFusion-DA2 on the LineMOD dataset was approximately 3.4% higher than that of DenseFusion. Furthermore, the estimation accuracy surpassed that of PoseCNN, PVNet, SSD6D, and PointFusion by 8.3%, 11.1%, 20.3%, and 23.8%, respectively. The estimation accuracy also shows a significant advantage on the Occluded LineMOD and HR-Vision datasets. This research not only presents a more efficient solution for robot perception but also introduces novel ideas and methods for technological advancements and applications in related fields.

Human Pose Classification Based on Pose Long Short-Term Memory

Objective: At present, pose classification research is a hot topic in the field of computer vision, playing a crucial role in surveillance security, and motion data mining. The research serves as a crucial follow-up to three dimensional (3D) pose estimation and makes a significant contribution to the advancement of artificial intelligence. Pose classification is a complex problem involving a large amount of complicated data, making its digital modeling and further processing challenging. Methods: This article proposes the Pose long short - term memory (LSTM) method to classify 3D poses reconstructed from 2D image sequences. Result: Compared with traditional methods, the experimental tests show that the performance of the Pose LSTM is more superior in all aspects. Conclusion: Pose LSTM is particularly effective for recognizing continuous action poses because they can capture temporal dependencies in sequential data. In the context of pose recognition, it can analyze the sequence of poses over time and understand how these poses transition from one to another, making it possible to accurately classify or predict actions that unfold over a series of frames.

3D Head Pose Estimation via Normal Maps: A Generalized Solution for Depth Image, Point Cloud, and Mesh

Head pose estimation plays a crucial role in various applications, including human–machine interaction, autonomous driving systems, and 3D reconstruction. Current methods address the problem primarily from a 2D perspective, which limits the efficient utilization of 3D information. Herein, a novel approach, called pose orientation‐aware network (POANet), which leverages normal maps for orientation information embedding, providing abundant and robust head pose information, is introduced. POANet incorporates the axial signal perception module and the rotation matrix perception module, these lightweight modules make the approach achieve state‐of‐the‐art (SOTA) performance with few computational costs. This method can directly analyze various topological 3D data without extensive preprocessing. For depth images, POANet outperforms existing methods on the Biwi Kinect head pose dataset, reducing the mean absolute error (MAE) by ≈30% compared to the SOTA methods. POANet is the first method to perform rigid head registration in a landmark‐free manner. It also incorporates few‐shot learning capabilities and achieves an MAE of about on the Headspace dataset. These features make POANet a superior alternative to traditional generalized Procrustes analysis for mesh data processing, offering enhanced convenience for human phenotype studies.

Automated yoga pose recognition using enhanced chicken swarm optimization with deep learning

Automated yoga pose recognition using enhanced chicken swarm optimization with deep learning

Dual Leap Motion Controller 2: A Robust Dataset for Multi-view Hand Pose Recognition

This paper presents Multi-view Leap2 Hand Pose Dataset (ML2HP Dataset), a new dataset for hand pose recognition, captured using a multi-view recording setup with two Leap Motion Controller 2 devices. This dataset encompasses a diverse range of hand poses, recorded from different angles to ensure comprehensive coverage. The dataset includes real images with the associated precise and automatic hand properties, such as landmark coordinates, velocities, orientations, and finger widths. This dataset has been meticulously designed and curated to maintain a balance in terms of subjects, hand poses, and the usage of right or left hand, ensuring fairness and parity. The content includes 714,000 instances from 21 subjects of 17 different hand poses (including real images and 247 associated hand properties). The multi-view setup is necessary to mitigate hand occlusion phenomena, ensuring continuous tracking and pose estimation required in real human-computer interaction applications. This dataset contributes to advancing the field of multimodal hand pose recognition by providing a valuable resource for developing advanced artificial intelligence human computer interfaces.

Data augmentation in human-centric vision

AbstractThis survey presents a comprehensive analysis of data augmentation techniques in human-centric vision tasks, a first of its kind in the field. It delves into a wide range of research areas including person ReID, human parsing, human pose estimation, and pedestrian detection, addressing the significant challenges posed by overfitting and limited training data in these domains. Our work categorizes data augmentation methods into two main types: data generation and data perturbation. Data generation covers techniques like graphic engine-based generation, generative model-based generation, and data recombination, while data perturbation is divided into image-level and human-level perturbations. Each method is tailored to the unique requirements of human-centric tasks, with some applicable across multiple areas. Our contributions include an extensive literature review, providing deep insights into the influence of these augmentation techniques in human-centric vision and highlighting the nuances of each method. We also discuss open issues and future directions, such as the integration of advanced generative models like Latent Diffusion Models, for creating more realistic and diverse training data. This survey not only encapsulates the current state of data augmentation in human-centric vision but also charts a course for future research, aiming to develop more robust, accurate, and efficient human-centric vision systems.

Image-Based Fitness Yoga Pose Recognition: Using Ensemble Learning and Multi-head Attention

With the increasing awareness of fitness, more and more people are choosing to participate in fitness activities. Yoga, as a form of exercise that improves both physical and mental health, is becoming increasingly popular worldwide. In order to assist yoga practitioners in more effective training through automated or semi automated systems, improve training effectiveness, assist professional athletes in training through intelligent recognition systems, correct movements, and improve athletic performance. This paper proposes a method that addresses the low accuracy issue of current yoga pose recognition algorithms by integrating multi-head attention mechanism and ensemble learning. Firstly, the Mixup algorithm is used to enhance yoga movement images. Subsequently, convolutional features are extracted from the images using the ResNet101 and VGGNet19 transfer learning models. Finally, the extracted convolutional features are combined and stacked using a multi-head attention mechanism. Model training, validation, and testing are performed using the Soft target cross-entropy loss function. Experimental results demonstrate that the proposed method achieves a training accuracy of 100%, a validation accuracy of 89.94%, a testing accuracy of 93.79%, and a detection speed of 297 frames per second. Overall, this method demonstrates high stability and robustness, providing a technological foundation for intelligent recognition of yoga poses.

3D human pose estimation with multi-hypotheses gated transformer

3D human pose estimation with multi-hypotheses gated transformer

Exploring event-based human pose estimation with 3D event representations

Human pose estimation is a fundamental and appealing task in computer vision. Although traditional cameras are commonly applied, their reliability decreases in scenarios under high dynamic range or heavy motion blur, where event cameras offer a robust solution. Predominant event-based methods accumulate events into frames, ignoring the asynchronous and high temporal resolution that is crucial for distinguishing distinct actions. To address this issue and to unlock the 3D potential of event information, we introduce two 3D event representations: the Rasterized Event Point Cloud (RasEPC) and the Decoupled Event Voxel (DEV). The RasEPC aggregates events within concise temporal slices at identical positions, preserving their 3D attributes along with statistical information, thereby significantly reducing memory and computational demands. Meanwhile, the DEV representation discretizes events into voxels and projects them across three orthogonal planes, utilizing decoupled event attention to retrieve 3D cues from the 2D planes. Furthermore, we develop and release EV-3DPW, a synthetic event-based dataset crafted to facilitate training and quantitative analysis in outdoor scenes. Our methods are tested on the DHP19 public dataset, MMHPSD dataset, and our EV-3DPW dataset, with further qualitative validation via a derived driving scene dataset EV-JAAD and an outdoor collection vehicle. Our code and dataset have been made publicly available at https://github.com/MasterHow/EventPointPose.

Hierarchical Keypoints Feature Alignment for Domain Adaptive Pose Estimation

Unsupervised Domain Adaptation (UDA) is a popular research topic in computer vision. One of the most common strategies in this field is to reduce domain shifts through global feature alignment. Unfortunately, we observe its failure on adaptive human pose estimation. From our analysis, we find out two major reasons: the extreme imbalance between keypoints and non-keypoints regions and the high diversity of human skeleton structures. To address these problems, we propose a simple yet effective approach named PoseDA. Our idea is to let the alignment focus on the feature of keypoint regions and enforce the model to learn domain-invariant representations for keypoint prediction. The key component of our PoseDA is a Hierarchical Feature Selection of Keypoints Regions (HFS) module, which consists of Coarse Feature Selection of Keypoints Regions (CFS) and Reliable Feature Selection of Keypoints Regions (RFS). By using HFS, we obtain reliable and compact keypoints features, allowing our model to achieve more effective feature alignment. Under three UDA scenarios, i.e. JTA→MPII, SURREAL→LSP, SURREAL→Human3.6, our PoseDA establishes new state-of-the-art performance. In particular, our PoseDA outperforms the previous best UDA methods by over 7% w.r.t. PCKh on JTA→MPII.

DM-HAP: Diffusion model for accurate hand pose prediction

Forecasting hand poses is a challenging task due to inherent uncertainties, occlusions, and inaccuracies in 3D pose estimation. Diffusion models provide a promising direction for predicting precise 3D hand poses under noisy conditions. In this work, we introduce Dual-diffusion, an innovative framework for the precise prediction of future hand poses. Our approach leverages the strengths of diffusion models by framing hand pose forecasting as a reverse diffusion process, effectively addressing the complexities of noisy hand joint movements and their subtleties. To enhance the learning of hand pose representations, we propose a unique neural architecture that simultaneously captures both local and global features. This is achieved through the deployment of Global and Local Diffusion (GLD) blocks within our network, which facilitate the exchange of information between local and global features. This diffusion-based interaction enables the integration of global hand actions and local finger actions, leading to a more powerful representation learning approach. We evaluate the effectiveness of our Dual-diffusion method on three public 3D hand pose estimation datasets (MSRA, F-PHAB, and BigHand2.2M), and our approach outperforms previous methods.

A multi-stage approach for desired part grasping under complex backgrounds in human-robot collaborative assembly

Human-robot collaborative assembly can leverage the unique capabilities of humans and robots to provide a flexible and efficient way for complex tasks. In this context, robots are expected to be endowed with the perception ability to collaborate with humans. For flexible processes, robotic grasping for a desired assembly part from variable multi-parts is essential but remains challenging, especially in situations of complex backgrounds including disordered placement, occlusion, similarity, and large-scale differences in size. To improve the perception and decision-making abilities of robots in collaborative assembly, firstly, a multi-stage approach integrating 2D part recognition and 6D pose estimation with perspective-focusing is proposed for desired part grasping under complex backgrounds. Secondly, to efficiently and accurately recognize a desired part from variable multi-parts under complex backgrounds, based on RGB data, an enhanced detector is adopted, which lays the foundation for the subsequent pose estimation. Thirdly, based on point cloud data, a pose estimation method based on perspective-focusing is proposed to enhance the success rate and efficiency of pose acquisition. Additionally, a modular autonomous robotic grasping system is designed. Finally, taking decelerator assembly as a case, the proposed approaches are comprehensively validated and compared on a real collaborative robot platform. Under complex backgrounds with occlusion, our approach shows stable performance with an average grasping success rate of 90.0 % and an average cycle time of 6.4175 s. Our work is expected to improve robots’ perception and decision-making capabilities while further reducing the cycle time of collaborative assembly tasks.

DeepArUco++: Improved detection of square fiducial markers in challenging lighting conditions

Fiducial markers are a computer vision tool used for object pose estimation and detection. These markers are highly useful in fields such as industry, medicine and logistics. However, optimal lighting conditions are not always available, and other factors such as blur or sensor noise can affect image quality. Classical computer vision techniques that precisely locate and decode fiducial markers often fail under difficult illumination conditions (e.g. extreme variations of lighting within the same frame). Hence, we propose DeepArUco++, a deep learning-based framework that leverages the robustness of Convolutional Neural Networks to perform marker detection and decoding in challenging lighting conditions. The framework is based on a pipeline using different Neural Network models at each step, namely marker detection, corner refinement and marker decoding. Additionally, we propose a simple method for generating synthetic data for training the different models that compose the proposed pipeline, and we present a second, real-life dataset of ArUco markers in challenging lighting conditions used to evaluate our system. The developed method outperforms other state-of-the-art methods in such tasks and remains competitive even when testing on the datasets used to develop those methods. Code available in GitHub: https://github.com/AVAuco/deeparuco/.

Improving Human Pose Estimation with Integrated Dual Self-Attention Mechanism in High-Resolution Network

Human Pose Estimation (HPE) in computer vision (CV) has garnered significant attention due to its diverse applications. Deep convolutional neural networks (CNNs) may be solutions for addressing this challenge, but still face several critical issues. Many existing models employ serial convolution with pooling, leading to low-resolution outputs that are suboptimal for the precise localisation required in HPE. They often prioritise local feature learning, overlooking crucial contextual relationships between key-points. This work addresses these challenges by proposing a novel approach for enhancing HPE. Firstly, the paper evaluates the high-resolution network (HRNet) and its comparative advantages over other CNN architectures. Secondly, it introduces a dual self-attention (DSA) mechanism designed to enhance the model’s global awareness, thereby enriching feature maps with contextual information. Thirdly, it integrates the DSA mechanism into HRNet, crafting DSA-HRNet. The model performance was tested on the COCO Val 2017 validation dataset, showing improvements of 2.3% in mean average precision (mAP), 3% in AP at 50 (AP50), and 2.7% in AP at 75 (AP75). Finally, the work includes an investigation into the effectiveness of the DSA mechanism within the HRNet framework, through a series of experiments, showing this work offers a streamlined and effective solution for improving HPE.

Dense depth alignment for human pose and shape estimation

Dense depth alignment for human pose and shape estimation

Full-Body Pose Estimation of Humanoid Robots Using Head-Worn Cameras for Digital Human-Augmented Robotic Telepresence

We envision a telepresence system that enhances remote work by facilitating both physical and immersive visual interactions between individuals. However, during robot teleoperation, communication often lacks realism, as users see the robot’s body rather than the remote individual. To address this, we propose a method for overlaying a digital human model onto a humanoid robot using XR visualization, enabling an immersive 3D telepresence experience. Our approach employs a learning-based method to estimate the 2D poses of the humanoid robot from head-worn stereo views, leveraging a newly collected dataset of full-body poses for humanoid robots. The stereo 2D poses and sparse inertial measurements from the remote operator are optimized to compute 3D poses over time. The digital human is localized from the perspective of a continuously moving observer, utilizing the estimated 3D pose of the humanoid robot. Our moving camera-based pose estimation method does not rely on any markers or external knowledge of the robot’s status, effectively overcoming challenges such as marker occlusion, calibration issues, and dependencies on headset tracking errors. We demonstrate the system in a remote physical training scenario, achieving real-time performance at 40 fps, which enables simultaneous immersive and physical interactions. Experimental results show that our learning-based 3D pose estimation method, which operates without prior knowledge of the robot, significantly outperforms alternative approaches requiring the robot’s global pose, particularly during rapid headset movements, achieving markerless digital human augmentation from head-worn views.

RETRACTED ARTICLEResearch Progress on 2D Human Pose Estimation Based on Deep Learning

Human pose estimation (HPE) is a significant area of study within computer vision and artificial intelligence, which locates the position and pose of human joints through images or videos. HPE has advanced progress in computer vision and deep learning fields at the theoretical level, improved the efficiency and accuracy of neural networks, and supported the development of natural and efficient human-computer interaction systems. At the application level, HPE has extensive value in many fields, including security monitoring, motion analysis and health management, VR&AR, robotics technology, film and television production and animation, intelligent driving and assisted driving. This paper divides 2D HPE tasks into three categories: two-stage (top-down and bottom-up), single-stage, and other improvement strategies (Transformer-based methods and post-processing optimization methods). Although deep learning-based pose estimation algorithms have achieved remarkable results, they still face challenges, such as prediction in complex scenes, crowd and joint occlusion problems, and lightweight design of models. Future work should pay more attention to these challenges and extend pose estimation technology from 2D to 3D contexts.

Clinical Significance of Pose Estimation Methods Compared with Radiographic Parameters in Adolescent Patients with Idiopathic Scoliosis.

Human pose estimation, a computer vision technique that identifies body parts and constructs human body representations from images and videos, has recently demonstrated high performance through deep learning. However, its potential application in clinical photography remains underexplored. This study aimed to establish photographic parameters for patients with adolescent idiopathic scoliosis (AIS) using pose estimation and to determine correlations between these photographic parameters and corresponding radiographic measures. We conducted a study involving 42 patients with AIS who had undergone spinal correction surgery and conservative treatment. Preoperative photographs were captured using an iPhone 13 Pro mounted on a tripod positioned at the head of an X-ray tube. From the outputs of pose estimation, we derived five photographic parameters and subsequently conducted a statistical analysis to assess their correlations with relevant conventional radiographic parameters. In the sagittal plane, we identified significant correlations between photographic and radiographic parameters measuring trunk tilt angles. In the coronal plane, significant correlations were found between photographic parameters measuring shoulder height and trunk tilt and corresponding radiographic measurements. The results suggest that pose estimation, achievable with common mobile devices, offers potential for AIS screening, early detection, and continuous posture monitoring, effectively mitigating the need for X-ray radiation exposure. Level of Evidence: 3.