Human Pose Estimation Research Articles

Image-Based Fitness Yoga Pose Recognition: Using Ensemble Learning and Multi-head Attention

With the increasing awareness of fitness, more and more people are choosing to participate in fitness activities. Yoga, as a form of exercise that improves both physical and mental health, is becoming increasingly popular worldwide. In order to assist yoga practitioners in more effective training through automated or semi automated systems, improve training effectiveness, assist professional athletes in training through intelligent recognition systems, correct movements, and improve athletic performance. This paper proposes a method that addresses the low accuracy issue of current yoga pose recognition algorithms by integrating multi-head attention mechanism and ensemble learning. Firstly, the Mixup algorithm is used to enhance yoga movement images. Subsequently, convolutional features are extracted from the images using the ResNet101 and VGGNet19 transfer learning models. Finally, the extracted convolutional features are combined and stacked using a multi-head attention mechanism. Model training, validation, and testing are performed using the Soft target cross-entropy loss function. Experimental results demonstrate that the proposed method achieves a training accuracy of 100%, a validation accuracy of 89.94%, a testing accuracy of 93.79%, and a detection speed of 297 frames per second. Overall, this method demonstrates high stability and robustness, providing a technological foundation for intelligent recognition of yoga poses.

3D human pose estimation with multi-hypotheses gated transformer

3D human pose estimation with multi-hypotheses gated transformer

Exploring event-based human pose estimation with 3D event representations

Human pose estimation is a fundamental and appealing task in computer vision. Although traditional cameras are commonly applied, their reliability decreases in scenarios under high dynamic range or heavy motion blur, where event cameras offer a robust solution. Predominant event-based methods accumulate events into frames, ignoring the asynchronous and high temporal resolution that is crucial for distinguishing distinct actions. To address this issue and to unlock the 3D potential of event information, we introduce two 3D event representations: the Rasterized Event Point Cloud (RasEPC) and the Decoupled Event Voxel (DEV). The RasEPC aggregates events within concise temporal slices at identical positions, preserving their 3D attributes along with statistical information, thereby significantly reducing memory and computational demands. Meanwhile, the DEV representation discretizes events into voxels and projects them across three orthogonal planes, utilizing decoupled event attention to retrieve 3D cues from the 2D planes. Furthermore, we develop and release EV-3DPW, a synthetic event-based dataset crafted to facilitate training and quantitative analysis in outdoor scenes. Our methods are tested on the DHP19 public dataset, MMHPSD dataset, and our EV-3DPW dataset, with further qualitative validation via a derived driving scene dataset EV-JAAD and an outdoor collection vehicle. Our code and dataset have been made publicly available at https://github.com/MasterHow/EventPointPose.

Hierarchical Keypoints Feature Alignment for Domain Adaptive Pose Estimation

Unsupervised Domain Adaptation (UDA) is a popular research topic in computer vision. One of the most common strategies in this field is to reduce domain shifts through global feature alignment. Unfortunately, we observe its failure on adaptive human pose estimation. From our analysis, we find out two major reasons: the extreme imbalance between keypoints and non-keypoints regions and the high diversity of human skeleton structures. To address these problems, we propose a simple yet effective approach named PoseDA. Our idea is to let the alignment focus on the feature of keypoint regions and enforce the model to learn domain-invariant representations for keypoint prediction. The key component of our PoseDA is a Hierarchical Feature Selection of Keypoints Regions (HFS) module, which consists of Coarse Feature Selection of Keypoints Regions (CFS) and Reliable Feature Selection of Keypoints Regions (RFS). By using HFS, we obtain reliable and compact keypoints features, allowing our model to achieve more effective feature alignment. Under three UDA scenarios, i.e. JTA→MPII, SURREAL→LSP, SURREAL→Human3.6, our PoseDA establishes new state-of-the-art performance. In particular, our PoseDA outperforms the previous best UDA methods by over 7% w.r.t. PCKh on JTA→MPII.

DM-HAP: Diffusion model for accurate hand pose prediction

Forecasting hand poses is a challenging task due to inherent uncertainties, occlusions, and inaccuracies in 3D pose estimation. Diffusion models provide a promising direction for predicting precise 3D hand poses under noisy conditions. In this work, we introduce Dual-diffusion, an innovative framework for the precise prediction of future hand poses. Our approach leverages the strengths of diffusion models by framing hand pose forecasting as a reverse diffusion process, effectively addressing the complexities of noisy hand joint movements and their subtleties. To enhance the learning of hand pose representations, we propose a unique neural architecture that simultaneously captures both local and global features. This is achieved through the deployment of Global and Local Diffusion (GLD) blocks within our network, which facilitate the exchange of information between local and global features. This diffusion-based interaction enables the integration of global hand actions and local finger actions, leading to a more powerful representation learning approach. We evaluate the effectiveness of our Dual-diffusion method on three public 3D hand pose estimation datasets (MSRA, F-PHAB, and BigHand2.2M), and our approach outperforms previous methods.

A multi-stage approach for desired part grasping under complex backgrounds in human-robot collaborative assembly

Human-robot collaborative assembly can leverage the unique capabilities of humans and robots to provide a flexible and efficient way for complex tasks. In this context, robots are expected to be endowed with the perception ability to collaborate with humans. For flexible processes, robotic grasping for a desired assembly part from variable multi-parts is essential but remains challenging, especially in situations of complex backgrounds including disordered placement, occlusion, similarity, and large-scale differences in size. To improve the perception and decision-making abilities of robots in collaborative assembly, firstly, a multi-stage approach integrating 2D part recognition and 6D pose estimation with perspective-focusing is proposed for desired part grasping under complex backgrounds. Secondly, to efficiently and accurately recognize a desired part from variable multi-parts under complex backgrounds, based on RGB data, an enhanced detector is adopted, which lays the foundation for the subsequent pose estimation. Thirdly, based on point cloud data, a pose estimation method based on perspective-focusing is proposed to enhance the success rate and efficiency of pose acquisition. Additionally, a modular autonomous robotic grasping system is designed. Finally, taking decelerator assembly as a case, the proposed approaches are comprehensively validated and compared on a real collaborative robot platform. Under complex backgrounds with occlusion, our approach shows stable performance with an average grasping success rate of 90.0 % and an average cycle time of 6.4175 s. Our work is expected to improve robots’ perception and decision-making capabilities while further reducing the cycle time of collaborative assembly tasks.

DeepArUco++: Improved detection of square fiducial markers in challenging lighting conditions

Fiducial markers are a computer vision tool used for object pose estimation and detection. These markers are highly useful in fields such as industry, medicine and logistics. However, optimal lighting conditions are not always available, and other factors such as blur or sensor noise can affect image quality. Classical computer vision techniques that precisely locate and decode fiducial markers often fail under difficult illumination conditions (e.g. extreme variations of lighting within the same frame). Hence, we propose DeepArUco++, a deep learning-based framework that leverages the robustness of Convolutional Neural Networks to perform marker detection and decoding in challenging lighting conditions. The framework is based on a pipeline using different Neural Network models at each step, namely marker detection, corner refinement and marker decoding. Additionally, we propose a simple method for generating synthetic data for training the different models that compose the proposed pipeline, and we present a second, real-life dataset of ArUco markers in challenging lighting conditions used to evaluate our system. The developed method outperforms other state-of-the-art methods in such tasks and remains competitive even when testing on the datasets used to develop those methods. Code available in GitHub: https://github.com/AVAuco/deeparuco/.

Improving Human Pose Estimation with Integrated Dual Self-Attention Mechanism in High-Resolution Network

Human Pose Estimation (HPE) in computer vision (CV) has garnered significant attention due to its diverse applications. Deep convolutional neural networks (CNNs) may be solutions for addressing this challenge, but still face several critical issues. Many existing models employ serial convolution with pooling, leading to low-resolution outputs that are suboptimal for the precise localisation required in HPE. They often prioritise local feature learning, overlooking crucial contextual relationships between key-points. This work addresses these challenges by proposing a novel approach for enhancing HPE. Firstly, the paper evaluates the high-resolution network (HRNet) and its comparative advantages over other CNN architectures. Secondly, it introduces a dual self-attention (DSA) mechanism designed to enhance the model’s global awareness, thereby enriching feature maps with contextual information. Thirdly, it integrates the DSA mechanism into HRNet, crafting DSA-HRNet. The model performance was tested on the COCO Val 2017 validation dataset, showing improvements of 2.3% in mean average precision (mAP), 3% in AP at 50 (AP50), and 2.7% in AP at 75 (AP75). Finally, the work includes an investigation into the effectiveness of the DSA mechanism within the HRNet framework, through a series of experiments, showing this work offers a streamlined and effective solution for improving HPE.

Dense depth alignment for human pose and shape estimation

Dense depth alignment for human pose and shape estimation

Full-Body Pose Estimation of Humanoid Robots Using Head-Worn Cameras for Digital Human-Augmented Robotic Telepresence

We envision a telepresence system that enhances remote work by facilitating both physical and immersive visual interactions between individuals. However, during robot teleoperation, communication often lacks realism, as users see the robot’s body rather than the remote individual. To address this, we propose a method for overlaying a digital human model onto a humanoid robot using XR visualization, enabling an immersive 3D telepresence experience. Our approach employs a learning-based method to estimate the 2D poses of the humanoid robot from head-worn stereo views, leveraging a newly collected dataset of full-body poses for humanoid robots. The stereo 2D poses and sparse inertial measurements from the remote operator are optimized to compute 3D poses over time. The digital human is localized from the perspective of a continuously moving observer, utilizing the estimated 3D pose of the humanoid robot. Our moving camera-based pose estimation method does not rely on any markers or external knowledge of the robot’s status, effectively overcoming challenges such as marker occlusion, calibration issues, and dependencies on headset tracking errors. We demonstrate the system in a remote physical training scenario, achieving real-time performance at 40 fps, which enables simultaneous immersive and physical interactions. Experimental results show that our learning-based 3D pose estimation method, which operates without prior knowledge of the robot, significantly outperforms alternative approaches requiring the robot’s global pose, particularly during rapid headset movements, achieving markerless digital human augmentation from head-worn views.

RETRACTED ARTICLEResearch Progress on 2D Human Pose Estimation Based on Deep Learning

Human pose estimation (HPE) is a significant area of study within computer vision and artificial intelligence, which locates the position and pose of human joints through images or videos. HPE has advanced progress in computer vision and deep learning fields at the theoretical level, improved the efficiency and accuracy of neural networks, and supported the development of natural and efficient human-computer interaction systems. At the application level, HPE has extensive value in many fields, including security monitoring, motion analysis and health management, VR&AR, robotics technology, film and television production and animation, intelligent driving and assisted driving. This paper divides 2D HPE tasks into three categories: two-stage (top-down and bottom-up), single-stage, and other improvement strategies (Transformer-based methods and post-processing optimization methods). Although deep learning-based pose estimation algorithms have achieved remarkable results, they still face challenges, such as prediction in complex scenes, crowd and joint occlusion problems, and lightweight design of models. Future work should pay more attention to these challenges and extend pose estimation technology from 2D to 3D contexts.

Clinical Significance of Pose Estimation Methods Compared with Radiographic Parameters in Adolescent Patients with Idiopathic Scoliosis.

Human pose estimation, a computer vision technique that identifies body parts and constructs human body representations from images and videos, has recently demonstrated high performance through deep learning. However, its potential application in clinical photography remains underexplored. This study aimed to establish photographic parameters for patients with adolescent idiopathic scoliosis (AIS) using pose estimation and to determine correlations between these photographic parameters and corresponding radiographic measures. We conducted a study involving 42 patients with AIS who had undergone spinal correction surgery and conservative treatment. Preoperative photographs were captured using an iPhone 13 Pro mounted on a tripod positioned at the head of an X-ray tube. From the outputs of pose estimation, we derived five photographic parameters and subsequently conducted a statistical analysis to assess their correlations with relevant conventional radiographic parameters. In the sagittal plane, we identified significant correlations between photographic and radiographic parameters measuring trunk tilt angles. In the coronal plane, significant correlations were found between photographic parameters measuring shoulder height and trunk tilt and corresponding radiographic measurements. The results suggest that pose estimation, achievable with common mobile devices, offers potential for AIS screening, early detection, and continuous posture monitoring, effectively mitigating the need for X-ray radiation exposure. Level of Evidence: 3.

A dual-channel network based on occlusion feature compensation for human pose estimation

Human pose estimation is an important technique in computer vision. Existing methods perform well in ideal environments, but there is room for improvement in occluded environments. The specific reasons are that the ambiguity of the features in the occlusion area makes the network pay insufficient attention to it, and the inadequate expressive ability of the features in the occlusion part cannot describe the true keypoint features. To address the occlusion issue, we propose a dual-channel network based on occlusion feature compensation. The dual channels are occlusion area enhancement channel based on convolution and occlusion feature compensation channel based on graph convolution, respectively. In the convolution channel, we propose an occlusion handling enhanced attention mechanism (OHE-attention) to improve the attention to the occlusion area. In the graph convolution channel, we propose a node feature compensation module that eliminates the obstacle features and integrates the shared and private attributes of the keypoints to improve the expressive ability of the node features. We conduct experiments on the COCO2017 dataset, COCO-Wholebody dataset, and CrowdPose dataset, achieving accuracy of 78.7%, 66.4%, and 77.9%, respectively. In addition, a series of ablation experiments and visualization demonstrations verify the performance of the dual-channel network in occluded environments.

Poses of People in Art: A Data Set for Human Pose Estimation in Digital Art History

Poses of People in Art: A Data Set for Human Pose Estimation in Digital Art History

Monocular 3D Multi-Person Pose Estimation for On-Site Joint Flexion Assessment: A Case of Extreme Knee Flexion Detection.

Work-related musculoskeletal disorders (WMSDs) represent a significant health challenge for workers in construction environments, often arising from prolonged exposure to ergonomic risks associated with manual labor, awkward postures, and repetitive motions. These conditions not only lead to diminished worker productivity but also incur substantial economic costs for employers and healthcare systems alike. Thus, there is an urgent need for effective tools to assess and mitigate these ergonomic risks. This study proposes a novel monocular 3D multi-person pose estimation method designed to enhance ergonomic risk assessments in construction environments. Leveraging advanced computer vision and deep learning techniques, this approach accurately captures and analyzes the spatial dynamics of workers' postures, with a focus on detecting extreme knee flexion, a critical indicator of work-related musculoskeletal disorders (WMSDs). A pilot study conducted on an actual construction site demonstrated the method's feasibility and effectiveness, achieving an accurate detection rate for extreme flexion incidents that closely aligned with supervisory observations and worker self-reports. The proposed monocular approach enables universal applicability and enhances ergonomic analysis through 3D pose estimation and group pose recognition for timely interventions. Future efforts will focus on improving robustness and integration with health monitoring to reduce WMSDs and promote worker health.

Research on the Cable-to-Terminal Connection Recognition Based on the YOLOv8-Pose Estimation Model

Substations, as critical nodes for power transmission and distribution, play a pivotal role in ensuring the stability and security of the entire power grid. With the ever-increasing demand for electricity and the growing complexity of grid structures, traditional manual inspection methods for substations can no longer meet the requirements for efficient and safe operation and maintenance. The advent of automated inspection systems has brought revolutionary changes to the power industry. These systems utilize advanced sensor technology, image processing techniques, and artificial intelligence algorithms to achieve real-time monitoring and fault diagnosis of substation equipment. Among these, the recognition of cable-to-terminal connection relationships is a key task for automated inspection systems, and its accuracy directly impacts the system’s diagnostic capabilities and fault prevention levels. However, traditional methods face numerous limitations when dealing with complex power environments, such as inadequate recognition performance under conditions of significant perspective angles and geometric distortions. This paper proposes a cable-to-terminal connection relationship recognition method based on the YOLOv8-pose model. The YOLOv8-pose model combines object detection and pose estimation techniques, significantly improving detection accuracy and real-time performance in environments with small targets and dense occlusions through optimized feature extraction algorithms and enhanced receptive fields. The model achieves an average inference time of 74 milliseconds on the test set, with an accuracy of 92.8%, a recall rate of 91.5%, and an average precision mean of 90.2%. Experimental results demonstrate that the YOLOv8-pose model performs excellently under different angles and complex backgrounds, accurately identifying the connection relationships between terminals and cables, providing reliable technical support for automated substation inspection systems. This research offers an innovative solution for automated substation inspection systems, with significant application prospects.

ESCAPE: Energy-based Selective Adaptive Correction for Out-of-distribution 3D Human Pose Estimation

Despite recent advances in human pose estimation (HPE), poor generalization to out-of-distribution (OOD) data remains a difficult problem. While previous works have proposed Test-Time Adaptation (TTA) to bridge the train-test domain gap by refining network parameters at inference, the absence of ground-truth annotations makes it highly challenging and existing methods typically increase inference times by one or more orders of magnitude. We observe that (1) not every test time sample is OOD, and (2) HPE errors are significantly larger on distal keypoints (wrist, ankle). To this end, we propose ESCAPE: a lightweight correction and selective adaptation framework which applies a fast, forward-pass correction on most data while reserving costly TTA for OOD data. The free energy function is introduced to separate OOD samples from incoming data and a correction network is trained to estimate the errors of pretrained backbone HPE predictions on the distal keypoints. For OOD samples, we propose a novel self-consistency adaptation loss to update the correction network by leveraging the constraining relationship between distal keypoints and proximal keypoints (shoulders, hips), via a second “reverse” network. ESCAPE improves the distal MPJPE of five popular HPE models by up to 7% on unseen data, achieves state-of-the-art results on two popular HPE benchmarks, and is significantly faster than existing adaptation methods.

Robotic grasping method with 6D pose estimation and point cloud fusion

Robotic grasping method with 6D pose estimation and point cloud fusion

FusionNetV2: Explicit Enhancement of Edge Features for 6D Object Pose Estimation

FusionNet is a hybrid model that incorporates convolutional neural networks and Transformers, achieving state-of-the-art performance in 6D object pose estimation while significantly reducing the number of model parameters. Our study reveals that FusionNet has local and global attention mechanisms for enhancing deep features in two paths and the attention mechanisms play a role in implicitly enhancing features around object edges. We found that enhancing the features around object edges was the main reason for the performance improvement in 6D object pose estimation. Therefore, in this study, we attempt to enhance the features around object edges explicitly and intuitively. To this end, an edge boosting block (EBB) is introduced that replaces the attention blocks responsible for local attention in FusionNet. EBB is lightweight and can be directly applied to FusionNet with minimal modifications. EBB significantly improved the performance of FusionNet in 6D object pose estimation in experiments on the LINEMOD dataset.

Enhancing Inter-AUV Perception: Adaptive 6-DOF Pose Estimation with Synthetic Images for AUV Swarm Sensing

The capabilities of AUV mutual perception and localization are crucial for the development of AUV swarm systems. We propose the AUV6D model, a synthetic image-based approach to enhance inter-AUV perception through 6D pose estimation. Due to the challenge of acquiring accurate 6D pose data, a dataset of simulated underwater images with precise pose labels was generated using Unity3D. Mask-CycleGAN technology was introduced to transform these simulated images into realistic synthetic images, addressing the scarcity of available underwater data. Furthermore, the Color Intermediate Domain Mapping strategy is proposed to ensure alignment across different image styles at pixel and feature levels, enhancing the adaptability of the pose estimation model. Additionally, the Salient Keypoint Vector Voting Mechanism was developed to improve the accuracy and robustness of underwater pose estimation, enabling precise localization even in the presence of occlusions. The experimental results demonstrated that our AUV6D model achieved millimeter-level localization precision and pose estimation errors within five degrees, showing exceptional performance in complex underwater environments. Navigation experiments with two AUVs further verified the model’s reliability for mutual 6D pose estimation. This research provides substantial technical support for more complex and precise collaborative operations for AUV swarms in the future.