Human Pose Estimation Research Articles

Transfer-Learning-Based Gesture and Pose Recognition System for Human–Robot Interaction: An Internet of Things Application

Transfer-Learning-Based Gesture and Pose Recognition System for Human–Robot Interaction: An Internet of Things Application

Motion Recognition Based on Deep Learning Algorithm

Advanced scientific and technological measurement methods are the basis of sports research and the progress of sports research methods. It promotes the continuous improvement of sports technology to meet athletes’ pursuit of excellent sports achievements. Therefore, the method of sports scientific research must be changed from the traditional empirical training mode to the programmed training method, from the qualitative analysis of the training effect to the nuanced analysis of the training process. Based on this, this paper uses in-depth learning to study sports motion capture. First, two-dimensional human joints are extracted based on Mask R-CNN. Then, the 3D human motion skeleton is constructed by using the binocular vision system, and the Mask R-CNN human pose estimation algorithm is optimized. On this basis, a sports motion capture system is designed, and the system’s accuracy is verified. The error of the depth information obtained by the sports capture system constructed in this paper is less than 3% in the experiment of about 2[Formula: see text]m. It has strong practicability.

Next-generation fall detection: harnessing human pose estimation and transformer technology

ABSTRACT Elderly falls are occurring at an alarming rate, with significant health risks for seniors. Current fall detection systems often lack accuracy, efficacy, and privacy considerations. This study examines three leading human pose estimation frameworks combined with transformer deep learning models to develop a lightweight, privacy-preserving fall detection system. Key features include: 1) It runs on low-power devices like Raspberry Pis; 2) It monitors seniors passively, without requiring active participation; 3) It can be deployed in any residential or senior care setting; 4) It does not rely on wearables; and 5) All processing occurs locally, ensuring privacy with only fall alerts transmitted to caregivers. In real-world tests, the model achieved 95.24% sensitivity, 89.80% specificity, 98.00% accuracy, a 90.91% F1 score, and 95.24% precision, highlighting its effectiveness in detecting falls among the elderly while maintaining privacy and security.

Investigating the efficacy of AI-enhanced telerehabilitation in sarcopenic older individuals.

This study explores the effectiveness of 3D pose estimation technology in Yi Jin Jing (a traditional Chinese exercise) interventions for sarcopenic older individuals. A randomized controlled trial involving 93 participants (mean age: 71.64 ± 7.09 years; 41 males and 52 females) divided into three groups: a face-to-face offline traditional training group (OFFG), a general remote online training group (ONG), and an AI-based online remote training group (AIONG). Participants in each group underwent their respective training programs. The effectiveness of the interventions was measured using Appendicular Skeletal Muscle Mass Index, Grip Strength, 6-meter Walking Speed, Timed-Up-and-Go Test, and Quality of Life assessments. Significant improvements were observed across all groups in ASMI, Grip Strength, 6-meter Walking Speed, TUGT, and QoL. However, there were no statistically significant differences between the groups in terms of the magnitude of these improvements. AIONG showed outcomes comparable to OFFG and ONG methods. AI-based telerehabilitation with 3D pose estimation is a viable and effective alternative for remote exercise interventions. It offers precise guidance and enhances the quality of rehabilitation training, demonstrating outcomes comparable to traditional and general online methods.

Cylinder object 6-DOF pose estimation via single perspective circle on cylindrical surface

Abstract Pose estimation is an important research topic in computer vision. In this paper, a novel method for six-degree-of-freedom (6-DOF) pose estimation of cylindrical objects via a single perspective circle on cylindrical surface is proposed. Unlike traditional methods that rely on planar circular features, a circular feature target which is affixed to the side of cylindrical objects is utilized. The geometric properties of the spatial curve formed by its contour are leveraged for pose estimation. Specifically, the imaging model of the spatial curve is established to derive the general equation of its projection curve. The initial pose is determined based on the perspective principle, and the final optimized pose is obtained with the least squares method. To avoid local optima, a verification and correction method is proposed to enhance estimation accuracy. Synthetic and physical experiments demonstrate that the proposed method can achieve real-time, high-precision pose estimation for cylindrical objects, exhibiting robustness against occlusion and noise interference.

3D skeleton aware driver behavior recognition framework for autonomous driving system

The recognition of the driver’s behaviors inside an autonomous vehicle can effectively address emergency handling in autonomous driving and is crucial for ensuring the driver’s safety. Driver behavior recognition is a challenging task due to factors such as variations, diversities, complexities, and strong interferences in behaviors. In this paper, to realize the application in the autonomous driving scenes, a novel 3D skeleton aware behavior recognition framework is proposed to recognize various driver behaviors in autonomous driving systems. First, a 3D human pose estimation network (Pose-GTFNet) with temporal Transformer and spatial graph convolutional network (GCN) is designed to infer 3D human poses from 2D pose sequences. Second, based on the obtained 3D human pose sequences, a behavior recognition network (Beh-MSFNet) with multi-skeleton feature fusion is designed to recognize driver behaviors. In the experiments, the Pose-GTFNet and Beh-MSFNet can get the best performance compared with most state-of-the-art (SOTA) methods on the Human3.6M human pose dataset, JHMDB and SHREC action recognition dataset, respectively. In addition, the proposed driver behavior recognition framework can achieve SOTA performance on the Drive&Act and Driver-Skeleton driver behavior datasets.

BodyFlow: An Open-Source Library for Multimodal Human Activity Recognition

Human activity recognition is a critical task for various applications across healthcare, sports, security, gaming, and other fields. This paper presents BodyFlow, a comprehensive library that seamlessly integrates human pose estimation and multiple-person estimation and tracking, along with activity recognition modules. BodyFlow enables users to effortlessly identify common activities and 2D/3D body joints from input sources such as videos, image sets, or webcams. Additionally, the library can simultaneously process inertial sensor data, offering users the flexibility to choose their preferred input, thus facilitating multimodal human activity recognition. BodyFlow incorporates state-of-the-art algorithms for 2D and 3D pose estimation and three distinct models for human activity recognition.

DenseFusion-DA2: End-to-End Pose-Estimation Network Based on RGB-D Sensors and Multi-Channel Attention Mechanisms.

Notably, 6D pose estimation is a critical technology that enables robotics to perceive and interact with their operational environment. However, occlusion causes a loss of local features, which, in turn, restricts the estimation accuracy. To address these challenges, this paper proposes an end-to-end pose-estimation network based on a multi-channel attention mechanism, DA2Net. Firstly, a multi-channel attention mechanism, designated as "DA2Net", was devised using A2-Nets as its foundation. This mechanism is constructed in two steps. In the first step, the essential characteristics are extracted from the global feature space through the second-order attention pool. In the second step, a feature map is generated by the integration of position and channel attention. Subsequently, the extracted key features are assigned to each position of the feature map, enhancing both the feature representation capacity and the overall performance. Secondly, the designed attention mechanism is introduced into both the feature fusion and pose iterative refinement networks to enhance the network's capacity to acquire local features thus improving its overall performance. The experimental results demonstrated that the estimation accuracy of DenseFusion-DA2 on the LineMOD dataset was approximately 3.4% higher than that of DenseFusion. Furthermore, the estimation accuracy surpassed that of PoseCNN, PVNet, SSD6D, and PointFusion by 8.3%, 11.1%, 20.3%, and 23.8%, respectively. The estimation accuracy also shows a significant advantage on the Occluded LineMOD and HR-Vision datasets. This research not only presents a more efficient solution for robot perception but also introduces novel ideas and methods for technological advancements and applications in related fields.

Human Pose Classification Based on Pose Long Short-Term Memory

Objective: At present, pose classification research is a hot topic in the field of computer vision, playing a crucial role in surveillance security, and motion data mining. The research serves as a crucial follow-up to three dimensional (3D) pose estimation and makes a significant contribution to the advancement of artificial intelligence. Pose classification is a complex problem involving a large amount of complicated data, making its digital modeling and further processing challenging. Methods: This article proposes the Pose long short - term memory (LSTM) method to classify 3D poses reconstructed from 2D image sequences. Result: Compared with traditional methods, the experimental tests show that the performance of the Pose LSTM is more superior in all aspects. Conclusion: Pose LSTM is particularly effective for recognizing continuous action poses because they can capture temporal dependencies in sequential data. In the context of pose recognition, it can analyze the sequence of poses over time and understand how these poses transition from one to another, making it possible to accurately classify or predict actions that unfold over a series of frames.

3D Head Pose Estimation via Normal Maps: A Generalized Solution for Depth Image, Point Cloud, and Mesh

Head pose estimation plays a crucial role in various applications, including human–machine interaction, autonomous driving systems, and 3D reconstruction. Current methods address the problem primarily from a 2D perspective, which limits the efficient utilization of 3D information. Herein, a novel approach, called pose orientation‐aware network (POANet), which leverages normal maps for orientation information embedding, providing abundant and robust head pose information, is introduced. POANet incorporates the axial signal perception module and the rotation matrix perception module, these lightweight modules make the approach achieve state‐of‐the‐art (SOTA) performance with few computational costs. This method can directly analyze various topological 3D data without extensive preprocessing. For depth images, POANet outperforms existing methods on the Biwi Kinect head pose dataset, reducing the mean absolute error (MAE) by ≈30% compared to the SOTA methods. POANet is the first method to perform rigid head registration in a landmark‐free manner. It also incorporates few‐shot learning capabilities and achieves an MAE of about on the Headspace dataset. These features make POANet a superior alternative to traditional generalized Procrustes analysis for mesh data processing, offering enhanced convenience for human phenotype studies.

Automated yoga pose recognition using enhanced chicken swarm optimization with deep learning

Automated yoga pose recognition using enhanced chicken swarm optimization with deep learning

Dual Leap Motion Controller 2: A Robust Dataset for Multi-view Hand Pose Recognition

This paper presents Multi-view Leap2 Hand Pose Dataset (ML2HP Dataset), a new dataset for hand pose recognition, captured using a multi-view recording setup with two Leap Motion Controller 2 devices. This dataset encompasses a diverse range of hand poses, recorded from different angles to ensure comprehensive coverage. The dataset includes real images with the associated precise and automatic hand properties, such as landmark coordinates, velocities, orientations, and finger widths. This dataset has been meticulously designed and curated to maintain a balance in terms of subjects, hand poses, and the usage of right or left hand, ensuring fairness and parity. The content includes 714,000 instances from 21 subjects of 17 different hand poses (including real images and 247 associated hand properties). The multi-view setup is necessary to mitigate hand occlusion phenomena, ensuring continuous tracking and pose estimation required in real human-computer interaction applications. This dataset contributes to advancing the field of multimodal hand pose recognition by providing a valuable resource for developing advanced artificial intelligence human computer interfaces.

Data augmentation in human-centric vision

AbstractThis survey presents a comprehensive analysis of data augmentation techniques in human-centric vision tasks, a first of its kind in the field. It delves into a wide range of research areas including person ReID, human parsing, human pose estimation, and pedestrian detection, addressing the significant challenges posed by overfitting and limited training data in these domains. Our work categorizes data augmentation methods into two main types: data generation and data perturbation. Data generation covers techniques like graphic engine-based generation, generative model-based generation, and data recombination, while data perturbation is divided into image-level and human-level perturbations. Each method is tailored to the unique requirements of human-centric tasks, with some applicable across multiple areas. Our contributions include an extensive literature review, providing deep insights into the influence of these augmentation techniques in human-centric vision and highlighting the nuances of each method. We also discuss open issues and future directions, such as the integration of advanced generative models like Latent Diffusion Models, for creating more realistic and diverse training data. This survey not only encapsulates the current state of data augmentation in human-centric vision but also charts a course for future research, aiming to develop more robust, accurate, and efficient human-centric vision systems.

Image-Based Fitness Yoga Pose Recognition: Using Ensemble Learning and Multi-head Attention

With the increasing awareness of fitness, more and more people are choosing to participate in fitness activities. Yoga, as a form of exercise that improves both physical and mental health, is becoming increasingly popular worldwide. In order to assist yoga practitioners in more effective training through automated or semi automated systems, improve training effectiveness, assist professional athletes in training through intelligent recognition systems, correct movements, and improve athletic performance. This paper proposes a method that addresses the low accuracy issue of current yoga pose recognition algorithms by integrating multi-head attention mechanism and ensemble learning. Firstly, the Mixup algorithm is used to enhance yoga movement images. Subsequently, convolutional features are extracted from the images using the ResNet101 and VGGNet19 transfer learning models. Finally, the extracted convolutional features are combined and stacked using a multi-head attention mechanism. Model training, validation, and testing are performed using the Soft target cross-entropy loss function. Experimental results demonstrate that the proposed method achieves a training accuracy of 100%, a validation accuracy of 89.94%, a testing accuracy of 93.79%, and a detection speed of 297 frames per second. Overall, this method demonstrates high stability and robustness, providing a technological foundation for intelligent recognition of yoga poses.

3D human pose estimation with multi-hypotheses gated transformer

3D human pose estimation with multi-hypotheses gated transformer

Exploring event-based human pose estimation with 3D event representations

Human pose estimation is a fundamental and appealing task in computer vision. Although traditional cameras are commonly applied, their reliability decreases in scenarios under high dynamic range or heavy motion blur, where event cameras offer a robust solution. Predominant event-based methods accumulate events into frames, ignoring the asynchronous and high temporal resolution that is crucial for distinguishing distinct actions. To address this issue and to unlock the 3D potential of event information, we introduce two 3D event representations: the Rasterized Event Point Cloud (RasEPC) and the Decoupled Event Voxel (DEV). The RasEPC aggregates events within concise temporal slices at identical positions, preserving their 3D attributes along with statistical information, thereby significantly reducing memory and computational demands. Meanwhile, the DEV representation discretizes events into voxels and projects them across three orthogonal planes, utilizing decoupled event attention to retrieve 3D cues from the 2D planes. Furthermore, we develop and release EV-3DPW, a synthetic event-based dataset crafted to facilitate training and quantitative analysis in outdoor scenes. Our methods are tested on the DHP19 public dataset, MMHPSD dataset, and our EV-3DPW dataset, with further qualitative validation via a derived driving scene dataset EV-JAAD and an outdoor collection vehicle. Our code and dataset have been made publicly available at https://github.com/MasterHow/EventPointPose.

Hierarchical Keypoints Feature Alignment for Domain Adaptive Pose Estimation

Unsupervised Domain Adaptation (UDA) is a popular research topic in computer vision. One of the most common strategies in this field is to reduce domain shifts through global feature alignment. Unfortunately, we observe its failure on adaptive human pose estimation. From our analysis, we find out two major reasons: the extreme imbalance between keypoints and non-keypoints regions and the high diversity of human skeleton structures. To address these problems, we propose a simple yet effective approach named PoseDA. Our idea is to let the alignment focus on the feature of keypoint regions and enforce the model to learn domain-invariant representations for keypoint prediction. The key component of our PoseDA is a Hierarchical Feature Selection of Keypoints Regions (HFS) module, which consists of Coarse Feature Selection of Keypoints Regions (CFS) and Reliable Feature Selection of Keypoints Regions (RFS). By using HFS, we obtain reliable and compact keypoints features, allowing our model to achieve more effective feature alignment. Under three UDA scenarios, i.e. JTA→MPII, SURREAL→LSP, SURREAL→Human3.6, our PoseDA establishes new state-of-the-art performance. In particular, our PoseDA outperforms the previous best UDA methods by over 7% w.r.t. PCKh on JTA→MPII.

DM-HAP: Diffusion model for accurate hand pose prediction

Forecasting hand poses is a challenging task due to inherent uncertainties, occlusions, and inaccuracies in 3D pose estimation. Diffusion models provide a promising direction for predicting precise 3D hand poses under noisy conditions. In this work, we introduce Dual-diffusion, an innovative framework for the precise prediction of future hand poses. Our approach leverages the strengths of diffusion models by framing hand pose forecasting as a reverse diffusion process, effectively addressing the complexities of noisy hand joint movements and their subtleties. To enhance the learning of hand pose representations, we propose a unique neural architecture that simultaneously captures both local and global features. This is achieved through the deployment of Global and Local Diffusion (GLD) blocks within our network, which facilitate the exchange of information between local and global features. This diffusion-based interaction enables the integration of global hand actions and local finger actions, leading to a more powerful representation learning approach. We evaluate the effectiveness of our Dual-diffusion method on three public 3D hand pose estimation datasets (MSRA, F-PHAB, and BigHand2.2M), and our approach outperforms previous methods.

A multi-stage approach for desired part grasping under complex backgrounds in human-robot collaborative assembly

Human-robot collaborative assembly can leverage the unique capabilities of humans and robots to provide a flexible and efficient way for complex tasks. In this context, robots are expected to be endowed with the perception ability to collaborate with humans. For flexible processes, robotic grasping for a desired assembly part from variable multi-parts is essential but remains challenging, especially in situations of complex backgrounds including disordered placement, occlusion, similarity, and large-scale differences in size. To improve the perception and decision-making abilities of robots in collaborative assembly, firstly, a multi-stage approach integrating 2D part recognition and 6D pose estimation with perspective-focusing is proposed for desired part grasping under complex backgrounds. Secondly, to efficiently and accurately recognize a desired part from variable multi-parts under complex backgrounds, based on RGB data, an enhanced detector is adopted, which lays the foundation for the subsequent pose estimation. Thirdly, based on point cloud data, a pose estimation method based on perspective-focusing is proposed to enhance the success rate and efficiency of pose acquisition. Additionally, a modular autonomous robotic grasping system is designed. Finally, taking decelerator assembly as a case, the proposed approaches are comprehensively validated and compared on a real collaborative robot platform. Under complex backgrounds with occlusion, our approach shows stable performance with an average grasping success rate of 90.0 % and an average cycle time of 6.4175 s. Our work is expected to improve robots’ perception and decision-making capabilities while further reducing the cycle time of collaborative assembly tasks.

Improving Human Pose Estimation with Integrated Dual Self-Attention Mechanism in High-Resolution Network

Human Pose Estimation (HPE) in computer vision (CV) has garnered significant attention due to its diverse applications. Deep convolutional neural networks (CNNs) may be solutions for addressing this challenge, but still face several critical issues. Many existing models employ serial convolution with pooling, leading to low-resolution outputs that are suboptimal for the precise localisation required in HPE. They often prioritise local feature learning, overlooking crucial contextual relationships between key-points. This work addresses these challenges by proposing a novel approach for enhancing HPE. Firstly, the paper evaluates the high-resolution network (HRNet) and its comparative advantages over other CNN architectures. Secondly, it introduces a dual self-attention (DSA) mechanism designed to enhance the model’s global awareness, thereby enriching feature maps with contextual information. Thirdly, it integrates the DSA mechanism into HRNet, crafting DSA-HRNet. The model performance was tested on the COCO Val 2017 validation dataset, showing improvements of 2.3% in mean average precision (mAP), 3% in AP at 50 (AP50), and 2.7% in AP at 75 (AP75). Finally, the work includes an investigation into the effectiveness of the DSA mechanism within the HRNet framework, through a series of experiments, showing this work offers a streamlined and effective solution for improving HPE.