Human Pose Estimation Research Articles

Multi-Scale Feature Refined Network for Human Pose Estimation

Occlusive keypoints has been a challenge for human pose estimation, especially the mutual occlusion of human bodies. One possible solution to this problem is to utilize multi-scale features, where small scale features are capable of identifying keypoints, while large-scale features can capture the relationship between keypoints. Feature fusion among multi-scale features allows for the exchange of information between keypoints, facilitating the inference of occluded keypoints based on the identified keypoints. However, it’s found that there are invalid features in feature fusion which will interfere valid feature. In this paper, we propose multi-scale feature refined network (MSFRNet) based on HRNet and a new attention module namely multi-resolution attention module (MRAM). The proposed MRAM is designed to strengthen the effective information while suppressing redundant information. It has multiple inputs and outputs and can learn the relationships between keypoints while retaining detailed information. The proposed MSFRNet outperforms HRNet, achieving a 1.4[Formula: see text]AP improvement on the COCO dataset with only a marginal computational increase of 0.35 GFLOPs. Additionally, it demonstrates superior performance with a 0.9[Formula: see text]AP, 0.7[Formula: see text]AP, and 1.8[Formula: see text]AP improvement on the MPII, CrowdPose and OCHuman datasets, respectively. Furthermore, compared with the latest attention mechanism PSA, the MSFRNet exhibits lower computational cost while maintaining the same pose-estimation accuracy.

Joint graph convolution networks and transformer for human pose estimation in sports technique analysis

Human pose estimation has various applications in domains such as sports technology analysis, virtual reality, and education. However, most previous studies focused on the respective feature representations of keypoints, but disregarded the topological relationship among keypoints. To address this challenge, we propose GTPose, a network structure that integrates graph convolutional networks and Transform. First of all, a set of multi-scale convolution operations are applied to extract local feature maps of images. Secondly, the positions of keypoints are roughly estimated by using Transform to process the sequential relations between feature maps. Finally, GCN is adopted to model the topological structure between keypoints to accurately locate the location of keypoints and learn feature representations. The performance of GTPose is evaluated on two real datasets: MS COCO and MPII. Experimental results demonstrate that GTPose outperforms other methods in human pose estimation tasks. In addition, experimental results also show that the spatial relationship between keypoints is effective for accurately characterizing keypoints.

Context-aware 6D pose estimation of known objects using RGB-D data

Context-aware 6D pose estimation of known objects using RGB-D data

An Efficient Pose Estimation Algorithm for Non-Cooperative Space Objects Based on Dual-Channel Transformer

Non-cooperative space object pose estimation is a key technique for spatial on-orbit servicing, where pose estimation algorithms based on low-quality, low-power monocular sensors provide a practical solution for spaceborne applications. The current pose estimation methods for non-cooperative space objects using monocular vision generally consist of three stages: object detection, landmark regression, and perspective-n-point (PnP) solver. However, there are drawbacks, such as low detection efficiency and the need for prior knowledge. To solve the above problems, an end-to-end non-cooperative space object pose estimation learning algorithm based on dual-channel transformer is proposed, a feature extraction backbone network based on EfficientNet is established, and two pose estimation subnetworks based on transformer are also established. A quaternion SoftMax-like activation function is designed to improve the precision of orientation error estimating. The method only uses RGB images, eliminating the need for a CAD model of the satellite, and simplifying the detection process by using an end-to-end network to directly detect satellite pose information. Experiments are carried out on the SPEED dataset provided by the European Space Agency (ESA). The results show that the proposed algorithm can successfully predict the satellite pose information and effectively decouple the spatial translation information and orientation information, which significantly improves the recognition efficiency compared with other methods.

STRFormer: Spatial–Temporal–ReTemporal Transformer for 3D human pose estimation

Transformer-based methods have emerged as the golden standard in 2D-3D human pose estimation from video sequences, largely thanks to their powerful spatial–temporal feature encoders. In the past, researchers have made concerted efforts to engineer spatial and temporal encoders using transformer blocks. This approach involved a dramatic reshaping of the input, transforming it from mere joint information to dynamic joint trajectories. Despite this, the inherent limitations of the spatial–temporal structure have resulted in an inadequate acquisition and subsequent utilization of temporal information. In an attempt to rectify this prevalent issue, our paper proposes a new model, dubbed Spatial–Temporal-ReTemporal Transformer (i.e., STRFormer). This model ingeniously employs two separate temporal transformer blocks to extract the essential temporal motion information from video sequences. Intriguingly, one temporal transformer block is dedicated to the original video sequence, while the other concerns itself with the reversed order video. This novel approach allows for a more thorough investigation and utilization of temporal information from the video sequences. In order to alternate the processing of these two blocks effectively with the spatial block, we focus on maximizing the extraction of temporal domain information. This method leads to a more comprehensive understanding of the pose estimation and its evolution over time. Furthermore, we introduce a novel error metric, Mean Per-Joint Position Acceleration Error (i.e., MPJAE). This advanced metric takes into account the body part velocity in adjacent predicted frames, allowing for a more detailed evaluation of the predicted poses. We conduct extensive experiments on various open benchmarks to evaluate the effectiveness of our proposed model. The results demonstrate that our STRFormer, coupled with the MPJAE loss, achieves highly competitive results when compared with other state-of-the-art models. This illustrates its promising potential and practical applicability in 2D-3D human pose estimation tasks. We plan to release our code publicly for further research.

Constraints on Optimising Encoder-Only Transformers for Modelling Sign Language with Human Pose Estimation Keypoint Data.

Supervised deep learning models can be optimised by applying regularisation techniques to reduce overfitting, which can prove difficult when fine tuning the associated hyperparameters. Not all hyperparameters are equal, and understanding the effect each hyperparameter and regularisation technique has on the performance of a given model is of paramount importance in research. We present the first comprehensive, large-scale ablation study for an encoder-only transformer to model sign language using the improved Word-level American Sign Language dataset (WLASL-alt) and human pose estimation keypoint data, with a view to put constraints on the potential to optimise the task. We measure the impact a range of model parameter regularisation and data augmentation techniques have on sign classification accuracy. We demonstrate that within the quoted uncertainties, other than ℓ2 parameter regularisation, none of the regularisation techniques we employ have an appreciable positive impact on performance, which we find to be in contradiction to results reported by other similar, albeit smaller scale, studies. We also demonstrate that the model architecture is bounded by the small dataset size for this task over finding an appropriate set of model parameter regularisation and common or basic dataset augmentation techniques. Furthermore, using the base model configuration, we report a new maximum top-1 classification accuracy of 84% on 100 signs, thereby improving on the previous benchmark result for this model architecture and dataset.

Robotic Continuous Grasping System by Shape Transformer-Guided Multiobject Category-Level 6-D Pose Estimation

Robotic grasping is one of the key functions for realizing industrial automation and human-machine interaction. However, current robotic grasping methods for unknown objects mainly focus on generating the 6D grasp poses, which cannot obtain rich object pose information and are not robust in challenging scenes. Based on this, we propose a robotic continuous grasping system that achieves end-to-end robotic grasping of intra-class unknown objects in 3D space by accurate category-level 6D object pose estimation. Specifically, to achieve object pose estimation, first, we propose a global shape extraction network (GSENet) based on ResNet1D to extract the global shape of an object category from the 3D models of intra-class known objects. Then, with the global shape as the prior feature, we propose a transformer-guided network to reconstruct the shape of intra-class unknown object. The proposed network can effectively introduce internal and mutual communication between the prior feature, current feature, and their difference feature. The internal communication is performed by self-attention. The mutual communication is performed by cross-attention to strengthen their correlation. To achieve robotic grasping for multiple objects, we propose a low-computation and effective grasping strategy based on the pre-defined vector orientation, and develop a GUI for monitoring and control. Experiments on two benchmark datasets demonstrate that our system achieves state-of-the-art (SOTA) 6D pose estimation accuracy. Moreover, the real-world experiments show that our system also achieves superior robotic grasping performance, with a grasping success rate of 81.6 <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\%$</tex-math></inline-formula> for multiple objects. Code and trained models are released at https://github.com/CNJianLiu/6D-CLGrasp

Pose Estimation of Point Sets Using Residual MLP in Intelligent Transportation Infrastructure

6D pose estimation of arbitrary objects is a crucial topic for intelligent transportation infrastructure measurement. However, some external environmental factors and the characteristics of the object itself impact the accuracy of the object’s pose estimation in practical applications. In this paper, we propose a new multi-class dataset ICD-4 (Industrial car Components Dataset) for 6D object pose estimation, which mainly includes four component categories, and every category takes 20,000 different scenarios. ICD-4 dataset delivers quite a few research challenges involving the range of object pose transformations and has significant research value for small-scale pose estimation tasks. We also propose an innovative method PoseMLP, a pose estimation network that uses residual MLP (multilayer perceptron) modules to predict the 6D pose estimation directly. Simultaneously, the experimental results demonstrate the effectiveness and reliability of the proposed method.

Recognition Method with Deep Contrastive Learning and Improved Transformer for 3D Human Motion Pose

Three-dimensional (3D) human pose recognition techniques based on spatial data have gained attention. However, existing models and algorithms fail to achieve desired precision. We propose a 3D human motion pose recognition method using deep contrastive learning and an improved Transformer. The improved Transformer removes noise between human motion RGB and depth images, addressing orientation correlation in 3D models. Two-dimensional (2D) pose features are extracted from de-noised RGB images using a kernel generation module in a graph convolutional network (GCN). Depth features are extracted from de-noised depth images. The 2D pose features and depth features are fused using a regression module in the GCN to obtain 3D pose recognition results. The results demonstrate that the proposed method captures RGB and depth images, achieving high recognition accuracy and fast speed. The proposed method demonstrates good accuracy in 3D human motion pose recognition.

An analysis of precision: occlusion and perspective geometry’s role in 6D pose estimation

Achieving precise 6 degrees of freedom (6D) pose estimation of rigid objects from color images is a critical challenge with wide-ranging applications in robotics and close-contact aircraft operations. This study investigates key techniques in the application of YOLOv5 object detection convolutional neural network (CNN) for 6D pose localization of aircraft using only color imagery. Traditional object detection labeling methods suffer from inaccuracies due to perspective geometry and being limited to visible key points. This research demonstrates that with precise labeling, a CNN can predict object features with near-pixel accuracy, effectively learning the distinct appearance of the object due to perspective distortion with a pinhole camera. Additionally, we highlight the crucial role of knowledge about occluded features. Training the CNN with such knowledge slightly reduces pixel precision, but enables the prediction of 3 times more features, including those that are not initially visible, resulting in an overall better performing 6D system. Notably, we reveal that the data augmentation technique of scale can interfere with pixel precision when used during training. These findings are crucial for the entire system, which leverages the Solve Perspective-N-Point (Solve-PnP) algorithm, achieving 6D pose accuracy within 1^circ and 7 cm at distances ranging from 7.5 to 35 m from the camera. Moreover, this solution operates in real-time, achieving sub-10ms processing times on a desktop PC.

Pose-guided self and external attention feature matching and aggregation network for person re-identification

Traditional methods that use pose information for person re-identification (ReID) often focus only on relationships between different parts of the same sample and ignore the correlation between poses of different samples, which hinders further improvement in recognition accuracy. In view of this, this paper proposes a pose-guided self and external attention feature matching and aggregation net which contains a visual context self and external attention module and a pose-guided feature matching and aggregation module. It can divide global features into local features without strict spatial feature alignment, and enables the model to focus on pose correlation between different samples. Thus, it not only learns pose-related identity identification features, but also avoids interference from the samples with large pose changes. The visual context self and external attention module realizes feature embedding through a transformer-based image classification model ViT and a human pose estimation model. Also, it extracts encoder features and local features containing rich pose information by adding an external attention. The pose-guided feature matching and aggregation module obtains a learnable part semantic view through a transformer decoder and pose heat maps. Then, it matches and aggregates with local features, adaptively learns pose-related features, and enhances the robustness of the model to background and occlusion. In this paper, experiments are conducted based on three data sets, including Market-1501, DukeMTMC-reID and MSMT17. The Rank-1 values are 95.3%, 90.1%, and 82.5%, respectively, and the mAP values are 88.3%, 81.1%, and 64.1%, respectively. In conclusion, our method introduces an external attention into the ReID task, and the ReID model can obtain more accurate results by extracting pose-related identity identification features and avoiding interference from samples with large pose changes.

SimCC coordinate based learning of human pose constraint information

This paper proposes a novel approach to improve the accuracy of human pose estimation, which is a technique to predict the coordinates of human body keypoints in an input image. Existing methods often focus on keypoint localization without considering their positional relationships. We introduce two regularization terms based on SimCC coordinates: Bone loss and Sum loss of visible keypoints. They constrain the spatial relationship and visible number of keypoints respectively. To better leverage these loss functions, we enhance the deconvolution module with a combination of convolution and bilinear interpolation, which effectively recovers image resolution and preserves more feature map details. We conduct a comprehensive evaluation of our approach on the COCO dataset. Our approach enhances the accuracy of the top-down method by incorporating HRNet-W48. We achieve 77.7AP on the COCO validation set, which is competitive with state-of-the-art methods.

Deep Learning-Based Standardized Evaluation and Human Pose Estimation: A Novel Approach to Motion Perception

Deep Learning-Based Standardized Evaluation and Human Pose Estimation: A Novel Approach to Motion Perception

Design of a Two-Dimensional Conveyor Platform with Cargo Pose Recognition and Adjustment Capabilities.

Linear conveyors, traditional tools for cargo transportation, have faced criticism due to their directional constraints, inability to adjust poses, and single-item conveyance, making them unsuitable for modern flexible logistics demands. This paper introduces a platform designed to convey and adjust cargo boxes according to their spatial positions and orientations. Additionally, a cargo pose recognition algorithm that integrates image and point cloud data are presented. By aligning depth camera data, the axis-aligned bounding box (AABB) point serves as the image's region of interest (ROI). Peaks extracted from the image's Hough transform are refined using RANSAC-based point cloud linear fitting, then integrated with the point cloud's oriented bounding box (OBB). Notably, the algorithm eliminates the need for deep learning and registration, enabling its use in rectangular cargo boxes of various sizes. A comparative experiment using accelerometer sensors for pose acquisition revealed a deviation of <0.7° between the two processes. Throughout the real-time adjustments controlled by the experimental platform, cargo angles consistently remained stable. The proposed two-dimensional conveyance platform, compared to existing methods, exhibits simplicity, accurate recognition, enhanced flexibility, and wide applicability.

YH-Pose: Human pose estimation in complex coal mine scenarios

Human pose estimation in continuous video frames captured from complex coal mine scenes is challenging. The video frames in this scene may suffer from problems such as uneven brightness, blurred image details, and excessive noise. Mainstream pose estimation methods have good detection performance for high-quality static images, but their accuracy and prediction rate can be significantly reduced in coal mine scenes. In this work, a yielding human pose estimation framework is proposed, termed YH-Pose. The framework strives to incorporate additional visual evidences from neighboring frames to facilitate the pose estimation of the current frame. Firstly, a human detector is introduced to locate the person’s position in the video frame and extract global features. This can provide a good initialization for the latter keypoint detection, making the training process converge quickly. Secondly, heatmaps are used to encode the joint locations as Gaussian peaks, and a temporal road module (TRM) is designed, which encodes video frames at intervals. The module efficiently fuses spatio-temporal information through frame-rate groups in a hierarchical manner. Lastly, the spatial road module (SRM) learns from fused keypoint context features and predicts the location of the keypoints in the next frame. In addition, a dataset called Colliery-1 is proposed, which derives from underground surveillance video from chinese coal mines and consists of 3600 video clips. The experimental results on the Colliery-1 dataset indicate that the framework achieved an average accuracy of 82% and 80% on the training and test sets, respectively. Moreover, the framework achieved a 94.2% prediction rate for pose estimation. To further evaluate the effectiveness of the proposed method, some comparisons have been made between it and various mainstream methods using different metrics.

"Self-care selfies": Patient-uploaded videos capture meaningful changes in dexterity over 6 months.

Upper extremity function reflects disease progression in multiple sclerosis (MS). This study evaluated the feasibility, validity, and sensitivity to change of remote dexterity assessments applying human pose estimation to patient-uploaded videos. A discovery cohort of 50 adults with MS recorded "selfie" videos of self-care tasks at home: buttoning, brushing teeth, and eating. Kinematic data were extracted using MediaPipe Hand pose estimation software. Clinical comparison tests were grip and pinch strength, 9 hole peg test (9HPT), and vibration, and patient-reported dexterity assessments (ABILHAND). Feasibility and acceptability were evaluated (Health-ITUES framework). A validation cohort (N = 35) completed 9HPT and videos. The modality was feasible: 88% of the 50 enrolled participants uploaded ≥3 videos, and 74% completed the study. It was also usable: assessments easy to access (95%), platform easy to use (97%), and tasks representative of daily activities (86%). The buttoning task revealed four metrics with strong correlations with 9HPT (nondominant: r = 0.60-0.69, dominant: r = 0.51-0.57, P < 0.05) and ABILHAND (r = -0.48, P = 0.05). Retest validity at 1 week was stable (r > 0.8). Cross-sectional correlations between video metrics and 9HPT were similar at 6 months, and in the validation cohort (nondominant: r = 0.46, dominant: r = 0.45, P < 0.05). Over 6 months, pinch strength (5.8-5.0 kg/cm2 , P = 0.05) and self-reported pinch (ABILHAND) decreased marginally. While only 15% of participants worsened by 20% on 9HPT, 70% worsened in key buttoning video metrics. Patient-uploaded videos represent a novel, patient-centered modality for capturing dexterity that appears valid and sensitive to change, enhancing its potential to be disseminated for neurological disease monitoring and treatment.

Posture Risk Assessment and Workload Estimation for Material Handling by Computer Vision

Workers in material handling tasks often suffer from work-related musculoskeletal disorders (WMSDs) caused by inaccurate work postures or the lifting of excessively heavy loads. Therefore, effective ergonomic assessment of workers is needed to improve worker productivity while reducing the risk of musculoskeletal disorders. This paper proposes a noninvasive method for evaluating posture risks and load analysis in manual material handling tasks. The study focuses on three main aspects: first, using 3D pose recognition technology to extract the 3D coordinates and joint angles of the human body. Second, the REBA method was improved by using fuzzy logic theory to more effectively capture the slow transition features of continuous movement by humans without abruptly altering risk scores, as well as to increase the accuracy and consistency of posture risk evaluation. Third, joint torque and workloads were estimated using biomechanical calculations by integrating pressure insoles and 3D joint coordinate data. Experiments show that this method can effectively evaluate posture risks and workloads in manual material handling tasks, with a correlation coefficient of 0.817 ( p &lt; 0.01 ) between fuzzy logic REBA and REBA and an error rate of 15% in estimating workloads of eight joints. This method can help reduce occupational health risks for workers and industries and improve work efficiency.

Real-time driver monitoring system with facial landmark-based eye closure detection and head pose recognition

This paper introduces a real-time Driver Monitoring System (DMS) designed to monitor driver behavior while driving, employing facial landmark estimation-based behavior recognition. The system utilizes an infrared (IR) camera to capture and analyze video data. Through facial landmark estimation, crucial information about the driver’s head posture and eye area is extracted from the detected facial region, obtained via face detection. The proposed method consists of two distinct modules, each focused on recognizing specific behaviors. The first module employs head pose analysis to detect instances of inattention. By monitoring the driver’s head movements along the horizontal and vertical axes, this module assesses the driver’s attention level. The second module implements an eye-closure recognition filter to identify instances of drowsiness. Depending on the continuity of eye closures, the system categorizes them as either occasional drowsiness or sustained drowsiness. The advantages of the proposed method lie in its efficiency and real-time capabilities, as it solely relies on IR camera video for computation and analysis. To assess its performance, the system underwent evaluation using IR-Datasets, demonstrating its effectiveness in monitoring and recognizing driver behavior accurately. The presented real-time Driver Monitoring System with facial landmark-based behavior recognition offers a practical and robust approach to enhance driver safety and alertness during their journeys.

Tomato flower pollination features recognition based on binocular gray value-deformation coupled template matching

It is necessary to recognize the tomato pollination features for the designing demand of intelligent and precise tomato supplementary pollination equipment. Mentioned pollination features include flower opening state and the three-dimensional position and pose of flower anther. Tomato flower pollination features recognition method is designed in this paper based on the deep learning full opened flower recognition model and binocular template matching three-dimensional information recognition method. First of all, a binocular stereo vision system is built to acquire the tomato flower images in greenhouse with natural lighting. Which can help vision system avoid the impact of inconsistent light intensity on image recognition. The acquired images were equalized with three groups parameters to labeled images. And the improved MC-AlexNet deep learning model is established to recognize the full opened tomato flowers in the left image of image pair acquired with binocular vision system. Then, the template is created with the full opened flower recognition result of deep learning model in left image based on gray value and deformation template matching method. And the template matching is established to recognize the corresponding full opened flowers in the right image of image pair. Finally, with the template matching result, the anther segmentation is conducted to calculate three-dimensional position and pose of anther. The experimental results show that the accuracy of full opened tomato flowers recognition model is 96.23%. The average position recognition deviation of anther is 5.94 mm. And the average anther pose recognition deviation in three plane is 6.24° for the images that anthers can be observed. And the average time consuming is about 143.18 ms per image pair. It can be concluded that the method of binocular template matching established in this paper can fulfill the demand of supplementary pollination equipment design, and the research result lays a foundation for designing and improvement of tomato supplementary pollination equipment in greenhouse.

The comparison of top-down and bottom-up methods in multi-person pose estimation

2D multi-person pose estimation is a process to detect all bodies in a two-dimensional picture. The main purpose of this report is to discuss the difference between top-down and bottom- up method in multi-person human pose estimation. This estimation will focus on many people in one picture. There are two popular methods in this area. The first is top-down method, which is to find people firstly and then to detect the body of each people. On the other hand, the bottom-up method is to detect body parts and then find each person. Their comparison and analyse in algorithm, speed and accuracy may help researchers to find more suitable methods when they research in human pose estimation. Generally, top-down methods have higher accuracy than bottom-up methods but have higher speed.