Human Pose Estimation Research Articles

Adversarial imitation learning-based network for category-level 6D object pose estimation

Adversarial imitation learning-based network for category-level 6D object pose estimation

A multi-channel spatial information feature based human pose estimation algorithm

Human pose estimation is an important task in computer vision, which can provide key point detection of human body and obtain bone information. At present, human pose estimation is mainly utilized for detection of large targets, and there is no solution for detection of small targets. This paper proposes a multi-channel spatial information feature based human pose (MCSF-Pose) estimation algorithm to address the issue of medium and small targets inaccurate detection of human key points in scenarios involving occlusion and multiple poses. The MCSF-Pose network is a bottom-up regression network. Firstly, an UP-Focus module is designed to expand the feature information while reducing parameter computation during the up-sampling process. Then, the channel segmentation strategy is adopted to cut the features, and the feature information of multiple dimensions is retained through different convolutional groups, which reduces the parameter lightweight network model and makes up for the loss of the feature information associated with the depth of the network. Finally, the three-layer PANet structure is designed to reduce the complexity of the model. With the aid of the structure, it also to improve the detection accuracy and anti-interference ability of human key points. The experimental results indicate that the proposed algorithm outperforms YOLO-Pose and other human pose estimation algorithms on COCO2017 and MPII human pose datasets.

Human pose estimation in complex background videos via Transformer-based multi-scale feature integration

Human posture estimation is still a hot research topic. Previous algorithms based on traditional machine learning have difficulties in feature extraction and low fusion efficiency. To address these problems, we proposed a Transformer-based method. We combined three techniques, namely the Transformer-based feature extraction module, the multi-scale feature fusion module, and the occlusion processing mechanism, to capture the human pose. The Transformer-based feature extraction module uses the self-attention mechanism to extract key features from the input sequence, the multi-scale feature fusion module fuses feature information of different scales to enhance the perception ability of the model, and the occlusion processing mechanism can effectively handle occlusion in the data and effectively remove background interference. Our method has shown excellent performance through verification on the standard dataset Human3.6M and the wild video dataset, achieving accurate pose prediction in both complex actions and challenging samples.

Advancing Global Pose Refinement: A Linear, Parameter-Free Model for Closed Circuits via Quaternion Interpolation.

Global pose refinement is a significant challenge within Simultaneous Localization and Mapping (SLAM) frameworks. For LIDAR-based SLAM systems, pose refinement is integral to correcting drift caused by the successive registration of 3D point clouds collected by the sensor. A divergence between the actual and calculated platform paths characterizes this error. In response to this challenge, we propose a linear, parameter-free model that uses a closed circuit for global trajectory corrections. Our model maps rotations to quaternions and uses Spherical Linear Interpolation (SLERP) for transitions between them. The intervals are established by the constraint set by the Least Squares (LS) method on rotation closure and are proportional to the circuit's size. Translations are globally adjusted in a distinct linear phase. Additionally, we suggest a coarse-to-fine pairwise registration method, integrating Fast Global Registration and Generalized ICP with multiscale sampling and filtering. The proposed approach is tested on three varied datasets of point clouds, including Mobile Laser Scanners and Terrestrial Laser Scanners. These diverse datasets affirm the model's effectiveness in 3D pose estimation, with substantial pose differences and efficient pose optimization in larger circuits.

EV-TIFNet: lightweight binocular fusion network assisted by event camera time information for 3D human pose estimation

EV-TIFNet: lightweight binocular fusion network assisted by event camera time information for 3D human pose estimation

Unveiling Structure-Enhanced Positional Embedding Module in Transformer-Based 3D Human Pose Estimation

Abstract With the advancement of computational capabilities, the realm of deep learning has expanded from 2D image tasks to encompass 3D video tasks. Human action recognition stands as a typical 3D video task, which emphasizes the accurately capture of local body structures. However, existing methods encounter limitations in precisely capturing these structures, especially in joints within the same body region that exhibits diverse static and dynamic features. To address this complexity, our study innovatively introduces a Structure-Enhanced Positional Embedding Module seamlessly integrated into the PoseFormer model, resulting in the innovative HStruPE-Former model. Experimental validation on the Human3.6M dataset demonstrates a significant performance improvement achieved by the HStruPE-Former model in the task of 3D human pose estimation. Particularly in scenarios involving intricate movements and postures, the module demonstrates improved stability and accuracy. Although the improvements are modest, this technology is expected to facilitate the exploration of untapped potential of structure-enhanced postion embedding modules in diverse neural network applications, thereby promoting further exploration in neural network research. This research outcome not only enhances the performance in current 3D video tasks but also paves the way for novel research directions in the field of deep learning.

A Two-Stage Monocular Vision Detection Method for 6D Pose Estimation in Multi-Heterogeneous Robot Systems

A Two-Stage Monocular Vision Detection Method for 6D Pose Estimation in Multi-Heterogeneous Robot Systems

Dual-COPE: A novel prior-based category-level object pose estimation network with dual Sim2Real unsupervised domain adaptation module

Category-level pose estimation offers the generalization ability to novel objects unseen during training, which has attracted increasing attention in recent years. Despite the advantage, annotating real-world data with pose label is intricate and laborious. Although using synthetic data with free annotations can greatly reduce training costs, the Synthetic-to-Real (Sim2Real) domain gap could result in a sharp performance decline on real-world test. In this paper, we propose Dual-COPE, a novel prior-based category-level object pose estimation method with dual Sim2Real domain adaptation to avoid expensive real pose annotations. First, we propose an estimation network featured with conjoined prior deformation and transformer-based matching to realize high-precision pose prediction. Upon that, an efficient dual Sim2Real domain adaptation module is further designed to reduce the feature distribution discrepancy between synthetic and real-world data both semantically and geometrically, thus maintaining superior performance on real-world test. Moreover, the adaptation module is loosely coupled with estimation network, allowing for easy integration with other methods without any additional inference overhead. Comprehensive experiments show that Dual-COPE outperforms existing unsupervised methods and achieves state-of-the-art precision under supervised settings.

TeaPoseNet: A deep neural network for tea leaf pose recognition

The estimation of tea leaf pose is an emerging research topic. Recognising the morphological features of tea leaves can help accurately categorise, grade, and determine their level of maturity. Therefore, this study proposes a deep neural network, TeaPoseNet, to estimate tea leaf poses. The algorithm was trained and validated using a dataset of one-bud-one-leaf images of Yinghong No.9 tea leaves and was compared with four other pose estimation networks. At the same time, the contribution of TKS_NMS to the algorithm was validated through ablation experiments. The results indicate that TKS_NMS improved the EPE accuracy of pose recognition by 16.33 %. More specifically, the algorithm achieved a good overall performance, with PCK, AUC, EPE, and NME reaching 0.9800, 0.8147, 9.0955, and 0.0644, respectively. The average running speed for detecting the pose of a single tea leaf image was 40.01 ms. To the best of our knowledge, this is the first application of pose estimation technology to the detection and analysis of Yinghong No.9 tea leaves. The results show that the proposed algorithm can effectively estimate the pose of tea leaves, thus providing a reference for subsequent tea research.

Advancing human pose estimation with transformer models: An experimental approach

This paper explores the integration of Transformer architectures into human pose estimation, a critical task in computer vision that involves detecting human figures and predicting their poses by identifying body joint positions. With applications ranging from enhancing interactive gaming experiences to advancing biomechanical analyses, human pose estimation demands high accuracy and flexibility, particularly in dynamic and partially occluded scenes. This study hypothesizes that Transformers, renowned for their ability to manage long-range dependencies and focus on relevant data parts through self-attention mechanisms, can significantly outperform existing deep learning methods such as Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs). We introduce the PoseTransformer, a hybrid model that combines the precise feature extraction capabilities of CNNs with the global contextual awareness of Transformers, aiming to set new standards for accuracy and adaptability in pose estimation tasks. The model's effectiveness is demonstrated through rigorous testing on benchmark datasets, showing substantial improvements over traditional approaches, especially in complex scenarios.

Lester: Rotoscope Animation through Video Object Segmentation and Tracking

This article introduces Lester, a novel method to automatically synthesize retro-style 2D animations from videos. The method approaches the challenge mainly as an object segmentation and tracking problem. Video frames are processed with the Segment Anything Model (SAM) and the resulting masks are tracked through subsequent frames with DeAOT, a method of hierarchical propagation for semi-supervised video object segmentation. The geometry of the masks’ contours is simplified with the Douglas–Peucker algorithm. Finally, facial traits, pixelation and a basic rim light effect can be optionally added. The results show that the method exhibits an excellent temporal consistency and can correctly process videos with different poses and appearances, dynamic shots, partial shots and diverse backgrounds. The proposed method provides a more simple and deterministic approach than diffusion models based video-to-video translation pipelines, which suffer from temporal consistency problems and do not cope well with pixelated and schematic outputs. The method is also more feasible than techniques based on 3D human pose estimation, which require custom handcrafted 3D models and are very limited with respect to the type of scenes they can process.

LiDARCapV2: 3D human pose estimation with human–object interaction from LiDAR point clouds

Human–object interactions in open environments are common in the real world. Estimating 3D human pose from data where objects occlude the human is a challenging task in biometrics. However, existing LiDAR-based human motion capture datasets lack occlusion scenarios between humans and objects. To overcome this limitation, we propose LiDARHuman51M, a new human–object interaction dataset captured by LiDAR in long-range outdoor scene. It includes human motion labels acquired by an IMU system and synchronous RGB images. Additionally, we present an occlusion-aware method, LiDARCapV2, for capturing human motion from LiDAR point clouds under human–object interaction settings. Our key insight is to overcome object interference in human feature extraction by introducing a module called AgNoise-Segment. A noise augmentation strategy introduced in the AgNoise-Segment module alleviates the dependency of the segmentation accuracy on the effectiveness of 3D human pose estimations. Furthermore, we propose a skeleton extraction module that integrates features learned from the AgNoise-Segment module and predicts the skeleton locations. Quantitative and qualitative experiments demonstrate that LiDARCapV2 can capture high-quality 3D human motion under human–object interaction settings. Experiments on the KITTI and Waymo datasets demonstrate that our method can be generalized to real-world open scenarios.

3D Human Pose Estimation Based on Wearable IMUs and Multiple Camera Views

The problem of 3D human pose estimation (HPE) has been the focus of research in recent years, yet precise estimation remains an under-explored challenge. In this paper, the merits of both multiview images and wearable IMUs are combined to enhance the process of 3D HPE. We build upon a state-of-the-art baseline while introducing three novelties. Initially, we enhance the precision of keypoint localization by substituting Gaussian kernels with Laplacian kernels in the generation of target heatmaps. Secondly, we incorporate orientation regularized network (ORN), which enhances cross-modal heatmap fusion by taking a weighted average of the top-scored values instead of solely relying on the maximum value. This not only improves robustness to outliers but also leads to higher accuracy in pose estimation. Lastly, we modify the limb length constraint in the conventional orientation regularized pictorial structure model (ORPSM) to improve the estimation of joint positions. Specifically, we devise a soft-coded binary term for limb length constraint, hence imposing a flexible and smoothed penalization and reducing sensitivity to hyperparameters. The experimental results using the TotalCapture dataset reveal a significant improvement, with a 10.3% increase in PCKh accuracy at the one-twelfth threshold and a 3.9 mm reduction in MPJPE error compared to the baseline.

DOT-SLAM: A Stereo Visual Simultaneous Localization and Mapping (SLAM) System with Dynamic Object Tracking Based on Graph Optimization.

Most visual simultaneous localization and mapping (SLAM) systems are based on the assumption of a static environment in autonomous vehicles. However, when dynamic objects, particularly vehicles, occupy a large portion of the image, the localization accuracy of the system decreases significantly. To mitigate this challenge, this paper unveils DOT-SLAM, a novel stereo visual SLAM system that integrates dynamic object tracking through graph optimization. By integrating dynamic object pose estimation into the SLAM system, the system can effectively utilize both foreground and background points for ego vehicle localization and obtain a static feature points map. To rectify the inaccuracies in depth estimation from stereo disparity directly on the foreground points of dynamic objects due to their self-similarity characteristics, a coarse-to-fine depth estimation method based on camera-road plane geometry is presented. This method uses rough depth to guide fine stereo matching, thereby obtaining the 3 dimensions (3D)spatial positions of feature points on dynamic objects. Subsequently, by establishing constraints on the dynamic object's pose using the road plane and non-holonomic constraints (NHCs) of the vehicle, reducing the initial pose uncertainty of dynamic objects leads to more accurate dynamic object initialization. Finally, by considering foreground points, background points, the local road plane, the ego vehicle pose, and dynamic object poses as optimization nodes, through the establishment and joint optimization of a nonlinear model based on graph optimization, accurate six degrees of freedom (DoFs) pose estimations are obtained for both the ego vehicle and dynamic objects. Experimental validation on the KITTI-360 dataset demonstrates that DOT-SLAM effectively utilizes features from the background and dynamic objects in the environment, resulting in more accurate vehicle trajectory estimation and a static environment map. Results obtained from a real-world dataset test reinforce the effectiveness.

LA-Net: An End-to-End Category-Level Object Attitude Estimation Network Based on Multi-Scale Feature Fusion and an Attention Mechanism

In category-level object pose estimation tasks, determining how to mitigate intra-class shape variations and improve pose estimation accuracy for complex objects remains a challenging problem to solve. To address this issue, this paper proposes a new network architecture, LA-Net, to efficiently ascertain object poses from features. Firstly, we extend the 3D graph convolution network architecture by introducing the LS-Layer (Linear Connection Layer), which enables the network to acquire features from different layers and perform multi-scale feature fusion. Secondly, LA-Net employs a novel attention mechanism (PSA) and a Max-Pooling layer to extract local and global geometric information, which enhances the network’s ability to perceive object poses. Finally, the proposed LA-Net recovers the rotation information of an object by decoupling the rotation mechanism. The experimental results show that LA-Net can has much better accuracy in object pose estimation compared to the baseline method (HS-Pose). Especially for objects with complex shapes, its performance is 8.2% better for the 10°5 cm metric and 5% better for the 10°2 cm metric.

Three-dimensional human pose estimation based on contact pressure

Three-dimensional human pose estimation based on contact pressure

Marker Data Enhancement For Markerless Motion Capture.

Human pose estimation models can measure movement from videos at a large scale and low cost; however, open-source pose estimation models typically detect only sparse keypoints, which leads to inaccurate joint kinematics. OpenCap, a freely available service for researchers to measure movement from videos, addresses this issue using a deep learning model-the marker enhancer-that transforms sparse keypoints into dense anatomical markers. However, OpenCap performs poorly on movements not included in the training data. Here, we create a much larger and more diverse training dataset and develop a more accurate and generalizable marker enhancer. We compiled marker-based motion capture data from 1176 subjects and synthesized 1433 hours of keypoints and anatomical markers to train the marker enhancer. We evaluated its accuracy in computing kinematics using both benchmark movement videos and synthetic data representing unseen, diverse movements. The marker enhancer improved kinematic accuracy on benchmark movements (mean error: 4.1°, max: 8.7°) compared to using video keypoints (mean: 9.6°, max: 43.1°) and OpenCap's original enhancer (mean: 5.3°, max: 11.5°). It also better generalized to unseen, diverse movements (mean: 4.1°, max: 6.7°) than OpenCap's original enhancer (mean: 40.4°, max: 252.0°). Our marker enhancer demonstrates both accuracy and generalizability across diverse movements. We integrated the marker enhancer into OpenCap, thereby offering its thousands of users more accurate measurements across a broader range of movements.

TCFormer: Visual Recognition via Token Clustering Transformer.

Transformers are widely used in computer vision areas and have achieved remarkable success. Most state-of-the-art approaches split images into regular grids and represent each grid region with a vision token. However, fixed token distribution disregards the semantic meaning of different image regions, resulting in sub-optimal performance. To address this issue, we propose the Token Clustering Transformer (TCFormer), which generates dynamic vision tokens based on semantic meaning. Our dynamic tokens possess two crucial characteristics: (1) Representing image regions with similar semantic meanings using the same vision token, even if those regions are not adjacent, and (2) concentrating on regions with valuable details and represent them using fine tokens. Through extensive experimentation across various applications, including image classification, human pose estimation, semantic segmentation, and object detection, we demonstrate the effectiveness of our TCFormer. The code and models for this work are available at https://github.com/zengwang430521/TCFormer.

Estimation of human body 3D pose for parent-infant interaction settings using azure Kinect and OpenPose

Automatic pose estimation has become a valuable tool for the study of human behavior, including dyadic interactions. It allows researchers to analyze the nuanced dynamics of interactions more effectively, and facilitates the integration of behavioral data with other modalities (EEG, etc.). However, many technical difficulties remain. Particularly, for parent-infant interactions, automatic pose estimation for infants is unpredictable; the immature proportions and smaller bodies of children may cause misdetections. OpenPose is one tool that has shown high performance in pose tracking from video, even in infants. However, OpenPose is limited to 2D (i.e., coordinates relative to the image space). This may be undesirable in a multitude of paradigms (e.g., naturalistic settings). We developed a method for expanding the functionality of OpenPose to 3D, tailored to parent-infant interaction paradigms. This method merges the estimations from OpenPose with the depth information from a depth camera to obtain a 3D pose that works even for young infants.•Video recordings of interactions of parents and infants are taken using a dual color-depth camera.•2D-positions of parents and their infants are estimated from the color video.•Using the depth camera, we transform the 2D estimations into real-world 3D positions, allowing movement analysis in full-3D space.

Proposal and validation of a new approach in tele-rehabilitation with 3D human posture estimation: a randomized controlled trial in older individuals with sarcopenia

ObjectiveThrough a randomized controlled trial on older adults with sarcopenia, this study compared the training effects of an AI-based remote training group using deep learning-based 3D human pose estimation technology with those of a face-to-face traditional training group and a general remote training group.MethodsSeventy five older adults with sarcopenia aged 60–75 from community organizations in Changchun city were randomly divided into a face-to-face traditional training group (TRHG), a general remote training group (GTHG), and an AI-based remote training group (AITHG). All groups underwent a 3-month program consisting of 24-form Taichi exercises, with a frequency of 3 sessions per week and each session lasting 40 min. The participants underwent Appendicular Skeletal Muscle Mass Index (ASMI), grip strength, 6-meter walking pace, Timed Up and Go test (TUGT), and quality of life score (QoL) tests before the experiment, during the mid-term, and after the experiment. This study used SPSS26.0 software to perform one-way ANOVA and repeated measures ANOVA tests to compare the differences among the three groups. A significance level of p < 0.05 was defined as having significant difference, while p < 0.01 was defined as having a highly significant difference.Results(1) The comparison between the mid-term and pre-term indicators showed that TRHG experienced significant improvements in ASMI, 6-meter walking pace, and QoL (p < 0.01), and a significant improvement in TUGT timing test (p < 0.05); GTHG experienced extremely significant improvements in 6-meter walking pace and QoL (p < 0.01); AITHG experienced extremely significant improvements in ASMI, 6-meter walking pace, and QoL (p < 0.01), and a significant improvement in TUGT timing test (p < 0.05). (2) The comparison between the post-term and pre-term indicators showed that TRHG experienced extremely significant improvements in TUGT timing test (p < 0.01); GTHG experienced significant improvements in ASMI and TUGT timing test (p < 0.05); and AITHG experienced extremely significant improvements in TUGT timing test (p < 0.01). (3) During the mid-term, there was no significant difference among the groups in all tests (p > 0.05). The same was in post-term tests (p > 0.05).ConclusionCompared to the pre-experiment, there was no significant difference at the post- experiment in the recovery effects on the muscle quality, physical activity ability, and life quality of patients with sarcopenia between the AI-based remote training group and the face-to-face traditional training group. 3D pose estimation is equally as effective as traditional rehabilitation methods in enhancing muscle quality, functionality and life quality in older adults with sarcopenia.Trial registrationThe trial was registered in ClinicalTrials.gov (NCT05767710).