Pose Dataset Research Articles

Multi-person 3D pose estimation from a single image captured by a fisheye camera

Multi-person 3D pose estimation with absolute depths for a fisheye camera is a challenging task but with valuable applications in daily life, especially for video surveillance. However, to the best of our knowledge, such problem has not been explored so far, leaving a gap in practical applications. In this work, we first propose a method for multi-person 3D pose estimation from a single image taken by a fisheye camera. Our method consists of two branches to estimate absolute 3D human poses: (1) a 2D-to-3D lifting module to predict root-relative 3D human poses (HPoseNet); (2) a root regression module to estimate absolute root locations in the camera coordinate (HRootNet). Finally, we propose a fisheye re-projection module without using ground-truth camera parameters to connect two branches, alleviating the impact of image distortions on 3D pose estimation and further regularizing prediction absolute 3D poses. Experimental results demonstrate that our method achieves the state-of-the-art performance on two public multi-person 3D pose datasets with synthetic fisheye images and our newly collected dataset with real fisheye images. The code and new dataset will be made publicly available.

Improved Graph Convolutional Neural Network for Dance Tracking and Pose Estimation.

Movement recognition technology is widely used in various practical application scenarios, but there are few researches on dance movement recognition at present. Aiming at the problem of low accuracy of dance movement recognition due to complex pose changes in dance movements, this paper designed an improved graph convolutional neural network algorithm for dance tracking and pose estimation. In this method, the spatial and temporal characteristics of motion are extracted from the skeleton joint diagram of human body. Then, GCN (graph convolutional neural) is used to extract potential spatial information between skeleton nodes. Finally, LSTM (long short-term memory) extracts the time series features before and after human actions as a supplement and performs late fusion of the prediction outputs of the two networks, respectively, to improve the problem of insufficient generalization ability of single network. The experimental results show that this method can effectively improve the accuracy of dance movement recognition in general movement recognition data set and dance pose data set. It has certain application value in dance self-help teaching, professional dancer movement correction, and other application scenarios.

Animal pose estimation: A closer look at the state-of-the-art, existing gaps and opportunities

Over the past few years, research on animal pose estimation in computer vision field has grown in many aspects such as 2D and 3D pose estimation, 3D mesh reconstruction, and behavior prediction. Promoted by deep learning, more and more animal pose estimation tools and animal pose datasets have also been made publicly available. However, compared to human pose estimation, which already has high accuracy and high applicability for complex scenes, animal pose estimation is still at a preliminary stage. The huge domain shift between each species, the scarce datasets, and uncooperative research subjects all pose intractable challenges to the development of robust and accurate animal pose estimation algorithms. In this review paper, we summarize the recent (from 2013 to 2021) work in animal pose estimation from computer vision perspective in order to present the state-of-the-art approaches and highlight the challenges they face in this field. We first categorize the various methods of animal pose estimation and present them according to several keywords. Also, we sort and introduce the released annotated image, video, and 3D models of animal poses as well as a promising substitute for real dataset. We also report the performances of the existing algorithms and visualize their results. Finally, we provide an in-depth analysis of the persisting obstacles in this field based on existing work, and offer potential solutions.

Structured Context Enhancement Network for Mouse Pose Estimation

Automated analysis of mouse behaviours is crucial for many applications in neuroscience. However, quantifying mouse behaviours from videos or images remains a challenging problem, where pose estimation plays an important role in describing mouse behaviours. Although deep learning based methods have made promising advances in human pose estimation, they cannot be directly applied to pose estimation of mice due to different physiological natures. Particularly, since mouse body is highly deformable, it is a challenge to accurately locate different keypoints on the mouse body. In this paper, we propose a novel Hourglass network based model, namely Graphical Model based Structured Context Enhancement Network (GM-SCENet) where two effective modules, i.e., Structured Context Mixer (SCM) and Cascaded Multi-level Supervision (CMLS) are subsequently implemented. SCM can adaptively learn and enhance the proposed structured context information of each mouse part by a novel graphical model that takes into account the motion difference between body parts. Then, the CMLS module is designed to jointly train the proposed SCM and the Hourglass network by generating multi-level information, increasing the robustness of the whole network. Using the multi-level prediction information from SCM and CMLS, we develop an inference method to ensure the accuracy of the localisation results. Finally, we evaluate our proposed approach against several baselines on our Parkinson’s Disease Mouse Behaviour (PDMB) and the standard DeepLabCut Mouse Pose datasets. The experimental results show that our method achieves better or competitive performance against the other state-of-the-art approaches.

A Large-Scale Mouse Pose Dataset for Mouse Pose Estimation

Mouse pose estimations have important applications in the fields of animal behavior research, biomedicine, and animal conservation studies. Accurate and efficient mouse pose estimations using computer vision are necessary. Although methods for mouse pose estimations have developed, bottlenecks still exist. One of the most prominent problems is the lack of uniform and standardized training datasets. Here, we resolve this difficulty by introducing the mouse pose dataset. Our mouse pose dataset contains 40,000 frames of RGB images and large-scale 2D ground-truth motion images. All the images were captured from interacting lab mice through a stable single viewpoint, including 5 distinct species and 20 mice in total. Moreover, to improve the annotation efficiency, five keypoints of mice are creatively proposed, in which one keypoint is at the center and the other two pairs of keypoints are symmetric. Then, we created simple, yet effective software that works for annotating images. It is another important link to establish a benchmark model for 2D mouse pose estimations. We employed modified object detections and pose estimation algorithms to achieve precise, effective, and robust performances. As the first large and standardized mouse pose dataset, our proposed mouse pose dataset will help advance research on animal pose estimations and assist in application areas related to animal experiments.

A Survey on GAN-Based Data Augmentation for Hand Pose Estimation Problem

Deep learning solutions for hand pose estimation are now very reliant on comprehensive datasets covering diverse camera perspectives, lighting conditions, shapes, and pose variations. While acquiring such datasets is a challenging task, several studies circumvent this problem by exploiting synthetic data, but this does not guarantee that they will work well in real situations mainly due to the gap between the distribution of synthetic and real data. One recent popular solution to the domain shift problem is learning the mapping function between different domains through generative adversarial networks. In this study, we present a comprehensive study on effective hand pose estimation approaches, which are comprised of the leveraged generative adversarial network (GAN), providing a comprehensive training dataset with different modalities. Benefiting from GAN, these algorithms can augment data to a variety of hand shapes and poses where data manipulation is intuitively controlled and greatly realistic. Next, we present related hand pose datasets and performance comparison of some of these methods for the hand pose estimation problem. The quantitative and qualitative results indicate that the state-of-the-art hand pose estimators can be greatly improved with the aid of the training data generated by these GAN-based data augmentation methods. These methods are able to beat the baseline approaches with better visual quality and higher values in most of the metrics (PCK and ME) on both the STB and NYU datasets. Finally, in conclusion, the limitation of the current methods and future directions are discussed.

Consensus-Based Optimization for 3D Human Pose Estimation in Camera Coordinates

3D human pose estimation is frequently seen as the task of estimating 3D poses relative to the root body joint. Alternatively, we propose a 3D human pose estimation method in camera coordinates, which allows effective combination of 2D annotated data and 3D poses and a straightforward multi-view generalization. To that end, we cast the problem as a view frustum space pose estimation, where absolute depth prediction and joint relative depth estimations are disentangled. Final 3D predictions are obtained in camera coordinates by the inverse camera projection. Based on this, we also present a consensus-based optimization algorithm for multi-view predictions from uncalibrated images, which requires a single monocular training procedure. Although our method is indirectly tied to the training camera intrinsics, it still converges for cameras with different intrinsic parameters, resulting in coherent estimations up to a scale factor. Our method improves the state of the art on well known 3D human pose datasets, reducing the prediction error by 32% in the most common benchmark. We also reported our results in absolute pose position error, achieving 80 mm for monocular estimations and 51 mm for multi-view, on average. Source code is available at https://github.com/dluvizon/3d-pose-consensus .

STEREO VISION INFORMATION SYSTEM USING MEDIAN THEOREM AND ATTITUDE COMPENSATION WITH NONLINEAR DIFFERENTIAL EQUATIONS

Using computer vision technology to obtain and analyze biomechanical information is an important research direction in recent years. However, the linear model in the computer vision system cannot accurately describe the geometric relationship of the camera imaging, so it is difficult to realize human posture recognition in high-precision mechanics information. Therefore, how to improve the recognition accuracy is very important. In this paper, we apply nonlinear differential equations to stereo computer vision (SCV) information systems. And based on the median theorem, a nonlinear posture recognition and error compensation algorithm based on BP neural network is proposed to reduce the recognition error. The test set uses the Leeds Motion Pose (LSP) dataset to verify the performance of the algorithm. Experimental results show that the compensated median filter of BP neural network can eliminate glitches in attitude data. Superimposing the output attitude error compensation value with the attitude estimation value can greatly reduce the root-mean-square error of the attitude angle. The result of gesture recognition is closer to reality. Compared with traditional algorithms, the cyclomatic complexity of the proposed BP neural network algorithm has a much lower growth rate in high-order calculations, which indicates that the proposed BP neural network algorithm is more concise and scalable.

Supine Infant Pose Estimation via Single Depth Image

objective: The infant pose estimation is very important for the evaluation of infants’ brain development and motor quality. Here, we propose a new method to quickly build a large-scale depth image pose dataset, and on this basis, establish the XJTU-IDP dataset, then we also propose a new method based on joint feature coding to estimate the infants’ posture from depth image. Methods: First, we use Kinect V2 to collect depth and color images synchronously, and use the proposed framework to build a large dataset. Then we encode the offset vector in the confidence region of each joint of the infant, and export the joint positioning and connection encoding through our convolutional neural network (CNN) model. The proposed method is tested from two datasets (including 16 and 27 clinical actual infants, respectively). Results: When applied to pose estimation, the median root mean square distance (RMSD) calculated among all limbs is 4.12 pixels, which is much lower than the baseline (9.06 pixels) on babyPose dataset and RMSD in XJTU-IDP dataset is 3.93 pixels, which achieves the best in comparison with the state-of-the-arts. Conclusion: The proposed framework of the building dataset in our work is effective, and the joint feature coding can enrich the network information. Significance: Our work not only provides a larger infant pose dataset to make up for the deficiencies of babyPose, but also proposes a new framework for quickly establishing high-quality dataset. The proposed pose estimation method significantly improves the state of art in automatic measurement of infants’ health status.

Implicit Semantic Data Augmentation for Hand Pose Estimation

Data augmentation is a well-known technique used for improving the generalization performance of modern neural networks. After the success of several traditional random data augmentation for images (including flipping, translation, or rotation), a recent surge of interest in implicit data augmentation techniques occurs to complement random data augmentation techniques. Implicit data augmentation augments training samples in feature space, rather than in pixel space, resulting in the generation of semantically meaningful data. Several techniques on implicit data augmentation have been introduced for classification tasks. However, few approaches have been introduced for regression tasks with continuous/structured labels, such as object pose estimation. Hence, we are motivated to propose a method for implicit semantic data augmentation for hand pose estimation. By considering semantic distances of hand poses, the proposed method implicitly generates extra training samples in feature space. We propose two additional techniques to improve the performance of this augmentation: metric learning and hand-dependent augmentation. Metric learning aims to learn feature representations to reflect the semantic distance of data. For hand pose estimation, the distribution of augmented hand poses can be regulated by managing the distribution of feature representations. Meanwhile, hand-dependent augmentation is specifically designed for hand pose estimation to prevent semantically meaningless hand poses from being generated (e.g., hands generated by simple interpolation between both hands). Further, we demonstrate the effectiveness of the proposed technique using two well-known hand pose datasets: STB and RHD.

Towards human-level performance on automatic pose estimation of infant spontaneous movements.

Assessment of spontaneous movements can predict the long-term developmental disorders in high-risk infants. In order to develop algorithms for automated prediction of later disorders, highly precise localization of segments and joints by infant pose estimation is required. Four types of convolutional neural networks were trained and evaluated on a novel infant pose dataset, covering the large variation in 1424 videos from a clinical international community. The localization performance of the networks was evaluated as the deviation between the estimated keypoint positions and human expert annotations. The computational efficiency was also assessed to determine the feasibility of the neural networks in clinical practice. The best performing neural network had a similar localization error to the inter-rater spread of human expert annotations, while still operating efficiently. Overall, the results of our study show that pose estimation of infant spontaneous movements has a great potential to support research initiatives on early detection of developmental disorders in children with perinatal brain injuries by quantifying infant movements from video recordings with human-level performance.

Single Image Depth Map Estimation for Improving Posture Recognition

Image-based posture recognition is a very challenging problem since it is difficult to acquire rich 3D information from the posture in color image. To address this issue, we present a novel and unified framework for human posture recognition, applying single image depth map estimation from color images. The proposed method includes two stages. The first stage estimates the depth map from the single-color image by an improved Pix2Pix generation module. The generation module is equipped with a hybrid loss function that captures the high-level features and recovers the sharp depth discontinuities, thus improving the depth estimation results. The second stage (the recognition stage) improves the color image-based recognition performance by incorporating the estimated depth map. Thereby, a two-stream CNN architecture that separately processes the color image and its estimated depth image is developed for robust posture recognition. To verify its effectiveness, we first test the proposed method on a novel pose dataset, which contains 13800 samples of paired color-and-depth of 6 subjects with 15 poses. The dataset used in this work is been created and released, is available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">http://media.ritsumei.ac.jp/iipl/database/pose/</uri> . Extensive experiments are also performed on the public OUHANDS hand gesture dataset. Experiments demonstrate that the proposed method achieves superior performance on both human pose and hand gesture recognition tasks.

Syntactic model-based human body 3D reconstruction and event classification via association based features mining and deep learning.

The study of human posture analysis and gait event detection from various types of inputs is a key contribution to the human life log. With the help of this research and technologies humans can save costs in terms of time and utility resources. In this paper we present a robust approach to human posture analysis and gait event detection from complex video-based data. For this, initially posture information, landmark information are extracted, and human 2D skeleton mesh are extracted, using this information set we reconstruct the human 2D to 3D model. Contextual features, namely, degrees of freedom over detected body parts, joint angle information, periodic and non-periodic motion, and human motion direction flow, are extracted. For features mining, we applied the rule-based features mining technique and, for gait event detection and classification, the deep learning-based CNN technique is applied over the mpii-video pose, the COCO, and the pose track datasets. For the mpii-video pose dataset, we achieved a human landmark detection mean accuracy of 87.09% and a gait event recognition mean accuracy of 90.90%. For the COCO dataset, we achieved a human landmark detection mean accuracy of 87.36% and a gait event recognition mean accuracy of 89.09%. For the pose track dataset, we achieved a human landmark detection mean accuracy of 87.72% and a gait event recognition mean accuracy of 88.18%. The proposed system performance shows a significant improvement compared to existing state-of-the-art frameworks.

Vogtareuth Rehab Depth Datasets: Benchmark for Marker-less Posture Estimation in Rehabilitation.

Posture estimation using a single depth camera has become a useful tool for analyzing movements in rehabilitation. Recent advances in posture estimation in computer vision research have been possible due to the availability of large-scale pose datasets. However, the complex postures involved in rehabilitation exercises are not represented in the existing benchmark depth datasets. To address this limitation, we propose two rehabilitation-specific pose datasets containing depth images and 2D pose information of patients, both adult and children, performing rehab exercises. We use a state-of-the-art marker-less posture estimation model which is trained on a non-rehab benchmark dataset. We evaluate it on our rehab datasets, and observe that the performance degrades significantly from non-rehab to rehab, highlighting the need for these datasets. We show that our dataset can be used to train pose models to detect rehab-specific complex postures. The datasets will be released for the benefit of the research community.

Risk assessment for musculoskeletal disorders based on the characteristics of work posture

Construction workers are at high risk of work-related musculoskeletal disorders (WMSDs). Although the existing observational assessment methods are easy to use, when it comes to a more in-depth statistical analysis of the dynamic characteristics of the worker's operation, the sample data to be processed turn out to be large, the labor cost high, and the analysis easily affected by the prejudice of the evaluator. This study examines a novel WMSD prediction method based on the dynamic characteristics of the working posture, which comprises three artificial intelligence algorithms in series. In this method, the posture detector identifies the limb angles and state in the working video, the posture risk evaluator evaluates the risk level of the working posture frame by frame, and the task risk predictor predicts the risk level of the current work process. The collected video data of common tasks of construction workers and the MPII Human Pose dataset were used for training and evaluation of the algorithms. The method achieved 87.0% accuracy of the joint point recognition. The micro-averaged accuracy, recall, and F1-score (harmonic average of accuracy and recall) reached 96.7%, 96.0%, and 96.6%, respectively. The results showed that the proposed method has great potential for real-time risk assessment. It can output all of the changes of the limb angles of workers in the work process frame by frame and predict the risk level of the whole work process.

Seeing under the cover with a 3D U-Net: point cloud-based weight estimation of covered patients

PurposeBody weight is a crucial parameter for patient-specific treatments, particularly in the context of proper drug dosage. Contactless weight estimation from visual sensor data constitutes a promising approach to overcome challenges arising in emergency situations. Machine learning-based methods have recently been shown to perform accurate weight estimation from point cloud data. The proposed methods, however, are designed for controlled conditions in terms of visibility and position of the patient, which limits their practical applicability. In this work, we aim to decouple accurate weight estimation from such specific conditions by predicting the weight of covered patients from voxelized point cloud data.MethodsWe propose a novel deep learning framework, which comprises two 3D CNN modules solving the given task in two separate steps. First, we train a 3D U-Net to virtually uncover the patient, i.e. to predict the patient’s volumetric surface without a cover. Second, the patient’s weight is predicted from this 3D volume by means of a 3D CNN architecture, which we optimized for weight regression.ResultsWe evaluate our approach on a lying pose dataset (SLP) under two different cover conditions. The proposed framework considerably improves on the baseline model by up to {16}{%} and reduces the gap between the accuracy of weight estimates for covered and uncovered patients by up to {52}{%}.ConclusionWe present a novel pipeline to estimate the weight of patients, which are covered by a blanket. Our approach relaxes the specific conditions that were required for accurate weight estimates by previous contactless methods and thus constitutes an important step towards fully automatic weight estimation in clinical practice.

Head pose estimation using deep neural networks and 3D point clouds

In this paper, we propose head pose estimation using deep neural networks and 3D point cloud. Unlike existing methods that either take 2D RGB image or 2D depth image as input, we adopt 3D point cloud data generated from depth to estimate 3D head poses. To further improve robustness and accuracy of head pose estimation, we classify 3D angles of head poses into 36 classes with 5 ∘ interval and predict the probability of each angle in a class based on multi-layer perceptron (MLP). While traditional iterative methods for head model construction require high computation and memory costs, the proposed method is lightweight and computationally efficient by utilizing a sampled 3D point cloud as input combined with a graph convolutional neural network (GCNN). Experimental results on Biwi Kinect Head Pose dataset show that the proposed method achieves outstanding performance in head pose estimation and outperforms state-of-the-art ones in terms of accuracy.

SD-Pose: Semantic Decomposition for Cross-Domain 6D Object Pose Estimation

The current leading 6D object pose estimation methods rely heavily on annotated real data, which is highly costly to acquire. To overcome this, many works have proposed to introduce computer-generated synthetic data. However, bridging the gap between the synthetic and real data remains a severe problem. Images depicting different levels of realism/semantics usually have different transferability between the synthetic and real domains. Inspired by this observation, we introduce an approach, SD-Pose, that explicitly decomposes the input image into multi-level semantic representations and then combines the merits of each representation to bridge the domain gap. Our comprehensive analyses and experiments show that our semantic decomposition strategy can fully utilize the different domain similarities of different representations, thus allowing us to outperform the state of the art on modern 6D object pose datasets without accessing any real data during training.

VRU Pose-SSD: Multiperson Pose Estimation For Automated Driving

We present a fast and efficient approach for joint person detection and pose estimation optimized for automated driving (AD) in urban scenarios. We use a multitask weight sharing architecture to jointly train detection and pose estimation. This modular architecture allows us to accommodate different downstream tasks in the future. By systematic large-scale experiments on the Tsinghua-Daimler Urban Pose Dataset (TDUP), we obtain multiple models with varying accuracy-speed trade-offs. We then quantize and optimize our network for deployment and present a detailed analysis of the efficacy of the algorithm. We introduce a two-stage evaluation strategy, which is more suitable for AD and achieve a significant performance improvement in comparison to state-of-the-art approaches. Our optimized model runs at 52~fps on full HD images and still reaches a competitive performance of 32.25~LAMR. We are confident that our work serves as an enabler to tackle higher-level tasks like VRU intention estimation and gesture recognition, which rely on stable pose estimates and will play a crucial role in future AD systems.

ASPset: An outdoor sports pose video dataset with 3D keypoint annotations

Recent advances in deep learning approaches to computer vision problems have led to renewed interest in the task of predicting 3D human joint locations from raw image data, with application areas including sports analysis, human-computer interaction, and physical rehabilitation. Although supervised learning of deep neural networks has proven to be effective for pose estimation, it requires a wealth of varied data to generalise well to previously unseen examples. Consequently, progress in 3D human pose estimation has been slowed by the fact that 3D keypoint annotations are notoriously difficult to obtain, traditionally requiring a large array of cameras and/or the use of wearable markers/sensors. In this paper we describe a methodology for obtaining 3D human pose annotations using only three video cameras and without any wearables. We apply this methodology to construct ASPset-510 (Australian Sports Pose Dataset), a large collection of natural sports-related video with 3D pose annotations. Using ASPset-510 as an additional source of training examples, we found that we could improve pose model generalisation on the established MPI-INF-3DHP benchmark. We make ASPset-510 publicly available, and provide strong baseline results for future work to compare against.