Three-dimensional Pose Estimation Research Articles

Fall-portent detection for construction sites based on computer vision and machine learning

PurposeFall accidents can cause casualties and economic losses in the construction industry. Fall portents, such as loss of balance (LOB) and sudden sways, can result in fatal, nonfatal or attempted fall accidents. All of them are worthy of studying to take measures to prevent future accidents. Detecting fall portents can proactively and comprehensively help managers assess the risk to workers as well as in the construction environment and further prevent fall accidents.Design/methodology/approachThis study focused on the postures of workers and aimed to directly detect fall portents using a computer vision (CV)-based noncontact approach. Firstly, a joint coordinate matrix generated from a three-dimensional pose estimation model is employed, and then the matrix is preprocessed by principal component analysis, K-means and pre-experiments. Finally, a modified fusion K-nearest neighbor-based machine learning model is built to fuse information from the x, y and z axes and output the worker's pose status into three stages.FindingsThe proposed model can output the worker's pose status into three stages (steady–unsteady–fallen) and provide corresponding confidence probabilities for each category. Experiments conducted to evaluate the approach show that the model accuracy reaches 85.02% with threshold-based postprocessing. The proposed fall-portent detection approach can extract the fall risk of workers in the both pre- and post-event phases based on noncontact approach.Research limitations/implicationsFirst, three-dimensional (3D) pose estimation needs sufficient information, which means it may not perform well when applied in complicated environments or when the shooting distance is extremely large. Second, solely focusing on fall-related factors may not be comprehensive enough. Future studies can incorporate the results of this research as an indicator into the risk assessment system to achieve a more comprehensive and accurate evaluation of worker and site risk.Practical implicationsThe proposed machine learning model determines whether the worker is in a status of steady, unsteady or fallen using a CV-based approach. From the perspective of construction management, when detecting fall-related actions on construction sites, the noncontact approach based on CV has irreplaceable advantages of no interruption to workers and low cost. It can make use of the surveillance cameras on construction sites to recognize both preceding events and happened accidents. The detection of fall portents can help worker risk assessment and safety management.Originality/valueExisting studies using sensor-based approaches are high-cost and invasive for construction workers, and others using CV-based approaches either oversimplify by binary classification of the non-entire fall process or indirectly achieve fall-portent detection. Instead, this study aims to detect fall portents directly by worker's posture and divide the entire fall process into three stages using a CV-based noncontact approach. It can help managers carry out more comprehensive risk assessment and develop preventive measures.

Three-Dimensional Human Pose Estimation from Sparse IMUs through Temporal Encoder and Regression Decoder

Three-dimensional (3D) pose estimation has been widely used in many three-dimensional human motion analysis applications, where inertia-based path estimation is gradually being adopted. Systems based on commercial inertial measurement units (IMUs) usually rely on dense and complex wearable sensors and time-consuming calibration, causing intrusions to the subject and hindering free body movement. The sparse IMUs-based method has drawn research attention recently. Existing sparse IMUs-based three-dimensional pose estimation methods use neural networks to obtain human poses from temporal feature information. However, these methods still suffer from issues, such as body shaking, body tilt, and movement ambiguity. This paper presents an approach to improve three-dimensional human pose estimation by fusing temporal and spatial features. Based on a multistage encoder-decoder network, a temporal convolutional encoder and human kinematics regression decoder were designed. The final three-dimensional pose was predicted from the temporal feature information and human kinematic feature information. Extensive experiments were conducted on two benchmark datasets for three-dimensional human pose estimation. Compared to state-of-the-art methods, the mean per joint position error was decreased by 13.6% and 19.4% on the total capture and DIP-IMU datasets, respectively. The quantitative comparison demonstrates that the proposed temporal information and human kinematic topology can improve pose accuracy.

PYNet: Poseclass and Yaw Angle Output Network for Object Pose Estimation

The issues of estimating the poses of simple-shaped objects, such as retail store goods, have been addresses to ease the grasping of objects by robots. Conventional methods to estimate poses with an RGBD camera mounted on robots have difficulty estimating the three-dimensional poses of simple-shaped objects with few shape features. Therefore, in this study, we propose a new class called “poseclass” to indicate the grounding face of an object. The poseclass is of discrete value and solvable as a classification problem; it can be estimated with high accuracy; in addition, the three-dimensional pose estimation problems can be simplified into one-dimensional pose-estimation problem to estimate the yaw angles on the grounding face. We have developed a new neural network (PYNet) to estimate the poseclass and yaw angle, and compared it with conventional methods to determine its ratio of estimating unknown simple-shaped object poses with an angle error of 30° or less. The ratio of PYNet (68.9%) is an 18.1 pt higher than that of the conventional methods (50.8%). Additionally, a PYNet-implemented robot successfully grasped convenience store goods.

OkeyDoggy3D: A Mobile Application for Recognizing Stress-Related Behaviors in Companion Dogs Based on Three-Dimensional Pose Estimation through Deep Learning

Dogs often express their stress through physical motions that can be recognized by their owners. We propose a mobile application that analyzes companion dog’s behavior and their three-dimensional poses via deep learning. As existing research on pose estimation has focused on humans, obtaining a large dataset comprising images showing animal joint locations is a challenge. Nevertheless, we generated such a dataset and used it to train an AI model. Furthermore, we analyzed circling behavior, which is associated with stress in companion dogs. To this end, we used the VideoPose3D model to estimate the 3D poses of companion dogs from the 2D pose estimation technique derived by the DeepLabCut model and developed a mobile app that provides analytical information on the stress-related behaviors, as well as the walking and isolation times, of companion dogs. Finally, we interviewed five certified experts to evaluate the validity and applicability of the app.

Three-dimensional pose estimation of deformable linear object tips based on a low-cost, two-dimensional sensor setup and AI-based evaluation

Handling and processing rigid objects is a common task for industrial robots. However, regarding flexible objects, this task is considerably more complex because gravity and other external forces influence the shape of the objects. Moreover, especially deformable linear objects like cables can show a large deformation. Therefore, regarding the automation of handling and assembly processes with robots, it is crucial to determine the current state of the shape of the cables. Furthermore, to significantly reduce the engineering cost and time, we introduce a method for utilizing synthetic data and thus reduce data acquisition and preparation efforts. The basic approach of detection is separated into four steps. Firstly, object detection identifies and classifies cable tips from two two-dimensional pictures taken from different perspectives. Secondly, keypoint detection identifies the pixel defining the start and endpoint of the identified object class. Thirdly, a triangulation merges the evaluations of the single images with the geometrical arrangement of the setup. Finally, a correction calculation compensates systematic errors caused by the inexactness of the physical camera setup based on an initially performed calibration algorithm performed for each new physical setup. The results presented in this paper prove the feasibility and advantage of the approach and show the impact of different characteristics on the accuracy of the three-dimensional pose estimation.

Deep Learning for Ballroom Dance Recognition: A Temporal and Trajectory-Aware Classification Model With Three-Dimensional Pose Estimation and Wearable Sensing

Dance performance recognition methods have been investigated and shown various applications such as picture-pose evaluation and synchronizing foot timing and direction. However, detailed analysis and feedback are still missing. To provide them, understanding the performance by component level is necessary. Specifically, we formulate it as a dance-figure classification problem using three-dimensional body joints and wearable sensors. Our model is based on long short-term memory (LSTM) and includes the temporal and trajectory-wise structure that uses the trajectory information in a timestep and the temporal masking module. As a result, we achieved 93% accuracy with our proposed method, which is highly overwhelming the baseline result (84.7%) and very close to the accuracy of the experienced dancers (93.6%). We have made the dataset of ballroom dance performance dataset open to researchers to develop the activity recognition field further.

Football Player Posture Detection Method Combining Foreground Detection and Neural Networks

In recent years, with the rapid development of artificial intelligence, information technology, intelligent digital video surveillance systems, real-time sports competition playback, and other technologies have emerged one after another, making the advantages of deep learning-based football posture detection tasks become more obvious. Related models and methods have been applied to the research field of sports posture estimation and have achieved great improvement, surpassing the traditional football posture estimation method based on manual design features in one fell swoop. In addition, the application of video foreground detection has developed rapidly and has great application value in sports analysis. Therefore, this paper proposes a novel football motion detection approach combining foreground detection and deep learning for real-time detection of football player posture. The main task of foreground target detection is to extract the interesting foreground target in the real monitoring scene and use it as the target of interest for subsequent analysis. Then, we propose a triple DetectNet detection framework based on deep learning technology, which can quickly and robustly realize the three-dimensional pose estimation of multiperson motion. For input, the triple DetectNet framework uses three neural networks and is executed in three stages; the first stage is to use the DetectNet (DN) network to detect the bounding box of each person separately, the second stage uses the 2DPoseNet (2DPN) network to estimate each of the corresponding two-dimensional poses of the individual, and the third stage uses the 3DPoseNet (3DPN) network to obtain the 3D pose of the person. This paper also conducted experiments on four datasets, and the results proved the superiority and success of this algorithm.

Three-Dimensional Pose Estimation of Infants Lying Supine Using Data From a Kinect Sensor With Low Training Cost

Early diagnosis of cerebral palsy in infants has produced promising results using tools like the General Movement Assessment (GMA). Pose estimation of infants lying supine is an important step towards an automated system for GMA. Developing methods for accurate, reliable, fast estimation of the three-dimensional (3D) position of human limbs and proposing motion features suitable for 3D models to classify typical and atypical movements are attracting increasing research interest lately. In this study, we propose a 3D pose estimation method with low training cost suitable for infants in lying positions. The method uses an existing two-dimensional human body keypoint detection method combined with the in-depth information in Red-Green-Blue Depth (RGB-D) data from a Kinect sensor. The method is evaluated using the Moving INfants In RGB-D (MINI-RGBD) open dataset. The results show that the average error of the estimated body part length is 13.76 mm, while the accuracy of the Percentage of Correctly-localized Parts (PCP) and Percentage of Correct Keypoint (PCK) is 80.7 and 86.1%, respectively. The results are comparable to those achieved in the baseline study performed by the researchers who generated this open dataset. The advantage of our method is its low training cost.

Three-dimensional pose estimation model for object with complex surface

The proposed three-dimensional pose estimation model for object with complex surface, which primarily absorbs the essence of scale-invariant feature transform and iterative closest point algorithm, includes two steps, off-line and online. At first, two kinds of feature databases are established in the off-line operations. Then, the online process mainly has three steps. The first one is two-dimensional edge extraction from red–green–blue (RGB) information based on scale-invariant feature transform algorithm. The second one is three-dimensional surface reconstruction from the previous two-dimensional edge and the depth information obtained from depth camera. The last one is three-dimensional pose estimation based on camera calibration and iterative closest point algorithm. The Kinect camera is selected as the information acquisition device which can produce red–green–blue information and depth information. In the experiment, the container twist-lock with complex surface is taken as the object. The result shows that the accuracy of the proposed model is very high.

Fast object localization and pose estimation in heavy clutter for robotic bin picking

We present a practical vision-based robotic bin-picking system that performs detection and three-dimensional pose estimation of objects in an unstructured bin using a novel camera design, picks up parts from the bin, and performs error detection and pose correction while the part is in the gripper. Two main innovations enable our system to achieve real-time robust and accurate operation. First, we use a multi-flash camera that extracts robust depth edges. Second, we introduce an efficient shape-matching algorithm called fast directional chamfer matching (FDCM), which is used to reliably detect objects and estimate their poses. FDCM improves the accuracy of chamfer matching by including edge orientation. It also achieves massive improvements in matching speed using line-segment approximations of edges, a three-dimensional distance transform, and directional integral images. We empirically show that these speedups, combined with the use of bounds in the spatial and hypothesis domains, give the algorithm sublinear computational complexity. We also apply our FDCM method to other applications in the context of deformable and articulated shape matching. In addition to significantly improving upon the accuracy of previous chamfer matching methods in all of the evaluated applications, FDCM is up to two orders of magnitude faster than the previous methods.

Robust solution to three-dimensional pose estimation using composite extended Kalman observer and Kalman filter

Three-dimensional (3D) pose estimation of a rigid object by only one camera has a vital role in visual servoing systems, and extended Kalman filter (EKF) is vastly used for this task in an unstructured environment. In this study, the stability of the EKF-based 3D pose estimators is analysed in detail. The most challenging issue of the state-of-the-art EKF-based 3D pose estimators is the possibility of its divergence because of the measurement and model noises. By analysing the stability of conventional EKF-based pose estimators a composite technique is proposed to guarantee the stability of the procedure. In the proposed technique, the non-linear-uncertain estimation problem is decomposed into a non-linear-certain observation in addition to a linear-uncertain estimation problem. The first part is handled using the extended Kalman observer and the second part is accomplished by a simple Kalman filter. Finally, some experimental and simulation results are given in order to verify the robustness of the method and compare the performance of the proposed method in noisy and uncertain environment to the conventional techniques.

Automated Three-Dimensional Pose Estimation of Objects using Multiple ID Devices by Autonomous Robot

This paper describes a method for automated estimation of three-dimensional pose (position and orientation) of objects using multiple ID devices such as radio frequency identification (RFID) devices read by autonomous robots. Robots achieve tasks based on the task information acquired from ID devices attached to the object. To acquire IDs from ID devices using an ID reader, a motion generation for acquisition of IDs is required; the ID devices must be within the detection range of the ID reader. The paper focuses on a method for acquiring pose data of an object by controlled motion of the stereo vision ID reader which detects the ID devices on the object. The data is then processed using a 3D geometric model. We show the feasibility of the method through an experiment for estimation of object pose.

Computationally efficient eigenspace decomposition of correlated images characterized by three parameters

Eigendecomposition is a common technique that is performed on sets of correlated images in a number of pattern recognition applications including object detection and pose estimation. However, many fast eigendecomposition algorithms rely on correlated images that are, at least implicitly, characterized by only one parameter, frequently time, for their computational efficacy. In some applications, e.g., three-dimensional pose estimation, images are correlated along multiple parameters and no natural one-dimensional ordering exists. In this work, a fast eigendecomposition algorithm that exploits the “temporal” correlation within image data sets characterized by one parameter is extended to improve the computational efficiency of computing the eigendecomposition for image data sets characterized by three parameters. The algorithm is implemented and evaluated using three-dimensional pose estimation as an example application. Its accuracy and computational efficiency are compared to that of the original algorithm applied to one-dimensional pose estimation.