Deep Learning-based Pose Estimation Research Articles

Machine learning-based gait analysis to predict clinical frailty scale in elderly patients with heart failure.

Although frailty assessment is recommended for guiding treatment strategies and outcome prediction in elderly patients with heart failure (HF), most frailty scales are subjective, and the scores vary among raters. We sought to develop a machine learning-based automatic rating method/system/model of the clinical frailty scale (CFS) for patients with HF. We prospectively examined 417 elderly (≥75 years) with symptomatic chronic HF patients from 7 centres between January 2019 and October 2023. The patients were divided into derivation (n = 194) and validation (n = 223) cohorts. We obtained body-tracking motion data using a deep learning-based pose estimation library, on a smartphone camera. Predicted CFS was calculated from 128 key features, including gait parameters, using the light gradient boosting machine (LightGBM) model. To evaluate the performance of this model, we calculated Cohen's weighted kappa (CWK) and intraclass correlation coefficient (ICC) between the predicted and actual CFSs. In the derivation and validation datasets, the LightGBM models showed excellent agreements between the actual and predicted CFSs [CWK 0.866, 95% confidence interval (CI) 0.807-0.911; ICC 0.866, 95% CI 0.827-0.898; CWK 0.812, 95% CI 0.752-0.868; ICC 0.813, 95% CI 0.761-0.854, respectively]. During a median follow-up period of 391 (inter-quartile range 273-617) days, the higher predicted CFS was independently associated with a higher risk of all-cause death (hazard ratio 1.60, 95% CI 1.02-2.50) after adjusting for significant prognostic covariates. Machine learning-based algorithms of automatically CFS rating are feasible, and the predicted CFS is associated with the risk of all-cause death in elderly patients with HF.

Abstract 16327: Machine Learning-Based Prediction of Clinical Frailty Scale Based on Gait Analysis in Elderly Patients With Heart Failure

Background: Although frailty assessment is recommended for guiding treatment strategies and outcome prediction in elderly patients with heart failure (HF), most frailty scales are subjective, and the scores vary among raters. We sought to develop a machine learning-based automatic rating method/system/model of the clinical frailty scale (CFS) for patients with HF. Methods: We prospectively examined 358 elderly (≥75 years) with symptomatic chronic HF patients (mean age 82.3±5.2 years, left ventricular ejection fraction 58 [IQR 45-67]%) from six sites between January 2019 and September 2022. The CFS was evaluated based on the modified Delphi method by 10 independent trained cardiologists as the gold standard. We obtained body-tracking motion data using OpenPose ® , a deep learning-based pose estimation library, on a smartphone camera. Predicted CFS was calculated from 128 key features, including gait parameters, using the extremely randomized trees (ERT) model. To evaluate the performance of this model, we calculated Cohen’s weighted kappa and intraclass correlation coefficient (ICC) between the predicted and actual CFSs. The studied patients were divided into derivation (from a university hospital) and validation (from other five sites) cohorts. Results: In the derivation (n = 181) and validation (n = 177) cohorts, the ERT model showed sufficient agreement between the actual and predicted CFSs (kappa 0.864, ICC 0.864 and kappa 0.796, ICC 0.796, respectively) ( Figure ). During a median follow-up period of 421 (IQR 280-616) days, the higher predicted CFS was independently associated with a higher risk of all-cause death (HR 1.97, 95% CI 1.21-3.20) adjusted for the MAGGIC mortality risk prediction score, N-terminal pro-brain natriuretic peptide, and serum albumin levels. Conclusions: Machine learning-based algorithms that automatically rate the CFS are feasible, and the predicted CFS is associated with the risk of all-cause death in elderly patients with HF.

Object Pose Estimation Using Edge Images Synthesized from Shape Information

This paper presents a method for estimating the six Degrees of Freedom (6DoF) pose of texture-less objects from a monocular image by using edge information. The deep learning-based pose estimation method needs a large dataset containing pairs of an image and ground truth pose of objects. To alleviate the cost of collecting a dataset, we focus on the method using a dataset made by computer graphics (CG). This simulation-based method prepares a thousand images by rendering the computer-aided design (CAD) data of the object and trains a deep-learning model. As an inference stage, a monocular RGB image is entered into the model, and the object's pose is estimated. The representative simulation-based method, Pose Interpreter Networks, uses silhouette images as the input, thereby enabling common feature (contour) extraction from RGB and CG images. However, estimating rotation parameters is less accurate. To overcome this problem, we propose a method to use edge information extracted from the object's ridgelines for training the deep learning model. Since edge distribution changes largely according to the pose, the estimation of rotation parameters becomes more robust. Through an experiment with simulation data, we quantitatively proved the accuracy improvement compared to the previous method (error rate decreases at a certain condition are translation 22.9% and rotation: 43.4%). Moreover, through an experiment with physical data, we clarified the issues of this method and proposed an effective solution by fine-tuning (error rate decrease at a certain condition are translation 20.1% and rotation 57.7%).

A Novel Deep Learning-Based Pose Estimation Method for Robotic Grasping of Axisymmetric Bodies in Industrial Stacked Scenarios

A vision-based intelligent robotic grasping system is essential for realizing unmanned operations in industrial manufacturing, and pose estimation plays an import role in this system. In this study, deep learning was used to obtain the 6D pose of an axisymmetric body which was optimal for robotic grasping in industrial stacked scenarios. We propose a method to obtain the 6D pose of an axisymmetric body by detecting the pre-defined keypoints on the side surface. To realize this method and solve other challenges in industrial stacked scenarios, we propose a multitask real-time convolutional neural network (CNN), named Key-Yolact, which involves object detection, instance segmentation, and multiobject 2D keypoint detection. A small CNN as a decision-making subsystem was designed to score multiple predictions of Key-Yolact, and the body with the highest score is considered the best for grasping. Experiments on a self-built stacked dataset showed that Key-Yolact has a practical tradeoff between inference speed and precision. The inference speed of Key-Yolact is higher by 10 FPS, whereas its precision is decreased by only 7% when compared with the classical multitask Keypoint R-CNN. Robotic grasping experiments showed that the proposed design is effective and can be directly applied to industrial scenarios.

Pose prediction of textureless objects for robot bin picking with deep learning approach

Six dimensional pose prediction of objects for robotic manipulation has received much attention in industrial applications. Despite considerable research on improving the performance of pose estimation, the problem remains challenging for industrial objects because of their textureless and mostly homogeneous properties in the presence of clutter and occlusion. This article proposes deep learning-based pose estimation by jointly detecting, segmenting and pose predicting. The binary classification branch as an attention module is suggested to improve the accuracy of instance segmentation. Using the instance information, the initial pose estimation network is designed by fusing the depth information into the grayscale image to strengthen the geometric features. Then, to obtain a highly accurate pose, an iterative network is constructed with point clouds as inputs to refine the initial pose. The networks are implemented to predict the pose of textureless object on the synthetic and real scenes. Experimental results indicate the pose estimation method is efficient and robust to pose prediction of textureless objects in cluttered and occluded scenes.

Accuracy of a markerless motion capture system in estimating upper extremity kinematics during boxing.

Kinematic analysis of the upper extremity can be useful to assess the performance and skill levels of athletes during combat sports such as boxing. Although marker-based approach is widely used to obtain kinematic data, it is not suitable for “in the field” activities, i.e., when performed outside the laboratory environment. Markerless video-based systems along with deep learning-based pose estimation algorithms show great potential for estimating skeletal kinematics. However, applicability of these systems in assessing upper-limb kinematics remains unexplored in highly dynamic activities. This study aimed to assess kinematics of the upper limb estimated with a markerless motion capture system (2D video cameras along with commercially available pose estimation software Theia3D) compared to those measured with marker-based system during “in the field” boxing. A total of three elite boxers equipped with retroreflective markers were instructed to perform specific sequences of shadow boxing trials. Their movements were simultaneously recorded with 12 optoelectronic and 10 video cameras, providing synchronized data to be processed further for comparison. Comparative assessment showed higher differences in 3D joint center positions at the elbow (more than 3 cm) compared to the shoulder and wrist (<2.5 cm). In the case of joint angles, relatively weaker agreement was observed along internal/external rotation. The shoulder joint revealed better performance across all the joints. Segment velocities displayed good-to-excellent agreement across all the segments. Overall, segment velocities exhibited better performance compared to joint angles. The findings indicate that, given the practicality of markerless motion capture system, it can be a promising alternative to analyze sports-performance.

Fusion Poser: 3D Human Pose Estimation Using Sparse IMUs and Head Trackers in Real Time.

The motion capture method using sparse inertial sensors is an approach for solving the occlusion and economic problems in vision-based methods, which is suitable for virtual reality applications and works in complex environments. However, VR applications need to track the location of the user in real-world space, which is hard to obtain using only inertial sensors. In this paper, we present Fusion Poser, which combines the deep learning-based pose estimation and location tracking method with six inertial measurement units and a head tracking sensor that provides head-mounted displays. To estimate human poses, we propose a bidirectional recurrent neural network with a convolutional long short-term memory layer that achieves higher accuracy and stability by preserving spatio-temporal properties. To locate a user with real-world coordinates, our method integrates the results of an estimated joint pose with the pose of the tracker. To train the model, we gathered public motion capture datasets of synthesized IMU measurement data, as well as creating a real-world dataset. In the evaluation, our method showed higher accuracy and a more robust estimation performance, especially when the user adopted lower poses, such as a squat or a bow.

Automated Motion Heatmap Generation for Bridge Navigation Watch Monitoring System

Abstract Most ship collisions and grounding accidents are due to errors made by watchkeeping personnel (WP) on the bridge. International Maritime Organization (IMO) adopts the resolution on the Bridge Navigation Watch Alarm System (BNWAS) detecting operator disability to avert these accidents. The defined system in the resolution is very basic and vulnerable to abuse. There is a need for a more advanced system of monitoring the behaviour of WP to mitigate watchkeeping errors. In this research, a Bridge Navigation Watch Monitoring System (BNWMS) is suggested to achieve this task. Architecture is proposed to train a model for BNWMS. The literature reveals that vision-based sensors can produce relevant input data required for model training. 2D body poses belonging to the same person are estimated from multiple camera views by using a deep learning-based pose estimation algorithm. Estimated 2D poses are projected into 3D space with a maximum 8 mm error by utilising multiple view computer vision techniques. Finally, the obtained 3D poses are plotted on a bird’s-eye view bridge plan to calculate a heatmap of body motions capturing temporal, as well as spatial, information. The results show that motion heatmaps present significant information about the behaviour of WP within a defined time interval. This automated motion heatmap generation is a novel approach that provides input data for the suggested BNWMS.

Automatic Measurement of Postural Abnormalities With a Pose Estimation Algorithm in Parkinson’s Disease

ObjectiveThis study aims to develop an automated and objective tool to evaluate postural abnormalities in Parkinson’s disease (PD) patients.MethodsWe applied a deep learning-based pose-estimation algorithm to lateral photos of prospectively enrolled PD patients (n = 28). We automatically measured the anterior flexion angle (AFA) and dropped head angle (DHA), which were validated with conventional manual labeling methods.Results The automatically measured DHA and AFA were in excellent agreement with manual labeling methods (intraclass correlation coefficient > 0.95) with mean bias equal to or less than 3 degrees.ConclusionThe deep learning-based pose-estimation algorithm objectively measured postural abnormalities in PD patients.

Stride-level analysis of mouse open field behavior using deep learning-based pose estimation

Stride-level analysis of mouse open field behavior using deep learning-based pose estimation

Stride-level analysis of mouse open field behavior using deep learning-based pose estimation

Stride-level analysis of mouse open field behavior using deep learning-based pose estimation

Stride-level analysis of mouse open field behavior using deep learning-based pose estimation

Stride-level analysis of mouse open field behavior using deep learning-based pose estimation

Progress on Computational Ethology

Animal behavior, mostly controlled by the central brain, has been studied in natural environments and controlled laboratory settings. In the early 20th century, researchers studied behavior in natural environments to reveal how it is built from components and organized over time in response to stimuli. In the laboratory settings, researchers study the ability of brain to generate behaviors in response to rewards and punishments. However, there are limitations in quantifying animal behaviors in these two approaches. Recent advances and breakthroughs in computer science provide an important opportunity for overcoming these limitations of behavioral studies. In this review, we focus on an emerging new discipline called “Computational Ethology”, which uses a wide variety of techniques, including computer vision and machine learning, to measure and analyze the patterns of animal behaviors. Computational Ethology allows quantitative analyses of animal behaviors with high efficiency, and has made significant progress in recent years towards a better understanding of behaviors as well as their underlying neural mechanism. Over the last decade, with the application of artificial intelligence in animal behavior study, numerous methods have been developed to automatically quantify animal behavior, including automatic tracking of movements. Classical computer vision methods estimate centroids and ellipses of animals, which reflect the locomotion and orientation, respectively. This estimation was then extended to multiple animals. Nevertheless, non-locomotion information of animal pose cannot be reliably captured. To solve this problem, deep learning-based pose estimation of behaviors in the laboratory setting has been developed for various animal species. In this review, we present a pipeline of collecting and analyzing behavioral data. High-dimensional raw behavioral data are first subjected to dimension reduction. The low-dimensional data are then segmented and analyzed by supervised classification or unsupervised clustering in order to produce behavioral modules. After this, transition probabilities between behavioral modules over time are calculated to elucidate the pattern of behaviors. We also review the application of artificial intelligence to analyze behaviors for diagnosing and evaluating neuropsychiatric diseases and discuss the opportunities and challenges in Computational Ethology. Computational Ethology, a result of substantial interdisciplinary research of neuroscience, psychology, physics, computer science, and ethology has great potential towards a deeper understanding of the nature of animal behaviors. Nevertheless, it is still in its infancy and many questions remain to be explored. This review provides a summary as well as a reference resource for this new yet rapidly advancing discipline. We hope that this review will be informative and useful for a wide interdisciplinary scientific community studying animal behaviors and artificial intelligence.

A novel dataset and deep learning-based approach for marker-less motion capture during gait

BackgroundThe deep learning-based human pose estimation methods, which can estimate joint centers position, have achieved promising results on the publicly available human pose datasets (e.g., Human3.6 M). However, these datasets may be less efficient for gait study, particularly for clinical applications, because of the limited number of subjects, their homogeneity (all asymptomatic adults), and the errors introduced by marker placement on subjects’ regular clothing. Research questionHow a new human pose dataset, adapted for gait study, could contribute to the advancement and evaluation of marker-less motion capture systems? MethodsA marker-less system, based on deep learning-based pose estimation methods, was proposed. A new dataset (ENSAM dataset) was collected. Twenty-two asymptomatic adults, one adult with scoliosis, one adult with spondylolisthesis, and seven children with bone disease performed ten walking trials, while being recorded both by the proposed marker-less system and a reference system – combining a marker-based motion capture system and a medical imaging system (EOS). The dataset was split into training and test sets. The pose estimation method, already trained on the Human3.6 M dataset, was evaluated on the ENSAM test set, then reevaluated after further training on the ENSAM training set. The joints coordinates were evaluated, using Bland-Altman bias and 95 % confidence interval, and joint position error (the Euclidean distance between the estimated joint centers and the corresponding reference values). ResultsThe Bland-Altman 95 % confidence intervals were substantially improved after finetuning the pose estimation method on the ENSAM training set (e.g., from 106.9 mm to 17.4 mm for the hip joint). With the new dataset and approach, the mean joint position error varied from 6.2 mm for ankles to 21.1 mm for shoulders. SignificanceThe proposed marker-less system achieved promising results in terms of joint position errors. Future studies are necessary to assess the system in terms of gait parameters.

Comparison of a Deep Learning-Based Pose Estimation System to Marker-Based and Kinect Systems in Exergaming for Balance Training.

Using standard digital cameras in combination with deep learning (DL) for pose estimation is promising for the in-home and independent use of exercise games (exergames). We need to investigate to what extent such DL-based systems can provide satisfying accuracy on exergame relevant measures. Our study assesses temporal variation (i.e., variability) in body segment lengths, while using a Deep Learning image processing tool (DeepLabCut, DLC) on two-dimensional (2D) video. This variability is then compared with a gold-standard, marker-based three-dimensional Motion Capturing system (3DMoCap, Qualisys AB), and a 3D RGB-depth camera system (Kinect V2, Microsoft Inc). Simultaneous data were collected from all three systems, while participants (N = 12) played a custom balance training exergame. The pose estimation DLC-model is pre-trained on a large-scale dataset (ImageNet) and optimized with context-specific pose annotated images. Wilcoxon’s signed-rank test was performed in order to assess the statistical significance of the differences in variability between systems. The results showed that the DLC method performs comparably to the Kinect and, in some segments, even to the 3DMoCap gold standard system with regard to variability. These results are promising for making exergames more accessible and easier to use, thereby increasing their availability for in-home exercise.

Performance Evaluation of Model-Based Gait on Multi-View Very Large Population Database With Pose Sequences

Model-based gait recognition is considered to be promising due to the robustness against some variations, such as clothing and baggage carried. Although model-based gait recognition has not been fully explored due to the difficulty of human body model fitting and the lack of a large-scale gait database, recent progress in deep learning-based approaches to human body model fitting and human pose estimation is mitigating the difficulty. In this paper, we, therefore, address the remaining issue by presenting a large-scale human pose-based gait database, OUMVLP-Pose, which is based on a publicly available multi-view large-scale gait database, OUMVLP. OUMVLP-Pose has many unique advantages compared with other public databases. First, OUMVLP-Pose is the first gait database that provides two datasets of human pose sequences extracted by two standard deep learning-based pose estimation algorithms, OpenPose and AlphaPose. Second, it contains multi-view large-scale data, i.e., over 10,000 subjects and 14 views for each subject. In addition, we also provide benchmarks in which different kinds of gait recognition methods, including model-based methods and appearance-based methods, have been evaluated comprehensively. The model-based gait recognition methods have shown promising performances. We believe this database, OUMVLP-Pose, will greatly promote model-based gait recognition in the next few years.

Deep Learning-Based Pose Estimation of Apples for Inspection in Logistic Centers Using Single-Perspective Imaging

Fruit packaging is a time-consuming task due to its low automation level. The gentle handling required by some kinds of fruits and their natural variations complicates the implementation of automated quality controls and tray positioning for final packaging. In this article, we propose a method for the automatic localization and pose estimation of apples captured by a Red-Green-Blue (RGB) camera using convolutional neural networks. Our pose estimation algorithm uses a cascaded structure composed of two independent convolutional neural networks: one for the localization of apples within the images and a second for the estimation of the three-dimensional rotation of the localized and cropped image area containing an apple. We used a single shot multi-box detector to find the bounding boxes of the apples in the images. Lie algebra is used for the regression of the rotation, which represents an innovation in this kind of application. We compare the performances of four different network architectures and show that this kind of representation is more suitable than using state-of-the-art quaternions. By using this method, we achieved a promising accuracy for the rotation regression of 98.36%, considering an error range lower than 15 degrees, forming a base for the automation of fruit packing systems.

A deep learning approach for pose estimation from volumetric OCT data.

Tracking the pose of instruments is a central problem in image-guided surgery. For microscopic scenarios, optical coherence tomography (OCT) is increasingly used as an imaging modality. OCT is suitable for accurate pose estimation due to its micrometer range resolution and volumetric field of view. However, OCT image processing is challenging due to speckle noise and reflection artifacts in addition to the images' 3D nature. We address pose estimation from OCT volume data with a new deep learning-based tracking framework. For this purpose, we design a new 3D convolutional neural network (CNN) architecture to directly predict the 6D pose of a small marker geometry from OCT volumes. We use a hexapod robot to automatically acquire labeled data points which we use to train 3D CNN architectures for multi-output regression. We use this setup to provide an in-depth analysis on deep learning-based pose estimation from volumes. Specifically, we demonstrate that exploiting volume information for pose estimation yields higher accuracy than relying on 2D representations with depth information. Supporting this observation, we provide quantitative and qualitative results that 3D CNNs effectively exploit the depth structure of marker objects. Regarding the deep learning aspect, we present efficient design principles for 3D CNNs, making use of insights from the 2D deep learning community. In particular, we present Inception3D as a new architecture which performs best for our application. We show that our deep learning approach reaches errors at our ground-truth label's resolution. We achieve a mean average error of 14.89 ± 9.3µm and 0.096 ± 0.072° for position and orientation learning, respectively.