Pose Estimation Algorithm Research Articles

An Efficient Pose Estimation Algorithm for Non-Cooperative Space Objects Based on Dual-Channel Transformer

Non-cooperative space object pose estimation is a key technique for spatial on-orbit servicing, where pose estimation algorithms based on low-quality, low-power monocular sensors provide a practical solution for spaceborne applications. The current pose estimation methods for non-cooperative space objects using monocular vision generally consist of three stages: object detection, landmark regression, and perspective-n-point (PnP) solver. However, there are drawbacks, such as low detection efficiency and the need for prior knowledge. To solve the above problems, an end-to-end non-cooperative space object pose estimation learning algorithm based on dual-channel transformer is proposed, a feature extraction backbone network based on EfficientNet is established, and two pose estimation subnetworks based on transformer are also established. A quaternion SoftMax-like activation function is designed to improve the precision of orientation error estimating. The method only uses RGB images, eliminating the need for a CAD model of the satellite, and simplifying the detection process by using an end-to-end network to directly detect satellite pose information. Experiments are carried out on the SPEED dataset provided by the European Space Agency (ESA). The results show that the proposed algorithm can successfully predict the satellite pose information and effectively decouple the spatial translation information and orientation information, which significantly improves the recognition efficiency compared with other methods.

Wearable Motion Capture: Reconstructing and Predicting 3D Human Poses from Wearable Sensors.

Reconstructing and predicting 3D human walking poses in unconstrained measurement environments have the potential to use for health monitoring systems for people with movement disabilities by assessing progression after treatments and providing information for assistive device controls. The latest pose estimation algorithms utilize motion capture systems, which capture data from IMU sensors and third-person view cameras. However, third-person views are not always possible for outpatients alone. Thus, we propose the wearable motion capture problem of reconstructing and predicting 3D human poses from the wearable IMU sensors and wearable cameras, which aids clinicians' diagnoses on patients out of clinics. To solve this problem, we introduce a novel Attention-Oriented Recurrent Neural Network (AttRNet) that contains a sensor-wise attention-oriented recurrent encoder, a reconstruction module, and a dynamic temporal attention-oriented recurrent decoder, to reconstruct the 3D human pose over time and predict the 3D human poses at the following time steps. To evaluate our approach, we collected a new WearableMotionCapture dataset using wearable IMUs and wearable video cameras, along with the musculoskeletal joint angle ground truth. The proposed AttRNet shows high accuracy on the new lower-limb WearableMotionCapture dataset, and it also outperforms the state-of-the-art methods on two public full-body pose datasets: DIP-IMU and TotalCaputre.

OpenCap: Human movement dynamics from smartphone videos.

Measures of human movement dynamics can predict outcomes like injury risk or musculoskeletal disease progression. However, these measures are rarely quantified in large-scale research studies or clinical practice due to the prohibitive cost, time, and expertise required. Here we present and validate OpenCap, an open-source platform for computing both the kinematics (i.e., motion) and dynamics (i.e., forces) of human movement using videos captured from two or more smartphones. OpenCap leverages pose estimation algorithms to identify body landmarks from videos; deep learning and biomechanical models to estimate three-dimensional kinematics; and physics-based simulations to estimate muscle activations and musculoskeletal dynamics. OpenCap's web application enables users to collect synchronous videos and visualize movement data that is automatically processed in the cloud, thereby eliminating the need for specialized hardware, software, and expertise. We show that OpenCap accurately predicts dynamic measures, like muscle activations, joint loads, and joint moments, which can be used to screen for disease risk, evaluate intervention efficacy, assess between-group movement differences, and inform rehabilitation decisions. Additionally, we demonstrate OpenCap's practical utility through a 100-subject field study, where a clinician using OpenCap estimated musculoskeletal dynamics 25 times faster than a laboratory-based approach at less than 1% of the cost. By democratizing access to human movement analysis, OpenCap can accelerate the incorporation of biomechanical metrics into large-scale research studies, clinical trials, and clinical practice.

6IMPOSE: bridging the reality gap in 6D pose estimation for robotic grasping.

6D pose recognition has been a crucial factor in the success of robotic grasping, and recent deep learning based approaches have achieved remarkable results on benchmarks. However, their generalization capabilities in real-world applications remain unclear. To overcome this gap, we introduce 6IMPOSE, a novel framework for sim-to-real data generation and 6D pose estimation. 6IMPOSE consists of four modules: First, a data generation pipeline that employs the 3D software suite Blender to create synthetic RGBD image datasets with 6D pose annotations. Second, an annotated RGBD dataset of five household objects was generated using the proposed pipeline. Third, a real-time two-stage 6D pose estimation approach that integrates the object detector YOLO-V4 and a streamlined, real-time version of the 6D pose estimation algorithm PVN3D optimized for time-sensitive robotics applications. Fourth, a codebase designed to facilitate the integration of the vision system into a robotic grasping experiment. Our approach demonstrates the efficient generation of large amounts of photo-realistic RGBD images and the successful transfer of the trained inference model to robotic grasping experiments, achieving an overall success rate of 87% in grasping five different household objects from cluttered backgrounds under varying lighting conditions. This is made possible by fine-tuning data generation and domain randomization techniques and optimizing the inference pipeline, overcoming the generalization and performance shortcomings of the original PVN3D algorithm. Finally, we make the code, synthetic dataset, and all the pre-trained models available on GitHub.

Accuracy of Video-Based Gait Analysis Using Pose Estimation During Treadmill Walking Versus Overground Walking in Persons After Stroke.

Video-based pose estimation is an emerging technology that shows significant promise for improving clinical gait analysis by enabling quantitative movement analysis with little costs of money, time, or effort. The objective of this study is to determine the accuracy of pose estimation-based gait analysis when video recordings are constrained to 3 common clinical or in-home settings (ie, frontal and sagittal views of overground walking and sagittal views of treadmill walking). Simultaneous video and motion capture recordings were collected from 30 persons after stroke during overground and treadmill walking. Spatiotemporal and kinematic gait parameters were calculated from videos using an open-source human pose estimation algorithm and from motion capture data using traditional gait analysis. Repeated-measures analyses of variance were then used to assess the accuracy of the pose estimation-based gait analysis across the different settings, and the authors examined Pearson and intraclass correlations with ground-truth motion capture data. Sagittal videos of overground and treadmill walking led to more accurate measurements of spatiotemporal gait parameters versus frontal videos of overground walking. Sagittal videos of overground walking resulted in the strongest correlations between video-based and motion capture measurements of lower extremity joint kinematics. Video-based measurements of hip and knee kinematics showed stronger correlations with motion capture versus ankle kinematics for both overground and treadmill walking. Video-based gait analysis using pose estimation provides accurate measurements of step length, step time, and hip and knee kinematics during overground and treadmill walking in persons after stroke. Generally, sagittal videos of overground gait provide the most accurate results. Many clinicians lack access to expensive gait analysis tools that can help identify patient-specific gait deviations and guide therapy decisions. These findings show that video-based methods that require only common household devices provide accurate measurements of a variety of gait parameters in persons after stroke and could make quantitative gait analysis significantly more accessible.

Gait analysis comparison between manual marking, 2D pose estimation algorithms, and 3D marker-based system.

Recent advances in Artificial Intelligence (AI) and Computer Vision (CV) have led to automated pose estimation algorithms using simple 2D videos. This has created the potential to perform kinematic measurements without the need for specialized, and often expensive, equipment. Even though there's a growing body of literature on the development and validation of such algorithms for practical use, they haven't been adopted by health professionals. As a result, manual video annotation tools remain pretty common. Part of the reason is that the pose estimation modules can be erratic, producing errors that are difficult to rectify. Because of that, health professionals prefer the use of tried and true methods despite the time and cost savings pose estimation can offer. In this work, the gait cycle of a sample of the elderly population on a split-belt treadmill is examined. The Openpose (OP) and Mediapipe (MP) AI pose estimation algorithms are compared to joint kinematics from a marker-based 3D motion capture system (Vicon), as well as from a video annotation tool designed for biomechanics (Kinovea). Bland-Altman (B-A) graphs and Statistical Parametric Mapping (SPM) are used to identify regions of statistically significant difference. Results showed that pose estimation can achieve motion tracking comparable to marker-based systems but struggle to identify joints that exhibit small, but crucial motion. Joints such as the ankle, can suffer from misidentification of their anatomical landmarks. Manual tools don't have that problem, but the user will introduce a static offset across the measurements. It is proposed that an AI-powered video annotation tool that allows the user to correct errors would bring the benefits of pose estimation to professionals at a low cost.

Comparison of video-based algorithms for 2D human kinematics estimation: a preliminary study

Many research efforts have been spent developing robust video-based algorithms for human pose estimation. Our goal was to compare video-based algorithms for pose estimation for gait analysis. We conducted an experiment with a healthy subject performing walking sessions on a treadmill at three different speeds: slow (3.6 km/h), medium (5 km/h), and high (7 km/h). An RGB 4k camera was placed laterally on the sagittal plane. Four algorithms were compared: (i) colour threshold filtering with blob-analysis, and three Deep Learning-based markerless algorithms (ii) TC-Former, (iii) FastPose and (iv) Blazepose. For colour threshold filtering with the blob-analysis algorithm, six magenta passive markers were placed over the joint centres of the subject’s lower limb. All selected deep learning-based markerless algorithms are supported by various open-source pose estimation toolboxes and are pre-trained on several whole-body keypoint datasets. The 2D trajectories of the joint centres were compared considering the root mean square error and Pearson’s coefficient. Preliminary results showed high correlations between marker and markerless algorithms for all walking speeds. TC-Former generally performed better with root mean square error on trajectories below 35 mm and did not suffer from self-occlusion issues.

Student Competition (Technology Innovation) ID 1970388

Background The evaluation of hand function after spinal cord injury (SCI) is conducted in clinical settings, which may not accurately reflect hand function in the real world, thereby limiting the efficacy assessment of new treatments. Wearable cameras, also known as egocentric video, are a novel method to evaluate hand function in non-clinical environments. Nonetheless, manual processing of vast quantities of complex video data is difficult, highlighting the need for automated data analysis. The objective of this study was to automatically identify distinct hand postures in egocentric video using unsupervised machine learning. Methods Seventeen participants with cervical SCI recorded activities of daily living in a home simulation laboratory. A hand pose estimation algorithm was applied on detected hands to determine 2D joint locations, which were lifted to 3D coordinates. The resulting hand posture information was subjected to a number of clustering techniques. Hand grasps were manually labelled into four categories for evaluation purposes: power, precision, intermediate, and non-prehensile. Results K-Means clustering consistently exhibited the highest Silhouette score, which reflects the presence of discrete clusters in the data. When comparing with manual annotations, Spectral Clustering applied to a feature space consisting of 2D pose estimation with confidence scores yield the best performance as quantified by maximum match (0.48), Fowlkes-Mallows score (0.46), and normalized mutual information (0.22). Conclusions This is the first attempt to develop an unsupervised, data-driven hand taxonomy for individuals with SCI using wearable technology. The findings suggest that the method is capable of grouping similar hand grasps.

An action analysis algorithm for teachers based on human pose estimation

Teachers serve as transmitters of cultural knowledge and play a crucial role in ongoing development and continuity of human society. Analyzing their behavior can assist educators in identifying their strengths and weaknesses within the classroom, thereby enhancing their teaching skills and methods. With the development of artificial intelligence, it is possible to automatically analyze teachers' behavior in the classroom by offering timely feedback and suggestions to improve teaching strategies. In this paper, we propose an action classification method based on human skeletal posture to analyze teacher-behavior in the classroom. First, by using the HRNet pose estimation algorithm to extract teacher skeleton information as features, the algorithm could remove redundant information in images and accurately reflect the teacher's posture. Second, by considering the relative positions of skeletal points, we establish a set of quantifiable indicators to analyze the interrelationships among these points and then obtain the teacher's local posture. Finally, relying on the local posture obtained based on the indicators, we encode and classify common behaviors of teachers in the classroom. The approach's effectiveness is evidenced by experimental results, which enables dynamic management and assessment of teacher behavior in educational environments.

Application of IoT technology based on neural networks in basketball training motion capture and injury prevention

Basketball players need to frequently engage in various physical movements during the game, which puts a certain burden on their bodies and can easily lead to various sports injuries. Therefore, it is crucial to prevent sports injuries in basketball teaching. This paper also studies basketball motion track capture. Basketball motion capture preserves the motion posture information of the target person in three-dimensional space. Because the motion capture system based on machine vision often encounters problems such as occlusion or self occlusion in the application scene, human motion capture is still a challenging problem in the current research field. This article designs a multi perspective human motion trajectory capture algorithm framework, which uses a two-dimensional human motion pose estimation algorithm based on deep learning to estimate the position distribution of human joint points on the two-dimensional image from each perspective. By combining the knowledge of camera poses from multiple perspectives, the three-dimensional spatial distribution of joint points is transformed, and the final evaluation result of the target human 3D pose is obtained. This article applies the research results of neural networks and IoT devices to basketball motion capture methods, further developing basketball motion capture systems.

Quantification of User Driving Behavior in Motorized Mobility Scooters using CNN-based Head Pose Estimation Algorithm.

There is a growing concern regarding the driving safety of Motorized Mobility Scooters (MMSs) for the elderly and mobility-impaired individuals. Although various research has made progress in sensor-based driving assistance systems to identify environmental hazards, few studies focus on investigating the impact of user behavior on MMS driving. In this paper, a driving status logging (DSL) system is developed to measure the user's behavior while driving. A cross-correlation analysis is implemented to quantify the temporal relationship between the head movement and steering operation in the driving of MMSs. The preliminary results suggest that the head movement can be used as an appropriate index to predict the intended steering operation in the driving of MMSs. Moreover, the quantified head-steering lag time provides the possibility to identify the hazardous driving pattern of MMS users.Clinical Relevance- The investigation of user behavior has the potential to improve the safety of MMSs. In this study, the user behavior in the driving of MMSs was quantitatively measured using the developed DSL system. Consequently, the temporal relationship between head movement and steering operation was first quantified in MMS-related research. These outcomes provide valuable insights into developing behavioral interventions to address the user's risky behavior patterns, thereby promoting the driving safety of MMSs.

Visual identification and pose estimation algorithms of nut tightening robot system

Visual identification and pose estimation algorithms of nut tightening robot system

CHAN: Skeleton based action recognition by multi‐level feature learning

AbstractSkeleton‐based action recognition has been continuously and intensively studied. However, dynamic 3D skeleton data are difficult to be popularized in practical applications due to the restricted data acquisition conditions. Although the action recognition method based on 2D pose information extracted from RGB video can effectively avoid the influence of complex background, it is susceptible to factors such as video jitter and joint overlap. To reduce the interference of the aforementioned factors, we use two‐dimensional skeletal joint coordinate modal information to represent the changes in human body posture. First, we use a target detector and pose estimation algorithm to obtain the joint coordinates of each frame sample from RGB video. Then the feature extraction network is combined to perform multi‐level feature learning to establish correspondence between actions and corresponding multi‐level features. Finally, the hierarchical attention mechanism is introduced to design the model named CHAN. By calculating the association between elements, the weight of the action classification is redistributed. Extensive experiments on three datasets demonstrate the effectiveness of our proposed method.

Multi-Angle Models and Lightweight Unbiased Decoding-Based Algorithm for Human Pose Estimation

When a top-down method is taken to the task of human pose estimation, the accuracy of joint point localization is often limited by the accuracy of human detection. In addition, conventional algorithms commonly encode the image to generate a heat map before processing, but the systematic error in decoding the heat map back to the original image has an impact on the positioning. Therefore, to address the two problems, we propose an algorithm that uses multiple angle models to generate the human boxes and then performs lightweight decoding to recover the image. The new boxes can better fit humans and the recovery error can be reduced. First, we split the backbone network into three sub-networks, the first sub-network is responsible for generating the original human box, the second sub-network is responsible for generating a coarse pose estimation in the boxes, and the third sub-network is responsible for a high-precision pose estimation. In order to make the human box fit the human body better, with only a small number of interfering pixels inside the box, models of the human boxes with multiple rotation angles are generated. The results from the second sub-network are used to select the best human box. Using this human box as input to the third sub-network can significantly improve the accuracy of the pose estimation. Then to reduce the errors arising from image decoding, we propose a lightweight unbiased decoding strategy that differs from traditional methods by combining multiple possible offsets to select the direction and size of the final offset. On the MPII dataset and the COCO dataset, we compare the proposed algorithm with 11 state-of-the-art algorithms. The experimental results show that the algorithm achieves a large improvement in accuracy for a wide range of image sizes and different metrics.

Motion Capture for Sporting Events Based on Graph Convolutional Neural Networks and Single Target Pose Estimation Algorithms

Human pose estimation refers to accurately estimating the position of the human body from a single RGB image and detecting the location of the body. It serves as the basis for several computer vision tasks, such as human tracking, 3D reconstruction, and autonomous driving. Improving the accuracy of pose estimation has significant implications for the advancement of computer vision. This paper addresses the limitations of single-branch networks in pose estimation. It presents a top-down single-target pose estimation approach based on multi-branch self-calibrating networks combined with graph convolutional neural networks. The study focuses on two aspects: human body detection and human body pose estimation. The human body detection is for athletes appearing in sports competitions, followed by human body pose estimation, which is divided into two methods: coordinate regression-based and heatmap test-based. To improve the accuracy of the heatmap test, the high-resolution feature map output from HRNet is used for deconvolution to improve the accuracy of single-target pose estimation recognition.

High-accuracy pose estimation algorithm with monocular camera based on residual network

Pose estimation is a typical problem encountered in computer vision. Therefore, it is important to improve the accuracy of this method. Aiming at the accuracy of pose estimation, we propose a high-accuracy pose estimation algorithm based on the you only look once network and residual network with a monocular camera as the sensor for visual acquisition. This algorithm uses ArUco markers as a reference for object localization and uses the red, green, and blue (RGB) image as the input. The input image is sampled 16 and 32 times to extract the feature image, and the feature image extracted by 16 times sampling is passed through the pass-through layer and then combined with the feature image extracted by 32 times sampling to accomplish the dimension expansion. The feature image is identified by the convolutional layer. The EPnP algorithm is used to solve the camera poses. The pose information of the target object in the RGB image is used as the output. By comparing the pose estimation accuracy for the LINEMOD dataset with three evaluation metrics—the 2D projection metric, ADD metric, and 5 cm to 5 deg metric—it can be observed that the pose estimation algorithm proposed has advantages in terms of accuracy compared with traditional pose estimation algorithms. When the target is very similar to the background objects, the algorithm also achieves good performance.

The Classification of Movement in Infants for the Autonomous Monitoring of Neurological Development

Neurodevelopmental delay following extremely preterm birth or birth asphyxia is common but diagnosis is often delayed as early milder signs are not recognised by parents or clinicians. Early interventions have been shown to improve outcomes. Automation of diagnosis and monitoring of neurological disorders using non-invasive, cost effective methods within a patient’s home could improve accessibility to testing. Furthermore, said testing could be conducted over a longer period, enabling greater confidence in diagnoses, due to increased data availability. This work proposes a new method to assess the movements in children. Twelve parent and infant participants were recruited (children aged between 3 and 12 months). Approximately 25 min 2D video recordings of the infants organically playing with toys were captured. A combination of deep learning and 2D pose estimation algorithms were used to classify the movements in relation to the children’s dexterity and position when interacting with a toy. The results demonstrate the possibility of capturing and classifying children’s complexity of movements when interacting with toys as well as their posture. Such classifications and the movement features could assist practitioners to accurately diagnose impaired or delayed movement development in a timely fashion as well as facilitating treatment monitoring.

Spacecraft Pose Estimation Based on Different Camera Models

Spacecraft pose estimation is an important technology to maintain or change the spacecraft orientation in space. For spacecraft pose estimation, when two spacecraft are relatively distant, the depth information of the space point is less than that of the measuring distance, so the camera model can be seen as a weak perspective projection model. In this paper, a spacecraft pose estimation algorithm based on four symmetrical points of the spacecraft outline is proposed. The analytical solution of the spacecraft pose is obtained by solving the weak perspective projection model, which can satisfy the requirements of the measurement model when the measurement distance is long. The optimal solution is obtained from the weak perspective projection model to the perspective projection model, which can meet the measurement requirements when the measuring distance is small. The simulation results show that the proposed algorithm can obtain better results, even though the noise is large.

Reliability and validity of pose estimation algorithm for measurement of knee range of motion after total knee arthroplasty.

We aimed to assess the reliability and validity of OpenPose, a posture estimation algorithm, for measurement of knee range of motion after total knee arthroplasty (TKA), in comparison to radiography and goniometry. In this prospective observational study, we analyzed 35 primary TKAs (24 patients) for knee osteoarthritis. We measured the knee angles in flexion and extension using OpenPose, radiography, and goniometry. We assessed the test-retest reliability of each method using intraclass correlation coefficient (1,1). We evaluated the ability to estimate other measurement values from the OpenPose value using linear regression analysis. We used intraclass correlation coefficients (2,1) and Bland-Altman analyses to evaluate the agreement and error between radiography and the other measurements. OpenPose had excellent test-retest reliability (intraclass correlation coefficient (1,1) = 1.000). The R2 of all regression models indicated large correlations (0.747 to 0.927). In the flexion position, the intraclass correlation coefficients (2,1) of OpenPose indicated excellent agreement (0.953) with radiography. In the extension position, the intraclass correlation coefficients (2,1) indicated good agreement of OpenPose and radiography (0.815) and moderate agreement of goniometry with radiography (0.593). OpenPose had no systematic error in the flexion position, and a 2.3° fixed error in the extension position, compared to radiography. OpenPose is a reliable and valid tool for measuring flexion and extension positions after TKA. It has better accuracy than goniometry, especially in the extension position. Accurate measurement values can be obtained with low error, high reproducibility, and no contact, independent of the examiner's skills.

Active Pose Relocalization for Intelligent Substation Inspection Robot

Intelligent inspection robots widely applied in substations are required to capture the inspection image consistent with the calibration image during routine inspection of electrical equipment. However, it is a challenging work for the inspection robot to capture the inspection image meeting the requirement due to navigation error and mechanical wear. To address this problem, an active pose relocalization (APR) method is proposed in this article. Specifically, an error model describing the relationship between the pixel error in the image plane and the robot pose error is established. Then, a decoupling three-stage PI control strategy based on the error model is provided to relocate the robot to the calibration pose, wherein, a translation error estimation algorithm based on homography transformation is proposed to compute the absolute translation scale between calibration and inspection poses, which avoid the degradation problem of the classic 2D-2D pose estimation algorithm. Finally, the performance of the proposed APR method is demonstrated through comparative relocalization experiments of 10 calibration points in virtual and real-world environments respectively.