Pose Estimation Research Articles

Axes-aligned non-linear optimized PnP algorithm

Pose estimation is an important component of many real-world computer vision systems. Most existing pose estimation algorithms need a large number of point correspondences to accurately determine the pose of an object. Since the number of point correspondences depends on the object’s appearance, lighting and other external conditions, detecting many points may not be feasible. In many real-world applications, the movement of objects is limited, e.g. due to gravity. Hence, detecting objects with only three degrees of freedom is usually sufficient. This allows us to improve the accuracy of pose estimation by changing the underlying equations of the perspective-n-point problem to three variables instead of six. By using the simplified equations, our algorithm is more robust against detection errors with limited point correspondences. In this article, we study three scenarios where such constraints apply. The first one is about parking a vehicle on a specific spot. Here, a stationary camera is detecting the vehicle to assist the driver. The second scenario describes the perspective of a moving camera detecting objects in its environment. This scenario is common for driver assistance systems, autonomous cars or mobile robots. Third, we describe a camera observing objects from a birds-eye view, which occurs in industrial applications. In all three scenarios, observed objects can only move in the ground plane and rotate around the vertical axis. Hence, three degrees of freedom are sufficient to estimate the pose. Experiments with synthetic data and real-world photographs have shown that our algorithm outperforms state-of-the-art pose estimation algorithms. Depending on the scenario, our algorithm is able to achieve 50% better accuracy, while being equally fast.

A Depth Awareness and Learnable Feature Fusion Network for Enhanced Geometric Perception in Semantic Correspondence.

Deep learning is becoming the most widely used technology for multi-sensor data fusion. Semantic correspondence has recently emerged as a foundational task, enabling a range of downstream applications, such as style or appearance transfer, robot manipulation, and pose estimation, through its ability to provide robust correspondence in RGB images with semantic information. However, current representations generated by self-supervised learning and generative models are often limited in their ability to capture and understand the geometric structure of objects, which is significant for matching the correct details in applications of semantic correspondence. Furthermore, efficiently fusing these two types of features presents an interesting challenge. Achieving harmonious integration of these features is crucial for improving the expressive power of models in various tasks. To tackle these issues, our key idea is to integrate depth information from depth estimation or depth sensors into feature maps and leverage learnable weights for feature fusion. First, depth information is used to model pixel-wise depth distributions, assigning relative depth weights to feature maps for perceiving an object's structural information. Then, based on a contrastive learning optimization objective, a series of weights are optimized to leverage feature maps from self-supervised learning and generative models. Depth features are naturally embedded into feature maps, guiding the network to learn geometric structure information about objects and alleviating depth ambiguity issues. Experiments on the SPair-71K and AP-10K datasets show that the proposed method achieves scores of 81.8 and 83.3 on the percentage of correct keypoints (PCK) at the 0.1 level, respectively. Our approach not only demonstrates significant advantages in experimental results but also introduces the depth awareness module and a learnable feature fusion module, which enhances the understanding of object structures through depth information and fully utilizes features from various pre-trained models, offering new possibilities for the application of deep learning in RGB and depth data fusion technologies. We will also continue to focus on accelerating model inference and optimizing model lightweighting, enabling our model to operate at a faster speed.

ADS–SLAM: a semantic SLAM based on adaptive motion compensation and semantic information for dynamic environments

Abstract Static environment assumptions are a prerequisite for simultaneous localization and mapping (SLAM), while interference from dynamic objects in the environment can seriously impair the system’s localization accuracy. Recently, many works have combined deep learning and geometric constraints to attenuate the interference of dynamic objects, but poor real-time performance and low accuracy in high dynamic scenes still exist. In this paper, we propose a semantic SLAM algorithm for complex dynamic scenes named ADS–SLAM. Our system combines the advantages of semantic information and motion constraints to remove dynamic points during tracking and localization. First, an adaptive dynamic point detection method based on epipolar constraint between adjacent frames is designed to adapt to the changes of object motion states and a motion area detection method based on Gaussian mixture model and Kalman Filter is utilized to effectively compensate the missed motion areas. Second, an object detection network with improved inference in the backend is utilized to extract prior object semantics. Lastly, the multi-level information is integrated in order to comprehensively screen all dynamic points in the environment and utilize only static points for pose estimation and optimization. Experimental evaluations on challenging public datasets and outdoor dynamic environments demonstrate that our algorithm achieves high localization accuracy in almost all dynamic scenarios compared to the current state-of-the-art SLAM algorithms, with the highest accuracy in high dynamic scenarios, and shows real-time performance for practical applications.

Research on the Detection Algorithm of Elderly Falling Behavior Based on AlphaPose Model

This study aims to address the challenge of quickly and accurately detecting high-risk behaviors such as falls in elderly people in hardware environments with limited power consumption and cost. Therefore, this article proposes an optimized behavior anomaly detection algorithm based on the AlphaPose model. The algorithm first improved the human target detection and pose estimation models to achieve faster pedestrian detection and pose joint point inference; Then, using the adjusted AlphaPose model, efficiently obtain the image coordinate data of each joint point in the human body; Finally, by analyzing the relationship between the linear velocities of the human head and hip joints, as well as the changes in the angle between the vertical line of the human center and the X-axis of the image, it is determined whether a fall event has occurred. This study deployed the proposed algorithm on the JetsonNano embedded development board and compared its performance with mainstream fall detection algorithms based on human posture, including YOLOv3+Pose, YOLOv4+Pose, YOLOv5+Pose, trtpose, and NanoDet+Pose. The experimental results show that on the set embedded platform, when the image resolution is 320×240, this algorithm achieves a detection rate of 8.83 frames per second and an accuracy of 0.913, both of which are superior to the compared algorithms. In summary, this algorithm has high real-time performance and accuracy, and can effectively identify the falling behavior of elderly people in a timely manner.

Human Pose Classification Based on Pose Long Short-Term Memory

Objective: At present, pose classification research is a hot topic in the field of computer vision, playing a crucial role in surveillance security, and motion data mining. The research serves as a crucial follow-up to three dimensional (3D) pose estimation and makes a significant contribution to the advancement of artificial intelligence. Pose classification is a complex problem involving a large amount of complicated data, making its digital modeling and further processing challenging. Methods: This article proposes the Pose long short - term memory (LSTM) method to classify 3D poses reconstructed from 2D image sequences. Result: Compared with traditional methods, the experimental tests show that the performance of the Pose LSTM is more superior in all aspects. Conclusion: Pose LSTM is particularly effective for recognizing continuous action poses because they can capture temporal dependencies in sequential data. In the context of pose recognition, it can analyze the sequence of poses over time and understand how these poses transition from one to another, making it possible to accurately classify or predict actions that unfold over a series of frames.

DenseFusion-DA2: End-to-End Pose-Estimation Network Based on RGB-D Sensors and Multi-Channel Attention Mechanisms.

Notably, 6D pose estimation is a critical technology that enables robotics to perceive and interact with their operational environment. However, occlusion causes a loss of local features, which, in turn, restricts the estimation accuracy. To address these challenges, this paper proposes an end-to-end pose-estimation network based on a multi-channel attention mechanism, DA2Net. Firstly, a multi-channel attention mechanism, designated as "DA2Net", was devised using A2-Nets as its foundation. This mechanism is constructed in two steps. In the first step, the essential characteristics are extracted from the global feature space through the second-order attention pool. In the second step, a feature map is generated by the integration of position and channel attention. Subsequently, the extracted key features are assigned to each position of the feature map, enhancing both the feature representation capacity and the overall performance. Secondly, the designed attention mechanism is introduced into both the feature fusion and pose iterative refinement networks to enhance the network's capacity to acquire local features thus improving its overall performance. The experimental results demonstrated that the estimation accuracy of DenseFusion-DA2 on the LineMOD dataset was approximately 3.4% higher than that of DenseFusion. Furthermore, the estimation accuracy surpassed that of PoseCNN, PVNet, SSD6D, and PointFusion by 8.3%, 11.1%, 20.3%, and 23.8%, respectively. The estimation accuracy also shows a significant advantage on the Occluded LineMOD and HR-Vision datasets. This research not only presents a more efficient solution for robot perception but also introduces novel ideas and methods for technological advancements and applications in related fields.

Human Fall Motion Prediction: Fall Motion Forecasting and Detection with GRU

The human fall motion prediction system is a preventive tool aimed at reducing the risk of falls. In our research, we developed a deep learning model that utilizes pose estimation to track human body posture and integrated this with a Gated Recurrent Unit (GRU) to forecast human motion and predict falls. GRU, an enhancement of Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) models offers improved memorization and more efficient memory usage and performance. Our study presents the human fall motion prediction, which combines the forecasting and classification of potential falls.The CAUCAFall dataset is used as the benchmark of this study, which contains the image sequences of single human motion with ten actions conducted by ten actors. We employed the YOLOv8 Pose model to track the 2D human body pose as the input in our system. A thorough evaluation of the CAUCAFall dataset highlights the effectiveness of our proposed system. Evaluation using the CAUCAFall dataset demonstrates that the model achieved a Mean Per Joint Position Error (MPJPE) of 4.65 pixels from the ground truth, with a 70% accuracy rate in fall prediction. However, the model also exhibited a Mean Relative Error (MRE) of 0.3, indicating that 30% of the predictions were incorrect. These findings underscore the potential of the GRU-based system in fall prevention

Enhanced Infant Movement Analysis Using Transformer-Based Fusion of Diverse Video Features for Neurodevelopmental Monitoring

Neurodevelopment is a highly intricate process, and early detection of abnormalities is critical for optimizing outcomes through timely intervention. Accurate and cost-effective diagnostic methods for neurological disorders, particularly in infants, remain a significant challenge due to the heterogeneity of data and the variability in neurodevelopmental conditions. This study recruited twelve parent–infant pairs, with infants aged 3 to 12 months. Approximately 25 min of 2D video footage was captured, documenting natural play interactions between the infants and toys. We developed a novel, open-source method to classify and analyse infant movement patterns using deep learning techniques, specifically employing a transformer-based fusion model that integrates multiple video features within a unified deep neural network. This approach significantly outperforms traditional methods reliant on individual video features, achieving an accuracy of over 90%. Furthermore, a sensitivity analysis revealed that the pose estimation contributed far less to the model’s output than the pre-trained transformer and convolutional neural network (CNN) components, providing key insights into the relative importance of different feature sets. By providing a more robust, accurate and low-cost analysis of movement patterns, our work aims to enhance the early detection and potential prediction of neurodevelopmental delays, whilst providing insight into the functioning of the transformer-based fusion models of diverse video features.

Validation of Automated Countermovement Vertical Jump Analysis: Markerless Pose Estimation vs. 3D Marker-Based Motion Capture System

This study aimed to validate the automated temporal analysis of countermovement vertical jump (CMJ) using MMPose, a markerless pose estimation framework, by comparing it with the gold-standard 3D marker-based motion capture system. Twelve participants performed five CMJ trials, which were simultaneously recorded using the marker-based system and two smartphone cameras capturing both sides of the body. Key kinematic points, including center of mass (CoM) and toe trajectories, were analyzed to determine jump phases and temporal variables. The agreement between methods was assessed using Bland–Altman analysis, root mean square error (RMSE), and Pearson’s correlation coefficient (r), while consistency was evaluated via intraclass correlation coefficient (ICC 3,1) and two-way repeated-measures ANOVA. Cohen’s effect size (d) quantified the practical significance of differences. Results showed strong agreement (r > 0.98) with minimal bias and narrow limits of agreement for most variables. The markerless system slightly overestimated jump height and CoM vertical velocity, but ICC values (ICC > 0.91) confirmed strong reliability. Cohen’s d values were near zero, indicating trivial differences, and no variability due to recording side was observed. Overall, MMPose proved to be a reliable alternative for in-field CMJ analysis, supporting its broader application in sports and rehabilitation settings.

3D Head Pose Estimation via Normal Maps: A Generalized Solution for Depth Image, Point Cloud, and Mesh

Head pose estimation plays a crucial role in various applications, including human–machine interaction, autonomous driving systems, and 3D reconstruction. Current methods address the problem primarily from a 2D perspective, which limits the efficient utilization of 3D information. Herein, a novel approach, called pose orientation‐aware network (POANet), which leverages normal maps for orientation information embedding, providing abundant and robust head pose information, is introduced. POANet incorporates the axial signal perception module and the rotation matrix perception module, these lightweight modules make the approach achieve state‐of‐the‐art (SOTA) performance with few computational costs. This method can directly analyze various topological 3D data without extensive preprocessing. For depth images, POANet outperforms existing methods on the Biwi Kinect head pose dataset, reducing the mean absolute error (MAE) by ≈30% compared to the SOTA methods. POANet is the first method to perform rigid head registration in a landmark‐free manner. It also incorporates few‐shot learning capabilities and achieves an MAE of about on the Headspace dataset. These features make POANet a superior alternative to traditional generalized Procrustes analysis for mesh data processing, offering enhanced convenience for human phenotype studies.

Applications of AI in martial arts: A survey

Artificial intelligence (AI) has become an interdisciplinary subject that integrates digital image processing, machine learning, and computer science, etc. A lot of work in martial arts benefits from AI technology. In this paper, we highlight and summarize the applications of AI in martial arts and indicate the main progress of related studies. Through the investigation of martial arts styles, tasks, data acquisition methods, and algorithm improvements, a paradigm that represents a possible technological process for an intelligent martial arts training system was summarized. These applications bring more scientific training and more effective technical analysis to martial arts. Furthermore, this paper delves into specific AI applications in areas such as action recognition, pose estimation, action evaluation, elite athlete support, sports betting, health care, animation generation, and competition field segmentation. By exploring these areas, the research prospects of AI applications in martial arts are comprehensively discussed. This paper inspires further research on martial arts by using AI, which includes helping researchers expand their research ideas and research methods, identify frontier issues, and innovate based on existing research.

Landmark-aware autonomous odometry correction and map pruning for planetary rovers

Planetary rover autonomous localization is paramount for a planetary surface exploration mission. However, existing methods demonstrate limited localization accuracy, mostly due to the unstructured texture characterization of planetary surface. In response, this study presents a novel Neural Radiance Field (NeRF) driven visual odometry correction method that allows for high-precision 6-DoF rover pose estimation and local map pruning. First, an innovative image saliency evaluation approach, combining binarization and feature detection, is introduced to meticulously select landmarks that are conducive to rover re-localization. Subsequently, we conduct 3D reconstruction and rendering of the chosen landmarks based on a-priori knowledge of planetary surface images and their Neural Radiance Field (NeRF) models. High-precision odometry correction is achieved through the optimization of photometric loss between NeRF rending images and real images. Simultaneously, the odometry correction mechanism is employed in an autonomous manner to refine the NeRF model of the corresponding landmark, leading to an improved local map and gradually enhanced rover localization accuracy. Numerical simulation and experiment trials are carried out to evaluate the performance of the proposed method, results of which demonstrate state-of-the-art rover re-localization accuracy and local map pruning.

Visual inspection and grasping methods based on deep learning

Aiming at the problems of existing robot grasping systems that have high hardware requirements, are difficult to adapt to different objects, and produce large harmful torques during the grasping process, a visual detection and grasping method based on deep learning is proposed. The channel attention mechanism is used to improve YOLO-V3, enhance the network's ability to extract image features, improve the effect of target detection in complex environments, and increase the average recognition rate compared with the original . Aiming at the problem of discreteness of the current pose estimation angle, a minimum area bounding rectangle (MABR) algorithm based on the main network embedded in the Visual Geometry Group 16 (VGG-16) is proposed to perform grasping pose estimation and angle optimization. The average error between the improved grasping angle and the actual angle of the target is less than , which greatly reduces the harmful torque applied by the two-finger manipulator to the object during the grasping process. A visual grasping system was built using UR5 robotic arm, pneumatic two-finger manipulator, Realsense D435 camera and ATI-Mini45 six-dimensional force sensor. Experiments show that the proposed method can effectively grasp and classify different objects, has low hardware requirements, and reduces harmful torque by about , thereby reducing damage to objects. It has good application prospects.

A systematic review of computer vision-based personal protective equipment compliance in industry practice: advancements, challenges and future directions

Computerized compliance of Personal Protective Equipment (PPE) is an emerging topic in academic literature that aims to enhance workplace safety through the automation of compliance and prevention of PPE misuse (which currently relies on manual employee supervision and reporting). Although trends in the scientific literature indicate a high potential for solving the compliance problem by employing computer vision (CV) techniques, the practice has revealed a series of barriers that limit their wider applications. This article aims to contribute to the advancement of CV-based PPE compliance by providing a comparative review of high-level approaches, algorithms, datasets, and technologies used in the literature. The systematic review highlights industry-specific challenges, environmental variations, and computational costs related to the real-time management of PPE compliance. The issues of employee identification and identity management are also discussed, along with ethical and cybersecurity concerns. Through the concept of CV-based PPE Compliance 4.0, which encapsulates PPE, human, and company spatio-temporal variabilities, this study provides guidelines for future research directions for addressing the identified barriers. The further advancements and adoption of CV-based solutions for PPE compliance will require simultaneously addressing human identification, pose estimation, object recognition and tracking, necessitating the development of corresponding public datasets.

Multi-Modal Pose Representations for 6-DOF Object Tracking

Pose estimation methods for robotics should return a distribution of poses rather than just a single pose estimate. Motivated by this, in this work we investigate multi-modal pose representations for reliable 6-DoF object tracking. A neural network architecture for simultaneous object segmentation and estimation of fiducial points of the object on RGB images is proposed. Given a priori probability distribution of object poses a particle filter is employed to estimate the posterior probability distribution of object poses. An advanced observation model relying on matching the projected 3D model with the segmented object and a distance transform-based object representation is used to weight samples representing the probability distribution. Afterwards, the object pose determined by the PnP algorithm is included in the probability distribution via replacing a particle with the smallest weight. Next, a k-means++ algorithm is executed to determine modes in a multi-modal probability distribution. A multi-swarm particle swarm optimization is then executed to determine the finest modes in the probability distribution. A subset of particles for final pose optimization is found in a multi-criteria analysis using the TOPSIS algorithm. They are verified using conflicting criteria that are determined on the basis of object keypoints, segmented object, and the distance transform. On the challenging YCB-Video dataset it outperforms recent algorithms for both object pose estimation and object pose tracking.

A biomimetic fruit fly robot for studying the neuromechanics of legged locomotion

For decades, the field of biologically inspired robotics has leveraged insights from animal locomotion to improve the walking ability of legged robots. Recently, 'biomimetic' robots have been developed to model how specific animals walk. By prioritizing biological accuracy to the target organism rather than the application of general principles from biology, these robots can be used to develop detailed biological hypotheses for animal experiments, ultimately improving our understanding of the biological control of legs while improving technical solutions. In this work, we report the development and validation of the robot Drosophibot II, a meso-scale robotic model of an adult fruit fly,Drosophila melanogaster. This robot is novel for its close attention to the kinematics and dynamics ofDrosophila, an increasingly important model of legged locomotion. Each leg's proportions and degrees of freedom have been modeled afterDrosophila3D pose estimation data. We developed a program to automatically solve the inverse kinematics necessary for walking and solve the inverse dynamics necessary for mechatronic design. By applying this solver to a fly-scale body structure, we demonstrate that the robot's dynamics fit those modeled for the fly. We validate the robot's ability to walk forward and backward via open-loop straight line walking with biologically inspired foot trajectories. This robot will be used to test biologically inspired walking controllers informed by the morphology and dynamics of the insect nervous system, which will increase our understanding of how the nervous system controls legged locomotion.

Pose Estimation of a Cobot Implemented on a Small AI-Powered Computing System and a Stereo Camera for Precision Evaluation.

The precision of robotic manipulators in the industrial or medical field is very important, especially when it comes to repetitive or exhaustive tasks. Geometric deformations are the most common in this field. For this reason, new robotic vision techniques have been proposed, including 3D methods that made it possible to determine the geometric distances between the parts of a robotic manipulator. The aim of this work is to measure the angular position of a robotic arm with six degrees of freedom. For this purpose, a stereo camera and a convolutional neural network algorithm are used to reduce the degradation of precision caused by geometric errors. This method is not intended to replace encoders, but to enhance accuracy by compensating for degradation through an intelligent visual measurement system. The camera is tested and the accuracy is about one millimeter. The implementation of this method leads to better results than traditional and simple neural network methods.

Dual Leap Motion Controller 2: A Robust Dataset for Multi-view Hand Pose Recognition

This paper presents Multi-view Leap2 Hand Pose Dataset (ML2HP Dataset), a new dataset for hand pose recognition, captured using a multi-view recording setup with two Leap Motion Controller 2 devices. This dataset encompasses a diverse range of hand poses, recorded from different angles to ensure comprehensive coverage. The dataset includes real images with the associated precise and automatic hand properties, such as landmark coordinates, velocities, orientations, and finger widths. This dataset has been meticulously designed and curated to maintain a balance in terms of subjects, hand poses, and the usage of right or left hand, ensuring fairness and parity. The content includes 714,000 instances from 21 subjects of 17 different hand poses (including real images and 247 associated hand properties). The multi-view setup is necessary to mitigate hand occlusion phenomena, ensuring continuous tracking and pose estimation required in real human-computer interaction applications. This dataset contributes to advancing the field of multimodal hand pose recognition by providing a valuable resource for developing advanced artificial intelligence human computer interfaces.

Balancing Efficiency and Accuracy: Enhanced Visual Simultaneous Localization and Mapping Incorporating Principal Direction Features

In visual Simultaneous Localization and Mapping (SLAM), operational efficiency and localization accuracy are equally crucial evaluation metrics. We propose an enhanced visual SLAM method to ensure stable localization accuracy while improving system efficiency. It can maintain localization accuracy even after reducing the number of feature pyramid levels by 50%. Firstly, we innovatively incorporate the principal direction error, which represents the global geometric features of feature points, into the error function for pose estimation, utilizing Pareto optimal solutions to improve the localization accuracy. Secondly, for loop-closure detection, we construct a feature matrix by integrating the grayscale and gradient direction of an image. This matrix is then dimensionally reduced through aggregation, and a multi-layer detection approach is employed to ensure both efficiency and accuracy. Finally, we optimize the feature extraction levels and integrate our method into the visual system to speed up the extraction process and mitigate the impact of the reduced levels. We comprehensively evaluate the proposed method on local and public datasets. Experiments show that the SLAM method maintained high localization accuracy after reducing the tracking time by 24% compared with ORB SLAM3. Additionally, the proposed loop-closure-detection method demonstrated superior computational efficiency and detection accuracy compared to the existing methods.

Rat movements reflect internal decision dynamics in an evidence accumulation task.

Perceptual decision-making involves multiple cognitive processes, including accumulation of sensory evidence, planning, and executing a motor action. How these processes are intertwined is unclear; some models assume that decision-related processes precede motor execution, whereas others propose that movements reflecting on-going decision processes occur before commitment to a choice. Here we combine and apply two complementary methods to study the relationship between decision processes and the movements leading up to a choice. The first is a free response pulse-based evidence accumulation task, in which stimuli continue until choice is reported and the second is a motion-based drift diffusion model (mDDM), in which movement variables from video pose estimation constrain decision parameters on a trial-by-trial basis. We find the mDDM provides a better model fit to rats' decisions in the free response accumulation task than traditional DDM models. Interestingly, on each trial we observed a period of time, prior to choice, that was characterized by head immobility. The length of this period was positively correlated with the rats' decision bounds and stimuli presented during this period had the greatest impact on choice. Together these results support a model in which internal decision dynamics are reflected in movements and demonstrate that inclusion of movement parameters improves the performance of diffusion-to-bound decision models.