Pose Estimation Research Articles

A LiDAR SLAM Algorithm Considering Dynamic Extraction of Feature Points in Underground Coal Mine

Abstract. To address the absence of GNSS signals in underground coal mines and the susceptibility of mainstream LiDAR SLAM to degradation due to insufficient feature constraints, this paper proposes a tightly-coupled SLAM algorithm incorporating LiDAR and IMU for use in such environments. The paper initially introduces a dynamic feature point extraction strategy, where the number of corner feature points can be dynamically adjusted based on the occurrence of degradation in the underground coal mine environment. This approach constructs a constraint matrix with rich and robust feature information, enhancing pose estimation accuracy in environments with inadequate feature constraints, mitigating degradation effects, and reducing global cumulative error. Subsequently, the consistent building of the underground coal mine environment is achieved through back-end factor graph optimization. Finally, to validate the effectiveness of the method, experimental analysis is conducted in an underground coal mine. The results demonstrate that LeGO_LOAM exhibits poor robustness in underground coal mines and fails to construct a globally consistent pose estimation and map. Conversely, the pose estimation error of the proposed method in this paper is 50.93% lower in the plane direction and 42.13% lower in the elevation direction compared to LIO_SAM. This underscores the method's significance as crucial technical support for the intelligent perception and positioning of robots in underground coal mines.

Ground-LIO: enhanced LiDAR-inertial odometry for ground robots based on ground optimization

Abstract In the field of LiDAR-based Simultaneous Localization and Mapping, the potential of ground point clouds to enhance pose estimation in mobile robots has yet to be fully realized. This paper focuses on leveraging ground point clouds to improve the performance of LiDAR-Inertial Odometry (LIO) systems for ground-based mobile robots. We begin by analyzing the characteristics of ground point clouds and the typical types of noise that affect their extraction and utilization. Ground point clouds are then extracted from denoised data. Given the generally local planar nature of ground point clouds, we propose a segmentation-and-refitting approach to process them. This method reduces the computational burden of residual calculation in pose estimation by avoiding redundant plane fitting. Additionally, we introduce a data structure designed for the efficient management and utilization of ground point clouds obtained through segmentation and refitting. This structure is particularly suited to the ground point cloud data produced by our method, enabling efficient access and registration through the continuous maintenance and consolidation of local plane parameters. Our method has been integrated into advanced LIO systems (Bai et al 2022 IEEE Robot. Autom. Lett. 7 4861–8), and experimental results on challenging datasets demonstrate its promising performance.

Cross-Spectral Navigation with Sensor Handover for Enhanced Proximity Operations with Uncooperative Space Objects

Close-proximity operations play a crucial role in emerging mission concepts, such as Active Debris Removal or small celestial bodies exploration. When approaching a non-cooperative target, the increased risk of collisions and reduced reliance on ground intervention necessitate autonomous on-board relative pose (position and attitude) estimation. Although navigation strategies relying on monocular cameras which operate in the visible (VIS) spectrum have been extensively studied and tested in flight for navigation applications, their accuracy is heavily related to the target’s illumination conditions, thus limiting their applicability range. The novelty of the paper is the introduction of a thermal-infrared (TIR) camera to complement the VIS one to mitigate the aforementioned issues. The primary goal of this work is to evaluate the enhancement in navigation accuracy and robustness by performing VIS-TIR data fusion within an Extended Kalman Filter (EKF) and to assess the performance of such navigation strategy in challenging illumination scenarios. The proposed navigation architecture is tightly coupled, leveraging correspondences between a known uncooperative target and feature points extracted from multispectral images. Furthermore, handover from one camera to the other is introduced to enable seamlessly operations across both spectra while prioritizing the most significant measurement sources. The pipeline is tested on Tango spacecraft synthetically generated VIS and TIR images. A performance assessment is carried out through numerical simulations considering different illumination conditions. Our results demonstrate that a combined VIS-TIR navigation strategy effectively enhances operational robustness and flexibility compared to traditional VIS-only navigation chains.

Autonomous Positioning for Mobile Vehicles Based on Visual‐Inertial Fusion in Challenging Dark Roadway Scenes

ABSTRACTAccurate positioning for autonomous driven underground mining vehicles (UMVs) and coal mine robots (CMRs) is indeed one of the cores in the intelligentization of coal mining. Completely different from positioning on the ground and in parking scenes, there will be great difficulties in realizing the accurate active positioning for UMVs shuttled in dark and narrow roadways. We propose an effective visual and inertial fusion positioning method for autonomous CMRs and roadheaders in challenging roadway scenarios based on the odometer‐aided inertial navigation system and visual pose estimation system. Velocity information of the odometer is adapted to restrain the error accumulation of inertial positioning based on a Kalman filter. The hybrid visual feature detection algorithm is put forward to improve the accuracy and robustness of visual observation information in a dark environment. Autonomous experiments for CMRs and roadheaders are separately performed in the narrow roadway and dark passageway to demonstrate the applicability of our localization method. The proposed approach outperforms the subsystems and existing methods in accuracy and has outstanding stability.

BodyFlow: An Open-Source Library for Multimodal Human Activity Recognition

Human activity recognition is a critical task for various applications across healthcare, sports, security, gaming, and other fields. This paper presents BodyFlow, a comprehensive library that seamlessly integrates human pose estimation and multiple-person estimation and tracking, along with activity recognition modules. BodyFlow enables users to effortlessly identify common activities and 2D/3D body joints from input sources such as videos, image sets, or webcams. Additionally, the library can simultaneously process inertial sensor data, offering users the flexibility to choose their preferred input, thus facilitating multimodal human activity recognition. BodyFlow incorporates state-of-the-art algorithms for 2D and 3D pose estimation and three distinct models for human activity recognition.

3D skeleton aware driver behavior recognition framework for autonomous driving system

The recognition of the driver’s behaviors inside an autonomous vehicle can effectively address emergency handling in autonomous driving and is crucial for ensuring the driver’s safety. Driver behavior recognition is a challenging task due to factors such as variations, diversities, complexities, and strong interferences in behaviors. In this paper, to realize the application in the autonomous driving scenes, a novel 3D skeleton aware behavior recognition framework is proposed to recognize various driver behaviors in autonomous driving systems. First, a 3D human pose estimation network (Pose-GTFNet) with temporal Transformer and spatial graph convolutional network (GCN) is designed to infer 3D human poses from 2D pose sequences. Second, based on the obtained 3D human pose sequences, a behavior recognition network (Beh-MSFNet) with multi-skeleton feature fusion is designed to recognize driver behaviors. In the experiments, the Pose-GTFNet and Beh-MSFNet can get the best performance compared with most state-of-the-art (SOTA) methods on the Human3.6M human pose dataset, JHMDB and SHREC action recognition dataset, respectively. In addition, the proposed driver behavior recognition framework can achieve SOTA performance on the Drive&Act and Driver-Skeleton driver behavior datasets.

ESFuse: Weak Edge Structure Perception Network for Infrared and Visible Image Fusion

Infrared and visible image fusion (IVIF) fully integrates the complementary features of different modal images, and the fused image provides a more comprehensive and objective interpretation of the scene compared to each source image, thus attracting extensive attention in the field of computer vision in recent years. However, current fusion methods usually center their attention on the extraction of prominent features, falling short of adequately safeguarding subtle and diminutive structures. To address this problem, we propose an end-to-end unsupervised IVIF method (ESFuse), which effectively enhances fine edges and small structures. In particular, we introduce a two-branch head interpreter to extract features from source images of different modalities. Subsequently, these features are fed into the edge refinement module with the detail injection module (DIM) to obtain the edge detection results of the source image, improving the network’s ability to capture and retain complex details as well as global information. Finally, we implemented a multiscale feature reconstruction module to obtain the final fusion results by combining the output of the DIM with the output of the head interpreter. Extensive IVIF fusion experiments on existing publicly available datasets show that the proposed ESFuse outperforms the state-of-the-art(SOTA) methods in both subjective vision and objective evaluation, and our fusion results perform well in semantic segmentation, target detection, pose estimation and depth estimation tasks. The source code has been availabled.

NeRF-based Localization and Meshing with Wearable Laser Scanning System: A Case Study in Underground Environment

Abstract. Due to the subterranean scene’s poor lighting conditions and variability of the environment, real-time localizing and meshing in complex underground scenes present a challenging task, with high-potential applications in mining and tunnel protection. In this work, we propose a method that combines SLAM and NeRF for mesh reconstruction in the underground environment with a wearable device. First, a LiDAR-Inertial odometry is used for pose estimation. The resulting poses and sequential laser frames are synchronized to generate a novel data structure, scan-block, which is crucial for improving efficiency and ensuring local precision. This structure is designed to enhance efficiency and ensure local precision. Finally, the generated scan-blocks are passed to the backend NeRF for real-time mesh reconstruction. The experiments are carried out in a complex underground space. The experimental results proved that this method achieved good results. The demo video can be found at https://youtu.be/_y1vbaW06EM?si= hFnAl12u-ffU933h.

Axes-aligned non-linear optimized PnP algorithm

Pose estimation is an important component of many real-world computer vision systems. Most existing pose estimation algorithms need a large number of point correspondences to accurately determine the pose of an object. Since the number of point correspondences depends on the object’s appearance, lighting and other external conditions, detecting many points may not be feasible. In many real-world applications, the movement of objects is limited, e.g. due to gravity. Hence, detecting objects with only three degrees of freedom is usually sufficient. This allows us to improve the accuracy of pose estimation by changing the underlying equations of the perspective-n-point problem to three variables instead of six. By using the simplified equations, our algorithm is more robust against detection errors with limited point correspondences. In this article, we study three scenarios where such constraints apply. The first one is about parking a vehicle on a specific spot. Here, a stationary camera is detecting the vehicle to assist the driver. The second scenario describes the perspective of a moving camera detecting objects in its environment. This scenario is common for driver assistance systems, autonomous cars or mobile robots. Third, we describe a camera observing objects from a birds-eye view, which occurs in industrial applications. In all three scenarios, observed objects can only move in the ground plane and rotate around the vertical axis. Hence, three degrees of freedom are sufficient to estimate the pose. Experiments with synthetic data and real-world photographs have shown that our algorithm outperforms state-of-the-art pose estimation algorithms. Depending on the scenario, our algorithm is able to achieve 50% better accuracy, while being equally fast.

A Depth Awareness and Learnable Feature Fusion Network for Enhanced Geometric Perception in Semantic Correspondence.

Deep learning is becoming the most widely used technology for multi-sensor data fusion. Semantic correspondence has recently emerged as a foundational task, enabling a range of downstream applications, such as style or appearance transfer, robot manipulation, and pose estimation, through its ability to provide robust correspondence in RGB images with semantic information. However, current representations generated by self-supervised learning and generative models are often limited in their ability to capture and understand the geometric structure of objects, which is significant for matching the correct details in applications of semantic correspondence. Furthermore, efficiently fusing these two types of features presents an interesting challenge. Achieving harmonious integration of these features is crucial for improving the expressive power of models in various tasks. To tackle these issues, our key idea is to integrate depth information from depth estimation or depth sensors into feature maps and leverage learnable weights for feature fusion. First, depth information is used to model pixel-wise depth distributions, assigning relative depth weights to feature maps for perceiving an object's structural information. Then, based on a contrastive learning optimization objective, a series of weights are optimized to leverage feature maps from self-supervised learning and generative models. Depth features are naturally embedded into feature maps, guiding the network to learn geometric structure information about objects and alleviating depth ambiguity issues. Experiments on the SPair-71K and AP-10K datasets show that the proposed method achieves scores of 81.8 and 83.3 on the percentage of correct keypoints (PCK) at the 0.1 level, respectively. Our approach not only demonstrates significant advantages in experimental results but also introduces the depth awareness module and a learnable feature fusion module, which enhances the understanding of object structures through depth information and fully utilizes features from various pre-trained models, offering new possibilities for the application of deep learning in RGB and depth data fusion technologies. We will also continue to focus on accelerating model inference and optimizing model lightweighting, enabling our model to operate at a faster speed.

ADS–SLAM: a semantic SLAM based on adaptive motion compensation and semantic information for dynamic environments

Abstract Static environment assumptions are a prerequisite for simultaneous localization and mapping (SLAM), while interference from dynamic objects in the environment can seriously impair the system’s localization accuracy. Recently, many works have combined deep learning and geometric constraints to attenuate the interference of dynamic objects, but poor real-time performance and low accuracy in high dynamic scenes still exist. In this paper, we propose a semantic SLAM algorithm for complex dynamic scenes named ADS–SLAM. Our system combines the advantages of semantic information and motion constraints to remove dynamic points during tracking and localization. First, an adaptive dynamic point detection method based on epipolar constraint between adjacent frames is designed to adapt to the changes of object motion states and a motion area detection method based on Gaussian mixture model and Kalman Filter is utilized to effectively compensate the missed motion areas. Second, an object detection network with improved inference in the backend is utilized to extract prior object semantics. Lastly, the multi-level information is integrated in order to comprehensively screen all dynamic points in the environment and utilize only static points for pose estimation and optimization. Experimental evaluations on challenging public datasets and outdoor dynamic environments demonstrate that our algorithm achieves high localization accuracy in almost all dynamic scenarios compared to the current state-of-the-art SLAM algorithms, with the highest accuracy in high dynamic scenarios, and shows real-time performance for practical applications.

Research on the Detection Algorithm of Elderly Falling Behavior Based on AlphaPose Model

This study aims to address the challenge of quickly and accurately detecting high-risk behaviors such as falls in elderly people in hardware environments with limited power consumption and cost. Therefore, this article proposes an optimized behavior anomaly detection algorithm based on the AlphaPose model. The algorithm first improved the human target detection and pose estimation models to achieve faster pedestrian detection and pose joint point inference; Then, using the adjusted AlphaPose model, efficiently obtain the image coordinate data of each joint point in the human body; Finally, by analyzing the relationship between the linear velocities of the human head and hip joints, as well as the changes in the angle between the vertical line of the human center and the X-axis of the image, it is determined whether a fall event has occurred. This study deployed the proposed algorithm on the JetsonNano embedded development board and compared its performance with mainstream fall detection algorithms based on human posture, including YOLOv3+Pose, YOLOv4+Pose, YOLOv5+Pose, trtpose, and NanoDet+Pose. The experimental results show that on the set embedded platform, when the image resolution is 320×240, this algorithm achieves a detection rate of 8.83 frames per second and an accuracy of 0.913, both of which are superior to the compared algorithms. In summary, this algorithm has high real-time performance and accuracy, and can effectively identify the falling behavior of elderly people in a timely manner.

Human Pose Classification Based on Pose Long Short-Term Memory

Objective: At present, pose classification research is a hot topic in the field of computer vision, playing a crucial role in surveillance security, and motion data mining. The research serves as a crucial follow-up to three dimensional (3D) pose estimation and makes a significant contribution to the advancement of artificial intelligence. Pose classification is a complex problem involving a large amount of complicated data, making its digital modeling and further processing challenging. Methods: This article proposes the Pose long short - term memory (LSTM) method to classify 3D poses reconstructed from 2D image sequences. Result: Compared with traditional methods, the experimental tests show that the performance of the Pose LSTM is more superior in all aspects. Conclusion: Pose LSTM is particularly effective for recognizing continuous action poses because they can capture temporal dependencies in sequential data. In the context of pose recognition, it can analyze the sequence of poses over time and understand how these poses transition from one to another, making it possible to accurately classify or predict actions that unfold over a series of frames.

DenseFusion-DA2: End-to-End Pose-Estimation Network Based on RGB-D Sensors and Multi-Channel Attention Mechanisms.

Notably, 6D pose estimation is a critical technology that enables robotics to perceive and interact with their operational environment. However, occlusion causes a loss of local features, which, in turn, restricts the estimation accuracy. To address these challenges, this paper proposes an end-to-end pose-estimation network based on a multi-channel attention mechanism, DA2Net. Firstly, a multi-channel attention mechanism, designated as "DA2Net", was devised using A2-Nets as its foundation. This mechanism is constructed in two steps. In the first step, the essential characteristics are extracted from the global feature space through the second-order attention pool. In the second step, a feature map is generated by the integration of position and channel attention. Subsequently, the extracted key features are assigned to each position of the feature map, enhancing both the feature representation capacity and the overall performance. Secondly, the designed attention mechanism is introduced into both the feature fusion and pose iterative refinement networks to enhance the network's capacity to acquire local features thus improving its overall performance. The experimental results demonstrated that the estimation accuracy of DenseFusion-DA2 on the LineMOD dataset was approximately 3.4% higher than that of DenseFusion. Furthermore, the estimation accuracy surpassed that of PoseCNN, PVNet, SSD6D, and PointFusion by 8.3%, 11.1%, 20.3%, and 23.8%, respectively. The estimation accuracy also shows a significant advantage on the Occluded LineMOD and HR-Vision datasets. This research not only presents a more efficient solution for robot perception but also introduces novel ideas and methods for technological advancements and applications in related fields.

Human Fall Motion Prediction: Fall Motion Forecasting and Detection with GRU

The human fall motion prediction system is a preventive tool aimed at reducing the risk of falls. In our research, we developed a deep learning model that utilizes pose estimation to track human body posture and integrated this with a Gated Recurrent Unit (GRU) to forecast human motion and predict falls. GRU, an enhancement of Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) models offers improved memorization and more efficient memory usage and performance. Our study presents the human fall motion prediction, which combines the forecasting and classification of potential falls.The CAUCAFall dataset is used as the benchmark of this study, which contains the image sequences of single human motion with ten actions conducted by ten actors. We employed the YOLOv8 Pose model to track the 2D human body pose as the input in our system. A thorough evaluation of the CAUCAFall dataset highlights the effectiveness of our proposed system. Evaluation using the CAUCAFall dataset demonstrates that the model achieved a Mean Per Joint Position Error (MPJPE) of 4.65 pixels from the ground truth, with a 70% accuracy rate in fall prediction. However, the model also exhibited a Mean Relative Error (MRE) of 0.3, indicating that 30% of the predictions were incorrect. These findings underscore the potential of the GRU-based system in fall prevention

Enhanced Infant Movement Analysis Using Transformer-Based Fusion of Diverse Video Features for Neurodevelopmental Monitoring

Neurodevelopment is a highly intricate process, and early detection of abnormalities is critical for optimizing outcomes through timely intervention. Accurate and cost-effective diagnostic methods for neurological disorders, particularly in infants, remain a significant challenge due to the heterogeneity of data and the variability in neurodevelopmental conditions. This study recruited twelve parent–infant pairs, with infants aged 3 to 12 months. Approximately 25 min of 2D video footage was captured, documenting natural play interactions between the infants and toys. We developed a novel, open-source method to classify and analyse infant movement patterns using deep learning techniques, specifically employing a transformer-based fusion model that integrates multiple video features within a unified deep neural network. This approach significantly outperforms traditional methods reliant on individual video features, achieving an accuracy of over 90%. Furthermore, a sensitivity analysis revealed that the pose estimation contributed far less to the model’s output than the pre-trained transformer and convolutional neural network (CNN) components, providing key insights into the relative importance of different feature sets. By providing a more robust, accurate and low-cost analysis of movement patterns, our work aims to enhance the early detection and potential prediction of neurodevelopmental delays, whilst providing insight into the functioning of the transformer-based fusion models of diverse video features.

Validation of Automated Countermovement Vertical Jump Analysis: Markerless Pose Estimation vs. 3D Marker-Based Motion Capture System

This study aimed to validate the automated temporal analysis of countermovement vertical jump (CMJ) using MMPose, a markerless pose estimation framework, by comparing it with the gold-standard 3D marker-based motion capture system. Twelve participants performed five CMJ trials, which were simultaneously recorded using the marker-based system and two smartphone cameras capturing both sides of the body. Key kinematic points, including center of mass (CoM) and toe trajectories, were analyzed to determine jump phases and temporal variables. The agreement between methods was assessed using Bland–Altman analysis, root mean square error (RMSE), and Pearson’s correlation coefficient (r), while consistency was evaluated via intraclass correlation coefficient (ICC 3,1) and two-way repeated-measures ANOVA. Cohen’s effect size (d) quantified the practical significance of differences. Results showed strong agreement (r > 0.98) with minimal bias and narrow limits of agreement for most variables. The markerless system slightly overestimated jump height and CoM vertical velocity, but ICC values (ICC > 0.91) confirmed strong reliability. Cohen’s d values were near zero, indicating trivial differences, and no variability due to recording side was observed. Overall, MMPose proved to be a reliable alternative for in-field CMJ analysis, supporting its broader application in sports and rehabilitation settings.

3D Head Pose Estimation via Normal Maps: A Generalized Solution for Depth Image, Point Cloud, and Mesh

Head pose estimation plays a crucial role in various applications, including human–machine interaction, autonomous driving systems, and 3D reconstruction. Current methods address the problem primarily from a 2D perspective, which limits the efficient utilization of 3D information. Herein, a novel approach, called pose orientation‐aware network (POANet), which leverages normal maps for orientation information embedding, providing abundant and robust head pose information, is introduced. POANet incorporates the axial signal perception module and the rotation matrix perception module, these lightweight modules make the approach achieve state‐of‐the‐art (SOTA) performance with few computational costs. This method can directly analyze various topological 3D data without extensive preprocessing. For depth images, POANet outperforms existing methods on the Biwi Kinect head pose dataset, reducing the mean absolute error (MAE) by ≈30% compared to the SOTA methods. POANet is the first method to perform rigid head registration in a landmark‐free manner. It also incorporates few‐shot learning capabilities and achieves an MAE of about on the Headspace dataset. These features make POANet a superior alternative to traditional generalized Procrustes analysis for mesh data processing, offering enhanced convenience for human phenotype studies.

Applications of AI in martial arts: A survey

Artificial intelligence (AI) has become an interdisciplinary subject that integrates digital image processing, machine learning, and computer science, etc. A lot of work in martial arts benefits from AI technology. In this paper, we highlight and summarize the applications of AI in martial arts and indicate the main progress of related studies. Through the investigation of martial arts styles, tasks, data acquisition methods, and algorithm improvements, a paradigm that represents a possible technological process for an intelligent martial arts training system was summarized. These applications bring more scientific training and more effective technical analysis to martial arts. Furthermore, this paper delves into specific AI applications in areas such as action recognition, pose estimation, action evaluation, elite athlete support, sports betting, health care, animation generation, and competition field segmentation. By exploring these areas, the research prospects of AI applications in martial arts are comprehensively discussed. This paper inspires further research on martial arts by using AI, which includes helping researchers expand their research ideas and research methods, identify frontier issues, and innovate based on existing research.

Landmark-aware autonomous odometry correction and map pruning for planetary rovers

Planetary rover autonomous localization is paramount for a planetary surface exploration mission. However, existing methods demonstrate limited localization accuracy, mostly due to the unstructured texture characterization of planetary surface. In response, this study presents a novel Neural Radiance Field (NeRF) driven visual odometry correction method that allows for high-precision 6-DoF rover pose estimation and local map pruning. First, an innovative image saliency evaluation approach, combining binarization and feature detection, is introduced to meticulously select landmarks that are conducive to rover re-localization. Subsequently, we conduct 3D reconstruction and rendering of the chosen landmarks based on a-priori knowledge of planetary surface images and their Neural Radiance Field (NeRF) models. High-precision odometry correction is achieved through the optimization of photometric loss between NeRF rending images and real images. Simultaneously, the odometry correction mechanism is employed in an autonomous manner to refine the NeRF model of the corresponding landmark, leading to an improved local map and gradually enhanced rover localization accuracy. Numerical simulation and experiment trials are carried out to evaluate the performance of the proposed method, results of which demonstrate state-of-the-art rover re-localization accuracy and local map pruning.