Pose Estimation Research Articles

U-Match: Exploring Hierarchy-Aware Local Context for Two-View Correspondence Learning.

Rejecting outlier correspondences is one of the critical steps for successful feature-based two-view geometry estimation, and contingent heavily upon local context exploration. Recent advances focus on devising elaborate local context extractors whereas typically adopting explicit neighborhood relationship modeling at a specific scale, which is intrinsically flawed and inflexible, because 1) severe outliers often populated in putative correspondences and 2) the uncertainty in the distribution of inliers and outliers make the network incapable of capturing adequate and reliable local context from such neighborhoods, therefore resulting in the failure of pose estimation. This prospective study proposes a novel network called U-Match that has the flexibility to enable implicit local context awareness at multiple levels, naturally circumventing the aforementioned issues that plague most existing studies. Specifically, to aggregate multi-level local context implicitly, a hierarchy-aware graph representation module is designed to flexibly encode and decode hierarchical features. Moreover, considering that global context always works collaboratively with local context, an orthogonal local-and-global information fusion module is presented to integrate complementary local and global context in a redundancy-free manner, thus yielding compact feature representations to facilitate correspondence learning. Thorough experimentation across relative pose estimation, homography estimation, visual localization, and point cloud registration affirms U-Match's remarkable capabilities.

Augmented reality safety warnings in roadway work zones: Evaluating the effect of modality on worker reaction times

Given the aging highway infrastructure requiring extensive rebuilding and enhancements, and the consequent rise in the number of work zones, there is an urgent need to develop advanced safety systems to protect workers. While Augmented Reality (AR) holds significant potential for delivering warnings to workers, its integration into roadway work zones remains relatively unexplored. The primary objective of this study is to improve safety measures within roadway work zones by conducting an extensive analysis of how different combinations of multimodal Augmented Reality (AR) warnings influence the reaction times of workers. This paper addresses this gap through a series of experiments that aim to replicate the distinctive conditions of roadway work zones, both in real-world and virtual reality environments. Our approach comprises three key components: an advanced AR system prototype, a VR simulation of AR functionality within the work zone environment, and the Wizard of Oz technique to synchronize user experiences across experiments. To assess reaction times, we leverage both the simple reaction time (SRT) technique and an innovative vision-based metric that utilizes real-time pose estimation. By conducting five experiments in controlled outdoor work zones and indoor VR settings, our study provides valuable information on how various multimodal AR warnings impact workers reaction times. Furthermore, our findings reveal the disparities in reaction times between VR simulations and real-world scenarios, thereby gauging VR’s capability to mirror the dynamics of roadway work zones. Furthermore, our results substantiate the potential and reliability of vision-based reaction time measurements. These insights resonate well with those derived using the SRT technique, underscoring the viability of this approach for tangible real-world uses.

OOD-CV-v2 : An Extended Benchmark for Robustness to Out-of-Distribution Shifts of Individual Nuisances in Natural Images.

Enhancing the robustness of vision algorithms in real-world scenarios is challenging. One reason is that existing robustness benchmarks are limited, as they either rely on synthetic data or ignore the effects of individual nuisance factors. We introduce OOD-CV-v2, a benchmark dataset that includes out-of-distribution examples of 10 object categories in terms of pose, shape, texture, context and the weather conditions, and enables benchmarking of models for image classification, object detection, and 3D pose estimation. In addition to this novel dataset, we contribute extensive experiments using popular baseline methods, which reveal that: 1) Some nuisance factors have a much stronger negative effect on the performance compared to others, also depending on the vision task. 2) Current approaches to enhance robustness have only marginal effects, and can even reduce robustness. 3) We do not observe significant differences between convolutional and transformer architectures. We believe our dataset provides a rich test bed to study robustness and will help push forward research in this area.

Adversarially trained RTMpose: A high-performance, non-contact method for detecting Genu valgum in adolescents

Genu valgum (GV), a prevalent postural deformity in adolescents, is traditionally diagnosed using methods that are complex, costly, and accompanied by radiation risks. To address these challenges, we evaluated 1,519 Chinese adolescents, collecting GV annotations from three medical professionals to establish a robust dataset. Leveraging these annotations, we developed an end-to-end GV prediction model using RTMpose for body landmark extraction from images. However, a key challenge was the inaccuracy of landmarks, which adversely affects downstream tasks. To mitigate this, we harnessed the parallels between pose estimation biases and adversarial perturbations, implementing adversarial training to bolster model robustness against noisy landmark data. Our model achieved a significant improvement, with an accuracy of 75%, compared to the baseline’s 64.25%. These results underscore the model’s efficacy as a high-performance, non-contact GV detection method and demonstrate the effectiveness of adversarial training in enhancing landmark-related tasks, providing a safer, cost-effective alternative for adolescent GV diagnosis.

Occluded human pose estimation based on limb joint augmentation

Occluded human pose estimation based on limb joint augmentation

Advancements in Fingerprint Recognition Through Deep Learning: A Comprehensive Analysis of Novel Algorithms

Abstract. This paper presents a comprehensive exploration of the application of deep learning technology in fingerprint recognition, focusing on the development and impact of several novel algorithms. These algorithms have been specifically designed to address challenges in key sub-fields such as pose estimation, direction field estimation, minutiae extraction, and minutiae matching. By integrating deep learning techniques, these new approaches significantly enhance the accuracy, stability, and efficiency of fingerprint recognition systems. The study demonstrates that these algorithms surpass traditional methods in several critical areas, offering improved precision in recognizing fingerprints, particularly in high-noise environments. Furthermore, the fully differentiable nature of these models contributes to their robustness, enabling more consistent and reliable performance across diverse scenarios. The results underscore the potential for these deep learning-based algorithms to set new benchmarks in the field, with broad implications for their application in security, law enforcement, and other areas requiring reliable biometric authentication. The use of these cutting-edge methods is anticipated to be vital in influencing the direction of fingerprint recognition technology as it develops further, guaranteeing increased security and precision.

A Lightweight Hand Attitude Estimation Method Based on GCN Feature Enhancement

In this study, a hand pose estimation method based on GCN feature enhancement is proposed to address the problem of the time-consuming nature and neglection of the internal relationships between hand joint points, which results in the low accuracy of hand pose estimation. Firstly, a lightweight feature extraction network RexNet is used, and deep separable convolutions are used instead of ordinary convolutions to reduce the model parameters and computational complexity. Secondly, deconvolution is added to the backend of the network to obtain preliminary estimation results of joint points. Finally, the GCN feature enhancement module is used to modify the preliminary estimation results to improve the accuracy of hand pose estimation. The proposed method is tested for accuracy on the CMU-Hand and RHD datasets. The results show that the proposed method achieves an AUC metric of 80.1% on the CMU-Hand dataset and 97.0% on the RHD dataset, and the accuracy of hand pose estimation is high.

Research on 3D humanoid robot pose estimation based on HRNet-Epipolar and CRF robot model by multiple view

Purpose This paper aims to present an unmarked method including entire two-dimensional (2D) and three-dimensional (3D) methods to recover absolute 3D humanoid robot poses from multiview images. Design/methodology/approach The method consists of two separate steps: estimating the 2D poses in multiview images and recovering the 3D poses from the multiview 2D heatmaps. The 2D one is conducted by High-Resolution Net with Epipolar (HRNet-Epipolar), and the Conditional Random Fields Humanoid Robot Pictorial Structure Model (CRF Robot Model) is proposed to recover 3D poses. Findings The performance of the algorithm is validated by experiments developed on data sets captured by four RGB cameras in Qualisys system. It illustrates that the algorithm has higher Mean Per Joint Position Error than Direct Linear Transformation and Recursive Pictorial Structure Model algorithms when estimating 14 joints of the humanoid robot. Originality/value A new unmarked method is proposed for 3D humanoid robot pose estimation. Experimental results show enhanced absolute accuracy, which holds important theoretical significance and application value for humanoid robot pose estimation and motion performance testing.

Efficient Encoding and Aligning Viewpoints for 6D Pose Estimation of Unseen Industrial Parts

Abstract Estimating the 6D poses of industrial parts is a fundamental task in automated industries. However, the scarcity of industrial part datasets and the effort involved to retrain networks present challenges when estimating poses for unseen parts. Although a few pre-trained networks demonstrate effectiveness on unseen objects, they often struggle to encode correct viewpoint for unseen industrial parts, which have significant geometrical differences compared to the pre-trained objects. Additionally, they overlook the viewpoint non-uniformity that frequently occurs in industrial settings, resulting in significant 3D rotation errors. To address these issues, a novel 6D pose estimator for unseen industrial parts is proposed. First, a Self-to-Inter (S2I) viewpoint encoder is introduced to efficiently generate discriminative descriptors that capture the viewpoint information of the observed image. The S2I viewpoint encoder utilizes an Inter-viewpoint attention module to facilitate prior viewpoint communication and leverages a saliency descriptor selection strategy to boost inference speed. Second, a Viewpoint Alignment module (VAM) is established and integrated with the ICP refiner. The VAM aligns non-uniform viewpoints in an analytical paradigm, leading to enhanced efficiency of the refinement process and more accurate final predictions. Experimental results on the LINEMOD dataset demonstrate competitive performance compared to state-of-the-art methods. Furthermore, the experiments conducted on eight unseen industrial parts validate the exceptional generalizability of our method, highlighting its potential in industrial applications.

PCNet: a human pose compensation network based on incremental learning for sports actions estimation

Human pose estimation has a wide range of applications. Existing methods perform well in conventional domains, but there are certain defects when they are applied to sports activities. The first is lack of estimation of the extremity posture, making it impossible to comprehensively evaluate the movement posture; the second is insufficient occlusion handling. Therefore, we propose a human pose compensation network based on incremental learning, which obtains shared weights to extract detailed features under the premise of limited extremity training data. We propose a higher-order feature compensator (HOF-compensator) to embed the attributes of the extremity into the torso and limbs topology structure, building a complete higher-order feature. In addition, to improve the occlusion handling performance, we propose an occlusion feature enhancement attention mechanism (OFE-attention) that can identify occluded keypoints and enhance attention to occlusion areas. We design comparative experiments on three public datasets and a self-built sports dataset, achieving the highest mean accuracy among all comparative methods. In addition, we design a series of ablation analysis and visualization displays to verify that our method performs best in sports pose estimation.

Enhancing manual inspection in semiconductor manufacturing with integrated augmented reality solutions

On-site routine inspection often remains a manual operation in the semiconductor manufacturing industry because implementing automated solutions can be costly and technically challenging in such a highly controlled and complex environment. The manual inspection is prone to errors due to the impact of demanding physical and mental workloads. This paper presents an integrated Augmented Reality (AR) solution developed to assist manual inspection tasks in the supporting areas of semiconductor manufacturing, referred to as the sub-fab. The solution is accessible to a human worker wearing an AR headset during the inspection process at the location. We propose a system framework to deploy computational intelligences of varying granularity provided by the solution across cloud, edge, and device levels, accommodating constraints within the sub-fab. A machine maintenance module helps estimate and monitor the health condition of running scrubbers. Incorrect intentions performed by the worker on the scrubber control panel are detected through hand gesture recognition. This instantly prompts warning messages in the AR headset to prevent subsequent wrong actions. The solution can also identify abnormal device states through 6D pose estimation of objects enabled by machine learning models. A test scenario demonstrates how these functional features enhance the inspection efficiency and quality by reducing human workloads. This work demonstrates that semiconductor manufacturing may require AR-assisted functions different from those needed or common in other industrial sectors. It also highlights the potential of AR technology for reducing operational human errors in manual tasks.

Real-time location of acupuncture points based on anatomical landmarks and pose estimation models

IntroductionPrecise identification of acupuncture points (acupoints) is essential for effective treatment, but manual location by untrained individuals can often lack accuracy and consistency. This study proposes two approaches that use artificial intelligence (AI) specifically computer vision to automatically and accurately identify acupoints on the face and hand in real-time, enhancing both precision and accessibility in acupuncture practices.MethodsThe first approach applies a real-time landmark detection system to locate 38 specific acupoints on the face and hand by translating anatomical landmarks from image data into acupoint coordinates. The second approach uses a convolutional neural network (CNN) specifically optimized for pose estimation to detect five key acupoints on the arm and hand (LI11, LI10, TE5, TE3, LI4), drawing on constrained medical imaging data for training. To validate these methods, we compared the predicted acupoint locations with those annotated by experts.ResultsBoth approaches demonstrated high accuracy, with mean localization errors of less than 5 mm when compared to expert annotations. The landmark detection system successfully mapped multiple acupoints across the face and hand even in complex imaging scenarios. The data-driven approach accurately detected five arm and hand acupoints with a mean Average Precision (mAP) of 0.99 at OKS 50%.DiscussionThese AI-driven methods establish a solid foundation for the automated localization of acupoints, enhancing both self-guided and professional acupuncture practices. By enabling precise, real-time localization of acupoints, these technologies could improve the accuracy of treatments, facilitate self-training, and increase the accessibility of acupuncture. Future developments could expand these models to include additional acupoints and incorporate them into intuitive applications for broader use.

Innovative hip-knee health evaluation: Leveraging pose estimation for single-leg stance postural stability assessment

The analysis of the Hip and Knee (HK) joint angles during single-leg stance (SLS) activity contributes to a great understanding of the bio-mechanical mechanisms and balance maintenance across different age groups. Comprehending how these joints operate in the dynamic state is critical for identifying age-related changes in joint control and stability which can contribute to reducing the risk of falls and improving mobility among people. The possibility of revealing the health of HK joints without resorting to an MRI scan is the significance of this work. By obtaining the data on joint angles during the SLS test, we succeeded in identifying those with a higher risk of HK issues and thus the possibility of early intervention and treatment. Our proposed work is going to use a pose estimation technique that will track the trajectories of HK angles followed by association with instabilities or compromised balance in this way, we can detect the affected joint and evaluate the severity of the issue. The study found stable mean and standard deviation values for HK joints in a young participant, both hips (107.14 ± 5, 96.42 ± 7) and both knees (36.76 ± 7, 44.30 ± 4), these values align with the expected norm (110–120° for the hip and 45–65° for the knee) which indicates stable results. while elderly participants showed high variability and low mean values for both hips (65.42 ± 77, 85 ± 76.67) and both knees (4.15 ± 10.8, 7 ± 18) indicating concerns about joint health and stability. The evaluation of HK joint angles through SLS activity offers new insights.

Remote Kinematic Analysis for Mobility Scooter Riders Leveraging Edge AI

Current kinematic analysis for patients with upper or lower extremity challenges is usually performed indoors at the clin- ics, which may not always be accessible for all patients. On the other hand, mobility scooter is a popular assistive tool used by people with mobility disabilities. In this study, we introduce a remote kinematic analysis system for mobility scooter riders to use in their local communities. In order to train the human pose estimation model for the kinematic anal- ysis application, we have collected our own mobility scooter riding video dataset which captures riders’ upper-body move- ments. The ground truth data is labeled by the collaborating clinicians. The evaluation results show high system accuracy both in the keypoints prediction and in the downstream kine- matic analysis, compared with the general-purpose pose mod- els. Our efficiency test results on NVIDIA Jetson Orin Nano also validate the feasibility of running the system in real-time on edge devices.

Attention-based hand pose estimation with voting and dual modalities

Hand pose estimation has recently emerged as a compelling topic in the robotic research community, because of its usefulness in learning from human demonstration or safe human–robot interaction. Although deep learning-based methods have been introduced for this task and have shown promise, it remains a challenging problem. To address this, we propose a novel end-to-end architecture for hand pose estimation using red-green-blue (RGB) and depth (D) data (RGB-D). Our approach processes the two data sources separately and utilizes a dense fusion network with an attention module to extract discriminative features. The features extracted include both spatial information and geometric constraints, which are fused to vote for the hand pose. We demonstrate that our voting mechanism in conjunction with the attention mechanism is particularly useful for solving the problem, especially when hands are heavily occluded by objects or are self-occluded. Our experimental results on benchmark datasets demonstrate that our approach outperforms state-of-the-art methods by a significant margin.

LightUrban: Similarity Based Fine‐grained Instancing for Lightweighting Complex Urban Point Clouds

AbstractLarge‐scale urban point clouds play a vital role in various applications, while rendering and transmitting such data remains challenging due to its large volume, complicated structures, and significant redundancy. In this paper, we present LightUrban, the first point cloud instancing framework for efficient rendering and transmission of fine‐grained complex urban scenes. We first introduce a segmentation method to organize the point clouds into individual buildings and vegetation instances from coarse to fine. Next, we propose an unsupervised similarity detection approach to accurately group instances with similar shapes. Furthermore, a fast pose and size estimation component is applied to calculate the transformations between the representative instance and the corresponding similar instances in each group. By replacing individual instances with their group's representative instances, the data volume and redundancy can be dramatically reduced. Experimental results on large‐scale urban scenes demonstrate the effectiveness of our algorithm. To sum up, our method not only structures the urban point clouds but also significantly reduces data volume and redundancy, filling the gap in lightweighting urban landscapes through instancing.

SMS-Net: Bridging the Gap Between High Accuracy and Low Computational Cost in Pose Estimation

Human pose estimation identifies and classifies key joints of the human body in images or videos. Existing pose estimation methods can precisely capture human movements in real time but require significant computational time and resources, which restricts their usage in specific conditions. Thus, we propose a lightweight pose estimation model—SMS-Net—based on the sequentially stacked structure of the hourglass network. The proposed model uses various lightweight techniques to enable high-speed pose estimation while requiring minimal storage space and computation. Specifically, a shuffle-gated block was introduced to reduce the computational load and number of parameters during the feature extraction process of the encoder composing each hourglass network. A multi-dilation block was used in the decoder to secure the receptive fields of various scales without increasing the computational load. The performance of the proposed model was assessed using the MPII and Common Objects in Context (COCO) datasets used for pose estimation and certain performance metrics and compared with state-of-the-art lightweight pose estimation models. Furthermore, an ablation study was performed to assess the impact of each module on network performance and efficiency. The results demonstrate that the proposed model achieved an improved balance between computational efficiency and performance compared to existing models in human pose estimation. Overall, the study findings can provide a basis for applications in computer vision technology.

Adaptive Granularity-Fused Keypoint Detection for 6D Pose Estimation of Space Targets

Estimating the 6D pose of a space target is an intricate task due to factors such as occlusions, changes in visual appearance, and background clutter. Accurate pose determination requires robust algorithms capable of handling these complexities while maintaining reliability under various environmental conditions. Conventional pose estimation for space targets unfolds in two stages: establishing 2D–3D correspondences using keypoint detection networks and 3D models, followed by pose estimation via the perspective-n-point algorithm. The accuracy of this process hinges critically on the initial keypoint detection, which is currently limited by predominantly singular-scale detection techniques and fails to exploit sufficient information. To tackle the aforementioned challenges, we propose an adaptive dual-stream aggregation network (ADSAN), which enables the learning of finer local representations and the acquisition of abundant spatial and semantic information by merging features from both inter-layer and intra-layer perspectives through a multi-grained approach, consolidating features within individual layers and amplifying the interaction of distinct resolution features between layers. Furthermore, our ADSAN implements the selective keypoint focus module (SKFM) algorithm to alleviate problems caused by partial occlusions and viewpoint alterations. This mechanism places greater emphasis on the most challenging keypoints, ensuring the network prioritizes and optimizes its learning around these critical points. Benefiting from the finer and more robust information of space objects extracted by the ADSAN and SKFM, our method surpasses the SOTA method PoET (5.8°, 8.1°/0.0351%, 0.0744%) by 0.5°, 0.9°, and 0.0084%, 0.0354%, achieving 5.3°, 7.2° in rotation angle errors and 0.0267%, 0.0390% in normalized translation errors on the Speed and SwissCube datasets, respectively.

High-fidelity pose estimation for real-time extended reality (XR) visualization for cardiac catheterization

Extended reality (XR) technologies are emerging as promising platforms for medical training and procedural guidance, particularly in complex cardiac interventions. This paper presents a high-fidelity methodology to perform real-time 3D catheter tracking and visualization during simulated cardiac interventions. A custom 3D-printed setup with mounted cameras enables biplane video capture of a catheter. A computer vision algorithm processes the biplane images in real-time to reconstruct the 3D catheter trajectory represented by any designated number of points along its length. This method accurately localizes the catheter tip within 1 mm and can reconstruct any arbitrary catheter configuration. The tracked catheter data is integrated into an interactive Unity-based scene rendered on the Meta Quest 3 headset. The visualization seamlessly combines a reconstructed 3D patient-specific heart model with the dynamically tracked catheter, creating an immersive extended reality training environment. Our experimental study, involving six participants, demonstrated that the 3D visualization provided by the proposed XR system significantly outperformed 2D visualization in terms of speed and user experience. This suggests that the XR system has the potential to enhance catheterization training by improving spatial comprehension and procedural skills. The proposed system demonstrates the potential of XR technologies to transform percutaneous cardiac interventions through improved visualization and interactivity.

Validation of a 3D Markerless Motion Capture Tool Using Multiple Pose and Depth Estimations for Quantitative Gait Analysis

Gait analysis is essential for evaluating walking patterns and identifying functional limitations. Traditional marker-based motion capture tools are costly, time-consuming, and require skilled operators. This study evaluated a 3D Marker-less Motion Capture (3D MMC) system using pose and depth estimations with the gold-standard Motion Capture (MOCAP) system for measuring hip and knee joint angles during gait at three speeds (0.7, 1.0, 1.3 m/s). Fifteen healthy participants performed gait tasks which were captured by both systems. The 3D MMC system demonstrated good accuracy (LCC > 0.96) and excellent inter-session reliability (RMSE < 3°). However, moderate-to-high accuracy with constant biases was observed during specific gait events, due to differences in sample rates and kinematic methods. Limitations include the use of only healthy participants and limited key points in the pose estimation model. The 3D MMC system shows potential as a reliable tool for gait analysis, offering enhanced usability for clinical and research applications.