Hand Tracking Method Research Articles

Enhanced 2D Hand Pose Estimation for Gloved Medical Applications: A Preliminary Model.

(1) Background: As digital health technology evolves, the role of accurate medical-gloved hand tracking is becoming more important for the assessment and training of practitioners to reduce procedural errors in clinical settings. (2) Method: This study utilized computer vision for hand pose estimation to model skeletal hand movements during in situ aseptic drug compounding procedures. High-definition video cameras recorded hand movements while practitioners wore medical gloves of different colors. Hand poses were manually annotated, and machine learning models were developed and trained using the DeepLabCut interface via an 80/20 training/testing split. (3) Results: The developed model achieved an average root mean square error (RMSE) of 5.89 pixels across the training data set and 10.06 pixels across the test set. When excluding keypoints with a confidence value below 60%, the test set RMSE improved to 7.48 pixels, reflecting high accuracy in hand pose tracking. (4) Conclusions: The developed hand pose estimation model effectively tracks hand movements across both controlled and in situ drug compounding contexts, offering a first-of-its-kind medical glove hand tracking method. This model holds potential for enhancing clinical training and ensuring procedural safety, particularly in tasks requiring high precision such as drug compounding.

Temporally enhanced graph convolutional network for hand tracking from an egocentric camera

We propose a robust 3D hand tracking system in various hand action environments, including hand-object interaction, which utilizes a single color image and a previous pose prediction as input. We observe that existing methods deterministically exploit temporal information in motion space, failing to address realistic diverse hand motions. Also, prior methods paid less attention to efficiency as well as robust performance, i.e., the balance issues between time and accuracy. The Temporally Enhanced Graph Convolutional Network (TE-GCN) utilizes a 2-stage framework to encode temporal information adaptively. The system establishes balance by adopting an adaptive GCN, which effectively learns the spatial dependency between hand mesh vertices. Furthermore, the system leverages the previous prediction by estimating the relevance across image features through the attention mechanism. The proposed method achieves state-of-the-art balanced performance on challenging benchmarks and demonstrates robust results on various hand motions in real scenes. Moreover, the hand tracking system is integrated into a recent HMD with an off-loading framework, achieving a real-time framerate while maintaining high performance. Our study improves the usability of a high-performance hand-tracking method, which can be generalized to other algorithms and contributes to the usage of HMD in everyday life. Our code with the HMD project will be available at https://github.com/UVR-WJCHO/TEGCN_on_Hololens2.

Towards Smartphone-based 3D Hand Pose Reconstruction Using Acoustic Signals

Accurately reconstructing 3D hand poses is a pivotal element for numerous Human-Computer Interaction applications. In this work, we propose SonicHand, the first Smartphone-based 3D Hand Pose Reconstruction system using purely inaudible acoustic signals. SonicHand incorporates signal processing techniques and a deep learning framework to address a series of challenges. Firstly, it encodes the topological information of the hand skeleton as prior knowledge and utilizes a deep learning model to realistically and smoothly reconstruct the hand poses. Secondly, the system employs adversarial training to enhance the generalization ability of our system to be deployed in a new environment or for a new user. Thirdly, we adopt a hand tracking method based on channel impulse response (CIR) estimation. It enables our system to handle the scenario where the hand performs gestures while moving arbitrarily as a whole. We conduct extensive experiments on a smartphone testbed to demonstrate the effectiveness and robustness of our system from various dimensions. The experiments involve 10 subjects performing up to 12 different hand gestures in 3 distinctive environments. When the phone is held in one of the user’s hand, the proposed system can track joints with an average error of 18.64 mm.

Gesture Recognition and Hand Tracking for Anti-Counterfeit Palmvein Recognition

At present, COVID-19 is posing a serious threat to global human health. The features of hand veins in infrared environments have many advantages, including non-contact acquisition, security, privacy, etc., which can remarkably reduce the risks of COVID-19. Therefore, this paper builds an interactive system, which can recognize hand gestures and track hands for palmvein recognition in infrared environments. The gesture contours are extracted and input into an improved convolutional neural network for gesture recognition. The hand is tracked based on key point detection. Because the hand gesture commands are randomly generated and the hand vein features are extracted from the infrared environment, the anti-counterfeiting performance is obviously improved. In addition, hand tracking is conducted after gesture recognition, which prevents the escape of the hand from the camera view range, so it ensures that the hand used for palmvein recognition is identical to the hand used during gesture recognition. The experimental results show that the proposed gesture recognition method performs satisfactorily on our dataset, and the hand tracking method has good robustness.

Bimanual Intravenous Needle Insertion Simulation Using Nonhomogeneous Haptic Device Integrated into Mixed Reality.

In this study, we developed a new haptic-mixed reality intravenous (HMR-IV) needle insertion simulation system, providing a bimanual haptic interface integrated into a mixed reality system with programmable variabilities considering real clinical environments. The system was designed for nursing students or healthcare professionals to practice IV needle insertion into a virtual arm with unlimited attempts under various changing insertion conditions (e.g., skin: color, texture, stiffness, friction; vein: size, shape, location depth, stiffness, friction). To achieve accurate hand-eye coordination under dynamic mixed reality scenarios, two different haptic devices (Dexmo and Geomagic Touch) and a standalone mixed reality system (HoloLens 2) were integrated and synchronized through multistep calibration for different coordinate systems (real world, virtual world, mixed reality world, haptic interface world, HoloLens camera). In addition, force-profile-based haptic rendering proposed in this study was able to successfully mimic the real tactile feeling of IV needle insertion. Further, a global hand-tracking method, combining two depth sensors (HoloLens and Leap Motion), was developed to accurately track a haptic glove and simulate grasping a virtual hand with force feedback. We conducted an evaluation study with 20 participants (9 experts and 11 novices) to measure the usability of the HMR-IV simulation system with user performance under various insertion conditions. The quantitative results from our own metric and qualitative results from the NASA Task Load Index demonstrate the usability of our system.

Evaluation and improvement of HMD-based and RGB-based hand tracking solutions in VR

Hand tracking has become a state-of-the-art technology in the modern generation of consumer VR devices. However, off-the-shelf solutions do not support hand detection for more than two hands at the same time at distances beyond arm’s length. The possibility to track multiple hands at larger distances would be beneficial for colocated multi-user VR scenarios, allowing user-worn devices to track the hands of other users and therefore reducing motion artifacts caused by hand tracking loss. With the global focus of enabling natural hand interactions in colocated multi-user VR, we propose an RGB image input-based hand tracking method, built upon the MediaPipe framework, that can track multiple hands at once at distances of up to 3 m. We compared our method’s accuracy to that of Oculus Quest and Leap Motion, at different distances from the tracking device and in static and dynamic settings. The results of our evaluation show that our method provides only slightly less accurate results than Oculus Quest or Leap motion in the near range (with median errors below 1.75 cm at distances below 75 cm); at larger distances, its accuracy remains stable (with a median error of 4.7 cm at the distance of 2.75 m) while Leap Motion and Oculus Quest either loose tracking or produce very inaccurate results. Taking into account the broad choice of suitable hardware (any RGB camera) and the ease of setup, our method can be directly applied to colocated multi-user VR scenarios.

Comparing Different Grasping Visualizations for Object Manipulation in VR using Controllers.

Virtual grasping is one of the most common and important interactions performed in a Virtual Environment (VE). Even though there has been substantial research using hand tracking methods exploring different ways of visualizing grasping, there are only a few studies that focus on handheld controllers. This gap in research is particularly crucial, since controllers remain the most used input modality in commercial Virtual Reality (VR). Extending existing research, we designed an experiment comparing three different grasping visualizations when users are interacting with virtual objects in immersive VR using controllers. We examine the following visualizations: the Auto-Pose (AP), where the hand is automatically adjusted to the object upon grasping; the Simple-Pose (SP), where the hand closes fully when selecting the object; and the Disappearing-Hand (DH), where the hand becomes invisible after selecting an object, and turns visible again after positioning it on the target. We recruited 38 participants in order to measure if and how their performance, sense of embodiment, and preference are affected. Our results show that while in terms of performance there is almost no significant difference in any of the visualizations, the perceived sense of embodiment is stronger with the AP, and is generally preferred by the users. Thus, this study incentivizes the inclusion of similar visualizations in relevant future research and VR experiences.

A Mobile Natural Human–Robot Interaction Method for Virtual Chinese Acupuncture

Most of the existing virtual acupuncture methods cannot allow operators to interact naturally in a large area. To address this problem, this article proposes a mobile natural human–robot interaction interface for virtual Chinese acupuncture teaching, including an automatic hand tracking method and an acupuncture interaction method combining vision and force feedback. The automatic hand tracking method can automatically track and capture the operator’s hand movements, allowing the operator to perform acupuncture in a large area. The acupuncture interaction method combining vision and force reproduces the real acupuncture operation mode. The proposed interaction method provides the operator with visual feedback and force feedback without limitation, which improves the naturalness and authenticity of the operation. Through a series of experiments, the proposed method was proven to be more natural and efficient in the virtual acupuncture teaching scene from the point view of interaction. The operators achieve a stronger sense of immersion using this method.

Assembly Line Quality Assurance Through Hand Tracking and Object Detection

This work presents a vision-based quality assurance system that does assembly line monitoring. The system is developed using machine learning hand tracking and object detection methods to monitor the worker’s hand movement while evaluating the correctness of the assembly. Feedback about the order of the steps the worker has taken is continuously shown to the user. This work has the potential to reduce the amount of manual work required for quality assurance in assembly line.

Vision-Based Intelligent Perceiving and Planning System of a 7-DoF Collaborative Robot.

In this paper, an intelligent perceiving and planning system based on deep learning is proposed for a collaborative robot consisting of a 7-DoF (7-degree-of-freedom) manipulator, a three-finger robot hand, and a vision system, known as IPPS (intelligent perceiving and planning system). The lack of intelligence has been limiting the application of collaborative robots for a long time. A system to realize “eye-brain-hand” process is crucial for the true intelligence of robots. In this research, a more stable and accurate perceiving process was proposed. A well-designed camera system as the vision system and a new hand tracking method were proposed for operation perceiving and recording set establishment to improve the applicability. A visual process was designed to improve the accuracy of environment perceiving. Besides, a faster and more precise planning process was proposed. Deep learning based on a new CNN (convolution neural network) was designed to realize intelligent grasping planning for robot hand. A new trajectory planning method of the manipulator was proposed to improve efficiency. The performance of the IPPS was tested with simulations and experiments in a real environment. The results show that IPPS could effectively realize intelligent perceiving and planning for the robot, which could realize higher intelligence and great applicability for collaborative robots.

A naked eye 3D display and interaction system for medical education and training.

To provide natural simulated objects and intuitive user interaction in medical education and training, we propose a naked eye 3D display and interaction system. The current 3D rendering algorithms for naked eye 3D displays are not suitable for medical use, due to the requirements of displaying and interacting with high quality medical images and simulating soft tissues. Because the traditional 3D rendering procedure and vertex indexing in collision detection require substantial computing power when using a naked eye 3D display, the current method cannot achieve fluent displays and interactions. Thus, we develop a novel octree-based 3D rendering and interaction algorithm for high quality medical models to improve the rendering rate and obtain smooth human machine interactions when using the naked eye 3D display device. We also valuate the soft-body phantom simulation of the naked eye 3D display device by combining the traditional 3D rendering algorithm with the elastic 3D simulation to simulate deformable tissues. We integrate an incremental interaction method and a Kalman filter-based hand tracking method to achieve a larger user interaction range and robust hand tracking. We used the proposed system to perform human-computer interactions with rigid phantoms and soft-body phantoms. The experimental results showed that the proposed rendering algorithm for rigid phantoms could achieve higher rendering performance (50 FPS) than the traditional rendering algorithm (9.8 FPS). The user experiments showed that the 3D simulation system equipped with the enhanced rendering algorithm could achieve fluent interactions when using the naked eye 3D display, thus promoting education experiences and reducing task completion times.

Hand pose estimation and tracking in real and virtual interaction:A review

Vision-based 3D hand tracking is a key and popular component for interaction studies in a broad range of domains such as virtual reality (VR), augmented reality (AR) and natural human-computer interaction (HCI). While this research field has been well studied in the last decades, most approaches have considered the human hand in isolation and not in action or in interaction with the surrounding environment. Even the common collaborative and strong interactions with the other hand have been ignored. However, many of today's computer applications require more and more hand-object interactions. Furthermore, employing contextual information about the object in the hand (e.g. the shape, the texture, and the pose) can remarkably constrain the tracking problem. The most studied contextual constraints involve interaction with real objects and not with virtual objects which is still a very big challenge. The goal of this survey is to develop an up-to-date taxonomy of the state-of-the-art vision-based hand pose estimation and tracking methods with a new classification scheme: hand-object interaction constraints. This taxonomy allows us to examine the strengths and weaknesses of the current state of the art and to highlight future trends in the domain.

Single-shot 3D hand pose estimation using radial basis function networks trained on synthetic data

In this work, we present a novel framework to perform single-shot hand pose estimation using depth data as input. The method follows a coarse to fine strategy and employs several radial basis function networks (RBFNs) that are trained on a dataset containing only synthetically generated depth maps. Thus, compared to most contemporary deep learning approaches, it does not require the laborious annotation of large, real-world datasets. At run time, an initialization RBFN is used to provide a rough estimation of the hand’s 3D pose. Subsequently, several specialized RBFNs are employed to improve that initial estimation in an iterative refinement scheme. To train the RBFNs, we select a set of hand poses from a real-world sequence that are as diverse as possible. We use this representative set, along with a dense sampling of all possible rotations, as a seed to generate a large synthetic training set. The method is parallelizable, taking advantage of the inherent data parallelism of RBFNs. Furthermore, the method requires few real-world data and virtually no manual annotation. We perform a quantitative evaluation of our method on a testing sequence of our own. We also present quantitative and qualitative results on a public dataset that is commonly used to evaluate hand pose estimation and tracking methods. We show that in all cases, our approach achieves promising results. Moreover, it can achieve comparable or even faster computational performance than current deep learning approaches but on a single CPU core, i.e., without requiring GPU processing.

Isolated sign language recognition using Convolutional Neural Network hand modelling and Hand Energy Image

This paper presents an isolated sign language recognition system that comprises of two main phases: hand tracking and hand representation. In the hand tracking phase, an annotated hand dataset is used to extract the hand patches to pre-train Convolutional Neural Network (CNN) hand models. The hand tracking is performed by the particle filter that combines hand motion and CNN pre-trained hand models into a joint likelihood observation model. The predicted hand position corresponds to the location of the particle with the highest joint likelihood. Based on the predicted hand position, a square hand region centered around the predicted position is segmented and serves as the input to the hand representation phase. In the hand representation phase, a compact hand representation is computed by averaging the segmented hand regions. The obtained hand representation is referred to as “Hand Energy Image (HEI)”. Quantitative and qualitative analysis show that the proposed hand tracking method is able to predict the hand positions that are closer to the ground truth. Similarly, the proposed HEI hand representation outperforms other methods in the isolated sign language recognition.

Fast hand and finger detection algorithm for interaction on smart display

This paper presents a fast and accurate hand detection and tracking algorithm for depth images using an object contour analysis for which pairs of horizontal edges of an object are used. The computational complexity for generating the pairs of edges and estimating the object shape is low. A shape analysis is performed in the raster scan order and an object which can be a finger is selected as the finger candidate during the line scan. The initial hand is detected by searching for an object with a shape that corresponds to the predefined shape of the initial hand in the entire depth range. Thus, there is no assumption of the hand location throughout the depth, and the cluttered background has little effect on the detection. This paper also proposes a low-complexity hand tracking method which achieves 97.11% in the detection rate and 10.39 ms in the average processing time for a frame, respectively, when a single hand is detected. For five hands, the average processing time is 21.15 ms. These results show that the proposed method can be applied to the real-time application of vision-based HCI especially for smart display.

A four dukkha state-space model for hand tracking

In this paper, we propose a hand tracking method which was inspired by the notion of the four dukkha: birth, aging, sickness and death (BASD) in Buddhism. Based on this philosophy, we formalize the hand tracking problem in the BASD framework, and apply it to hand track hand gestures in isolated sign language videos. The proposed BASD method is a novel nature-inspired computational intelligence method which is able to handle complex real-world tracking problem. The proposed BASD framework operates in a manner similar to a standard state-space model, but maintains multiple hypotheses and integrates hypothesis update and propagation mechanisms that resemble the effect of BASD. The survival of the hypothesis relies upon the strength, aging and sickness of existing hypotheses, and new hypotheses are birthed by the fittest pairs of parent hypotheses. These properties resolve the sample impoverishment problem of the particle filter. The estimated hand trajectories show promising results for the American sign language.

ウェアラブルビジュアルインターフェースのための機能分散型ハンドトラッキング手法とその応用(「ウェアラブル・ユビキタス技術」特集)

ウェアラブルビジュアルインターフェースのための機能分散型ハンドトラッキング手法とその応用(「ウェアラブル・ユビキタス技術」特集)

REAL-TIME HAND DETECTION BY DEPTH IMAGES: A SURVEY

Human hand detection can enable human to communicate with a machine and interact without any external device. Human hands play an important role in different applications such as medical image processing, sign language translator, gesture recognition and augmented reality. A human hand has different length and breadth for male and female. So, it is a complex articulated object consisting many connected parts and joints. Traditional methods for hand detection and tracking used color and shape information from RGB camera. Using a depth camera for hand detection and tracking is a challenging and interesting domain in computer vision. Some research has shown that using depth data for hand detection can improve human computer interaction. Recently, researchers used depth data in different hand detection and tracking methods in real time application. This paper explains different types of methods which are used for human hand detection. Various techniques and methods are explored and analyzed in this survey to determine the shortfalls and future directions in the field of hand detection from depth data.

Online hand gesture recognition using enhanced $N recogniser based on a depth camera

In this paper, we propose a hand gesture recognition system using a depth camera for user notes correction. For this system, we developed a gesture recognition and hand tracking method. In tracking, we focus on the index finger tip point. To extract the point, we detect the hand region using depth information and determine the top point of the region. For recognition, we use the $N recogniser. However, the recogniser has a problem that it is too insensitive in rotation. Therefore, we propose an enhanced $N recogniser. We include the process of matching the angle between the starting gesture pose and the ending gesture pose. Through experimental results, we show that the performance improves with our methods.

RGBD Video Based Human Hand Trajectory Tracking and Gesture Recognition System

The task of human hand trajectory tracking and gesture trajectory recognition based on synchronized color and depth video is considered. Toward this end, in the facet of hand tracking, a joint observation model with the hand cues of skin saliency, motion and depth is integrated into particle filter in order to move particles to local peak in the likelihood. The proposed hand tracking method, namely, salient skin, motion, and depth based particle filter (SSMD-PF), is capable of improving the tracking accuracy considerably, in the context of the signer performing the gesture toward the camera device and in front of moving, cluttered backgrounds. In the facet of gesture recognition, a shape-order context descriptor on the basis of shape context is introduced, which can describe the gesture in spatiotemporal domain. The efficient shape-order context descriptor can reveal the shape relationship and embed gesture sequence order information into descriptor. Moreover, the shape-order context leads to a robust score for gesture invariant. Our approach is complemented with experimental results on the settings of the challenging hand-signed digits datasets and American sign language dataset, which corroborate the performance of the novel techniques.