3D Hand Posture Research Articles

Estimation of 3D anatomically précised hand poses using single shot corrective CNN

A crucial component of human-computer interaction is 3D hand posture assessment. The most recent advancements in computer vision have made estimating 3D hand positions simpler by using deep sensors. The main challenge still stems from unrealistic 3D hand poses because the existing models only use the training dataset to learn the kinematic rules, which is ambiguous, and it is a difficult task to estimate realistic 3D hand poses from datasets because they are not free from anatomical errors. The suggested model in this study is trained using a closed-form expression that encodes the biomechanical rules, thus it does not entirely reliant on the pictures from the annotated dataset. This work also used a Single Shot Detection and Correction convolutional neural network (SSDC-CNN) to handle the issues in imposing anatomically correctness from the architecture level. The ResNetPlus is implemented to improve representation capability with enhanced the efficiency of error back-propagation of the network. The datasets of the Yoga Mudras, like HANDS2017, and MSRA have been used to train and test the future model. As observed from the ground truth the previous hand models have many anatomical errors but, the proposed hand model is anatomically error free hand model compared to previous hand models. By considering the ground truth hand pose, the recommended hand model has shown good accuracy when compared to the state-of-art hand models.

Examining the Success of Information Gain, Pearson Correlation, and Symmetric Uncertainty Ranking Methods on 3D Hand Posture Data for Metaverse Systems

Metaverse is a hardware and software interface space that can connect people's social lives as in the real-natural world and provide the feeling of being there at the maximum level. In order for metaverse systems to be efficient, many independent accessories have to work holistically. One of these accessories is wearable gloves called meta gloves and equipped with sensors. Thanks to it, an important stage of metaverse systems is completed with the detection of 3-dimensional (3D) hand postures. In this study, the success of Information Gain, Pearson’s Correlation, and Symmetric Uncertainty ranking methods on 3D hand posture data for metaverse systems were investigated. For this purpose, various preprocessing was performed on the 3D data, and a dataset consisting of 15 features in total was created. The created dataset was ranked by 3 different methods mentioned and the features that the methods determined effectively were classified separately. Obtained results were interpreted with various statistical evaluation criteria. According to the experimental results obtained, it has been seen that the Symmetric Uncertainty ranking algorithm produces successful results for metaverse systems. As a result of the classification made with the active features determined using this method, there has been an increase in statistical performance criteria compared to other methods. In addition, it has been proven that time loss can be avoided in the classification of big data similar to the data used.

A 3D Hand Attitude Estimation Method for Fixed Hand Posture Based on Dual-View RGB Images

This work provides a 3D hand attitude estimation approach for fixed hand posture based on a CNN and LightGBM for dual-view RGB images to facilitate the application of hand posture teleoperation. First, using dual-view cameras and an IMU sensor, we provide a simple method for building 3D hand posture datasets. This method can quickly acquire dual-view 2D hand image sets and automatically append the appropriate three-axis attitude angle labels. Then, combining ensemble learning, which has strong regression fitting capabilities, with deep learning, which has excellent automatic feature extraction capabilities, we present an integrated hand attitude CNN regression model. This model uses a Bayesian optimization based LightGBM in the ensemble learning algorithm to produce 3D hand attitude regression and two CNNs to extract dual-view hand image features. Finally, a mapping from dual-view 2D images to 3D hand attitude angles is established using a training approach for feature integration, and a comparative experiment is run on the test set. The results of the experiments demonstrate that the suggested method may successfully solve the hand self-occlusion issue and accomplish 3D hand attitude estimation using only two normal RGB cameras.

3D Gesture Estimation from RGB Images Based on DB-InterNet

The booming of deep learning has made it possible to estimate 3D gestures from ordinary color images. However, the high accuracy of inferring 3D hand postures from RGB images is still not available due to the high flexibility of the gestures themselves. This paper aims to address the problem of low accuracy of keypoint coordinates position in InterNet gesture estimation network, by means of improving the confidence coordinate function, and selecting different suppression factor β according to different keypoints to fit t interacting hand estimation. By doing so, the maximum position coordinates are more precise and the keypoint prediction is more accurate. With respect to good measures, enhancing the representation capability of the model and employing dynamic activation function at different locations of the network to learn features in a dynamic way so as to boost the learning of hidden joints. In this way, different layers dynamically adjusted the segmental activation function according to the input to improve the performance of the model by dynamically learning features in a more flexible way. The experimental results indicate that compared with the baseline algorithm, the MPJPE and MRRPE of this algorithm are reduced by 0.73% and 2.04%, respectively, and the accuracy of single hand estimation is higher while the accuracy of interacting hand estimation is also effectively improved.

Three-Dimensional Input System Employing Pinching Gestures for Robot Design

Several studies of input interfaces capable of recognizing the gestures have been conducted but most of them use the user’s fingers to enter the position data. These finger-based input interfaces are difficult to provide a so-called click & drag function (as in a mouse) and some of them request for the user to take uncomfortable gestures. When people pinch any objects, however, basically their thumb and index finger come into contact with each other or separate them from each other. These pinching gestures provide superior benefits as the gestures, which may contribute to the input interfaces. This study proposes the method for detecting 3D finger positions and estimating 3D hand postures in pinching gestures based on information on depth images captured by a depth sensor, especially from the viewpoint of robot design. That produces benefits including button-clicking-like input operation by means of contact between the fingers; user’s comfortable gestures as in daily life; clicking action independent of input of positions and postures; and clear identification between ON and OFF. As the evaluation of the 3D input interface proposed here, the authors design real products with the system and a 3D printer, suggesting that the users can design precise and fine 3D objects with his/her comfortable daily gestures with highest usability.

Hand Posture Recognition Using a Three-Dimensional Light Field Camera

This paper used a light field camera to capture two types of 3D hand posture depth images, contour hand posture and solid hand posture, without a complicated setup and superfluous preprocess. The images were captured under two recognition conditions: in-plane and out-of-plane rotations. The posture recognition features two methods: the first method is the principal component analysis (PCA), which is used to obtain the required feature vectors, associated with the k-nearest neighbor (k-NN) algorithm as a classifier; the second method involves using 2D optimal-PCA (2DOPCA) combined with a genetic algorithm (GA) for feature selection, and the Mahalanobis distance is then used for classification. The variations in the test images include in-plane rotation, out-of-plane rotation, and Gaussian noise added to simulate the lighting interference in a real situation. The results showed that the PCA combined with the k-NN yields high recognition rates for grayscale contour images with in-plane and out-of-plane rotations and color solid posture images with in-plane rotation. For color solid posture images with out-of-plane rotation, the projection color space was combined with the PCA and k-NN methods to obtain high recognition rates. Moreover, the contour and color solid posture images with noise more than 5% require the 2DOPCA combined with the GA to obtain a satisfactory result and maintain recognition rate stability.

3D Finger CAPE: Clicking Action and Position Estimation under Self-Occlusions in Egocentric Viewpoint.

In this paper we present a novel framework for simultaneous detection of click action and estimation of occluded fingertip positions from egocentric viewed single-depth image sequences. For the detection and estimation, a novel probabilistic inference based on knowledge priors of clicking motion and clicked position is presented. Based on the detection and estimation results, we were able to achieve a fine resolution level of a bare hand-based interaction with virtual objects in egocentric viewpoint. Our contributions include: (i) a rotation and translation invariant finger clicking action and position estimation using the combination of 2D image-based fingertip detection with 3D hand posture estimation in egocentric viewpoint. (ii) a novel spatio-temporal random forest, which performs the detection and estimation efficiently in a single framework. We also present (iii) a selection process utilizing the proposed clicking action detection and position estimation in an arm reachable AR/VR space, which does not require any additional device. Experimental results show that the proposed method delivers promising performance under frequent self-occlusions in the process of selecting objects in AR/VR space whilst wearing an egocentric-depth camera-attached HMD.

Towards Real-Time and Rotation-Invariant American Sign Language Alphabet Recognition Using a Range Camera

The automatic interpretation of human gestures can be used for a natural interaction with computers while getting rid of mechanical devices such as keyboards and mice. In order to achieve this objective, the recognition of hand postures has been studied for many years. However, most of the literature in this area has considered 2D images which cannot provide a full description of the hand gestures. In addition, a rotation-invariant identification remains an unsolved problem, even with the use of 2D images. The objective of the current study was to design a rotation-invariant recognition process while using a 3D signature for classifying hand postures. A heuristic and voxel-based signature has been designed and implemented. The tracking of the hand motion is achieved with the Kalman filter. A unique training image per posture is used in the supervised classification. The designed recognition process, the tracking procedure and the segmentation algorithm have been successfully evaluated. This study has demonstrated the efficiency of the proposed rotation invariant 3D hand posture signature which leads to 93.88% recognition rate after testing 14,732 samples of 12 postures taken from the alphabet of the American Sign Language.

ROBUST REAL-TIME AND ROTATION-INVARIANT AMERICAN SIGN LANGUAGE ALPHABET RECOGNITION USING RANGE CAMERA

Abstract. The automatic interpretation of human gestures can be used for a natural interaction with computers without the use of mechanical devices such as keyboards and mice. The recognition of hand postures have been studied for many years. However, most of the literature in this area has considered 2D images which cannot provide a full description of the hand gestures. In addition, a rotation-invariant identification remains an unsolved problem even with the use of 2D images. The objective of the current study is to design a rotation-invariant recognition process while using a 3D signature for classifying hand postures. An heuristic and voxelbased signature has been designed and implemented. The tracking of the hand motion is achieved with the Kalman filter. A unique training image per posture is used in the supervised classification. The designed recognition process and the tracking procedure have been successfully evaluated. This study has demonstrated the efficiency of the proposed rotation invariant 3D hand posture signature which leads to 98.24% recognition rate after testing 12723 samples of 12 gestures taken from the alphabet of the American Sign Language.

Quick and accurate estimation of human 3D hand posture

We propose a three-dimensional hand posture estimation system that can retrieve a hand posture image most similar to the input data from a non-multilayer database. Our system uses, at the first stage, coarse screening by the proportional information on the hand images, which roughly correspond to forearm rotation or bending of the thumb or four fingers; then, at the second stage, performs a detailed search for similarity for selected candidates. To describe forearm rotation, and wrist’s internal and external rotations, bending and stretching, no separate processes were used for estimating the corresponding joint angles. By estimating the sequential images of the finger shape using this method, we successfully realized a process involving a joint angle estimation error within two or three degrees, a processing time of approximately 80 fps or more, using only one Note PC and high-speed camera, even when the wrist was freely rotating. Since the image information and the joint angle information are paired in the database, as well as the wrist joint, the system can generate the imitative motions as those of the fingers and wrist of a human being with no time delay by means of a robot, by outputting the estimation results to the robot hand.

Model-based 3D hand posture estimation from a single 2D image

Abstract Passive sensing of the 3D geometric posture of the human hand has been studied extensively over the past decade. However, these research efforts have been hampered by the computational complexity caused by inverse kinematics and 3D reconstruction. In this paper, our objective focuses on 3D hand posture estimation based on a single 2D image. We introduce the human hand model with 27 degrees of freedom (DOFs) and analyze some of its constraints to reduce the 27 to 12 DOFs without any significant degradation of performance. A novel algorithm to estimate the 3D hand posture from eight 2D projected feature points is proposed. Experimental results using real images confirm that our algorithm gives good estimates of the 3D hand pose.

Model-based analysis of hand posture

Researchers have successfully recognized specific fingers of the hand by silhouette images and distinguished a small set of hand signs by contour features of images. However, the silhouette or contour features recovered from the images do not provide sufficient information to generate a 3D hand posture with the fingers positioned properly. This failure led to our study, which developed a new method employing a hand model that can automatically analyze 3D hand postures using static stereo images. Guided by internal constraints and external forces, the model is automatically fitted to the hand image. Our method differs from previous methods in that it captures 27 interacting hand parameters, including finger joint angles, enabling suitable reconstruction of 3D hand posture. >