3D Hand Pose Research Articles

Improve Regression Network on Depth Hand Pose Estimation With Auxiliary Variable

The regression based deep neural networks have achieved state-of-the-arts performance on depth 3D hand pose estimation task. This paper focuses on improving the regression mapping between features and pose joints. Inspired by the distribution modeling ability of Variational Autoencoders, we introduce an auxiliary variable into the regression network. During training, the auxiliary variable is modeled by an inference distribution that learns the underlying structural kinematics of human hand. Different with other regression methods on hand poses, our network estimates the pose joints from input depth features and the learned auxiliary variable as well. We show that by introducing the auxiliary variable, the regression is benefited from 1) regularization modeled by inference distribution; and 2) prior information carried by the auxiliary model. The effectiveness of the proposed regression method is evaluated with extensively self-comparative experiments and in comparison with other regression methods on hand pose datasets. The proposed network is easy to train in an end-to-end manner and can work with various feature extraction methods. We apply the proposed regression method to an existing hand pose estimation system, and improves the estimation accuracy by 18.35% and 16.65% on public hand pose datasets.

3D Hand Pose Estimation Using Synthetic Data and Weakly Labeled RGB Images.

Compared with depth-based 3D hand pose estimation, it is more challenging to infer 3D hand pose from monocular RGB images, due to the substantial depth ambiguity and the difficulty of obtaining fully-annotated training data. Different from the existing learning-based monocular RGB-input approaches that require accurate 3D annotations for training, we propose to leverage the depth images that can be easily obtained from commodity RGB-D cameras during training, while during testing we take only RGB inputs for 3D joint predictions. In this way, we alleviate the burden of the costly 3D annotations in real-world dataset. Particularly, we propose a weakly-supervised method, adaptating from fully-annotated synthetic dataset to weakly-labeled real-world single RGB dataset with the aid of a depth regularizer, which serves as weak supervision for 3D pose prediction. To further exploit the physical structure of 3D hand pose, we present a novel CVAE-based statistical framework to embed the pose-specific subspace from RGB images, which can then be used to infer the 3D hand joint locations. Extensive experiments on benchmark datasets validate that our proposed approach outperforms baselines and state-of-the-art methods, which proves the effectiveness of the proposed depth regularizer and the CVAE-based framework.

CFAM: Estimating 3D Hand Poses from a Single RGB Image with Attention

Precise 3D hand pose estimation can be used to improve the performance of human–computer interaction (HCI). Specifically, computer-vision-based hand pose estimation can make this process more natural. Most traditional computer-vision-based hand pose estimation methods use depth images as the input, which requires complicated and expensive acquisition equipment. Estimation through a single RGB image is more convenient and less expensive. Previous methods based on RGB images utilize only 2D keypoint score maps to recover 3D hand poses but ignore the hand texture features and the underlying spatial information in the RGB image, which leads to a relatively low accuracy. To address this issue, we propose a channel fusion attention mechanism that combines 2D keypoint features and RGB image features at the channel level. In particular, the proposed method replans weights by using cascading RGB images and 2D keypoint features, which enables rational planning and the utilization of various features. Moreover, our method improves the fusion performance of different types of feature maps. Multiple contrast experiments on public datasets demonstrate that the accuracy of our proposed method is comparable to the state-of-the-art accuracy.

DeepAirSig: End-to-End Deep Learning Based in-Air Signature Verification

In-air signature verification is vital for biometric user identification in contact-less mode. The state-of-the-art methods use heuristics for signature acquisition, and provide insufficient data to train neural networks for the verification. In this article, we present a novel method for end-to-end deep learning based in-air signature verification using a depth sensor. In this regard, we propose a new medium-scale in-air signature dataset which is created using an accurate convolutional neural network (CNN) based 3D hand pose estimation algorithm. The proposed dataset offers a total of 1800 signatures collected from 40 subjects. So far, dynamic time warping (DTW) has been the most effective and commonly used method for verification. Keeping in view the significant advancement in deep learning, we present a more accurate deep learning based alternative which outperforms DTW by 67.6%. To this end, we train a personalized autoencoder to reconstruct the signature of each subject. Thereafter, the signature is verified by thresholding the reconstruction loss. We perform extensive experiments by formulating spatial and depth features of the signature in images and point clouds based representations. Moreover, for comparisons, we implement several deep learning algorithms (i.e. linear autoencoder, convolutional autoencoder, and Deep One-Class classifier). Our verification approach achieves an EER of 0.055%. The dataset is available at https://bit.ly/2mEJzOw.

Fast and Accurate 3D Hand Pose Estimation via Recurrent Neural Network for Capturing Hand Articulations

3D hand pose estimation from a single depth image plays an important role in computer vision and human-computer interaction. Although recent hand pose estimation methods using convolution neural network (CNN) have shown notable improvements in accuracy, most of them have a limitation that they rely on a complex network structure without fully exploiting the articulated structure of the hand. A hand, which is an articulated object, is composed of six local parts: the palm and five independent fingers. Each finger consists of sequential-joints that provide constrained motion, referred to as a kinematic chain. In this paper, we propose a hierarchically-structured convolutional recurrent neural network (HCRNN) with six branches that estimate the 3D position of the palm and five fingers independently. The palm position is predicted via fully-connected layers. Each sequential-joint, i.e. finger position, is obtained using a recurrent neural network (RNN) to capture the spatial dependencies between adjacent joints. Then the output features of the palm and finger branches are concatenated to estimate the global hand position. HCRNN directly takes the depth map as an input without a time-consuming data conversion, such as 3D voxels and point clouds. Experimental results on public datasets demonstrate that the proposed HCRNN not only outperforms most 2D CNN-based methods using the depth image as their inputs but also achieves competitive results with state-of-the-art 3D CNN-based methods with a highly efficient running speed of 285 fps on a single GPU.

Attention-Based Pose Sequence Machine for 3D Hand Pose Estimation

Most of the existing methods for 3D hand pose estimation are performed from a single depth map. In that case, the depth missing challenges from input frames caused by hand self-occlusions and imaging quality lead to multi-valued mapping phenomenon and sub-optimal model. In this paper, we proposed a novel recurrent architecture named Attention-based Pose Sequence Machine (APSM) to alleviate challenges by introducing temporal consistency. As for recurrent unit (RU), we extend traditional Gated Recurrent Unit (GRU) with 3D convolutional neural networks (CNNs) to handle voxelized inputs and features, and a novel RU named Deep Gated Recurrent Unit (DGRU) was proposed by rebuilding deeper gates based on GRU. To improve the model performance, a novel spatial attention mechanism denoted as Attention Model (AM) was proposed. Ablation experiments are designed to validate each contribution of our work, and experiments on two publicly available dataset show that our work outperforms state-of-the-art on hand pose estimation.

3D hand pose estimation method using CNN and high-speed vision system

3D hand pose estimation method using CNN and high-speed vision system

MVPointNet: Multi-View Network for 3D Object Based on Point Cloud

The research of hand pose estimation is a hot topic in computer vision, robotics and virtual reality. Compared with using data glove, vision based methods show great advantage for its contactless property, low-cost and convenience. With the commercial depth cameras became widely available and the great success of Convolution Neural Network (CNN) on images, various works focused on hand pose estimation have achieved promising performance. This research is inspired by the recent work that directly perform 3D classification and segmentation tasks on point cloud. In this paper, Multi-View PointNet (MVPointNet) is proposed which takes several views of point cloud as input source. Then, they fed into the well-performing point cloud-based architecture. In addition, to better capture the hand context structure and improve the performance, more features between centroid and local neighborhood points (norm, edge, angle) are extracted and fed into a deep CNN architecture. To our knowledge, our proposed method achieved good performance on the ModelNet40 dataset for 3D shape classification. Besides, it achieved superior performance over other deep learning methods for 3D hand pose estimation based on point cloud, which is evaluated on MSRA dataset.

A multi-branch hand pose estimation network with joint-wise feature extraction and fusion

The study of 3D hand pose estimation from a single depth image is regarded as a detection-based or regression-based problem among most of the existing deep learning-based methods, and this approach does not fully exploit the geometry of the hand, such as its structural and physical constraints. To overcome these weaknesses, we design a network with three simple parallel branches that correspond to the three functional parts of the hand. This observation is motivated by the biological viewpoint that each finger plays a different role in performing grasping and manipulation. In each branch, we perform a more detailed regression in two stages – top-down joint location regression followed by bottom-up hand pose regression – which fully exploits both the local and global structure of a hand. Finally, we further make use of the hand structure and physical constraints to refine each joint by its auxiliary points. The proposed network is a unified structure and function model that is more appropriate for hand pose estimation. Our system does not require pose pre-processing or feedback since it can directly perform training and predicting from end-to-end. The experimental results on three public datasets demonstrate that the proposed system achieves performance comparable to state-of-the-art methods.

Simple and effective deep hand shape and pose regression from a single depth image

Simultaneously estimating the 3D shape and pose of a hand in real time is a new and challenging computer graphics problem, which is important for animation and interactions with 3D objects in virtual environments with personalized hand shapes. CNN-based direct hand pose estimation methods are the state-of-the-art approaches, but they can only regress a 3D hand pose from a single depth image. In this study, we developed a simple and effective real-time CNN-based direct regression approach for simultaneously estimating the 3D hand shape and pose, as well as structure constraints for both egocentric and third person viewpoints by learning from the synthetic depth. In addition, we produced the first million-scale egocentric synthetic dataset called SynHandEgo, which contains egocentric depth images with accurate shape and pose annotations, as well as color segmentation of the hand parts. Our network is trained based on combined real and synthetic datasets with full supervision of the hand pose and structure constraints, and semi-supervision of the hand mesh. Our approach performed better than the state-of-the-art methods based on the SynHand5M synthetic dataset in terms of both the 3D shape and pose recovery. By learning simultaneously using real and synthetic data, we demonstrated the feasibility of hand mesh recovery from two real hand pose datasets, i.e., BigHand2.2M and NYU. Moreover, our method obtained more accurate estimates of the 3D hand poses based on the NYU dataset compared with the existing methods that output more than joint positions. The SynHandEgo dataset has been made publicly available to promote further research in the emerging domain of hand shape and pose recovery from egocentric viewpoints (https://bit.ly/2WMWM5u).

Synthesizing Depth Hand Images with GANs and Style Transfer for Hand Pose Estimation.

Hand pose estimation is a critical technology of computer vision and human-computer interaction. Deep-learning methods require a considerable amount of tagged data. Accordingly, numerous labeled training data are required. This paper aims to generate depth hand images. Given a ground-truth 3D hand pose, the developed method can generate depth hand images. To be specific, a ground truth can be 3D hand poses with the hand structure contained, while the synthesized image has an identical size to that of the training image and a similar visual appearance to the training set. The developed method, inspired by the progress in the generative adversarial network (GAN) and image-style transfer, helps model the latent statistical relationship between the ground-truth hand pose and the corresponding depth hand image. The images synthesized using the developed method are demonstrated to be feasible for enhancing performance. On public hand pose datasets (NYU, MSRA, ICVL), comprehensive experiments prove that the developed method outperforms the existing works.

Accurate and efficient 3D hand pose regression for robot hand teleoperation using a monocular RGB camera

In this paper, we present a novel deep learning-based architecture, which is under the scope of expert and intelligent systems, to perform accurate real-time tridimensional hand pose estimation using a single RGB frame as an input, so there is no need to use multiple cameras or points of view, or RGB-D devices. The proposed pipeline is composed of two convolutional neural network architectures. The first one is in charge of detecting the hand in the image. The second one is able to accurately infer the tridimensional position of the joints retrieving, thus, the full hand pose. To do this, we captured our own large-scale dataset composed of images of hands and the corresponding 3D joints annotations.The proposal achieved a 3D hand pose mean error of below 5 mm on both the proposed dataset and Stereo Hand Pose Tracking Benchmark, which is a public dataset. Our method also outperforms the state-of-the-art methods.We also demonstrate in this paper the application of the proposal to perform a robotic hand teleoperation with high success.

Pose Anchor: A Single-Stage Hand Keypoint Detection Network

This paper presents an effective single network for hand keypoint detection, instead of relying on the frequently-used two-stage pipeline consisting of localizing the hand and detecting the key points. Our method trains a fully convolutional neural network in an end-to-end manner, based on a novelly proposed pose anchor network, which can be deemed as an extension of the region proposal network (RPN) in Faster Region-based convolutional network. Moreover, we generate our pose anchor in a data-driven way, i.e., a K-means cluster algorithm based on object keypoint similarity (OKS), instead of manually design. In this way, we can obtain multiple representative pose anchors with various gestures, angles, and scales. By introducing the pose anchor, we are capable of utilizing the prior knowledge of the hand structure, mitigating the problem of occlusion to some extent. We demonstrate the feasibility and effectiveness of our method with extensive experiments on the challenging large-scale multiview 3D hand pose dataset (LSM-HPD) and New Zealand Sign Language Dataset (NZSL).

Learning hand latent features for unsupervised 3D hand pose estimation

<p align="justify">Recent hand pose estimation methods require large numbers of annotated training data to extract the dynamic information from a hand representation. Nevertheless, precise and dense annotation on the real data is difficult to come by and the amount of information passed to the training algorithm is significantly higher. This paper presents an approach to developing a hand pose estimation system which can accurately regress a 3D pose in an unsupervised manner. The whole process is performed in three stages. Firstly, the hand is modelled by a novel latent tree dependency model (LTDM) which transforms internal joints location to an explicit representation. Secondly, we perform predictive coding of image sequences of hand poses in order to capture latent features underlying a given image without supervision. A mapping is then performed between an image depth and a generated representation. Thirdly, the hand joints are regressed using convolutional neural networks to finally estimate the latent pose given some depth map. Finally, an unsupervised error term which is a part of the recurrent architecture ensures smooth estimations of the final pose. To demonstrate the performance of the proposed system, a complete experiment is conducted on three challenging public datasets, ICVL, MSRA, and NYU. The empirical results show the significant performance of our method which is comparable or better than state-of-the-art approaches.</p>

A survey on 3D hand pose estimation: Cameras, methods, and datasets

3D Hand pose estimation has received an increasing amount of attention, especially since consumer depth cameras came onto the market in 2010. Although substantial progress has occurred recently, no overview has kept up with the latest developments. To bridge the gap, we provide a comprehensive survey, including depth cameras, hand pose estimation methods, and public benchmark datasets. First, a markerless approach is proposed to evaluate the tracking accuracy of depth cameras with the aid of a numerical control linear motion guide. Traditional approaches focus only on static characteristics. The evaluation of dynamic tracking capability has been long neglected. Second, we summarize the state-of-the-art methods and analyze the lines of research. Third, existing benchmark datasets and evaluation criteria are identified to provide further insight into the field of hand pose estimation. In addition, realistic challenges, recent trends, dataset creation and annotation, and open problems for future research directions are also discussed.

Generalized Feedback Loop for Joint Hand-Object Pose Estimation.

We propose an approach to estimating the 3D pose of a hand, possibly handling an object, given a depth image. We show that we can correct the mistakes made by a Convolutional Neural Network trained to predict an estimate of the 3D pose by using a feedback loop. The components of this feedback loop are also Deep Networks, optimized using training data. This approach can be generalized to a hand interacting with an object. Therefore, we jointly estimate the 3D pose of the hand and the 3D pose of the object. Our approach performs en-par with state-of-the-art methods for 3D hand pose estimation, and outperforms state-of-the-art methods for joint hand-object pose estimation when using depth images only. Also, our approach is efficient as our implementation runs in real-time on a single GPU.

Feature Boosting Network For 3D Pose Estimation.

In this paper, a feature boosting network is proposed for estimating 3D hand pose and 3D body pose from a single RGB image. In this method, the features learned by the convolutional layers are boosted with a new long short-term dependence-aware (LSTD) module, which enables the intermediate convolutional feature maps to perceive the graphical long short-term dependency among different hand (or body) parts using the designed Graphical ConvLSTM. Learning a set of features that are reliable and discriminatively representative of the pose of a hand (or body) part is difficult due to the ambiguities, texture and illumination variation, and self-occlusion in the real application of 3D pose estimation. To improve the reliability of the features for representing each body part and enhance the LSTD module, we further introduce a context consistency gate (CCG) in this paper, with which the convolutional feature maps are modulated according to their consistency with the context representations. We evaluate the proposed method on challenging benchmark datasets for 3D hand pose estimation and 3D full body pose estimation. Experimental results show the effectiveness of our method that achieves state-of-the-art performance on both of the tasks.

A Soft Proposal Segmentation Network (SPS-Net) for Hand Segmentation on Depth Videos

Hand segmentation is an important prerequisite for acquiring accurate 3D hand poses on depth images, as it can significantly reduce the complicity of hand pose estimation. However, it has been ignored or treated as a trivial problem for a long time, since most of the hand pose estimation works suppose the hand part is given or can be easily segmented by a depth threshold, which is not practical in many realistic scenarios (e.g., side/egocentric view). In order to perform the robust hand segmentation in various scenarios, we propose a Soft Proposal Segmentation Network (SPS-Net). A key difference from existing hand segmentation methods is that the SPS-Net can utilize the temporal information on depth videos. As a benefit, significant performance gains over the existing methods can be obtained, as demonstrated on two popular public datasets with diverse camera viewpoints and background. Moreover, integrating SPS-Net with a simple 3D hand pose estimator, we achieved the leading result on the 3D Hand Pose Tracking Task of the Hand2017 Challenge - the only world-wide open challenge of its kind.

Robust 3D Hand Pose Estimation From Single Depth Images Using Multi-View CNNs.

Articulated hand pose estimation is one of core technologies in human-computer interaction. Despite the recent progress, most existing methods still cannot achieve satisfactory performance, partly due to the difficulty of the embedded high-dimensional nonlinear regression problem. Most existing data-driven methods directly regress 3D hand pose from 2D depth image, which cannot fully utilize the depth information. In this paper, we propose a novel multi-view convolutional neural network (CNN)-based approach for 3D hand pose estimation. To better exploit 3D information in the depth image, we project the point cloud generated from the query depth image onto multiple views of two projection settings and integrate them for more robust estimation. Multi-view CNNs are trained to learn the mapping from projected images to heat-maps, which reflect probability distributions of joints on each view. These multi-view heat-maps are then fused to estimate the optimal 3D hand pose with learned pose priors, and the unreliable information in multi-view heat-maps is suppressed using a view selection method. Experimental results show that the proposed method is superior to the state-of-the-art methods on two challenging data sets. Furthermore, a cross-data set experiment also validates that our proposed approach has good generalization ability.

Real-Time 3D Hand Pose Estimation with 3D Convolutional Neural Networks.

In this paper, we present a novel method for real-time 3D hand pose estimation from single depth images using 3D Convolutional Neural Networks (CNNs). Image-based features extracted by 2D CNNs are not directly suitable for 3D hand pose estimation due to the lack of 3D spatial information. Our proposed 3D CNN-based method, taking a 3D volumetric representation of the hand depth image as input and extracting 3D features from the volumetric input, can capture the 3D spatial structure of the hand and accurately regress full 3D hand pose in a single pass. In order to make the 3D CNN robust to variations in hand sizes and global orientations, we perform 3D data augmentation on the training data. To further improve the estimation accuracy, we propose applying the 3D deep network architectures and leveraging the complete hand surface as intermediate supervision for learning 3D hand pose from depth images. Extensive experiments on three challenging datasets demonstrate that our proposed approach outperforms baselines and state-of-the-art methods. A cross-dataset experiment also shows that our method has good generalization ability. Furthermore, our method is fast as our implementation runs at over 91 frames per second on a standard computer with a single GPU.