Hand Pose Research Articles

3D Hand Tracking in the Presence of Excessive Motion Blur.

We present a sensor-fusion method that exploits a depth camera and a gyroscope to track the articulation of a hand in the presence of excessive motion blur. In case of slow and smooth hand motions, the existing methods estimate the hand pose fairly accurately and robustly, despite challenges due to the high dimensionality of the problem, self-occlusions, uniform appearance of hand parts, etc. However, the accuracy of hand pose estimation drops considerably for fast-moving hands because the depth image is severely distorted due to motion blur. Moreover, when hands move fast, the actual hand pose is far from the one estimated in the previous frame, therefore the assumption of temporal continuity on which tracking methods rely, is not valid. In this paper, we track fast-moving hands with the combination of a gyroscope and a depth camera. As a first step, we calibrate a depth camera and a gyroscope attached to a hand so as to identify their time and pose offsets. Following that, we fuse the rotation information of the calibrated gyroscope with model-based hierarchical particle filter tracking. A series of quantitative and qualitative experiments demonstrate that the proposed method performs more accurately and robustly in the presence of motion blur, when compared to state of the art algorithms, especially in the case of very fast hand rotations.

CFAM: Estimating 3D Hand Poses from a Single RGB Image with Attention

Precise 3D hand pose estimation can be used to improve the performance of human–computer interaction (HCI). Specifically, computer-vision-based hand pose estimation can make this process more natural. Most traditional computer-vision-based hand pose estimation methods use depth images as the input, which requires complicated and expensive acquisition equipment. Estimation through a single RGB image is more convenient and less expensive. Previous methods based on RGB images utilize only 2D keypoint score maps to recover 3D hand poses but ignore the hand texture features and the underlying spatial information in the RGB image, which leads to a relatively low accuracy. To address this issue, we propose a channel fusion attention mechanism that combines 2D keypoint features and RGB image features at the channel level. In particular, the proposed method replans weights by using cascading RGB images and 2D keypoint features, which enables rational planning and the utilization of various features. Moreover, our method improves the fusion performance of different types of feature maps. Multiple contrast experiments on public datasets demonstrate that the accuracy of our proposed method is comparable to the state-of-the-art accuracy.

Light and Fast Hand Pose Estimation From Spatial-Decomposed Latent Heatmap

We present a light and efficient approach named Latent Fusion network for fast and accurate hand pose estimation from a single depth image. Our method innovatively decomposes 3D joint regression into 2D plane localization and 1D axis estimation from different spatial perspectives. We design multiple latent heatmap regression branches to predict hand pose separately and a fusion network to output the final result. Experiments on three public hand pose datasets (ICVL, NYU, MSRA) demonstrate that our system achieves state-of-the-art accuracy. Moreover, our method outperforms all top-ranked approaches by a large margin both in terms of inference speed (nearly a thousand frames per second) and model size (less than 10 MB).

Embedding Gesture Prior to Joint Shape Optimization Based Real-Time 3D Hand Tracking

In this paper, we present a novel approach for 3D hand tracking in real-time from a set of depth images. In each frame, our approach initializes hand pose with learning and then jointly optimizes the hand pose and shape. For pose initialization, we propose a gesture classification and root location network (GCRL), which can capture the meaningful topological structure of the hand to estimate the gesture and root location of the hand. With the per-frame initialization, our approach can rapidly recover from tracking failures. For optimization, unlike most existing methods that have been using a fixed-size hand model or manual calibration, we propose a hand gesture-guided optimization strategy to estimate pose and shape iteratively, which makes the tracking results more accuracy. Experiments on three challenging datasets show that our proposed approach achieves similar accuracy as state-of-the-art approaches, while runs on a low computational resource (without GPU).

Learning Task-Oriented Grasping From Human Activity Datasets

We propose to leverage a real-world, human activity RGB dataset to teach a robot Task-Oriented Grasping (TOG). We develop a model that takes as input an RGB image and outputs a hand pose and configuration as well as an object pose and a shape. We follow the insight that jointly estimating hand and object poses increases accuracy compared to estimating these quantities independently of each other. Given the trained model, we process an RGB dataset to automatically obtain the data to train a TOG model. This model takes as input an object point cloud and outputs a suitable region for task-specific grasping. Our ablation study shows that training an object pose predictor with the hand pose information (and vice versa) is better than training without this information. Furthermore, our results on a real-world dataset show the applicability and competitiveness of our method over state-of-the-art. Experiments with a robot demonstrate that our method can allow a robot to preform TOG on novel objects.

SPH Fluid Tactile Rendering for Ultrasonic Mid-Air Haptics.

In our daily lives, we interact with fluids by touching them directly with our hands. Fluids produce a pressure field against the surface of our hands, and we experience fluid dynamics over our skin temporally and spatially at varying pressure distributions depending on fluid properties as well as on the interacting hand's poses and motions. To improve the realism of fluid simulation together with user interaction, we propose a real-time fluid tactile rendering technique that computes the pressure field on a virtual hand surface to be delivered to the user's actual hand via ultrasound-based mid-air haptic display. Our haptic rendering algorithm computes the feedback force in two stages: First, the pressure distribution of the rigid-fluid interaction is computed from a real-time Lagrangian fluid simulation, and then a set of focal points that reflects the generated pressure field is extracted by using a hill-climbing method which gives the local extrema of the pressure field of simulation. We implement a real-time smoothed-particle hydrodynamics fluid simulator and the proposed haptic rendering algorithm using adaptive amplitude modulation approach to demonstrate the effectiveness of our method in fluid tactile rendering in various scenarios.

DeepAirSig: End-to-End Deep Learning Based in-Air Signature Verification

In-air signature verification is vital for biometric user identification in contact-less mode. The state-of-the-art methods use heuristics for signature acquisition, and provide insufficient data to train neural networks for the verification. In this article, we present a novel method for end-to-end deep learning based in-air signature verification using a depth sensor. In this regard, we propose a new medium-scale in-air signature dataset which is created using an accurate convolutional neural network (CNN) based 3D hand pose estimation algorithm. The proposed dataset offers a total of 1800 signatures collected from 40 subjects. So far, dynamic time warping (DTW) has been the most effective and commonly used method for verification. Keeping in view the significant advancement in deep learning, we present a more accurate deep learning based alternative which outperforms DTW by 67.6%. To this end, we train a personalized autoencoder to reconstruct the signature of each subject. Thereafter, the signature is verified by thresholding the reconstruction loss. We perform extensive experiments by formulating spatial and depth features of the signature in images and point clouds based representations. Moreover, for comparisons, we implement several deep learning algorithms (i.e. linear autoencoder, convolutional autoencoder, and Deep One-Class classifier). Our verification approach achieves an EER of 0.055%. The dataset is available at https://bit.ly/2mEJzOw.

Multihead Self Attention Hand Pose Estimation

In This paper, we propose a hand pose estimation neural networks architecture named MSAHP which can improve PCK (percentage correct keypoints) greatly by fusing self-attention module in CNN (Convolutional Neural Networks). The proposed network is based on a ResNet (Residual Neural Network) backbone and concatenate discriminative features through multiple different scale feature maps, then multiple head self-attention module was used to focus on the salient feature map area. In recent years, self-attention mechanism was applicated widely in NLP and speech recognition, which can improve greatly key metrics. But in compute vision especially for hand pose estimation, we did not find the application. Experiments on hand pose estimation dataset demonstrate the improved PCK of our MSAHP than the existing state-of-the-art hand pose estimation methods. Specifically, the proposed method can achieve 93.68% PCK score on our mixed test dataset.

Fast and Accurate 3D Hand Pose Estimation via Recurrent Neural Network for Capturing Hand Articulations

3D hand pose estimation from a single depth image plays an important role in computer vision and human-computer interaction. Although recent hand pose estimation methods using convolution neural network (CNN) have shown notable improvements in accuracy, most of them have a limitation that they rely on a complex network structure without fully exploiting the articulated structure of the hand. A hand, which is an articulated object, is composed of six local parts: the palm and five independent fingers. Each finger consists of sequential-joints that provide constrained motion, referred to as a kinematic chain. In this paper, we propose a hierarchically-structured convolutional recurrent neural network (HCRNN) with six branches that estimate the 3D position of the palm and five fingers independently. The palm position is predicted via fully-connected layers. Each sequential-joint, i.e. finger position, is obtained using a recurrent neural network (RNN) to capture the spatial dependencies between adjacent joints. Then the output features of the palm and finger branches are concatenated to estimate the global hand position. HCRNN directly takes the depth map as an input without a time-consuming data conversion, such as 3D voxels and point clouds. Experimental results on public datasets demonstrate that the proposed HCRNN not only outperforms most 2D CNN-based methods using the depth image as their inputs but also achieves competitive results with state-of-the-art 3D CNN-based methods with a highly efficient running speed of 285 fps on a single GPU.

Hand pose estimation system based on combined features for mobile devices

Hand pose estimation system based on combined features for mobile devices

Attention-Based Pose Sequence Machine for 3D Hand Pose Estimation

Most of the existing methods for 3D hand pose estimation are performed from a single depth map. In that case, the depth missing challenges from input frames caused by hand self-occlusions and imaging quality lead to multi-valued mapping phenomenon and sub-optimal model. In this paper, we proposed a novel recurrent architecture named Attention-based Pose Sequence Machine (APSM) to alleviate challenges by introducing temporal consistency. As for recurrent unit (RU), we extend traditional Gated Recurrent Unit (GRU) with 3D convolutional neural networks (CNNs) to handle voxelized inputs and features, and a novel RU named Deep Gated Recurrent Unit (DGRU) was proposed by rebuilding deeper gates based on GRU. To improve the model performance, a novel spatial attention mechanism denoted as Attention Model (AM) was proposed. Ablation experiments are designed to validate each contribution of our work, and experiments on two publicly available dataset show that our work outperforms state-of-the-art on hand pose estimation.

3D hand pose estimation method using CNN and high-speed vision system

3D hand pose estimation method using CNN and high-speed vision system

Hand pose estimation system based on combined features for mobile devices

Today's mobile devices or smartphones have a revolutionary impact on how we communicate, especially after the advent of devices like smart watches and Google glasses that require new ways to interact with them. To optimise the use of mobile devices, special input and output peripherals have been designed over the years to facilitate communication with them. The well known peripherals are the multi-touch screens. Smartphones are too small to work freely using their input screens. To solve this problem, recent research has focused on contactless and natural gestural interfaces. In this context, we propose a hand gesture recognition system for mobile devices as a simple way of communication with smartphones. In this work, we describe a hand gesture recognition system for Android devices based on a combination of HOG and LBP features and the SVM classifier. An accuracy rate of about 95% is obtained on the improved 'NUS database I'. In addition, we conduct experiments on different Android devices to know the impact of using such a recognition task on embedded systems like smartphones.

Accurate Hand Detection from Single-Color Images by Reconstructing Hand Appearances.

Hand detection is a crucial pre-processing procedure for many human hand related computer vision tasks, such as hand pose estimation, hand gesture recognition, human activity analysis, and so on. However, reliably detecting multiple hands from cluttering scenes remains to be a challenging task because of complex appearance diversities of dexterous human hands (e.g., different hand shapes, skin colors, illuminations, orientations, and scales, etc.) in color images. To tackle this problem, an accurate hand detection method is proposed to reliably detect multiple hands from a single color image using a hybrid detection/reconstruction convolutional neural networks (CNN) framework, in which regions of hands are detected and appearances of hands are reconstructed in parallel by sharing features extracted from a region proposal layer, and the proposed model is trained in an end-to-end manner. Furthermore, it is observed that the generative adversarial network (GAN) could further boost the detection performance by generating more realistic hand appearances. The experimental results show that the proposed approach outperforms the state-of-the-art on public challenging hand detection benchmarks.

MVPointNet: Multi-View Network for 3D Object Based on Point Cloud

The research of hand pose estimation is a hot topic in computer vision, robotics and virtual reality. Compared with using data glove, vision based methods show great advantage for its contactless property, low-cost and convenience. With the commercial depth cameras became widely available and the great success of Convolution Neural Network (CNN) on images, various works focused on hand pose estimation have achieved promising performance. This research is inspired by the recent work that directly perform 3D classification and segmentation tasks on point cloud. In this paper, Multi-View PointNet (MVPointNet) is proposed which takes several views of point cloud as input source. Then, they fed into the well-performing point cloud-based architecture. In addition, to better capture the hand context structure and improve the performance, more features between centroid and local neighborhood points (norm, edge, angle) are extracted and fed into a deep CNN architecture. To our knowledge, our proposed method achieved good performance on the ModelNet40 dataset for 3D shape classification. Besides, it achieved superior performance over other deep learning methods for 3D hand pose estimation based on point cloud, which is evaluated on MSRA dataset.

Estimation of 3D human hand poses with structured pose prior

Here, the authors present multistage estimation model embedding with structured pose prior (SPP), a novel coarse-to-fine framework for real-time 3D hand estimation from single depth image. Authors’ main contributions can be summarised as follows: (i) The authors proposed SPP to enforce constraints of canonical hand pose instead of original hand pose. (ii) The authors are the first to adopt under-complete stacked denoising auto-encoder (SDA) to construct pose prior by mapping canonical hand pose to latent representation. In the case of enforcing constraints of canonical hand pose, the authors empirically validate that under-complete SDA outperforms over-complete SDA in improving the hand estimation accuracy. (iii) The authors propose candidate keypoints patches (CKP) as intermediate data to conduct further hand pose refinement. Experimental evaluation on two publically available datasets shows that authors’ model is competitive both in accuracy and computation time. Especially, authors’ method placed first in the location of palm key-point on both two datasets, and the high accuracy of hand palm key-point plays an important role in many applications, such as that manipulator can grasp objects to specific coordinates with the guiding of human hand palm.

SRHandNet: Real-time 2D Hand Pose Estimation with Simultaneous Region Localization.

This paper introduces a novel method for real-time 2D hand pose estimation from monocular color images, which is named as SRHandNet. Existing methods can not time efficiently obtain appropriate results for small hand. Our key idea is to simultaneously regress the hand region of interests (RoIs) and hand keypoints for a given color image, and iteratively take the hand RoIs as feedback information for boosting the performance of hand keypoints estimation with a single encoder-decoder network architecture. Different from previous region proposal network (RPN), a new lightweight bounding box representation, which is called region map, is proposed. The proposed bounding box representation map together with hand keypoints heatmaps are combined into the unified multi-channel feature maps, which can be easily acquired with only one forward network inference and thus improve the runtime efficiency of the network. Our proposed SRHandNet can run at 40fps for hand bounding box detection and up to 30fps accurate hand keypoints estimation under the desktop environment without implementation optimization. Experiments demonstrate the effectiveness of the proposed method. State-of-the-art results are also achieved out competing all recent methods.

Submerged Urlinien, Shadow Urlinien, and Urlinie Play in the Left-Hand Melody Piece

Abstract Piano pieces with the principal melody in the left hand pose interesting challenges for Schenkerian analysis, which generally favors outer voices for the components of the Ursatz. In the left-hand melody piece, the accompaniment is positioned above the melody, and bass lines may not be literally present. Middleground sketches of such pieces often featured a submerged Urlinie, which is a fundamental line that is positioned in an inner voice of the work’s texture (after Schachter 1994). These pieces also quite frequently involve “shadow Urlinien,” which are potentially viable fundamental lines in an upper voice that are perceptually weaker than Urlinien positioned in the principal melodic register. “Urlinie play” describes the analytical and phenomenological interplay between two potential fundamental lines, such as a submerged Urlinie and a shadow Urlinie. The analyses in this article examine aspects of Urlinie play in left-hand melody pieces by Schumann, Massenet, Beethoven, and Chopin, and describe that play in terms of four narrative categories: shadowing, role reversion, denial, and competition.

Robust hand pose estimation using visual sensor in IoT environment

In Internet of Things (IoT) environments, visual sensors with good performance have been used to create and apply various kinds of image data. Particularly, in the field of human–computer interaction, the image sensor interface using human hands is applicable to sign language recognition, games, object operation in virtual reality, and remote surgery. With the popularization of depth cameras, there has been a new interest in the research conducted in RGB images. Nevertheless, hand pose estimation is hard. Research on hand pose estimation has multiple issues, including high-dimensional degrees of freedom, shape changes, self-occlusion, and real-time condition. To address the issues, this study proposes the random forests-based method of hierarchically estimating hand pose in depth images. In this study, the hierarchical estimation method that individually handles hand palms and fingers with the use of an inverse matrix is utilized to address high-dimensional degrees of freedom, shape changes, and self-occlusion. For real-time execution, random forests using simple characteristics are applied. As shown in the experimental results of this study, the proposed hierarchical estimation method estimates the hand pose in input depth images more robustly and quickly than other existing methods.

A multi-branch hand pose estimation network with joint-wise feature extraction and fusion

The study of 3D hand pose estimation from a single depth image is regarded as a detection-based or regression-based problem among most of the existing deep learning-based methods, and this approach does not fully exploit the geometry of the hand, such as its structural and physical constraints. To overcome these weaknesses, we design a network with three simple parallel branches that correspond to the three functional parts of the hand. This observation is motivated by the biological viewpoint that each finger plays a different role in performing grasping and manipulation. In each branch, we perform a more detailed regression in two stages – top-down joint location regression followed by bottom-up hand pose regression – which fully exploits both the local and global structure of a hand. Finally, we further make use of the hand structure and physical constraints to refine each joint by its auxiliary points. The proposed network is a unified structure and function model that is more appropriate for hand pose estimation. Our system does not require pose pre-processing or feedback since it can directly perform training and predicting from end-to-end. The experimental results on three public datasets demonstrate that the proposed system achieves performance comparable to state-of-the-art methods.