Improving articulated hand pose detection for static finger sign recognition in RGB-D images

Abstract

With the emergence of consumer RGB-D sensors, discriminative modeling has been shown to perform well in estimating human body pose. However, articulated hand pose estimation remains a challenging problem, mostly due to its high flexibility, occlusions, noisy data, and small area of the fingertips. In this paper, we present an efficient discriminative-based scheme to improve the performance of hand pose estimation from a single depth image. The proposed scheme is inspired by decision forest-based framework, but with several well-motivated modifications. Specifically, we propose a method to estimate 2D in-plane orientation of the hand, which is then utilized to enforce the depth comparison features and make them invariant to in-plane rotation. Subsequently, we investigate the use of random decision forests (RDF) and mean shift algorithm to predict a primary version of hand parts and joint locations. Based on this primary prediction, an adaptive spatial clustering method is applied to correct the misclassified regions, and to deliver the final estimation of hand pose. Along with the proposed scheme, we further develop a new set of highly-distinctive features for static finger sign recognition by utilizing the estimated hand pose configurations and RGB information. The proposed features are straightforward and can effectively capture different aspects of hand pose, such as links from each joint to the closest joints and orientation of each hand part. Extensive experiments on several challenging datasets demonstrate that our approach, compared to decision forest-based methods, is able to provide more precise estimation of hand poses (with up to 21% improvement in joint localization accuracy), and can efficiently recognize more complex static finger signs (93.85% mean recognition accuracy on a challenging 34-finger sign dataset). Our approach is also robust to illumination, inter-hand occlusion, scale, and rotation variance.