A combination of user-guide scheme and kernel descriptor on RGB-D data for robust and realtime hand posture recognition

Abstract

This paper presents a robust and real-time hand posture recognition system. To obtain this, key elements of the proposed system contain an user-guide scheme and a kernel-based hand posture representation. We firstly describe a three-stage scheme to train an end-user. This scheme aims to adapt environmental conditions (e.g., background images, distance from device to hand/human body) as well as to learn appearance-based features such as hand-skin color. Thanks to the proposed user-guide scheme, we could precisely estimate heuristic parameters which play an important role for detecting and segmenting hand regions. Based on the segmented hand regions, we utilize a kernel-based hand representation in which three levels of feature are extracted. Whereas pixel-level and patch-level are conventional extractions, we construct image-level which presents a hand pyramid structure. These representations contribute to a Multi-class support vector machine classifier. We evaluate the proposed system in term of the learning time versus the robustness and real time performances. Averagely, the proposed system requires 14s in advanced to guide an end-user. However, the hand posture recognition rate obtains 91.2% accuracy. Performance of the proposed system is comparable with state-of-the-art methods (e.g. Pisharady et al., 2012) but it is a real time system. To recognize a posture, its computational cost is only 0.15s. This is significantly faster than works in Pisharady et al. (2012), which required approximately 2min. The proposed methods therefore are feasible to embed into smart devices, particularly, consumer electronics in domain of home-automation such as televisions, game consoles, or lighting systems.