Hand gesture recognition using kinect via deterministic learning

Abstract

Hand gestures are spatio-temporal patterns which can be characterized by collections of spatio-temporal features. However, in real world scenarios, hand gesture recognition suffers from huge challenges with variations of illumination, poses and occlusions. The Microsoft Kinect device provides an effective way to solve the above issues and extract discriminative features for hand gesture recognition. The recognition approach consists of two stages: a training stage and a recognition stage. In the training stage, hand gesture features representing hand motion dynamics, including spatial position and direction of fingertips, are derived from Kinect. Hand motion dynamics underlying motion patterns of different gestures which represent Arabic numbers (0–9) are locally accurately modeled and approximated by radial basis function (RBF) neural networks. The obtained knowledge of approximated hand motion dynamics is stored in constant RBF networks. In the recognition stage, a bank of dynamical estimators is constructed for all the training patterns. Prior knowledge of hand motion dynamics represented by the constant RBF networks is embedded in the estimators. By comparing the set of estimators with a test gesture pattern to be recognized, a set of recognition errors are generated. The average L 1 norms of the errors are taken as the recognition measure between the dynamics of the training gesture patterns and the dynamics of the test gesture pattern according to the smallest error principle. By using the 2-fold and 10-fold cross-validation styles, the correct recognition rates are reported to be 95.83% and 97.25%, respectively.