Spatio-temporal 3D surface matching for hand gesture recognition using ICP algorithm

Abstract

With emerging new applications like virtual reality, different algorithms for human action and gesture recognition have been proposed. In this paper, a new method for the recognition of moving hand gestures is presented. The proposed algorithm is based on the representation of hand motion as spatio-temporal 3D surfaces. Then, 3D surface matching is used to recognize the hand gesture. To form the spatio-temporal 3D surface of hand motion, we first apply the necessary preprocessing to video frames and extract hand contours. Then, by normalizing and overlapping hand contours in different frames, we construct spatio-temporal 3D surface of the hand gesture. To recognize hand gesture, we match the input 3D surface with surfaces in the database. For this purpose, we utilize ICP algorithm to find and compensate for 3D transformation between surfaces as well as the similarity measure between them. In real-world applications, hand motion is continuous and results in a sequence of disjointed hand gestures, which is called continuous hand gesture. To recognize continuous hand gestures, we propose an algorithm which first estimates probable disjointed gestures in the continuous gesture and then divides iteratively continuous gestures to true disjointed gestures. Finally, by applying a robust algorithm, the continuous gesture is recognized. We tested the proposed algorithm with hand gestures of American sign language and results showed the recognition rate of 94 % for disjointed gestures and 93.9 % for continuous gestures. The experimental results showed the efficiency of the proposed algorithm for hand gestures with noise as well.