A Deep Learning Approach for Hybrid Hand Gesture Recognition

Abstract

Emerging depth sensors and new interaction paradigms enable to create more immersive and intuitive Natural User Interfaces by recognizing body gestures. One of the vision-based devices that has received plenty of attention is the Leap Motion Controller (LMC). This device models the 3D position of hands and fingers and provides more than 50 features such as palm center and fingertips. In spite of the fact that the LMC provides such useful information of the hands, developers still have to deal with the hand gesture recognition problem. For this reason, several researchers approached this problem using well-known machine learning techniques used for gesture recognition such as SVM for static gestures and DTW for dynamic gestures. At this point, we propose an approach that applies a resampling technique based on fast Fourier Transform algorithm to feed a CNN+LSTM neural network in order to identify both static and dynamic gestures. As far as our knowledge, there is no full dataset based on the LMC that includes both types of gestures. Therefore, we also introduce the Hybrid Hand Gesture Recognition database, which consists of a large set of gestures generated with the LMC, including both type of gestures with different temporal sizes. Experimental results showed the robustness of our approach in terms of recognizing both type of gestures. Moreover, our approach outperforms other well-known algorithms of the gesture recognition field.