An Integrated CNN-LSTM Model for Micro Hand Gesture Recognition

Abstract

Vision based micro gesture recognition systems enable the development of HCI (Human Computer Interaction) interfaces to mirror real-world experiences. It is unlikely that a gesture recognition method will be suitable for every application, as each gesture recognition system rely on the user cultural background and application domain. This research is an attempt to develop a micro gesture recognition system suitable for the asian culture. However, hands vary in shapes and sizes while gesture varies in orientation and motion. For accurate feature extraction, deep learning approaches are considered. Here, an integrated CNN-LSTM (Convolutional Neural Network- Long Short-Term Memory) model is proposed for building micro gesture recognition system. To demonstrate the applicability of the system two micro hand gesture-based datasets namely, standard and local dataset consisting of ten significant classes are used. Besides, the model is tested against both augmented and unaugmented datasets. The accuracy achieved for standard data with augmentation is 99.0%, while the accuracy achieved for local data with augmentation is 96.1% by applying CNN-LSTM model. In case of both the datasets, the proposed CNN-LSTM model appears to perform better than the other pre-trained CNN models including ResNet, MobileNet, VGG16 and VGG9 as well as CNN excluding LSTM.