Dynamic Hand Gesture Based Sign Word Recognition Using Convolutional Neural Network with Feature Fusion

Abstract

Gesture-based sign language recognition systems play an important role in human-computer interaction to develop communication between deaf communities and other people. Where the deaf community, hard of hearing, and deaf family members express their feelings and communicate with others. In this case, hand gestures have been a promising subject and applied to the very practical application of sign language recognition (SLR). SLR is highly influenced by the recognition of hand, as the sign word is a form of communicative gesture. However, the diversity and complexities of the gestures of the hand can greatly affect reliability and recognition rates. To solve this problem, this paper introduces an effective sign word recognition system using a deep learning technique, including feature fusion convolutional neural network. In the proposed system, the input image is captured from the live video using a low cost device, such as a webcam and preprocessed hand gesture image. The pre-processing is accomplished with the conversion of YCbCr, binarization, erosion and finally hole fillings. Two channels of CNN are used to extract the features from preprocessed images. The feature fusion is performed at the fully connected layer and this feature is used for gesture classification by the softmax classifier. An experimental setup established in our laboratory environment and the user can recognize the signs of fifteen common words in real-time. The experimental results show high recognition accuracy in gesture-based sign word recognition compared with the state-of-art systems.