Indian Sign Language Recognition System for Emergency Words by Using Shape and Deep Features

Abstract

Sign Language Recognition Systems (SLRS) are an advanced communication tool that provides a platform for the deaf community to communicate with each other or with the hearing community. In the past decade, SLRS has made significant advances; however, they are still far from realworld implications. Most existing techniques for Sign Language (SL) recognition using the Convolution Neural Network (CNN) based approach provide satisfactory results for isolated word recognition; however, the accuracy can be improved by incorporating handcrafted features with CNN features and eliminating redundant frames. Thus, this paper proposes an Expert System for Indian Sign Language Recognition (ESISLR) for accurately and efficiently predicting isolated sign words. The ESISLR incorporates a keyframe extraction technique for eliminating redundant frames from an extensive sequence of frames. Then the system uses a combination of CNN and handcrafted features, which are given to a stacked Bi Directional Long Short Term Memory (BiLSTM) network for sequence learning. The proposed model uses VGG-19 for CNN feature extraction and Hu Moments (HM) and Zernike Moments (ZM) for extracting handcrafted features. The proposed model achieved an average accuracy of 93.68% with HM and 94.17% with ZM.