Stacking Ensemble of Convolutional Neural Networks for Sign Language Recognition

Abstract

Sign language involves the use of various hand gestures and motions for conveying information. It is predominantly used by the deaf community, while very rarely understood by others. Machine learning and deep learning models have been widely reported for automatic recognition of sign language from wearable and vision-based sensor data. In this paper, data from three multi-modality wireless sensors placed on the dominant hand of the signer is processed using a homogeneous ensemble of convolutional neural networks (CNN) to recognize the performed sign. 50 signs from the Indian sign language are performed by 5 subjects multiple times to develop the sensor signal database. The features extracted for the multi-modality data are given as input to randomly initialized CNN models to enable extraction of high-level features. The class probabilities at the output of the CNN models are stacked and interpreted by a meta-learner to determine the performed sign. Experimental results are presented to analyze the effect of including varying number of CNN models in the ensemble, as well to determine the best meta-learner for classification of the performed signs using the stacked CNN ensemble. The best average classification accuracy of 87.6% is obtained for a stacked ensemble of 25 CNN models with multi-layer perceptron as the meta-learner, which is significantly better as compared to that of individual CNN models, were a maximum accuracy of just 79.5% was obtained.