Transfer-Learning Based User-Personalization of Indian Sign Language Recognition System

Abstract

Differently abled people who cannot speak or hear often face difficulties when communicating with others, due to unavailability of translators, and sign languages not being common knowledge. Several research works report Sign Language Recognition system leveraging deep learning algorithms. However, till date, a well-performing, highly generalizable, and cost-effective system is not available for commercial use. In this work, classification of 50 signs from the Indian Sign Language (ISL) is carried out using convolutional neural network on time-series data. Data for SLR is captured using wireless multi-modality wearable sensors. A novel deep transfer learning algorithm is proposed for personalization of the sign language recognition (SLR) system for a new user. The performance of the proposed approach is tested on 5 new users. The proposed model yields the best average accuracy of 95.6% after applying transfer-learning based personalization of the network with six samples of each sign from the new user. This is significantly better as compared to the 3% accuracy obtained when the model is tested without applying transfer learning, proving the effectiveness of proposed approach in handling subject variability.