An Intelligent Arabic Sign Language Recognition System Using a Pair of LMCs With GMM Based Classification

Abstract

In this paper, we propose a dual leap motion controllers (LMC)-based Arabic sign language recognition system. More specifically, we introduce the concept of using both front and side LMCs to cater for the challenges of finger occlusions and missing data. For feature extraction, an optimum set of geometric features is selected from both controllers, while for classification, we used both a Bayesian approach with a Gaussian mixture model (GMM) and a simple linear discriminant analysis (LDA) approach. To combine the information from the two LMCs, we introduce an evidence-based fusion approach; namely, the Dempster–Shafer (DS) theory of evidence. Data were collected from two native adult signers, for 100 isolated Arabic dynamic signs. Ten observations were collected for each of the signs. The proposed framework uses an intelligent strategy to handle the case of missing data from one or both controllers. A recognition accuracy of about 92% was achieved. The proposed system outperforms state-of-the-art glove-based systems and single-sensor-based techniques.