Arabie sign language recognition using the Microsoft Kinect

Abstract

Several studies have been carried on sign language recognition systems, however, practically deployable system for real-time use is still a challenge. Traditionally, sign language recognition systems have either used sensor gloves or digital cameras to acquire and process hand gestures. Both approaches exhibit some disadvantages for real time deployment that hindered it large scale adoption. With the growth witnessed in gaming systems, two new instruments have been introduced namely, the Microsoft Kinect (MK) and the leap motion controller. The MK system has been developed to interact with video games by tracking full body movements and gestures. To overcome some of the disadvantages of the classical methods, we propose here to develop an Arabic sign language recognition system based on MK system. The developed system was tested with 20 signs from the Arabic sign language dictionary. Therefore, in this paper, we present our experiment carried out using the MK setup on 20 Arabic sign language words. Video samples of both true color images and depth images were collected from native deaf signer. Linear Discriminant analysis was used for feature dimension reduction and sign classification. Furthermore, fusion from RGB and depth sensor was carried at feature and decision level giving an overall best accuracy of 99.8%.