Extraction of texture and geometrical features from informative facial regions for sign language recognition

Abstract

In general, the most common form of gestures is made up of movements of the hand and/or arm associated with facial expressions. In this, the hand is used to make different message signs, while facial movements are used to reflect the mood and emotion of the person. In this paper, some sign language gestures are recognized only with the help of associated facial expressions. Existing facial expression based sign language recognition (SLR) methods only used facial geometric features to recognize sign language gestures. However, the performance of geometric feature-based SLR methods depends on the accuracy of tracking algorithms and the number of facial landmark points. Additionally, facial textures are more informative as compared to the geometric features of a face. Inspiring from these facts, we propose to recognize sign language gestures with the help of spatio-temporal characteristics of facial texture patterns. For this, a new face model is proposed by extracting texture features only from the informative regions of a face. The proposed face model can also be employed to extract the geometrical features of a face. The features extracted from the informative regions of a face are significantly discriminative, and so the proposed face model can track/encode the facial dynamics of the associated facial expressions of a sign. Finally, a 3-state hidden conditional random field is employed to model the texture variations of facial gestures. Experimental results on RWTH-BOSTON data-set show that proposed method can achieve upto 80.06% recognition rate.