Motionlets Matching With Adaptive Kernels for 3-D Indian Sign Language Recognition

Abstract

Recognizing human gestures in sign language is a complex and challenging task. Human sign language gestures are a combination of independent hand and finger articulations, which are sometimes performed in coordination with the head, face, and body. 3-D motion capture of sign language involves recording 3-D sign videos that are often affected by interobject or self-occlusions, lighting, and background. This paper proposes characterization of sign language gestures articulated at different body parts as 3-D motionlets, which describe the signs with a subset of joint motions. A two-phase fast algorithm identifies 3-D query signs from an adaptively ranked database of 3-D sign language. Phase-I process clusters all human joints into motion joints (MJ) and nonmotion joints (NMJ). The relation between MJ and NMJ is analyzed to categorically segment the database into four motionlet classes. Phase-II process investigates the relation within the motion joints to represent shape information of a sign as 3-D motionlets. The 4-class sign database features three adaptive motionlet kernels. A simple kernel matching algorithm is used to rank the database according to the highest-ranked query sign. The proposed method is sign invariant to temporal misalignment and can characterize sign language based on a 3-D spatiotemporal framework. In this paper, five 500-word Indian sign language data sets were used to evaluate the proposed model. The experimental results reveal that the method proposed here improved recognition compared with the state-of-the-art 3-D action recognition methods.