Vision-Based Continuous Sign Language Spotting Using Gaussian Hidden Markov Model

Abstract

The presence of movement epenthesis ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">me</i> ) between two consecutive signs in a continuous sign language makes sign language recognition (SLR) a challenging task. In this letter, we propose a vision-based continuous SL spotting system, which separates the meaningful signs from the sign sequences by removing the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">me</i> components from H.264/AVC compressed videos. The work is based on a two-state hidden Markov model (HMM) with Gaussian emission probability. The HMM is trained by the feature set extracted from the entire sign sequence video and finally, the hidden state-sequence is decoded using the Viterbi algorithm. From the decoded state-sequence, the sign spotting is done. The feature set comprises features extracted from the compressed domain as well as uncompressed domain analysis of the sign video. The video database is composed of American SL videos collected from Boston University database. From the experimental results, it is seen that the proposed system can spot the sign and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">me</i> frames with a spotting rate of about 83%.