Live demonstration: A HMM-based real-time sign language recognition system with multiple depth sensors

Abstract

Automatic sign language recognition plays an important role in communications for sign language users. Most existing sign language recognition systems use single sensor input. However, such systems may fail to recognize hand gestures correctly due to occluded regions of hand gestures. In this work, we propose a novel system for real-time recognition of the digits in American Sign Language (ASL) [1]. The proposed system [2] utilizes two Leap Motion sensors [3] to capture hand gestures from different angles. Sensory data are preprocessed using a multi-sensor data fusion approach and ASL digits are recognized in real-time from the fused data using Hidden Markov models (HMM) [4]. Experimental results of the proposed sign language recognition system demonstrate its improved performance over single sensor systems. With a low implementation cost and a high recognition accuracy, the proposed system can be widely adopted in many real world applications and bring conveniences to world-wide ASL users.