Real-Time American Sign Language Recognition Using Skin Segmentation and Image Category Classification with Convolutional Neural Network and Deep Learning

Abstract

A real-time sign language translator is an important milestone in facilitating communication between the deaf community and the general public. We hereby present the development and implementation of an American Sign Language (ASL) fingerspelling translator based on skin segmentation and machine learning algorithms. We present an automatic human skin segmentation algorithm based on color information. The YCbCr color space is employed because it is typically used in video coding and provides an effective use of chrominance information for modeling the human skin color. We model the skin-color distribution as a bivariate normal distribution in the CbCr plane. The performance of the algorithm is illustrated by simulations carried out on images depicting people of different ethnicity. Then Convolutional Neural Network (CNN) is used to extract features from the images and Deep Learning Method is used to train a classifier to recognize Sign Language.