Real-time fingerspelling recognition system design based on RGB-D image information

Abstract

This paper provides a fingerspelling recognition system with high accuracy rate based on RGB-D image. The system are separated into three parts, including hand region detection, hand feature extraction, and fingerspelling recognition. For the hand regions detection, the regions of hand and face are first obtained by skin color detection and connect component labeling (CCL), and then the hand, the region-of-interest (ROI), is determined by the feature point extraction based on distance transform. Followed is the hand feature extraction which consists of the hand structure and the hand texture. From the feature points of ROI, the locations of palm and fingertips, palm direction, and finger vectors are formed as the hand structure. In addition to the hand structure, this paper adopts the LBP operator to generate the hand texture to deal with the fingerspelling not recognizable by the hand structure. Finally, the extracted hand features are sent into the fingerspelling recognition system, which is built with several different neural network classifiers. The experimental results show that this system is an effective real-time recognition system whose accuracy is higher than 80% for most of the fingerspelling in American Sign Language (ASL).