Hand Sign Recognition for Thai Finger Spelling: an Application of Convolution Neural Network

Abstract

The finger spelling is a necessary part of Sign Language—an important means of communication among people with hearing disability. The finger spelling is used to spell out names, places or signs that have not yet been defined. A sign recognition system attempts to allow better communication between hearing majority and hearing disability people. Our study investigates Thai Finger Spelling(TFS), its unique characteristics, a design of automatic TFS recognition, and approaches to handle a TFS key potential issue. Our research designs automatic TFS recognition as a two-stage pipeline: (1) locating and extracting a signing hand on the image and (2) classifying the signing image into the valid TFS sign. Signing hand is located and extracted based on color scheme and contour area using Green’s Theorem. Two approaches are examined for signing image classification: Convolution Neural Network(CNN)-based and Histogram of Oriented Gradients(HOG)-based approaches. Our experimental results have shown the viability of the proposed pipeline, which achieves mean Average Precision (mAP) at 91.26. The proposed design outperforms state-of-the-arts in automatic visual TFS recognition. In a practical sign recognition system, invalid TFS signs may appear in sign transition or simply from unaware hand postures. We proposed a formulation, called confidence ratio. Confidence ratio is simple to compute and generally compatible with multi-class classifiers. The confidence ratio has been found to be a promising mechanism for identifying invalid TFS signs. Our findings reveal challenging issues related to TFS recognition, practical design for TFS sign transcription, formulation and effectiveness of confidence ratio.