CRNN Based OCR for American and British Sign Language Fingerspelling

Abstract

Optical Character Recognition (OCR) technology is mostly used to convert image containing written text (typed, handwritten, printed or scanned) into machine-readable text data. This work explores the first investigation of American Sign Language (ASL) and British Sign Language (BSL) fingerspelling font images to the corresponding English text conversion system. The proposed system is implemented by the Convolutional Recurrent Neural Network (CRNN) model with three different feature extraction methods. We also investigated two types of hyper-parameters such as hidden size and number of iterations. The experimental results show that our system achieved significantly higher conversion quality on the open-test dataset for both ASL and BSL fingerspelling. Our proposed technique can also be used in deaf education, for example, to extract fingerspelling images from exam answer sheets.