Arabic Sign Language Recognition through Deep Neural Networks Fine-Tuning

Yaser Saleh,Ghassan Farid Issa

doi:10.3991/ijoe.v16i05.13087

Abstract

<p class="0abstract">Sign Language is considered the main communication tool for deaf or hearing impaired people. It is a visual language that uses hands and other parts of the body to provide people who are in need to full access of communication with the world. Accordingly, the automation of sign language recognition has become one of the important applications in the areas of Artificial Intelligence and Machine learning. Specifically speaking, Arabic sign language recognition has been studied and applied using various intelligent and traditional approaches, but with few attempts to improve the process using deep learning networks. This paper utilizes transfer learning and fine tuning deep convolutional neural networks (CNN) to improve the accuracy of recognizing 32 hand gestures from the Arabic sign language. The proposed methodology works by creating models matching the VGG16 and the ResNet152 structures, then, the pre-trained model weights are loaded into the layers of each network, and finally, our own soft-max classification layer is added as the final layer after the last fully connected layer. The networks were fed with normal 2D images of the different Arabic Sign Language data, and was able to provide accuracy of nearly 99%.</p>

Highlights

According to the World Health Organization in 2019, the number of people with hearing disability is around 466 million with 34 million of these are children [1]
Sign Language has been improved and standardized by many different countries and different cultures resulting in different standards such as the American Sign Language (ASL), British Sign Language (BSL), Arabic Sign Language, and others
This paper presented a model utilizing fine-tuning with deep learning for the specific task of recognizing Arabic Sign Language, hoping to improve areas of related research such as sign language to sound techniques, translation of Arabic sign language to other languages, and many other areas

Summary

Introduction

According to the World Health Organization in 2019, the number of people with hearing disability is around 466 million with 34 million of these are children [1]. It is estimated that this number is expected to double in the 30 years. These numbers show the importance of sign language as a tool for communication between hearing impaired people and the rest of the world. As this paper focuses on Arabic sign language, we found that there is no one standard Arabic sign language, but instead many variations as many as the Arabic-speaking countries. Enough, these different Arabic sign languages share the same signs for alphabets [2]

Objectives

Methods

Findings

Conclusion