American Sign Language Alphabet Recognition by Extracting Feature from Hand Pose Estimation.

Jungpil Shin,Md Al Mehedi Hasan,Akitaka Matsuoka,Azmain Yakin Srizon

doi:10.3390/s21175856

Abstract

Sign language is designed to assist the deaf and hard of hearing community to convey messages and connect with society. Sign language recognition has been an important domain of research for a long time. Previously, sensor-based approaches have obtained higher accuracy than vision-based approaches. Due to the cost-effectiveness of vision-based approaches, researchers have been conducted here also despite the accuracy drop. The purpose of this research is to recognize American sign characters using hand images obtained from a web camera. In this work, the media-pipe hands algorithm was used for estimating hand joints from RGB images of hands obtained from a web camera and two types of features were generated from the estimated coordinates of the joints obtained for classification: one is the distances between the joint points and the other one is the angles between vectors and 3D axes. The classifiers utilized to classify the characters were support vector machine (SVM) and light gradient boosting machine (GBM). Three character datasets were used for recognition: the ASL Alphabet dataset, the Massey dataset, and the finger spelling A dataset. The results obtained were 99.39% for the Massey dataset, 87.60% for the ASL Alphabet dataset, and 98.45% for Finger Spelling A dataset. The proposed design for automatic American sign language recognition is cost-effective, computationally inexpensive, does not require any special sensors or devices, and has outperformed previous studies.

Highlights

Sign language is a form of communication that utilizes visual–manual methodologies such as expressions, hand gestures, and body movements to interact among the deaf and hard of hearing community, yield opinions, and convey meaningful conversations [1]
support vector machine (SVM) has been used as the main classifier, while light gradient boosting machine (GBM) has been utilized for comparison
Features were generated from the estimated coordinates, and character recognition was performed based on these features

Summary

Introduction

Sign language is a form of communication that utilizes visual–manual methodologies such as expressions, hand gestures, and body movements to interact among the deaf and hard of hearing community, yield opinions, and convey meaningful conversations [1]. The term deaf and hard of hearing is employed to identify a person who is either deaf or incapable to speak an oral language or have some level of speaking ability but prefer to not speak to bypass negative or undesired attention that atypical voices seldom attract. Deafness is often expressed as hearing loss or injury which is an entire or moderate inability to hear which may appear in one or both ears of an individual [2,3]. The main reasons for hearing loss involve aging, genetics, noise exposure, a variety of infections, such as chronic ear infections, and certain toxins or medications [2]. The main reasons for mutism include organic, psychological, developmental, or neurological trauma, physical disability, communication disorder, and so on [6]

Objectives

Methods

Results

Conclusion