Abstract

<p class="Abstract"><span lang="EN-US">Gesture recognition is a fundamental step to enable efficient communication for the deaf through the automated translation of sign language. This work proposes the usage of a high-precision magnetic positioning system for 3D positioning and orientation tracking of the fingers and hands palm. The gesture is reconstructed by the MagIK (magnetic and inverse kinematics) method and then processed by a deep learning gesture classification model trained to recognize the gestures associated with the sign language alphabet. Results confirm the limits of vision-based systems and show that the proposed method based on hand skeleton reconstruction has good generalization properties. The proposed system, which combines sensor-based gesture acquisition and deep learning techniques for gesture recognition, provides a 100% classification accuracy, signer independent, after a few hours of training using transfer learning technique on well-known ResNet CNN architecture. The proposed classification model training method can be applied to other sensor-based gesture tracking systems and other applications, regardless of the specific data acquisition technology.</span></p>

Highlights

  • Sign language recognition (SLR) is a research area that involves gesture tracking, pattern matching, computer vision, natural language processing, linguistics, and machine learning [1]

  • The 100% accuracy was confirmed after deployment with test cases from the American Sign Language MNIST

  • Combining sensor-based acquisition, visual reconstruction of the skeleton, and a deep convolutional neural network (CNN) classification model, the proposed system achieves 100% inference accuracy on gestures performed by different people after a few epochs of training

Read more

Summary

INTRODUCTION

Sign language recognition (SLR) is a research area that involves gesture tracking, pattern matching, computer vision, natural language processing, linguistics, and machine learning [1]. The final goal of SLR is to develop methods and algorithms to build an SRL system (SLRS) capable of identifying signs, decoding their meaning, and producing some output that the intended receiver can understand (Figure 1). Available literature surveys [2]-[5] report that recent research achieved accuracy in the range of 80–100% for the first two tasks using vision-based and sensor-based approaches. We compare the performance of the two systems we developed: a vision-based system and a hybrid system with sensor-based data acquisition and vision-based classification stages

SLRS Performance Assessment
Sign Language
VISION-BASED SIGN LANGUAGE GESTURE RECOGNITION
Classic Machine Learning and Convolutional Neural Network on MNIST Dataset
Vision-based Classification Accuracy
Efficient Deep CNN Training for Sign Language Recognition
Gesture Classification Inference with MPS
Findings
CONCLUSIONS
Full Text
Published version (Free)

Talk to us

Join us for a 30 min session where you can share your feedback and ask us any queries you have

Schedule a call