Hand Gesture Recognition Using FMCW Radar and Deep Learning for Understanding Deaf Sign Language

Abstract

People have difficulty in understanding sign language of deaf people therefore a technology is needed to translate sign language to help deaf people. In Indonesia, society lacks fluency in national sign languages like Indonesian Sign Language (SIBI), which can affect their self-confidence and social interaction skills with others. Previous studies generally used a camera in dataset retrieval in sign language recognition. There is a weakness in using the camera because it has to pay attention to environmental conditions, such as being influenced by light intensity. Radar can correct this deficiency because it uses electromagnetic waves. In this study proposed, the use of uRAD radar by Antheral based on radar Frequency Modulated Continuous Wave (FMCW) to detect and then collect words of SIBI for raw data that is IQ signals. The proposed technique applies the preprocessing methods for the frequency-domain IQ signals of radar echoes as input to a convolutional neural network (CNN). The system designed is able to classify five words of Indonesian Sign Language into five classes. Thereafter compare three self-constructed architectures to find the best model with differences in the number of convolution and pooling layers in each architecture. The results of the proposed architectural classification show the movement classification accuracy exceeds more than 98%. This study has high accuracy because of the proposed preprocessing method and compares several CNN architectures whose accuracy is higher than previous studies which also used radar. Based on the results, the proposed system can be helpful in understanding deaf sign language with SIBI.