REAL-TIME HAND GESTURE CLASSIFICATION USING CRNN WITH SCALE AVERAGE WAVELET TRANSFORM

Abstract

It is very useful in the human computer interface to quickly and accurately recognize human hand movements in real time. In this paper, we aimed to robustly recognize hand gestures in real time using Convolutional Recurrent Neural Network (CRNN) with pre-processing and overlapping window. The CRNN is a deep learning model that combines Long Short-Term Memory (LSTM) for time-series information classification and Convolutional Neural Network (CNN) for feature extraction. The sensor for hand gesture detection uses Myo-armband, and six hand gestures are recognized and classified, including two grips, three hand signs, and one rest. As the essential pre-processing due to the characteristics of EMG data, the existing Short Time Fourier Transform (STFT), Continuous-time Wavelet Transform (CWT), and newly proposed Scale Average Wavelet Transform (SAWT) are used, and thus, the SAWT showed relatively high accuracy in the stationary environmental test. The CRNN with overlapping window has been proposed that can improve the degradation of real-time prediction accuracy, which is caused by inconsistent start time and hand motion speed when acquiring the EMG signal. In the stationary environmental test, the CRNN model with SAWT and overlapping window showed the highest accuracy of 92.5%. In the real-time environmental test, for all subjects learning, 80% accuracy and 0.99 s time delay were obtained on average, and for individual learning, 91.5% accuracy and 0.32 s time delay were obtained on average. As a result, in both stationary and real-time tests, the CRNN with SAWT and overlapping window showed better performance than the other methods.