Deep Learning Based Real-Time Recognition of Dynamic Finger Gestures Using a Data Glove

Minhyuk Lee,Joonbum Bae

doi:10.1109/access.2020.3039401

Abstract

In this article, a real-time dynamic finger gesture recognition using a soft sensor embedded data glove is presented, which measures the metacarpophalangeal (MCP) and proximal interphalangeal (PIP) joint angles of five fingers. In the gesture recognition field, a challenging problem is that of separating meaningful dynamic gestures from a continuous data stream. Unconscious hand motions or sudden tremors, which can easily lead to segmentation ambiguity, makes this problem difficult. Furthermore, the hand shapes and speeds of users differ when performing the same dynamic gesture, and even those made by one user often vary. To solve the problem of separating meaningful dynamic gestures, we propose a deep learning-based gesture spotting algorithm that detects the start/end of a gesture sequence in a continuous data stream. The gesture spotting algorithm takes window data and estimates a scalar value named gesture progress sequence (GPS). GPS is a quantity that represents gesture progress. Moreover, to solve the gesture variation problem, we propose a sequence simplification algorithm and a deep learning-based gesture recognition algorithm. The proposed three algorithms (gesture spotting algorithm, sequence simplification algorithm, and gesture recognition algorithm) are unified into the real-time gesture recognition system and the system was tested with 11 dynamic finger gestures in real-time. The proposed system took only 6 ms to estimate a GPS and no more than 12 ms to recognize the completed gesture in real-time.

Highlights

D EVICES and techniques that facilitate human-computer interaction (HCI) have attracted a great deal of interest
Hand gesture recognition is accepted as an effective natural interface, and both static and dynamic gesture recognition have been studied
Static hand gesture recognition can be achieved by applying standard pattern recognition techniques such as template matching [4], whereas dynamic hand gesture recognition requires time-series pattern recognition algorithms such as hidden Markov models (HMMs) or dynamic time warping (DTW) algorithms [5]

Summary

Introduction

D EVICES and techniques that facilitate human-computer interaction (HCI) have attracted a great deal of interest. Hand gesture recognition is accepted as an effective natural interface, and both static and dynamic gesture recognition have been studied. Hand gesture recognition can be achieved via vision-based and data glove-based approaches. Most work on hand gesture recognition has employed vision sensors because few data gloves are commercially available, and most are expensive and hinder natural hand motion [1]– [3]. Static hand gesture recognition can be achieved by applying standard pattern recognition techniques such as template matching [4], whereas dynamic hand gesture recognition requires time-series pattern recognition algorithms such as hidden Markov models (HMMs) or dynamic time warping (DTW) algorithms [5]. There have been some studies that focused on recognition of dynamic hand gestures by applying recurrent neural networks (RNNs) [6]

Results

Discussion

Conclusion