Dynamic Hand Gesture Recognition using Convolutional Neural Network with RGB-D Fusion

Abstract

Listen

Translate article

In this paper, we propose a novel, real-time dynamic hand gesture recognition framework using convolutional neural network with depth and RGB data fusion. Hand gestures are a natural form of communication between humans as well as between human and machine. They also find important applications in areas such as sign language recognition, man-machine interaction and behavior understanding. Natural hand gestures are complex hand movements in space and time and are challenging to recognize. In our proposed framework, we use both RGB and depth data to automatically recognize dynamic hand gestures. Initially, we work with RGB and depth data separately. We find the motion history of the gesture performed with RGB data and independently with depth data to store the motion information of the moving hands. Motion history of the performed gesture stores the rich information of the movement. Then, we use transfer learning on two separate VGG16 networks, where one network is fine-tuned using RGB motion history while the other network is fine-tuned using depth motion history, to configure them for dynamic hand gesture recognition problem. Then, using the two fine-tunned VGG16 networks, we extract the features of both the motion history images obtained from RGB and depth data separately, for each dynamic hand gesture. We then, integrate the features obtained from both the networks using weighted summation, to accurately and robustly recognize the dynamic hand gesture. We perform experiments on standard and the publicly available dynamic hand gesture datasets and show that our method outperforms state of the art methods.