Online Dynamic Hand Gesture Recognition Including Efficiency Analysis

Abstract

Online dynamic hand gesture recognition is challenging mainly due to three reasons: (i) There is no indication when a gesture starts and ends in the video, (ii) performed gestures should only be recognized once, and (iii) the entire architecture should be designed considering the memory and power budget. In this paper, a two-level hierarchical structure consisting of a detector and a classifier is proposed which enables offline-working convolutional neural network (CNN) architectures to operate online efficiently by using sliding window approach. For efficiency analysis, different CNN architectures are applied to compare these architectures over offline classification accuracy, number of parameters and computation complexity. In order to evaluate the single-time activations of the detected gestures, we used Levenshtein distance as an evaluation metric since it can measure misclassifications, multiple detections, and missing detections at the same time. The performance of the approach is evaluated on two public datasets - EgoGesture and NVIDIA Dynamic Hand Gesture Datasets - which require temporal detection and classification of the performed hand gestures. ResNeXt-101 model achieves the state-of-the-art offline classification accuracy of 94.03% on EgoGesture benchmark and competitive results on NVIDIA benchmarks. In online recognition, we obtain very good performances with considerable early detections.