Deep signature-based isolated and large scale continuous gesture recognition approach

Abstract

In recent years, recognition of hand gestures has received considerable attention from the research community, with an increased interest in advanced decision support systems. In applications of this type of system, recognition of hand gestures based on vision is used in the design and development of interfaces such as that of automobile users. This type of interface allows drivers to interact with the vehicle, without affecting their concentrations. This also increases driver comfort, without compromising safety. However, the video-based gesture recognition problem is not trivial and poses problems due to: (i) intra and inter-person variations in the movement of human hand gestures; (ii) variations between people in the shape and size of the human hand; and (iii) variations in lighting; and background noise.In this paper, a new recognition system is proposed to address the problem of large scale continuous gesture recognition with depth and gray-scale input videos. The proposed recognition system contains 3 main phases. Firstly continuous gesture sequences are segmented into isolated gesture using mean of velocity information calculated based on the estimation of deep optical flow. For each isolated segment a set of relevant descriptors named deep signature features is extracted in order to characterize different intensity and spatial information describing the location, the velocity and the orientation of the movement. The features constructed for both depth and gray-scale sequences are fed to linear SVM for classification. To evaluate the effectiveness of our recognition model, it was compared with other methods using two challenging datasets: KTH and Chalearn. Indeed, a first experimental study has shown that, using the KTH dataSet and the SVM classifier, results given by our method outperforms previously published methods and it reaches 95.6% in term of accuarcy and 96% in term of precision. The second experimental study, carried out on the ChaLearn LAP ConGD data set, also showed a superiority of our method from the point of view of precision by codelivery over other methods with mean Jaccard Index of 0.5011.