Static and Dynamic Hand Gesture Recognition in Depth Data Using Dynamic Time Warping

Abstract

This paper discusses the development of a natural gesture user interface that tracks and recognizes in real time hand gestures based on depth data collected by a Kinect sensor. The interest space corresponding to the hands is first segmented based on the assumption that the hand of the user is the closest object in the scene to the camera. A novel algorithm is proposed to improve the scanning time in order to identify the first pixel on the hand contour within this space. Starting from this pixel, a directional search algorithm allows for the identification of the entire hand contour. The $k$ -curvature algorithm is then employed to locate the fingertips over the contour, and dynamic time warping is used to select gesture candidates and also to recognize gestures by comparing an observed gesture with a series of prerecorded reference gestures. The comparison of results with state-of-the-art approaches shows that the proposed system outperforms most of the solutions for the static recognition of sign digits and is similar in terms of performance for the static and dynamic recognition of popular signs and for the sign language alphabet. The solution simultaneously deals with static and dynamic gestures as well as with multiple hands within the interest space. An average recognition rate of 92.4% is achieved over 55 static and dynamic gestures. Two possible applications of this work are discussed and evaluated: one for interpretation of sign digits and gestures for a friendlier human-machine interaction and the other one for the natural control of a software interface.