Turkish Sign Language recognition using spatio-temporal features on Kinect RGB video sequences and depth maps

Abstract

This paper presents a Turkish Sign Language recognition system that uses spatio-temporal features on Kinect sensor RGB video sequences and depth maps. Proposed system uses cumulative motion images which based on motion differences and represent the temporal characteristics of dynamic signs in motion sequences. Cumulative motion images represent the whole motions of signers. 2-D Discrete Cosine Transform (DCT) is applied to cumulative sign images in order to obtain spatial features of signs and transformed images that represent the energy density of signs are obtained. Two transform images are obtained by applying referred methods to both of RGB video sequences and depth maps seperately. Feature vectors of dynamic signs are produced by combining a certain amount of DCT coefficients that contain higher energy via zig-zag scanning on transform images. K-Nearist Neighbor classifier with Manhattan distance used for recognition process. System performance is evaluated on a sign database that contains 1002 signs belongs to 111 words in three different categories of Turkish Sign Language (TID). Proposed sign language recognition system has a recognition rate about %90.