Vision-based gesture modeling for signed expressions recognition

Abstract

A vision-based method for signed expressions recognition consisting of searching for previously prepared word models in the analyzed utterances was developed. The models were created as temporal sequences of grouped descriptors of the place of articulation and the shape of the hand. The descriptors were created using the skeletons determined by the deep neural network and histogram of oriented gradients features. The classification was carried out using the Wagner–Fischer algorithm for determining Levenshtein distance, which was modified to search for fragments in sequences of discrete symbols. Considering the practical application of the developed method, even in the case of partially inaccurate recognition results, it was proposed to visualize the answers in the form of a word cloud. The method was tested for a challenging set of sign language expressions used in the office when applying for an ID card.