A framework for live and cross platform fingerspelling recognition using modified shape matrix variants on depth silhouettes

Abstract

Automatic recognition of fingerspelling postures in a live environment is a challenging task primarily due to the complex computation of popular moment-based and spectral descriptors. Shape matrix offers a time-efficient alternative that samples the shape region through the intersection points of adjacent log-polar sections. However, sparse sampling of the region by discrete log-polar intersection points cannot capture salience of the shape. This manuscript proposes modified forms of the shape matrix which can capture salience of the fingerspelling postures by the precise sampling of contours and regions. For effective segmentation and subsequent description, hand postures are acquired through the depth sensor. Proposed shape matrix variants are evaluated for fingerspelling recognition with one-handed and two-handed postures. Experiments are rigorously performed on three datasets including one-handed signs of American Sign Language (ASL), NTU hand digits, and both one-handed and two-handed signs of Indian Sign Language (ISL). Proposed shape matrix variants supersede the benchmark shape context and Gabor features by obtaining 94.15% accuracy on ISL dataset with minimum mean running time of 0.029 s. On ASL and NTU datasets, 91.86% and 95.11% accuracies are obtained with 0.0172 and 0.0483 s mean running times, respectively.