Data Augmentation Using Feature Interpolation of Individual Words for Compound Word Recognition of Sign Language

Abstract

Sign language recognition is an important topic to improve communication between native signers and native speakers. In this paper, we propose one of the data augmentations to recognize compound words represented by sequences of two individual sign language words. The proposed method generates two-dimensional tracking points of the compound words by concatenating the two individual sign language words’ tracking points. Linear interpolation is applied to concatenate the tracking points of the individual sign language words. The evaluation was conducted using a conventional sign feature extraction and recognition model based on the encoder-decoder Recurrent Neural Network with Attention. The minimum average word error rate of the model, which trained with the generated compound words, was 18.85%. Moreover, the minimum average word error rate of the model, which trained with the generated and recorded compound words, was 7.15%, and it was improved by 3.08% from that of the recorded compound words.