Towards Realizing Sign Language to Emotional Speech Conversion by Deep Learning

Abstract

This paper proposes a framewrok for realizing sign language to emotional speech conversion by deep learning. We firstly adopt a deep belief network (DBN) model to extract the features of sign language and a deep neural network (DNN) to extract the features of facial expression. Then we train two support vector machines (SVM) to classify the sign language and facial expression for recognizing the text of sign language and emotional tags of facial expression. We also train a set of DNN-based emotional speech acoustic models by speaker adaptive training with an multi-speaker emotional speech corpus. Finally, we select the DNN-based emotional speech acoustic models with emotion tags to synthesize emotional speech from the text recognized from the sign language. Objective tests show that the recognition rate for static sign language is 92.8%. The recognition rate of facial expression achieves 94.6% on the extended Cohn-Kanade database (CK+) and 80.3% on the JAFFE database respectively. Subjective evaluation demonstrates that synthesized emotional speech can get 4.2 of the emotional mean opinion score. The pleasure-arousal-dominance (PAD) evaluation shows that the PAD values of facial expression are close to the PAD values of synthesized emotional speech.