MDS

Abstract

This paper describes MDS: a Multimodal-based Dialog System that supports communication between the hearing impaired and hearing-abled. The system converts sign language to speech, and combines speech with gesture and lip motion using a human face. The features of the human face are derived by doing a 3D feature extraction of the speaker's face, so that the “virtual face” similar to the actual speaker. The main technologies associated with the system include sign language recognition, sign language synthesis and the synchrony of the lip movement and speech. Integration of the sign language recognition, sign language synthesis, speech recognition, speech synthesis and 3D virtual human technologies provides a new way to interact with computers for the hearing impaired.