CNN Based Feature Extraction for Visual Speech Recognition in Malayalam

Abstract

Abstract Visual speech recognition is the technique of recognizing speech by using visual cues obtained during speech. The current research in this area makes use of visual cues such as mouth or lip movement for the recognition of speech. This paper introduced a method of visual speech recognition from the face by giving importance to the facial expressions while a person speaking. Facial expression is an important feature for hearing impaired speech recognition because they are more expressive while speaking. As an initial study in this area, we have introduced a Malayalam audio-visual speech emotion database of hearing people. The experimental studies in this database prove that facial expressions play a pivotal role in visual speech recognition.KeywordsAudio visual speech recognitionConvolutional neural networkHearing impairedVisual speech recognition