Audio–Visual (Multimodal) Speech Recognition System Using Deep Neural Network

Abstract

Abstract Recognizing the voice of a person by a smart device has become the need of the hour. Our cell phones are equipped with such features that converts speech to text, receives a command as a voice instruction instead of screen touch or pressing of a button. There are numerous intelligent machines such as automatic cars, robots, smart phones that use speech recognition features. These features are also very useful for people who are disabled and cannot use the smart devices with their hands. There are numerous techniques available to implement a speech recognition system. The need of such a system is to accurately recognize human voice at any circumstances, because the accent of each human differs for a single language, and also, the human voice is never devoid of noise and other emotional elements. So, the aim of such systems is to recognize human voices of different accents, even in a noisy environment. Because of this, an effective and robust speech recognition system has become important for machine and human interaction. Much voice-recognizing software depends purely on audio to recognize a voice, and one cannot guarantee that the audio will be noise free. In order to improve the accuracy and performance of traditional speech recognition system in a noisy environment, a new, nontraditional approach called a multimodal speech recognition system that uses both the audio and video features such as lip movement facial expression to recognize a speech is used. This article discusses the advantage of using video features along with the audio features with an experiment that proves it.