Abstract
Abstract. Visual information plays a key role in automatic speech recognition (ASR) when audio is corrupted by background noise, or even inaccessible. Speech recognition using visual information is called lip-reading. The initial idea of visual speech recognition comes from humans’ experience: we are able to recognize spoken words from the observation of a speaker's face without or with limited access to the sound part of the voice. Based on the conducted experimental evaluations as well as on analysis of the research field we propose a novel task-oriented approach towards practical lip-reading system implementation. Its main purpose is to be some kind of a roadmap for researchers who need to build a reliable visual speech recognition system for their task. In a rough approximation, we can divide the task of lip-reading into two parts, depending on the complexity of the problem. First, if we need to recognize isolated words, numbers or small phrases (e.g. Telephone numbers with a strict grammar or keywords). Or second, if we need to recognize continuous speech (phrases or sentences). All these stages disclosed in detail in this paper. Based on the proposed approach we implemented from scratch automatic visual speech recognition systems of three different architectures: GMM-CHMM, DNN-HMM and purely End-to-end. A description of the methodology, tools, step-by-step development and all necessary parameters are disclosed in detail in current paper. It is worth noting that for the Russian speech recognition, such systems were created for the first time.
Highlights
The initial idea of visual speech recognition comes from humans’ experience: we are able to recognize spoken words from the observation of a speaker's face without or with limited access to the sound part of the voice
Visual information plays a key role in automatic speech recognition (ASR) when audio is corrupted by background noise, or even inaccessible
In this paper we present the developed task-oriented approach for creating practical visual speech recognition systems
Summary
The initial idea of visual speech recognition comes from humans’ experience: we are able to recognize spoken words from the observation of a speaker's face without or with limited access to the sound part of the voice. In quiet office environments, for a variety of tasks speech recognition can approach almost hundred percent of accuracy It is often achieved under the condition of a limited vocabulary and a stricted grammar. Since the early 90s, there have been several attempts to use visual information about speech in addition to acoustic information, to improve the accuracy and reliability of automatic recognition systems. In a number of studies, the developed audio-visual speech recognition systems have demonstrated better. There is little research on the effect of acoustically noisy environments on the performance of visual speech recognition systems, and quite a few studies have focused on inflectional languages (such as Russian). There is a huge difference between the recognition of analytical languages (for example, English) and inflected languages, due to the presence in the latter of a much larger number of word forms and grammatical rules
Sign-in/Register to view highlights & summary Sign-in/Register to access full text options 
Free) 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a call
