Visual Speech Recognition with Lightweight Psychologically Motivated Gabor Features.

Xuejie Zhang,Yan Xu,Roger Watt,Andrew K Abel,Leslie S Smith,Amir Hussain,Chengxiang Gao

doi:10.3390/e22121367

Abstract

Extraction of relevant lip features is of continuing interest in the visual speech domain. Using end-to-end feature extraction can produce good results, but at the cost of the results being difficult for humans to comprehend and relate to. We present a new, lightweight feature extraction approach, motivated by human-centric glimpse-based psychological research into facial barcodes, and demonstrate that these simple, easy to extract 3D geometric features (produced using Gabor-based image patches), can successfully be used for speech recognition with LSTM-based machine learning. This approach can successfully extract low dimensionality lip parameters with a minimum of processing. One key difference between using these Gabor-based features and using other features such as traditional DCT, or the current fashion for CNN features is that these are human-centric features that can be visualised and analysed by humans. This means that it is easier to explain and visualise the results. They can also be used for reliable speech recognition, as demonstrated using the Grid corpus. Results for overlapping speakers using our lightweight system gave a recognition rate of over 82%, which compares well to less explainable features in the literature.

Highlights

The brain handles both auditory and visual speech information
Motivated by psychological research into how humans recognise faces, we present Gabor-based lip feature extraction
This paper extends an initial conference paper presented at IEEE SSCI 2019 [17] by demonstrating that as well as being able to visualise the features, we can use these simple and lightweight features for visual speech recognition

Summary

Introduction

The brain handles both auditory and visual speech information. Visual information was shown to be able to influence interpretation, as demonstrated by the McGurk Effect [1]. Some lipreading systems use deep learning methods, and are end-to-end systems not separated into two stages, instead relying on data-intensive pre-trained models, and lacking clearly defined explanatory features. As well as a variety of network topologies, datasets, and tasks, the different approaches summarised in Table 1 have different training and test conditions. Other research uses the BBC program based corpora, meaning that the training and test sets are divided according to broadcast date. This means that there may be some speaker overlap, depending on the dataset. This includes research by Chung and Zisserman [23,24], Chung et al [16], Stafylakis and Tzimiropoulos [26], Petridis et al [10] and Weng [27]

Objectives

Results

Discussion

Conclusion