Lip Reading Using Various Deep Learning Models with Visual Turkish Data

Abstract

In the Human-Computer Interaction field, lip reading is essential and still an open research problem. In the last decades, there have been many studies in the field of Automatic Lip-Reading (ALR) in different languages which is important for societies where the essential applications developed. Similarly to other machine learning and artificial intelligence applications, Deep Learning (DL) based classification algorithms have been applied for ALR in order to improve the performance of ALR. In the field of ALR, few studies have been done on the Turkish language. In this study, firstly an original data set was provided. Also, three image data augmentation techniques, which are sigmoidal transform, horizontal flip, and inverse transform, were applied to increase the data quality and variety. Then three deep learning models: Convolutional Neural Networks (CNN), Long-Short Term Memory (LSTM), and Bidirectional Gated Recurrent Unit (BGRU), were performed with a visual Turkish lip reading dataset. The performance of the applied method has been compared regarding precision, recall, and F1 metrics. According to experiment results, BGRU and LSTM models gave the same results up to the fifth decimal, and BGRU had the fastest training time.