Dual-flow Spatio-temporal Separation Network for Lip Reading

Abstract

Lip reading is a task of predicting the corresponding language information in a silent video, which has attracted a lot of attention in recent years. Its key is to capture temporal and spatial features from lip motion videos and decode them. In the past, lip reading methods based on deep learning mostly adopt the form of spatio-temporal series connection, which first extracts spatial features, and then carries out global time-domain modeling on this basis. The spatial information extracted by the current approach is insufficient. To get more abundant spatio-temporal video representation and fully integrate the features from different viewpoints, this paper proposes a novel lip motion feature extraction framework, Dual-flow Spatio-temporal Separation Network (DSSN). Specifically, we adopt an end-to-end double tower structure to model the temporal information and spatial information respectively, and carry out feature fusion through collaborative learning. Finally, we evaluate our proposed model on the OuluVS2 lip reading dataset. Experiments show that our method outperforms baseline models.