Automatic translation of sign language with multi-stream 3D CNN and generation of artificial depth maps

Abstract

Sign languages play an essential role in the cognitive and social development of the deaf, consisting of a natural form of communication and being a symbol of identity and culture. However, hearing loss has a severe social impact due to an existing communication barrier, preventing access to essential services such as education and health. A bi-directional sign language translation may be the solution to bridging the communication gap between the deaf and the listener, completing a two-way communication cycle. Virtual personal assistants can benefit from this technology by extending how users interact with the intelligent system. With this idea, in this work we develop a multi-stream deep learning model to recognize signs of Brazilian (BSL), Indian (ISL), and Korean (KSL) Sign Languages. We combine different types of information for the classification task, using single-stream and multi-stream 3D Convolutional Neural Networks. In addition, considering the largest source of sign data globally – the internet – we propose a depth sensor-free classification method, with depth maps artificially generated through Generative Adversarial Networks. In order to consider the main parameters that encode sign languages, the final architecture is composed of a multi-stream network that receives the segmented hands, the faces, the distances and speeds of the points of articulation, and the RGB frames associated with artificial depth maps. Finally, we provide a visual explanation to understand which regions were important for model decision-making. The best models were obtained using the multi-stream network, presenting an accuracy of 0.91 ± 0.07, and f1-score of 0.90 ± 0.08 on publicly available BSL data set. The results suggest that the multi-stream network with artificially generated depth maps is suitable for the task of sign recognition in different languages.