TMS-Net: A multi-feature multi-stream multi-level information sharing network for skeleton-based sign language recognition

Abstract

Sign Language Recognition (SLR) is an increasingly popular research topic due to its extensive potential applications, such as education, healthcare, emergency response, and social interaction. Sign language is a complex and dynamic language comprising hand gestures, facial expressions, and body motions. This high level of variability poses a significant obstacle for SLR tasks, which must accurately identify and respond to numerous gestures. To address these challenges, an end-to-end skeleton-based multi-feature multi-stream multi-level information sharing network (three multi information sharing network (TMS-Net)) is proposed. Specifically, in order to input more rich information to TMS-Net, we use joint feature pair with global features, bone feature pair with local features, and angle feature pair with scale invariance. In terms of network structure, to efficiently extract multiple features from inputs, we build a multi-stream structure and design a multi-level information sharing mechanism based on this structure to ensure the full utilization of skeleton feature information. From the experiment results of the WLASL-2000 dataset (56.4%), AUTSL dataset (96.62%) and MSASL(65.13%), TMS-Net surpass the state-of-the-art (SOTA) methods with single modality as input. In addition, a SLR-based human–robot interaction (HRI) experiment using our proposed TMS-Net is conducted, which proves the practical performance of the TMS-Net.