Exploration of Chinese Sign Language Recognition Using Wearable Sensors Based on Deep Belief Net

Abstract

In this paper, deep belief net (DBN) was applied into the field of wearable-sensor based Chinese sign language (CSL) recognition. Eight subjects were involved in the study, and all of the subjects finished a five-day experiment performing CSL on a target word set consisting of 150 CSL subwords. During the experiment, surface electromyography (sEMG), accelerometer (ACC), and gyroscope (GYRO) signals were collected from the participants. In order to obtain the optimal structure of the network, three different sensor fusion strategies, including data-level fusion, feature-level fusion, and decision-level fusion, were explored. In addition, for the feature-level fusion strategy, two different feature sources, which are hand-crafted features and network generated features, and two different network structures, which are fully-connected net and DBN, were also compared. The result showed that feature level fusion could achieve the best recognition accuracy among the three fusion strategies, and feature-level fusion with network generated features and DBN could achieve the best recognition accuracy. The best recognition accuracy realized in this study was 95.1% for the user-dependent test and 88.2% for the user-independent test. The significance of the study is that it applied the deep learning method into the field of wearable sensors-based CSL recognition, and according to our knowledge it's the first study comparing human engineered features with the network generated features in the correspondent field. The results from the study shed lights on the method of using network-generated features during sensor fusion and CSL recognition.