Neural Decoding of Chinese Sign Language With Machine Learning for Brain-Computer Interfaces.

Pengpai Wang,Daoqiang Zhang,Zhongnian Li,Shuo Huang,Yueying Zhou

doi:10.1109/tnsre.2021.3137340

Abstract

Limb motion decoding is an important part of brain-computer interface (BCI) research. Among the limb motion, sign language not only contains rich semantic information and abundant maneuverable actions but also provides different executable commands. However, many researchers focus on decoding the gross motor skills, such as the decoding of ordinary motor imagery or simple upper limb movements. Here we explored the neural features and decoding of Chinese sign language from electroencephalograph (EEG) signal with motor imagery and motor execution. Sign language not only contains rich semantic information, but also has abundant maneuverable actions, and provides us with more different executable commands. In this paper, twenty subjects were instructed to perform movement execution and movement imagery based on Chinese sign language. Seven classifiers are employed to classify the selected features of sign language EEG. L1 regularization is used to learn and select features that contain more information from the mean, power spectral density, sample entropy, and brain network connectivity. The best average classification accuracy of the classifier is 89.90% (imagery sign language is 83.40%). These results have shown the feasibility of decoding between different sign languages. The source location reveals that the neural circuits involved in sign language are related to the visual contact area and the pre-movement area. Experimental evaluation shows that the proposed decoding strategy based on sign language can obtain outstanding classification results, which provides a certain reference value for the subsequent research of limb decoding based on sign language.

Highlights

To verify that the sign language-based limb movement can be accurately classified and provide reliable and rich instruction set for brain-computer interface (BCI), we introduce a robust model and select the sign language EEG features to achieve high classification accuracy
We adopt the support vector machine (SVM), linear discriminant analysis (LDA), long short term memory network (LSTM), EEGNet, dynamic graph convolution network (DGCN), spiking neural network (SNN), and dendritic morphological neural network (DMNN) classifiers based on L1 regularization to learn and select relevant features, to obtain effective sparse features and high classification accuracy
We introduce SVM, LDA, LSTM, EEGNet, DGCN, SNN, and DMNN classifiers to learn the features of executive sign language

Summary

Introduction

Zhang are with the College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics, MIIT Key Laboratory of Pattern Analysis and Machine Intelligence, Nanjing 211106, China

Objectives

Methods

Results

Discussion

Conclusion