Multimodal Fusion Convolutional Neural Network Based on sEMG and Accelerometer Signals for Intersubject Upper Limb Movement Classification

Abstract

The variation in the distributions of recorded data between individuals leads to low classification accuracy. To address this issue, we introduce a multimodal fusion convolutional neural network (MFCNN). This network extracts common information from surface electromyography (sEMG) and accelerometer signals of different subjects using a two-stream CNN. To enhance the classification accuracy of a particular subject, a fine-tuning approach was implemented. The performance of the proposed method was assessed in four different scenarios, which include inter-subject classification, inter-subject classification when training data from multiple subjects, fine-tuned inter-subject classification, and fine-tuned inter-subject classification when training data from multiple subjects. The results demonstrate that in the inter-subject scenario, when multiple subjects are available for training, the MFCNN achieves higher classification accuracy (p < 0.05) than other neural networks and support vector machines (SVMs) that use sEMG signals (NN and SVM), accelerometer signals (accNN and accSVM), sEMG and accelerometer signals (MFNN and MFSVM) as inputs, as well as a CNN that uses sEMG signals as input after fine-tuning. Furthermore, compared with an MFCNN model trained with data from a single subject and an accCNN model trained with data from a single subject or multiple subjects, an MFCNN trained with multiple subjects demonstrated better performance on new subjects after fine-tuning (p < 0.05). This method can learn common features among different subjects and improve the performance of classification among subjects. Our proposed method demonstrates the innovation of using a multimodal fusion approach and two-stream CNN to improve inter-subject classification accuracy in upper limb movements.