Advancing classroom fatigue recognition: A multimodal fusion approach using self-attention mechanism

Abstract

This paper presents a novel multimodal tensor fusion network based on the self-attention mechanism (MTFN-SAM) for classroom fatigue recognition. The LSTM network is first employed to learn the fatigue information of different modalities over time. Then, a self-attention mechanism is utilized to capture the connections within the modal features. Finally, a multimodal tensor fusion layer is employed to fuse features from multiple modalities, including electroencephalogram (EEG), heart rate (HR), galvanic skin response (GSR), and acceleration (ACC). The classification accuracy of the experimental results was compared with TFN, LWF, LSTM, and other fusion algorithms, confirming that the proposed MTFN-SAM network had better performance in multimodal feature fusion. Additionally, we found that as the number of modalities increased, the information contained in the multimodal fusion became more abundant, as confirmed by the experimental results (94.11% accuracy after EEG+HR+GSR+ACC fusion). Thus, the method could efficiently identify students’ classroom fatigue.