Data-driven learning fatigue detection system: A multimodal fusion approach of ECG (electrocardiogram) and video signals

Abstract

Fatigue could lead to low efficiency and even serious disaster. In the educational field, detecting fatigue could help adjust teaching strategies accordingly when a student is inactive, which can potentially improve learning efficiency. Despite numerous studies in fatigue detection, there is still a lack of multiple classifier systems capable of detecting fatigue in daily life (without specific stimulations). To initially alleviate this problem, this study develops a learning fatigue detection system using a multimodal approach with ECG and video signals, classifying a learner’s state into three categories: alert, normal, and fatigued. To validate performance, the proposed system is tested on (i) an open-source dataset DROZY (n = 35) and (ii) a self-collected dataset captured in a learning environment (n = 92). The experimental results based on 10-fold cross-validation demonstrate that the system outperforms the state-of-the-art approaches, achieving a detection accuracy of 99.6% and 91.8% on the two datasets, respectively.