Using Multimodal Bio-Signals for Prediction of Physiological Cognitive State Under Free-Living Conditions

Abstract

Drowsiness is a common human physiological response. Research suggests that insufficient sleep results in low energy levels, various negative physiological and psychological effects on the body, and abnormal cognitive functioning. Studies have predominantly focused on driving while drowsy, using brainwave measurement and facial detection techniques to address this topic, whereas few have discussed the physiological prediction of drowsiness. In addition to driving, working conditions and environments as well as daily activities also correspond to the risk of accidents occurring when people are drowsy. This study designed an experiment consisting of five tests in which a brainwave sensor, eye tracker, heart rate sensor, and galvanic skin response sensor were used to record physiological changes in participants. The data indicated the binary outcomes of patients either being or not being in a state of drowsiness or sleepiness. During various states of drowsiness or sleepiness, brain wave activity, eye movement, heart rate, and GSR were measured, and differences in these physiological responses were analyzed. A classification model was also used to predict a participant’s state of drowsiness or sleepiness. Data on physiological characteristics included eye movement and the heart rate value, which was calculated during various states of drowsiness or sleepiness to obtain a value. Brain wave and GSR signals were converted through software development kit (SDK) programming at the sensor end. Subsequently, the data were processed through an artificial neural network (ANN), back propagation network, and support vector machine (SVM). In the experiment, the SVM hyperparameters were adjusted, the ANN model was added to the Adam optimization model, and overfitting was avoided to ensure that the results were comprehensive. According to the experiment results, the use of SVM yields the optimal classification performance, reaching an accuracy of 89.1%; 90% of participants were also categorized more accurately through SVM than ANN.