Detecting slow eye movements with bimodal-LSTM for recognizing drivers’ sleep onset period

Abstract

Slow eye movements (SEMs) indicate sleep onset period but are rarely studied in the field of driver fatigue detection. Through visual observation and statistical analysis we find that SEMs occur during eye-closed periods and have a high overall coverage rate during driving, which makes it feasible to utilize detecting SEMs to recognize drivers’ sleep onset period. To detect SEMs, we adopt a bimodal Long Short-Term Memory (LSTM) network to deal with temporal information and multimodal information in physiological signals. To extend the distinguishing information, we define a novel horizontal sum (HSUM) signal, which is sum of signals from two horizontal electrooculogram (EOG) channels. Electroencephalogram (EEG)-related features are extracted from the HSUM signal in contrast to those from the traditional O2 signal. EOG features are extracted from the horizontal EOG (HEOG) signals. The results demonstrate that features from multimodal signals (HSUM and HEOG signals, or O2 and HEOG signals) achieve better classification results than features from the single HEOG signal. And the EEG-related features extracted from self-defined HSUM signals achieve comparable results to those from traditional O2 signals, thus avoiding using extra O2 channel. The proposed method of detecting SEMs using the bimodal LSTM to classify features from HSUM and HEOG signals achieves the average F-score of 76.5%, which is higher than the classic support vector machine by 7.5%. This method of detecting SEMs using only two channels helps to build a user acceptable and feasible system for recognizing drivers’ sleep onset period.