SleepPrintNet: A Multivariate Multimodal Neural Network Based on Physiological Time-Series for Automatic Sleep Staging

Abstract

Sleep is one of the most fundamental physiological activities of human beings. Sleep assessment based on physiological time-series can efficiently assist human experts to diagnose the sleep health of people. However, most of the existing methods only considered one or two kinds of time-domain, frequency-domain, and spatial-domain information from electroencephalogram (EEG). Besides, existing deep learning methods share the feature extraction module of EEG with other modalities, which ignore the discriminative features of electrooculogram (EOG) and electromyography (EMG). Therefore, how to make full use of the complementarity of different features of EEG and capture the discriminative features from other modalities is challenging. To tackle this challenge, we design SleepPrintNet to capture the SleepPrint in physiological time-series, which represents the complementarity among different features of EEG and discriminative features from other modalities in different sleep stages. SleepPrintNet consists of an EEG temporal feature extraction module, an EEG spectral-spatial feature extraction module for the temporal-spectral-spatial representation of EEG signals, and two multimodal feature extraction modules including EOG and EMG feature extraction module. To the best of our knowledge, it is the first attempt to integrate EEG temporal-spectral-spatial as well as the multimodal features simultaneously in a unified model for sleep staging. Experiments on the benchmark dataset MASS-SS3 demonstrate that SleepPrintNet outperforms all baseline models. The implementation code of SleepPrintNet is available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/xiyangcai/SleepPrintNet</uri> . <p xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><i>Impact Statement–</i> Sleep staging helps sleep experts assess sleep quality and diagnose sleep health. The polysomnography, which contains the physiological signal recordings during sleep, is a kind of multimodal multivariant physiological time-series for sleep staging. However, existing methods treat physiological time-series from different parts of the body equally, which ignore the abundant information in multimodal signals. The SleepPrintNet proposed in this article has improved the accuracy of sleep staging with the help of the discriminative characteristics from different physiological time-series, which is made up of several independent modules for the extraction of different modalities signals. SleepPrintNet is also a universal framework for the classification of multivariate and multimodal signals. It can be applied in the diagnosis and treatment of other diseases based on physiological signals. The high-accuracy classification of physiological signals has great significance in the field of intelligent medical diagnostics.