TFormer: A time–frequency Transformer with batch normalization for driver fatigue recognition

Abstract

Within the framework of the advanced human-cybernetic interfaces (HCI), Cross-subject electroencephalogram (EEG)-based driver fatigue recognition is emerging as a pivotal application in the paradigm of Industry 5.0. Recognizing the importance of ensuring driver safety through proactive monitoring, it is essential to offer a general EEG decoding system to improve road safety. This work investigated the use of Transformers for the challenging cross-subject EEG decoding task due to the great success the Transformers have achieved in various applications. Previous research focused on using Transformers to capture global temporal information, but less work targeted global frequency-domain patterns. Furthermore, in order to leverage a standard Transformer architecture grounded in natural language processing for EEG decoding, it is imperative to account for inherent characteristics in EEG and make pertinent adjustments accordingly. In this work, we proposed a time–frequency Transformer (TFormer) that can automatically learn the global time–frequency patterns from raw EEG data. TFormer consisted of three components: convolutional stems for input embedding, time–frequency multi-head cross-attention (TF-MCA) for integrating time-domain patterns into frequency points, and self-attention to further learn global time-frequency patterns. Moreover, we analyzed TFormer’s internal settings and found batch normalization (BN) more suitable for cross-subject EEG decoding than layer normalization (LN). The experiment results demonstrated the superiority of the proposed model compared to existing methods. Overall, our work contributes to the development of Transformer models in EEG decoding and illustrates a different way to leverage Transformers for decoding raw EEG data.