Abstract
Non-invasive brain-computer interfaces (BCIs) have been widely used for neural decoding, linking neural signals to control devices. Hybrid BCI systems using electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) have received significant attention for overcoming the limitations of EEG- and fNIRS-standalone BCI systems. However, most hybrid EEG-fNIRS BCI studies have focused on late fusion because of discrepancies in their temporal resolutions and recording locations. Despite the enhanced performance of hybrid BCIs, late fusion methods have difficulty in extracting correlated features in both EEG and fNIRS signals. Therefore, in this study, we proposed a deep learning-based early fusion structure, which combines two signals before the fully-connected layer, called the fNIRS-guided attention network (FGANet). First, 1D EEG and fNIRS signals were converted into 3D EEG and fNIRS tensors to spatially align EEG and fNIRS signals at the same time point. The proposed fNIRS-guided attention layer extracted a joint representation of EEG and fNIRS tensors based on neurovascular coupling, in which the spatially important regions were identified from fNIRS signals, and detailed neural patterns were extracted from EEG signals. Finally, the final prediction was obtained by weighting the sum of the prediction scores of the EEG and fNIRS-guided attention features to alleviate performance degradation owing to delayed fNIRS response. In the experimental results, the FGANet significantly outperformed the EEG-standalone network. Furthermore, the FGANet has 4.0% and 2.7% higher accuracy than the state-of-the-art algorithms in mental arithmetic and motor imagery tasks, respectively.
Highlights
T HE brain-computer interfaces (BCIs) provide a direct interface between neural activities containing the user’s intention and control signals for external devices [1]–[3]
We proposed the functional near-infrared spectroscopy (fNIRS)-guided attention network (FGANet) as a deep learning-based early fusion structure
In the FGA layer, fNIRS features were used to create the FGA map that identifies the important regions of the EEG features for reliable EEG decoding
Summary
T HE brain-computer interfaces (BCIs) provide a direct interface between neural activities containing the user’s intention and control signals for external devices [1]–[3]. In non-invasive BCI systems, various neural activities have been utilized, such as electroencephalography (EEG) [5], [6], [9], [10], magnetoencephalography (MEG) [11], [12], functional near-infrared spectroscopy (fNIRS) [13]– [16], and functional magnetic resonance imaging (fMRI) [17], [18]. Despite MEG and fMRI having excellent spatial resolution to study the underlying neuronal activities and cerebral blood flow changes, they are inappropriate for real-world BCI systems because of their large size and high cost. EEG- or fNIRS-based BCI systems have been extensively investigated
Sign-in/Register to view highlights & summary 
Published Version (
Free)
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a callMore From: IEEE transactions on neural systems and rehabilitation engineering : a publication of the IEEE Engineering in Medicine and Biology Society
Disclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.
