MULTIMODAL INTEGRATION OF MOTOR IMAGERY-BASED SIGNATURES FOR NEURAL RESPONSE CLASSIFICATION

Abstract

Decoding motor imagery (MI) brain responses from multimodal neural information sources, including electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS), has the transformative potential to advance brain-computer interface (BCI) research. To date, these systems fall short of their users’ expectations due to their modest performance improvement attributed to the lack of computational frameworks that exploit the discriminative electrical-vascular features both on unimodal and multimodal levels. The major goal of this dissertation is to propose a multimodal EEG-fNIRS data fusion framework to decode MI neural responses for an optimized BCI classification performance. We first investigate the feasibility of relying solely on EEG to control BCI systems for ALS patients. We explore the spatio-spectral-temporal dynamics of the EEG sensorimotor oscillations during MI for ALS and healthy controls using state-of-the-art neural signal analysis techniques including time-frequency based decompositions, and topographic correlation analysis. Our findings revealed potential disease-specific alterations in MI electrophysiological responses for ALS highlighting the importance of investigating alternative neuroimaging modalities in BCI research. Then, we investigate the ability of patients with ALS, to utilize fNIRS-based hemodynamic responses to efficiently control an MI-based BCI for the first time. We quantify subject-specific spatio-temporal characteristics of ALS patients' MI hemodynamic responses, and investigate the feasibility of using these responses as a means of communication. For this purpose, the generalized linear model (GLM) analysis is conducted to statistically estimate and evaluate individualized spatial activation and selected channel sets are statistically optimized for classification. Subject-specific discriminative features, and optimized classification parameters are identified and used to further evaluate the performance using various classification methods including linear support vector machine (SVM) classifier. Our findings indicate a promising application of fNIRS-based MI hemodynamic responses to control a binary BCI by ALS patients and highlight the primary role of subject-specific data-driven approaches for an optimized BCI performance. Finally, we propose a multimodal data fusion framework to decode MI neural responses that expands the information content beyond single modalities, then adopts a fused feature selection strategy to identify the most discriminative fused features. We hypothesize that exploiting the nonlinear dynamics underlying the MI neural response complements the traditionally combined EEG-fNIRS features for an enhanced hBCI performance. The nonlinear dynamics underlying the MI responses are quantified by extracting graph-based recurrence quantification analysis (RQA) features to complement the classical spectral EEG features and statistical fNIRS features. The high-dimensional multimodal features are given to a feature selection algorithm that relies on least absolute shrinkage