Signal models for fusion of multiple brain responses and context evidence sources in EEG-based BCIs

Abstract

Amyotrophic lateral sclerosis is a progressive neuro-degenerative condition that results in gradual loss of communication between the brain and the body. In late stages, these individuals have no ability to move or speak, but they may retain normal cognitive function. The individual is conscious and cognitively intact to be able to communicate, but lack of muscle control is a serious barrier. This condition, referred to as "locked-in syndrome" (LIS), is encountered in patients who have various other conditions, including spinal cord injury and cerebral palsy. Brain-computer interfaces (BCIs) can provide them with alternative communication channels, independent of muscle control. This technology can be especially important for individuals with severe neuromuscular diseases who cannot use standard communication pathways or other assistive technology. In our research, we study how people can use electrical brain activity acquired on the scalp in a non-invasive fashion, known as electroencephalogram (EEG), to communicate with people and their environments. By detecting specific features of the EEG, we can translate brain activity into actions on a virtual keyboard. While some BCI typing systems have shown encouraging results, there is still much work to be done to produce real-world-worthy systems that can be comfortably, conveniently, and reliably used by individuals with LIS. This work presents several improvements to the RSVP Keyboard™, as well as similar designs that depend on visually evoked potentials. The baseline system fuses text/language and EEG evidence to infer user intent in EEG-controlled spelling to generate expressive language. First, we study the benefits of fusing feedback related potentials, a form of error-related potentials (ErrP), with event-related potentials (ERP) and context information (language model) in a Bayesian fashion. Specifically, we perform Bayesian fusion of ERP, ErrP, and non-EEG evidence using different probabilistic generative models. We present results from EEG data acquired from healthy participants using RSVP Keyboard™ to complete a copy-phrase task. The results demonstrate that Bayesian fusion of these three types of evidence (ErrP, ERP, context information) yields faster speeds without compromising accuracy. Second, we study Gaussian Process models for multichannel EEG in the context of non-invasive BCIs. The relatively high dimensionality of EEG feature vectors with respect to the number of calibration trials leads to rank deficient covariance matrix estimates for multivariate Gaussian, class-conditioned, feature density models. While multivariate EEG time-series can be assumed to be a Gaussian process with unknown mean and autocovariance sequences, typically BCIs ignore the structure imposed by the time-series nature when estimating parameters. In this work, we build spatio-temporal signal models for EEG and show that under certain assumptions these models lead to a Kronecker product structure for said covariance matrices. We demonstrate that imposing this structure on covariance matrices improves estimation accuracy and consequently BCI performance. Finally, previous works assumed that trial EEG-evidence from each trial is statistically independent, which is clearly inaccurate at least due to the overlapping time windows that are used for trial-EEGevidence extraction. Considering the temporal dependency of brain responses to serial stimuli can improve the system performance so we propose a generative signal model that captures the temporal dependency of EEG signals evoked with rapid sequential stimulation. The model describes the EEG evoked by a sequence of stimuli as a superposition of impulse responses time-locked to stimuli, corrupted by an autoregressive noise process. EEG data obtained for model calibration from healthy participants is used to fit and compare two models: the proposed sequence based EEG model and the trial-based feature-class-conditional distribution model that ignores temporal dependencies. Results from the experimental data indicate that the new model that includes temporal dependencies results in important improvements in information transfer rate (ITR).