Passive BCI Research Articles

Modeling of anticipation using instance-based learning: application to automation surprise in aviation using passive BCI and eye-tracking data

Modeling of anticipation using instance-based learning: application to automation surprise in aviation using passive BCI and eye-tracking data

K-Fold Cross-Validation Can Significantly Over-Estimate True Classification Accuracy in Common EEG-Based Passive BCI Experimental Designs: An Empirical Investigation.

In passive BCI studies, a common approach is to collect data from mental states of interest during relatively long trials and divide these trials into shorter "epochs" to serve as individual samples in classification. While it is known that using k-fold cross-validation (CV) in this scenario can result in unreliable estimates of mental state separability (due to autocorrelation in the samples derived from the same trial), k-fold CV is still commonly used and reported in passive BCI studies. What is not known is the extent to which k-fold CV misrepresents true mental state separability. This makes it difficult to interpret the results of studies that use it. Furthermore, if the seriousness of the problem were clearly known, perhaps more researchers would be aware that they should avoid it. In this work, a novel experiment explored how the degree of correlation among samples within a class affects EEG-based mental state classification accuracy estimated by k-fold CV. Results were compared to a ground-truth (GT) accuracy and to "block-wise" CV, an alternative to k-fold which is purported to alleviate the autocorrelation issues. Factors such as the degree of true class separability and the feature set and classifier used were also explored. The results show that, under some conditions, k-fold CV inflated the GT classification accuracy by up to 25%, but block-wise CV underestimated the GT accuracy by as much as 11%. It is our recommendation that the number of samples derived from the same trial should be reduced whenever possible in single-subject analysis, and that both the k-fold and block-wise CV results are reported.

Toward passive BCI: asynchronous decoding of neural responses to direction- and angle-specific perturbations during a simulated cockpit scenario

Neuroimaging studies have provided proof that loss of balance evokes specific neural transient wave complexes in electroencephalography (EEG), called perturbation evoked potentials (PEPs). Online decoding of balance perturbations from ongoing EEG signals can establish the possibility of implementing passive brain-computer interfaces (pBCIs) as a part of aviation/driving assistant systems. In this study, we investigated the feasibility of identifying the existence and expression of perturbations in four different conditions by using EEG signals. Fifteen healthy participants experienced four various postural changes while they sat in a glider cockpit. Sudden perturbations were exposed by a robot connected to a glider and moved to the right and left directions with tilting angles of 5 and 10 degrees. Perturbations occurred in an oddball paradigm in which participants were not aware of the time and expression of the perturbations. We employed a hierarchical approach to separate the perturbation and rest, and then discriminate the expression of perturbations. The performance of the BCI system was evaluated by using classification accuracy and F1 score. Asynchronously, we achieved average accuracies of 89.83 and 73.64% and average F1 scores of 0.93 and 0.60 for binary and multiclass classification, respectively. These results manifest the practicality of pBCI for the detection of balance disturbances in a realistic situation.

Simultaneous Classification of Both Mental Workload and Stress Level Suitable for an Online Passive Brain-Computer Interface.

Research studies on EEG-based mental workload detection for a passive BCI generally focus on classifying cognitive states associated with the performance of tasks at different levels of difficulty, with no other aspects of the user’s mental state considered. However, in real-life situations, different aspects of the user’s state such as their cognitive (e.g., level of mental workload) and affective (e.g., level of stress/anxiety) states will often change simultaneously, and performance of a BCI system designed considering just one state may be unreliable. Moreover, multiple mental states may be relevant to the purposes of the BCI—for example both mental workload and stress level might be related to an aircraft pilot’s risk of error—and the simultaneous prediction of states may be critical in maximizing the practical effectiveness of real-life online BCI systems. In this study we investigated the feasibility of performing simultaneous classification of mental workload and stress level in an online passive BCI. We investigated both subject-specific and cross-subject classification approaches, the latter with and without the application of a transfer learning technique to align the distributions of data from the training and test subjects. Using cross-subject classification with transfer learning in a simulated online analysis, we obtained accuracies of 77.5 ± 6.9% and 84.1 ± 5.9%, across 18 participants for mental workload and stress level detection, respectively.

EEG-Based Auditory Attention Detection and Its Possible Future Applications for Passive BCI

The ability to discriminate and attend one specific sound source in a complex auditory environment is a fundamental skill for efficient communication. Indeed, it allows us to follow a family conversation or discuss with a friend in a bar. This ability is challenged in hearing-impaired individuals and more precisely in those with a cochlear implant (CI). Indeed, due to the limited spectral resolution of the implant, auditory perception remains quite poor in a noisy environment or in presence of simultaneous auditory sources. Recent methodological advances allow now to detect, on the basis of neural signals, which auditory stream within a set of multiple concurrent streams an individual is attending to. This approach, called EEG-based auditory attention detection (AAD), is based on fundamental research findings demonstrating that, in a multi speech scenario, cortical tracking of the envelope of the attended speech is enhanced compared to the unattended speech. Following these findings, other studies showed that it is possible to use EEG/MEG (Electroencephalography/Magnetoencephalography) to explore auditory attention during speech listening in a Cocktail-party-like scenario. Overall, these findings make it possible to conceive next-generation hearing aids combining customary technology and AAD. Importantly, AAD has also a great potential in the context of passive BCI, in the educational context as well as in the context of interactive music performances. In this mini review, we firstly present the different approaches of AAD and the main limitations of the global concept. We then expose its potential applications in the world of non-clinical passive BCI.

A passive BCI for monitoring the intentionality of the gaze-based moving object selection

Objective. The use of an electroencephalogram (EEG) anticipation-related component, the expectancy wave (E-wave), in brain–machine interaction was proposed more than 50 years ago. This possibility was not explored for decades, but recently it was shown that voluntary attempts to select items using eye fixations, but not spontaneous eye fixations, are accompanied by the E-wave. Thus, the use of the E-wave detection was proposed for the enhancement of gaze interaction technology, which has a strong need for a mean to decide if a gaze behavior is voluntary or not. Here, we attempted at estimating whether this approach can be used in the context of moving object selection through smooth pursuit eye movements. Approach. Eighteen participants selected, one by one, items which moved on a computer screen, by gazing at them. In separate runs, the participants performed tasks not related to voluntary selection but also provoking smooth pursuit. A low-cost consumer-grade eye tracker was used for item selection. Main results. A component resembling the E-wave was found in the averaged EEG segments time-locked to voluntary selection events of every participant. Linear discriminant analysis with shrinkage regularization classified the intentional and spontaneous smooth pursuit eye movements, using single-trial 300 ms long EEG segments, significantly above chance in eight participants. When the classifier output was averaged over ten subsequent data segments, median group ROC AUC of 0.75 was achieved. Significance. The results suggest the possible usefulness of the E-wave detection in the gaze-based selection of moving items, e.g. in video games. This technique might be more effective when trial data can be averaged, thus it could be considered for use in passive interfaces, for example, in estimating the degree of the user’s involvement during gaze-based interaction.

Tracing Pilots' Situation Assessment by Neuroadaptive Cognitive Modeling.

This study presents the integration of a passive brain-computer interface (pBCI) and cognitive modeling as a method to trace pilots’ perception and processing of auditory alerts and messages during operations. Missing alerts on the flight deck can result in out-of-the-loop problems that can lead to accidents. By tracing pilots’ perception and responses to alerts, cognitive assistance can be provided based on individual needs to ensure they maintain adequate situation awareness. Data from 24 participating aircrew in a simulated flight study that included multiple alerts and air traffic control messages in single pilot setup are presented. A classifier was trained to identify pilots’ neurophysiological reactions to alerts and messages from participants’ electroencephalogram (EEG). A neuroadaptive ACT-R model using EEG data was compared to a conventional normative model regarding accuracy in representing individual pilots. Results show that passive BCI can distinguish between alerts that are processed by the pilot as task-relevant or irrelevant in the cockpit based on the recorded EEG. The neuroadaptive model’s integration of this data resulted in significantly higher performance of 87% overall accuracy in representing individual pilots’ responses to alerts and messages compared to 72% accuracy of a normative model that did not consider EEG data. We conclude that neuroadaptive technology allows for implicit measurement and tracing of pilots’ perception and processing of alerts on the flight deck. Careful handling of uncertainties inherent to passive BCI and cognitive modeling shows how the representation of pilot cognitive states can be improved iteratively for providing assistance.

Toward a Hybrid Passive BCI for the Modulation of Sustained Attention Using EEG and fNIRS.

We report results of a study that utilizes a BCI to drive an interactive interface countermeasure that allows users to self-regulate sustained attention while performing an ecologically valid, long-duration business logistics task. An engagement index derived from EEG signals was used to drive the BCI while fNIRS measured hemodynamic activity for the duration of the task. Participants (n = 30) were split into three groups (1) no countermeasures (NOCM), (2) continuous countermeasures (CCM), and (3) event synchronized, level-dependent countermeasures (ECM). We hypothesized that the ability to self-regulate sustained attention through a neurofeedback mechanism would result in greater task engagement, decreased error rate and improved task performance. Data were analyzed by wavelet coherence analysis, statistical analysis, performance metrics and self-assessed cognitive workload via RAW-TLX. We found that when the BCI was used to deliver continuous interface countermeasures (CCM), task performance was moderately enhanced in terms of total 14,785 (σ = 423) and estimated missed sales 7.46% (σ = 1.76) when compared to the NOCM 14,529 (σ = 510), 9.79% (σ = 2.75), and the ECM 14,180 (σ = 875), 9.62% (σ = 4.91) groups. An “actions per minute” (APM) metric was used to determine interface interaction activity which showed that overall the CCM and ECM groups had a higher APM of 3.460 (SE = 0.140) and 3.317 (SE = 0.139) respectively when compared with the NOCM group 2.65 (SE = 0.097). Statistical analysis showed a significant difference between ECM - NOCM and CCM - NOCM (p < 0.001) groups, but no significant difference between the ECM – CCM groups. Analysis of the RAW-TLX scores showed that the CCM group had lowest total score 7.27 (σ = 3.1) when compared with the ECM 9.7 (σ = 3.3) and NOCM 9.2 (σ = 3.4) groups. No statistical difference was found between the RAW-TLX or the subscales, except for self-perceived performance (p < 0.028) comparing the CCM and ECM groups. The results suggest that providing a means to self-regulate sustained attention has the potential to keep operators engaged over long periods, and moderately increase on-task performance while decreasing on-task error.

Analyzing Passive BCI Signals to Control Adaptive Automation Devices.

Brain computer interfaces are currently considered to greatly enhance assistive technologies and improve the experiences of people with special needs in the workplace. The proposed adaptive control model for smart offices provides a complete prototype that senses an environment’s temperature and lighting and responds to users’ feelings in terms of their comfort and engagement levels. The model comprises the following components: (a) sensors to sense the environment, including temperature and brightness sensors, and a headset that collects electroencephalogram (EEG) signals, which represent workers’ comfort levels; (b) an application that analyzes workers’ feelings regarding their willingness to adjust to a space based on an analysis of collected data and that determines workers’ attention levels and, thus, engagement; and (c) actuators to adjust the temperature and/or lighting. This research implemented independent component analysis to remove eye movement artifacts from the EEG signals and used an engagement index to calculate engagement levels. This research is expected to add value to research on smart city infrastructures and on assistive technologies to increase productivity in smart offices.

Distinguishing mental attention states of humans via an EEG-based passive BCI using machine learning methods

Recent advances in technology bring about novel operating environments where the role of human participants is reduced to passive observation. While opening new frontiers in productivity and lifestyle, such environments also create hazards related to the inability of human individuals to maintain focus and concentration during passive control tasks. A passive brain-computer interface for monitoring mental attention states of human individuals (focused, unfocused, and drowsy) by using electroencephalographic (EEG) brain activity imaging and machine learning data analysis methods is developed in this work. An EEG data processing pipeline and a machine learning mental state detection algorithm using the Support Vector Machine (SVM) method were designed and compared with k-Nearest Neighbor and Adaptive Neuro-Fuzzy System methods. To collect 25 h of EEG data from 5 participants, a classic EEG headset was modified. We found that the changes in EEG activity in frontal and parietal lobes occurring at 1–5 Hz and 10–15 Hz frequency bands were associated with the changes in individuals’ attention state. We demonstrated the ability to use such changes to identify individuals’ attention state with 96.70% (best) and 91.72% (avg.) accuracy in experimental settings using a version of continuous performance task with SVM-based mental state detector. The findings help guide the design of future systems for monitoring the state of human individuals by means of EEG brain activity data.

EEG-based neglect assessment: A feasibility study

BackgroundSpatial neglect (SN) is a neuropsychological syndrome that impairs automatic attention orienting to stimuli in the contralesional visual space of stroke patients. SN is commonly assessed using paper and pencil tests. Recently, computerized tests have been proposed to provide a dynamic assessment of SN. However, both paper- and computer-based methods have limitations. New methodElectroencephalography (EEG) shows promise for overcoming the limitations of current assessment methods. The aim of this work is to introduce an objective passive BCI system that records EEG signals in response to visual stimuli appearing in random locations on a screen with a dynamically changing background. Our preliminary experimental studies focused on validating the system using healthy participants with intact brains rather than employing it initially in more complex environments with patients having cortical lesions. Therefore, we designed a version of the test in which we simulated SN by hiding target stimuli appearing on the left side of the screen so that the subject’s attention is shifted to the right side. ResultsResults showed that there are statistically significant differences between EEG responses due to right and left side stimuli reflecting different processing and attention levels towards both sides of the screen. The system achieved average accuracy, sensitivity and specificity of 74.24%, 75.17% and 71.36% respectively. Comparison with existing methodsThe proposed test can examine both presence and severity of SN, unlike traditional paper and pencil tests and computer-based methods. ConclusionsThe proposed test is a promising objective SN evaluation method.

Towards a hybrid passive BCI for the modulation of sustained attention using EEG and fNIRS

Towards a hybrid passive BCI for the modulation of sustained attention using EEG and fNIRS

Passive BCI in Operational Environments: Insights, Recent Advances, and Future Trends.

This minireview aims to highlight recent important aspects to consider and evaluate when passive brain-computer interface (pBCI) systems would be developed and used in operational environments, and remarks future directions of their applications. Electroencephalography (EEG) based pBCI has become an important tool for real-time analysis of brain activity since it could potentially provide covertly-without distracting the user from the main task-and objectively-not affected by the subjective judgment of an observer or the user itself-information about the operator cognitive state. Different examples of pBCI applications in operational environments and new adaptive interface solutions have been presented and described. In addition, a general overview regarding the correct use of machine learning techniques (e.g., which algorithm to use, common pitfalls to avoid, etc.) in the pBCI field has been provided. Despite recent innovations on algorithms and neurotechnology, pBCI systems are not completely ready to enter the market yet, mainly due to limitations of the EEG electrodes technology, and algorithms reliability and capability in real settings. High complexity and safety critical systems (e.g., airplanes, ATM interfaces) should adapt their behaviors and functionality accordingly to the user' actual mental state. Thus, technologies (i.e., pBCIs) able to measure in real time the user's mental states would result very useful in such "high risk" environments to enhance human machine interaction, and so increase the overall safety.

Classification of unconscious like/dislike decisions: First results towards a novel application for BCI technology.

More and more applications for BCI technology emerge that are not restricted to communication or control, like gaming, rehabilitation, Neuro-IS research, neuro-economics or security. In this context a so called passive BCI, a system that derives its outputs from arbitrary brain activity for enriching a human-machine interaction with implicit information on the actual user state will be used. Concretely EEG-based BCI technology enables the use of signals related to attention, intentions and mental state, without relying on indirect measures based on overt behavior or other physiological signals which is an important point e.g. in Neuromarketing research. The scope of this pilot EEG-study was to detect like/dislike decisions on car stimuli just by means of ERP analysis. Concretely to define user preferences concerning different car designs by implementing an offline BCI based on shrinkage LDA classification. Although classification failed in the majority of participants the elicited early (sub) conscious ERP components reflect user preferences for cars. In a broader sense this study should pave the way towards a "product design BCI" suitable for neuromarketing research.

The Space Between Us: Evaluating a multi-user affective brain-computer music interface

Music as a mechanism for neuro-feedback presents an interesting medium for artistic exploration, especially with regard to passive BCI control. Passive control in a brain-computer music interface (BCMI) provides a means for approximating mental states that can be mapped to select musical phrases, creating a system for real-time musical neuro-feedback. This article presents a BCMI for measuring the affective states of two users, a performer and an audience member, during a live musical performance of the piece titled The Space Between Us. The system adapts to the affective states of the users and selects sequences of a pre-composed musical score. By affect-matching music to mood and subsequently plotting affective musical trajectories across a two-dimensional model of affect, the system attempts to measure the affective interactions of the users, derived from arousal and valence recorded in EEG. An Affective Jukebox, the work of a previous study, validates the method used to read emotions across two dimensions in EEG in response to music. Results from a live performance of The Space Between Us indicate that measures of arousal may be manipulated by emotionally charged music, and that measures of valence are less responsive to musical stimuli, across both users. As such, an affective BCMI presents a novel platform for designing individualized musical performance and composition tools where the selection of music can reflect and induce affect in users. Furthermore, an affective channel of communication shows potential for enhancing collaboration in music-making in a wider context, for example in the roles of therapist and patient.

Artistic brain-computer interfaces: the expression and stimulation of the user’s affective state

Science, technology, engineering, and mathematics (STEM) is rapidly transitioning to STEAM, which is the integration of the arts and design into the sciences. This transition is due to the need of adapting creativity in the sciences and engineering fields. This demonstrates the growing importance of the arts contribution towards the sciences in various ways. An area where art shows promise is in brain-computer interfaces. In this paper, we propose a definition for artistic brain-computer interfaces (artistic BCI) from a passive BCI perspective. We defined its composition in four fields: human-computer interaction, neurophysiology, art, and computing. This definition is based on the user’s state stimulation and expression and how art could help us and the end user understand the user’s affective state. We also discuss its applicability towards different areas and how these areas can benefit from artistic BCI.

Estimating endogenous changes in task performance from EEG.

Brain wave activity is known to correlate with decrements in behavioral performance as individuals enter states of fatigue, boredom, or low alertness.Many BCI technologies are adversely affected by these changes in user state, limiting their application and constraining their use to relatively short temporal epochs where behavioral performance is likely to be stable. Incorporating a passive BCI that detects when the user is performing poorly at a primary task, and adapts accordingly may prove to increase overall user performance. Here, we explore the potential for extending an established method to generate continuous estimates of behavioral performance from ongoing neural activity; evaluating the extended method by applying it to the original task domain, simulated driving; and generalizing the method by applying it to a BCI-relevant perceptual discrimination task. Specifically, we used EEG log power spectra and sequential forward floating selection (SFFS) to estimate endogenous changes in behavior in both a simulated driving task and a perceptual discrimination task. For the driving task the average correlation coefficient between the actual and estimated lane deviation was 0.37 ± 0.22 (μ ± σ). For the perceptual discrimination task we generated estimates of accuracy, reaction time, and button press duration for each participant. The correlation coefficients between the actual and estimated behavior were similar for these three metrics (accuracy = 0.25 ± 0.37, reaction time = 0.33 ± 0.23, button press duration = 0.36 ± 0.30). These findings illustrate the potential for modeling time-on-task decrements in performance from concurrent measures of neural activity.

Designing a passive brain computer interface using real time classification of functional near-infrared spectroscopy

Passive brain-computer interfaces consider brain activity as an additional source of information, to augment and adapt the interface instead of controlling it. We have developed a software system that allows for real time brain signal analysis and machine learning classification of affective and workload states measured with functional near-infrared spectroscopy (fNIRS) called the online fNIRS analysis and classification (OFAC). Our system reproduces successful offline procedures, adapting them for real-time input to a user interface. Our first evaluation compares a previous offline analysis with our online analysis. While results show an accuracy decrease, they are outweighed by the new ability of interface adaptation. The second study demonstrates OFAC's online features through real-time classification of two tasks, and interface adaptation according to the predicted task. Accuracy averaged over 85%. We have created the first working real time passive BCI using fNIRS, opening the door to build adaptive user interfaces.