Fast mental states decoding in mixed reality.

Daniele De Massari,Alberto Betella,Daniel Pacheco,Paul F M J Verschure,Rahim Malekshahi,Andrea Caria,Niels Birbaumer

doi:10.3389/fnbeh.2014.00415

Abstract

The combination of Brain-Computer Interface (BCI) technology, allowing online monitoring and decoding of brain activity, with virtual and mixed reality (MR) systems may help to shape and guide implicit and explicit learning using ecological scenarios. Real-time information of ongoing brain states acquired through BCI might be exploited for controlling data presentation in virtual environments. Brain states discrimination during mixed reality experience is thus critical for adapting specific data features to contingent brain activity. In this study we recorded electroencephalographic (EEG) data while participants experienced MR scenarios implemented through the eXperience Induction Machine (XIM). The XIM is a novel framework modeling the integration of a sensing system that evaluates and measures physiological and psychological states with a number of actuators and effectors that coherently reacts to the user's actions. We then assessed continuous EEG-based discrimination of spatial navigation, reading and calculation performed in MR, using linear discriminant analysis (LDA) and support vector machine (SVM) classifiers. Dynamic single trial classification showed high accuracy of LDA and SVM classifiers in detecting multiple brain states as well as in differentiating between high and low mental workload, using a 5 s time-window shifting every 200 ms. Our results indicate overall better performance of LDA with respect to SVM and suggest applicability of our approach in a BCI-controlled MR scenario. Ultimately, successful prediction of brain states might be used to drive adaptation of data representation in order to boost information processing in MR.

Highlights

Mixed Reality (MR) is a type of virtual reality-related technology where real and virtual worlds are merged so that real-time interaction with both physical and digital objects (Milgram, 1994; Bohil et al, 2011) is achievable
The linear discriminant analysis (LDA) based classifier generated on average the highest accuracy (83.30%, Matthews correlation coefficient (MCC) = 0.72) across all subjects, with peaks of 89.72% for accuracy and 0.84 for MCC in subject 2
To this end we explored continuous EEG data decoding during performance of MR relevant tasks such as spatial navigation, calculation and reading in XIM, using LDA and support vector machine (SVM) based classifiers

Summary

Introduction

Mixed Reality (MR) is a type of virtual reality-related technology where real and virtual worlds are merged so that real-time interaction with both physical and digital objects (Milgram, 1994; Bohil et al, 2011) is achievable. A promising MR system is the eXperience Induction Machine (XIM) (Bernardet et al, 2011; Omedas et al, 2014). This technology permits to model representational elements analog to real phenomena as well as highly abstract non-representation forms describing complex high-dimensional data in a controlled environment. BCI-based real-time analysis of brain signals, with no need of participants to learn their control (sometimes referred to as “passive” BCI), can be utilized to manipulate behavioral response by delivering information according to specific mental states

Methods

Results

Conclusion