Novel Hybrid Brain-Computer Interface for Virtual Reality Applications Using Steady-State Visual-Evoked Potential-Based Brain-Computer Interface and Electrooculogram-Based Eye Tracking for Increased Information Transfer Rate.

Jisoo Ha,Seonghun Park,Chang-Hwan Im

doi:10.3389/fninf.2022.758537

Abstract

Brain–computer interfaces (BCIs) based on electroencephalogram (EEG) have recently attracted increasing attention in virtual reality (VR) applications as a promising tool for controlling virtual objects or generating commands in a “hands-free” manner. Video-oculography (VOG) has been frequently used as a tool to improve BCI performance by identifying the gaze location on the screen, however, current VOG devices are generally too expensive to be embedded in practical low-cost VR head-mounted display (HMD) systems. In this study, we proposed a novel calibration-free hybrid BCI system combining steady-state visual-evoked potential (SSVEP)-based BCI and electrooculogram (EOG)-based eye tracking to increase the information transfer rate (ITR) of a nine-target SSVEP-based BCI in VR environment. Experiments were repeated on three different frequency configurations of pattern-reversal checkerboard stimuli arranged in a 3 × 3 matrix. When a user was staring at one of the nine visual stimuli, the column containing the target stimulus was first identified based on the user’s horizontal eye movement direction (left, middle, or right) classified using horizontal EOG recorded from a pair of electrodes that can be readily incorporated with any existing VR-HMD systems. Note that the EOG can be recorded using the same amplifier for recording SSVEP, unlike the VOG system. Then, the target visual stimulus was identified among the three visual stimuli vertically arranged in the selected column using the extension of multivariate synchronization index (EMSI) algorithm, one of the widely used SSVEP detection algorithms. In our experiments with 20 participants wearing a commercial VR-HMD system, it was shown that both the accuracy and ITR of the proposed hybrid BCI were significantly increased compared to those of the traditional SSVEP-based BCI in VR environment.

Highlights

Brain–computer interface (BCI) is a technology that directly translates brain activities into commands to provide users with a new communication channel toward outside world (Vallabhaneni et al, 2013)
We first investigated whether the accuracy of classifying three directions using horizontal EOG (hEOG) was high enough to be employed for our hybrid BCI system
We proposed a new hybrid BCI system combining state visual-evoked potential (SSVEP)-based BCI and EOG-based eye tracking to effectively improve the BCI performance in virtual reality (VR) environment

Summary

Introduction

Brain–computer interface (BCI) is a technology that directly translates brain activities into commands to provide users with a new communication channel toward outside world (Vallabhaneni et al, 2013). Previous BCIs were mostly applied to patients with severe paralyzes (Daly and Wolpaw, 2008), their applications have been gradually expanded to the general public in recent years, one of the representative example of which includes hands-free operation of external devices (Coogan and He, 2018) such as drones (Nourmohammadi et al, 2018), robots (Liu et al, 2021), and game characters (Vasiljevic and De Miranda, 2019). Among the various EEG-based BCI paradigms including motor imagery (MI), P300, and steady-state visualevoked potential (SSVEP) (Yin et al, 2015), SSVEP-based BCIs have been successfully applied to VR applications as a new hands-free communication tool, thanks to its high information transfer rate (ITR) and no (or less) training requirement (Coogan and He, 2018; Stawicki et al, 2018; Choi et al, 2019; Grichnik et al, 2019; Monteiro et al, 2021). As will be shown in the results of this study, the increment of the number of visual stimuli (e.g., 9) greatly degrades the overall classification accuracy compared to previous studies because the SSVEP-based BCIs implemented with VRHMDs are more prone to be affected by the peripheral vision due to the relatively short distance between the eyes and the display

Methods

Results

Conclusion