Interface Paradigm Research Articles

Learning active sensing strategies using a sensory brain–machine interface

Diverse organisms, from insects to humans, actively seek out sensory information that best informs goal-directed actions. Efficient active sensing requires congruity between sensor properties and motor strategies, as typically honed through evolution. However, it has been difficult to study whether active sensing strategies are also modified with experience. Here, we used a sensory brain-machine interface paradigm, permitting both free behavior and experimental manipulation of sensory feedback, to study learning of active sensing strategies. Rats performed a searching task in a water maze in which the only task-relevant sensory feedback was provided by intracortical microstimulation (ICMS) encoding egocentric bearing to the hidden goal location. The rats learned to use the artificial goal direction sense to find the platform with the same proficiency as natural vision. Manipulation of the acuity of the ICMS feedback revealed distinct search strategy adaptations. Using an optimization model, the different strategies were found to minimize the effort required to extract the most salient task-relevant information. The results demonstrate that animals can adjust motor strategies to match novel sensor properties for efficient goal-directed behavior.

Influence of User Tasks on EEG-based Classification Performance in a Hazard Detection Paradigm.

Attention-based brain-computer interface (BCI) paradigms offer a way to exert control, but also to provide insight into a user's perception and judgment of the environment. For a sufficient classification performance, user engagement and motivation are critical aspects. Consequently, many paradigms require the user to perform an auxiliary task, such as mentally counting subsets of stimuli or pressing a button when encountering them. In this work, we compare two user tasks, mental counting and button-presses, in a hazard detection paradigm in driving videos. We find that binary classification performance of events based on the electroencephalogram as well as user preference are higher for button presses. Amplitudes of evoked responses are higher for the counting task-an observation which holds even after projecting out motor-related potentials during the data preprocessing. Our results indicate that the choice of button-presses can be a preferable choice in such BCIs based on prediction performance as well as user preference.

Enhancing the Usability of Brain-Computer Interface Systems.

Brain-computer interfaces (BCIs) aim to enable people to interact with the external world through an alternative, nonmuscular communication channel that uses brain signal responses to complete specific cognitive tasks. BCIs have been growing rapidly during the past few years, with most of the BCI research focusing on system performance, such as improving accuracy or information transfer rate. Despite these advances, BCI research and development is still in its infancy and requires further consideration to significantly affect human experience in most real-world environments. This paper reviews the most recent studies and findings about ergonomic issues in BCIs. We review dry electrodes that can be used to detect brain signals with high enough quality to apply in BCIs and discuss their advantages, disadvantages, and performance. Also, an overview is provided of the wide range of recent efforts to create new interface designs that do not induce fatigue or discomfort during everyday, long-term use. The basic principles of each technique are described, along with examples of current applications in BCI research. Finally, we demonstrate a user-friendly interface paradigm that uses dry capacitive electrodes that do not require any preparation procedure for EEG signal acquisition. We explore the capacitively measured steady-state visual evoked potential (SSVEP) response to an amplitude-modulated visual stimulus and the auditory steady-state response (ASSR) to an auditory stimulus modulated by familiar natural sounds to verify their availability for BCI. We report the first results of an online demonstration that adopted this ergonomic approach to evaluating BCI applications. We expect BCI to become a routine clinical, assistive, and commercial tool through advanced EEG monitoring techniques and innovative interface designs.

Current State and Future Prospects of EEG and fNIRS in Robot-Assisted Gait Rehabilitation: A Brief Review.

Gait and balance impairments are frequently considered as the most significant concerns among individuals suffering from neurological diseases. Robot-assisted gait training (RAGT) has shown to be a promising neurorehabilitation intervention to improve gait recovery in patients following stroke or brain injury by potentially initiating neuroplastic changes. However, the neurophysiological processes underlying gait recovery through RAGT remain poorly understood. As non-invasive, portable neuroimaging techniques, electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) provide new insights regarding the neurophysiological processes occurring during RAGT by measuring different perspectives of brain activity. Due to spatial information about changes in cortical activation patterns and the rapid temporal resolution of bioelectrical changes, more features correlated with brain activation and connectivity can be identified when using fused EEG-fNIRS, thus leading to a detailed understanding of neurophysiological mechanisms underlying motor behavior and impairments due to neurological diseases. Therefore, multi-modal integrations of EEG-fNIRS appear promising for the characterization of neurovascular coupling in brain network dynamics induced by RAGT. In this brief review, we surveyed neuroimaging studies focusing specifically on robotic gait rehabilitation. While previous studies have examined either EEG or fNIRS with respect to RAGT, a multi-modal integration of both approaches is lacking. Based on comparable studies using fused EEG-fNIRS integrations either for guiding non-invasive brain stimulation or as part of brain-machine interface paradigms, the potential of this methodologically combined approach in RAGT is discussed. Future research directions and perspectives for targeted, individualized gait recovery that optimize the outcome and efficiency of RAGT in neurorehabilitation were further derived.

Synchronization of Slow Cortical Rhythms During Motor Imagery-Based Brain-Machine Interface Control.

Modulation of sensorimotor rhythm (SMR) power, a rhythmic brain oscillation physiologically linked to motor imagery, is a popular Brain-Machine Interface (BMI) paradigm, but its interplay with slower cortical rhythms, also involved in movement preparation and cognitive processing, is not entirely understood. In this study, we evaluated the changes in phase and power of slow cortical activity in delta and theta bands, during a motor imagery task controlled by an SMR-based BMI system. In Experiment I, EEG of 20 right-handed healthy volunteers was recorded performing a motor-imagery task using an SMR-based BMI controlling a visual animation, and during task-free intervals. In Experiment II, 10 subjects were evaluated along five daily sessions, while BMI-controlling same visual animation, a buzzer, and a robotic hand exoskeleton. In both experiments, feedback received from the controlled device was proportional to SMR power (11-14 Hz) detected by a real-time EEG-based system. Synchronization of slow EEG frequencies along the trials was evaluated using inter-trial-phase coherence (ITPC). Results: cortical oscillations of EEG in delta and theta frequencies synchronized at the onset and at the end of both active and task-free trials; ITPC was significantly modulated by feedback sensory modality received during the tasks; and ITPC synchronization progressively increased along the training. These findings suggest that phase-locking of slow rhythms and resetting by sensory afferences might be a functionally relevant mechanism in cortical control of motor function. We propose that analysis of phase synchronization of slow cortical rhythms might also improve identification of temporal edges in BMI tasks and might help to develop physiological markers for identification of context task switching and practice-related changes in brain function, with potentially important implications for design and monitoring of motor imagery-based BMI systems, an emerging tool in neurorehabilitation of stroke.

Intrinsic Variable Learning for Brain-Machine Interface Control by Human Anterior Intraparietal Cortex

Although animal studies provided significant insights in understanding the neural basis of learning and adaptation, they often cannot dissociate between different learning mechanisms due to the lack of verbal communication. To overcome this limitation, we examined the mechanisms of learning and its limits in a human intracortical brain-machine interface (BMI) paradigm. A tetraplegic participant controlled a 2D computer cursor by modulating single-neuron activity in the anterior intraparietal area (AIP). By perturbing the neuron-to-movement mapping, the participant learned to modulate the activity of the recorded neurons to solve the perturbations by adopting a target re-aiming strategy. However, when no cognitive strategies were adequate to produce correct responses, AIP failed to adapt to perturbations. These findings suggest that learning is constrained by the pre-existing neuronal structure, although it is possible that AIP needs more training time to learn to generate novel activity patterns when cognitive re-adaptation fails to solve the perturbations.

A Novel Single-Character Visual BCI Paradigm With Multiple Active Cognitive Tasks.

To introduce a novel event-related potential (ERP)-based brain-computer interface (BCI) paradigm with active mental tasks multiplying precise judgment and visual cognitive capacities and evaluate its performance. This study employed a paradigm with three types of targets (true-, pseudo-, and non-), double flash codes, colors and color terms, and four test conditions. The primary hypothesis was that active mental tasks combining multiple cognitive capacities and clear judgment for different categories of stimuli increase the BCI performance and evoke stronger or specific ERPs. Classification methods were proposed and evaluated, and two were used in online experiments. The modes containing active mental tasks provided higher accuracy than the control mode (by up to 19.06%). The color-word matching mode had the highest judgment level and achieved the best performance. True-stimuli evoked strong P3b, while pseudotarget signals provided obvious N4, but the control mode seemed less sensitive to both of them. Different types of stimuli evoked distinctive N2 and P3a components. An appropriate boost in the judgment level using multiple stimuli and cognitive approaches could be investigated to improve the BCI performance and evoke or enhance ERPs. Utilizing active mental tasks may be a promising way to promote BCIs. Active mental tasks combining multiple cognitive capacities and precise judgments were adopted in an ERP-based BCI. Color and color words were introduced as stimuli to construct an alternative paradigm, and the judgment levels of different conditions were calculated. High accuracies and the participants' preferences were obtained, which may promote the effective use of BCIs.

Distinct types of neural reorganization during long-term learning.

What are the neural mechanisms of skill acquisition? Many studies find that long-term practice is associated with a functional reorganization of cortical neural activity. However, the link between these changes in neural activity and the behavioral improvements that occur is not well understood, especially for long-term learning that takes place over several weeks. To probe this link in detail, we leveraged a brain-computer interface (BCI) paradigm in which rhesus monkeys learned to master nonintuitive mappings between neural spiking in primary motor cortex and computer cursor movement. Critically, these BCI mappings were designed to disambiguate several different possible types of neural reorganization. We found that during the initial phase of learning, lasting minutes to hours, rapid changes in neural activity common to all neurons led to a fast suppression of motor error. In parallel, local changes to individual neurons gradually accrued over several weeks of training. This slower timescale cortical reorganization persisted long after the movement errors had decreased to asymptote and was associated with more efficient control of movement. We conclude that long-term practice evokes two distinct neural reorganization processes with vastly different timescales, leading to different aspects of improvement in motor behavior. NEW & NOTEWORTHY We leveraged a brain-computer interface learning paradigm to track the neural reorganization occurring throughout the full time course of motor skill learning lasting several weeks. We report on two distinct types of neural reorganization that mirror distinct phases of behavioral improvement: a fast phase, in which global reorganization of neural recruitment leads to a quick suppression of motor error, and a slow phase, in which local changes in individual tuning lead to improvements in movement efficiency.

Motor Imagery based Brain Computer Interface Paradigm for Upper Limb Stroke Rehabilitation.

Motor Imagery (MI) based Brain Computer Interface (BCI) systems have shown potential to serve as a tool for neurorehabilitation for post stroke patients to complement the standard therapy. The aim of this study was to develop an MI based BCI system that could potentially be used in neurorehabilitation of hand motor function in stroke patients. Two co-adaptive, three-class MI based BCI systems for realtime processing were developed and compared using the publicly available data from the BCI Competition III Dataset V as well as our own data. The first algorithm utilizes the Filterbank Common Spatial Pattern (FBCSP) for feature extraction, and the other utilizes the Separable Common Spatio-Spectral Pattern (SCSSP) - both combined with a Multi-layer Perceptron (MLP) for classification. The proposed system proved successful when using the competition data showing an average accuracy of 64.71 % for the SCSSP compared to 60.48% for the FBCSP. This proved superior to a related study using the same feature extraction methods, but with other classification methods. The proposed system, however did show results around chance level for the 3-class MI experimental data that we have collected in our laboratory. Further studies needs to be conducted to improve the performance as well as to realize such a system to put in use.

Development of an electrooculogram-based human-computer interface using involuntary eye movement by spatially rotating sound for communication of locked-in patients

Individuals who have lost normal pathways for communication need augmentative and alternative communication (AAC) devices. In this study, we propose a new electrooculogram (EOG)-based human-computer interface (HCI) paradigm for AAC that does not require a user’s voluntary eye movement for binary yes/no communication by patients in locked-in state (LIS). The proposed HCI uses a horizontal EOG elicited by involuntary auditory oculogyric reflex, in response to a rotating sound source. In the proposed HCI paradigm, a user was asked to selectively attend to one of two sound sources rotating in directions opposite to each other, based on the user’s intention. The user’s intentions could then be recognised by quantifying EOGs. To validate its performance, a series of experiments was conducted with ten healthy subjects, and two patients with amyotrophic lateral sclerosis (ALS). The online experimental results exhibited high-classification accuracies of 94% in both healthy subjects and ALS patients in cases where decisions were made every six seconds. The ALS patients also participated in a practical yes/no communication experiment with 26 or 30 questions with known answers. The accuracy of the experiments with questionnaires was 94%, demonstrating that our paradigm could constitute an auxiliary AAC system for some LIS patients.

Towards using fNIRS recordings of mental arithmetic for the detection of residual cognitive activity in patients with disorders of consciousness (DOC)

BackgroundRecently, fNIRS has been proposed as a promising approach for awareness detection, and a possible method to establish basic communication in patients with disorders of consciousness (DOC). AimUsing fNIRS, the present study evaluated the applicability of auditory presented mental-arithmetic tasks in this respect. MethodsWe investigated the applicability of active attention to serial subtractions for awareness detection in ten healthy controls (HC, 21–32 y/o), by comparing the measured patterns to patterns induced by self-performance of the same task. Furthermore, we examined the suitability of ignoring the given task as additional control signal to implement a two-class brain-computer interface (BCI) paradigm. Finally, we compared our findings in HC with recordings in one DOC patient (78 y/o). Results and conclusionResults of the HC revealed no differences between the self-performance and the attention condition, making the attention task suitable for awareness detection. However, there was no general difference between the ignore and attend condition, making the tasks less suitable for BCI control. Despite inconsistent correlations between the patient data and the HC group, single runs of the patient recordings revealed task-synchronous patterns – however, we cannot conclude whether the measured activation derives from instruction based task performance and thus awareness.

Optically-Controlled 'Living Electrodes' with Long-Projecting Axon Tracts for a Synaptic Brain-Machine Interface

Achievements in intracortical brain-machine interfaces are compromised by limitations in long-term performance and information transfer rate. A biological intermediary between devices and the brain based on synaptic integration may offer a specificity and permanence that has eluded neural interfaces to date. Accordingly, we have developed the first living electrodes comprised of implantable axonal tracts protected within soft hydrogel cylinders to enable biologically-mediated monitoring and modulation of brain activity. Here we demonstrate the controlled fabrication, rapid axonal outgrowth, reproducible cytoarchitecture, and axonal conduction of these engineered constructs in vitro. We also present simultaneous optical stimulation and recording of neuronal activity in vitro, transplantation in rat cortex, and their survival, integration, and activity over time in vivo as a proof-of-concept for this neural interface paradigm. The creation and functional validation of living electrodes is a critical step towards developing a new class of neural interfaces using targeted, synaptic-based integration with native circuitry.

Pedagogical Agents: Back to the Future

Back in the 1990s, we started work on pedagogical agents — a novel paradigm for interactive learning. Pedagogical agents are autonomous characters that inhabit learning environments to engage with learners in rich, face‐to‐face interactions. Building on this work, in 2000, together with our colleague Jeff Rickel, we published an article on pedagogical agents (Johnson, Rickel, and Lester 2000) that surveyed and discussed the potential of this new paradigm. We made the case that pedagogical agents that interact with learners in natural, lifelike ways can help learning environments achieve improved learning outcomes. This article has been widely cited, and was a winner of the 2017 IFAAMAS Award for Influential Papers in Autonomous Agents and Multiagent Systems.1On the occasion of receiving the IFAAMAS award, and after 20 years of work on pedagogical agents, we take another look at the future of the field. We start by revisiting the predictions we made in 2000 for pedagogical agents, and examine which predictions panned out. Then, informed by what we have learned since then, we take another look at emerging trends and reexamine the future of pedagogical agents. Advances in natural language dialogue, affective computing, machine learning, virtual environments, and robotics are making possible even more lifelike and effective pedagogical agents, with potentially profound effects on the way people learn.

A rodent brain-machine interface paradigm to study the impact of paraplegia on BMI performance

BackgroundMost brain machine interfaces (BMI) focus on upper body function in non-injured animals, not addressing the lower limb functional needs of those with paraplegia. A need exists for a novel BMI task that engages the lower body and takes advantage of well-established rodent spinal cord injury (SCI) models to study methods to improve BMI performance. New methodA tilt BMI task was designed that randomly applies different types of tilts to a platform, decodes the tilt type applied and rights the platform if the decoder correctly classifies the tilt type. The task was tested on female rats and is relatively natural such that it does not require the animal to learn a new skill. It is self-rewarding such that there is no need for additional rewards, eliminating food or water restriction, which can be especially hard on spinalized rats. Finally, task difficulty can be adjusted by making the tilt parameters. ResultsThis novel BMI task bilaterally engages the cortex without visual feedback regarding limb position in space and animals learn to improve their performance both pre and post-SCI.Comparison with Existing Methods: Most BMI tasks primarily engage one hemisphere, are upper-body, rely heavily on visual feedback, do not perform investigations in animal models of SCI, and require nonnaturalistic extrinsic motivation such as water rewarding for performance improvement. Our task addresses these gaps. ConclusionsThe BMI paradigm presented here will enable researchers to investigate the interaction of plasticity after SCI and plasticity during BMI training on performance.

An approach for brain-controlled prostheses based on Scene Graph Steady-State Visual Evoked Potentials

Brain control technology can restore communication between the brain and a prosthesis, and choosing a Brain-Computer Interface (BCI) paradigm to evoke electroencephalogram (EEG) signals is an essential step for developing this technology. In this paper, the Scene Graph paradigm used for controlling prostheses was proposed; this paradigm is based on Steady-State Visual Evoked Potentials (SSVEPs) regarding the Scene Graph of a subject’s intention. A mathematic model was built to predict SSVEPs evoked by the proposed paradigm and a sinusoidal stimulation method was used to present the Scene Graph stimulus to elicit SSVEPs from subjects. Then, a 2-degree of freedom (2-DOF) brain-controlled prosthesis system was constructed to validate the performance of the Scene Graph-SSVEP (SG-SSVEP)-based BCI. The classification of SG-SSVEPs was detected via the Canonical Correlation Analysis (CCA) approach.To assess the efficiency of proposed BCI system, the performances of traditional SSVEP-BCI system were compared. Experimental results from six subjects suggested that the proposed system effectively enhanced the SSVEP responses, decreased the degradation of SSVEP strength and reduced the visual fatigue in comparison with the traditional SSVEP-BCI system. The average signal to noise ratio (SNR) of SG-SSVEP was 6.31 ± 2.64 dB, versus 3.38 ± 0.78 dB of traditional-SSVEP. In addition, the proposed system achieved good performances in prosthesis control. The average accuracy was 94.58% ± 7.05%, and the corresponding high information transfer rate (IRT) was 19.55 ± 3.07 bit/min. The experimental results revealed that the SG-SSVEP based BCI system achieves the good performance and improved the stability relative to the conventional approach.

BlindSense: An Accessibility-inclusive Universal User Interface for Blind People

A large number of blind people use smartphone-based assistive technology to perform their common activities. In order to provide a better user experience the existing user interface paradigm needs to be revisited. A new user interface model has been proposed in this paper. A simplified, semantically consistent, and blind-friendly adaptive user interface is provided. The proposed solution is evaluated through an empirical study on 63 blind people leveraging an improved user experience in performing common activities on a smartphone.

Interface paradigm of Bots-based human-computer interaction

Bots, usually known as chatbots, have given rise to a brand-new pattern of user-centered interaction. Owing to the development of multi-modal natural language interaction, Bots no longer rely on the graphical desktop for interaction. Consequently, a new interface paradigm is required to normalize the bots-based human-computer interaction, which is no longer limited to the WIMP (windows, icon, menu, and pointer) paradigm of the desktop paradigm. In this paper, a new ASLI (avatar, scenario, language, and instrument) interface paradigm is proposed on the basis of the extended avatar metaphor. The framework of the Bots-based interaction is provided as well as the corresponding interaction instances, which provide guidance for the interface design of Bots-based interaction. The platforms of Bots and ASLI are also discussed and are believed to be able to derive a Bots-based ecosystem in the near future.

Interaction paradigm in intelligent systems

This paper discusses the user interface design paradigm for intelligent systems. We first introduce the concepts of paradigm in science and human-computer interaction (HCI). After discussing the WIMP (windows, icons, menus, a pointing device) paradigm, the most influential user interface design paradigm in HCI, we summarize the major components a user interface paradigm usually takes into consideration, and based on this summarization, we analyze the features as well as limitations of the user interfaces of some intelligent systems currently available on the market. We also propose a user interface design paradigm, RMCP (role, modal, commands, and presentation style), for intelligent systems, which emphasizes the consideration of the role(s) that an intelligent system plays in a users life and work, the interaction modal(s) that the system can accommodate, the commands that the system supports, and the presentation style(s) that the system adopts for information outputs.

Visual Stimuli for P300-Based Brain-Computer Interfaces: Color, Shape, and Mobility

The purpose of this study was to identify the impact of different discriminative features of stimuli in a P300 brain-computer interface paradigm on overall performance and evoked potentials. It has been shown that stimuli sets with a greater number of discriminative features yield better target selection accuracy. Target selection accuracy was significantly higher for the stimuli that differ from each other by color, shape, and semantics. Highest performance was achieved with the stimuli set containing the largest number of discriminative features, namely a set of nine different-colored letters. This result is mainly due to higher mean P300 peak amplitude for stimuli sets that contain more discriminative features. The results of the study can be used for designing a better user experience in brain-computer interfacing (BCI). Motion of the stimuli presentation point and characteristics of this motion (linear or pseudorandom) did not have any impact on BCI performance. This result is promising for future BCI designs with rapid serial visual presentation using mobile robots or augmented reality as stimuli presentation environment.

A Speedy Calibration Method Using Riemannian Geometry Measurement and Other-Subject Samples on A P300 Speller.

P300 spellers are among the most popular brain-computer interface paradigms, and they are used for many clinical applications. However, building the classifier for identifying event-related potential (ERP) responses, i.e., calibrating the P300 speller, is still a time-consuming and user-dependent problem. This paper proposes a novel method to reduce calibration times significantly. In the proposed method, a small number of ERP epochs from the current user were used to build a reference epoch. Based on this reference, the Riemannian distance measurement was used to select similar ERP samples from an existing data pool, which contained other-subject ERP responses. Linear discriminant analysis (LDA), support vector machine, and stepwise LDA were trained as ERP classifiers on the selected database and then were used to identify the user-attended character. With only 12 s of EEG data to calibrate, an average character recognition accuracy for 55 subjects of up to 87.82% was obtained. The LDA that built on other-subject samples that were selected by Riemannian distance outperformed the other classifiers. Compared with other state-of-the-art studies, this method significantly reduces P300 speller calibration times, while maintaining the character recognition accuracy.