Based Brain-machine Interface Research Articles

Brain–machine interface based on deep learning to control asynchronously a lower-limb robotic exoskeleton: a case-of-study

BackgroundThis research focused on the development of a motor imagery (MI) based brain–machine interface (BMI) using deep learning algorithms to control a lower-limb robotic exoskeleton. The study aimed to overcome the limitations of traditional BMI approaches by leveraging the advantages of deep learning, such as automated feature extraction and transfer learning. The experimental protocol to evaluate the BMI was designed as asynchronous, allowing subjects to perform mental tasks at their own will.MethodsA total of five healthy able-bodied subjects were enrolled in this study to participate in a series of experimental sessions. The brain signals from two of these sessions were used to develop a generic deep learning model through transfer learning. Subsequently, this model was fine-tuned during the remaining sessions and subjected to evaluation. Three distinct deep learning approaches were compared: one that did not undergo fine-tuning, another that fine-tuned all layers of the model, and a third one that fine-tuned only the last three layers. The evaluation phase involved the exclusive closed-loop control of the exoskeleton device by the participants’ neural activity using the second deep learning approach for the decoding.ResultsThe three deep learning approaches were assessed in comparison to an approach based on spatial features that was trained for each subject and experimental session, demonstrating their superior performance. Interestingly, the deep learning approach without fine-tuning achieved comparable performance to the features-based approach, indicating that a generic model trained on data from different individuals and previous sessions can yield similar efficacy. Among the three deep learning approaches compared, fine-tuning all layer weights demonstrated the highest performance.ConclusionThis research represents an initial stride toward future calibration-free methods. Despite the efforts to diminish calibration time by leveraging data from other subjects, complete elimination proved unattainable. The study’s discoveries hold notable significance for advancing calibration-free approaches, offering the promise of minimizing the need for training trials. Furthermore, the experimental evaluation protocol employed in this study aimed to replicate real-life scenarios, granting participants a higher degree of autonomy in decision-making regarding actions such as walking or stopping gait.

Real-Time Deep Neurolinguistic Learning Enhances Noninvasive Neural Language Decoding for Brain-Machine Interaction.

Electroencephalogram (EEG)-based brain-machine interface (BMI) has been utilized to help patients regain motor function and has recently been validated for its use in healthy people because of its ability to directly decipher human intentions. In particular, neurolinguistic research using EEGs has been investigated as an intuitive and naturalistic communication tool between humans and machines. In this study, the human mind directly decoded the neural languages based on speech imagery using the proposed deep neurolinguistic learning. Through real-time experiments, we evaluated whether BMI-based cooperative tasks between multiple users could be accomplished using a variety of neural languages. We successfully demonstrated a BMI system that allows a variety of scenarios, such as essential activity, collaborative play, and emotional interaction. This outcome presents a novel BMI frontier that can interact at the level of human-like intelligence in real time and extends the boundaries of the communication paradigm.

Characteristics of Event-related Potentials during a Target Selection Task with Fingerspelling Images

We have been developing a cognitive assessment system based on an electroencephalography (EEG)-based brain-machine interface (BMI) that uses event-related potentials (ERPs) as a virtual “Mind Switch.” ERPs reflect temporal changes in attention and are also known as potential biomarkers for assessing cognitive function. We recorded EEG data from 10 healthy subjects who performed target-selection tasks based upon the visual discrimination of fingerspelling images with or without motor responses (key release). We found differential responses between target and non-target images, which were quantitatively evaluated by the decoding accuracy of the target. We also found that ERPs were enhanced when accompanied by key releases, which might indicate that a more attentive state was involved. These results suggest that fingerspelling images are a feasible tool for cognitive assessment. Future studies should be conducted to demonstrate the applicability of our system in preventing frailty in older people.

Tailoring brain-machine interface rehabilitation training based on neural reorganization: towards personalized treatment for stroke patients.

Electroencephalogram (EEG)-based brain-machine interface (BMI) has the potential to enhance rehabilitation training efficiency, but it still remains elusive regarding how to design BMI training for heterogeneous stroke patients with varied neural reorganization. Here, we hypothesize that tailoring BMI training according to different patterns of neural reorganization can contribute to a personalized rehabilitation trajectory. Thirteen stroke patients were recruited in a 2-week personalized BMI training experiment. Clinical and behavioral measurements, as well as cortical and muscular activities, were assessed before and after training. Following treatment, significant improvements were found in motor function assessment. Three types of brain activation patterns were identified during BMI tasks, namely, bilateral widespread activation, ipsilesional focusing activation, and contralesional recruitment activation. Patients with either ipsilesional dominance or contralesional dominance can achieve recovery through personalized BMI training. Results indicate that personalized BMI training tends to connect the potentially reorganized brain areas with event-contingent proprioceptive feedback. It can also be inferred that personalization plays an important role in establishing the sensorimotor loop in BMI training. With further understanding of neural rehabilitation mechanisms, personalized treatment strategy is a promising way to improve the rehabilitation efficacy and promote the clinical use of rehabilitation robots and other neurotechnologies.

Decoding multiclass motor imagery EEG from the same upper limb by combining Riemannian geometry features and partial least squares regression

Objective. Due to low spatial resolution and poor signal-to-noise ratio of electroencephalogram (EEG), high accuracy classifications still suffer from lots of obstacles in the context of motor imagery (MI)-based brain-machine interface (BMI) systems. Particularly, it is extremely challenging to decode multiclass MI EEG from the same upper limb. This research proposes a novel feature learning approach to address the classification problem of 6-class MI tasks, including imaginary elbow flexion/extension, wrist supination/pronation, and hand close/open within the unilateral upper limb. Approach. Instead of the traditional common spatial pattern (CSP) or filter-bank CSP (FBCSP) manner, the Riemannian geometry (RG) framework involving Riemannian distance and Riemannian mean was directly adopted to extract tangent space (TS) features from spatial covariance matrices of the MI EEG trials. Subsequently, to reduce the dimensionality of the TS features, the algorithm of partial least squares regression was applied to obtain more separable and compact feature representations. Main results. The performance of the learned RG feature representations was validated by a linear discriminative analysis and support vector machine classifier, with an average accuracy of 80.50% and 79.70% on EEG dataset collected from 12 participants, respectively. Significance. These results demonstrate that compared with CSP and FBCSP features, the proposed approach can significantly increase the decoding accuracy for multiclass MI tasks from the same upper limb. This approach is promising and could potentially be applied in the context of MI-based BMI control of a robotic arm or a neural prosthesis for motor disabled patients with highly impaired upper limb.

Decoding Movement-Related Cortical Potentials Based on Subject-Dependent and Section-Wise Spectral Filtering.

An important challenge in developing a movement-related cortical potential (MRCP)-based brain-machine interface (BMI) is an accurate decoding of the user intention for real-world environments. However, the performance remains insufficient for real-time decoding owing to the endogenous signal characteristics compared to other BMI paradigms. This study aims to enhance the MRCP decoding performance from the perspective of preprocessing techniques (i.e., spectral filtering). To the best of our knowledge, existing MRCP studies have used spectral filters with a fixed frequency bandwidth for all subjects. Hence, we propose a subject-dependent and section-wise spectral filtering (SSSF) method that considers the subjects' individual MRCP characteristics for two different temporal sections. In this study, MRCP data were acquired under a powered exoskeleton environments in which the subjects conducted self-initiated walking. We evaluated our method using both our experimental data and a public dataset (BNCI Horizon 2020). The decoding performance using the SSSF was 0.86 (±0.09), and the performance on the public dataset was 0.73 (±0.06) across all subjects. The experimental results showed a statistically significant enhancement ( ) compared with the fixed frequency bands used in previous methods on both datasets. In addition, we presented successful decoding results from a pseudo-online analysis. Therefore, we demonstrated that the proposed SSSF method can involve more meaningful MRCP information than conventional methods.

Real-Time EEG–EMG Human–Machine Interface-Based Control System for a Lower-Limb Exoskeleton

This article presents a rehabilitation technique based on a lower-limb exoskeleton integrated with a human-machine interface (HMI). HMI is used to record and process multimodal signals collected using a foot motor imagery (MI)-based brain-machine interface (BMI) and multichannel electromyographic (EMG) signals recorded from leg muscles. Current solutions of HMI-equipped rehabilitation assistive technologies tested under laboratory conditions demonstrated a great deal of success, but faced several difficulties caused by the limited accuracy of detecting MI electroencephalography (EEG) and the reliability of online control when executing a movement by patients dressed in an exoskeleton. In the case of lower-limb representation, there is still the problem of reliably distinguishing leg movement intentions and differentiating them in BMI systems. Targeting the design of a rehabilitation technique replicating the natural mode of motor control in exoskeleton walking patients, we have shown how the combined use of multimodal signals can improve the accuracy, performance, and reliability of HMI. The system was tested on healthy subjects operating the exoskeleton under different conditions. The study also resulted in algorithms of multimodal HMI data collection, processing, and classification. The developed system can analyze up to 15 signals simultaneously in real-time during a movement. Foot MI is extracted from EEG signals (seven channels) using the event-related (de)synchronization effect. Supplemented by EMG signals reflecting motor intention, the control system can initiate and differentiate the movement of the right and left legs with a high degree of reliability. The classification and control system permits one to work online when the exoskeleton is executing a movement.

A Bayesian Shared Control Approach for Wheelchair Robot With Brain Machine Interface.

To enhance the performance of the brain-actuated robot system, a novel shared controller based on Bayesian approach is proposed for intelligently combining robot automatic control and brain-actuated control, which takes into account the uncertainty of robot perception, action and human control. Based on maximum a posteriori probability (MAP), this method establishes the probabilistic models of human and robot control commands to realize the optimal control of a brain-actuated shared control system. Application on an intelligent Bayesian shared control system based on steady-state visual evoked potential (SSVEP)-based brain machine interface (BMI) is presented for all-time continuous wheelchair navigation task. Moreover, to obtain more accurate brain control commands for shared controller and adapt the proposed system to the uncertainty of electroencephalogram (EEG), a hierarchical brain control mechanism with feedback rule is designed. Experiments have been conducted to verify the proposed system in several scenarios. Eleven subjects participated in our experiments and the results illustrate the effectiveness of the proposed method.

The Advantage of Low-Delta Electroencephalogram Phase Feature for Reconstructing the Center-Out Reaching Hand Movements

It is an emerging frontier of research on the use of neural signals for prosthesis control, in order to restore lost function to amputees and patients after spinal cord injury. Compared to the invasive neural signal based brain-machine interface (BMI), a non-invasive alternative, i.e., the electroencephalogram (EEG)-based BMI would be more widely accepted by the patients above. Ideally, a real-time continuous neuroprosthestic control is required for practical applications. However, conventional EEG-based BMIs mainly deal with the discrete brain activity classification. Until recently, the literature has reported several attempts for achieving the real-time continuous control by reconstructing the continuous movement parameters (e.g., speed, position, etc.) from the EEG recordings, and the low-frequency band EEG is consistently reported to encode the continuous motor control information. Previous studies with executed movement tasks have extensively relied on the amplitude representation of such slow oscillations of EEG signals for building models to decode kinematic parameters. Inspired by the recent successes of instantaneous phase of low-frequency invasive brain signals in the motor control and sensory processing domains, this study examines the extension of such a slow-oscillation phase representation to the reconstructing two-dimensional hand movements, with the non-invasive EEG signals for the first time. The data for analysis are collected on five healthy subjects performing 2D hand center-out reaching along four directions in two sessions. On representative channels over the cortices encoding the execution information of reaching movements, we show that the low-delta EEG phase representation is characterized by higher signal-to-noise ratio and stronger modulation by the movement tasks, compared to the low-delta EEG amplitude representation. Furthermore, we have tested the low-delta EEG phase representation with two commonly used linear decoding models. The results demonstrate that the low-delta EEG phase based decoders lead to superior performance for 2D executed movement reconstruction to its amplitude based counterparts, as well as the other-frequency band amplitude and power based features. Thus, our study contributes to improve the movement reconstruction from EEG by introducing a new feature set based on the low-delta EEG phase patterns, and demonstrates its potential for continuous fine motion control of neuroprostheses.

Single-Trial Classification of Different Movements on One Arm Based on ERD/ERS and Corticomuscular Coherence

Electroencephalography (EEG)-based brain-machine interface (BMI) is widely applied to control external devices like a wheel chair or a robotic arm, to restore motor function. EEG is useful to distinguish between left arm and right arm movements, however, it is difficult to classify the different movements on one arm. In this paper, a two-step single-trial classification method is proposed to recognize three movements (make a fist, hand extension and elbow flexion) of left and right arms: (1) distinguish between left arm and right arm movements by decoding event-related (de) synchronization (ERD/ERS) and (2) recognize the specific movement of this arm using corticomuscular coherence as features. Four healthy subjects are employed in a cue-based motor execution (ME) experiment. In Step one, ERD and post-movement ERS are found over the contralateral sensorimotor area; in Step two, for each movement, only the beta-band coherence between C3/C4 and the corresponding agonistic muscle is significant. The classification results show the best accuracy of Step one and Step two is 88.10% and 93.33%, respectively. This proposed method achieves a total accuracy of 82.22%. This study demonstrates that our method is effective to classify different movements on one arm, and provides the theoretic basis and technical support for the practical development of BMI-based motor restoration applications.

Use of high-frequency visual stimuli above the critical flicker frequency in a SSVEP-based BMI

ObjectiveThis study presents a new steady-state visual evoked potential (SSVEP)-based brain–machine interface (BMI) using flickering visual stimuli at frequencies greater than the critical flicker frequency (CFF). MethodsWe first asked participants to fixate on a green/blue flicker (30–70Hz), and SSVEP amplitude was evaluated. Participants were asked to indicate whether the stimulus was visibly flickering and to report their subjective level of discomfort. We then assessed visibly (41, 43, and 45Hz) vs. invisibly (61, 63, and 65Hz) flickering stimulus in an SSVEP-based BMI. Visual fatigue was assessed via the flicker test before and after operation of the BMI. ResultsHigher frequency stimuli reduced participants’ subjective discomfort. Participants successfully controlled the SSVEP-based BMI using both the visibly and invisibly flickering stimuli (93.1% and 88.0%, respectively); the flicker test revealed a decrease in CFF (i.e., visual fatigue) under the visible condition only (–5.7%, P<0.001). ConclusionsThe use of high-frequency visual stimuli above the CFF led to high classification accuracy and decreased visual fatigue in an SSVEP-based BMI. SignificanceHigh-frequency flicker stimuli above the CFF were able to induce SSVEPs and may prove useful in the development of BMI-based assistive products.

Learned EEG-based brain self-regulation of motor-related oscillations during application of transcranial electric brain stimulation: feasibility and limitations

Objective: Transcranial direct current stimulation (tDCS) improves motor learning and can affect emotional processing and attention. However, it is unclear whether learned electroencephalography (EEG)-based brain-machine interface (BMI) control during tDCS is feasible, how application of transcranial electric currents during BMI control would interfere with feature-extraction of physiological brain signals and how it affects brain control performance. Here we tested this combination and evaluated stimulation-dependent artifacts across different EEG frequencies and stability of motor imagery-based BMI control.Approach: Ten healthy volunteers were invited to two BMI-sessions, each comprising two 60-trial blocks. During the trials, learned desynchronization of mu-rhythms (8–15 Hz) associated with motor imagery (MI) recorded over C4 was translated into online cursor movements on a computer screen. During block 2, either sham (session A) or anodal tDCS (session B) was applied at 1 mA with the stimulation electrode placed 1 cm anterior of C4.Main results: tDCS was associated with a significant signal power increase in the lower frequencies most evident in the signal spectrum of the EEG channel closest to the stimulation electrode. Stimulation-dependent signal power increase exhibited a decay of 12 dB per decade, leaving frequencies above 9 Hz unaffected. Analysis of BMI control performance did not indicate a difference between blocks and tDCS conditions.Conclusion: Application of tDCS during learned EEG-based self-regulation of brain oscillations above 9 Hz is feasible and safe, and might improve applicability of BMI systems.

3P2-P01 ヘッドマウントディスプレイベースドBMI構築と定常状態視覚誘発電位によるマイクロヘリコプタ操作(脳・神経・認知ロボティクス)

This paper presents development of a head mounted display based brain machine interface (BMI) and its application to micro helicopter control via steady state visual evoked potentials (SSVEP). The head mounted display based BMI system developed in this paper brings us the possibility of being easy to use due to its portability and no training requirements. We demonstrate that a micro helicopter is well controlled using our BMI system, that is, using brain signals, although helicopter control is very difficult due to nonlinear coupling in addition to instability.

Suggestibility and hemispheric asymmetry of 40-Hz event-related potentials during depth reversal

This study presents a new steady-state visual evoked potential (SSVEP)-based brain–machine interface (BMI) using flickering visual stimuli at frequencies greater than the critical flicker frequency (CFF).We first asked participants to fixate on a green/blue flicker (30–70 Hz), and SSVEP amplitude was evaluated. Participants were asked to indicate whether the stimulus was visibly flickering and to report their subjective level of discomfort. We then assessed visibly (41, 43, and 45 Hz) vs. invisibly (61, 63, and 65 Hz) flickering stimulus in an SSVEP-based BMI. Visual fatigue was assessed via the flicker test before and after operation of the BMI.Higher frequency stimuli reduced participants’ subjective discomfort. Participants successfully controlled the SSVEP-based BMI using both the visibly and invisibly flickering stimuli (93.1% and 88.0%, respectively); the flicker test revealed a decrease in CFF (i.e., visual fatigue) under the visible condition only (–5.7%, P < 0.001).The use of high-frequency visual stimuli above the CFF led to high classification accuracy and decreased visual fatigue in an SSVEP-based BMI.High-frequency flicker stimuli above the CFF were able to induce SSVEPs and may prove useful in the development of BMI-based assistive products.