Cursor Control Research Articles

Диагностические маркеры двигательной активности человека в электронной образовательной среде

The article aims to study the dynamics of psychological and psychophysiological indicators of students’ cognitive activity in a digital learning environment. As a research platform, we used the well-known task of speed–accuracy tradeoff, adapted for work in the LMS Moodle using a set of plug-ins that implement both the stimulus situation itself and the ability to monitor the characteristics of the user’s motor activity displayed in the track of the cursor control device (mouse tracking). As part of the study, LMS users performed a cognitive test, included in the digital learning environment and demonstrating the manifestation of Fits’s law in two versions: with instructions for the accuracy and speed of searching for the center of a reference figure that appears on a computer screen. The results were obtained in the form of tracks of the movement of a computer mouse, the main characteristics of the motor activity of the subjects – the time to complete both parts of the test, the distance traveled, the accuracy of hitting targets, and speed and acceleration – were analyzed and evaluated. All participants’ indicators significantly differed depending on the stages of the experiment. These studies were presented in the form of a set of formal statistical generalizations and visualization, reflecting the characteristic features of the subjects’ actions, the distance traveled, speed and acceleration on all segments of the path. As a result of data analysis, 6 groups of users were identified, differing in the degree of severity of indicators in terms of the characteristics of speed, accuracy and amplitude of movement. Of particular interest were such traces of a person’s actions, which, on the one hand, were an expression of his skills or experiences, and, on the other hand, were available not only for subjective, but also for objective analysis. As an example, we can point to the handwriting of a person who became one of the promising objects of study by psychologists of the early twentieth century. The concept of “handwriting” was easily transferred to other ways of transmitting information, for example, to the work of an operator with a key when transmitting messages using the “Morse code” method. Thus, the use of a computer mouse monitoring system in a digital learning environment provides an opportunity to quickly and adaptively provide interaction with the user, taking into account their psychophysiological characteristics and emotional state. The proposed diagnostic system allows us not only to determine the quantitative characteristics of the path traveled and the click map of a computer mouse, but also to classify the user into one or another group, taking into account their psychological characteristics.

Архітектура систем реабілітації з біологічним зворотним зв’язком

Non-invasiveness, non-toxicity, reliability, and efficiency make the systems with biological feedback (BF) the most promising for improvement. The structures of the vast majority of BF-systems have been studied and the most advanced one has been proposed. The technical essence of the BF-method lies in the computer registration of certain physiological indicators that are not available for direct human perception and their transformation into a form understandable to people. Since all systems of the human organism are directly subject to the constant influ-ence of the central nervous system, the most promising type of BF is neurobiocontrol of self-learning of your own brain. The general BCI (Brain-Computer-Interface) scheme contains an encephalograph whose inputs are connected to the patient's scalp and the outputs – to a comput-er connected to a controlled device. Using AI, BCI options are greatly expanded to include cur-sor control, auditory sensations, limb control, spelling devices, somatic sensations, and visual prostheses. The National Center for Adaptive Neurotechnologies has developed the BCI200 software system for fundamental and clinical neurophysiological studies of brain-computer in-teraction. As a result, all existing implementations and applications of BF-based systems for medical rehabilitation do not meet modern trends of creating unified, portable and safe new generation rehabilitation systems which can be used not only in medical institutions but also at home. To cardinally solve this problem, a new structure of the hardware and software complex for the implementation of the advanced BF-method of rehabilitation of people with disabilities has been suggested. The improved BF-procedure consists of three steps: entering and accumu-lating knowledge in the rehabilitation field, setting personal rehabilitation goals, and searching for a solution to these goals.

Mouse Cursor Control System using Gaze and Eye Opening and Closing

Mouse Cursor Control System using Gaze and Eye Opening and Closing

Vigilance Estimating in SSVEP-Based BCI Using Multimodal Signals.

Brain-computer interface (BCI) is a communication system that allows a direct connection between the human brain and external devices. With the application of BCI, it is important to estimate vigilance for BCI users. In order to investigate the vigilance changes of the subjects during BCI tasks and develop a multimodal method to estimate the vigilance level, a high-speed 4-target BCI system for cursor control was built based on steady-state visual evoked potential (SSVEP). 18 participants were recruited and underwent a 90-min continuous cursor-control BCI task, when electroencephalogram (EEG), electrooculogram (EOG), electrocardiography (ECG), and electrodermal activity (EDA) were recorded simultaneously. Then, we extracted features from the multimodal signals and applied regression models to estimate vigilance. Experimental results showed that the differential entropy (DE) feature could effectively reflect the change of vigilance. The vigilance estimation method, which integrates DE and EOG features into the support vector regression (SVR) model, achieved a better performance than the compared methods. These results demonstrate the feasibility of our methods for estimating vigilance levels in BCI.

The dorsolateral pre-frontal cortex bi-polar error-related potential in a locked-in patient implanted with a daily use brain\u2013computer interface

While brain computer interfaces (BCIs) offer the potential of allowing those suffering from loss of muscle control to once again fully engage with their environment by bypassing the affected motor system and decoding user intentions directly from brain activity, they are prone to errors. One possible avenue for BCI performance improvement is to detect when the BCI user perceives the BCI to have made an unintended action and thus take corrective actions. Error-related potentials (ErrPs) are neural correlates of error awareness and as such can provide an indication of when a BCI system is not performing according to the user’s intentions. Here, we investigate the brain signals of an implanted BCI user suffering from locked-in syndrome (LIS) due to late-stage ALS that prevents her from being able to speak or move but not from using her BCI at home on a daily basis to communicate, for the presence of error-related signals. We first establish the presence of an ErrP originating from the dorsolateral pre-frontal cortex (dLPFC) in response to errors made during a discrete feedback task that mimics the click-based spelling software she uses to communicate. Then, we show that this ErrP can also be elicited by cursor movement errors in a continuous BCI cursor control task. This work represents a first step toward detecting ErrPs during the daily home use of a communications BCI.

Voice and Touch Based Error-tolerant Multimodal Text Editing and Correction for Smartphones.

Editing operations such as cut, copy, paste, and correcting errors in typed text are often tedious and challenging to perform on smartphones. In this paper, we present VT, a voice and touch-based multi-modal text editing and correction method for smartphones. To edit text with VT, the user glides over a text fragment with a finger and dictates a command, such as “bold” to change the format of the fragment, or the user can tap inside a text area and speak a command such as “highlight this paragraph” to edit the text. For text correcting, the user taps approximately at the area of erroneous text fragment and dictates the new content for substitution or insertion. VT combines touch and voice inputs with language context such as language model and phrase similarity to infer a user’s editing intention, which can handle ambiguities and noisy input signals. It is a great advantage over the existing error correction methods (e.g., iOS’s Voice Control) which require precise cursor control or text selection. Our evaluation shows that VT significantly improves the efficiency of text editing and text correcting on smartphones over the touch-only method and the iOS’s Voice Control method. Our user studies showed that VT reduced the text editing time by 30.80%, and text correcting time by 29.97% over the touch-only method. VT reduced the text editing time by 30.81%, and text correcting time by 47.96% over the iOS’s Voice Control method.

Nose Tracking Assistive Robot Control for the People with Motor Dysfunctions

<h3>Research Objectives</h3> To Implement Human Computer Interface (HCI) with digital image processing for a new approach to control assistive robots. <h3>Design</h3> Experimental Study. <h3>Setting</h3> In UWM BioRobotics Lab. <h3>Participants</h3> Healthy participants. <h3>Interventions</h3> Assistive robot control for daily living activities(Eating, Picking/Placing Objects), Nose tracking cursor control, User interface. <h3>Main Outcome Measures</h3> #Control the computer cursor with nose movements. #App Interface is developed to control the robotic arm. #Control the interface using nose and teeth (showing teeth triggered the click events) to control the assistive robotic arm. <h3>Results</h3> Result contains the demonstration of the control of a robotic arm with nose tracking control through an user interface. External webcam was used to get the real-time video for tracking the nose for cursor movement. We conducted experiments on healthy subjects. The feedbacks are recorded in Tables. We compared the results with other two techniques which are eye tracking and camera mouse for cursor movement and clicking events in terms of error pixels and other factors. <h3>Conclusions</h3> The nose tracking cursor control is mainly for the disabled people who are struggling with or unable to do their basic day to day tasks for their motor dysfunctions. This method will help them to access and control over a computer as well as a robotic assistive arm to ease their lives. <h3>Author(s) Disclosures</h3> No Conflict

Generalizable cursor click decoding using grasp-related neural transients

Objective. Intracortical brain–computer interfaces (iBCI) have the potential to restore independence for individuals with significant motor or communication impairments. One of the most realistic avenues for clinical translation of iBCI technology is enabling control of a computer cursor—i.e. movement-related neural activity is interpreted (decoded) and used to drive cursor function. Here we aim to improve cursor click decoding to allow for both point-and-click and click-and-drag control. Approach. Using chronic microelectrode arrays implanted in the motor cortex of two participants with tetraplegia, we identified prominent neural responses related to attempted hand grasp. We then developed a new approach for decoding cursor click (hand grasp) based on the most salient responses. Main results. We found that the population-wide response contained three dominant components related to hand grasp: an onset transient response, a sustained response, and an offset transient response. The transient responses were larger in magnitude—and thus more reliably detected—than the sustained response, and a click decoder based on these transients outperformed the standard approach of binary state classification. Significance. A transient-based approach for identifying hand grasp can provide a high degree of cursor click control for both point-and-click and click-and-drag applications. This generalized click functionality is an important step toward high-performance cursor control and eventual clinical translation of iBCI technology.

Brainwave assistive system for paralyzed individuals

The Brain-Computer Interface (BCI) is a system based on brainwaves that can be used to translate and comprehend the innumerable activities of the brain. Brainwave refers to the bioelectric impulses invariably produced in the human brain during neurotransmission, often measured as the action potential. Moreover, BCI essentially uses the widely studied Electroencephalography (EEG) technique to capture brainwave data. Paralysis generally occurs when there is a disturbance in the central nervous system prompted by a neurodegenerative or unforeseen event. To overcome the obstacles associated with paralysis, this paper on the brainwave-assistive system is based on the BCI incorporated with Internet-of-things. BCI can be implemented to achieve control over external devices and applications. For instance, the process of cursor control, motor control, neuroprosthetics and wheelchair control, etc. In this paper, the OpenBCI Cyton-biosensing board has been used for the collection of the EEG data. The accumulated EEG data is executed subsequently to obtain control over the respective systems in real-time. Hence, it can be concluded that the experiments of the paper support the idea of controlling an interfaced system through the real-time application of EEG data.

Home Use of a Percutaneous Wireless Intracortical Brain-Computer Interface by Individuals With Tetraplegia.

Objective.Individuals with neurological disease or injury such as amyotrophic lateral sclerosis, spinal cord injury or stroke may become tetraplegic, unable to speak or even locked-in. For people with these conditions, current assistive technologies are often ineffective. Brain-computer interfaces are being developed to enhance independence and restore communication in the absence of physical movement. Over the past decade, individuals with tetraplegia have achieved rapid on-screen typing and point-and-click control of tablet apps using intracortical brain-computer interfaces (iBCIs) that decode intended arm and hand movements from neural signals recorded by implanted microelectrode arrays. However, cables used to convey neural signals from the brain tether participants to amplifiers and decoding computers and require expert oversight, severely limiting when and where iBCIs could be available for use. Here, we demonstrate the first human use of a wireless broadband iBCI.Methods.Based on a prototype system previously used in pre-clinical research, we replaced the external cables of a 192-electrode iBCI with wireless transmitters and achieved high-resolution recording and decoding of broadband field potentials and spiking activity from people with paralysis. Two participants in an ongoing pilot clinical trial completed on-screen item selection tasks to assess iBCI-enabled cursor control.Results:Communication bitrates were equivalent between cabled and wireless configurations. Participants also used the wireless iBCI to control a standard commercial tablet computer to browse the web and use several mobile applications. Within-day comparison of cabled and wireless interfaces evaluated bit error rate, packet loss, and the recovery of spike rates and spike waveforms from the recorded neural signals. In a representative use case, the wireless system recorded intracortical signals from two arrays in one participant continuously through a 24-hour period at home.Significance.Wireless multi-electrode recording of broadband neural signals over extended periods introduces a valuable tool for human neuroscience research and is an important step toward practical deployment of iBCI technology for independent use by individuals with paralysis. On-demand access to high-performance iBCI technology in the home promises to enhance independence and restore communication and mobility for individuals with severe motor impairment.

Benefits of deep learning classification of continuous noninvasive brain–computer interface control

Objective. Noninvasive brain–computer interfaces (BCIs) assist paralyzed patients by providing access to the world without requiring surgical intervention. Prior work has suggested that EEG motor imagery based BCI can benefit from increased decoding accuracy through the application of deep learning methods, such as convolutional neural networks (CNNs). Approach. Here, we examine whether these improvements can generalize to practical scenarios such as continuous control tasks (as opposed to prior work reporting one classification per trial), whether valuable information remains latent outside of the motor cortex (as no prior work has compared full scalp coverage to motor only electrode montages), and the existing challenges to the practical implementation of deep-learning based continuous BCI control. Main results. We report that: (1) deep learning methods significantly increase offline performance compared to standard methods on an independent, large, and longitudinal online motor imagery BCI dataset with up to 4-classes and continuous 2D feedback; (2) our results suggest that a variety of neural biomarkers for BCI, including those outside the motor cortex, can be detected and used to improve performance through deep learning methods, and (3) tuning neural network output will be an important step in optimizing online BCI control, as we found the CNN models trained with full scalp EEG also significantly reduce the average trial length in a simulated online cursor control environment. Significance. This work demonstrates the benefits of CNNs classification during BCI control while providing evidence that electrode montage selection and the mapping of CNN output to device control will be important design choices in CNN based BCIs.

An artificial intelligence that increases simulated brain–computer interface performance

Objective. Brain–computer interfaces (BCIs) translate neural activity into control signals for assistive devices in order to help people with motor disabilities communicate effectively. In this work, we introduce a new BCI architecture that improves control of a BCI computer cursor to type on a virtual keyboard. Approach. Our BCI architecture incorporates an external artificial intelligence (AI) that beneficially augments the movement trajectories of the BCI. This AI-BCI leverages past user actions, at both long (100 s of seconds ago) and short (100 s of milliseconds ago) timescales, to modify the BCI’s trajectories. Main results. We tested our AI-BCI in a closed-loop BCI simulator with nine human subjects performing a typing task. We demonstrate that our AI-BCI achieves: (1) categorically higher information communication rates, (2) quicker ballistic movements between targets, (3) improved precision control to ‘dial in’ on targets, and (4) more efficient movement trajectories. We further show that our AI-BCI increases performance across a wide control quality spectrum from poor to proficient control. Significance. This AI-BCI architecture, by increasing BCI performance across all key metrics evaluated, may increase the clinical viability of BCI systems.

Effects of Peripheral Haptic Feedback on Intracortical Brain-Computer Interface Control and Associated Sensory Responses in Motor Cortex.

Intracortical brain-computer interfaces (iBCIs) provide people with paralysis a means to control devices with signals decoded from brain activity. Despite recent impressive advances, these devices still cannot approach able-bodied levels of control. To achieve naturalistic control and improved performance of neural prostheses, iBCIs will likely need to include proprioceptive feedback. With the goal of providing proprioceptive feedback via mechanical haptic stimulation, we aim to understand how haptic stimulation affects motor cortical neurons and ultimately, iBCI control. We provided skin shear haptic stimulation as a substitute for proprioception to the back of the neck of a person with tetraplegia. The neck location was determined via assessment of touch sensitivity using a monofilament test kit. The participant was able to correctly report skin shear at the back of the neck in 8 unique directions with 65% accuracy. We found motor cortical units that exhibited sensory responses to shear stimuli, some of which were strongly tuned to the stimuli and well modeled by cosine-shaped functions. In this article, we also demonstrated online iBCI cursor control with continuous skin-shear feedback driven by decoded command signals. Cursor control performance increased slightly but significantly when the participant was given haptic feedback, compared to the purely visual feedback condition.

Electrooculogram (EOG) based Mouse Cursor Controller Using the Continuous Wavelet Transform and Statistic Features

This study design a system prototype to control a mouse cursor's movement on a computer using an electrooculogram (EOG) signal. The EOG signal generated from eye movement was processed utilizing a microcontroller with an analog to the digital conversion process, which communicates with the computer through a USB port. The signal was decomposed using continuous wavelet transform (CWT), followed by feature extraction processes using statistic calculation, and then classified using K-Nearest Neighbors (k-NN) to decide the movement and direction of the mouse cursor. The test was carried out with 110 EOG signals then separated, 0.5 as training data and 0.5 as test data with eight categories of directional movement patterns, including up, bottom, right, left, top right, top left, bottom right bottom left. The highest accuracy that can be achieved using CWT-bump and kurtosis is 100%, while the time needed to translate the eye movement to the cursor movement is 1.9792 seconds. It is hoped that the proposed system can help assistive devices, particularly for Amyotrophic Lateral Sclerosis (ALS) sufferers.&#x0D;

Effects of Augmented Feedback and Motor Learning Adaptation on Human–Automation Interaction Factors

Increased interest in body–machine interfaces necessitates understanding how to train users to use nontraditional inputs. In this study, a control task driven by subject-activated surface electromyography was developed as a testbed to observe the effects of automated training methodologies on the development of performance, workload, and trust. Forty-eight subjects learned to use a surface-electromyography-based command system to perform a Fitts’s-law-style cursor-to-target task with 120 training trials and 40 evaluation trials. Subjects were divided into four groups: control, concurrent feedback, terminal feedback, and adaptive threshold. The control group trained and learned through repetition using the visual feedback of the cursor position. The concurrent feedback group received additional concurrent visual feedback during command input, and the terminal feedback group had supplementary visual feedback after command input. The adaptive threshold group did not have any additional feedback, but experienced changes in the cursor control designed to induce motor learning adaptation. The results indicate that 1) additional visual feedback improves task performance, workload, and trust during training, and 2) the groups converged in their command proficiency by the end of training.

Effects of Long-Term Meditation Practices on Sensorimotor Rhythm-Based Brain-Computer Interface Learning.

Sensorimotor rhythm (SMR)-based brain–computer interfaces (BCIs) provide an alternative pathway for users to perform motor control using motor imagery. Despite the non-invasiveness, ease of use, and low cost, this kind of BCI has limitations due to long training times and BCI inefficiency—that is, the SMR BCI control paradigm may not work well on a subpopulation of users. Meditation is a mental training method to improve mindfulness and awareness and is reported to have positive effects on one’s mental state. Here, we investigated the behavioral and electrophysiological differences between experienced meditators and meditation naïve subjects in one-dimensional (1D) and two-dimensional (2D) cursor control tasks. We found numerical evidence that meditators outperformed control subjects in both tasks (1D and 2D), and there were fewer BCI inefficient subjects in the meditator group. Finally, we also explored the neurophysiological difference between the two groups and showed that the meditators had a higher resting SMR predictor, more stable resting mu rhythm, and a larger control signal contrast than controls during the task.

Developing a Novel Hands-Free Interaction Technique Based on Nose and Teeth Movements for Using Mobile Devices

Human-mobile interaction is aimed at facilitating interaction with the smartphone devices. The conventional way to interact with mobile devices is through manual input where most of the applications are made assuming that the end user has full control over their hand movements. However, this assumption excludes people who are unable to use their hands or have suffered limb damage. In this paper, we proposed a nose and teeth based interaction system, which allows the users to control their mobile devices completely hands free. The proposed system uses the front facing camera of the smartphone to track the position of the nose for cursor control on the smartphone screen. The system detects teeth for performing the touch screen events such as tap, scroll, long press, and drag. Viola-Jones algorithm is used to detect the face and teeth based on the Haar features. After detecting the face, the nose position is calculated and tracked continuously using Lucas Kanade’s method for optical flow estimation. All the touch screen events have been implemented in the system so that the user can execute all the operations of the smartphone. To evaluate the performance and the effect of (smartphone) device type on the execution time, the proposed system was installed in 3 smartphone devices and 7 trials for each device were performed by 3 different able-bodied elderly persons. The result shows a significant success rate for the detection of nose and teeth, and for the execution of the operations. The execution time of each operation slightly varies by 0.72s on average because of the configuration of the smartphones.

Performance Analysis of a Head and Eye Motion-Based Control Interface for Assistive Robots.

Assistive robots support people with limited mobility in their everyday life activities and work. However, most of the assistive systems and technologies for supporting eating and drinking require a residual mobility in arms or hands. For people without residual mobility, different hands-free controls have been developed. For hands-free control, the combination of different modalities can lead to great advantages and improved control. The novelty of this work is a new concept to control a robot using a combination of head and eye motions. The control unit is a mobile, compact and low-cost multimodal sensor system. A Magnetic Angular Rate Gravity (MARG)-sensor is used to detect head motion and an eye tracker enables the system to capture the user’s gaze. To analyze the performance of the two modalities, an experimental evaluation with ten able-bodied subjects and one subject with tetraplegia was performed. To assess discrete control (event-based control), a button activation task was performed. To assess two-dimensional continuous cursor control, a Fitts’s Law task was performed. The usability study was related to a use-case scenario with a collaborative robot assisting a drinking action. The results of the able-bodied subjects show no significant difference between eye motions and head motions for the activation time of the buttons and the throughput, while, using the eye tracker in the Fitts’s Law task, the error rate was significantly higher. The subject with tetraplegia showed slightly better performance for button activation when using the eye tracker. In the use-case, all subjects were able to use the control unit successfully to support the drinking action. Due to the limited head motion of the subject with tetraplegia, button activation with the eye tracker was slightly faster than with the MARG-sensor. A further study with more subjects with tetraplegia is planned, in order to verify these results.

Spatial-temporal aspects of continuous EEG-based neurorobotic control

Objective. The goal of this work is to identify the spatio-temporal facets of state-of-the-art electroencephalography (EEG)-based continuous neurorobotics that need to be addressed, prior to deployment in practical applications at home and in the clinic. Approach. Nine healthy human subjects participated in five sessions of one-dimensional (1D) horizontal (LR), 1D vertical (UD) and two-dimensional (2D) neural tracking from EEG. Users controlled a robotic arm and virtual cursor to continuously track a Gaussian random motion target using EEG sensorimotor rhythm modulation via motor imagery (MI) commands. Continuous control quality was analyzed in the temporal and spatial domains separately. Main results. Axis-specific errors during 2D tasks were significantly larger than during 1D counterparts. Fatigue rates were larger for control tasks with higher cognitive demand (LR, left- and right-hand MI) compared to those with lower cognitive demand (UD, both hands MI and rest). Additionally robotic arm and virtual cursor control exhibited equal tracking error during all tasks. However, further spatial error analysis of 2D control revealed a significant reduction in tracking quality that was dependent on the visual interference of the physical device. In fact, robotic arm performance was significantly greater than that of virtual cursor control when the users’ sightlines were not obstructed. Significance. This work emphasizes the need for practical interfaces to be designed around real-world tasks of increased complexity. Here, the dependence of control quality on cognitive task demand emphasizes the need for decoders that facilitate the translation of 1D task mastery to 2D control. When device footprint was accounted for, the introduction of a physical robotic arm improved control quality, likely due to increased user engagement. In general, this work demonstrates the need to consider both the physical footprint of devices, the complexity of training tasks, and the synergy of control strategies during the development of neurorobotic control.

A hybrid Body-Machine Interface integrating signals from muscles and motions

Objective. Body-Machine Interfaces (BoMIs) establish a way to operate a variety of devices, allowing their users to extend the limits of their motor abilities by exploiting the redundancy of muscles and motions that remain available after spinal cord injury or stroke. Here, we considered the integration of two types of signals, motion signals derived from inertial measurement units (IMUs) and muscle activities recorded with electromyography (EMG), both contributing to the operation of the BoMI. Approach. A direct combination of IMU and EMG signals might result in inefficient control due to the differences in their nature. Accordingly, we used a nonlinear-regression-based approach to predict IMU from EMG signals, after which the predicted and actual IMU signals were combined into a hybrid control signal. The goal of this approach was to provide users with the possibility to switch seamlessly between movement and EMG control, using the BoMI as a tool for promoting the engagement of selected muscles. We tested the interface in three control modalities, EMG-only, IMU-only and hybrid, in a cohort of 15 unimpaired participants. Participants practiced reaching movements by guiding a computer cursor over a set of targets. Main results. We found that the proposed hybrid control led to comparable performance to IMU-based control and significantly outperformed the EMG-only control. Results also indicated that hybrid cursor control was predominantly influenced by EMG signals. Significance. We concluded that combining EMG with IMU signals could be an efficient way to target muscle activations while overcoming the limitations of an EMG-only control.