Human-computer Interface System Research Articles

EOG Based Text and Voice Controlled Remote Interpreter for Quadriplegic Patients

Electrooculography is considered as one of the significant electro-physiological signals. These signals carry data of eye movements which can be employed in human-computer interface (HCL) as a control signal. This project focuses on creating a text and voice-based interpreter for quadriplegic patients using electrooculography (EOG) signals. EOG is a technique that measures the electrical activity of the eye muscles responsible for eye movements and can be used to track changes in eye location to reveal information about human eye activities. The EOG signal is commonly used in human-computer interface (HCI) systems as an alternative input for patients suffering from quadriplegia, ALS, and locked-in syndrome. The BioAmp EXG Pill Sensor is used to acquire EOG signals of left and right eye movement, as well as up and down eye movement. The signals are processed using an ESP32 microcontroller and Arduino IDE, and an algorithm is created to analyze the observed ranges and generate text and voice-based outputs. The accuracy of the system was tested by asking 10 healthy participants to perform each of the four types of motions ten times, and the results showed an overall accuracy of 81.04%. The system involves detecting EOG signals using sensors that are placed around the patient's eyes, and the text-based output is displayed on an LCD screen, while the voice-based output is played on an MP3 player. The output is then displayed on an application enabling communication with the patient remotely, potentially improving the quality of care and increasing the patient’s sense of security. Future developments could include increasing the degrees of motion and addition of an eye-blink sensor for more convenient user experience. This project provides a valuable solution for quadriplegic patients, enabling them to communicate effectively and empowering them with a sense of independence. However, further research and testing are needed to fully evaluate the efficacy of the system on actual quadriplegic patients.

Development of electrooculogram based human computer interface system using deep learning

The patients with diseases that cause severe movement disabilities was noticeably increasing. These disabilities made patients unable to carry out their daily activities or interact with their external environment. However, the existence of human-computer interfaces (HCI) gave those patients a new hope to be able to interact once again. HCI enabled these patients to communicate with their environment by recognizing the movement of their eyes. Eye movements are recorded by an electro-oculogram (EOG) through some electrodes that are put vertically and horizontally on the eyes. In this paper, EOG vertical and horizontal signals were analyzed to detect six eye movements (up, down, right, left, double blinking, and center). Three deep learning models namely convolution neural network (CNN), visual geometry group (VGG), and inception had been examined on filtered EOG signals. The experimental results reveal the superiority of the inception model in providing the best average accuracy 96.4%. Accordingly, a writing system is presented based on the detected movements.

Development of electrooculogram based human computer interface system using deep learning

The patients with diseases that cause severe movement disabilities was noticeably increasing. These disabilities made patients unable to carry out their daily activities or interact with their external environment. However, the existence of human-computer interfaces (HCI) gave those patients a new hope to be able to interact once again. HCI enabled these patients to communicate with their environment by recognizing the movement of their eyes. Eye movements are recorded by an electro-oculogram (EOG) through some electrodes that are put vertically and horizontally on the eyes. In this paper, EOG vertical and horizontal signals were analyzed to detect six eye movements (up, down, right, left, double blinking, and center). Three deep learning models namely convolution neural network (CNN), visual geometry group (VGG), and inception had been examined on filtered EOG signals. The experimental results reveal the superiority of the inception model in providing the best average accuracy 96.4%. Accordingly, a writing system is presented based on the detected movements.

Identification of emotions evoked by music via spatial-temporal transformer in multi-channel EEG signals.

Emotion plays a vital role in understanding activities and associations. Due to being non-invasive, many experts have employed EEG signals as a reliable technique for emotion recognition. Identifying emotions from multi-channel EEG signals is evolving into a crucial task for diagnosing emotional disorders in neuroscience. One challenge with automated emotion recognition in EEG signals is to extract and select the discriminating features to classify different emotions accurately. In this study, we proposed a novel Transformer model for identifying emotions from multi-channel EEG signals. Note that we directly fed the raw EEG signal into the proposed Transformer, which aims at eliminating the issues caused by the local receptive fields in the convolutional neural networks. The presented deep learning model consists of two separate channels to address the spatial and temporal information in the EEG signals, respectively. In the experiments, we first collected the EEG recordings from 20 subjects during listening to music. Experimental results of the proposed approach for binary emotion classification (positive and negative) and ternary emotion classification (positive, negative, and neutral) indicated the accuracy of 97.3 and 97.1%, respectively. We conducted comparison experiments on the same dataset using the proposed method and state-of-the-art techniques. Moreover, we achieved a promising outcome in comparison with these approaches. Due to the performance of the proposed approach, it can be a potentially valuable instrument for human-computer interface system.

Design of subject independent 3D VAD emotion detection system using EEG signals and machine learning algorithms

This work aims to develop a subject-independent Emotion Detection System (EDS) based on EEG signals and the 3D Valence-Arousal-Dominance (VAD) model. The DEAP database physiological signals are considered for the system design. A multi-domain feature extraction is performed using the Wavelet-based Atomic Function time–frequency domain technique; and various time and frequency domain feature extraction techniques. Further, principal component analysis reduces the data dimensionality and redundancy in the obtained feature set. The minimal feature set is analysed using machine learning classifiers, i.e., gradient boosting, decision tree, and random forest. Additionally, the hyperparameters of machine learning algorithms are tuned using Optuna to improve the performance of the proposed model. Three EDS are designed in this work; EDS1 considers the 3D VAD model for three class classifications. 2D VA model is used in EDS2 to determine 9 discrete emotions, and EDS3 detects 12 discrete emotions using the 3D VAD model. Results reveal the highest classification accuracy of about 99% with EDS1, whereas an average accuracy of 99.82% and 98.44% is obtained with EDS2 and EDS3, respectively. The results reveal that both EDS1 and EDS2 show improvement as compared to the existing literature. Also, the proposed EDS3 provides the classification of the highest number of discrete emotions, which may facilitate the design of an efficient human–computer interface system.

Monopolar and bipolar electrooculography signal characteristics due to target displacements—have we seen the whole picture?

The development of electrooculography (EOG)-based human-computer interface systems is generally based on the processing of the commonly referred to horizontal and vertical bipolar EOG channels, which are computed from a horizontally-aligned and another vertically-aligned pair of electrodes, respectively. Horizontal (vertical) target displacements are assumed to result in changes in the horizontal (vertical) EOG channel only, and any cross-talk between the bipolar channels is often neglected or incorrectly attributed solely to electrode misalignment with respect to the ocular rotation axes. Objective. The aim of this work is to demonstrate that such cross-talk is intrinsic to the geometric relationship between the orientation of the verging ocular globes and the planar displacement of the gaze target with respect to the primary gaze position. Approach. Since it is difficult to record actual EOG data with electrodes which are perfectly-aligned with the ocular rotation axes, this is studied by simulating the EOG potential values for various horizontally- and vertically-displacing targets using a dipole model of the eye. Main results. We show that cross-talk between the horizontal and vertical bipolar EOG channels is manifested even if the electrodes are aligned with the ocular rotation axes. Specifically, for a horizontally- (vertically-)displaced target, while the monopolar EOG signals obtained from the horizontally- (vertically-)aligned electrodes exhibit an expected predominant potential displacement, a smaller displacement is also exhibited in the monopolar EOG signals obtained from the vertically- (horizontally-)aligned electrodes. These unexpected displacements in the vertically- (horizontally-)aligned monopolar channels may have different magnitudes, resulting in an effective potential displacement in the vertical (horizontal) bipolar EOG channel. Significance. This is significant as it shows that, unlike in many works published so far for EOG-based ocular pose estimation, it is not sufficient to only use the horizontal (vertical) bipolar EOG channel to estimate the horizontal (vertical) displacement of the ocular pose.

Fish Swarm Optimized Deep Hopfield Neural Network-Assisted HCI System for Augmentative Communication Using a Visual Feedback System

In this context, the uses of computers, the Human-computer interface (HCI) system, can assist interaction on-demand services. HCI method facilitates ventilated patients to interact with computers about their needs using their brain’s electrical activity. To accomplish this, an HCI framework is developed in this research to facilitate visual feedback system (VFS) using an augmentative communication approach. Augmentative communication (AC.) or icon-based services are incorporated with a portable monitor placed in front of a patient; they can look at the screen to select (ask) their appropriate needs-related icons. The services have been achieved by capturing and processing patients’ electromagnetic brain activates during the icon selection by their eye flickering moment recording using wearable Electroencephalogram (EEG). The flickering icons on the screen conveying an appropriate message to the monitoring unit computer, and the monitoring unit can respond to the patient’s request using VFS. The HCI system is comprised of the following methodologies to achieve augmentative communication-based services such as EEG signals acquisition, filtering, partition-based feature extraction, and fusion and fish swarm optimized Deep Hopfield neural network FSODHNN based classifier. The evolution results of the VSF based HCI framework are demonstrated successfully. It obtained the highest accuracy of 99.11%, specificity of 99.05%, the sensitivity of 99.09%, and the lowest RMSE of 0.98, MSE of 0.92 in icon identification/ selection.

Intelligent Alarm System-Based Devices Designed for People with Disabilities, Caused by Various Chronic Diseases

Context: Chronic diseases (CD) are defined as symptoms or disabilities, caused by diseases, genetic factors, and injury requiring long-term treatment. Intelligent alarm systems, which collect patient health data and transfer it to a medical server, help track and avoid future incidents. Method: The search terms were “computer network” OR “information systems” OR “wireless technology” OR “decision support systems” AND “chronic disease” OR “chronic disease” in major electronic databases, including Pubmed/Medline, Scopus, Embase, ISI Web of Science, and Cochrane Central. Results: The search resulted in 1275 articles with 11 specific to intelligence-based systems in chronic medical conditions until 08 June 2021. The creation of different access levels for care providers in the system and application customization according to CD conditions were the goals that can be achieved in future research. The human-computer interface (HCI) systems, smart home, and software, such as Fitbit using IoMT to monitor health metrics in people with different CDs, are introduced so far. Conclusions: These systems, if provided on the web and mobile platform, can be accessed at any time and place and are more efficient. Finally, the combination of clinical decision support systems with artificial intelligence has beneficial effects on physician's systems, increases the accuracy in CD diagnosis, and improves the pain management. This intelligent system demonstrates factors influencing back to work and allows identifying high-risk patients and their potential to handle activities of daily living.

A Transfer Learning Model for Gesture Recognition Based on the Deep Features Extracted by CNN

The surface electromyogram (sEMG) based hand gesture recognition is prevalent in human–computer interface systems. However, the generalization of the recognition model does not perform well on cross-subject and cross-day. Transfer learning, which applies the pretrained model to another task, has demonstrated its effectiveness in solving this kind of problem. In this regard, this article first proposes a multiscale kernel convolutional neural network (MKCNN) model to extract and fuse multiscale features of the multichannel sEMG signals. Based on the proposed MKCNN model, a transfer learning model named TL-MKCNN combines the MKCNN and its Siamese network by a custom distribution normalization module (DNM) and a distribution alignment module (DAM) to realize domain adaptation. The DNM can cluster the deep features extracted from different domains to their category center points embedded in the feature space, and the DAM further aligns the overall distribution of the deep features from different domains. The MKCNN model and the TL-MKCNN model are tested on various benchmark databases to verify the effectiveness of the transfer learning framework. The experimental results show that, on the benchmark database NinaPro DB6, the average accuracies of TL-MKCNN can achieve 97.22% on within-session, 74.48% on cross-subject, and 90.30% on cross-day, which are 4.31%, 11.58%, and 5.51% higher than those of the MKCNN model on within-session, cross-subject, and cross-day, respectively. Compared with the state-of-the-art works, the TL-MKCNN obtains 13.38% and 37.88% accuracy improvement on cross-subject and cross-day, respectively.

Design of a Wearable Eye-Movement Detection System Based on Electrooculography Signals and Its Experimental Validation.

In the assistive research area, human–computer interface (HCI) technology is used to help people with disabilities by conveying their intentions and thoughts to the outside world. Many HCI systems based on eye movement have been proposed to assist people with disabilities. However, due to the complexity of the necessary algorithms and the difficulty of hardware implementation, there are few general-purpose designs that consider practicality and stability in real life. Therefore, to solve these limitations and problems, an HCI system based on electrooculography (EOG) is proposed in this study. The proposed classification algorithm provides eye-state detection, including the fixation, saccade, and blinking states. Moreover, this algorithm can distinguish among ten kinds of saccade movements (i.e., up, down, left, right, farther left, farther right, up-left, down-left, up-right, and down-right). In addition, we developed an HCI system based on an eye-movement classification algorithm. This system provides an eye-dialing interface that can be used to improve the lives of people with disabilities. The results illustrate the good performance of the proposed classification algorithm. Moreover, the EOG-based system, which can detect ten different eye-movement features, can be utilized in real-life applications.

Human Reliability Evaluation Based on Objective and Subjective Comprehensive Method Used for Ergonomic Interface Design

In the process of machining, the operator’s incorrect operation using the control human-computer interaction interface may lead to a disastrous effect on the whole production chain such as task failure and system failure. The eight types of using errors about human-computer interaction interface system (misperception, memory lapse, carelessness, improper decision-making, improper operation, improper interface layout design, improper icon and text display design, and improper prompt feedback) are proposed based on the cognition from users. According to the eight types of using error, the human reliability and safety evaluation method is proposed. Considering the variability objective and uncertainty subjective factors in the human-machine interaction process, the human reliability evaluation method is proposed based on objective and subjective comprehensive weighting under ergonomic principles. Firstly, based on the investigation of relevant machine tool interface design, the comprehensive evaluation index system for man-machine reliability of complex interface design is established. Secondly, an objective and subjective comprehensive weight assignment method as the composition of Theilʼs entropy weight method combined with TOPSIS (Technique for Order Preference by Similarity to an Ideal Solution) and improved AHP (analytic hierarchy process) method combined with fuzzy mathematics is proposed. It can not only effectively avoid the impact of excessive subjective factors on empowerment but also fully excavate the real information in the survey data and improve the objectivity and accuracy of the weight data of evaluation indicators at all levels. Thirdly, taking the interface design of CNC machine tool equipment for example, the best human-computer interaction interface design schemes are chosen according to the human reliability evaluation method based on objective and subjective comprehensive method calculated by Entropy-TOPSIS-AHP. Finally, based on eye movement experiments, the use efficiency and accuracy of the human-computer interaction interface design scheme chosen by the assessment method are better than the other design schemes or original one to verify the assessment method validity.

Emotion Recognition from ECG Signals Using Wavelet Scattering and Machine Learning

Affect detection combined with a system that dynamically responds to a person’s emotional state allows an improved user experience with computers, systems, and environments and has a wide range of applications, including entertainment and health care. Previous studies on this topic have used a variety of machine learning algorithms and inputs such as audial, visual, or physiological signals. Recently, a lot of interest has been focused on the last, as speech or video recording is impractical for some applications. Therefore, there is a need to create Human–Computer Interface Systems capable of recognizing emotional states from noninvasive and nonintrusive physiological signals. Typically, the recognition task is carried out from electroencephalogram (EEG) signals, obtaining good accuracy. However, EEGs are difficult to register without interfering with daily activities, and recent studies have shown that it is possible to use electrocardiogram (ECG) signals for this purpose. This work improves the performance of emotion recognition from ECG signals using wavelet transform for signal analysis. Features of the ECG signal are extracted from the AMIGOS database using a wavelet scattering algorithm that allows obtaining features of the signal at different time scales, which are then used as inputs for different classifiers to evaluate their performance. The results show that the proposed algorithm for extracting features and classifying the signals obtains an accuracy of 88.8% in the valence dimension, 90.2% in arousal, and 95.3% in a two-dimensional classification, which is better than the performance reported in previous studies. This algorithm is expected to be useful for classifying emotions using wearable devices.

Low power CMOS based Self Controlled Precharge Free Content Addressable Memory with Minimum area for Image Processing Devices

Image processing devices plays a vital role in several applications like medical, security, biometric etc. The devices ranges from portable size to larger machines with and without Human Computer Interface possibilities. As the image processing and human computer interface system application requires higher memory requirements, the power and area should be small. Searching of data is a high priority work in image classification. To perform high speed search through hardware Content Addressable Memory is used. But the circuit suffers from higher power consumption, precharging issues and low performance. For longer word length the elimination of precharge is needed. So for high speed applications self-controlled precharge-free CAM (SCPF-CAM) is suitable. A 4T hybrid self controlled pre charge free Content Addressable Memory is proposed in this paper using CMOS 32nm technology. The observation shows that the circuit works at high speed, minimizes the search time and has high performance operation. When compared to the conventional SCPF-CAM, 8T CAM the proposed design reduces the number of transistors. The reduction in area is about approximately 20% and can be used in low power and low energy applications. In Synopsis HSPICE Predictive technology models were used for the implementation in 32nm CMOS technology. The work will be extended in future using FinFET technology where the leakage current can be minimized.

EOG-Based HCI System for Quadcopter Navigation

Human–computer interface (HCI) systems are extending their boundaries in our daily life and becoming an important subject in biomedical engineering. Electrooculogram (EOG) signal as an input for such systems stems from the corneoretinal standing potential, which can be used for monitoring human eye rotation. Higher amplitude, better signal-to-noise ratio, and much easier recording conditions compared with electroencephalography, make it an important input modality for HCI systems. In this article, real-time processing and cost-effective (< 100$) HCI system was designed and developed based on the EOG signals. The required electrodes were embedded in updated eyeglasses for easy electrode placement over the subject’s face. EOG signals were acquired by the subject’s eye movement toward the four middle parts of the screen edges of a laptop placed in front of them. The system training for each subject ameliorated system withstanding against the blink and wrinkle artifact. Finally, the required commands for quadcopter navigation (up, down, left, and right) generated with 0.6-s total delay and 94.8% of system accuracy in detecting the correct eye movements. A real quadcopter navigation experiment based on the standard navigation interface and the developed HCI system represented the system accuracy, robustness, and usefulness.

Multi-reservoirs EEG signal feature sensing and recognition method based on generative adversarial networks

EEG emotion recognition is one of the interesting and challenging tasks in the research based emotion human–computer interface system. In this paper, a multi-reservoirs feature coding continuous label fusion semi-supervised Generative Adversarial Networks (MCLFS-GAN) is proposed by using permutation phase transfer entropy as the EEG signal feature. Firstly, the obtained features are encapsulated in time series, and then the features are sent into multi-reservoirs according to the division of brain intra, brain interval or frequency band. After convolution optimization, the feature expression with time sequence relationship is obtained. The generic representation between the features and pseudo effective feature expression are iteratively learned in encoder E and generator G in the generative adversarial way. In addition, the continuous fusion for class intra tags can help to form continuous differences between classes. The experimental results show that the accuracy for the four classification is 81.32% and 54.87% respectively by using SAP and LOSO in DEAP database. Compared with other models, this algorithm can effectively improve the recognition performance.

Empirical mean curve decomposition with multiwavelet transformation for eye movements recognition using electrooculogram signals.

Many research works are in progress in classification of the eye movements using the electrooculography signals and employing them to control the human-computer interface systems. This article introduces a new model for recognizing various eye movements using electrooculography signals with the help of empirical mean curve decomposition and multiwavelet transformation. Furthermore, this article also adopts a principal component analysis algorithm to reduce the dimension of electrooculography signals. Accordingly, the dimensionally reduced decomposed signal is provided to the neural network classifier for classifying the electrooculography signals, along with this, the weight of the neural network is fine-tuned with the assistance of the Levenberg-Marquardt algorithm. Finally, the proposed method is compared with the existing methods and it is observed that the proposed methodology gives the better performance in correspondence with accuracy, sensitivity, specificity, precision, false positive rate, false negative rate, negative predictive value, false discovery rate, F1 score, and Mathews correlation coefficient.

Optimised multi‐wavelet domain for decomposed electrooculogram‐based eye movement classification

The human eye movement tracking is possible with the assistance of the electrooculography (EOG) signals. The human eye tracking system allows researchers to analyse the participant's eye movements during certain activities. This study offers the EOG signals to control the human–computer interface systems with the help of Empirical Mean Curve Decomposition (EMCD) decomposition model. At first, the input EOG signal is provided as input to the EMCD decomposition model, later the resultant signal is given to principal component analysis for dimensional reduction, and then the dimensional reduced signal is offered to multi-wavelet decomposition model. The resultant dimensionally reduced multi-wavelet decomposed signal is passed to the proposed Feature Mapping (FM) model, using the k-means clustering model. Then, the Grey Wolf Optimization (GWO) algorithm is utilised to tune the margin. Next to mapping, the obtained features are provided to the nearest neighbour classifier, to obtain the eye movement. Next to the implementation, the proposed method is compared with the existing methods, and it is witnessed that the proposed methodology gives the superior performance in correspondence with accuracy, sensitivity, specificity, precision, false positive rate, false negative rate, negative predictive value, false discovery rate, F1 score and Mathews correlation coefficient.

Open Software/Hardware Platform for Human-Computer Interface Based on Electrooculography (EOG) Signal Classification.

Electrooculography (EOG) signals have been widely used in Human-Computer Interfaces (HCI). The HCI systems proposed in the literature make use of self-designed or closed environments, which restrict the number of potential users and applications. Here, we present a system for classifying four directions of eye movements employing EOG signals. The system is based on open source ecosystems, the Raspberry Pi single-board computer, the OpenBCI biosignal acquisition device, and an open-source python library. The designed system provides a cheap, compact, and easy to carry system that can be replicated or modified. We used Maximum, Minimum, and Median trial values as features to create a Support Vector Machine (SVM) classifier. A mean of 90% accuracy was obtained from 7 out of 10 subjects for online classification of Up, Down, Left, and Right movements. This classification system can be used as an input for an HCI, i.e., for assisted communication in paralyzed people.

Detecting Affect States Using VGG16, ResNet50 and SE-ResNet50 Networks

Affect detection is a key component in developing intelligent human computer interface systems. State-of-the-art affect detection systems assume the availability of full un-occluded face images. This work uses convolutional neural networks with transfer learning to detect 7 basic affect states, viz. Angry, Contempt, Disgust, Fear, Happy and Sad. The paper compares three pre-trained networks, viz. VGG16, ResNet50 and a SE-ResNet50, in which a new architectural block of squeeze and excitation has been integrated with ResNet50. Modified VGG-16, ResNet50 and SE-ResNet50 networks are trained on images from the dataset, and the results are compared. We have been able to achieve validation accuracies of 96.8%, 99.47%, and 97.34% for VGG16, ResNet50 and SE-ResNet50, respectively. Apart from accuracy, the other performance matrices used in this work are precision and recall. Our evaluation, based on these performance matrices, shows that accurate affect detection is obtained from all the three networks with Resnet50 being the most accurate.

Affect State Classification from Face Segments Using Resnet-50 and SE-Resnet-50

One of the important components of an intelligent Human computer Interface system is accurate classification of the various affect states. Such interface systems are however plagued by a recurring problem of image occlusion. The challenge hence is to be able to classify the various affect states accurately from whatever portions of the face are available to the system. This paper attempts to investigate if there are segments within the facial region which carry sufficient information about the affect states. In this paper we have used two pre-defined Convolutional Neural networks (CNN). We have implemented a ResNet-50 network and a modified version of ResNet-50 which has a Squeeze and Excitation network connected to ResNet-50. This is called SE-ResNet-50. We use these two networks to classify seven basic affect states of Angry, Contempt, Disgust, Fear, Happy, Sad and Surprise from various segments of the face. We partition the face into four regions with each region comprising of only 50% of the original data. The results obtained are compared with that obtained using the full face. The validation accuracy values are obtained for full face as well as the four segments of the face. The paper also calculates precision and recall for each partitioned area for each of the affect states using the two networks. Our evaluation shows that both, ResNet-50 as well as SE-ResNet-50 are successful in accurately classifying all the 7 affect state from the Right segment, Left segment Lower segment and Upper segment of the face. While ResNet-50 performs marginally better compared to the SE-ResNet-50 in identifying the various affect states form the right, left and lower segments of the face, SE-ResNet-50 performs better in identifying the affect states from the upper segment of the face. We can thus conclude that right segment, left segment, lower segment and upper segments of the face contain sufficient information to correctly classify the seven affect states. The experimental results presented in this paper show that pre-defined Convolutional Neural Networks gives us very high accuracy, precision and recall values and hence can be used to accurately classify affect states even when there are occlusions present in the image and only certain portions of the face are available for analysis.