Human Computer Interface Applications Research Articles

Formation of Anisotropic Conducting Interlayer for High-Resolution Epidermal Electromyography Using Mixed-Conducting Particulate Composite.

Epidermal electrophysiology is a non-invasive method used in research and clinical practices to study the electrical activity of the brain, heart, nerves, and muscles. However, electrode/tissue interlayer materials such as ionically conducting pastes can negatively affect recordings by introducing lateral electrode-to-electrode ionic crosstalk and reducing spatial resolution. To overcome this issue, biocompatible, anisotropic-conducting interlayer composites (ACI) that establish an electrically anisotropic interface with the skin are developed, enabling the application of dense cutaneous sensor arrays. High-density, conformable electrodes are also microfabricated that adhere to the ACI and follow the curvilinear surface of the skin. The results show that ACI significantly enhances the spatial resolution of epidermal electromyography (EMG) recording compared to conductive paste, permitting the acquisition of single muscle action potentials with distinct spatial profiles. The high-density EMG in developing mice, non-human primates, and humans is validated. Overall, high spatial-resolution epidermal electrophysiology enabled by ACI has the potential to advance clinical diagnostics of motor system disorders and enhance data quality for human-computer interface applications.

Time-Frequency Image-based Speech Emotion Recognition using Artificial Neural Network

Automatic Speech Emotion Recognition (ASER) is a state-of-the-art application in artificial intelligence. Speech recognition intelligence is employed in various applications such as digital assistance, security, and other human-machine interactive products. In the present work, three open-source acoustic datasets, namely SAVEE, RAVDESS, and EmoDB, have been utilized (Haq et al., 2008, Livingstone et al., 2005, Burkhardt et al., 2005). From these datasets, six emotions namely anger, disgust, fear, happy, neutral, and sad, are selected for automatic speech emotion recognition. Various types of algorithms are already reported for extracting emotional content from acoustic signals. This work proposes a time-frequency (t-f) image-based multiclass speech emotion classification model for the six emotions mentioned above. The proposed model extracts 472 grayscale image features from the t-f images of speech signals. The t-f image is a visual representation of the time component and frequency component at that time in the two-dimensional space, and differing colors show its amplitude. An artificial neural network-based multiclass machine learning approach is used to classify selected emotions. The experimental results show that the above-mentioned emotions' average classification accuracy (CA) of 88.6%, 85.5%, and 93.56% is achieved using SAVEE, RAVDESS, and EmoDB datasets, respectively. Also, an average CA of 83.44% has been achieved for the combination of all three datasets. The maximum reported average classification accuracy (CA) using spectrogram for SAVEE, RAVDESS, and EmoDB dataset is 87.8%, 79.5 %, and 83.4%, respectively (Wani et al., 2020, Mustaqeem and Kwon, 2019, Badshah et al., 2017). The proposed t-f image-based classification model shows improvement in average CA by 0.91%, 7.54%, and 12.18 % for SAVEE, RAVDESS, and EmoDB datasets, respectively. This study can be helpful in human-computer interface applications to detect emotions precisely from acoustic signals.

Modified Earthworm Optimization With Deep Learning Assisted Emotion Recognition for Human Computer Interface

Among the most prominent field in the human-computer interface (HCI) is emotion recognition using facial expressions. Posed variations, facial accessories, and non-uniform illuminations are some of the difficulties in the emotion recognition field. Emotion detection with the help of traditional methods has the shortcoming of mutual optimization of feature extraction and classification. Computer vision (CV) technology improves HCI by visualizing the natural world in a digital platform like the human brain. In CV technique, advances in machine learning and artificial intelligence result in further enhancements and changes, which ensures an improved and more stable visualization. This study develops a new Modified Earthworm Optimization with Deep Learning Assisted Emotion Recognition (MEWODL-ER) for HCI applications. The presented MEWODL-ER technique intends to categorize different kinds of emotions that exist in the HCI applications. To do so, the presented MEWODL-ER technique employs the GoogleNet model to extract feature vectors and the hyperparameter tuning process is performed via the MEWO algorithm. The design of automated hyperparameter adjustment using the MEWO algorithm helps in attaining an improved emotion recognition process. Finally, the quantum autoencoder (QAE) model is implemented for the identification and classification of emotions related to the HCI applications. To exhibit the enhanced recognition results of the MEWODL-ER approach, a wide-ranging simulation analysis is performed. The experimental values indicated that the MEWODL-ER technique accomplishes promising performance over other models with maximum accuracy of 98.91%.

Wearable Technology vs. Implantable Technology: Evaluating Human-Computer Interface and Healthcare Applications

With an emphasis on their effects on human-computer interface (HCI) and healthcare applications, this study compares and contrasts wearable and implantable technologies. Implantable technology involves placing electronic equipment inside the body, such as pacemakers, cochlear implants, and brain interfaces. Wearable technology describes gadgets that may be worn on the body, such smartwatches, fitness trackers, and augmented reality glasses. In user experience mobility, data collecting, and healthcare applications, the study assesses the benefits and drawbacks of both strategies. They also examines the uses of wearable and implanted technologies in healthcare, such as assistive devices, tailored medicine, and disease monitoring. The study also explores the privacy and ethical ramifications of these technologies.

Noninvasive Human-Computer Interface Methods and Applications for Robotic Control: Past, Current, and Future.

The purpose of this study is to explore the noninvasive human-computer interaction methods that have been widely used in various fields, especially in the field of robot control. To have a deep understanding of the development of the methods, this paper employs “Mapping Knowledge Domains” (MKDs) to find research hotspots in the area to show the future potential development. Through the literature review, this paper found that there was a paradigm shift in the research of noninvasive BCI technologies for robotic control, which has occurred from early 2010 since the rapid development of machine learning, deep learning, and sensory technologies. This study further provides a trend analysis that the combination of data-driven methods with optimized algorithms and human-sensory-driven methods will be the key areas for the future noninvasive method development in robotic control. Based on the above findings, the paper provides a potential developing way of noninvasive HCI methods for related areas including health care, robotic system, and media.

Human-Computer Interaction with Detection of Speaker Emotions Using Convolution Neural Networks.

Emotions play an essential role in human relationships, and many real-time applications rely on interpreting the speaker's emotion from their words. Speech emotion recognition (SER) modules aid human-computer interface (HCI) applications, but they are challenging to implement because of the lack of balanced data for training and clarity about which features are sufficient for categorization. This research discusses the impact of the classification approach, identifying the most appropriate combination of features and data augmentation on speech emotion detection accuracy. Selection of the correct combination of handcrafted features with the classifier plays an integral part in reducing computation complexity. The suggested classification model, a 1D convolutional neural network (1D CNN), outperforms traditional machine learning approaches in classification. Unlike most earlier studies, which examined emotions primarily through a single language lens, our analysis looks at numerous language data sets. With the most discriminating features and data augmentation, our technique achieves 97.09%, 96.44%, and 83.33% accuracy for the BAVED, ANAD, and SAVEE data sets, respectively.

Development of a Multichannel Bioinstrumentation System for Human-Computer Interface Applications

A multi-channel measurement system used to measure electrocardiogram (ECG), electroencephalogram (EEG), electromyogram (EMG) and electrooculogram (EOG) biosignals has been designed and prototyped. The designed system has 16 configurable measurement channels. Of the 16 channels the developed system has, 8 have been designed for EEG, 2 for EMG, 2 for EOG, 1 for ECG measurements, the remaining 3 have been reserved as backup channels. In circuit design, biosignal amplifier design principles have been applied by taking into account the characteristics of the biosignal to be measured for each channel, such as bandwidth, frequency, amplitude, noise level. Modules such as instrumentation amplifier, filter, DC suppression unit, amplifier, DC level determination unit, analog-digital converter, optical isolation unit, power supply have been designed to perform biosignal measurements through these channels. Biosignals measured by the developed system can be shifted to the desired threshold level with the help of the analog output reference voltage, converted to digital data 10-bit resolution and transferred to the computer environment in real time. The data transferred to the computer can be used in C#, Excel, MATLAB, and LabVIEW platforms. The novelty of the developed system is that any of the four desired biosignal types can be measured from any channel. In addition, another feature of the system is that it can work with real-time data without being dependent on the databases serving for human-computer interface applications. In experimental studies with some researchers for the performance tests of the system, ECG, EEG, EMG and EOG signals have been recorded with different module configurations, and signal processing stages were carried out to be used for human-computer applications.

Comparing subject-to-subject transfer learning methods in surface electromyogram-based motion recognition with shallow and deep classifiers

Surface electromyogram (sEMG)-based human-computer interface (HCI) is an effective tool for detecting human movements. Because sEMG-based motion recognition usually requires prolonged data measurements from the user (target), transfer learning reusing pre-measured (source) data from other users and pre-trained classifiers can be applied to sEMG data to reduce the measurement time. However, little knowledge is available regarding the combination of transfer learning methods and classifiers in sEMG data applications. Thus, we investigated the classification accuracy of data- and parameter-space-based transfer learning with shallow or deep classifiers in cross-subject sEMG classification. The dataset contains eight classes of forearm motions recorded from 25 volunteer participants. We used a support vector machine (SVM) as a shallow classifier as well as a deep neural network architecture, referred to as an artificial neural network (ANN), as a deep classifier. In addition, we used style transfer mapping (STM) as a data-space-based transfer learning method and fine-tuning (FT) as a parameter-space-based transfer learning method. Consequently, the classification accuracy of the ANN was higher than that of the SVM, regardless of the combinational use of transfer learning. STM and FT significantly improved the classification accuracy compared with non-transfer cases regardless of the classifier (note that FT can only be used with the ANN). In particular, the combined use of FT and the ANN yielded the best accuracy. These findings suggest that parameter-space-based transfer learning and deep classifiers are suitable for cross-subject sEMG classification. The combined use of parameter-space-based transfer learning and deep classifiers can effectively reduce the data measurement time of sEMG-based HCI applications.

American Sign Language Recognition for Alphabets Using MediaPipe and LSTM

With the advancement of today's technologies in artificial intelligence, humans tend to use hand gestures in their communication to convey their ideas. Gesture recognition is an active area of research in the human-computer interface (HCI). Gesture recognition is important for communication between deaf-mute people, HCI, robot control, home automation, and medical applications. In this article, a simple and efficient vision-based approach for American Sign Language (ASL) alphabets recognition has been discussed to recognize both static and dynamic gestures. Mediapipe introduced by Google had been used to get hand landmarks and a custom dataset has been created and used for the experimental study. Hand gesture recognition has been done by using Long short-term memory (LSTM). The proposed system has been investigated with 26 alphabets and an accuracy of 99% has been achieved. This work can be used to convert hand gestures into text.

Spectro-spatial Profile for Gender Identification Using Emotional-based EEG Signals

Identifying gender has become essential specially to support automatic human-computer interface applications and to customize interactions based on affective responses. The electroencephalogram (EEG) has been adopted for recording the neuronal information as waveforms from the scalp. The objective of this study was twofold. First, to identify genders from four different emotional states using spectral relative power biomarkers. Second, to develop Spectro-spatial profiles that afford additional information for gender identification using emotional-based EEGs. The dataset has been collected from ten healthful volunteer students from the University of Vienna while watching short emotional audio-visual clips of angry, happiness, sadness, and neutral emotions. Wavelet (WT) has been used as a denoising technique, the spectral relative power features of delta ( ), theta ( ), alpha ( ), beta ( ) and gamma ( ) were extracted from each recorded EEG channel. In the subsequent steps, analysis of variance (ANOVA) and Pearson’s correlation analysis were performed to characterize the emotional-based EEG biomarkers towards developing the Spectro-spatial profile to identify gender differences. The results show that the spectral set of features may provide and convey reliable biomarkers for identifying Spectro-spatial profiles from four different emotional states. EEG biomarkers and profiles enable more comprehensive insights into various human behavior effects and as an intervention on the brain. The results revealed that almost high relative powers from all emotional states appear in females compared to males. Particularly, was the most prominent for anger, and were widely observed in happiness, was the most appears in sadness, and were the powers that appears widely in neutral. Moreover, in females, neut was correlated with and _ang, _neut was mostly correlated with _ang. Besides, _neut was correlated with _ang, _neut was correlated with _ang, _neut was mostly correlated with _sad. Moreover, in males, _neut showed a very strong correlation with _sadness whereas _neut was correlated with _hap and _neut wa s correlated with _hap. Therefore, the proposed system using the WT denoising method, spectral relative power markers, and the spectro-spatial profile plays a crucial role in characterizing the emotional-based EEGs towards gender identification. The classification results were 89.46% for SVM and 90% for the KNN. Therefore, the proposed system using the WT denoising method, spectral relative powers features, SVM, and KNN classifiers were crucial in gender identification and characterizing the emotional EEG signals.

Enhancing the Performance of Fabric-Based Triboelectric Nanogenerators by Structural and Chemical Modification

Fabric-based triboelectric nanogenerators (TENGs) are promising candidates as wearable energy-harvesting devices and self-powered sensors. Booting the power generation performance is an eternal pursuit for TENGs. Herein, an efficient approach was proposed to enhance the triboelectric performance of commercial velvet fabric by enriching the fiber surface with hierarchical structures and amide bonds through chemical grafting of carbon nanotube (CNT) and poly(ethylenimine) (PEI) via a polyamidation reaction. With an optimized modifier concentration, the fabric-based TENG easily achieved over 10 times improvement in output voltage and current at a low modifier content of less than 1 wt %. The modified-fabric-based TENG was fully washable and exhibited excellent robustness and long-term stability. With a maximum power density of 3.2 W/m2 achieved on a 5 × 106 Ω external resistor, the TENG was able to serve as a power source for various small electronics such as pedometer, digital watch, calculator, and digital timer. In addition, the TENG demonstrated capability in self-powered tactile and pressure sensing and promising potential in human-computer interface applications. The approach proposed provides a feasible path for boosting the triboelectric performance of fabric-based TENGs and gives insights into the design of fabric-based nanogenerators and smart textiles.

Automatic Personality Traits Perception Using Asymmetric Auto-Encoder

On account of an increase in the human-computer interface applications, the study of automatic personality perception has become more and more prevalent than speech signal processing in recent years. These studies have shown that personality traits derived from psychology theories mainly affect acoustic features. However, some obstacles remain in the automatic personality perception classification, and the most important one is to extract the features related to each personality trait. Previous studies have shown that the personality effect differs from one acoustic feature to the others. Additionally, there are many features one can extract from speech signals. Curse of dimensionality in features also makes the classification difficult. This paper aimed to introduce and examine a novel and efficient automatic feature extraction method to classify the well-known big five personality traits. In this regard, three data augmentation methods for increasing data samples were examined. Afterwards, 6,373 statistical features were extracted from the nonverbal features of the SSPNet Speaker Personality Corpus. Finally, an innovative stacked asymmetric auto-encoder was utilized to extract useful features automatically to improve classification results. Compared with the conventional stacked auto-encoder and convolutional neural network, the proposed method exhibited an average improvement of 12.40%(10.14%) and 14.36%(1.42%) in terms of the unweighted average recall (accuracy), respectively. In comparison with other published works, classification results also revealed a notable average enhancement (11.78%) for unweighted average recall for all five traits and an average improvement of 5.1% for accuracy in two out of five personality traits.

In-water handwriting in multi-medium using electrical impedance imaging

A novel approach to human–computer interaction, in water handwriting is presented in this work. The novel handwriting method is based on tomographic imaging of electrical resistivity changes. The new method allows the users to write inside of a water ball in a natural, unconstrained way. In-water handwriting suffers no friction and allows handwriting or drawing in a more relaxed way. This work shows how an electrical impedance tomography (EIT) system and sensor/phantom can be used to create a robust handwriting in water. It is well known that EIT has low special resolution but offers a very good functional imaging with high temporal resolution. The work shows an example of a good quality functional imaging aspect of the EIT in a novel human computer interface (HCI) application. Although it is almost impossible to produce images of letters or complex shapes using EIT sensor, it is totally possible to produce the same using many EIT frames. The system used in this study has a frame rate of 15 frames per second so the actual handwriting of letters and shapes can be done without noticing any delays. Analysis of image similarity between actual handwriting and the one follow the EIT scheme shows a good correlation.

Evaluation of Novel Entropy-Based Complex Wavelet Sub-bands Measures of PPG in an Emotion Recognition System

Nowadays, the increasing demand for human–computer interface applications shows the social need to provide an accurate, intelligent emotion recognition system. Computer-aided emotion recognition using physiological signals remains a challenging task in “affective computing”. In this paper, the entropy-based complex wavelet sub-bands (ECWS) measures are suggested for the classification of 14 emotion categories from the photoplethysmograph (PPG). PPG data available at DEAP (a Database for Emotion Analysis using Physiological signals) were selected when subjects were watching the fun, exciting, joy, happy, cheerful, love, sentimental, melancholy, sad, depressing, mellow, hate, shock, and terrible music videos. Using the dual-tree complex wavelet transforms, each PPG signal was decomposed into six levels. Four entropy measures, including approximate entropy, sample entropy, permutation entropy, and the improved multi-scale permutation entropy, were extracted from each sub-band coefficient. Then, the normalized ECWS features were input to the probabilistic neural network. The role of sigma adjustment was also considered in classifier performance. The results indicated the accuracy rates of 92 to 100% for the classification of 14 emotional states. The maximum accuracy rate was 100% for sigma < 0.25. Our findings establish the proposed system as a superior framework compared to the state-of-the-art PPG emotion recognition tool.

Speech Emotion Recognition Using Scalogram Based Deep Structure

Speech Emotion Recognition (SER) is an important part of speech-based Human-Computer Interface (HCI) applications. Previous SER methods rely on the extraction of features and training an appropriate classifier. However, most of those features can be affected by emotionally irrelevant factors such as gender, speaking styles and environment. Here, an SER method has been proposed based on a concatenated Convolutional Neural Network (CNN) and a Recurrent Neural Network (RNN). The CNN can be used to learn local salient features from speech signals, images, and videos. Moreover, the RNNs have been used in many sequential data processing tasks in order to learn long-term dependencies between the local features. A combination of these two gives us the advantage of the strengths of both networks. In the proposed method, CNN has been applied directly to a scalogram of speech signals. Then, the attention-mechanism-based RNN model was used to learn long-term temporal relationships of the learned features. Experiments on various data such as RAVDESS, SAVEE, and Emo-DB demonstrate the effectiveness of the proposed SER method.

Implementation of Hand Gesture Recognition Device Applicable to Smart Watch Based on Flexible Epidermal Tactile Sensor Array.

Ever since the development of digital devices, the recognition of human gestures has played an important role in many Human-Computer interface applications. Various wearable devices have been developed, and inertial sensors, magnetic sensors, gyro sensors, electromyography, force-sensitive resistors, and other types of sensors have been used to identify gestures. However, there are different drawbacks for each sensor, which affect the detection of gestures. In this paper, we present a new gesture recognition method using a Flexible Epidermal Tactile Sensor based on strain gauges to sense deformation. Such deformations are transduced to electric signals. By measuring the electric signals, the sensor can estimate the degree of deformation, including compression, tension, and twist, caused by movements of the wrist. The proposed sensor array was demonstrated to be capable of analyzing the eight motions of the wrist, and showed robustness, stability, and repeatability throughout a range of experiments aimed at testing the sensor array. We compared the performance of the prototype device with those of previous studies, under the same experimental conditions. The result shows our recognition method significantly outperformed existing methods.

Writing in the Air with WiFi Signals for Virtual Reality Devices

Recently, handwriting recognition approaches has been widely applied to Human-Computer Interface (HCI) applications. The emergence of the novel mobile terminals urges a more man-machine friendly interface mode. The previous air-writing recognition approaches have been accomplished by virtue of cameras and sensors. However, the vision based approaches are susceptible to the light condition and sensor based methods have disadvantages in deployment and highcost. The latest researches have demonstrated that the pervasive wireless signals can be used to identify different gestures. In this paper, we attempt to utilize channel state information (CSI) derived from wireless signals to realize the device-free air-write recognition called Wri-Fi . Compared to the gesture recognition, the increased diversity and complexity of characters of the alphabet make it challenging. The Principle Component Analysis (PCA) is used for denoising effectively and the energy indicator derived from the Fast Fourier Transform (FFT) is to detect action continuously. The unique CSI waveform caused by unique writing patterns of 26 letters serve as feature space. Finally, the Hidden Markov model (HMM) is used for character modeling and classification. We conduct experiments in our laboratory and get the average accuracy of the Wri-Fi are 86.75 and 88.74 percent in two writing areas, respectively.

A New feature extraction method to Improve Emotion Detection Using EEG Signals

Since emotion plays an important role in human life, demand and importance of automatic emotion detection have grown with increasing role of human computer interface applications. In this research, the focus is on the emotion detection from the electroencephalogram (EEG) signals. The system derives a mechanism of quantification of basic emotions using. So far, several methods have been reported, which generally use different processing algorithms, evolutionary algorithms, neural networks and classification algorithms. The aim of this paper is to develop a smart method to improve the accuracy of emotion detection by discrete signal processing techniques and applying optimized support vector machine classifier with genetic evolutionary algorithm. The obtained results show that the proposed method provides the accuracy of 93.86% in detection of 4 emotions which is higher than state-of-the-art methods.

Overview Electrotactile Feedback for Enhancing Human Computer Interface

To achieve effective interaction between a human and a computing device or machine, adequate feedback from the computing device or machine is required. Recently, haptic feedback is increasingly being utilised to improve the interactivity of the Human Computer Interface (HCI). Most existing haptic feedback enhancements aim at producing forces or vibrations to enrich the user’s interactive experience. However, these force and/or vibration actuated haptic feedback systems can be bulky and uncomfortable to wear and only capable of delivering a limited amount of information to the user which can limit both their effectiveness and the applications they can be applied to. To address this deficiency, electrotactile feedback is used. This involves delivering haptic sensations to the user by electrically stimulating nerves in the skin via electrodes placed on the surface of the skin. This paper presents a review and explores the capability of electrotactile feedback for HCI applications. In addition, a description of the sensory receptors within the skin for sensing tactile stimulus and electric currents alsoseveral factors which influenced electric signal to transmit to the brain via human skinare explained.

Multi angle optimal pattern-based deep learning for automatic facial expression recognition

Abstract Facial Expression Recognition (FER) plays the vital role in the Human Computer Interface (HCI) applications. The illumination and pose variations affect the FER adversely. The projection of complex 3D actions on the image plane and the inaccurate alignment are the major issues in the FER process. This paper presents the novel Multi-Angle Optimal Pattern-based Deep Learning (MAOP-DL) method to rectify the problem from sudden illumination changes, find the proper alignment of a feature set by using multi-angle-based optimal configurations. The proposed method includes the five major processes as Extended Boundary Background Subtraction (EBBS), Multi-Angle Texture Pattern+STM, Densely Extracted SURF+Local Occupancy Pattern (LOP), Priority Particle Cuckoo Search Optimization (PPCSO) and Long Short-Term Memory -Convolutional Neural Network (LSTM-CNN). Initially, the EBBS algorithm subtracts the background and isolates the foreground from the images which overcome the illumination and pose variation. Then, the MATP-STM extracts the texture patterns and DESURF-LOP extracts the relevant key features of the facial points. The PPCSO algorithm selects the relevant features from the MATP-STM feature set to speed up the classification. The employment of LSTM-CNN predicts the required label for the facial expressions.The major key findings of the proposed work are clear image analysis, effective handling of pose/illumination variations and the facial alignment. The proposed MAOP-DL validates its effectiveness on two standard databases such as CK+ and MMI regarding various metrics and confirm their assurance of wide applicability in recent applications.