Human-machine Interaction Systems Research Articles

A review of physiological measures for mental workload assessment in aviation

AbstractThe relevant growth of human-machine interaction (HMI) systems in recent years is leading to the necessity of being constantly aware of the cognitive workload level of an operator, especially in a safety-critical context such as aviation. Since the confusion in the definition of this concept, this paper clarifies this terminology and also highlights its relationship with stress. Thus, we analysed the state-of-the-art of cognitive workload evaluations, showing three up-to-date methodologies: subjective, behavioural and physiological. In particular, the physiological approach is increasingly gaining attention in the literature due to today’s exponential growth of biomedical sensors. Therefore, a review of the most adopted physiological signals in the workload evaluation is provided, focusing on the aeronautical field. We conclude by highlighting the necessity of a multimodal approach for mental workload assessment as a result of this analysis.

Multi-characteristic tannic acid-reinforced polyacrylamide/sodium carboxymethyl cellulose ionic hydrogel strain sensor for human-machine interaction

Big data and cloud computing are propelling research in human-computer interface within academia. However, the potential of wearable human-machine interaction (HMI) devices utilizing multiperformance ionic hydrogels remains largely unexplored. Here, we present a motion recognition-based HMI system that enhances movement training. We engineered dual-network PAM/CMC/TA (PCT) hydrogels by reinforcing polyacrylamide (PAM) and sodium carboxymethyl cellulose (CMC) polymers with tannic acid (TA). These hydrogels possess exceptional transparency, adhesion, and remodelling features. By combining an elastic PAM backbone with tunable amounts of CMC and TA, the PCT hydrogels achieve optimal electromechanical performance. As strain sensors, they demonstrate higher sensitivity (GF = 4.03), low detection limit (0.5 %), and good linearity (0.997). Furthermore, we developed a highly accurate (97.85 %) motion recognition system using machine learning and hydrogel-based wearable sensors. This system enables contactless real-time training monitoring and wireless control of trolley operations. Our research underscores the effectiveness of PCT hydrogels for real-time HMI, thus advancing next-generation HMI systems.

Hidden Markov Model Deep Learning Architecture for Virtual Reality Assessment to Compute Human–Machine Interaction-Based Optimization Model

Virtual Reality (VR) is a technology that immerses users in a simulated, computer-generated environment. It creates a sense of presence, allowing individuals to interact with and experience virtual worlds. Human-Machine Interaction (HMI) refers to the communication and interaction between humans and machines. Optimization plays a crucial role in Virtual Reality (VR) and Human-Machine Interaction (HMI) to enhance the overall user experience and system performance. This paper proposed an architecture of the Hidden Markov Model with Grey Relational Analysis (GRA) integrated with Salp Swarm Algorithm (SSA) for the automated Human-Machine Interaction. The proposed architecture is stated as a Hidden Markov model Grey Relational Salp Swarm (HMM_ GRSS). The proposed HMM_GRSS model estimates the feature vector of the variables in the virtual reality platform and compute the feature spaces. The HMM_GRSS architecture aims to estimate the feature vector of variables within the VR platform and compute the feature spaces. Hidden Markov Models are used to model the temporal behavior and dynamics of the system, allowing for predictions and understanding of the interactions. Grey Relational Analysis is employed to evaluate the relationship and relevance between variables, aiding in feature selection and optimization. The SSA helps optimize the feature spaces by simulating the collective behavior of salp swarms, improving the efficiency and effectiveness of the HMI system. The proposed HMM_GRSS architecture aims to enhance the automated HMI process in a VR platform, allowing for improved interaction and communication between humans and machines. Simulation analysis provides a significant outcome for the proposed HMM_GRSS model for the estimation Human-Machine Interaction.

Efficient Facial Expression Recognition With Representation Reinforcement Network and Transfer Self-Training for Human–Machine Interaction

The emotion perception capability could endow machines with deeper feedback evaluation of human–machine interaction (HMI), which makes the experience of HMI more harmonious and natural. Currently, how to obtain the robust surface and semantic facial representation features has not been fully explored, moreover, the parameters and computation of conventional convolutional neural network models, which are adopted in facial expression recognition (FER) are too heavy, it is not conducive to the deployment of the HMI system. Therefore, this article proposes a representation reinforcement network (RRN) and transfer self-training (TST) based efficient FER method. Our designed RRN is mainly composed of two modules, i.e., surface representation reinforcement (SurRR) module and semantic representation reinforcement (SemaRR) module. SurRR module fuses the facial features in different dimensions and highlights key feature messaging nodes in feature maps. SemaRR module establishes the deep semantic representation of global facial regions on spatial and channel dimensions. Therefore, RRN has stronger feature extraction capability when the parameters and computational complexity are significantly reduced, and TST is proposed for contributing to the training effect of RRN at the condition of insufficient training data. We tested our method on CK+, RaFD, FERPLUS, and RAF-DB datasets, and the recognition accuracy are 100%, 98.62%, 89.64%, and 88.72%, respectively. Furthermore, the preliminary application experiment, which demonstrates the feasibility of our method in practice during HMI.

Research on Machine Vision-Based Control System for Cold Storage Warehouse Robots

In recent years, the global cold chain logistics market has grown rapidly, but the level of automation remains low. Compared to traditional logistics, automation in cold storage logistics requires a balance between safety and efficiency, and the current detection algorithms are poor at meeting these requirements. Therefore, based on YOLOv5, this paper proposes a recognition and grasping system for cartons in cold storage warehouses. A human–machine interaction system is designed for the cold storage environment, enabling remote control and unmanned grasping. At the algorithm level, the CA attention mechanism is introduced to improve accuracy. The Ghost lightweight module replaces the CBS structure to enhance runtime speed. The Alpha-DIoU loss function is utilized to improve detection accuracy. With the comprehensive improvements, the modified algorithm in this study achieves a 0.711% increase in mAP and a 0.7% increase in FPS while maintaining accuracy. Experimental results demonstrate that the CA attention mechanism increases fidelity by 2.32%, the Ghost lightweight module reduces response time by 13.89%, and the Alpha-DIoU loss function enhances positioning accuracy by 7.14%. By incorporating all the improvements, the system exhibits a 2.16% reduction in response time, a 4.67% improvement in positioning accuracy, and a significant overall performance enhancement.

Finger–coding intelligent human–machine interaction system based on all–fabric ionic capacitive pressure sensors

The comprehensive application of human–machine interaction (HMI) has promoted humanization in digital life. Wearable electronic devices and artificial intelligence are driving the development and transformation of HMI technology. Nevertheless, the fabrication of superior devices assisted by artificial intelligence for HMI remains challenging. We introduce an all–fabric ionic capacitive pressure sensor (AFICPS) using laser–induced graphene/fabric as the electrode and ion/fabric as the dielectric layer. The designed sensor exhibits high performance, including ultrahigh sensitivity (17.8 kPa−1@0–60 kPa, 73.3 kPa−1@60–100 kPa, and 33.8 kPa−1@100–150 kPa), wide detection range (0–150 kPa), ultrafast response and recovery times (11 and 8 ms, respectively), low detection limit (∼ 5 Pa), good mechanical stability (only 4.4 % performance loss after 5000 cycles), and excellent breathability and biocompatibility (mouse fibroblast cell survival exceeding 99 % after 48 h). All these factors contribute to the accurate detection of human physiological signals, particularly finger movements. Hence, we demonstrate a finger–coding intelligent HMI system that converts multiple gestures into 26 English letters and 10 Arabic numerals with a recognition accuracy above 98 % by using machine learning. This type of system may enhance information privacy and be of great significance in HMI and encrypted communication.

Modeling Human–Machine Interaction System Reliability with Multiple Dependent Degradation Processes and Situation Awareness

With the advancement in automation, the roles of machine operators have shifted from traditional physical commitments to controlling the automation process. The operators need to understand and assess the incoming information and make decisions accordingly, which will determine the next move of the process and further affect the machine’s health status/performance. Such awareness and understanding of the situation, as one of the critical prerequisites of decision-making and known as situation awareness (SA), needs to be addressed appropriately in accessing the reliability of the system with human–machine interaction (HMI). Hence, we propose the HMI system reliability model considering the machine has multiple dependent degradation processes, and each degradation process is affected by the interactions of internal machine degradation, random shocks, and SA. In addition, the impacts of SA and external factors on the system are reflected in the amount and rate of the machine degradation process. The proposed model is demonstrated by a simulated case, and the sensitivity analysis is conducted to analyze the impacts of model parameters on system reliability prediction. Finally, the performance of the proposed model is demonstrated by comparing it with the existing model.

Design and Development of a Non-Contact ECG-Based Human Emotion Recognition System Using SVM and RF Classifiers.

Emotion recognition becomes an important aspect in the development of human-machine interaction (HMI) systems. Positive emotions impact our lives positively, whereas negative emotions may cause a reduction in productivity. Emotionally intelligent systems such as chatbots and artificially intelligent assistant modules help make our daily life routines effortless. Moreover, a system which is capable of assessing the human emotional state would be very helpful to assess the mental state of a person. Hence, preventive care could be offered before it becomes a mental illness or slides into a state of depression. Researchers have always been curious to find out if a machine could assess human emotions precisely. In this work, a unimodal emotion classifier system in which one of the physiological signals, an electrocardiogram (ECG) signal, has been used is proposed to classify human emotions. The ECG signal was acquired using a capacitive sensor-based non-contact ECG belt system. The machine-learning-based classifiers developed in this work are SVM and random forest with 10-fold cross-validation on three different sets of ECG data acquired for 45 subjects (15 subjects in each age group). The minimum classification accuracies achieved with SVM and RF emotion classifier models are 86.6% and 98.2%, respectively.

Neuroscience in business-to-business marketing research: A literature review, co-citation analysis and research agenda

Neuroscience has led the way in bridging the gap between the social and natural sciences. The ongoing transformations in the business landscape have brought human-machine interaction and multi-agent systems to the forefront. This underscores the importance of theories that bridge the gap between human brain processes and business-to-business marketing practices. Despite the potential of neuroscience in business-to-business marketing, such interdisciplinary research is extensively lacking. This paper employs systematic literature reviews and co-citation analyses to: a) establish the absence of business-to-business marketing–neuroscience research; b) investigate the ways in which the broader business and management discipline has bridged with neuroscience to inspire such integration between business-to-business marketing and neuroscience; c) describe bridging approaches; and d) propose a research agenda for business-to-business marketing. The co-citation analyses show that the business and management discipline has focused on the ‘within-brain’ or behavioural aspects of neuroscience. In addition, the discipline has continually integrated new neuroscience insights or increasingly developed theories that are internalised within the discipline, independent of neuroscience. The paper's main contributions are the reviews and co-citation analyses, as well as an account of bridging in interdisciplinary research. Furthermore, the proposed research agenda enhances our understanding of the potential of neuroscience in business-to-business marketing.

Multimodal integrated flexible electronic skin for physiological perception and contactless kinematics pattern recognition

Multimodal integrated flexible electronic skin offers exceptional processing capability and versatility in various application scenarios. Researchers are devoting to developing nanomaterials or nano-devices with multidisciplinary functions. Herein, a trinary-mode integrated flexible electronic skin (E-Skin) was designed for physiological perception and contactless kinematics pattern recognition, which serves as real-time health monitor, electrode patch and contactless sensor. The composite film of liquid metal particles and thermoplastic polyurethane elastomers (LMPs@TPU) was prepared by a combination of electrostatic spinning, electrostatic spraying, and knife coating, which can be stretched up to 404.7 % at stress of 1Mpa. The film was endowed three functional modes through different forms of encapsulation, which allows for (1) detecting human arthrosis movements, muscle stretching, and inspiration, (2) achieving a real-time capture of Electrocardiogram (ECG) and surface electromyography (sEMG), and a Human-Machine Interaction (HMI) system for game control based on sEMG signals was also developed, (3) a contactless sensor combining with a machine learning model which realizes the recognition of human motion patterns with a receiver operating characteristic curve and area under curve score (ROC-AUC-score) of 0.99. The three models of this E-Skin highlight the significant potential for industrialization in various fields, such as HMI, physiological perception, virtual reality, and motion analysis.

Multiplication Circuit Architecture for Error- Tolerant CNN-Based Keywords Speech Recognition

Approximate multiplication is an emerging circuit design technique for AI-based IoT devices which can reduce the energy consumption. A precision reconfigurable tensor multiplication unit (TMU) is proposed to explore the advantages for energy efficient approximate computing for error-tolerant convolutional neural network (CNN) based keywords speech recognition, which is a widely used human-machine interaction system. A positive-negative encoding method and a reorganization structure of partial products are proposed for the tensor multiplication. The partial products are further optimized by a hierarchical addition tree structure with fine-grained precision reconfigurable approximate computing. The imprecision parts of the addition tree are optimized with a low supply voltage. The effects of circuit configuration variations on accuracy and power consumption have been evaluated under 22nm process technology. The proposed approximate designs are applied to a CNN-based keywords speech recognition system. With the proposed approach, the energy efficiency of the CNN accelerator can be greatly improved with a power reduction of up to 44.1% at a loss in accuracy of less than 2%.

Highly stable and reliable capacitive strain sensor for wearable electronics based on anti-dry hydrogel electrode

Hydrogel-based capacitive strain sensors have been rapidly developed benefiting from their robust structure, excellent flexibility, and high linearity. However, the existing capacitive strain sensors usually suffer from the water evaporation of hydrogel electrodes, leading to poor long-term stability. In this work, a dry-resistant hydrogel electrode with 250 μm thickness was prepared on the dielectric elastomer by a simple and versatile template method through the synergistic water-retention effect of glycerin and LiCl, which can maintain high mechanical stability in a dry environment with relative humidity of 10% RH. The fabricated capacitive strain sensor can be stored in a normal environment for 45 days and can continuously operate under the environment with relative humidity of 20% RH for 40 h (over 10,000 cycles). Most importantly, the capacitive strain sensor possessed ultra-high linearity (R2 > 0.998), good uniformity (>93.56%), and robustness at different humidity. With these favorable advantages, the sensor was successfully applied as a highly reliable wearable sensing system for human motion detection and human-machine interaction in real-time.

Wearable patch with direction-aware sensitivity of in-plane force for self-powered and single communication channel based human-machine interaction

The merging of flexible technologies with human machine interaction (HMI) is now optimizing the way people communicates with electrical terminals. Compared with “press” and “strain”, “sliding” is a directional operation which requires the interface to identify the directions of the applied force for accurate interaction. Previous efforts that explore the “directional force” in HMI system are mainly based on sensor array, which brings concern of complex electrode design and multiple communication channels to prevent cross-talk. In view of this, we developed a self-powered and wearable HMI interface that can distinguish the axial directions of in-plane force based on Faraday’s law of induction. The interface consists of well-orientated magnetized micropillars, and a conductive coil that collects and transmits the electrical signals during the interaction process. When in-plane sliding force was applied, distinguishable signals were generated to reflect the different axial directions (+X/−X/+Y/−Y) according to the polarity and number of voltage peaks. With this unique behavior, the HMI process can be completed with two electrodes and one communication channel in an interference-free manner. Through the systematic optimization, the intrinsic oscillation from the micropillars results in obviously enhanced signals for a high accuracy and reliability towards real application. The MMPs-based interface was successfully established for HMI platforms such as intelligent robot control, and Morse code communication, etc. Owing to the robustness, humid resistance, accuracy and reliability, we expect that the interface design can inspire the development of flexible and wearable devices in HMI especially for scenarios that require a high command capacity.

Recognizing Command Words using Deep Recurrent Neural Network for Both Acoustic and Throat Speech

The importance of speech command recognition in a human-machine interaction system is increased in recent years. In this study, we propose a deep neural network-based system for acoustic and throat command speech recognition. We apply a preprocessed pipeline to create the input of the deep learning model. Firstly, speech commands are decomposed into components using well-known signal decomposition techniques. The Mel-frequency cepstral coefficients (MFCC) feature extraction method is applied to each component of the speech commands to obtain the feature inputs for the recognition system. At this stage, we apply and compare performance using different speech decomposition techniques such as wavelet packet decomposition (WPD), continuous wavelet transform (CWT), and empirical mode decomposition (EMD) in order to find out the best technique for our model. We observe that WPD shows the best performance in terms of classification accuracy. This paper investigates long short-term memory (LSTM)-based recurrent neural network (RNN), which is trained using the extracted MFCC features. The proposed neural network is trained and tested using acoustic speech commands. Moreover, we also train and test the proposed model using a throat mic. speech commands as well. Lastly, the transfer learning technique is employed to increase the test accuracy for throat speech recognition. The weights of the model train with the acoustic signal are used to initialize the model used for throat speech recognition. Overall, we have found significant classification accuracy for both acoustic and throat command speech. We obtain LSTM is much better than the GMM-HMM model, convolutional neural networks such as CNN-tpool2 and residual networks such as res15 and res26 with an accuracy score of over 97% on Google’s Speech Commands dataset and we achieve 95.35% accuracy on our throat speech data set using the transfer learning technique.

Polyvinylidene Fluoride/Aromatic Hyperbranched Polyester of Third-Generation-Based Electrospun Nanofiber as a Self-Powered Triboelectric Nanogenerator for Wearable Energy Harvesting and Health Monitoring Applications.

Flexible pressure sensors have played an increasingly important role in the Internet of Things and human-machine interaction systems. For a sensor device to be commercially viable, it is essential to fabricate a sensor with higher sensitivity and lower power consumption. Polyvinylidene fluoride (PVDF)-based triboelectric nanogenerators (TENGs) prepared by electrospinning are widely used in self-powered electronics owing to their exceptional voltage generation performance and flexible nature. In the present study, aromatic hyperbranched polyester of the third generation (Ar.HBP-3) was added into PVDF as a filler (0, 10, 20, 30 and 40 wt.% w.r.t. PVDF content) to prepare nanofibers by electrospinning. The triboelectric performances (open-circuit voltage and short-circuit current) of PVDF-Ar.HBP-3/polyurethane (PU)-based TENG shows better performance than a PVDF/PU pair. Among the various wt.% of Ar.HBP-3, a 10 wt.% sample shows maximum output performances of 107 V which is almost 10 times that of neat PVDF (12 V); whereas, the current slightly increases from 0.5 μA to 1.3 μA. The self-powered TENG is also effective in measuring human motion. Overall, we have reported a simpler technique for producing high-performance TENG using morphological alteration of PVDF, which has the potential for use as mechanical energy harvesters and as effective power sources for wearable and portable electronic devices.

An Intelligent Glove of Synergistically Enhanced ZnO/PAN-Based Piezoelectric Sensors for Diversified Human–Machine Interaction Applications

Human–machine interaction is now deeply integrated into our daily lives. However, the rigidity and high-power supply of traditional devices limit their further development. Herein, a high-performance flexible piezoelectric sensor (HFPS) based on a novel zinc oxide/polyacrylonitrile/Ecoflex (ZnO/PAN/Ecoflex) composite membrane is proposed. Due to the synergistic piezoelectricity of ZnO and PAN, the output voltage/current of the HFPS is increased by 140%/100% compared to the pure Zno/Ecoflex composite membrane. Furthermore, the fabricated HFPSs also have excellent sensitivity, linearity, stability and flexibility under periodic pressure. On this basis, due to its flexibility, stretchability and battery-free characteristics, a self-powered HFPS-based intelligent glove is proposed to wirelessly control diverse electronic systems through human hand gestures. In the meanwhile, the intelligent glove has been successfully applied to car two-dimensional motion, light bulb control and fan control. With the advantages of simple operation, portability and low power consumption, the glove is expected to provide new application prospects for human–machine interaction systems.

Human–machine interaction for motorized wheelchair based on single-channel electroencephalogram headband

Human machine interaction (HMI) allows persons to control and interact with devices. Starting from elementary apparatus which acquires input bio-signals to controlling various applications. Medical applications are amongst the very important applications of HMI. One of these medical applications is assisting fully/partially paralyzed patients to restore movements or freely move using exoskeletons or motorized wheelchairs. Helping patients with spinal cord injury or serious neurological diseases to restore their movements is a key role objective for most researchers in this field. In this paper, an EEG-based HMI system is proposed to assist patients with tetraplegia/quadriplegia to mentally control a motorized wheelchair so they can move freely and independently. EEG power spectrum (α, β, δ, θ, and γ) from the frontal lobe of brain is recorded, filtered and wirelessly sent to the wheelchair to control directions and engine status. Four different experiments were conducted using the proposed system in order to validate the performance. Two different GUIs scenarios (cross-shaped and horizontal bar) were used with the experiments. Results showed that the horizontal bar scenario considered more user friendly while the cross-shaped is the more suitable for navigation. The implemented system can be equipped with modules and sensors such as GPS, ultrasound and accelerometer that improve the system performance and reliability.

Chat With ChatGPT on Interactive Engines for Intelligent Driving

ChatGPT, a chatbot launched by OpenAI, is making waves across the internet recently. Its potential capabilities and how it can impact various industries have been widely discussed. In the field of smart vehicles, integrating human-machine interaction systems with ChatGPT can be a game changer. For example, Jidu Auto, the electric vehicle arm of Baidu, announced in February 2023 that they will integrate ERNIE Bot [1], Baidu's ChatGPT-like service, into their cars. While ChatGPT has boosted the development of intelligent interaction, some limitations still exist that hinder its practical application. In this article, we discuss its potential applications in intelligent vehicles and analyze the challenges it faces.

How much is a “feedback” worth? User engagement and interaction for computer-supported adaptive quizzing

ABSTRACT The digital transformation of education is greatly accelerating in various computer-supported applications. As a particularly prominent application of the human-machine interactive system, intelligent learning systems aim to capture users’ current intentions and provide recommendations through real-time feedback. However, we have a limited understanding on what is the reality of user feedback for computer-supported adaptive quizzing. This present study proposes formalization for user feedback including engagement and interactions to understand and improve intelligent adaptive quizzing. This work starts by taking individual system’s action transition patterns as a temporally evolving action graph and derives patterns in terms of informing future user engagement. Furthermore, considering the student knowledge preference for a certain moment is closely related to personal experience of past learning,a novel framework EICQ (Engagement and Interaction-based Computer-Supported Adaptive Quizzing) is proposed for learning and adapting to students’ current preference based on engagement trends and interaction by collecting feedback data in real-time during a learning session. This established framework captures question characters, student dynamic interactions and engagement trends, which yield a better learning experience with adaptive practice question (PQ) recommendations. Finally, much more endeavors have been made to integrate the proposed EICQ into XXXU intelligent learning systems that are co-developed with the YYY Company.

Research on performance optimization method of human-machine physical interaction system considering exoskeleton wearing comfort

In order to improve the wearing comfort and bearing effectiveness of the exoskeleton, based on the prototype and working mechanism analysis of a relaxation wearable system for knee exoskeleton robot, the static optimization synthesis and its method are studied. Firstly, based on the construction of the virtual prototype model of the system, a comprehensive wearable comfort evaluation index considering the factors such as stress, deformation and the proportion of stress nodes was constructed. Secondly, based on the static simulation and evaluation index of system virtual prototype, multi-objective genetic optimization and local optimization synthesis of armor layer topology were carried out. Finally, the model reconstruction simulation data confirmed that the system had good wearing comfort. Our study provides a theoretical basis for the bearing performance and prototype construction of the subsequent wearable system.