Intelligent Human-machine Interface Research Articles

AI-empowered neural processing for intelligent human-machine interface and biomedical devices

AI-empowered neural processing for intelligent human-machine interface and biomedical devices Jie Gu, Associate Professor from Northwestern University, examines AI-empowered neural processing for intelligent human-machine interface and biomedical devices. Most conventional wearable devices rely on motion detection or image classifications to capture users’ activities. However, they lack the ability to decode neural signals generated by the human body. Neural signals, such as EEG, ECG, and EMG, offer a rich amount of information on a person’s physiological and psychological activities. Recognition and use of such signals present many new opportunities for applications in medical and daily commercial usage. Recently, artificial intelligence (AI) has been applied to neural signal processing, leading to a new generation of intelligent human-machine interfaces and biomedical devices.

An Intelligent Human–Machine Interface Architecture for Long-Term Remote Robot Handling in Fusion Reactor Environments

This paper addresses the intricate challenge posed by remote handling (RH) operations in facilities with operational lifespans surpassing 30 years. The extended RH task horizon necessitates a forward-looking strategy to accommodate the continuous evolution of RH equipment. Confronted with diverse and evolving hardware interfaces, a critical requirement emerges for a flexible and adaptive software architecture based on changing situations and past experiences. The paper explores the inherent challenges associated with sustaining and upgrading RH equipment within an extended operational context. In response to this challenge, a groundbreaking, flexible, and maintainable human–machine interface (HMI) architecture named MAMIC is designed, guaranteeing seamless integration with a diverse range of RH equipment developed over the years. Embracing a modular and extensible design, the MAMIC architecture facilitates the effortless incorporation of new equipment without compromising system integrity. Moreover, by adopting this approach, nuclear facilities can proactively steer the evolution of RH equipment, guaranteeing sustained performance and compliance throughout the extended operational lifecycle. The proposed adaptive architecture provides a scalable and future-proof solution, addressing the dynamic landscape of remote handling technology for decades.

Correction: Platform Supporting Intelligent Human–Machine Interface (HMI) Applications for Smart Machine Tools

Correction: Platform Supporting Intelligent Human–Machine Interface (HMI) Applications for Smart Machine Tools

Methodology based on machine learning through neck motion and POF-based pressure sensors for wheelchair operation

Polymer Optical Fiber (POF)-based sensors have gained recognition in recent years for biomedical applications because of their low cost, physical properties, and feasibility. A novel methodology is proposed here for classifying neck movements using POF- based pressure sensors and machine learning algorithms. To address this, signal pre-processing, feature extraction, and selection methods are implemented, considering variance, root mean square, and Hjorth parameters. Linear Discriminant Analysis, Support Vector Machine, k-Nearest Neighbors (kNN), and Decision Tree (DT) were used for classification. A maximum accuracy of 0.91 was obtained with kNN and DT for recognizing four neck movements by using the best discriminant nine features. These findings indicate that the proposed methodology is suitable for neck-motion classification using POF-based pressure sensors. Future work will focus on the implementation of this strategy for the design of intelligent Human Machine Interfaces based on electric-powered wheelchairs, which would allow for more independence for people with upper- and lower-limb disabilities.

Platform Supporting Intelligent Human–Machine Interface (HMI) Applications for Smart Machine Tools

As the Internet of Things, artificial intelligence, and the fourth industrial revolution advance, smart factories and machines increasingly gain intelligent features that enable the integration of more sophisticated functionalities. Approaches to achieving this intelligence involve both internal systems, such as human–machine interface (HMI), and external systems, such as big data platforms and cloud services. Although current research leans toward studying external systems, accomplishing intelligent functions through such means poses more challenges in achieving real-time responses during machining processes than using internal systems. When intellectualizing machine tools through internal HMI systems, three critical issues must be addressed. First, HMI functions are structured to depend on the HMI itself, leading to a ripple effect where a problem occurring in one HMI function impacts the entire system. Second, owing to differences in development tools and programming languages, the interconnectivity between functions developed by multiple stakeholders to be loaded onto the HMI may suffer, leading to potential inefficiencies and increased maintenance costs. Third, although various types of computer numerical control (CNC) machines need to communicate with the HMI function, the diverse communication methods and development tools used by each CNC manufacturer result in identical intelligent functions being developed separately for each CNC type. To address these challenges, this study proposes an innovative HMI platform capable of executing and developing various intelligent functions. The HMI platform and its major components are designed and implemented through component-based development (CBD). Subsequently, the performance and effectiveness of the platform are validated using quality attribute scenarios.

Microbial biofilms for self-powered noncontact sensing

Noncontact sensing technology plays a vital role in the intelligent human–machine interface, as the essential medium for exchanging information between human and electronic devices. To date, several inorganic materials-based noncontact sensing techniques have been used to accurately detect touch, electrical property, and physical motion. However, limited available materials, dependence on additional power supplies, and poor power production performance, have seriously obstructed the practical applications of noncontact sensing technology. Here, we developed simple self-powered noncontact sensors (SNSs) assembled using a typical G. sulfurreducens biofilm as the core component. In noncontact mode, the sensor demonstrated excellent self-powered sensing performance with maximum voltage output of 10 V and a current of 60 nA, a maximum sensing range of 40 cm which is the farthest reported to date. Depending on its excellent sensing characteristic, the SNSs was used to monitor human breathing in this work. Furthermore, an array of united SNSs was able to localize external electric fields and effectively extend the sensing area by increasing the number of devices. Compared to traditional inorganic materials, microbial biofilms have the advantages of wide existence, self-proliferation, low cost, environmental friendliness, and ultra-fast self-healing property (seconds level). The proposed biofilm SNSs in our work provides new insights for noncontact power generation of biomaterials and self-driven sensing.

Stretchable, skin‐conformable neuromorphic system for tactile sensory recognizing and encoding

AbstractExpanding wearable technologies to artificial tactile perception will be of significance for intelligent human–machine interface, as neuromorphic sensing devices are promising candidates due to their low energy consumption and highly effective operating properties. Skin‐compatible and conformable features are required for the purpose of realizing wearable artificial tactile perception. Here, we report an intrinsically stretchable, skin‐integrated neuromorphic system with triboelectric nanogenerators as tactile sensing and organic electrochemical transistors as information processing. The integrated system provides desired sensing, synaptic, and mechanical characteristics, such as sensitive response (~0.04 kPa−1) to low‐pressure, short‐ and long‐term synaptic plasticity, great switching endurance (>10 000 pulses), symmetric weight update, together with high stretchability of 100% strain. With neural encoding, demonstrations are capable of recognizing, extracting, and encoding features of tactile information. This work provides a feasible approach to wearable, skin‐conformable neuromorphic sensing system with great application prospects in intelligent robotics and replacement prosthetics.

Micro-fabrication Based Epidermal E-tattoo with Conformability and Sensitivity as Human-Machine Interface

To fulfill intelligent human-machine interface, the crucial information about the human health condition tends to be acquired with great sensitivity, stability and durability. Despite some remarkable progress for the bioelectric signal perception in recent years, the epidermal electrode still suffers from the motion artifacts, inability and sweat permeability. So the evolution of high-performance epidermal electrode has been the research focus and receives much attention in recent years. To satisfy the growing pursuit for smarter sensing, this article proposes a cost-effective strategy to fabricate e-tattoo epidermal electrode for acquisition of the epidermal bioelectric signal and provides demonstrations of electrocardiograph and electromyography applications. The maximum current change is only 1.6% in the 3-dimensional strain electrical test, and the maximum strain range could reach 40% for a normal usage. The mechanism relies upon conformal contact between the electrode and skin texture with ultra-thin process scale, which makes the device achieves high sensitivity and durability in a human-machine interface. The gesture recognition and wearable device demonstration further concludes by discussing the outlook and current challenges of these technologies as a human-machine interface and bring up the potential of clinical application.

Flexible, Permeable, and Recyclable Liquid-Metal-Based Transient Circuit Enables Contact/Noncontact Sensing for Wearable Human-Machine Interaction.

The past several years have witnessed a rapid development of intelligent wearable devices. However, despite the splendid advances, the creation of flexible human-machine interfaces that synchronously possess multiple sensing capabilities, wearability, accurate responsivity, sensitive detectivity, and fast recyclability remains a substantial challenge. Herein, a convenient yet robust strategy is reported to craft flexible transient circuits via stencil printing liquid metal conductor on the water-soluble electrospun film for human-machine interaction. Due to the inherent liquid conductor within porous substrate, the circuits feature high-resolution, customized patterning viability, attractive permeability, excellent electroconductivity, and superior mechanical stability. More importantly, such circuits display appealing noncontact proximity capabilities while maintaining compelling tactile sensing performance, which is unattainable by traditional systems with compromised contact sensing. As such, the flexible circuit is utilized as wearable sensors with practical multifunctionality, including information transfer, smart identification, and trajectory monitoring. Furthermore, an intelligent human-machine interface composed of the flexible sensors is fabricated to realize specific goals such as wireless object control and overload alarm. The transient circuits are quickly and efficiently recycled toward high economic and environmental values. This work opens vast possibilities of generating high-quality flexible and transient electronics for advanced applications in soft and intelligent systems.

Intelligent Human–Machine Interface: An Agile Operation and Decision Support for an ANAMMOX SBR System at a Pilot-Scale Wastewater Treatment Plant

Eco-efficient anaerobic ammonium oxidation (ANAMMOX) can eliminate toxic nutrients from wastewater and has been used in several nutrient removal technologies. However, its implementation for robust operation remains challenging because of process nonlinearity and time-variant characteristic, higher energy consumption, excess sludge produced, and biomass loss during sludge pumping. Also, sensor failure, process startup, and shut down present additional difficulties. In this article, an intelligent human–machine interface using an advanced numerical solution for a knowledge-based system (called ANKSys) was developed by integrating the fully optimized-functionality (soft sensing, decision making, and simulating model) data driven by supervisory control. This control consists of advanced algorithms (artificial neural network, Kalman filter, principal component analysis, least-square technique/renowned root-mean-squared error) using commercial software (MATLAB R2018a, Microsoft Visual Studio IDE 2016, Microsoft SQL Server 2014, OPC Automation with XGT series programmable logic controller). The developed <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">ANKSys</i> can help in online monitoring and optimal process operation by assessing risk and failure occurrences, acquiring data for data analysis, and managing operating expenditure. In real-time implementation, ANKSys enhanced the energy efficiency, i.e., 16% of a pilot-scale “LEAOX” wastewater treatment plant located at Daegu, Republic of Korea. Using this strategy, an optimal and sustainable operation for the removal of biological nitrogen was achieved.

All electrospun fabrics based piezoelectric tactile sensor

Tactile sensors have been widely used in the areas of health monitoring and intelligent human-machine interface. Flexible tactile sensors based on nanofiber mats made by electrospinning can meet the requirements of comfortability and breathability for wearing the body very well. Here, we developed a flexible and self-powered tactile sensor that was sandwich assembled by electrospun organic electrodes and a piezoelectric layer. The metal-free organic electrodes of thermal plastic polyurethane (PU) nanofibers decorated with multi-walled carbon nanotubes were fabricated by electrospinning followed by ultrasonication treatment. The electrospun polyvinylidene fluoride-trifluoroethylene (PVDF-TrFE) mat was utilized as the piezoelectric layer, and it was found that the piezoelectric performance of PVDF-TrFE nanofiber mat added with barium titanate (BaTiO3) nanoparticles was enhanced about 187% than that of the pure PVDF-TrFE nanofiber mat. For practical application, the as-prepared piezoelectric tactile sensor exhibited an approximative linear relationship between the external force and the electrical output. Then the array of fabricated sensors was attached to the fingertips of a glove to grab a cup of water for tactile sensing, and the mass of water can be directly estimated according to the outputs of the sensor array. Attributed to the integrated merits of good flexibility, enhanced piezoelectric performance, light weight, and efficient gas permeability, the developed tactile sensor could be widely used as wearable devices for robot execution end or prosthesis for tactile feedback.

Learning Spatio-Temporal Representations With a Dual-Stream 3-D Residual Network for Nondriving Activity Recognition

Accurate recognition of nondriving activity (NDA) is important for the design of intelligent human machine interface to achieve a smooth and safe control transition in the conditionally automated driving vehicle. However, some characteristics of such activities like limited-extent movement and similar background pose a challenge to the existing 3-D convolutional neural network based action recognition methods. In this article, we propose a dual-stream 3-D residual network, named DS3D residual network (ResNet), to enhance the learning of spatio-temporal representation and improve the activity recognition performance. Specifically, a parallel two-stream structure is introduced to focus on the learning of short-time spatial representation and small-region temporal representation. A two-feed driver behavior monitoring framework is further build to classify four types of NDAs and two types of driving behavior based on the driver's head and hand movement. A novel NDA dataset has been constructed for the evaluation, where the proposed DS3D ResNet achieves 83.35% average accuracy, at least 5% above three selected state-of-the-art methods. Furthermore, this study investigates the spatio-temporal features learned in the hidden layer through the saliency map, which explains the superiority of the proposed model on the selected NDAs.

Recent advancements in flexible and wearable sensors for biomedical and healthcare applications

With the increasing awareness of personal health management in recent decades, various types of flexible and wearable body sensors have been developed. Thanks to the superiority of advanced wearable technologies, including miniaturization, portability, stretchability, comfort, intelligent human-machine interface, etc, flexible and wearable body sensors hold great promise for next generation biomedicine and healthcare applications. Unfortunately, the data precision, response speed, sensitivity and selectivity, durability, compatibility with flexible substrates, and preparation techniques still need to be enhanced and refined to meet the requirements of clinical evaluations or even commercialization. According to working principles, flexible and wearable sensing platforms can be roughly divided into four categories: physical sensors, chemical sensors, biosensors, and the combination of different types of sensors. Here, a brief review focused on the recent developments of these flexible and wearable sensors applied especially to biomedicine and healthcare is presented. In addition, the existing challenges and potential opportunities ahead in flexible and wearable sensor technologies are discussed. Finally, an outlook of wearable sensing platforms in biomedicine and healthcare is proposed. We hope this review can provide guidance for superior flexible and wearable sensing technologies in the future and can provide an outlook for commercial wearable sensors.

The Identification of Non-Driving Activities with Associated Implication on the Take-Over Process.

In conditionally automated driving, the engagement of non-driving activities (NDAs) can be regarded as the main factor that affects the driver’s take-over performance, the investigation of which is of great importance to the design of an intelligent human–machine interface for a safe and smooth control transition. This paper introduces a 3D convolutional neural network-based system to recognize six types of driver behaviour (four types of NDAs and two types of driving activities) through two video feeds based on head and hand movement. Based on the interaction of driver and object, the selected NDAs are divided into active mode and passive mode. The proposed recognition system achieves 85.87% accuracy for the classification of six activities. The impact of NDAs on the perspective of the driver’s situation awareness and take-over quality in terms of both activity type and interaction mode is further investigated. The results show that at a similar level of achieved maximum lateral error, the engagement of NDAs demands more time for drivers to accomplish the control transition, especially for the active mode NDAs engagement, which is more mentally demanding and reduces drivers’ sensitiveness to the driving situation change. Moreover, the haptic feedback torque from the steering wheel could help to reduce the time of the transition process, which can be regarded as a productive assistance system for the take-over process.

A Robust and Wearable Triboelectric Tactile Patch as Intelligent Human-Machine Interface.

The human–machine interface plays an important role in the diversified interactions between humans and machines, especially by swaping information exchange between human and machine operations. Considering the high wearable compatibility and self-powered capability, triboelectric-based interfaces have attracted increasing attention. Herein, this work developed a minimalist and stable interacting patch with the function of sensing and robot controlling based on triboelectric nanogenerator. This robust and wearable patch is composed of several flexible materials, namely polytetrafluoroethylene (PTFE), nylon, hydrogels electrode, and silicone rubber substrate. A signal-processing circuit was used in this patch to convert the sensor signal into a more stable signal (the deviation within 0.1 V), which provides a more effective method for sensing and robot control in a wireless way. Thus, the device can be used to control the movement of robots in real-time and exhibits a good stable performance. A specific algorithm was used in this patch to convert the 1D serial number into a 2D coordinate system, so that the click of the finger can be converted into a sliding track, so as to achieve the trajectory generation of a robot in a wireless way. It is believed that the device-based human–machine interaction with minimalist design has great potential in applications for contact perception, 2D control, robotics, and wearable electronics.

Self-Powered Intelligent Human-Machine Interaction for Handwriting Recognition.

Handwritten signatures widely exist in our daily lives. The main challenge of signal recognition on handwriting is in the development of approaches to obtain information effectively. External mechanical signals can be easily detected by triboelectric nanogenerators which can provide immediate opportunities for building new types of active sensors capable of recording handwritten signals. In this work, we report an intelligent human-machine interaction interface based on a triboelectric nanogenerator. Using the horizontal-vertical symmetrical electrode array, the handwritten triboelectric signal can be recorded without external energy supply. Combined with supervised machine learning methods, it can successfully recognize handwritten English letters, Chinese characters, and Arabic numerals. The principal component analysis algorithm preprocesses the triboelectric signal data to reduce the complexity of the neural network in the machine learning process. Further, it can realize the anticounterfeiting recognition of writing habits by controlling the samples input to the neural network. The results show that the intelligent human-computer interaction interface has broad application prospects in signature security and human-computer interaction.

Dimensional Evolution of Intelligent Cars Human-Machine Interface considering Take-Over Performance and Drivers’ Perception on Urban Roads

The study analyzed the drivers’ take-over behaviors in intelligent cars when driving on urban roads and tried to find reasonable dimensions of the human-machine interface. Firstly, the main driving assistance functions in the process of take-over were analyzed based on the entropy theory, and the weight values of each function for the consumer’s purchase intention were calculated. Secondly, we explored the perceived comfortable dimensions of the interactive components under typical interaction modes. By means of experiments using a within-subjects design, the initial population of the evolutionary computation was obtained. The evolutionary mechanism of dimensions driven by users’ perception was constructed with a genetic algorithm. After debugging the parameters of the model, we verified the rationality of the model and evolved appropriate dimensions. Finally, the validity of the evolved dimensions was proved by a controlled experiment and paired-sample t-test. The results indicated that the completion time of most take-over tasks under the HMI with the evolved dimensions was significantly shorter, which ensured the HMI could be more conducive to the take-over quality and traffic efficiency.

Influence of Adaptive Human–Machine Interface on Electric-Vehicle Range-Anxiety Mitigation

The electrification of vehicles is without a doubt one of the milestones of today’s automotive technology. Even though industry actors perceive it as a future standard, acceptance, and adoption of this kind of vehicles by the end user remain a huge challenge. One of the main issues is the range anxiety related to the electric vehicle’s remaining battery level. In the scope of the H2020 ADAS&amp;ME project, we designed and developed an intelligent Human Machine Interface (HMI) to ease acceptance of Electric Vehicle (EV) technology. This HMI is mounted on a fake autonomous vehicle piloted by a hidden joystick (called Wizard of Oz (WoZ) driving). We examined 22 inexperienced EV drivers during a one-hour driving task tailored to generate range anxiety. According to our protocol, once the remaining battery level started to become critical after manual driving, the HMI proposed accurate coping techniques to inform the drivers how to reduce the power consumption of the vehicle. In the following steps of the protocol, the vehicle was totally out of battery, and the drivers had to experience an emergency stop. The first result of this paper was that an intelligent HMI could reduce the range anxiety of the driver by proposing adapted coping strategies (i.e., transmitting how to save energy when the vehicle approaches a traffic light). The second result was that such an HMI and automated driving to a safe spot could reduce the stress of the driver when an emergency stop is necessary.

Triboelectric Touch‐Free Screen Sensor for Noncontact Gesture Recognizing

AbstractAn intelligent human–machine interface (HMI) is a crucial medium for exchanging information between people and electronics. As one of the most important HMI devices, touch screen sensors are widely applied in personal electronics in daily life. However, as the most commonly used touch screen sensor, capacitive sensors can only detect limited kinds of gestures such as touching and sliding. Here, a triboelectric touch‐free screen sensor (TSS) is reported for recognizing diverse gestures in a noncontact operating mode by utilizing the charges naturally carried on the human body. Compared with conventional capacitive sensors, the TSS is capable of detecting various gestures such as the drop and lift of finger with different speeds, making a fist, opening palm, and flipping palm with different directions. Based on the TSS, an intelligent noncontact screen control system is further developed, which is used to unlock the smartphone interface by the noncontact operating mode. This research for the first time proposes the concept that taking the human body itself to participate in triboelectric self‐powered noncontact sensing and provides a touch‐free design concept to develop the next generation of screen sensors. It can alter the usual way that people operating their personal electronics.

Toward Comprehensive Decision Support Using Multilevel Flow Modeling

Abstract The complexity of modern industrial plants poses significant challenges for the design of effective operator interfaces. Although established practices can significantly reduce the frequency of alarms, operators often cannot resolve the failure cascades commonly occurring during emergency situations. Automating control rooms by incorporating design and operation knowledge about the systems can significantly improve operator efficacy. Intelligent support systems should reduce the amount of information and provide more context to the operators. The operators focus should be shifted from information acquisition to taking informed decisions about mitigation steps. This contribution gives a brief review of the development of Multilevel Flow Modeling (MFM) and its application to provide operators with decision support and situation awareness, focusing on implementations directly utilising the knowledge represented in MFM. Finally, current efforts toward a comprehensive intelligent human machine interface for operators are outlined.