Human-machine Interface Devices Research Articles

Liquid‐Metal‐Based Soft Pressure Sensor and Multidirectional Detection by Machine Learning

AbstractElectronic skin (e‐skin) is an emerging technology with promising applications in various fields, including human–machine interfaces, prosthetics, and robotics. Soft and flexible sensors are vital components for the e‐skin that can mimic human skin's sensing capabilities. Among soft sensors, liquid‐metal‐based sensors have gained attention owing to their unique properties, such as high electrical conductivity, stretchability, and elasticity. Herein, a novel approach is presented that enables multidirectional pressure sensing with a machine‐learning approach from the transient response of the liquid‐metal‐based soft pressure sensor for the e‐skins. In this study, a soft sensor is developed that utilizes liquid metal and has an array of microchannels on a dome‐shaped structure to detect pressures from multiple directions. The transient response from six microchannels of the sensor is used as the input for a convolutional neural network (CNN) to predict the direction (classification accuracy of 99.1%) and magnitude (regression error of 20.13%) of the applied pressures in real time. Finally, a potential application of the developed liquid‐metal‐based soft sensor as a human–machine interface device is demonstrated by using it to control an RC model car through multidirectional predictions (pressure direction and magnitude) through machine learning in real time.

Solvent-Triggered, Ultra-Adhesive, Conductive, and Biocompatible Transition Gels for Wearable Devices.

The development of robust adhesive, conductive, and flexible materials has garnered significant attention in the realm of human-machine interface and electronic devices. Conventional preparation methods to achieve these exceptional properties rely on incorporating highly polar raw materials, multiple components, or solvents. However, the overexposure of functional groups and the inherent toxicity of organic solvents often render gels non-stick or potentially biocompatible making them unsuitable for human-contact devices. In this study, a straightforward three-step strategy is devised for preparing responsive adhesive gels without complex components. Structurally conductive poly(N-(2-hydroxyethyl)-acrylamide-co-p-styrene sulfonate hydrate) (PHEAA-NaSS) gels are synthesized by integrating ionic and hydrophilic networks with distinct solvent effects. Initially, the in-suit formed PHEAA-NaSS networks are activated by dimethyl sulfoxide, which substantially increases intramolecular hydrogen bonding and enhances the matrix stretchability and interfacial adhesion. Subsequently, ethanol exchange reduced solvent impact and led to a compact network that limited surface exposure of ionic and hydrophilic groups, resulting in nonstick, robust for convenient storage. Finally, upon contacting with water, the network demonstrates rehydration, resulting in favorable adhesion, biocompatibility, and conductivity. The proposed PHEAA-NaSS/W gels can stably and reliably capture joint motion and electrophysiological signals. Furthermore, this uncomplicated gel preparation method is also applicable to other electrolyte monomers.

Revolutionizing Human-Computer Interaction through Brain-Machine Interface: An EEG-Based Approach

Abstract: Human-Machine Interface (HMI) devices, such as keyboards and mice, have been the primary means of input for computers since their inception. However, these devices have limitations that make them unsuitable for individuals with motor disabilities, motor impairment, or diseases such as paralysis, muscular dystrophy, polio, cerebral palsy, and others. This limits their ability to fully engage in computer-related activities, which can have a significant impact on their quality of life. This research paper proposes that new and emerging technologies, such as Brain-Machine Interface (BMI) and Machine Learning (ML), could be utilized to design a more convenient and accessible HMI solution that improves the quality of life for individuals with physical disabilities. BMI technology enables communication between the brain and external devices, while ML can analyse data and make predictions based on patterns and models. Combining these technologies can provide a more intuitive and adaptive interface that can detect and respond to the user's intentions and needs

A Methodology for the Systematic Selection of Human-Machine Interface Device Types in Production Machinery Development

Progressive digitization accelerates the evolution of human-machine interface (HMI) technologies. In mechanical engineering, HMIs are decisive for accomplishing user-friendly, efficient, and effective machine control. Although various technologies, such as AR-glasses, have already been applied in industrial practice, mechanical engineering companies still face tremendous challenges in selecting and implementing suitable HMI devices. Therefore, this paper presents a novel methodology for systematically evaluating and selecting a suitable HMI device type for production machinery in mechanical engineering product development. The methodology provides a use case-specific process considering the company's resources, innovation strategy, and other relevant criteria enabling the systematic selection of a suitable HMI device type out of a predefined set of existing options. A case study demonstrates the applicability and practicability of the methodology.

Design and Implementation of a Regenerative Mode Electric Vehicle Test Platform for Engineering Education

In engineering education, traditional teaching approaches cannot sufficiently help students to learn electric vehicle (EV) concepts. In this study, the design of an educational test setup including all the components and all dynamics of EVs is implemented, and case studies for engineering education with practical applications are discussed. The proposed test and training platform not only provides hands-on experience for engineering students, but also the opportunity for expert users to test their own designed algorithms on the test setup through a computer and human–machine interface device. The aim of the study is to show students the effects of road slope, vehicle weight and energy recovery parameters on a light EV. In this context, five case studies have been carried out by the students, and a survey was conducted with them. The survey results show that the test setup can help them better comprehend any EV system and develop their professional knowledge and skills.

Toward a Wireless Wearable System for Bidirectional Human-Machine Interface With Gesture Recognition and Vibration Feedback

The research on bidirectional human-machine interface (BHMI) is of great significance, which can realize the interaction between human and machine. As the traditional Human-Machine Interface (HMI) device is limited by its wired connection and unidirectional functionality, this paper intends to develop a wireless wearable solution for BHMI with gesture recognition and vibration feedback. Compact design and wireless connection improve the flexibility and wearability of bidirectional interface node (BIN). By establishing a Distributed Wireless Sensor Network (DWSN), the proposed system allows up to 8 BINs to detect muscle activities simultaneously and control each Linear Resonant Actuator (LRA) independently. The data transmission throughput of the proposed system can be reduced by transmitting signal features instead of raw signals. With a sample rate of 2000 Hz and a resolution of 12 bits, BIN can achieve 94.48%±4.56% classification accuracy of six gestures. To evaluate the performance of the system’s bidirectional functionality, virtual control experiment was carried out. The experimental results show that the completion rate of the virtual control experiment reached 87% within 3.5s with vibration feedback. Overall, the outcomes of this study have great potential to promote the practical applications of bidirectional human-machine interaction, including rehabilitation therapy, fatigue prevention, virtual reality/augmented reality (VR/AR), and sensorimotor control of artificial devices.

A Novel Method for Displaying Systems Datums Based on “Cognitive Load”

<div class="section abstract"><div class="htmlview paragraph">Operator attention has been a significant focus of human factors research in recent years. This research has clarified how electronic devices and other stimuli can become distractions for vehicle operators. The research has defined a condition known as “distracted driving,” characterized by interruption of the sequence of cognitive processes essential for safe operation of a vehicle. Although “attention” has been the most often mentioned of these cognitive processes, they also include perception, memory, cognition, and planful behavior. These processes are the “cognitive demands” of safe vehicle operation. There is another issue, similar to distracted driving, that may hamper safe operation of a vehicle. That issue is the “cognitive load” of human-machine interface devices, including instrument clusters. The present paper explores the effects of cognitive load on operator response speed. It describes a novel method for displaying systems datums designed to manage cognitive load. The paper documents a pilot study where participants performed modified driving simulations, responding to a “traditional” instrument cluster and an instrument cluster based on the novel method. Average response times proved significant (1.201 seconds faster) for the novel instrument cluster (p = 4.722E-06). Two-way ANOVA identified a significant “instrument cluster” effect in participants’ responses. The results support a conclusion that the novel method manages cognitive load. The functional significance of the novel method is discussed, including an inference that it allows operators to return their eyes to the direction of travel 105 feet faster at a speed of 60 miles per hour.</div></div>

High-performance keyboard typing motion driven hybrid nanogenerator

Keyboard is the most common input device in computer and human-machine interface devices. Each keystroke during keyboard typing provides a huge amount of biomechanical energy that can be converted into useful electrical energy. Herein, we propose a high-performance electromagnetic-triboelectric hybrid nanogenerator for harvesting biomechanical energy from keyboard typing motion (KTMEH). Combining the effects of electromagnetic induction and triboelectric contact electrification, the KTMEH can simultaneously scavenge mechanical energy from keystroke motion and deliver a maximum power of 7.04 mW from electromagnetic and 1.8 mW from the triboelectric unit, from a single Typewriter key. The output performances were analyzed for different typing speeds, typing force, and key shapes. For an average typing speed of 4 characters per second (CPS), the KTMEH can scavenge a power of 5.6 mW and 1.5 mW from the electromagnetic and triboelectric unit, respectively. This hybrid configuration and highly miniaturized design enable versatile energy harvesting capability from existing commercial keyboards along with facile installation and durability. From the experimental results, it was found that the harvesting typing motion energy is capable to charge the capacitor and drive commercial electronic loads such as a hygrometer. This work expresses notable advantages of hybrid nanogenerators to develop next-generation self-powered keyboards, self-powered keystroke dynamics monitoring and sensing systems, and security systems with biometric authentication. • A high-performance keyboard typing motion-driven hybrid nanogenerator is presented. • Contact-separation of nano-wires and nanowires improves triboelectric performance by six times. • A single key can deliver a maximum power of 7.04 mW from a single keystroke. • Output performance was analyzed for different typing speeds, typing force, and key shapes. • The proposed hybrid nanogenerator can be easily installed on commercial keyboards.

A low voltage-powered soft electromechanical stimulation patch for haptics feedback in human-machine interfaces

One grand challenge in haptic human-machine interface devices is to electromechanically stimulate sensations on the human skin wirelessly by thin and soft patches under a low driving voltage. Here, we propose a soft haptics-feedback system using highly charged, polymeric electret films with an annulus-shape bump structure to induce mechanical sensations on the fingertip of volunteers under an applied voltage range of 5–20 V. As an application demonstration, a 3 × 3 actuators array is used for transmitting patterned haptic information, such as letters of ‘T’, ‘H’, ‘U’ letters and numbers of ‘0’, ‘1’, ‘2’. Moreover, together with flexible lithium batteries and a flexible circuit board, an untethered stimulation patch is constructed for operations of 1 h. The analytical model, design principle, and performance characterizations can be applicable for the integration of other wearable electronics toward practical applications in the fields of AR (augmented reality), VR (virtual reality) and robotics.

Multi-sized planar capacitive pressure sensor with ultra-high sensitivity

It is now widely accepted that the capacitive pressure sensor has potential benefits for future robotic technologies, human-machine interface, artificial intelligence and health monitoring devices. Despite this fact, the conventional stacked structure including an intermediate dielectric layer between the neighboring electrode layers usually possesses considerable device thickness and size, thus imposing limits on the applicable scale. Here we develop a planar structured pressure sensor without the integrated elastic dielectric layer, and ultra-high sensitivity is achieved enabled by the emerging charge exchange channels between neighboring electrodes induced by external touching stimuli. Each pixel can be fabricated within a several-micrometer range and a tiny pressure of 0.02 Pa would result in a 750% increase in the relative capacitance (equivalent sensitivity of 3.75 × 10 5 kPa −1 for 0–0.05 Pa, exceeding all the previous reports to date). The working mechanisms are quantitatively investigated and proved to be applicable to macro-sized devices. Simultaneously, we illustrate the capabilities of such sensors by using them to perform static pressure mapping, generating capacitance signals to reflect its own surface morphology when the AFM tip passes by, detecting fingers touching, recording real-time human breath, and distinguishing the contact position. This work develops a planar structured pressure sensor without the integrated elastic dielectric layer, and ultra-high sensitivity (3.75 × 10 5 kPa −1 for 0–0.05 Pa) is achieved enabled by the emerging charge exchange channels between neighboring electrodes induced by external touching stimuli. Such devices can sense external stimuli based on the emerging charge exchange channels and be fabricated within wide scale ranges (from several micrometers to several millimeters). The practical application, like revealing the sample surface morphology in atomic force microscope (AFM), detecting fingers touching, recording real-time human breath, and distinguishing the contact position is also demonstrated in this research. • A planar capacitive pressure sensor is developed based on the emerging charge exchange channels between electrodes. • Multi-sized devices were fabricated from several micrometers to several millimeters without the limits of elastomer layers. • Micro-sized pressure sensor can reach an equivalent sensitivity of 3.75 × 10 5 kPa −1 for 0–0.05 Pa.

Industrial control system software patterns in manufacturing

The technique of pattern-designing software for an industrial control system for standard engineering processes has been implemented, where the same type of equipment is used and the same workpiece processing sequence is implemented. Patterns for the user controller program and supervisory control and monitoring software have been developed. Pattern elements are represented by the same type of sensors and actuators, which have the same algorithms for data processing and calculating of control actions. For them data types, data storage, functionality and organization of the exchange between programmable logic controllers and human-machine interface devices are defined. The necessary correspondence was established between template projects of various levels of the control system, which made it possible to organize a network exchange by data blocks.

Large-Area Integrated Triboelectric Sensor Array for Wireless Static and Dynamic Pressure Detection and Mapping.

Large-area flexible pressure sensors are of paramount importance for various future applications, such as electronic skin, human-machine interfacing, and health-monitoring devices. Here, a self-powered and large-area integrated triboelectric sensor array (ITSA) based on coupling a triboelectric sensor array and an array chip of CD4066 through a traditional connection is reported. Enabled by a simple and cost-effective fabrication process, the size of the ITSA can be scaled up to 38 × 38 cm2 . In addition, unlike previously proposed triboelectric sensors arrays, which can only react to the dynamic interaction, this ITSA is able to detect static and dynamic pressure. Moreover, through integrating the ITSA with a signal processing circuit, a complete wireless sensing system is present. Diverse applications of the system are demonstrated in detail, including detecting pressure, identifying position, tracking trajectory, and recognizing the profile of external contact objects. Thus, the ITSA in this work opens a new route in the direction of large-area, self-powered, and wireless triboelectric sensing systems.

Triboelectric Touch‐Free Screen Sensor for Noncontact Gesture Recognizing

AbstractAn intelligent human–machine interface (HMI) is a crucial medium for exchanging information between people and electronics. As one of the most important HMI devices, touch screen sensors are widely applied in personal electronics in daily life. However, as the most commonly used touch screen sensor, capacitive sensors can only detect limited kinds of gestures such as touching and sliding. Here, a triboelectric touch‐free screen sensor (TSS) is reported for recognizing diverse gestures in a noncontact operating mode by utilizing the charges naturally carried on the human body. Compared with conventional capacitive sensors, the TSS is capable of detecting various gestures such as the drop and lift of finger with different speeds, making a fist, opening palm, and flipping palm with different directions. Based on the TSS, an intelligent noncontact screen control system is further developed, which is used to unlock the smartphone interface by the noncontact operating mode. This research for the first time proposes the concept that taking the human body itself to participate in triboelectric self‐powered noncontact sensing and provides a touch‐free design concept to develop the next generation of screen sensors. It can alter the usual way that people operating their personal electronics.

A High Precision Real-time Premature Ventricular Contraction Assessment Method based on the Complex Feature Set.

This paper presents a high precision and low computational complexity premature ventricular contraction (PVC) assessment method for the ECG human-machine interface device. The original signals are preprocessed by integrated filters. Then, R points and surrounding feature points are determined by corresponding detection algorithms. On this basis, a complex feature set and feature matrices are obtained according to the position feature points. Finally, an exponential Minkowski distance method is proposed for PVC recognition. Both public dataset and clinical experiments were utilized to verify the effectiveness and superiority of the proposed method. The results show that our R peak detection algorithm can substantially reduce the error rate, and obtained 98.97% accuracy for QRS complexes. Meanwhile, the accuracy of PVC recognition was 98.69% for the MIT-BIH database and 98.49% for clinical tests. Moreover, benefiting from the lightweight of our model, it can be easily applied to portable healthcare devices for human-computer interaction.

Mobile TWG

Abstract Invited Session on HETEROGENEOUS INTEGRATION ROADMAP – Mobile TWG. Smart phones continue to be the major human human-machine interface devices with >5Bn data subscribers and still growing! More functions & intelligence added to interact with human, machine and environment in the Global Digital Society

Self-powered wearable touchpad composed of all commercial fabrics utilizing a crossline array of triboelectric generators

In the internet of things (IoT) era, electronic textile (E-textile), which combines diverse functional parts on a fabric substrate, has attracted much attention. Among the various components for E-textile, the human-machine interface device is one of the most important. Here, a wearable fabric touchpad based on a triboelectric generator (TEG) is demonstrated. It is created only with inexpensive commercialized fabrics. Long string-shaped fabric TEGs act as the rows and columns of a crossline array, and each cross-point of a unit cell at each row and column constructs a unit pixel. The proposed touchpad accurately traces basic motions such as the vertical tapping and lateral dragging of a polytetrafluoroethylene (PTFE) stylus and generates corresponding output signals. Based on this capability, the proposed touchpad can be utilized for the handwriting of digits and for their recognition as well. Digits from 0 to 9 written on the touchpad are successfully classified by a pre-trained neural network with very high accuracy of 98%. This implies that the proposed wearable touchpad has immediate practicality. Considering its cost-effectiveness, compatibility with the textile industry, and its accurate operation, the proposed touchpad is a strong candidate for use in interface devices in the future E-textile industry.

Breath-based human–machine interaction system using triboelectric nanogenerator

Human-machine interface (HMI) is an emerging technology to exchange information between humans and electric devices. In current HMI devices, their crucial sensing components are mainly based on the interaction of physical movements and languages. However, these conventional HMI interacting methods are not friendly to some disabled persons owing to the difficulty to clearly express their intentions with gestures, limb motions, or languages. Additionally, these HMI sensing parts rely on traditional batteries, which require frequent charging or replacing. Here, we report a breath-driven single-electrode triboelectric nanogenerator (TENG) serving as a self-powered sensor to deliver control command through breathing for the HMI interaction. The TENG is on the basis of a flexible nanowire-structured polyethylene terephthalate (PET) polymer thin film, which can harvest mechanical energy of the airflow from human breathing and produce responsive electrical signals. By using this TENG as a self-powered sensor together with signal processing and transmitting circuits, a smart wireless breath-driven HMI system is further developed. Compared with the conventional HMI interacting methods, this smart system can convert real-time human breathing into command signals to control electrical appliances without relying on physical movements or languages. This work introduces a self-powered breath-based interacting method to the field of HMI technology. It could greatly reduce the threshold of using modern electrical equipment and personal electronics for disabled people, and make the HMI interaction become more convenient and captivating.

High sensitivity knitted fabric bi-directional pressure sensor based on conductive blended yarn

Flexible pressure sensors have a broad range of applications, such as physiological signal monitoring, human-machine interfacing devices and soft robotics. The main challenges are laborious fabrication process, complex structure, poor conforming to human body. In this study, we report a wearable pressure sensor based on conductive blended yarn that is made into a plain knit fabric construction. The knitted fabric can be used as both wale direction sensor and course direction sensor with different sensitivities. The underlying sensing mechanisms for the two directions are analyzed using equivalent circuit networks. The pressure sensor demonstrates high sensitivity (0.73 kPa−1) for a wide range of pressure from nearly zero to 100 kPa and a very short response time (less than 0.4 s).

Self-powered wearable keyboard with fabric based triboelectric nanogenerator

In Internet of Things (IoT) era, electronic textiles (E-textiles), which combine various functional devices on a fabric, have attracted attention. Among various components for E-textiles, the human-machine interface device, for example, a keyboard, is one of the most important parts. The triboelectric nanogenerator (TENG) can be a powerful sensing component for the interface device due to its cost-effectiveness, design flexibility and self-powered operation. Previous studies reported fabric-based sensor devices with TENG, but as yet, no device that harnesses commercial compatibility with the textile industry has been reported. It is timely to explore a low-cost TENG-based keyboard made completely of commercial fabric for early commercialization. This paper proposes a TENG-based wearable keyboard that uses only cheap commercial fabrics. Each cell in the proposed keyboard generates electrical signals according to an external touch without any power supply. After an appropriate filtering process, the proposed keyboard can detect a key stroke without any ambiguity. We verify the keyboard operation by typing a word and playing music. The material dependency of the proposed keyboard is experimentally validated with various material pairs. Finally, the endurance of the proposed keyboard against folding, repeated touches and washing is experimentally confirmed for actual applications in a real environment.

Human Machine Interface Glove Using Piezoresistive Textile Based Sensors

Human machine interface technology is focused upon new ways of interaction between human beings and machines. Gesture recognition gloves are getting increasingly popular as human-machine interface devices. Conventionally, these gloves use electronic sensors to sense different hand gestures. As electronic sensors are bulky and uncomfortable, we propose a glove with textile based piezoresistive yarn sensors. This glove is flexible, more comfortable and cheap as compared to the conventional human machine interface gloves. We have examined that this glove can effectively sense our gestures and can be used for teleoperations, sign language to speech conversion systems and gaming.