Human-machine Interaction Systems Research Articles

A bioinspired MXene-based flexible sensory neuron for tactile near-sensor computing

Multifunctional near-sensor computing sensory neurons based on the integration of flexible sensors and memristors hold promise to achieve unprecedented development in large-scale neuromorphic perception circuits and human-machine interaction systems. However, the traditional sensory-memory integrated systems are based on the integration of different device modules, which are subjected to redundant analog-to-digital conversion circuits and poor flexibility. Here, an all MXene-based flexible sensory neuron fabricated by a room temperature energy-efficient, and controllable oxidation preparation technique is reported. The near-sensor computing system is seamlessly integrated a MXene-based pressure sensor array for acquisition of tactile signals with a flexible MXene-based memristor array for simulating the implementation of perceived behavior. The tactile sensor array shows a wide range (0.1–100 kPa), high linear sensitivity (23.9 kPa−1), and the memristor array exhibits repeatable and stable bipolar resistive switching behavior (retention >100 cycles, endurance >103 s) and excellent flexibility (R=1.0 mm). As a conceptual demonstration, our devices realize high-precision recognition of handwritten digit recognition (93.21%) and human motion (97.96%), which demonstrate the feasibility for real-time health monitoring based on signals collected by sensors and artificial neural networks constructed by the memristors. The MXene-based sensor-memristor system offers a prominent contribution towards advanced application of intelligence in electronic skin and wearable electronic equipment.

Real-time multi-camera 3D human pose estimation at the edge for industrial applications

There is an increasing interest in exploiting human pose estimation (HPE) software in human–machine interaction systems. Nevertheless, adopting such a computer vision application in real industrial scenarios is challenging. To overcome occlusion limitations, it requires multiple cameras, which in turn require multiple, distributed, and synchronized HPE software nodes running on resource-constrained edge devices. We address this challenge by presenting a real-time distributed 3D HPE platform, which consists of a set of 3D HPE software nodes on edge devices (i.e., one per camera) to redundantly extrapolate the human pose from different points of view. A centralized aggregator collects the pose information through a shared communication network and merges them, in real time, through a pipeline of filtering, clustering and association algorithms. It addresses network communication issues (e.g., delay and bandwidth variability) through a two-levels synchronization, and supports both single and multi-person pose estimation. We present the evaluation results with a real case of study (i.e., HPE for human–machine interaction in an intelligent manufacturing line), in which the platform accuracy and scalability are compared with state-of-the-art approaches and with a marker-based infra-red motion capture system.

An on-skin triboelectric sensor system enabled by heterogeneous parallel channel fusion model for human-machine interactions

The highly flexible wearable device for human-machine interaction is a burgeoning technology that has garnered considerable attention among researchers. However, current wearable devices such as data gloves, wrist bands and exoskeleton devices require additional fixation aids, which interfere with the users’ movements and cause discomfort. Here, a flexible on-skin triboelectric sensor is developed for high-precision human-machine interaction, which is in the form of a thin film containing a soft substrate and two triboelectric layers with mismatched elastic modulus. The on-skin triboelectric sensors can be mounted on the forearm epidermis by their own adhesion, which is rarely perceptible and does not impede hand and wrist mobility. On the basis, the sensors can accurately measure the tiny deformations caused by muscle movement. Meanwhile, a heterogeneous parallel channel fusion (HPCF) model is proposed for advanced signal processing and recognition of sensor data, where varying signal features are extracted by different-sized convolutional kernels. The human-machine interaction system by combining the on-skin triboelectric sensors and algorithm achieves up to 99.12% accuracy when identifying 26 distinct gestures, which holds vast potential in several areas, such as gesture recognition, virtual reality, and teleoperation.

Intelligent Tribotronic Transistors Toward Tactile Near‐Sensor Computing

AbstractFor the next generation of human‐machine interaction (HMI) systems, the development of a tactile interaction unit with multimodal, high sensitivity, and real‐time perception and recognition is the key. Herein, an artificial tactile near‐sensor computing (ATNSC) unit based on a triboelectric tactile sensor and an organic synaptic transistor is reported. By introducing multi‐peak microstructures, the mechanical performance of the tactile sensor is optimized, showing a high sensitivity of 0.98 V kPa−1 in the pressure range of 0–10 kPa and maintaining 0.11 V kPa−1 at high pressures up to 350 kPa. Additionally, by designing stripe‐like convex structures on the top surface, the sensor is capable of bimodal perception in both pressure and sliding sensations. Furthermore, the organic synaptic transistor, which can be driven by tactile sensing stimuli in a variety of circumstances, is achieved utilizing an ion‐rich gelatin dielectric covered by a hydrophobic polymer coating layer. The ATNSC unit well demonstrates the stimuli‐dependent short‐term memory effect, and it enables tactile near‐sensor computing for feature action recognition in an HMI system, laying a solid foundation for the construction of intelligent interaction devices.

Antifreeze protein-based ultrastretchable and highly sensitive hydrogel for human-machine interaction

Intelligent and convenient human–computer interfaces realize efficient and close cooperation between people and the digital world. As windows of human–machine interaction (HMI), sensors directly determine the ability and quality of acquired signals and the utilization potential of HMI systems. However, it still remains challenging to develop antifreeze hydrogel sensors with freeze-resistant properties while avoiding the loss of other properties as much as possible to achieve stable HMI applications in low-temperature environments. Here, we present an Agar/PAAm dual-network hydrogel (Na+-Agar/PAAm-AFP) doped with both antifreeze protein (AFP) and sodium chloride (NaCl), which exhibits excellent antifreeze capability (−20 °C). The hydrogel integrates transparency (>90%), ultra stretchability (up to 2100%), good adhesion properties to metal, glass, and plastic, high electrical conductivity (3.41 S m−1), high sensitivity (GF = 10.23), and non-irritating to the skin. The hydrogel can be used for monitoring daily activities and glove-based HMI systems for intelligent control. In addition, based on this hydrogel, we have successfully developed an interactive system for real-time information transfer over long distances. It is expected to provide better health management monitoring in several fields and also help to advance the development of innovative and IoT technologies.

A crosstalk-free interdigital electrode piezoresistive sensor matrix-based human-machine interaction system for automatic sitting posture recognition

Sedentarism with poor posture has been a significant concern for students and office workers. In this work, for inappropriate sitting posture monitoring, we developed a human machine interaction (HMI) system based on a piezoresistive sensor matrix, which achieves crosstalk-free pressure distribution detection by the inserting diodes method (IDM) with a structure simplified by the interdigital electrode (IDE) configuration. To measure the pressure distribution, a sensor matrix of 9×14 piezoresistive sensing pixels is fabricated by the standard flexible printed circuit (FPC) technique. The single sensing pixel shows a high sensitivity in the pressure below 20 kPa, covering the range of the human-seat contact pressure, and demonstrates the highest sensitivity of 9.865% kPa−1. The combination of the IDE and the IDM eliminates the crosstalk error completely, enabling the sensor matrix to accurately measure the human-chair contact pressure distribution. To develop an HMI system for sitting posture detection, pressure distribution data from subjects of different body shapes, including sitting upright, leaning forward, slumped sitting, crossing right legs, and crossing left legs, is collected using the proposed sensor matrix and used for training and evaluation of the convolution neural network (CNN) model. Accurate posture classification is achieved both in within-group and cross-group tests, showing the highest accuracy of 99.14% and an excellent generalization performance. With an excellent performance of crosstalk-free pressure distribution detection, the device in this study exhibits great application potential in daily sitting posture monitoring.

Ultraflexible Temperature-Strain Dual-Sensor Based on Chalcogenide Glass-Polymer Film for Human-Machine Interaction.

Skin-like thermoelectric (TE) films with temperature- and strain-sensing functions are highly desirable for human-machine interaction systems and wearable devices. However, current TE films still face challenges in achieving high flexibility and excellent sensing performance simultaneously. Herein, for the first time, a facile roll-to-roll strategy is proposed to fabricate an ultraflexible chalcogenide glass-polytetrafluoroethylene composite film with superior temperature- and strain-sensing performance. The unique reticular network of the composite film endows it with efficient Seebeck effect and flexibility, leading to a high Seebeck coefficient (731 µV/K), rapid temperature response (≈0.7 s), and excellent strain sensitivity (gauge factor = 836). Based on this high-performance composite film, an intelligent robotic hand for action feedback and temperature alarm is fabricated, demonstrating its great potential in human-machine interaction. Such TE film fabrication strategy not only brings new inspiration for wearable inorganic TE devices, but also sets the stage for a wide implementation of multifunctional human-machine interaction systems.

Ultrafast universal fabrication of configurable porous silicone-based elastomers by Joule heating chemistry

Silicone-based elastomers (SEs) have been extensively applied in numerous cutting-edge areas, including flexible electronics, biomedicine, 5G smart devices, mechanics, optics, soft robotics, etc. However, traditional strategies for the synthesis of polymer elastomers, such as bulk polymerization, suspension polymerization, solution polymerization, and emulsion polymerization, are inevitably restricted by long-time usage, organic solvent additives, high energy consumption, and environmental pollution. Here, we propose a Joule heating chemistry method for ultrafast universal fabrication of SEs with configurable porous structures and tunable components (e.g., graphene, Ag, graphene oxide, TiO2, ZnO, Fe3O4, V2O5, MoS2, BN, g-C3N4, BaCO3, CuI, BaTiO3, polyvinylidene fluoride, cellulose, styrene-butadiene rubber, montmorillonite, and EuDySrAlSiOx) within seconds by only employing H2O as the solvent. The intrinsic dynamics of the in situ polymerization and porosity creation of these SEs have been widely investigated. Notably, a flexible capacitive sensor made from as-fabricated silicone-based elastomers exhibits a wide pressure range, fast responses, long-term durability, extreme operating temperatures, and outstanding applicability in various media, and a wireless human-machine interaction system used for rescue activities in extreme conditions is established, which paves the way for more polymer-based material synthesis and wider applications.

Smart touchless human–machine interaction based on crystalline porous cages

The rise of touchless technology, driven by the recent pandemic, has transformed human-machine interaction (HMI). Projections indicate a substantial growth in the touchless technology market, nearly tripling from $13.6 billion in 2021 to an estimated $37.6 billion by 2026. In response to the pandemic-driven shift towards touchless technology, here we show an organic cage-based humidity sensor with remarkable humidity responsiveness, forming the basis for advanced touchless platforms in potential future HMI systems. This cage sensor boasts an ultrafast response/recovery time (1 s/3 s) and exceptional stability (over 800 cycles) across relative humidity (RH) changes from 11% to 95%. The crystal structure’s 3D pore network and luxuriant water-absorbing functional groups both inside and outside of the cage contribute synergistically to superior humidity sensing. Demonstrating versatility, we showcase this cage in smart touchless control screens and touchless password managers, presenting cost-effective and easily processable applications of molecularly porous materials in touchless HMI.

Estimating and approaching the maximum information rate of noninvasive visual brain-computer interface

An essential priority of visual brain-computer interfaces (BCIs) is to enhance the information transfer rate (ITR) to achieve high-speed communication. Despite notable progress, noninvasive visual BCIs have encountered a plateau in ITRs, leaving it uncertain whether higher ITRs are achievable. In this study, we used information theory to study the characteristics and capacity of the visual-evoked channel, which leads us to investigate whether and how we can decode higher information rates in a visual BCI system. Using information theory, we estimate the upper and lower bounds of the information rate with the white noise (WN) stimulus. Consequently, we found out that the information rate is determined by the signal-to-noise ratio (SNR) in the frequency domain, which reflects the spectrum resources of the channel. Based on this discovery, we propose a broadband WN BCI by implementing stimuli on a broader frequency band than the steady-state visual evoked potentials (SSVEPs)-based BCI. Through validation, the broadband BCI outperforms the SSVEP BCI by an impressive 7 bps, setting a record of 50 bps. The integration of information theory and the decoding analysis presented in this study offers valuable insights applicable to general sensory-evoked BCIs, providing a potential direction of next-generation human-machine interaction systems.

Neuro-Symbolic Reasoning for Multimodal Referring Expression Comprehension in HMI Systems

AbstractConventional Human–Machine Interaction (HMI) interfaces have predominantly relied on GUI and voice commands. However, natural human communication also consists of non-verbal communication, including hand gestures like pointing. Thus, recent works in HMI systems have tried to incorporate pointing gestures as an input, making significant progress in recognizing and integrating them with voice commands. However, existing approaches often treat these input modalities independently, limiting their capacity to handle complex multimodal instructions requiring intricate reasoning of language and gestures. On the other hand, multimodal tasks requiring complex reasoning are being challenged in the language and vision domain, but these typically do not include gestures like pointing. To bridge this gap, we explore one of the challenging multimodal tasks, called Referring Expression Comprehension (REC), within multimodal HMI systems incorporating pointing gestures. We present a virtual setup in which a robot shares an environment with a user and is tasked with identifying objects based on the user’s language and gestural instructions. Furthermore, to address this challenge, we propose a hybrid neuro-symbolic model combining deep learning’s versatility with symbolic reasoning’s interpretability. Our contributions include a challenging multimodal REC dataset for HMI systems, an interpretable neuro-symbolic model, and an assessment of its ability to generalize the reasoning to unseen environments, complemented by an in-depth qualitative analysis of the model’s inner workings.

Kinematic-driven human-robot interaction system with deep learning for flexible acupuncture needling manipulations

Acupuncture is an important therapeutic method in traditional Chinese medicine and essentially characterized by flexible needling manipulations that are complicated for repetitions. In this work, we designed a bio-inspired acupuncture robot combined with human–machine interaction system that can implement flexible and precise needle manipulations as manual acupuncture. In the proposed framework, the manual needling process was recorded with online imaging techniques. With a deep learning-based acupuncture estimator, kinetic parameters of lift insertion (LI) and twist rotation (TR) manipulations were extracted in three-dimensional space and the precent of correct key points for operator’s hand pose estimation achieved 94.03%. It is founded that the variation of operator’s finger vector angle for LI manipulation was much weaker than TR manipulation. Furthermore, inspired by kinematic characteristics of manual acupuncture, a six-degree-of-freedom robot was used to simulate the coordinated motion of hand joints in needling manipulations through robot kinematics analysis and trajectory planning with polynomial interpolation method. Numerical and experiential results showed that the acupuncture robot can mimic various needling manipulations as manual acupuncture. The flexibility and stability of the robotic acupuncture framework were evaluated for a wide range of needling parameters. It was demonstrated that the robot can follow the periodic dynamics of LI and TR manipulations with various frequencies and amplitudes, and perform accurate acupuncture operations with smooth needling trajectories as joint jitters during needle manipulations were completely eliminated. The presented results provide a new strategy for modern applications of acupuncture treatment with robotic and flexible needling manipulations to improve the therapeutic efficacy.

Syllable-rate-adjusted-modulation (SRAM) predicts clear and conversational speech intelligibility.

Objectively predicting speech intelligibility is important in both telecommunication and human-machine interaction systems. The classic method relies on signal-to-noise ratios (SNR) to successfully predict speech intelligibility. One exception is clear speech, in which a talker intentionally articulates as if speaking to someone who has hearing loss or is from a different language background. As a result, at the same SNR, clear speech produces higher intelligibility than conversational speech. Despite numerous efforts, no objective metric can successfully predict the clear speech benefit at the sentence level. We proposed a Syllable-Rate-Adjusted-Modulation (SRAM) index to predict the intelligibility of clear and conversational speech. The SRAM used as short as 1 s speech and estimated its modulation power above the syllable rate. We compared SRAM with three reference metrics: envelope-regression-based speech transmission index (ER-STI), hearing-aid speech perception index version 2 (HASPI-v2) and short-time objective intelligibility (STOI), and five automatic speech recognition systems: Amazon Transcribe, Microsoft Azure Speech-To-Text, Google Speech-To-Text, wav2vec2 and Whisper. SRAM outperformed the three reference metrics (ER-STI, HASPI-v2 and STOI) and the five automatic speech recognition systems. Additionally, we demonstrated the important role of syllable rate in predicting speech intelligibility by comparing SRAM with the total modulation power (TMP) that was not adjusted by the syllable rate. SRAM can potentially help understand the characteristics of clear speech, screen speech materials with high intelligibility, and convert conversational speech into clear speech.

Performance-Enhanced Flexible Self-Powered Tactile Sensor Arrays Based on Lotus Root-Derived Porous Carbon for Real-Time Human-Machine Interaction of the Robotic Snake.

Flexible tactile sensors play an important role in the development of wearable electronics and human-machine interaction (HMI) systems. However, poor sensing abilities, an indispensable external energy supply, and limited material selection have significantly constrained their advancement. Herein, a self-powered flexible triboelectric sensor (TES) is proposed by integrating lotus-root-derived porous carbon (PC) into polydimethylsiloxane (PDMS). Owing to the superior charge capturing capability of PC, the PDMS/PC (PPC)-based TES exhibits an open-circuit voltage (Voc) of 22.8 V when it is periodically patted by skin at the pressure of 2 N and the frequency of 1 Hz, which is 5 times higher than that of a pristine PDMS-based TES. Furthermore, the as-prepared self-powered TES exhibits a high sensitivity of 3.24 V kPa-1 below 15 kPa for detecting human motion signals, such as finger clicks, joint bends, etc. Last but not the least, after the assembly of a PPC-based TES array and construction of an HMI system, the robotic snake can be controlled remotely by recognizing finger touching signals. This work shows broad potential applications for the self-powered TES in the fields of intelligent robotics, flexible electronics, disaster relief, and intelligence spying.

Machine Learning-Enabled Environmentally Adaptable Skin-Electronic Sensor for Human Gesture Recognition.

Stretchable sensors have been widely investigated and developed for the purpose of human motion detection, touch sensors, and healthcare monitoring, typically converting mechanical/structural deformation into electrical signals. The viscoelastic strain of stretchable materials often results in nonlinear stress-strain characteristics over a broad range of strains, consequently making the stretchable sensors at the body joints less accurate in predicting and recognizing human gestures. Accurate recognition of human gestures can be further deteriorated by environmental changes such as temperature and humidity. Here, we demonstrated an environment-adaptable high stress-strain linearity (up to ε = 150%) and high-durability (>100,000 cycles) stretchable sensor conformally laminated onto the body joints for human gesture recognition. The serpentine configuration of our ionic liquid-based stretchable film enabled us to construct broad data sets of mechanical strain and temperature changes for machine learning-based gesture recognition. Signal recognition and training of distinct strains and environmental stimuli using a machine learning-based algorithm analysis successfully measured and predicted the joint motion in a temperature-changing environment with an accuracy of 92.86% (R-squared). Therefore, we believe that our serpentine-shaped ion gel-based stretchable sensor harmonized with machine-learning analysis will be a significant achievement toward environmentally adaptive and multianalyte sensing applications. Our proposed machine learning-enabled multisensor system may enable the development of future electronic devices such as wearable electronics, soft robotics, electronic skin, and human-machine interaction systems.

Human-machine interactions based on hand gesture recognition using deep learning methods

Human interaction with computers and other machines is becoming an increasingly important and relevant topic in the modern world. Hand gesture recognition technology is an innovative approach to managing computers and electronic devices that allows users to interact with technology through gestures and hand movements. This article presents deep learning methods that allow you to efficiently process and classify hand gestures and hand gesture recognition technologies for interacting with computers. This paper discusses modern deep learning methods such as convolutional neural networks (CNN) and recurrent neural networks (RNN), which show excellent results in gesture recognition tasks. Next, the development and implementation of a human-machine interaction system based on hand gesture recognition is discussed. System architectures are described, as well as technical and practical aspects of their application. In conclusion, the article summarizes the research results and outlines the prospects for the development of hand gesture recognition technology to improve human-machine interaction. The advantages and limitations of the technology are analyzed, as well as possible areas of its application in the future.

Single Channel Based Interference-Free and Self-Powered Human-Machine Interactive Interface Using Eigenfrequency-Dominant Mechanism.

The recent development of wearable devices is revolutionizing the way of human-machine interaction (HMI). Nowadays, an interactive interface that carries more embedded information is desired to fulfill the increasing demand in era of Internet of Things. However, present approach normally relies on sensor arrays for memory expansion, which inevitably brings the concern of wiring complexity, signal differentiation, power consumption, and miniaturization. Herein, a one-channel based self-powered HMI interface, which uses the eigenfrequency of magnetized micropillar (MMP) as identification mechanism, is reported. When manually vibrated, the inherent recovery of the MMP causes a damped oscillation that generates current signals because of Faraday's Law of induction. The time-to-frequency conversion explores the MMP-related eigenfrequency, which provides a specific solution to allocate diverse commands in an interference-free behavior even with one electric channel. A cylindrical cantilever model is built to regulate the MMP eigenfrequencies via precisely designing the dimensional parameters and material properties. It is shown that using one device and two electrodes, high-capacity HMI interface can be realized when the magnetic micropillars (MMPs) with different eigenfrequencies have been integrated. This study provides the reference value to design the future HMI system especially for situations that require a more intuitive and intelligent communication experience with high-memory demand.

Recent Progress of Nature Materials Based Triboelectric Nanogenerators for Electronic Skins and Human–Machine Interaction

The current field of human‐machine interaction (HMI) and electronic skin (e‐skin) is confronting challenges associated with energy supply, sensitivity, and biocompatibility. Traditional HMI systems, though rich in features, are constrained in their applications in the medical and wearable technology sectors due to their rigid structure and reliance on third‐party power inputs. Triboelectric nanogenerators (TENGs) based on natural materials are emerging as a focal point of research in this domain, attributed to their self‐powering capabilities, flexibility, and biocompatibility. These natural material‐based TENGs emulate the mechanical and biological characteristics of human skin, adapt to various body shapes and movements, and offer an efficient and reliable solution for the precise detection and analysis of complex, subtle physiological signals, thereby fostering innovations in wearable devices and robotic technology. This article provides a comprehensive review of recent advancements in natural material‐based TENGs, detailing the materials employed, including proteins, chitosan, and cellulose. It encapsulates their applications in the realms of electronic skin and HMI. Finally, the challenges confronted by natural material‐based TENGs are outlined and future trajectories for development in this burgeoning field are projected.

Recent Progress of Bioinspired Triboelectric Nanogenerators for Electronic Skins and Human–Machine Interaction

Advances in biomimetic triboelectric nanogenerators (TENGs) have significant implications for electronic skin (e-skin) and human–machine interaction (HMI). Emphasizing the need to mimic complex functionalities of natural systems, particularly human skin, TENGs leverage triboelectricity and electrostatic induction to bridge the gap in traditional electronic devices’ responsiveness and adaptability. The exploration begins with an overview of TENGs’ operational principles and modes, transitioning into structural and material biomimicry inspired by plant and animal models, proteins, fibers, and hydrogels. Key applications in tactile sensing, motion sensing, and intelligent control within e-skins and HMI systems are highlighted, showcasing TENGs’ potential in revolutionizing wearable technologies and robotic systems. This review also addresses the challenges in performance enhancement, scalability, and system integration of TENGs. It points to future research directions, including optimizing energy conversion efficiency, discovering new materials, and employing micro-nanostructuring techniques for enhanced triboelectric charges and energy conversion. The scalability and cost-effectiveness of TENG production, pivotal for mainstream application, are discussed along with the need for versatile integration with various electronic systems. The review underlines the significance of making bioinspired TENGs more accessible and applicable in everyday technology, focusing on compatibility, user comfort, and durability. Conclusively, it underscores the role of bioinspired TENGs in advancing wearable technology and interactive systems, indicating a bright future for these innovations in practical applications.

EMG-based Multi-User Hand Gesture Classification via Unsupervised Transfer Learning Using Unknown Calibration Gestures.

The poor generalization performance and heavy training burden of the gesture classification model contribute as two main barriers that hinder the commercialization of sEMG-based human-machine interaction (HMI) systems. To overcome these challenges, eight unsupervised transfer learning (TL) algorithms developed on the basis of convolutional neural networks (CNNs) were explored and compared on a dataset consisting of 10 gestures from 35 subjects. The highest classification accuracy obtained by CORrelation Alignment (CORAL) reaches more than 90%, which is 10% higher than the methods without using TL. In addition, the proposed model outperforms 4 common traditional classifiers (KNN, LDA, SVM, and Random Forest) using the minimal calibration data (two repeated trials for each gesture). The results also demonstrate the model has a great transfer robustness/flexibility for cross-gesture and cross-day scenarios, with an accuracy of 87.94% achieved using calibration gestures that are different with model training, and an accuracy of 84.26% achieved using calibration data collected on a different day, respectively. As the outcomes confirm, the proposed CNN TL method provides a practical solution for freeing new users from the complicated acquisition paradigm in the calibration process before using sEMG-based HMI systems.