Human-machine Interaction Research Articles

Teleoperation system for multiple robots with intuitive hand recognition interface

Robotic teleoperation is essential for hazardous environments where human safety is at risk. However, efficient and intuitive human–machine interaction for multi-robot systems remains challenging. This article aims to demonstrate a robotic teleoperation system, denominated AutoNav, centered around autonomous navigation and gesture commands interpreted through computer vision. The central focus is on recognizing the palm of the hand as a control interface to facilitate human–machine interaction in the context of multi-robots. The MediaPipe framework was integrated to implement gesture recognition from a USB camera. The system was developed using the Robot Operating System, employing a simulated environment that includes the Gazebo and RViz applications with multiple TurtleBot 3 robots. The main results show a reduction of approximately 50% in the execution time, coupled with an increase in free time during teleoperation, reaching up to 94% of the total execution time. Furthermore, there is a decrease in collisions. These results demonstrate the effectiveness and practicality of the robotic control algorithm, showcasing its promise in managing teleoperations across multi-robots. This study fills a knowledge gap by developing a hand gesture-based control interface for more efficient and safer multi-robot teleoperation. These findings enhance human–machine interaction in complex robotic operations. A video showing the system working is available at https://youtu.be/94S4nJ3IwUw.

Nerve-Inspired Optical Waveguide Stretchable Sensor Fusing Wireless Transmission and AI Enabling Smart Tele-Healthcare.

Flexible strain monitoring of hand and joint muscle movement is recognized as an effective method for the diagnosis and rehabilitation of neurological diseases such as stroke and Parkinson's disease. However, balancing high sensitivity and large strain, improving wearing comfort, and solving the separation of diagnosis and treatment are important challenges for further building tele-healthcare systems. Herein, a hydrogel-based optical waveguide stretchable (HOWS) sensor is proposed in this paper. A double network structure is adopted to allow the HOWS sensor to exhibit high stretchability of the tensile strain up to 600% and sensitivity of 0.685mV %-1. Additionally, the flexible smart bionic fabric embedding the HOWS sensor, produced through the warp and weft knitting, significantly enhances wearing comfort. A small circuit board is prepared to enable wireless signal transmission of the designed sensor, thereby improving the daily portability. A speech recognition and human-machine interaction system, based on sensor signal acquisition, is constructed, and the convolutional neural network algorithm is integrated for disease assessment. By integrating sensing, wireless transmission, and artificial intelligence (AI), a smart tele-healthcare system based on HOWS sensors is demonstrated to hold great potential for early warning and rehabilitation monitoring of neurological diseases.

Cognitive digital twin in manufacturing process: integrating the knowledge graph for enhanced human-centric Industry 5.0

Industry 5.0 emphasises human-centric intelligent manufacturing, posing challenges in integrating human expertise with advanced machine capabilities. To address these challenges, a novel three-layer cognitive digital twin model based on knowledge graphs is proposed, designed to integrate workers’ knowledge and experience into intelligent manufacturing processes. This model comprises three layers: an ontology layer that constructs a foundational process knowledge ontology library; a knowledge layer that maps real-time data to dynamically update digital models; and a cognitive layer that utilises machine learning, knowledge reasoning, and knowledge mining for advanced analysis, state understanding, and model evolution. The model promotes user interaction through intuitive interfaces and a Q&A system, leveraging techniques such as knowledge reasoning and querying to support decision-making and enhance worker engagement. Validated through a system implemented for aero-engine blade production, this cognitive digital twin model leverages human expertise and machine capabilities to enhance process control, quality management, and overall efficiency. The proposed approach demonstrates significant potential for advancing personalised human-machine interaction in manufacturing, truly embodying the value of a human-centric approach and paving the way for future developments in the field.

Magnetron sputtering preparation of flexible ZnO/AlN thin-films sensors with hybrid piezoelectric effect for broad-range human motions detection

Wearable sensors with excellent comfort, flexibility and fast response are widely applied in human-machine interactions, artificial skin, motion detection and healthcare monitoring. However, preparation processes of the most of current piezoelectric sensors are relatively expensive and complicate, limiting their mass productions in the applications. This work demonstrates the superior piezoelectric performance and stability of ZnO/AlN flexible thin films prepared by magnetron sputtering for broad-range human motions detection, and first-principles calculation are applied to reveal the complicate piezoelectric effect of the hybrid system. The ZnO/AlN hybrid thin films sensor exhibits superior piezoelectric performance and excellent reliability, with presenting high outputs voltage (3.63 V), high degree of response speed (42.33 ms), water stability and robust performance upon 10000 cycles. First-principles calculations reveal that formation of the ZnO/AlN interface facilitates charge transport and improves efficiency of carrier transport, and results in reduced in band gap and enhanced electrical conductivity. By applying the sensors in human health monitoring, the underlying behaviors can be determined by output voltage of the sensor in real-time. This work offers a sensitive, simple structured and robust solution for human motion detection.

Optimized multi-layer self-attention network for feature-level data fusion in emotion recognition

<p>Understanding human emotions across diverse data sources presents challenges in various applications including healthcare, human-machine interaction, security, marketing, and gaming. Prior research has explored fusion techniques to address multimodal data heterogeneity, yet often overlooks the importance of discriminative unimodal information and potential complementarity among fusion strategies. Recognizing emotions from video and audio data poses challenges such as non-verbal cues interpretation, varying expression, ambiguity in context, and the need for nuanced feature extraction to capture subtle emotional nuances accurately. To tackle these issues, it is imperative to employ efficient emotion representation and multimodal fusion techniques, as these tasks have significant importance within the realm of multifaceted recognizing study. This study introduced a novel approach, optimized multi-layer self-attention network for emotion recognition (OMSN-ER), focusing on feature-level data fusion. OMSN-ER precisely assesses emotional states by merging facial and voice data, utilizing a multi-layer progressive dense residual fusion network and a self-attention mountain gazelle convolution neural network. Implemented in Python with the RAVDESS dataset, the methodology achieves exceptional accuracy (0.9908), surpassing benchmarks and demonstrating efficacy in multimodal emotion recognition. This research represents promising advancements in the intricate field of emotion recognition.</p>

Environment-tolerant, inherently conductive and self-adhesive gelatin-based supramolecular eutectogel for flexible sensor

Although hydrogels have attracted increasing attention in the stretchable devices, the low adhesion properties and poor environmental adaptation still seriously restrict their development and application. Herein, we focused on the interaction between polymer networks with disperse media and their resultant influence on gel performance, and constructed self-adhesive and environment-tolerant gelatin/polyacrylamide supramolecular-polymer double-network (Gelatin/PAM SP-DN) eutectogels using multiple supramolecular interactions between natural macromolecule and well-designed deep eutectic solvent (DES). The dual networks of Gelatin/PAM SP-DN eutectogels produced significant supramolecular forces with DES, including hydrogen bonding and electrostatic interaction, contributing to enhance the energy dissipation capacity. Additionally, the Gelatin-PAM SP-DN eutectogels were more prone to generate strong bonding force to various substrates, showcasing both in-situ and ex-situ adhesion performance, and even being used for wet and underwater adhesion. The eutectogels revealed excellent environmental tolerance to maintain excellent mechanical flexibility, conductivity and adhesion at high and low temperatures, ensuring the constructed sensor to sensitively and reliably perceive strain, pressure and human motions over a wide temperature range. Also, the eutectogel demonstrated great potential as a temperature sensor. This work opens up a new horizon in the design of multifunctional and environment-tolerant natural macromolecule-based gel materials for flexible electronics, human-machine interaction and health diagnosis.

Patient activity recognition for robust performance under varied illumination and cluttered background in indoor environment

Human Activity Recognition (HAR) holds significant importance in health and human-machine interaction. However, recognizing actions from 2D information faces challenges like occlusion, illumination variation, cluttered backgrounds, and view invariance. These hurdles are particularly pronounced in indoor patient monitoring settings due to fluctuating lighting conditions and cluttered backgrounds, which compromise the accuracy of activity recognition systems. A new architecture named illuminationRevive has been proposed to tackle this issue. Which utilizes an encoder-decoder convolutional neural network and image post-processing blocks to enhance the image's visual appearance. A new dataset comprising seven indoor physical activities has been proposed, created with contributions from thirty individuals aged 20 to 45. A hybrid fusion architecture is proposed to classify activities, integrating motion sequence information and body joint features. The proposed classification model incorporates generated Skeleton Motion History Images (SMHIs), collected human joint motion features from video frames, and novel kinematic and geometric features within window frames as inputs. The model can extract spatial and temporal feature vectors by integrating ResNet50-ViT (Residual Network-50 layers, Vision Transformer) and CNN-BiLSTM (Convolutional Neural Network-Bidirectional Long Short-Term Memory) layers. The suggested classification model was evaluated alongside state-of-the-art models using the LNMIIT-SMD (The LNM Institute of Information Technology-Skelton Motion Dataset) and established NTU-RGBD (Nanyang Technological University's Red Blue Green and Depth information) dataset. The evaluation aimed to assess the effectiveness of the proposed classification model architecture. Results demonstrate the superiority of the proposed model, achieving impressive accuracies of 98.21% on real-time data, 98.45% on the proposed dataset, and 97.12% on the NTU-RGBD dataset.

Hydrogel Strain Sensors Based on Ordered Iron Nanowires for Human–Machine Interaction

Hydrogel Strain Sensors Based on Ordered Iron Nanowires for Human–Machine Interaction

Wave-transparent and eco-friendly flame-retardant phosphorus-based phthalonitrile/quartz composites by a synchronized self-curing strategy with cyano and alkynyl groups

The robust human–machine interaction systems of the aerospace industry necessitate the development of thermal resistant wave-transparent composites, with a pivotal focus on organic resin matrix. However, enhancing the heat resistance of such materials while preserving other critical attributes has been a formidable challenge. In this study, a novel designed strategy of a dual-curing functional phosphorus-based phthalonitrile monomer (P-ALK-PN) has been proposed. The optimized curing temperatures for intra- or intermolecular Diels-Alder reactions of alkynyl groups and polyaddition reaction of cyano groups resulted in a homogeneous and highly crosslinked N-enriched phthalonitrile system. After curing at 450 °C, the resin, namely P-ALK-PN-450 °C, demonstrated exceptional thermal stability (T5% = 644 °C), thermo-oxidative stability (T5% = 554 °C), and char yield at 800 °C (90.7 % in N2 and 67.5 % in air). Their quartz fiber reinforced composites (P-ALK-PN/QFs) were subsequently fabricated by the hot-pressing process. Upon post-curing at 420 °C, P-ALK-PN-420 °C/QF displayed elevated glass transition temperature (550 °C), flexural strength (319 MPa), and relatively low dielectric properties with a dielectric constant (Dk) of approximately 4.2 and a dissipation factor (Df) less than 0.02 across a wide temperature ranging from 25 °C to 600 °C. Especially, it exhibited excellent flame retardancy (only a 6.7 % weight loss at ambient temperatures exceeding 1300 °C for 200 s) as well, stemming from release of eco-friendly inert gases (NH3) and formation of a graphitized protective layer during pyrolysis. These promising results highlight the potential applications of P-ALK-PN resins in aerospace and high-performance engineering fields.

Multilevel Cu-LIG Tactile Sensing Arrays for 3D Touch Human-Machine Interaction.

High-performance flexible tactile sensors have attracted significant attention in the domains of human-machine interactions. However, the efficient fabrication of sensors with highly sensitive responses over a broad load range still remains a challenge. Here, we propose a one-step laser writing route to construct a distinctive multilevel piezoresistive structure, consisting of Cu nanoparticle-doped graphene protrusions and surrounding porous Cu sheets. This multilevel structure enables the assembled tactile sensors to exhibit superior sensitivity at both low-pressure (1468 kPa-1 at 0-200 kPa) and high-pressure (1345 kPa-1 at 600-800 kPa) stimulations. Its enhancement mechanism for piezoresistive sensing has been investigated. The programmable laser writing process facilitates the development of human-machine interaction devices that recognize multidimensional gestures such as sliding, clicking, and pressing. This advancement serves to promote the development of high-performance interactive sensing technologies.

Trap Assisted Dynamic Mechanoluminescence Toward Self-Referencing and Visualized Strain Sensing.

Strain sensors utilizing mechanoluminescent (ML) materials have garnered significant attention and application due to their advantages, such as self-powering, non-contact operation, and real-time response. However, ML-based strain sensing techniques typically rely on the establishing of a mathematical relationship between ML intensity and mechanical parameters. The absolute ML intensity is vulnerable to environmental factors, which can result in measurement errors. Herein, an color-resolved visualized dynamic ML and self-referencing strain sensing is investigated in Ca9Al(PO4)7: Tb3+, Mn2+. By analyzing the ML performance under various mechanical stimulations and adjustable strain parameters, a relationship between strain and the ML intensity ratio of Tb3+/Mn2+ is aimed to bed established. This will enable the development of a self-referencing and visualized strain sensing technology. Through a comparison of luminescence characteristics under continuous mechanical stimulation (stretching) and continuous X-ray irradiation, it is discovered that the ratiometric dynamic ML is primarily driven by the dynamic filling and continuous release of carriers form traps, which compensates for the ML of Mn2+. Leveraging the self-referencing and color-resolved (from green to red) visualized ML characteristics, an application scenario for monitoring human joint movement is developed. This approach offers new insights into the use of dynamic ML materials in strain sensing and human-machine interaction.

High fundamental frequency QCM humidity sensor based on C60-OH/Ti3C2TX nanocomposite with superior response

The development of high-response quartz crystal microbalance (QCM) humidity sensors typically requires the use of substantial amounts of moisture-sensitive materials, which can lead to slow response and recovery times, as well as inadequate hysteresis performance. In this study, a humidity sensor based on a high fundamental frequency (19.6 MHz) QCM transducer is prepared with fullerenol (C60-OH) nanodot-modified Ti3C2TX as the moisture-sensitive material. With only a tiny amount (1 μg) of material coated, the prepared high-frequency (19.6 MHz) QCM sensor exhibits an unexpectedly strong response to humidity, manifesting a maximum frequency shift up to −12,839 Hz in the range of 2–97 % RH, which is 10 times higher than a low-frequency (10 MHz)-based QCM sensor. Additionally, high-frequency (19.6 MHz) QCM sensor demonstrates excellent sensing performance in low humidity, with a sensitivity of −40.2 Hz/%RH in the range of 0.2–10 % RH. This high response is primarily ascribed to low initial mass of high fundamental frequency QCM and the improvement in moisture-sensing capabilities of Ti3C2TX due to the incorporation of C60-OH. With its outstanding properties, the high-frequency QCM sensor holds great potential for application in human-machine interaction.

Research Progress and Key Technology Analysis of Lower Limb Rehabilitation Robots

In China, there is a large number of patients with lower limb movement disorders and limited medical resources, while the market for rehabilitation technologies is growing rapidly. The technology of lower limb rehabilitation exoskeleton robots has been successfully applied and shows significant development potential. Both domestic and international research on lower limb rehabilitation exoskeleton robots focuses on the design aspects of comfort, bionics, lightweight construction, and safety. Notable domestic and international companies and universities are committed to enhancing personalization and human-machine interaction experiences, reducing costs, and expanding application fields. A preliminary analysis of the forward and inverse kinematics of lower limb rehabilitation exoskeleton robots has been conducted to derive the kinematic equations and velocity Jacobian matrices, facilitating better optimization of the robot's control algorithms. Existing control algorithms are diverse, including position-based, force information-based, bioelectric signal-based, and intelligent control strategies, each with its own advantages and disadvantages. Among these, control algorithms that integrate artificial intelligence and other cutting-edge technologies hold the most promise. As the aging population continues to grow, this technology is expected not only to play a role in rehabilitation therapy but also to expand into other areas, such as assistive walking, along with the optimization of mechanisms and the evolution of control algorithms.

Exploring the Potential of Augmented Reality (AR) and Virtual Reality (VR) in Developing Immersive Experiences for Education, Entertainment, and Training: Innovations, Applications, and Future Prospects

Augmented reality (AR) and virtual reality (VR) have become groundbreaking technologies that are revolutionizing the way we interact with digital environments. These innovations have great potential in education, entertainment, and teaching by providing a fantastic and interactive experience to bridge the gap between the real and virtual worlds. AR and VR in education create engaging learning environments that encourage deeper understanding through simulations and hands-on virtual experiments. The entertainment industry is leveraging technology to create immersive games, videos, and live events that increase user engagement. AR and VR educational tools also enable safe, effective, and realistic simulations, especially in high-risk fields such as medicine, aviation, and defense. This technology often relies on hardware such as headsets, audio sensors, and haptics, supported by advanced software that can create dynamic 3D environments. Challenges include high construction costs, accessibility, and ensuring long-term user comfort. However, ongoing advances in hardware miniaturization, cloud computing, and artificial intelligence continue to overcome these challenges and encourage the integration of AR and VR into real-world applications. This study explores current innovations, applications, and future prospects for AR and VR, and highlights their potential to transform industries and redefine human-machine interactions.

Bioinspired Magnetized String with Tension-Dependent Eigenfrequencies for Wearable Human-Machine Interactions.

Flexible and wearable devices have exhibited potential for applications in the fields of human-machine interactions (HMIs) and Internet of Things. However, challenges remain in the improvement of the communication storage capacity with a simplified architecture. Inspired by tension regulation in natural tendons, a single-channel wearable HMI strategy is proposed using the eigenfrequency of magnetized strings as a sensing solution. Based on electromagnetic induction, mechanical vibration of the magnetized string can electrically induce periodical damping signals in the coil that are associated with the intrinsic eigenfrequency property of the string. Using a theoretical vibration model, nonoverlapping eigenfrequencies are precisely customized by designing the dimension/modulus or tension status of the string to broaden the eigenfrequency library. By integrating strings with different eigenfrequencies, multiple commands can be realized with a single communication channel. Moreover, identifiable commands can be flexibly tuned with only one magnetized string by customizing the tensile length (string tension) for eigenfrequency regulation. Demonstrations such as tactile addressing, authentication systems, and robotic control indicate the potential of the interface for multifunctional HMI applications. We expect that this strategy will provide a valuable reference for the future design of wearable HMI interfaces with high storage capacity and controllability in an accessible architecture.

Interactive Haptic System with Multimodal Tactile Sensing and Hydraulic Feedback for Realistic Human–Machine Interaction

With the increasing demand for immersive haptic experiences, multimodal tactile sensing and feedback have become crucial for enhancing user interaction. In this work, an interactive haptic system comprising a robotic hand and wearable tactile feedback device is introduced to facilitate tactile information exchange between humans and robots. The robotic hand, equipped with force and temperature sensors, mimics human movements through vision recognition and captures detailed tactile data. The data are then transmitted to the user through a hydraulic feedback device, providing realistic touch sensations. The dual capabilities of the hydraulic mechanism to simultaneously simulate both pressure and temperature sensations simplify the device structure and enhance reliability compared to systems that require multiple tactile actuators. The force feedback system delivers pressures ranging from 0.5 to 10.5 N across five levels while the temperature feedback mechanism provides thermal sensations ranging from 3 to 50 °C across three levels. Extensive experimental validations demonstrate the system's effectiveness in applications such as remote robotic surgery and telepresence, enabling users to perceive real‐time contact pressure and temperature during remote manipulation tasks. The proposed system, capable of delivering multimodal tactile feedback through a single actuator, represents a significant step toward more immersive and realistic haptic experiences, highlighting its broad applicability.

Human-CoMo: A Combination of Cognitive Knowledge Engineering and Data Modelling for Human-Cyber-Physical Systems in Intelligent Manufacturing

Many cyber-physical systems face the challenge of appropriately integrating domain-specific human expert knowledge into the cyber part to create a shared sphere of knowledge and intelligent interactions between humans and the semi-autonomous technical system. Cognitive engineering contributes methods and insights into higher-order cognition that help to embed human knowledge in an appropriate way. The original research introduces a novel transdisciplinary framework called Human-CoMo, which demonstrates a systematic modelling process, different human perspectives, and the integration of expert knowledge at multiple hierarchical levels. Fundamental principles inspired by human cognition, such as conceptual chunking and knowledge precision, are characterised. Furthermore, it is shown how knowledge hierarchies can be methodically reflected in appropriate data analysis and modelling levels for small and big data applications including artificial intelligence approaches. Combined knowledge- and data-based modelling approaches offer more flexibility to integrate the strengths of humans and technology in a complementary way. The cognitive foundations and their computational reflections are outlined for the technical example process electroplating from the field of materials and surface engineering. The possibilities and limitations of integrating human knowledge through formalisation and implications for future forms of human-machine interaction are discussed.

Overall Prompting Effectiveness for Optimising Human-Machine Interaction in Cyber-Physical Systems

This paper presents the Overall Prompting Effectiveness (OPE) framework, designed to enhance interactions with systems employing generative AI(genAI) through prompt engineering. Leveraging principles from Total Productive Maintenance (TPM): Overall Equipment Effectiveness (OEE), the OPE framework provides a systematic tool for iterative refinement of prompts, crucial for optimising human-AI interactions in cyber-physical-social-human systems. Evaluating prompting results in three actionable categories: Availability, Performance, and Quality, OPE aims to generate more accurate, relevant, and effective AI responses. Preliminary applications and workshops with prompt engineers underscore the potential of OPE to significantly improve the efficiency and effectiveness of AI engagements. The integration of OPE within Large Language Models (LLMs) or other genAI systems promises to expand automation capabilities, enhancing the intellectualisation and personalisation of CPS. This framework fosters continuous improvement in prompt quality and contributes to the synthetic knowledge management and smart reasoning mechanisms essential for advanced CPS. Our findings suggest that OPE could play a significant role in cognitive engineering, offering a novel methodology for enhancing decision-making and problem-solving in intelligent systems. This aligns with the current research directions in CPS, emphasising the need for tools supporting technology's socialisation and personalisation in complex integrated environments.

Modular Stimuli‐Responsive Valves for Pneumatic Soft Robots

Pneumatic soft robots have several advantages, including facile fabrication, versatile deformation modes, and safe human–machine interaction. However, pneumatic soft robots typically rely on mechatronics to interact with their environment, which can limit their form factors and reliability. Researchers have considered how to achieve autonomous behaviors using the principles of mechanical computing and physical intelligence. Herein, modular responsive valves that can autonomously regulate airflow within pneumatic soft robots in response to various environmental stimuli, including light, water, and mechanical forces, are described. By combining multiple types of valves, autonomous logic gates and more advanced logical operations can be realized. Finally, it is demonstrated that responsive valves can be integrated with pneumatic soft robots, allowing autonomous morphing and navigation. This framework provides a strategy for creating autonomous pneumatic robots that can respond to multiple stimuli in their environment.

Algorithmically Enhanced Wearable Multimodal Emotion Sensor

Sensing diverse human emotions offers significant potential for understanding deeper cognitive processes and diagnosing neurological diseases. However, capturing the full spectrum of emotions remains a significant challenge due to their intricate and subjective nature. Traditional emotion‐sensing techniques often focus on singular emotions, limiting their ability to grasp the complexity of emotional experiences. Herein, a fully printed, organic wearable sensor capable of multimodal emotion sensing by noninvasively monitoring physiological indicators such as heart rate, breathing patterns, and voice signatures is presented. The recorded signals are processed using a long short‐term memory (LSTM) neural network, achieving over 91% classification accuracy in distinguishing different emotions through a data fusion approach, showing >9% enhancement in accuracy as compared to feature fusion. As a proof of concept, Q‐learning is implemented with the data to simulate emotional responses in a robotic model. The study's approach provides a pathway to understanding complex human emotions and enhances the capabilities of effective human–machine interaction.