Human-machine Interaction Research Articles

Magnetron sputtering preparation of flexible ZnO/AlN thin-films sensors with hybrid piezoelectric effect for broad-range human motions detection

Wearable sensors with excellent comfort, flexibility and fast response are widely applied in human-machine interactions, artificial skin, motion detection and healthcare monitoring. However, preparation processes of the most of current piezoelectric sensors are relatively expensive and complicate, limiting their mass productions in the applications. This work demonstrates the superior piezoelectric performance and stability of ZnO/AlN flexible thin films prepared by magnetron sputtering for broad-range human motions detection, and first-principles calculation are applied to reveal the complicate piezoelectric effect of the hybrid system. The ZnO/AlN hybrid thin films sensor exhibits superior piezoelectric performance and excellent reliability, with presenting high outputs voltage (3.63 V), high degree of response speed (42.33 ms), water stability and robust performance upon 10000 cycles. First-principles calculations reveal that formation of the ZnO/AlN interface facilitates charge transport and improves efficiency of carrier transport, and results in reduced in band gap and enhanced electrical conductivity. By applying the sensors in human health monitoring, the underlying behaviors can be determined by output voltage of the sensor in real-time. This work offers a sensitive, simple structured and robust solution for human motion detection.

Optimized multi-layer self-attention network for feature-level data fusion in emotion recognition

<p>Understanding human emotions across diverse data sources presents challenges in various applications including healthcare, human-machine interaction, security, marketing, and gaming. Prior research has explored fusion techniques to address multimodal data heterogeneity, yet often overlooks the importance of discriminative unimodal information and potential complementarity among fusion strategies. Recognizing emotions from video and audio data poses challenges such as non-verbal cues interpretation, varying expression, ambiguity in context, and the need for nuanced feature extraction to capture subtle emotional nuances accurately. To tackle these issues, it is imperative to employ efficient emotion representation and multimodal fusion techniques, as these tasks have significant importance within the realm of multifaceted recognizing study. This study introduced a novel approach, optimized multi-layer self-attention network for emotion recognition (OMSN-ER), focusing on feature-level data fusion. OMSN-ER precisely assesses emotional states by merging facial and voice data, utilizing a multi-layer progressive dense residual fusion network and a self-attention mountain gazelle convolution neural network. Implemented in Python with the RAVDESS dataset, the methodology achieves exceptional accuracy (0.9908), surpassing benchmarks and demonstrating efficacy in multimodal emotion recognition. This research represents promising advancements in the intricate field of emotion recognition.</p>

Wet-adaptive Strain Sensor Based on Hierarchical Core-sheath Yarns for Underwater Motion Monitoring and Energy Harvesting

Bifunctional flexible electronics with motion monitoring and energy harvesting have broad application prospects in day-to-day activities of human society. Nevertheless, conventional electronics are difficult to adapt to the wear comfortability and anti-sensing interference properties in wet environments. Herein, a wet-adaptive spandex/graphene@cotton/polyurethane yarn (SGCPY) sensor is fabricated with excellent sensing and triboelectric performance, which comprises core layer of spandex fibers, middle layer of graphene@cotton sensing fibers and outer layer of spindle-knotted polyurethane nanofibers. Benefiting from its unique structure, as-prepared SGCPY sensor exhibits high mechanical properties (~80%), superhydrophobic performance (>130°), good strain sensitivity (1.82) and fatigue resistance (12000 cycles) even under water condition. The usage of the SGCPY sensor is demonstrated for the stable monitoring of human body motions and human-machine interaction in both air and water environments. When SGCPY sensor weave as plain fabric, the textile also can convert various mechanical energy into electric power for driving electronic device, showing a maximum voltage of ~3.9V and ~0.7V with solid-solid and liquid-solid contact. This work fosters the in-depth study of textile-based electronics for underwater applications and highlights the promising prospects of multi-functional flexible electronics based on textiles.

Environment-tolerant, inherently conductive and self-adhesive gelatin-based supramolecular eutectogel for flexible sensor

Although hydrogels have attracted increasing attention in the stretchable devices, the low adhesion properties and poor environmental adaptation still seriously restrict their development and application. Herein, we focused on the interaction between polymer networks with disperse media and their resultant influence on gel performance, and constructed self-adhesive and environment-tolerant gelatin/polyacrylamide supramolecular-polymer double-network (Gelatin/PAM SP-DN) eutectogels using multiple supramolecular interactions between natural macromolecule and well-designed deep eutectic solvent (DES). The dual networks of Gelatin/PAM SP-DN eutectogels produced significant supramolecular forces with DES, including hydrogen bonding and electrostatic interaction, contributing to enhance the energy dissipation capacity. Additionally, the Gelatin-PAM SP-DN eutectogels were more prone to generate strong bonding force to various substrates, showcasing both in-situ and ex-situ adhesion performance, and even being used for wet and underwater adhesion. The eutectogels revealed excellent environmental tolerance to maintain excellent mechanical flexibility, conductivity and adhesion at high and low temperatures, ensuring the constructed sensor to sensitively and reliably perceive strain, pressure and human motions over a wide temperature range. Also, the eutectogel demonstrated great potential as a temperature sensor. This work opens up a new horizon in the design of multifunctional and environment-tolerant natural macromolecule-based gel materials for flexible electronics, human-machine interaction and health diagnosis.

Wave-transparent and eco-friendly flame-retardant phosphorus-based phthalonitrile/quartz composites by a synchronized self-curing strategy with cyano and alkynyl groups

The robust human–machine interaction systems of the aerospace industry necessitate the development of thermal resistant wave-transparent composites, with a pivotal focus on organic resin matrix. However, enhancing the heat resistance of such materials while preserving other critical attributes has been a formidable challenge. In this study, a novel designed strategy of a dual-curing functional phosphorus-based phthalonitrile monomer (P-ALK-PN) has been proposed. The optimized curing temperatures for intra- or intermolecular Diels-Alder reactions of alkynyl groups and polyaddition reaction of cyano groups resulted in a homogeneous and highly crosslinked N-enriched phthalonitrile system. After curing at 450 °C, the resin, namely P-ALK-PN-450 °C, demonstrated exceptional thermal stability (T5% = 644 °C), thermo-oxidative stability (T5% = 554 °C), and char yield at 800 °C (90.7 % in N2 and 67.5 % in air). Their quartz fiber reinforced composites (P-ALK-PN/QFs) were subsequently fabricated by the hot-pressing process. Upon post-curing at 420 °C, P-ALK-PN-420 °C/QF displayed elevated glass transition temperature (550 °C), flexural strength (319 MPa), and relatively low dielectric properties with a dielectric constant (Dk) of approximately 4.2 and a dissipation factor (Df) less than 0.02 across a wide temperature ranging from 25 °C to 600 °C. Especially, it exhibited excellent flame retardancy (only a 6.7 % weight loss at ambient temperatures exceeding 1300 °C for 200 s) as well, stemming from release of eco-friendly inert gases (NH3) and formation of a graphitized protective layer during pyrolysis. These promising results highlight the potential applications of P-ALK-PN resins in aerospace and high-performance engineering fields.

Multifunctional PVA/PNIPAM conductive hydrogel sensors enabled human-machine interaction intelligent rehabilitation training

Multifunctional PVA/PNIPAM conductive hydrogel sensors enabled human-machine interaction intelligent rehabilitation training

Flexible Anisotropic Magnetic Micropillar Arrays for Precision Directional Recognition in Human–Machine Interactions

Flexible Anisotropic Magnetic Micropillar Arrays for Precision Directional Recognition in Human–Machine Interactions

Frontal EEG correlation based human emotion identification and classification.

Humans express their feelings and intentions of their actions or communication through emotions. Recent advancements in technology involve machines in human communication in day-to-day life. Thus, understanding of human emotions by machines will be very helpful in assisting the user in a far better way. Various physiological and non-physiological signals can be used to make the machines to recognize the emotion of a person. The identification of emotional content in the signals is crucial to understand emotion and the machines act with emotional intelligence at appropriate times, thus providing a better human machine interaction with emotion identification system and mental health monitoring for psychiatric patients. This work includes the creation of an emotion EEG dataset, the development of an algorithm for identifying the emotion elicitation segments in the EEG signal, and the classification of emotions from EEG signals. The EEG signals are divided into 3s segments, and the segments with emotional content are selected based on the decrease in correlation between the frontal electrodes. The selected segments are validated with the facial expressions of the subjects in the appropriate time segments of the face video. EEGNet is used to classify the emotion from the EEG signal. The classification accuracy with the selected emotional EEG segments is higher compared to the accuracy using all the EEG segments. In subject-specific classification, an average accuracy of 80.87% is obtained from the network trained with selected EEG segments, and 70.5% is obtained from training with all EEG segments. In subject-independent classification, the accuracy of classification is 67% and 63.8% with and without segment selection, respectively. The proposed method of selection of EEG segments is validated using the DEAP dataset, and classification accuracies and F1-scores of subject dependent and subject-independent methods are presented.

Self-Healing, Electrically Conductive, Antibacterial, and Adhesive Eutectogel Containing Polymerizable Deep Eutectic Solvent for Human Motion Sensing and Wound Healing.

Flexible electronic devices such as wearable sensors are essential to advance human-machine interactions. Conductive eutectogels are promising for wearable sensors, despite their challenges in self-healing and adhesion properties. This study introduces a multifunctional eutectogel based on a novel polymerizable deep eutectic solvent (PDES) prepared by the incorporation of diallyldimethylammonium chloride (DADMAC) and glycerol in the presence of polycyclodextrin (PCD)/dopamine-grafted gelatin (Gel-DOP)/oxidized sodium alginate (OSA). The synthesized eutectogel has reversible Schiff-base bonds, hydrogen bonds, and host-guest interactions, which enable rapid self-healing upon network disruption. GPDO-15 eutectogel has significant tissue adhesion, high stretchability (419%), good ionic conductivity (0.79 mS·cm-1), and favorable antibacterial and self-healing properties. These eutectogels achieve 90% antibacterial effect, show excellent biocompatibility, and can be used as sensors to monitor human activities with strong stability and durability. The in vivo studies indicate that the eutectogels can improve the wound healing process which makes them an effective option for biological dressings.

Minimalist Design for Multi-Dimensional Pressure-Sensing and Feedback Glove with Variable Perception Communication

Immersive human–machine interaction relies on comprehensive sensing and feedback systems, which enable transmission of multiple pieces of information. However, the integration of increasing numbers of feedback actuators and sensors causes a severe issue in terms of system complexity. In this work, we propose a pressure-sensing and feedback glove that enables multi-dimensional pressure sensing and feedback with a minimalist design of the functional units. The proposed glove consists of modular strain and pressure sensors based on films of liquid metal microchannels and coin vibrators. Strain sensors located at the finger joints can simultaneously project the bending motion of the individual joint into the virtual space or robotic hand. For subsequent tactile interactions, the design of two symmetrically distributed pressure sensors and vibrators at the fingertips possesses capabilities for multi-directional pressure sensing and feedback by evaluating the relationship of the signal variations between two sensors and tuning the feedback intensities of two vibrators. Consequently, both dynamic and static multi-dimensional pressure communication can be realized, and the vibrational actuation can be monitored by a liquid-metal-based sensor via a triboelectric sensing mechanism. A demonstration of object interaction indicates that the proposed glove can effectively detect dynamic force in varied directions at the fingertip while offering the reconstruction of a similar perception via the haptic feedback function. This device introduces an approach that adopts a minimalist design to achieve a multi-functional system, and it can benefit commercial applications in a more cost-effective way.

Societal Impact of Machine Learning on Human Emotions

Modern networking conversations generate annotated metadata, necessitating a method for synthesizing insights from statistics. Emotion detection is crucial for practical conversations, distinguishing joy, grief, and wrath. Corpora are becoming the standard for human–machine interaction, aiming to make interactions feel natural and real. A paradigm that identifies debates and customer views can provide a human touch to these interactions. Researchers developed a machine learning framework for assessing emotions in English phrases, utilizing Long Short Term Memory perspective and real-time emotion recognition in idiomatic speech. Emotion recognition rule is created using ontologies like Word Net and Concept Net, Naive Bayes, and Random Forest. Real-time analysis of written words and facial expressions significantly outperforms current algorithms and commandment classifiers in identifying emotional states.

Human in Loop Machine Learning: A Paradigm Shift

This paper explores the transformative concept of Human-in-Loop (HIL) Machine Learning, which integrates human expertise into the machine learning process to enhance data quality, model accuracy, and ethical decision-making. Human interaction is added to traditional machine learning steps such data collection, preprocessing, model training, evaluation, and deployment through continuous feedback loops, data annotation, and model change. This integration makes use of human intuition and experience to enhance algorithm performance and model interpretability, hence mitigating the drawbacks of entirely automated approaches. HIL Machine Learning allows AI systems to adjust to changing obstacles and makes more accurate forecasts by promoting human-machine interaction. This study emphasizes HIL Machine Learning as a reliable method for successfully handling dynamic, complicated problems across a range of areas.

Collaborative Intelligence for Safety-Critical Industries: A Literature Review

While AI-driven automation can increase the performance and safety of systems, humans should not be replaced in safety-critical systems but should be integrated to collaborate and mitigate each other’s limitations. The current trend in Industry 5.0 is towards human-centric collaborative paradigms, with an emphasis on collaborative intelligence (CI) or Hybrid Intelligent Systems. In this survey, we search and review recent work that employs AI methods for collaborative intelligence applications, specifically those that focus on safety and safety-critical industries. We aim to contribute to the research landscape and industry by compiling and analyzing a range of scenarios where AI can be used to achieve more efficient human–machine interactions, improved collaboration, coordination, and safety. We define a domain-focused taxonomy to categorize the diverse CI solutions, based on the type of collaborative interaction between intelligent systems and humans, the AI paradigm used and the domain of the AI problem, while highlighting safety issues. We investigate 91 articles on CI research published between 2014 and 2023, providing insights into the trends, gaps, and techniques used, to guide recommendations for future research opportunities in the fast developing collaborative intelligence field.

Bioinspired Tilted Magnetized Flakes as a Self-Powered and Antislip Smart Outsole for Healthcare Monitoring and Human-Machine Interaction.

Footwear smart devices capable of reliably capturing body actions and conveniently transmitting human-made information are of great interest to advance healthcare monitoring, human-machine interactions (HMIs), etc. while remaining challenging. Herein, we present a self-powered, antislip, and multifunctional smart outsole based on the gecko toe-inspired tilted magnetized flakes (TMFs) and underlying flexible coils. With the pressure-induced flake deflection and the built-in magnetic moment alignment, the TMF can produce a variable magnetic field to induce the voltage signals in coils for precise pressure perception and linear velocity sensing. The TMF-based smart outsole can thus serve as a real-time footwear recorder to monitor various body actions for exercise analysis and to track the abnormal landing speed for alerting potential injuries. The gecko toe-like flakes also enable the excellent antislip capability of the outsole with a much higher friction coefficient than the standard one of the low slip risk. By programming the magnetic moment alignments of the TMFs, a single-circuit outsole can further output multiple signals as encoded instructions for controlling the racing game. Along with excellent abrasion resistance and environmental immunity, the proposed outsole exhibits great potential as a convenient platform for reliable healthcare monitoring and efficient HMI.

RISH Assistance: Voice Activated Digital Assistant for Developers

The increasing use of computing devices brings convenience but also challenges. Traditional interfaces can lead to repetitive strain, slow interactions, and accessibility problemsfor software developers, who often find it hard to turn complex ideas into code, especially with unfamiliar APIs. In response, there is a growing interest in automation and smart machines that act like humans. However, a major obstacle remains—the gap in human-machine interaction. Introducing “RISH Assistance,” a voice-activated digital assis- tant designed for developers. RISH enables users to navigate tasks more efficiently, reducing reliance on screens and tra- ditional input methods. Utilizing advanced voice recognition and Natural Language Processing (NLP), RISH transforms the developer experience by providing easy access to resources, seamless code suggestions, and natural troubleshooting—all through voice commands. Additionally, it can manage phone functions, allowing calls and messages directly from the desktop interface, enhancing productivity. Security is ensured with face authentication, and current session chat history tracking allowsusers to revisit previous interactions. RISH boosts productivity through hands-free interaction, en- abling developers to focus on creative problem-solving. It im- proves accessibility by offering an alternative input method and streamlines workflows, reducing cognitive load while fostering a more intuitive development environment. By facilitating natural and efficient engagement with machines, RISH Assistance aims to enhance human-machine collaboration and create a more inclusive software development landscape.

Improving the Classification of Defect Levels in Main Electrical Equipment Through Advanced Prompting on a Large Language Model

The stable operation of the power system is closely related to the national economy and people’s livelihoods. Therefore, the timely detection, qualitative assessment, and handling of major equipment defects are crucial. The classification of defect levels in main electrical equipment is a fundamental task in this process, which is often manually completed, supplemented by knowledge bases or expert systems. However, this approach is time-consuming, labor-intensive, involves challenging human–machine interaction, and relies on expert experience. Conversational large language models, such as ChatGPT, ERNIE Bot, ChatGLM, have garnered widespread recognition in various domains. However, these models may have errors in the reasoning process, resulting in biased or even erroneous outputs, which is referred to as the “hallucinations”. The hallucinations’ problem of large language models poses challenges in specific fields. To mitigate the hallucinations in large language models, researchers often seek to incorporate domain-specific knowledge into these models through methods like fine-tuning or prompt learning. In order to enhance model performance while minimizing computational costs, this study adopts the prompt learning approach. Specifically, we propose a large language model prompt learning framework based on knowledge graphs, aiming to provide the large language model with reasoning support by leveraging specific information stored within the knowledge graph and receive explainable reasoning result. Experimental results demonstrate that our module achieved superior results on the power defect dataset compared to the non-prompt method.

Deep learning-assisted object recognition with hybrid triboelectric-capacitive tactile sensor

Tactile sensors play a critical role in robotic intelligence and human-machine interaction. In this manuscript, we propose a hybrid tactile sensor by integrating a triboelectric sensing unit and a capacitive sensing unit based on porous PDMS. The triboelectric sensing unit is sensitive to the surface material and texture of the grasped objects, while the capacitive sensing unit responds to the object’s hardness. By combining signals from the two sensing units, tactile object recognition can be achieved among not only different objects but also the same object in different states. In addition, both the triboelectric layer and the capacitor dielectric layer were fabricated through the same manufacturing process. Furthermore, deep learning was employed to assist the tactile sensor in accurate object recognition. As a demonstration, the identification of 12 samples was implemented using this hybrid tactile sensor, and an recognition accuracy of 98.46% was achieved. Overall, the proposed hybrid tactile sensor has shown great potential in robotic perception and tactile intelligence.

Deep emotion recognition in textual conversations: a survey

Emotion Recognition in Conversations (ERC) is a key step towards successful human–machine interaction. While the field has seen tremendous advancement in the last few years, new applications and implementation scenarios present novel challenges and opportunities. These range from leveraging the conversational context, speaker, and emotion dynamics modelling, to interpreting common sense expressions, informal language, and sarcasm, addressing challenges of real-time ERC, recognizing emotion causes, different taxonomies across datasets, multilingual ERC, and interpretability. This survey starts by introducing ERC, elaborating on the challenges and opportunities of this task. It proceeds with a description of the emotion taxonomies and a variety of ERC benchmark datasets employing such taxonomies. This is followed by descriptions comparing the most prominent works in ERC with explanations of the neural architectures employed. Then, it provides advisable ERC practices towards better frameworks, elaborating on methods to deal with subjectivity in annotations and modelling and methods to deal with the typically unbalanced ERC datasets. Finally, it presents systematic review tables comparing several works regarding the methods used and their performance. Benchmarking these works highlights resorting to pre-trained Transformer Language Models to extract utterance representations, using Gated and Graph Neural Networks to model the interactions between these utterances, and leveraging Generative Large Language Models to tackle ERC within a generative framework. This survey emphasizes the advantage of leveraging techniques to address unbalanced data, the exploration of mixed emotions, and the benefits of incorporating annotation subjectivity in the learning phase.

The Issues and Optimization Strategies in the Modernization of Vocational Education in the Era of Chat GPT

The emergence of Chat GPT technology represents a significant breakthrough in natural language processing, revolutionizing human-machine interaction and presenting profound challenges to vocational education modernization in China. In the Chat GPT era, vocational education undergoes transformative changes, including personalized learning reaching new heights, big data-driven intelligent decision-making entering a new phase, and the integration of education resources on unprecedented scales. However, this modernization effort also encounters several critical issues, such as overreliance on technology and the associated risks of unemployment, threats to data privacy and security, erosion of ethical values, and the widening digital divide in education. To ensure the high-quality development of vocational education modernization with the support of artificial intelligence tools like Chat GPT, it is crucial to enhance supervision and regulation of Chat GPT technology applications, promoting its safe and responsible use. Moreover, prioritizing technology ethics and moral education becomes imperative to nurture students’ ethical awareness and moral compass in the utilization and development of technology. Lastly, optimizing the distribution of digital educational resources is essential to promote educational equity and bridge the digital divide among students.

MCNN-CMCA: A multiscale convolutional neural networks with cross-modal channel attention for physiological signal-based mental state recognition

Human mental state recognition (MSR) has significant implications for human-machine interactions. Although mental state recognition models based on single-modality signals, such as electroencephalogram (EEG) or peripheral physiological signals (PPS), have achieved encouraging progress, methods leveraging multimodal physiological signals still need to be explored. In this study, we present MCNN-CMCA, a generic model that employs multiscale convolutional neural networks (CNNs) with cross-modal channel attention to realize physiological signals-based MSR. Specifically, we first design an innovative cross-modal channel attention mechanism that adaptively adjusting the weights of each signal channel, effectively learning both intra-modality and inter-modality correlation and expanding the channel information to the depth dimension. Additionally, the study utilizes multiscale temporal CNNs for obtaining short-term and long-term time-frequency features across different modalities. Finally, the multimodal fusion module integrates the representations of all physiological signals and the classification layer implements sparse connections by setting the mask weights to 0. We evaluate the proposed method on the SEED-VIG, DEAP, and self-made datasets, achieving superior results compared to existing state-of-the-art methods. Furthermore, we conduct ablation studies to demonstrate the effectiveness of each component in the MCNN-CMCA and show the use of multimodal physiological signals outperforms single-modality signals.