Machine Interaction Research Articles

Flexible Anisotropic Magnetic Micropillar Arrays for Precision Directional Recognition in Human–Machine Interactions

Flexible Anisotropic Magnetic Micropillar Arrays for Precision Directional Recognition in Human–Machine Interactions

Frontal EEG correlation based human emotion identification and classification.

Humans express their feelings and intentions of their actions or communication through emotions. Recent advancements in technology involve machines in human communication in day-to-day life. Thus, understanding of human emotions by machines will be very helpful in assisting the user in a far better way. Various physiological and non-physiological signals can be used to make the machines to recognize the emotion of a person. The identification of emotional content in the signals is crucial to understand emotion and the machines act with emotional intelligence at appropriate times, thus providing a better human machine interaction with emotion identification system and mental health monitoring for psychiatric patients. This work includes the creation of an emotion EEG dataset, the development of an algorithm for identifying the emotion elicitation segments in the EEG signal, and the classification of emotions from EEG signals. The EEG signals are divided into 3s segments, and the segments with emotional content are selected based on the decrease in correlation between the frontal electrodes. The selected segments are validated with the facial expressions of the subjects in the appropriate time segments of the face video. EEGNet is used to classify the emotion from the EEG signal. The classification accuracy with the selected emotional EEG segments is higher compared to the accuracy using all the EEG segments. In subject-specific classification, an average accuracy of 80.87% is obtained from the network trained with selected EEG segments, and 70.5% is obtained from training with all EEG segments. In subject-independent classification, the accuracy of classification is 67% and 63.8% with and without segment selection, respectively. The proposed method of selection of EEG segments is validated using the DEAP dataset, and classification accuracies and F1-scores of subject dependent and subject-independent methods are presented.

Minimalist Design for Multi-Dimensional Pressure-Sensing and Feedback Glove with Variable Perception Communication

Immersive human–machine interaction relies on comprehensive sensing and feedback systems, which enable transmission of multiple pieces of information. However, the integration of increasing numbers of feedback actuators and sensors causes a severe issue in terms of system complexity. In this work, we propose a pressure-sensing and feedback glove that enables multi-dimensional pressure sensing and feedback with a minimalist design of the functional units. The proposed glove consists of modular strain and pressure sensors based on films of liquid metal microchannels and coin vibrators. Strain sensors located at the finger joints can simultaneously project the bending motion of the individual joint into the virtual space or robotic hand. For subsequent tactile interactions, the design of two symmetrically distributed pressure sensors and vibrators at the fingertips possesses capabilities for multi-directional pressure sensing and feedback by evaluating the relationship of the signal variations between two sensors and tuning the feedback intensities of two vibrators. Consequently, both dynamic and static multi-dimensional pressure communication can be realized, and the vibrational actuation can be monitored by a liquid-metal-based sensor via a triboelectric sensing mechanism. A demonstration of object interaction indicates that the proposed glove can effectively detect dynamic force in varied directions at the fingertip while offering the reconstruction of a similar perception via the haptic feedback function. This device introduces an approach that adopts a minimalist design to achieve a multi-functional system, and it can benefit commercial applications in a more cost-effective way.

Societal Impact of Machine Learning on Human Emotions

Modern networking conversations generate annotated metadata, necessitating a method for synthesizing insights from statistics. Emotion detection is crucial for practical conversations, distinguishing joy, grief, and wrath. Corpora are becoming the standard for human–machine interaction, aiming to make interactions feel natural and real. A paradigm that identifies debates and customer views can provide a human touch to these interactions. Researchers developed a machine learning framework for assessing emotions in English phrases, utilizing Long Short Term Memory perspective and real-time emotion recognition in idiomatic speech. Emotion recognition rule is created using ontologies like Word Net and Concept Net, Naive Bayes, and Random Forest. Real-time analysis of written words and facial expressions significantly outperforms current algorithms and commandment classifiers in identifying emotional states.

Collaborative Intelligence for Safety-Critical Industries: A Literature Review

While AI-driven automation can increase the performance and safety of systems, humans should not be replaced in safety-critical systems but should be integrated to collaborate and mitigate each other’s limitations. The current trend in Industry 5.0 is towards human-centric collaborative paradigms, with an emphasis on collaborative intelligence (CI) or Hybrid Intelligent Systems. In this survey, we search and review recent work that employs AI methods for collaborative intelligence applications, specifically those that focus on safety and safety-critical industries. We aim to contribute to the research landscape and industry by compiling and analyzing a range of scenarios where AI can be used to achieve more efficient human–machine interactions, improved collaboration, coordination, and safety. We define a domain-focused taxonomy to categorize the diverse CI solutions, based on the type of collaborative interaction between intelligent systems and humans, the AI paradigm used and the domain of the AI problem, while highlighting safety issues. We investigate 91 articles on CI research published between 2014 and 2023, providing insights into the trends, gaps, and techniques used, to guide recommendations for future research opportunities in the fast developing collaborative intelligence field.

Improving the Classification of Defect Levels in Main Electrical Equipment Through Advanced Prompting on a Large Language Model

The stable operation of the power system is closely related to the national economy and people’s livelihoods. Therefore, the timely detection, qualitative assessment, and handling of major equipment defects are crucial. The classification of defect levels in main electrical equipment is a fundamental task in this process, which is often manually completed, supplemented by knowledge bases or expert systems. However, this approach is time-consuming, labor-intensive, involves challenging human–machine interaction, and relies on expert experience. Conversational large language models, such as ChatGPT, ERNIE Bot, ChatGLM, have garnered widespread recognition in various domains. However, these models may have errors in the reasoning process, resulting in biased or even erroneous outputs, which is referred to as the “hallucinations”. The hallucinations’ problem of large language models poses challenges in specific fields. To mitigate the hallucinations in large language models, researchers often seek to incorporate domain-specific knowledge into these models through methods like fine-tuning or prompt learning. In order to enhance model performance while minimizing computational costs, this study adopts the prompt learning approach. Specifically, we propose a large language model prompt learning framework based on knowledge graphs, aiming to provide the large language model with reasoning support by leveraging specific information stored within the knowledge graph and receive explainable reasoning result. Experimental results demonstrate that our module achieved superior results on the power defect dataset compared to the non-prompt method.

Deep emotion recognition in textual conversations: a survey

Emotion Recognition in Conversations (ERC) is a key step towards successful human–machine interaction. While the field has seen tremendous advancement in the last few years, new applications and implementation scenarios present novel challenges and opportunities. These range from leveraging the conversational context, speaker, and emotion dynamics modelling, to interpreting common sense expressions, informal language, and sarcasm, addressing challenges of real-time ERC, recognizing emotion causes, different taxonomies across datasets, multilingual ERC, and interpretability. This survey starts by introducing ERC, elaborating on the challenges and opportunities of this task. It proceeds with a description of the emotion taxonomies and a variety of ERC benchmark datasets employing such taxonomies. This is followed by descriptions comparing the most prominent works in ERC with explanations of the neural architectures employed. Then, it provides advisable ERC practices towards better frameworks, elaborating on methods to deal with subjectivity in annotations and modelling and methods to deal with the typically unbalanced ERC datasets. Finally, it presents systematic review tables comparing several works regarding the methods used and their performance. Benchmarking these works highlights resorting to pre-trained Transformer Language Models to extract utterance representations, using Gated and Graph Neural Networks to model the interactions between these utterances, and leveraging Generative Large Language Models to tackle ERC within a generative framework. This survey emphasizes the advantage of leveraging techniques to address unbalanced data, the exploration of mixed emotions, and the benefits of incorporating annotation subjectivity in the learning phase.

Challenges in service robot devices for elderly motion assistance

Abstract Motion assistance for elderly people is a field of application for service robotic systems that can be characterized by requirements and constraints of human–machine interaction and by the specificity of the user’s conditions. The main aspects of characterization and constraints are examined for the application of service systems that can be specifically conceived or adapted for elderly motion assistance by having to consider conditions of motion deficiency and muscular strength weakness as well as psychological aptitudes of users. The analysis is discussed in general terms with reference to elderly people who may not even suffer from specific pathologies. Therefore, the discussion focuses on the need for motion exercise in proper environments, including domestic ones and frame familiar to a user. The challenges of such applications oriented toward elderly users are discussed as requiring research and design of solutions in terms of specific portability, user-oriented operation, low costs, and clinical-physiotherapeutic functionality. Results of the author’s team experiences are presented as an example of problems and attempted solutions to meet the new challenges of service systems for motion assistance applications for elderly people.

Beyond Human and Machine: An Architecture and Methodology Guideline for Centaurian Design

The concept of the centaur, symbolizing the fusion of human and machine intelligence, has intrigued visionaries for decades. Recent advancements in artificial intelligence have made this concept not only realizable but also actionable. This synergistic partnership between natural and artificial intelligence promises superior outcomes by leveraging the strengths of both entities. Tracing its origins back to early pioneers of human–computer interaction in the 1960s, such as J.C.R. Licklider and Douglas Engelbart, the idea initially manifested in centaur chess but faced challenges as technological advances began to overshadow human contributions. However, the resurgence of generative AI in the late 2010s, exemplified by conversational agents and text-to-image chatbots, has rekindled interest in the profound potential of human–AI collaboration. This article formalizes the centaurian model, detailing properties associated with various centaurian designs, evaluating their feasibility, and proposing a design methodology that integrates human creativity with artificial intelligence. Additionally, it compares this model with other integrative theories, such as the Theory of Extended Mind and Intellectology, providing a comprehensive analysis of its place in the landscape of human–machine interaction.

Flexible antibacterial strain sensor with low electrical hysteresis, ultralow detection limit, and wide linear sensing range for human motion monitoring and human–machine interaction

Achieving low electrical hysteresis, an ultralow detection limit, and a wide linear sensing range is critical for the practical application of flexible strain sensors. However, the intrinsic viscoelasticity of the elastomer macromolecules can adversely affect the behavior of the conductive sensing network during stretching and relaxation, rending it challenging to simultaneously achieve these properties. Herein, a novel strategy is proposed to design a hybrid covalent-ionic crosslinking network within the elastomer and to construct a robust conductive carbon nanotube (CNT) layer on the elastomer surface using a swelling-driven penetration method. The hybrid crosslinking network gives the elastomer enhanced mechanical strength and exceptional stretchability, while reducing the dynamic losses of the macromolecules. It also reduces the impact of the elastomer macromolecules on the conductive layer during stretching. Even minimal strains are sufficient to induce slippage of the CNTs, leading to changes in resistance. As the strain increases, further disentanglement and slippage of the CNTs results in crack formation, which enables greater resistance variation and dominates the sensing mechanism. As a result, the as-prepared conductive elastomer-based strain sensors exhibit exceptional sensing performance, including low electrical hysteresis (2.3 %), ultralow detection limit (0.1 %), wide linear working range (690 %), and excellent durability (>10,000 cycles). The strain sensors have been successfully employed in human motion monitoring, human–machine interaction, and angle measurement. In addition, the incorporation of the ionic crosslinking network endows the sensor with excellent antibacterial properties. Consequently, the developed conductive elastomers hold significant promise for applications in smart wearable electronics.

Humidity Sensing Using Polymers: A Critical Review of Current Technologies and Emerging Trends

In the post-pandemic era, human demand for a healthy lifestyle and a smart society has surged, leading to vibrant growth in the field of flexible electronic sensor technology for health monitoring. Flexible polymer humidity sensors are not only capable of the real-time monitoring of human respiration and skin moisture information but also serve as a non-contact human–machine interaction method. In addition, the development of moist-electric generation technology is expected to break free from the traditional reliance of flexible electronic devices on power equipment, which is of significant importance for the miniaturization, reliability, and environmentally friendly development of flexible devices. Currently, flexible polymer humidity sensors are playing a significant role in the field of wearable electronic devices and thus have attracted considerable attention. This review begins by introducing the structural types and working principles of various humidity sensors, including the types of capacitive, impedance/resistive, frequency-based, fiber optic, and voltage-based sensors. It mainly focuses on the latest research advancements in flexible polymer humidity sensors, particularly in the modification of humidity-sensitive materials, sensor fabrication, and hygrosensitivity mechanisms. Studies on material composites including different types of polymers, polymers combined with porous nanostructured materials, polymers combined with metal oxides, and two-dimensional materials are reviewed, along with a comparative summary of the fabrication and performance mechanisms of related devices. This paper concludes with a discussion on the current challenges and opportunities faced by flexible polymer humidity sensors, providing new research perspectives for their future development.

Bacterial film-based degradable triboelectric nanogenerator for both contact and non-contact sensing

As an emerging environmentally friendly energy conversion device, degradable triboelectric nanogenerators (TENGs) play an important role in self-powered sensing, health care and human–machine interaction. However, traditional degradable materials used in TENGs often suffer from limited material sources and complex preparation processes, restricting large-scale production for widespread applications. Here, we propose a simple and efficient way to prepare degradable TENGs with inactivated bacterial film, taking advantage of wide source, self-proliferation and easy culture properties of bacteria. The prepared bacterial film-based TENG (BF-TENG) has stable electrical output, good durability and fatigue resistance, along with superior capabilities in both contact and non-contact sensing. In contact mode, the BF-TENG can generate a peak voltage of up to 83.3 V, which is employed for accurate Morse code transmission. In non-contact mode, the BF-TENG is capable of effectively perceiving the target polymer at a distance of 150 cm. The non-contact LED control circuit and a sensing array based on BF-TENG further demonstrate its practical potential for gesture recognition and spatial position perception. Furthermore, the fully degradable nature of BF-TENG ensures effortless disposal after use without environmental impact, serving as a promising solution for the new generation of eco-friendly and sustainable sensing devices.

Self-adhesive, mechanically strong, and conductive organogel for flexible/wearable electronics and underwater electronic skin

Due to their flexibility, self-healing ability, and biocompatibility, conductive hydrogels have become excellent candidates for flexible electronics. However, challenges remain, such as water evaporation when exposed to air, resulting in a loss of flexibility and conductivity. Additionally, traditional hydrogels quickly swell underwater, which causes irreversible damage to their structure and electrical properties. As a promising alternative, organogels have gained attention for overcoming these drawbacks. Here, we report a transparent, multifunctional organogel using ultraviolet-initiated polymerization, which exhibits excellent adhesion, robustness, electrical and interfacial stability, and super stretchability. By adjusting the concentration of the lithium bis(trifluoromethanesulfonyl)imide (LiTFSI), the conductivity of organogel could be easily regulated from non-conductive to conductive, obtaining non-conducting elastomer (NCE, without LiTFSI) and ion-conducting elastomer (ICE, with LiTFSI) with identical mechanical properties and superior adhesion due to the same composing polymer chains. This unique characteristic overcomes the interface mismatch problem of traditional flexible electronic devices. Moreover, a high-performance triboelectric nanogenerator (TENG) was designed (ICE-TENG) to achieve a maximum power density of 151.3 μW cm−2, which was utilized to power small electronics. Additionally, a strain sensor was constructed using the ICE, realizing underwater sensing and communication with high stability. This multifunctional organogel is promising for multi-functional human–machine interaction.

From hardware to software integration: A comparative study of usability and safety in vehicle interaction modes

The increasing advancement of human–machine interaction (HMI) technology has brought the modes of vehicle HMI into focus, as they are closely related to driver and passenger safety and directly affect the travel experiences. This study compared the usability and safety of three vehicle HMI modes: hardware interaction (HI), hardware and software interaction (HSI), and software interaction (SI). The evaluation comprised two dimensions: usability and safety. Sixty participants’ performance on these tasks was evaluated at two driving speeds (30 km/h and 60 km/h). The results of the nonparametric tests indicated significant differences between the three interaction modes: (1) HI was the highest safety-oriented interaction mode with participants had the highest average vehicle speed and maximum acceleration measured at 60 km/h and the lowest glance frequency at both speeds; (2) HSI was the most usable interaction mode. Participants had the shortest task-completion time measured at 60 km/h and the highest score on the NASA-TLX and SUS scales taken for both speeds; (3) SI was the lowest secure and usable in-vehicle interaction mode. Participants had the longest task-completion time at 60 km/h, the highest error frequency under 30 and 60 km/h and the highest glance frequency, the longest total glance duration and the longest average glance time. In conclusion, HI and HSI were more secure and usable in-vehicle interaction modes than SI. From a theoretical exploration perspective, this paper elaborates on some exploratory thoughts and innovative ideas for practical application to the screen HMI mode selection and design in intelligent vehicle cabins.

Application of Machine Learning in Human-Computer Intelligent Confrontation

Human-computer intelligent confrontation is a widely used technique in a lot of game scenarios, and the artificial intelligence makes a lot of success in different fields. The level of artificial intelligence improves quickly with the development of machine learning. Machine learning refers to enhancing the capabilities of artificial intelligence through the interaction of machines and data. Among all kinds of machine learning, the most famous one is deep reinforced learning. This technology is currently widely used in human-computer confrontations. This article introduces the technical principles of machine learning and reviews the development history of machine learning theory. Besides, this article introduces the application of machine learning in the two important competitive fields of Go and Texas Hold’em, as well as its development and achievements in these two areas. This article can enable other scholars to have a more comprehensive and systematic understanding of machine learning and its applications in Go and Texas Hold’em.

Filiform Papillae‐Inspired Wearable Pressure Sensor with High Sensitivity and Wide Detection Range

AbstractFlexible pressure sensor (FPS) has promising applications in fields like health monitoring and human–machine interactions. The achieving of both high sensitivity and wide detection range in FPS remains highly challenging. Here, inspired by the filiform papillae on cat tongue, a FPS (noted as FPSp) with a sensitivity up to 504.5 kPa−1, a detection range from 30 Pa to 350 kPa, a fast response time of 83 ms, and a high stability over 8000 cycles is developed. The papilla‐like structure continuously shifts the location of stress concentration under increasing pressure, which avoids the accumulation of stress at papillae tips, resulting in a high sensitivity and wide detection range. Moreover, FPSp demonstrates capabilities in monitoring human physiological signals and movement status and can serve as the human‐machine interaction interface. The work not only presents a promising wearable pressure sensor but also establishes a design strategy for high‐performance wearable bioelectronics.

3D Digital Modeling Technology for Thermal Power Plant Boilers Based on Digital Twins

With the rapid advancement of Digital Twins (DTs) technologies, they have emerged as viable solutions for enhancing the control and optimization of thermal power combustion processes. This paper presents a pioneering approach that leverages DTs to revolutionize the combustion process control system within thermal power generating units. Focused on bridging the physical and digital realms, our research establishes DTs of the combustion process control system (CPCS), situated within the Networked Control System Laboratory (NCSLab). We propose a novel five-dimensional (5D) DTs model encompassing the physical, data, service, connection, and virtual models, thereby facilitating a comprehensive fusion of physical and digital information. By analyzing the demands and technical intricacies of the combustion process control system, we design and implement a virtual 3D model, enriching the visualization and analytical capabilities of the DT environment. Furthermore, through controlled experiments on the combustion process and rigorous comparison with offline simulations, we validate the efficacy and feasibility of our approach. Importantly, our research opens avenues for future exploration, particularly in harnessing industrial real-time data to drive the virtual 3D model. Through the integration of Artificial Intelligence (AI) techniques such as Deep Learning, we elevate the capabilities of our DT framework. This enhanced predictive analytics, dynamic optimization, human–machine interaction, and virtual model realism. This research contributes to the advancement of DT technology in the context of thermal power plants and holds promise for driving innovation and efficiency across diverse industrial sectors.

Nonknowledge in Computation. Reflecting on Irrevocable Uncertainty

Abstract My paper approaches the theme of computational creativity by looking at uncertainty as an epistemic and aesthetic tool that must be examined to address the challenges brought to critical practice by planetary computation. It positions uncertainty as central to how the encounter of the human practitioner with non-human machines is conceptualized, and as a resource for building speculative-pragmatic paths of resistance against algorithmic capture. It proposes ways to cultivate uncertainty and use it as a design material to produce new types of knowledge that question machines’ pre-emptying manoeuvres and resist their capture of potential. The argument proposed is that uncertainty affords the production of new imaginaries of the human-machine encounter that can resist the foreclosure of futures (what will be) and are sustained instead by the uncertainty of potential (what might be) (Munster 2013). Dwelling in a space of potential – Deleuze’s virtual, or what I call a space of ‘maybes’, requires of the practitioner a repositioning of their epistemic perspective and reflecting on the following questions: how can material knowledge be made by engaging with modes of un-knowing and not-knowing in machine interaction? How can these modes of un-knowing and not-knowing be fostered as a critical and political onto-epistemological project of reinventing critical practice for the algorithmic age? (Horl et al. 2021; Hansen 2021, 2015, Pasquinelli and Joler 2020). The paper argues that the machinic unknown should be engaged with - not through the conventional paradigm that pitches human vs machine creativity and attempts to rank and score them through similarities, but rather through a (paradoxical) deepening of the unknowability at the core of the machine (Parisi) and machine’s own incommensurability (Fazi 2020). It then proposes the Chinese notion of wu wei (active non action) (Jullien 2011, 2004, 2000, 1995; Allen 2015, 2011) as a stratagem to experiment with to craft speculative-pragmatic interventions, and to augment the ‘power of maybes’ as a space of anti-production, and resisting reduction (Ito 2019).

Scenario Design for the Evaluation of Human–Machine Interaction in Modular Process Plants

AbstractOperators in modular process plants face new demands due to features like increasing automation and digitalization combined with an architecture allowing for reconfigurations in short time spans. However, operators must be able to master process control tasks at all times. For research purposes on topics like supporting operators in their performance in process control of modular process plants, this article aims at developing scenarios of such plants. Based on three conceptual levels describing the relation of chemical processes and their modularized realization and a neutralization process as use case, four scenarios were developed. Having the common task to increase the throughput, the scenarios differ in terms of complexity and constraints that need to be considered for process control.

Variable Admittance Control of High Compatibility Exoskeleton Based on Human–Robotic Interaction Force

The wearable exoskeleton system is a typical strongly coupled human–robotic system. Human–robotic is the environment for each other. The two support each other and compete with each other. Achieving high human–robotic compatibility is the most critical technology for wearable systems. Full structural compatibility can improve the intrinsic safety of the exoskeleton, and precise intention understanding and motion control can improve the comfort of the exoskeleton. This paper first designs a physiologically functional bionic lower limb exoskeleton based on the study of bone and joint functional anatomy and analyzes the drive mapping model of the dual closed-loop four-link knee joint. Secondly, an exoskeleton dual closed-loop controller composed of a position inner loop and a force outer loop is designed. The inner loop of the controller adopts the PID control algorithm, and the outer loop adopts the adaptive admittance control algorithm based on human–robot interaction force (HRI). The controller can adaptively adjust the admittance parameters according to the HRI to respond to dynamic changes in the mechanical and physical parameters of the human–robot system, thereby improving control compliance and the wearing comfort of the exoskeleton system. Finally, we built a joint simulation experiment platform based on SolidWorks/Simulink to conduct virtual prototype simulation experiments and recruited volunteers to wear rehabilitation exoskeletons to conduct related control experiments. Experimental results show that the designed physiologically functional bionic exoskeleton and adaptive admittance controller can significantly improve the accuracy of human–robotic joint motion tracking, effectively reducing human–machine interaction forces and improving the comfort and safety of the wearer. This paper proposes a dual-closed loop four-link knee joint exoskeleton and a variable admittance control method based on HRI, which provides a new method for the design and control of exoskeletons with high compatibility.