Machine Interaction Research Articles

Neuro-Symbolic Reasoning for Multimodal Referring Expression Comprehension in HMI Systems

AbstractConventional Human–Machine Interaction (HMI) interfaces have predominantly relied on GUI and voice commands. However, natural human communication also consists of non-verbal communication, including hand gestures like pointing. Thus, recent works in HMI systems have tried to incorporate pointing gestures as an input, making significant progress in recognizing and integrating them with voice commands. However, existing approaches often treat these input modalities independently, limiting their capacity to handle complex multimodal instructions requiring intricate reasoning of language and gestures. On the other hand, multimodal tasks requiring complex reasoning are being challenged in the language and vision domain, but these typically do not include gestures like pointing. To bridge this gap, we explore one of the challenging multimodal tasks, called Referring Expression Comprehension (REC), within multimodal HMI systems incorporating pointing gestures. We present a virtual setup in which a robot shares an environment with a user and is tasked with identifying objects based on the user’s language and gestural instructions. Furthermore, to address this challenge, we propose a hybrid neuro-symbolic model combining deep learning’s versatility with symbolic reasoning’s interpretability. Our contributions include a challenging multimodal REC dataset for HMI systems, an interpretable neuro-symbolic model, and an assessment of its ability to generalize the reasoning to unseen environments, complemented by an in-depth qualitative analysis of the model’s inner workings.

Perceived interactivity in real estate APP increases consumers' psychological well-being: a moderated mediation model

PurposeThis study aims to empirically analyse the influence mechanism of perceived interactivity in real estate APP which affects consumers' psychological well-being. With the growing application of human–machine interaction in real estate APP, it is crucial to utilize human–machine interaction to stimulate perceived interactivity between humans and machines to positively impact consumers' psychological well-being and sustainable development of real estate APP. However, it is unclear whether perceived interactivity improves consumers' psychological well-being.Design/methodology/approachThis study proposes and examines a theoretical model grounded in the perceived interactivity theory, considers the relationship between perceived interactivity and consumers' psychological well-being and explores the mediating effect of perceived value and the moderating role of privacy concerns. It takes real estate APP as the research object, analyses the data of 568 consumer samples collected through questionnaires and then employs structural equation modelling to explore and examine the proposed theoretical model of this study.FindingsThe findings are that perceived interactivity (i.e. human–human interaction and human–information interaction) positively influences perceived value, which in turn affects psychological well-being, and that perceived value partially mediates the effect of perceived interaction on psychological well-being. More important findings are that privacy concerns not only negatively moderate human–information interaction on perceived value, but also negatively moderate the indirect effects of human–information interaction on users' psychological well-being through perceived value.Originality/valueThis study expands the context on perceived interaction and psychological well-being in the field of real estate APP, validating the mediating role and boundary conditions of perceived interactivity created by human–machine interaction on consumers' psychological well-being, and suggesting positive implications for practitioners exploring human–machine interaction technologies to improve the perceived interaction between humans and machines and thus enhance consumer psychological well-being and span sustainable development of real estate APP.

Flexible antibacterial degradable bioelastomer nanocomposites for ultrasensitive human–machine interaction sensing enabled by machine learning

AbstractFlexible wearables have attracted extensive interests for personal human motion sensing, intelligent disease diagnosis, and multifunctional electronic skins. However, the reported flexible sensors, mostly exhibited narrow detection range, low sensitivity, limited degradability to aggravate environmental pollution from vast electronic wastes, and poor antibacterial performance to hardly improve skin discomfort and skin inflammation from bacterial growth under long‐term wearing. Herein, bioinspired from human skin featuring highly sensitive tactile sensation with spinous microstructures for amplifying sensing sensitivity between epidermis and dermis, a wearable antibacterial degradable electronics is prepared from degradable elastomeric substrate with MXene‐coated spinous microstructures templated from lotus leaf assembled with the interdigitated electrode. The degradable elastomer is facilely obtained with tunable modulus to match the modulus of human skin with improved hydrophilicity for rapid degradation. The as‐obtained sensor displays ultra‐low detection limit (0.2 Pa), higher sensitivity (up to 540.2 kPa−1), outstanding cycling stability (>23,000 cycles), a wide detection range, robust degradability, and excellent antibacterial capability. Facilitated by machine learning, the collected sensing signals from the integrated sensors on volunteer's fingers to the related American Sign Language are effectively recognized with an accuracy up to 99%, showing excellent potential in wireless human movement sensing and smart machine learning‐enabled human–machine interaction.

Critical analysis of the use of extended reality XR for training in civil engineering

AbstractConstruction 4.0 promotes digital transformation through automation, robotisation, and the integration of systems and processes into digital environments, with direct links to real systems, using a wide range of technologies. The risk here is centred on having very advanced machines with people not prepared to use them. If the training is centred on teaching people, however, the risk is transferred to having overqualified equipment. In search of this balance, the study, analysis, and evaluation of human–machine interaction are crucial, as are correctly identifying the tools through which this interaction is achieved. Extended reality (XR), emerging technology within Construction 4.0, seems to be a tool that offers an environment conducive to achieving these interactions and meeting the objectives sought. In civil engineering, efforts have been directed towards the study and development of applications of XR experiences rather than the application of this technology in a transcendental way in civil engineering training. This research identifies developments in XR experiences and analyses their use, application methodologies, and training areas that include immersive training, as well as the relationship between XR and construction industry methodologies and technologies, such as building information modelling.

Kinematic-driven human-robot interaction system with deep learning for flexible acupuncture needling manipulations

Acupuncture is an important therapeutic method in traditional Chinese medicine and essentially characterized by flexible needling manipulations that are complicated for repetitions. In this work, we designed a bio-inspired acupuncture robot combined with human–machine interaction system that can implement flexible and precise needle manipulations as manual acupuncture. In the proposed framework, the manual needling process was recorded with online imaging techniques. With a deep learning-based acupuncture estimator, kinetic parameters of lift insertion (LI) and twist rotation (TR) manipulations were extracted in three-dimensional space and the precent of correct key points for operator’s hand pose estimation achieved 94.03%. It is founded that the variation of operator’s finger vector angle for LI manipulation was much weaker than TR manipulation. Furthermore, inspired by kinematic characteristics of manual acupuncture, a six-degree-of-freedom robot was used to simulate the coordinated motion of hand joints in needling manipulations through robot kinematics analysis and trajectory planning with polynomial interpolation method. Numerical and experiential results showed that the acupuncture robot can mimic various needling manipulations as manual acupuncture. The flexibility and stability of the robotic acupuncture framework were evaluated for a wide range of needling parameters. It was demonstrated that the robot can follow the periodic dynamics of LI and TR manipulations with various frequencies and amplitudes, and perform accurate acupuncture operations with smooth needling trajectories as joint jitters during needle manipulations were completely eliminated. The presented results provide a new strategy for modern applications of acupuncture treatment with robotic and flexible needling manipulations to improve the therapeutic efficacy.

Математическая модель тренажера «конь – наездник» и ее программная реализация в системе компьютерной математики Wolfram Mathematica

Riding simulation machines allow training horse-back riders regardless of environmental and weather conditions, horse fatigue, stable remoteness and other factors. Riders can practice complex movements without fear of potential injury related to sometimes unpredictable behavior of the horse. This type of training machines requires special software for selecting the relevant action mode for the person during the rider’s practice. Thus, the selected behavior of the horse can be simulated and the injuries of horse-back rider can be avoided. For this purpose, the model of horse-back rider, taking into account its training level and the horse motion, is required. The model of “person – simulation machine” combination is proposed in the article. The proposed model allows developing operating modes taking into account the sportsman-training machine interaction. The mechanical model is an exosuit with a mobile pole, consisting of four links, the rider’s feet, shins, hips, and the corps are attached to. The pole corresponds to the mass center of the horse, the rider interacts with by standing on the stirrups. The pole motion is implemented by a telescopic link attached to immobile base. The relative rotations of the links are implemented by cylindrical hinges with negligible friction. The proposed model has been implemented as software in the “Wolfram Mathematica 11.3” environment. It has been designed for training machine dynamics simulation of the “horse ­– rider” system. The software includes several modules: 1) the module for specifying the structure of training machine mathematical model, generalized coordinates, and for auto-compilation of the corresponding system of differential equations; 2) the module for specifying the programmed model motion and calculation of the required control torques in the hinges; 3) the module for the Cauchy problem numerical solution; 4) the module for the animated visualizing of the model motion, and for exporting the obtained graphic results and numerical calculations. The developed software allows conducting dynamics analysis of mathematical model for the considered system based on the solution of both direct and inverse dynamics problems. Also it can be suggested for designing training machines with programmed operation mode. It has been shown that application of this software accelerates the development of training machines.

“The machine doesn't judge”: Counternarratives on surveillance among people accessing a safer opioid supply via biometric machines

People who use illegal drugs experience routine surveillance, including in healthcare and harm reduction settings. The MySafe Project - a safer supply pilot project that dispenses prescription opioids via a biometric vending machine - exists in the Canadian province of British Columbia. The machine scans a participant's palmprint and has a built-in camera that records every machine interaction. The aim of this paper is to understand participants' experiences of surveillance, privacy, and personal security when accessing this novel program. An integrative case study and grounded theory methodology was employed. Qualitative one-to-one interviews were conducted with 46 MySafe participants across three different program sites in Vancouver. We used a team-based approach to code interview transcripts and utilized directed and conventional content analyses for deductive and inductive analyses. While participants described negative experiences of surveillance in other public and harm reduction settings, they did not have concerns regarding cameras, collection of personal information, tracking, nor staff issues associated with MySafe. Similarly, while some participants had privacy concerns in other settings, very few privacy and confidentiality concerns were expressed regarding accessing the machine in front of others. Lastly, while some participants reported being targeted by others when accessing the machines, most participants described how cameras, staff, and machine locations helped ensure a sense of safety. Despite negative experiences of surveillance and privacy issues elsewhere, participants largely lacked concern regarding the MySafe program and machines. The machine-human interaction was characterized as different than some human-human interactions as the machine is completing tasks in a manner that is acceptable and comfortable to participants, leading to a social preference toward the machines in comparison to other surveilled means of accessing medications. These findings provide an opportunity to rethink how we conceptualize surveillance, medication access, and harm reduction programs targeting people who use drugs.

Enhancing Emotion Recognition through Federated Learning: A Multimodal Approach with Convolutional Neural Networks

Human–machine interaction covers a range of applications in which machines should understand humans’ commands and predict their behavior. Humans commonly change their mood over time, which affects the way we interact, particularly by changing speech style and facial expressions. As interaction requires quick decisions, low latency is critical for real-time processing. Edge devices, strategically placed near the data source, minimize processing time, enabling real-time decision-making. Edge computing allows us to process data locally, thus reducing the need to send sensitive information further through the network. Despite the wide adoption of audio-only, video-only, and multimodal emotion recognition systems, there is a research gap in terms of analyzing lightweight models and solving privacy challenges to improve model performance. This motivated us to develop a privacy-preserving, lightweight, CNN-based (CNNs are frequently used for processing audio and video modalities) audiovisual emotion recognition model, deployable on constrained edge devices. The model is further paired with a federated learning protocol to preserve the privacy of local clients on edge devices and improve detection accuracy. The results show that the adoption of federated learning improved classification accuracy by ~2%, as well as that the proposed federated learning-based model provides competitive performance compared to other baseline audiovisual emotion recognition models.

River valley-inspired, high-sensitivity, and rapid-response capacitive three-dimensional force tactile sensor based on U-shaped groove structure

High-performance three-dimensional force (3D-force) tactile sensors with the capability of distinguishing normal and tangential forces in sync play a vital role in emerging wearable devices and smart electronics. And there is an urgent need for 3D-force tactile sensors with fast response and high flexibility. Herein, we design a capacitive 3D-force tactile sensors inspired by the U-shaped river valley surface morphology, which has satisfactory performance in terms of rapid response/recovery time (∼36 ms/∼ 36 ms), low hysteresis (4.2%), and high sensitivity (0.487 N−1). A theoretical model of general value for congener sensors is also proposed, obtaining a higher sensitivity through optimizing parameters. To verify the application potential of our device in actual scenarios, the robustness testing and gripping gamepad application were carried out. And it can recognize different motions in humans. Furthermore, principal component analysis is also conducted to demonstrate the distinct classification of different motions. Therefore, our work is eligible for the applications in wearable electronics, human–machine interaction, and soft intelligent robots.

Exploring behavior patterns in human and machine interactions

Exploring behavior patterns in human and machine interactions

Roadmap to Human–Machine Interaction through Triboelectric Nanogenerator and Machine Learning Convergence

Roadmap to Human–Machine Interaction through Triboelectric Nanogenerator and Machine Learning Convergence

An Intelligent Glove System for Real‐Time Multiple Electrical Signal Acquisition and Display

AbstractRealizing multimodal tactile perception for robots is highly desirable for achieving more intelligent human–machine interaction. A real‐time intelligent glove system that can capture and monitor hand tactile information, including pressure, bending, and temperature is proposed. This system processes the collected data and visualizes the tactile information in a 2D image. The intelligent glove system integrates a palm‐shaped sensor unit, a multi‐channel data acquisition board, and a data analysis module. The glove consists of 34 flexible sensors, including 28 pressure sensors, five bending sensors, and one temperature sensor, which convert biological information into electrical signals. By enhancing the performance of the system, the intelligent glove achieves high sensitivity and good fault tolerance, allowing it to analyze the multimodal tactile information of the entire hand with fast responsivity. By analyzing the acquired multi‐channel electrical signals, the system can analyze object shapes and various gripping states. This work paves the way for wearable devices used in multimodal tactile perception and promotes the emergence of new modes of human–machine interaction.

Optimizing Exoskeleton Assistance: Muscle Synergy-Based Actuation for Personalized Hip Exoskeleton Control

Exoskeleton robots hold promising prospects for rehabilitation training in individuals with weakened muscular conditions. However, achieving improved human–machine interaction and delivering customized assistance remains a challenging task. This paper introduces a muscle synergy-based human-in-the-loop (HIL) optimization framework for hip exoskeletons to offer more personalized torque assistance. Initially, we propose a muscle synergy similarity index to quantify the similarity of synergy while walking with and without the assistance of an exoskeleton. By integrating surface electromyography (sEMG) signals to calculate metrics evaluating muscle synergy and iteratively optimizing assistance parameters in real time, a muscle synergy-based HIL optimized torque configuration is presented and tested on a portable hip exoskeleton. Iterative optimization explores the optimal and suboptimal assistance torque profiles for six healthy volunteers, simultaneously testing zero torque and predefined assistance configurations, and verified the corresponding muscle synergy similarity indices through experimental testing. In our validation experiments, the assistance parameters generated through HIL optimization significantly enhance muscle synergy similarity during walking with exoskeletal assistance, with an optimal average of 0.80 ± 0.04 (mean ± std), marking a 6.3% improvement over prior assistive studies and achieving 96.4% similarity compared with free walking. This demonstrates that the proposed muscle synergy-based HIL optimization can ensure robotic exoskeleton-assisted walking as “natural” as possible.

An architectural framework of elderly healthcare monitoring and tracking through wearable sensor technologies

The growing elderly population in smart home environments necessitates increased remote medical support and frequent doctor visits. To address this need, wearable sensor technology plays a crucial role in designing effective healthcare systems for the elderly, facilitating human–machine interaction. However, wearable technology has not been implemented accurately in monitoring various vital healthcare parameters of elders because of inaccurate monitoring. In addition, healthcare providers encounter issues regarding the acceptability of healthcare parameter monitoring and secure data communication within the context of elderly care in smart home environments. Therefore, this research is dedicated to investigating the accuracy of wearable sensors in monitoring healthcare parameters and ensuring secure data transmission. An architectural framework is introduced, outlining the critical components of a comprehensive system, including Sensing, Data storage, and Data communication (SDD) for the monitoring process. These vital components highlight the system's functionality and introduce elements for monitoring and tracking various healthcare parameters through wearable sensors. The collected data is subsequently communicated to healthcare providers to enhance the well-being of elderly individuals. The SDD taxonomy guides the implementation of wearable sensor technology through environmental and body sensors. The proposed system demonstrates the accuracy enhancement of healthcare parameter monitoring and tracking through smart sensors. This study evaluates state-of-the-art articles on monitoring and tracking healthcare parameters through wearable sensors. In conclusion, this study underscores the importance of delineating the SSD taxonomy by classifying the system's major components, contributing to the analysis and resolution of existing challenges. It emphasizes the efficiency of remote monitoring techniques in enhancing healthcare services for the elderly in smart home environments.

Design of Virtual Hands for Natural Interaction in the Metaverse.

The emergence of the Metaverse is raising important questions in the field of human-machine interaction that must be addressed for a successful implementation of the new paradigm. Therefore, the exploration and integration of both technology and human interaction within this new framework are needed. This paper describes an innovative and technically viable proposal for virtual shopping in the fashion field. Virtual hands directly scanned from the real world have been integrated, after a retopology process, in a virtual environment created for the Metaverse, and have been integrated with digital nails. Human interaction with the Metaverse has been carried out through the acquisition of the real posture of the user's hands using an infrared-based sensor and mapping it in its virtualized version, achieving natural identification. The technique has been successfully tested in an immersive shopping experience with the Meta Quest 2 headset as a pilot project, where a transactions mechanism based on the blockchain technology (non-fungible tokens, NFTs) has allowed for the development of a feasible solution for massive audiences. The consumers' reactions were extremely positive, with a total of 250 in-person participants and 120 remote accesses to the Metaverse. Very interesting technical guidelines are raised in this project, the resolution of which may be useful for future implementations.

Towards interactive explanation-based nutrition virtual coaching systems

The awareness about healthy lifestyles is increasing, opening to personalized intelligent health coaching applications. A demand for more than mere suggestions and mechanistic interactions has driven attention to nutrition virtual coaching systems (NVC) as a bridge between human–machine interaction and recommender, informative, persuasive, and argumentation systems. NVC can rely on data-driven opaque mechanisms. Therefore, it is crucial to enable NVC to explain their doing (i.e., engaging the user in discussions (via arguments) about dietary solutions/alternatives). By doing so, transparency, user acceptance, and engagement are expected to be boosted. This study focuses on NVC agents generating personalized food recommendations based on user-specific factors such as allergies, eating habits, lifestyles, and ingredient preferences. In particular, we propose a user-agent negotiation process entailing run-time feedback mechanisms to react to both recommendations and related explanations. Lastly, the study presents the findings obtained by the experiments conducted with multi-background participants to evaluate the acceptability and effectiveness of the proposed system. The results indicate that most participants value the opportunity to provide feedback and receive explanations for recommendations. Additionally, the users are fond of receiving information tailored to their needs. Furthermore, our interactive recommendation system performed better than the corresponding traditional recommendation system in terms of effectiveness regarding the number of agreements and rounds.

Flexible triboelectric sensor array based on 3D printed bead-on-string sacrificial layer for human-machine interactions

Flexible sensor array has attracted widespread attention and brought many opportunities in future applications, such as electronic skin, human–machine interactions, and healthcare monitoring. In this study, we report the flexible triboelectric sensor array (TSA) with cross bar configurated sensing units for human-machine interaction. Bead-on-string sacrificial structure is fabricated by 3D printing directly and optimized based on experimental and numerical studies, and therefore affords excellent performance in constructing the enhanced electrode lines for TSA. With the especially patterning design and optimization, the number of data channels in TSA is reduced, and high output performance can be achieved effectively. Meanwhile, TSA platform exhibits excellent durability, reliability, waterproofness, and presents the ability in real-time sensing and locating of keystroke position and trajectory tracking. For application scenarios, a TSA-based intelligent digital controller is established for importer identification based on temporal perceiver network. We use this controller in robot manipulation to control gestures of both rigid manipulator and soft pneumatic actuator conveniently. The proposed TSA in this work opens a new route in promising applications in security identification, robot manipulation and artificial intelligence.

Recent Progress of Nature Materials Based Triboelectric Nanogenerators for Electronic Skins and Human–Machine Interaction

The current field of human‐machine interaction (HMI) and electronic skin (e‐skin) is confronting challenges associated with energy supply, sensitivity, and biocompatibility. Traditional HMI systems, though rich in features, are constrained in their applications in the medical and wearable technology sectors due to their rigid structure and reliance on third‐party power inputs. Triboelectric nanogenerators (TENGs) based on natural materials are emerging as a focal point of research in this domain, attributed to their self‐powering capabilities, flexibility, and biocompatibility. These natural material‐based TENGs emulate the mechanical and biological characteristics of human skin, adapt to various body shapes and movements, and offer an efficient and reliable solution for the precise detection and analysis of complex, subtle physiological signals, thereby fostering innovations in wearable devices and robotic technology. This article provides a comprehensive review of recent advancements in natural material‐based TENGs, detailing the materials employed, including proteins, chitosan, and cellulose. It encapsulates their applications in the realms of electronic skin and HMI. Finally, the challenges confronted by natural material‐based TENGs are outlined and future trajectories for development in this burgeoning field are projected.

Recent Progress of Bioinspired Triboelectric Nanogenerators for Electronic Skins and Human–Machine Interaction

Advances in biomimetic triboelectric nanogenerators (TENGs) have significant implications for electronic skin (e-skin) and human–machine interaction (HMI). Emphasizing the need to mimic complex functionalities of natural systems, particularly human skin, TENGs leverage triboelectricity and electrostatic induction to bridge the gap in traditional electronic devices’ responsiveness and adaptability. The exploration begins with an overview of TENGs’ operational principles and modes, transitioning into structural and material biomimicry inspired by plant and animal models, proteins, fibers, and hydrogels. Key applications in tactile sensing, motion sensing, and intelligent control within e-skins and HMI systems are highlighted, showcasing TENGs’ potential in revolutionizing wearable technologies and robotic systems. This review also addresses the challenges in performance enhancement, scalability, and system integration of TENGs. It points to future research directions, including optimizing energy conversion efficiency, discovering new materials, and employing micro-nanostructuring techniques for enhanced triboelectric charges and energy conversion. The scalability and cost-effectiveness of TENG production, pivotal for mainstream application, are discussed along with the need for versatile integration with various electronic systems. The review underlines the significance of making bioinspired TENGs more accessible and applicable in everyday technology, focusing on compatibility, user comfort, and durability. Conclusively, it underscores the role of bioinspired TENGs in advancing wearable technology and interactive systems, indicating a bright future for these innovations in practical applications.

A dynamics model for driving behavior based on coupling actuation of bounded rational cognition and diverse emotions

Precise comprehension about the impacts of drivers’ rational and perceptual characteristics on their behavioral decisions is crucial for the accurate prediction of driving behavior. In the previous research on driving behavior, drivers were regarded as homogeneous and absolutely rational individuals. To overcome this limitation, the coupling effects of bounded rational cognition and diverse emotions are considered, and a driving behavior model is proposed based on Dynamics Psychology. There are two parts in this research. In Study 1, the information entropy theory is applied to describe the vehicle cluster situation, and a method is established to quantify the cognitive uncertainty of the vehicle cluster situation for drivers in diverse emotions. In Study 2, with consideration of bounded rational and emotional cognition, the impacts of the vehicle cluster situation and drivers' features, which are their demands for the driving goals including safety, efficiency and comfort, emotional states, and cognitive characteristics, on driving behavior are uniformly expressed as the behavioral driving force, and a prediction model for driving behavior is proposed based on Dynamics Psychology. The results of validation based on the virtual driving data show the prediction accuracy of the proposed model for various driving behaviors of drivers in diverse emotions is over 80%. The results of verification based on NGSIM data suggest that the prediction accuracy of the proposed model for the natural driving behaviors is 82.07%. The research results can contribute to the study on the intrinsic mechanism of driving behavior and provide theoretical support for the development of traffic simulation, personalized active safety systems, human–machine interaction, and brain-inspired autonomous driving.