Robot Interaction Research Articles

Gaze-action coupling, gaze-gesture coupling, and exogenous attraction of gaze in dyadic interactions

In human interactions, gaze may be used to acquire information for goal-directed actions, to acquire information related to the interacting partner’s actions, and in the context of multimodal communication. At present, there are no models of gaze behavior in the context of vision that adequately incorporate these three components. In this study, we aimed to uncover and quantify patterns of within-person gaze-action coupling, gaze-gesture and gaze-speech coupling, and coupling between one person’s gaze and another person’s manual actions, gestures, or speech (or exogenous attraction of gaze) during dyadic collaboration. We showed that in the context of a collaborative Lego Duplo-model copying task, within-person gaze-action coupling is strongest, followed by within-person gaze-gesture coupling, and coupling between gaze and another person’s actions. When trying to infer gaze location from one’s own manual actions, gestures, or speech or that of the other person, only one’s own manual actions were found to lead to better inference compared to a baseline model. The improvement in inferring gaze location was limited, contrary to what might be expected based on previous research. We suggest that inferring gaze location may be most effective for constrained tasks in which different manual actions follow in a quick sequence, while gaze-gesture and gaze-speech coupling may be stronger in unconstrained conversational settings or when the collaboration requires more negotiation. Our findings may serve as an empirical foundation for future theory and model development, and may further be relevant in the context of action/intention prediction for (social) robotics and effective human–robot interaction.

Safety in Wearable Robotic Exoskeletons: Design, Control, and Testing Guidelines

Abstract Exoskeletons, wearable robotic devices designed to enhance human strength and endurance, find applications in various fields such as healthcare and industry; however, stringent safety measures should be adopted in such settings. This paper presents a comprehensive exploration of challenges associated with exoskeleton technology, ranging from mechanical issues to regulatory and ethical considerations. The enumerated challenges include joint hyper-extension or flexion, rapid or sudden motion, misalignment, fit, and comfort issues, mechanical failure, weight and mobility limitations, environmental challenges, power supply issues, high energy consumption and regeneration, fall risk or stability concerns, sensor failures, control algorithm malfunctions, machine-learning model challenges, communication disconnection, actuator malfunctions, unexpected human–robot interactions, and regulatory and ethical considerations. The paper outlines possible risks and suggests practical solutions based on design, control, and testing methods for each challenge. The objective is to offer a guideline for developers and users, emphasizing safety, reliability, and optimal performance in the ever-evolving landscape of exoskeleton technology. The guideline covers preoperation checks, user training, emergency response, real-time monitoring, and user interaction to ensure responsible innovation and user-centricity in exoskeleton development and deployment.

Autonomous Vehicle Control Through Socially Compliant Human–Robot Interactions With an Application to Eco-Driving

Autonomous Vehicle Control Through Socially Compliant Human–Robot Interactions With an Application to Eco-Driving

Transfer-Learning-Based Gesture and Pose Recognition System for Human–Robot Interaction: An Internet of Things Application

Transfer-Learning-Based Gesture and Pose Recognition System for Human–Robot Interaction: An Internet of Things Application

Exploring the Implementation Methods of AI-Driven Robots in Media Interaction Experience Design

The purpose of this study is to deeply explore the innovative implementation methods of artificial intelligence (AI)-driven robots in media interactive experience design. Firstly, this paper expounds how AI-driven robot technology can provide users with a more natural and intelligent interactive experience through natural language processing (NLP), speech recognition, computer vision and other technologies, thus enhancing users' sense of participation and immersion. In order to realize the efficient application of AI-driven robot in media interaction, this study proposes a series of implementation methods, including improving the accuracy and personalization of robot interaction by using NLP and deep learning algorithm; Identify user expressions and gestures through computer vision technology to enhance the naturalness of interaction; Multi-modal interactive technology is used to integrate multi-sensory channel information to provide comprehensive interactive experience; Use deep learning technology to train the robot's ability to perceive and understand users' needs, and introduce emotion analysis technology to realize more humanized service; By constructing user portraits, we design personalized interaction schemes that adapt to different scenarios and users. In the experimental part, three different scenes, library, shopping mall and museum, are selected to verify the effectiveness of the design scheme. The results show that the AI-driven robot significantly improves the user's satisfaction and emotional experience under the personalized interaction scheme.

Polyacrylamide/starch hydrogels doped with layered double hydroxides towards strain sensing applications

Electrically conductive hydrogels have attracted enormous attention due to the rapid development of flexible electronics. In this paper, the application of layered double hydroxides (LDHs) in the field of conductive hydrogel for strain sensing is firstly explored. LDHs are introduced to the polyacrylamide (PAM)/starch (St) semi-interpenetrating network (SIPN) to fabricate conductive PAM/St/LDHs (PSL) hydrogels for strain sensing applications. The results show that LDHs incorporated into PAM/St SIPN as inorganic nano fillers not only improve the mechanical strength, but also innovatively endow the hydrogel with electrical conductivity properties. The PSL hydrogels exhibit excellent mechanical properties (with an elongation strain of 1750 % and a fracture strength of 0.22 MPa) and decent sensing properties (with a gauge factor of 2.73). As a proof of concept, an 8*8 sensing array based on PSL hydrogels is designed to realize the visualization of pressure sensing, demonstrating the broad prospect of PSL hydrogels in applications of human-computer interaction, flexible wearable and soft robotics.

Human-Centered Coordination for Robot-Assisted Equipment Transport

Abstract This work explores how to use an unmanned ground vehicle (UGV) to offload the physical burdens of equipment from humans. This work formulates dynamic alignment following and compares it to position-based following techniques. We describe the control strategies of both following methods and implement them in a dynamic simulation and a physical prototype. We test the performance of the two following methods and show that dynamic alignment following can reduce robot positional error and interaction force between the human and the robot. We then analyze the energetics and the performance of the human–UGV team for candidate transportation tasks. The presence of the robot can make some tasks take longer to perform. Nonetheless, the results show that for the candidate tasks, the robot can reduce human average metabolic power and average overall task energy.

Multimodal flexible electronic skin with adjustable module size for robot collision detection

Purpose This paper aims to present a modular multimodal flexible electronic skin that can be used for robot collision detection in human–robot interactions. This type of electronic skin will meet the requirements of performance indicators such as sensing mode, sensing domain coverage and dynamic data update rate in human–robot interactions. Design/methodology/approach The electronic skin uses a modular architecture, and the sensing module is designed to be adjustable in size so that it can be easily deployed on complex robot surfaces, increasing area coverage, reducing power consumption, and improving data update rates. Findings The authors evaluated electronic skin through experiments using a UR5 robot. Electronic skin has high static scene perception differentiation and dynamic scene perception abilities. Moreover, the robot realizes a high-speed collision response based on the electronic skin proposed in this study. Originality/value The proposed electronic skin provides crucial technical support for advancing robotic technologies, and holds promising prospects for industrial applications.

Digital-twin co-simulation framework to support informed decision in healthcare planning and management

This manuscript aims to show the proof-of-concept for the integration of digital twin (DT) and discrete event co-simulation through its application to forward simulation of physical activities of care providers and robots in an intensive care unit (ICU). Co-simulation focuses on modular integration of heterogeneous simulation components to capture the complexity inherent in systems. Advancing this conception, the proposed co-simulation attempted to narrow gaps with real-world systems by using DT while simultaneously leveraging the forward simulation capacity of discrete event simulation (DES). In particular, we created a co-simulation for human–robot interactions in a simulated ICU facility, which modeled the hospital layout including the facility, robot, avatars for healthcare professionals, equipment, patient bed, and vital signs, which comprise physical twin (PT). In addition, specific patient-care scenarios were defined to drive PT for which the states of interest were represented in the DT, and then used for a dynamic input to the discrete event co-simulation. At the onset of the co-simulation, the parameters needed to run DES were manually configured to align the parameters with those of the DT. For internal validation, this DES was replicated under different work-system policies, and the co-simulation outcomes were interpreted in clinical context. Our co-simulation approach has potential for application to a broader class of problems in healthcare planning and management by supporting informed decision-making. Methodologically, timely information exchanges of PT with DT, and ensuing model updates in the Discrete Event System Specification (DEVS) is a promising approach for co-simulation, to maintain robustness against perturbations to real-world systems.

I am frail and my robot is stuck on the corridor: cohabitation of people with frailty and robots

Abstract UK and European policies promote research and innovation for people living with frailty (PLWF), as realistic strategies for independent living. Yet homes are not designed for PLWF or robots, even though robots could support PLWF with activities, i.e., exercise and companionship. Adoption of robots in homes is limited to devices such as automated vacuum cleaners and voice assistants. Robotics aimed at assisting PLWF have been thus far been tested in laboratories and the study of real homes for the co-habitation of robots and older people is often neglected. We draw on expertise from human-robot interaction, built environment, public health and clinical practice to facilitate optimisation of human and robot cohabitation for PLWF. For this, we have reviewed different robots-for-frailty, in terms of, ability to fit and move within homes. We utilised a range of methods and sources of information including meetings with allied health professionals, visited the Robot House and explored the available robots from a fit for purpose perspective, audited five flats for PLWF in Gloucester, England in accordance with the needs of those with frailty, simulated robot’s function within the home environment, using persona’s for different use case scenarios, trial testing of selected robots in the home environment for comparison with simulation, using knowledge gained to extrapolate this data to estimate suitability for frail individuals in their homes. Most robots currently available would not adequately fit in an independent living accommodation or the smaller ones could be a tripping hazard. The communal areas of independent living would be more realistic for such human robot interactions, rather than the actual households. The research informs the process facilitating the adoption of robots to benefit PLWF both feasible and desired. The research produced a framework that can lead to future research exploring robot and PLWF co-habitation in real home settings. Key messages • The cohabitation of humans and robots currently faces barriers relevant to independent living built-environment spatial restrictions. • Technologies that support independent living and frailty must develop in spatially-unobtrusive ways.

Modeling, Simulation, and Kinematic Validation of Transfemoral Prosthetic Mechanism With Ankle Varus–Valgus Characteristic

Abstract Inspired by the kinesiology of human bionic joints, a transfemoral prosthetic mechanism based on a functional structure of parallel mechanism is developed for the transfemoral amputees. The walking interactive simulation is implemented based on human-prosthesis modeling to verify the kinematics of the designed prosthetic mechanism, as well as to explore compatibility between the amputees and prosthesis. Then, simulation-based prosthetic optimization is performed to pursue an optimized human-prosthesis model with economic metabolic consumption while eliminating compatibility errors including the joints' misalignment error between the affected limb and healthy limb, and the assembly error between human and prosthesis, so that the potential physical health problems can be avoided efficiently. This method is valuable for the optimal design of interactive rehabilitation robots. Finally, a developed proportional-integral-derivative-based (PID-based) finite-state machine (FSM) strategy is used, and the kinematic validation is carried out. The results show that the designed prosthesis possesses ankle varus–valgus characteristic, and it has a high human-like motion accuracy due to the FSM control can track prosthetic motion in each gait event. What's more, the prosthetic optimization can be an efficient method to enhance the biomechanical performance of human-prosthetic model so that the amputees have a more natural and symmetry gait.

Human-robot interaction: predicting research agenda by long short-term memory.

The article addresses the identification and prediction of research topics in human-robot interaction (HRI), fundamental in Industry 4.0 (I4.0) and future Industry 5.0 (I5.0). In the absence of research agendas in the scientific literature, the study proposes a multilayered model to create a precise agenda to guide the scientific community in new developments in collaborative robotics and HRI technologies. The methodology is divided into four stages, which make up the three layers of the model. In the first two stages, scientific articles on HRI for the period 2020-2021 were collected and analyzed using data mining techniques together with VantagePoint and Gephi software to identify keywords and their relationships. These initial stages form layer 1 of the model, where the main scientific themes are recognized. In the third stage, article titles and abstracts are cleaned and processed using natural language processing (NLP) techniques, generating word embeddings models that highlight relevant HRI-related terms, forming layer 2. The fourth and final stage uses Recurrent Neural Networks (RNN) with long short-term memory (LSTM) architecture to predict future topics, consolidating the previously identified terms and forming layer 3 of the model. The results show that in layer 1 HRI has intensive application in various sectors through advanced computational algorithms, with trust as a key feature. In layer 2, terms such as vision, sensors, communication, collaboration and anthropomorphic aspects are fundamental, while layer 3 anticipates future topics such as design, performance, method and controllers, essential to improve robot interaction. The study concludes that the methodology is effective in defining a robust and relevant research agenda. By identifying future trends and needs, this work fills a gap in the scientific literature, providing a valuable tool for the research community in the field of HRI.

A Human-Centered View of Continual Learning: Understanding Interactions, Teaching Patterns, and Perceptions of Human Users Toward a Continual Learning Robot in Repeated Interactions

Continual learning (CL) has emerged as an important avenue of research in recent years, at the intersection of Machine Learning (ML) and Human–Robot Interaction (HRI), to allow robots to continually learn in their environments over long-term interactions with humans. Most research in CL, however, has been robot-centered to develop CL algorithms that can quickly learn new information on systematically collected static datasets. In this article, we take a human-centered approach to CL, to understand how humans interact with, teach, and perceive CL robots over the long term, and if there are variations in their teaching styles. We developed a socially guided CL system that integrates CL models for object recognition with a mobile manipulator robot and allows humans to directly teach and test the robot in real time over multiple sessions. We conducted an in-person study with 60 participants who interacted with the CL robot in 300 sessions with 5 sessions per participant. In this between-participant study, we used three different CL models deployed on a mobile manipulator robot. An extensive qualitative and quantitative analysis of the data collected in the study shows that there is significant variation among the teaching styles of individual users indicating the need for personalized adaptation to their distinct teaching styles. Our analysis shows that the constrained experimental setups that have been widely used to test most CL models are not adequate, as real users interact with and teach CL robots in a variety of ways. Finally, our analysis shows that although users have concerns about CL robots being deployed in our daily lives, they mention that with further improvements CL robots could assist older adults and people with disabilities in their homes.

Body Movement Mirroring and Synchrony in Human–Robot Interaction

This review article provides an overview of papers that have studied body movement mirroring and synchrony within the field of human-robot interaction. The papers included in this review cover system studies, which focus on evaluating the technical aspects of mirroring and synchrony robotic systems, and user studies, which focus on measuring particular interaction outcomes or attitudes towards robots expressing mirroring and synchrony behaviors. We review the papers in terms of the employed robotic platforms and the focus on parts of the body, the techniques used to sense and react to human motion, the evaluation methods, the intended applications of the human-robot interaction systems and the scenarios utilized in user studies. Finally, challenges and possible future directions are considered and discussed.

An Assessment of an Inpatient Robotic Nurse Assistant: A Mixed-Method Study

The worldwide nursing shortage has led to the exploration of using robotics to support care delivery and reduce nurses’ workload. In this observational, mixed-method study, we examined the implementation of a robotic nurse assistant (RNA) in a hospital ward to support vital signs measurements, medication, and item delivery. Human–robot interaction was assessed in four domains: usability, social acceptance, user experience, and its societal impact. Patients in a general medicine ward were recruited to participate in a one-time trial with the RNA and a post-trial 75-question survey. Patients’ interactions with the RNA were video recorded for analysis including patients’ behaviours, facial emotions, and visual attention. Focus group discussions with nurses elicited their perceptions of working with the RNA, areas for improvement, and scalability. Sixty-seven patients aged 21–79 participated in the trial. Eight in 10 patients reported positive interactions with the RNA. When the RNA did not perform to expectations, only 25% of patients attributed fault to the RNA. Video analysis showed patients at ease interacting with the RNA despite some technical problems. Nurses saw potential for the RNA taking over routine tasks. However, they were sceptical of real time savings and were concerned with the RNA’s ability to work well with older patients. Patients and nurses suggested greater interactivity between RNA and patients. Future studies should examine potential timesaving and whether time saved translated to nurses performing higher value clinical tasks. The utility of improved RNA’s social capability in a hospital setting should be explored as well.

THÖR-MAGNI: A large-scale indoor motion capture recording of human movement and robot interaction

We present a new large dataset of indoor human and robot navigation and interaction, called THÖR-MAGNI, that is designed to facilitate research on social human navigation: for example, modeling and predicting human motion, analyzing goal-oriented interactions between humans and robots, and investigating visual attention in a social interaction context. THÖR-MAGNI was created to fill a gap in available datasets for human motion analysis and HRI. This gap is characterized by a lack of comprehensive inclusion of exogenous factors and essential target agent cues, which hinders the development of robust models capable of capturing the relationship between contextual cues and human behavior in different scenarios. Unlike existing datasets, THÖR-MAGNI includes a broader set of contextual features and offers multiple scenario variations to facilitate factor isolation. The dataset includes many social human–human and human–robot interaction scenarios, rich context annotations, and multi-modal data, such as walking trajectories, gaze-tracking data, and lidar and camera streams recorded from a mobile robot. We also provide a set of tools for visualization and processing of the recorded data. THÖR-MAGNI is, to the best of our knowledge, unique in the amount and diversity of sensor data collected in a contextualized and socially dynamic environment, capturing natural human–robot interactions.

A lightweight network-based sign language robot with facial mirroring and speech system

Human–robot interaction is an essential capability for humanoid robots to enter the physical world and become companions in people’s lives, learning, and work. While the majority of current research focuses on the voice-based interactions of robots, yet over 60% of communication occurs through nonverbal behaviors, such as facial expressions and hand gestures. Endowing robots with the ability to communicate through nonverbal behavior not only enhances the interactive experience with robots but also provides a potential communication tool for individuals with hearing or speech impairments. Here, we develop a humanoid robot capable of adjusting facial movements by driving servos, and design a novel framework for the robot to integrate sign language recognition and facial landmark detection algorithms. This framework facilitates the robot recognize sign language and translate it into spoken language, while also imitating the facial expressions of the signers. To achieve this, we also propose a lightweight deep learning network called RealTimeSignNet for real-time sign language recognition. Leveraging lightweight 3D convolution modules and time-dependent constraints, this model adapts to various time scales, ensuring efficient processing of sign language recognition tasks. Experimental results demonstrate the outstanding performance of the RealTimeSignNet model on mainstream sign language datasets, achieving an accuracy of 88.1% on the large continuous sign language dataset (continuous SLR), 98.2% on the isolated sign language dataset (SLR 500), and 91.50% on the English sign language dataset (WLAS). The overall assessment demonstrates that our humanoid robot is capable of recognizing sign language and translating it into spoken language, while imitating the facial emotions, providing a comprehensive solution to the communication challenges faced by individuals with hearing and speech impairments.

Learning representations for robust human–robot interaction

AbstractThis article summarizes the author's presentation in the New Faculty Highlight at the Thirty‐Eighth AAAI Conference on Artificial Intelligence. It discusses the desired properties of representations for enabling robust human–robot interaction. Examples from the author's work are presented to show how to build these properties into models for performing tasks with natural language guidance and engaging in social interactions with other agents.

A Two-Stage Facial Kinematic Control Strategy for Humanoid Robots Based on Keyframe Detection and Keypoint Cubic Spline Interpolation

A plentiful number of facial expressions is the basis of natural human–robot interaction for high-fidelity humanoid robots. The facial expression imitation of humanoid robots involves the transmission of human facial expression data to servos situated within the robot’s head. These data drive the servos to manipulate the skin, thereby enabling the robot to exhibit various facial expressions. However, since the mechanical transmission rate cannot keep up with the data processing rate, humanoid robots often suffer from jitters in the imitation. We conducted a thorough analysis of the transmitted facial expression sequence data and discovered that they are extremely redundant. Therefore, we designed a two-stage strategy for humanoid robots based on facial keyframe detection and facial keypoint detection to achieve more natural and smooth expression imitation. We first built a facial keyframe detection model based on ResNet-50, combined with optical flow estimation, which can identify key expression frames in the sequence. Then, a facial keypoint detection model is used on the keyframes to obtain the facial keypoint coordinates. Based on the coordinates, the cubic spline interpolation method is used to obtain the motion trajectory parameters of the servos, thus realizing the robust control of the humanoid robot’s facial expression. Experiments show that, unlike before where the robot’s imitation would stutter at frame rates above 25 fps, our strategy allows the robot to maintain good facial expression imitation similarity (cosine similarity of 0.7226), even at higher frame rates.

A Human–Security Robot Interaction Literature Review

As advances in robotics continue, security robots are increasingly integrated into public and private security, enhancing protection in locations such as streets, parks, and shopping malls. To be effective, security robots must interact with civilians and security personnel, underscoring the need to enhance our knowledge of their interactions with humans. To investigate this issue, the authors systematically reviewed 47 studies on human interaction with security robots, covering 2003 to 2023. Papers in this domain have significantly increased over the last seven years. The paper provides three contributions. First, it comprehensively summarizes existing literature on human interaction with security robots. Second, it employs the Human–Robot Integrative Framework (HRIF) to categorize this literature into three main thrusts: human, robot, and context. The framework is leveraged to derive insights into the methodologies, tasks, predictors, and outcomes studied. Last, the paper synthesizes and discusses the findings from the reviewed literature, identifying avenues for future research in this domain.