Natural Human-robot Interaction Research Articles

THÖR-MAGNI: A large-scale indoor motion capture recording of human movement and robot interaction

We present a new large dataset of indoor human and robot navigation and interaction, called THÖR-MAGNI, that is designed to facilitate research on social human navigation: for example, modeling and predicting human motion, analyzing goal-oriented interactions between humans and robots, and investigating visual attention in a social interaction context. THÖR-MAGNI was created to fill a gap in available datasets for human motion analysis and HRI. This gap is characterized by a lack of comprehensive inclusion of exogenous factors and essential target agent cues, which hinders the development of robust models capable of capturing the relationship between contextual cues and human behavior in different scenarios. Unlike existing datasets, THÖR-MAGNI includes a broader set of contextual features and offers multiple scenario variations to facilitate factor isolation. The dataset includes many social human–human and human–robot interaction scenarios, rich context annotations, and multi-modal data, such as walking trajectories, gaze-tracking data, and lidar and camera streams recorded from a mobile robot. We also provide a set of tools for visualization and processing of the recorded data. THÖR-MAGNI is, to the best of our knowledge, unique in the amount and diversity of sensor data collected in a contextualized and socially dynamic environment, capturing natural human–robot interactions.

A Two-Stage Facial Kinematic Control Strategy for Humanoid Robots Based on Keyframe Detection and Keypoint Cubic Spline Interpolation

A plentiful number of facial expressions is the basis of natural human–robot interaction for high-fidelity humanoid robots. The facial expression imitation of humanoid robots involves the transmission of human facial expression data to servos situated within the robot’s head. These data drive the servos to manipulate the skin, thereby enabling the robot to exhibit various facial expressions. However, since the mechanical transmission rate cannot keep up with the data processing rate, humanoid robots often suffer from jitters in the imitation. We conducted a thorough analysis of the transmitted facial expression sequence data and discovered that they are extremely redundant. Therefore, we designed a two-stage strategy for humanoid robots based on facial keyframe detection and facial keypoint detection to achieve more natural and smooth expression imitation. We first built a facial keyframe detection model based on ResNet-50, combined with optical flow estimation, which can identify key expression frames in the sequence. Then, a facial keypoint detection model is used on the keyframes to obtain the facial keypoint coordinates. Based on the coordinates, the cubic spline interpolation method is used to obtain the motion trajectory parameters of the servos, thus realizing the robust control of the humanoid robot’s facial expression. Experiments show that, unlike before where the robot’s imitation would stutter at frame rates above 25 fps, our strategy allows the robot to maintain good facial expression imitation similarity (cosine similarity of 0.7226), even at higher frame rates.

LQR-based control strategy for improving human-robot companionship and natural obstacle avoidance

In the dynamic and unstructured environment of human–robot symbiosis, companion robots require natural human–robot interaction and autonomous intelligence through multimodal information fusion to achieve effective collaboration. Nevertheless, the control precision and coordination of the accompanying actions are not satisfactory in practical applications. This is primarily attributed to the difficulties in the motion coordination between the accompanying target and the mobile robot. This paper proposes a companion control strategy based on the Linear Quadratic Regulator (LQR) to enhance the coordination and precision of robot companion tasks. This method enables the robot to adapt to sudden changes in the companion target’s motion. Besides, the robot could smoothly avoid obstacles during the companion process. Firstly, a human–robot companion interaction model based on nonholonomic constraints is developed to determine the relative position and orientation between the robot and the companion target. Then, an LQR-based companion controller incorporating behavioral dynamics is introduced to simultaneously avoid obstacles and track the companion target’s direction and velocity. Finally, various simulations and real-world human–robot companion experiments are conducted to regulate the relative position, orientation, and velocity between the target object and the robot platform. Experimental results demonstrate the superiority of this approach over conventional control algorithms in terms of control distance and directional errors throughout system operation. The proposed LQR-based control strategy ensures coordinated and consistent motion with target persons in social companion scenarios.

How an Android Expresses “Now Loading…”: Examining the Properties of Thinking Faces

The “thinking face” is a facial signal used to convey being in thought. For androids, the thinking face may be important to achieve natural human–robot interaction. However, the facial pattern necessary for portraying the thinking face remains unclear and has not yet been investigated in androids. The current study aims to (a) identify the facial patterns when people are engaged in answering complex questions (i.e., thinking face) and (b) clarify whether implementing the observed thinking faces in an android can facilitate natural human–robot interaction. In Study 1, we analyze the facial movements of 40 participants after they are prompted with difficult questions and indicate five facial patterns that corresponded to thinking faces. In Study 2, we further focus on the pattern of furrowing of the brows and narrowing of the eyes among the observed thinking facial patterns and implement this pattern in an android. The results show that thinking faces enhance the perception of being in thought, genuineness, human-likeness, and appropriateness in androids while decreasing eeriness. The free-description data also revealed that negative emotions are attributed to the thinking face. In Study 3, we compared the thinking vs. neutral faces in a question–answer situation. The results showed that the android's thinking face facilitated the perception of being in thought and human-likeness. These findings suggest that the thinking face of androids can facilitate natural human–robot interaction.

The Effects of Human-Robot Interactions and the Human-Robot Relationship on Robot Competence, Trust, and Acceptance

As social robots may be used by a single user or multiple users different social scenarios are becoming more important for defining human-robot relationships. Therefore, this study explored human-robot relationships between robots and users in different interaction modes to improve user interaction experience. Specifically, education and companion were selected as the most common areas in the use of social robots. The interaction modes used include single-user interaction and multi-user interaction. The three human-robot relationships were adopted. The robot competence scale, human-robot trust scale, and acceptance of robot scale were used to evaluate subjects’ views on robots. The results demonstrate that in the two scenarios, people were more inclined to maintain a more familiar and closer relationship with the social robot when the robot interacted with a single user. When multiple persons interact in an education scenario, setting the robot to Acquaintance relationships is recommended to improve its competence and people’s trust in the robot. Similarly, in multi-person interaction, Acquaintance relationships would be more accepted and trusted by people in a companion scenario. Based on these results, robot sensors can be added to further optimize human-robot interaction sensing systems. By identifying the number of users in the interaction environment, robots can automatically employ the best human-robot relationship for interaction. Optimizing human-robot interaction sensing systems can also improve robot performance perceived in the interaction to meet different users’ needs and achieve more natural human-robot interaction experiences.

Self-administered questionnaires enhance emotion estimation of individuals with autism spectrum disorders in a robotic interview setting.

Robots offer many unique opportunities for helping individuals with autism spectrum disorders (ASD). Determining the optimal motion of robots when interacting with individuals with ASD is important for achieving more natural human-robot interactions and for exploiting the full potential of robotic interventions. Most prior studies have used supervised machine learning (ML) of user behavioral data to enable robot perception of affective states (i.e., arousal and valence) and engagement. It has previously been suggested that including personal demographic information in the identification of individuals with ASD is important for developing an automated system to perceive individual affective states and engagement. In this study, we hypothesized that assessing self-administered questionnaire data would contribute to the development of an automated estimation of the affective state and engagement when individuals with ASD are interviewed by an Android robot, which will be linked to implementing long-term interventions and maintaining the motivation of participants. Participants sat across a table from an android robot that played the role of the interviewer. Each participant underwent a mock job interview. Twenty-five participants with ASD (males 22, females 3, average chronological age = 22.8, average IQ = 94.04) completed the experiment. We collected multimodal data (i.e., audio, motion, gaze, and self-administered questionnaire data) to train a model to correctly classify the state of individuals with ASD when interviewed by an android robot. We demonstrated the technical feasibility of using ML to enable robot perception of affect and engagement of individuals with ASD based on multimodal data. For arousal and engagement, the area under the curve (AUC) values of the model estimates and expert coding were relatively high. Overall, the AUC values of arousal, valence, and engagement were improved by including self-administered questionnaire data in the classification. These findings support the hypothesis that assessing self-administered questionnaire data contributes to the development of an automated estimation of an individual's affective state and engagement. Given the efficacy of including self-administered questionnaire data, future studies should confirm the effectiveness of such long-term intervention with a robot to maintain participants' motivation based on the proposed method of emotion estimation.

A Robotic Teleoperation System Enhanced by Augmented Reality for Natural Human–Robot Interaction

Telekinesis, as commonly portrayed in science fiction literature and cinema, is a super power wherein users control and manipulate objects absent in physical interaction. In real world, enhancing human–robot interaction needs the synthesis of human intuitive processes with robotic arm. This paper introduces a robotic teleoperation system achieving the essence of telekinetic operations, combining the profound capabilities of augmented reality (AR) with the robotic arm operations. Utilizing AR, the proposed methodology offers operators with a visual feedback, facilitating a level of control surpassing the capacities of natural interfaces. By using AR-driven visual recognition, this system achieves operations in a virtual environment, subsequently actualized in the real world through the robotic arm. Through multiple experiments, we found that the system has a small margin of error in telekinesis operations, meeting the needs of remote operation. Furthermore, our system can operate on objects in the real world. These experiments underscore the capability of the remote control system to assist humans in accomplishing a wider range of tasks through the integration of AR and robotic arms, providing a natural human–robot interaction approach.

An emotion-driven and topic-aware dialogue framework for human–robot interaction

ABSTRACT Developing dialogue services for robots has been promoted nowadays for providing natural human–robot interactions to enhance user experiences as conversation is a key instrument for creating and maintaining their mutual relationships. In this work, we present a trainable framework for modeling context-aware human–robot dialogues, and apply it to a live streaming application to demonstrate the need for social robots. Our social chatting robot framework takes into account both emotional context and topic-focused content information to generate appropriate responses. This framework has some unique features. It adopts a multimodal deep learning model to recognize the participants’ emotions. Moreover, our work includes a topic-aware neural model that enables the social robot to follow certain conversation topics to fulfill the specific dialoguing goals and to enhance the chatting coherence. Most importantly, we design a strategy able to take emotions as the drive to control over the neural method of generating utterances. To evaluate the proposed approach, we have conducted several sets of quantitative and quantitative experiments. The results highlight the importance of multimodal emotion recognition and topic-awareness in dialoguing and confirm the feasibility and effectiveness of our framework for human–robot interaction.

A high-sensitivity wearable flexible strain sensor based on three-dimensional twist-like network structure

With the rapid development of the flexible electronics field, flexible pressure sensors are widely used inwearable devices, medical monitoring and other fields. However, it is still a huge challenge to design a sensor with highsensitivity, low cost and a simple manufacturing process. In this work, we report a flexible strain sensor based on discarded mask straps (MS) with a unique three-dimensional (3D) twist-like network structure. Multi-walled carbon nanotubes (MWCNTs)/carbon nanoparticles (CN)/MS composites were prepared by a simple dip-drying method and encapsulated using a medical Polyurethane (PU) film. It is founded that the fabricated MWCNTs/CN/MS/PU flexible strain sensor (MCMP-FSS) exhibits a high gauge factor (GF) of 13,928 and advanced performance with a large sensing range of 0%–72%. Moreover, good water-repellent harmony and various types of stimuli over 1000 cycles were obtained for the MCMP-FSS. Benefiting from the unique 3D twist-like network structure and the MWCNTs/CN synergistic conductive network, MCMP-FSS also has broad application prospects in the fields of human motion detection, encrypted communication, and natural human-robot interaction.

Computer vision-based hand gesture recognition for human-robot interaction: a review

As robots have become more pervasive in our daily life, natural human-robot interaction (HRI) has had a positive impact on the development of robotics. Thus, there has been growing interest in the development of vision-based hand gesture recognition for HRI to bridge human-robot barriers. The aim is for interaction with robots to be as natural as that between individuals. Accordingly, incorporating hand gestures in HRI is a significant research area. Hand gestures can provide natural, intuitive, and creative methods for communicating with robots. This paper provides an analysis of hand gesture recognition using both monocular cameras and RGB-D cameras for this purpose. Specifically, the main process of visual gesture recognition includes data acquisition, hand gesture detection and segmentation, feature extraction and gesture classification, which are discussed in this paper. Experimental evaluations are also reviewed. Furthermore, algorithms of hand gesture recognition for human-robot interaction are examined in this study. In addition, the advances required for improvement in the present hand gesture recognition systems, which can be applied for effective and efficient human-robot interaction, are discussed.

Adaptive timing in a dynamic field architecture for natural human–robot interactions

A close temporal coordination of actions and goals is crucial for natural and fluent human–robot interactions in collaborative tasks. How to endow an autonomous robot with a basic temporal cognition capacity is an open question. In this paper, we present a neurodynamics approach based on the theoretical framework of dynamic neural fields (DNF) which assumes that timing processes are closely integrated with other cognitive computations. The continuous evolution of neural population activity towards an attractor state provides an implicit sensation of the passage of time. Highly flexible sensorimotor timing can be achieved through manipulations of inputs or initial conditions that affect the speed with which the neural trajectory evolves. We test a DNF-based control architecture in an assembly paradigm in which an assistant hands over a series of pieces which the operator uses among others in the assembly process. By watching two experts, the robot first learns the serial order and relative timing of object transfers to subsequently substitute the assistant in the collaborative task. A dynamic adaptation rule exploiting a perceived temporal mismatch between the expected and the realized transfer timing allows the robot to quickly adapt its proactive motor timing to the pace of the operator even when an additional assembly step delays a handover. Moreover, the self-stabilizing properties of the population dynamics support the fast internal simulation of acquired task knowledge allowing the robot to anticipate serial order errors.

Multi-Level Attention-Based Categorical Emotion Recognition Using Modulation-Filtered Cochleagram

Speech emotion recognition is a critical component for achieving natural human–robot interaction. The modulation-filtered cochleagram is a feature based on auditory modulation perception, which contains multi-dimensional spectral–temporal modulation representation. In this study, we propose an emotion recognition framework that utilizes a multi-level attention network to extract high-level emotional feature representations from the modulation-filtered cochleagram. Our approach utilizes channel-level attention and spatial-level attention modules to generate emotional saliency maps of channel and spatial feature representations, capturing significant emotional channel and feature space from the 3D convolution feature maps, respectively. Furthermore, we employ a temporal-level attention module to capture significant emotional regions from the concatenated feature sequence of the emotional saliency maps. Our experiments on the Interactive Emotional Dyadic Motion Capture (IEMOCAP) dataset demonstrate that the modulation-filtered cochleagram significantly improves the prediction performance of categorical emotion compared to other evaluated features. Moreover, our emotion recognition framework achieves comparable unweighted accuracy of 71% in categorical emotion recognition by comparing with several existing approaches. In summary, our study demonstrates the effectiveness of the modulation-filtered cochleagram in speech emotion recognition, and our proposed multi-level attention framework provides a promising direction for future research in this field.

MMFN: Emotion recognition by fusing touch gesture and facial expression information

Human social interaction is a multimodal process that integrates several communication channels, such as speech, facial expressions, body gestures, and touch. For intelligent robots, the ability to recognize human emotions based on these channels helps in realizing natural human-robot interaction. The existing works on multimodal emotion recognition (MER) mainly focus on facial expression, speech, text, and electrophysiological signals. However, there are few works on the implementation of MER based on touch. In this work, we established a facial expression and touch gesture emotion (FETE) dataset comprising six basic discrete emotions, which was made publicly available for the first time. In addition, a multi-eigenspace based multimodal fusion network (MMFN) was proposed for tactile-visual bimodal emotion recognition. The proposed MMFN projects the tactile and visual modalities in multi-eigenspace to learn the specific and shared representations and employs multiple classifiers to decode these features. We demonstrated the effectiveness of our network through experiments on the FETE dataset. The tactile-visual dual-modal emotion recognition using MMFN achieved an accuracy of 80.18% and an improvement of approximately 15% over the single-modality results.

Touchless and nonverbal human-robot interfaces: An overview of the state-of-the-art

Human-robot interfaces encompass developing technologies that determine how a user interacts with robot systems. Touchless and nonverbal interfaces are a subset of these in which the user can interact with robots without the use of buttons, speech, or other physical methods. The two driving forces behind the development of these methods are the desire for more natural Human-Robot Interaction (HRI) and the need for more efficacious assistive technologies. Many interface methods inspired by natural human interaction can improve HRIs. These include gaze/eye contact, emotion detection, and gesture recognition. Assistive robot systems can efficiently utilize these methods, as well as other interfaces that are uncommon in human interactions. These includes several unexplored methods such as biopotential signals, the electroencephalogram (EEG) being one example. Research is also underway in exploring the use of involuntary or subconscious signals in human interaction, such as lie detection or behavior analysis. This paper compiles research findings in several of these areas and discusses the benefits, limitations, and potential for future development in several essential touchless and nonverbal human-robot interface methods. This includes a discussion of Internet-of-Things (IoT) devices as they relate to how robot systems can gather information through the use of sensors. The paper also examines cognitive robots and explores possible advancements that can be explored in this area using next-generation mobile networking (5G and 6G) to allow for high data transfer rates, highlighting important information gathered by robots through different interfacing methods. This leads to several new avenues of future research.

Real-Time Robotic Mirrored Behavior of Facial Expressions and Head Motions Based on Lightweight Networks

The ability of a humanoid robot to imitate facial expressions with simultaneous head motions is crucial to natural human–robot interaction. This mirrored behavior from human beings to humanoid robots has high demands of similarity and real-time performance. To fulfill these needs, this article proposes a real-time robotic mirrored behavior of facial expressions and head motions based on lightweight networks. First, a humanoid robot that can change the state of its facial organs and neck through servo displacement is developed to achieve the mirrored behavior of facial expressions and head motions. Second, to overcome the high latency caused by deep learning models running in embedded devices, a lightweight deep learning network is constructed for detecting facial feature points, which can reduce model size and improve running speed without affecting the performance of the model. Finally, a mapping relationship of 68 facial feature points to optimal servo displacements is established to realize the mirrored behavior from human beings to humanoid robots. The experimental results show that the facial feature point recognition method based on the lightweight model performs better than other state-of-the-art methods, and our head motion tracking method can maintain high accuracy compared with the gold standard optical motion capture system NOKOV. Overall, our method ensures the accurate and real-time generation of robot mirrored behavior and has a certain reference value for the efficient and natural interaction between humans and robots.

IRWoZ: Constructing an Industrial Robot Wizard-of-OZ Dialoguing Dataset

Enabling a flexible and natural human-robot interaction (HRI) for industrial robots is a critical yet challenging task that can be facilitated by the use of conversational artificial intelligence (AI). Prior research has concentrated on strengthening interactions through the deployment of social robots, while disregarding the capabilities required to boost the flexibility and user experience associated with human-robot collaboration (HRC) on manufacturing tasks. One of the main challenges is the lack of publicly available industrial-oriented dialogue datasets for the training of conversational AI. In this work, we present an industrial robot wizard-of-Oz dialoguing dataste (IRWoZ) focused on enabling HRC in manufacturing tasks. The dataset covers four domains: assembly, transportation, position, and relocation. It is created using the Wizard-of-Oz technique to be less noisy. We manually constructed, annotated and validated dialogue segments (e.g., intentions, slots, annotations), as well as the responses. Building upon the proposed dataset, we benchmark it on the state-of-the-art (SoTA) language models, generative pre-trained (GPT-2) models, on dialogue state tracking and response generation tasks. We expect that the IRWoZ dataset will facilitate exciting ongoing dialogue research and we provide it freely accessible at https://github.com/lcroy/ToD4IR/tree/main/dataset.

A Mobile Natural Human–Robot Interaction Method for Virtual Chinese Acupuncture

Most of the existing virtual acupuncture methods cannot allow operators to interact naturally in a large area. To address this problem, this article proposes a mobile natural human–robot interaction interface for virtual Chinese acupuncture teaching, including an automatic hand tracking method and an acupuncture interaction method combining vision and force feedback. The automatic hand tracking method can automatically track and capture the operator’s hand movements, allowing the operator to perform acupuncture in a large area. The acupuncture interaction method combining vision and force reproduces the real acupuncture operation mode. The proposed interaction method provides the operator with visual feedback and force feedback without limitation, which improves the naturalness and authenticity of the operation. Through a series of experiments, the proposed method was proven to be more natural and efficient in the virtual acupuncture teaching scene from the point view of interaction. The operators achieve a stronger sense of immersion using this method.

Noise-tolerant zeroing neurodynamic algorithm for upper limb motion intention-based human–robot interaction control in non-ideal conditions

Human–robot interaction control, a crucial technology in industrial manufacturing and rehabilitation robot, has been widely explored in recent years. However, there are still some problems in the existing strategy need to be solved for the rehabilitation robot: First, the human–robot confrontation (HRC) in motion control affects the information interaction. Second, in non-ideal conditions, the safety and accuracy of the control system will be disturbed by external unknown factors. Third, the existing continuous-time controllers are hard to be applied to hardware platforms. In this paper, an upper limb-exoskeleton coupling dynamics model based on human active motion intention is established to overcome HRC, and promote active and natural human–robot interaction control. Specifically, the Elman neural network (ENN) model is devised and utilized to estimate human active motion intention using the surface electromyography (sEMG) signals. On the basis of human active motion intention, the discrete-time controller predicated on noise-tolerant zeroing neurodynamic model (termed as NTZND controller) is proposed for constructing the discrete-time human–robot interaction control system (DTHRICS) under non-ideal conditions. In addition, the zeroing neurodynamic (ZND) controller, the gradient neural network (GNN) and proportional–derivative (PD) controllers are designed for comparisons. Theoretical analyses demonstrate that the 0-stability, consistency and convergence of the designed DTHRICS. The performance indicators root mean square errors (RMSEs) and coefficient of determination (R2) of the DTHRICS with different controllers are analyzed and compared under ideal and non-ideal conditions. Ultimately, numerical results illustrate that the NTZND-based DTHRICS (with the lowest performance RMSEs →0 and the highest performance R2→1, p<0.05) is significantly superior to the DTHRICS with the ZND, the GNN and the PD controllers in terms of control accuracy, noise-tolerant property and real time performance, while utilized for upper limb multi-joint motion control under non-ideal conditions.

Bilateral personalized dialogue generation with contrastive learning

Generating personalized responses is one of the major challenges in natural human-robot interaction. Current researches in this field mainly focus on generating responses consistent with the robot's pre-assigned persona, while ignoring the user's persona. Such responses may be inappropriate or even offensive, which may lead to the bad user experience. Therefore, we propose a Bilateral Personalized Dialogue Generation (BPDG) method for dyadic conversation, which integrates user and robot personas into dialogue generation via designing a dynamic persona-aware fusion method. To bridge the gap between the learning objective function and evaluation metrics, the Conditional Mutual Information Maximum (CMIM) criterion is adopted with contrastive learning to select the proper response from the generated candidates. Moreover, a bilateral persona accuracy metric is designed to measure the degree of bilateral personalization. Experimental results demonstrate that, compared with several state-of-the-art methods, the final results of the proposed method are more personalized and consistent with bilateral personas in terms of both automatic and manual evaluations.

A Bio-Inspired Endogenous Attention-Based Architecture for a Social Robot.

A robust perception system is crucial for natural human–robot interaction. An essential capability of these systems is to provide a rich representation of the robot’s environment, typically using multiple sensory sources. Moreover, this information allows the robot to react to both external stimuli and user responses. The novel contribution of this paper is the development of a perception architecture, which was based on the bio-inspired concept of endogenous attention being integrated into a real social robot. In this paper, the architecture is defined at a theoretical level to provide insights into the underlying bio-inspired mechanisms and at a practical level to integrate and test the architecture within the complete architecture of a robot. We also defined mechanisms to establish the most salient stimulus for the detection or task in question. Furthermore, the attention-based architecture uses information from the robot’s decision-making system to produce user responses and robot decisions. Finally, this paper also presents the preliminary test results from the integration of this architecture into a real social robot.