Natural Human-robot Interaction Research Articles

How an Android Expresses “Now Loading…”: Examining the Properties of Thinking Faces

The “thinking face” is a facial signal used to convey being in thought. For androids, the thinking face may be important to achieve natural human–robot interaction. However, the facial pattern necessary for portraying the thinking face remains unclear and has not yet been investigated in androids. The current study aims to (a) identify the facial patterns when people are engaged in answering complex questions (i.e., thinking face) and (b) clarify whether implementing the observed thinking faces in an android can facilitate natural human–robot interaction. In Study 1, we analyze the facial movements of 40 participants after they are prompted with difficult questions and indicate five facial patterns that corresponded to thinking faces. In Study 2, we further focus on the pattern of furrowing of the brows and narrowing of the eyes among the observed thinking facial patterns and implement this pattern in an android. The results show that thinking faces enhance the perception of being in thought, genuineness, human-likeness, and appropriateness in androids while decreasing eeriness. The free-description data also revealed that negative emotions are attributed to the thinking face. In Study 3, we compared the thinking vs. neutral faces in a question–answer situation. The results showed that the android's thinking face facilitated the perception of being in thought and human-likeness. These findings suggest that the thinking face of androids can facilitate natural human–robot interaction.

The Effects of Human-Robot Interactions and the Human-Robot Relationship on Robot Competence, Trust, and Acceptance

As social robots may be used by a single user or multiple users different social scenarios are becoming more important for defining human-robot relationships. Therefore, this study explored human-robot relationships between robots and users in different interaction modes to improve user interaction experience. Specifically, education and companion were selected as the most common areas in the use of social robots. The interaction modes used include single-user interaction and multi-user interaction. The three human-robot relationships were adopted. The robot competence scale, human-robot trust scale, and acceptance of robot scale were used to evaluate subjects’ views on robots. The results demonstrate that in the two scenarios, people were more inclined to maintain a more familiar and closer relationship with the social robot when the robot interacted with a single user. When multiple persons interact in an education scenario, setting the robot to Acquaintance relationships is recommended to improve its competence and people’s trust in the robot. Similarly, in multi-person interaction, Acquaintance relationships would be more accepted and trusted by people in a companion scenario. Based on these results, robot sensors can be added to further optimize human-robot interaction sensing systems. By identifying the number of users in the interaction environment, robots can automatically employ the best human-robot relationship for interaction. Optimizing human-robot interaction sensing systems can also improve robot performance perceived in the interaction to meet different users’ needs and achieve more natural human-robot interaction experiences.

Self-administered questionnaires enhance emotion estimation of individuals with autism spectrum disorders in a robotic interview setting.

Robots offer many unique opportunities for helping individuals with autism spectrum disorders (ASD). Determining the optimal motion of robots when interacting with individuals with ASD is important for achieving more natural human-robot interactions and for exploiting the full potential of robotic interventions. Most prior studies have used supervised machine learning (ML) of user behavioral data to enable robot perception of affective states (i.e., arousal and valence) and engagement. It has previously been suggested that including personal demographic information in the identification of individuals with ASD is important for developing an automated system to perceive individual affective states and engagement. In this study, we hypothesized that assessing self-administered questionnaire data would contribute to the development of an automated estimation of the affective state and engagement when individuals with ASD are interviewed by an Android robot, which will be linked to implementing long-term interventions and maintaining the motivation of participants. Participants sat across a table from an android robot that played the role of the interviewer. Each participant underwent a mock job interview. Twenty-five participants with ASD (males 22, females 3, average chronological age = 22.8, average IQ = 94.04) completed the experiment. We collected multimodal data (i.e., audio, motion, gaze, and self-administered questionnaire data) to train a model to correctly classify the state of individuals with ASD when interviewed by an android robot. We demonstrated the technical feasibility of using ML to enable robot perception of affect and engagement of individuals with ASD based on multimodal data. For arousal and engagement, the area under the curve (AUC) values of the model estimates and expert coding were relatively high. Overall, the AUC values of arousal, valence, and engagement were improved by including self-administered questionnaire data in the classification. These findings support the hypothesis that assessing self-administered questionnaire data contributes to the development of an automated estimation of an individual's affective state and engagement. Given the efficacy of including self-administered questionnaire data, future studies should confirm the effectiveness of such long-term intervention with a robot to maintain participants' motivation based on the proposed method of emotion estimation.

A Robotic Teleoperation System Enhanced by Augmented Reality for Natural Human–Robot Interaction

Telekinesis, as commonly portrayed in science fiction literature and cinema, is a super power wherein users control and manipulate objects absent in physical interaction. In real world, enhancing human–robot interaction needs the synthesis of human intuitive processes with robotic arm. This paper introduces a robotic teleoperation system achieving the essence of telekinetic operations, combining the profound capabilities of augmented reality (AR) with the robotic arm operations. Utilizing AR, the proposed methodology offers operators with a visual feedback, facilitating a level of control surpassing the capacities of natural interfaces. By using AR-driven visual recognition, this system achieves operations in a virtual environment, subsequently actualized in the real world through the robotic arm. Through multiple experiments, we found that the system has a small margin of error in telekinesis operations, meeting the needs of remote operation. Furthermore, our system can operate on objects in the real world. These experiments underscore the capability of the remote control system to assist humans in accomplishing a wider range of tasks through the integration of AR and robotic arms, providing a natural human–robot interaction approach.

An emotion-driven and topic-aware dialogue framework for human–robot interaction

ABSTRACT Developing dialogue services for robots has been promoted nowadays for providing natural human–robot interactions to enhance user experiences as conversation is a key instrument for creating and maintaining their mutual relationships. In this work, we present a trainable framework for modeling context-aware human–robot dialogues, and apply it to a live streaming application to demonstrate the need for social robots. Our social chatting robot framework takes into account both emotional context and topic-focused content information to generate appropriate responses. This framework has some unique features. It adopts a multimodal deep learning model to recognize the participants’ emotions. Moreover, our work includes a topic-aware neural model that enables the social robot to follow certain conversation topics to fulfill the specific dialoguing goals and to enhance the chatting coherence. Most importantly, we design a strategy able to take emotions as the drive to control over the neural method of generating utterances. To evaluate the proposed approach, we have conducted several sets of quantitative and quantitative experiments. The results highlight the importance of multimodal emotion recognition and topic-awareness in dialoguing and confirm the feasibility and effectiveness of our framework for human–robot interaction.

A high-sensitivity wearable flexible strain sensor based on three-dimensional twist-like network structure

With the rapid development of the flexible electronics field, flexible pressure sensors are widely used inwearable devices, medical monitoring and other fields. However, it is still a huge challenge to design a sensor with highsensitivity, low cost and a simple manufacturing process. In this work, we report a flexible strain sensor based on discarded mask straps (MS) with a unique three-dimensional (3D) twist-like network structure. Multi-walled carbon nanotubes (MWCNTs)/carbon nanoparticles (CN)/MS composites were prepared by a simple dip-drying method and encapsulated using a medical Polyurethane (PU) film. It is founded that the fabricated MWCNTs/CN/MS/PU flexible strain sensor (MCMP-FSS) exhibits a high gauge factor (GF) of 13,928 and advanced performance with a large sensing range of 0%–72%. Moreover, good water-repellent harmony and various types of stimuli over 1000 cycles were obtained for the MCMP-FSS. Benefiting from the unique 3D twist-like network structure and the MWCNTs/CN synergistic conductive network, MCMP-FSS also has broad application prospects in the fields of human motion detection, encrypted communication, and natural human-robot interaction.

Computer vision-based hand gesture recognition for human-robot interaction: a review

As robots have become more pervasive in our daily life, natural human-robot interaction (HRI) has had a positive impact on the development of robotics. Thus, there has been growing interest in the development of vision-based hand gesture recognition for HRI to bridge human-robot barriers. The aim is for interaction with robots to be as natural as that between individuals. Accordingly, incorporating hand gestures in HRI is a significant research area. Hand gestures can provide natural, intuitive, and creative methods for communicating with robots. This paper provides an analysis of hand gesture recognition using both monocular cameras and RGB-D cameras for this purpose. Specifically, the main process of visual gesture recognition includes data acquisition, hand gesture detection and segmentation, feature extraction and gesture classification, which are discussed in this paper. Experimental evaluations are also reviewed. Furthermore, algorithms of hand gesture recognition for human-robot interaction are examined in this study. In addition, the advances required for improvement in the present hand gesture recognition systems, which can be applied for effective and efficient human-robot interaction, are discussed.

Adaptive timing in a dynamic field architecture for natural human–robot interactions

A close temporal coordination of actions and goals is crucial for natural and fluent human–robot interactions in collaborative tasks. How to endow an autonomous robot with a basic temporal cognition capacity is an open question. In this paper, we present a neurodynamics approach based on the theoretical framework of dynamic neural fields (DNF) which assumes that timing processes are closely integrated with other cognitive computations. The continuous evolution of neural population activity towards an attractor state provides an implicit sensation of the passage of time. Highly flexible sensorimotor timing can be achieved through manipulations of inputs or initial conditions that affect the speed with which the neural trajectory evolves. We test a DNF-based control architecture in an assembly paradigm in which an assistant hands over a series of pieces which the operator uses among others in the assembly process. By watching two experts, the robot first learns the serial order and relative timing of object transfers to subsequently substitute the assistant in the collaborative task. A dynamic adaptation rule exploiting a perceived temporal mismatch between the expected and the realized transfer timing allows the robot to quickly adapt its proactive motor timing to the pace of the operator even when an additional assembly step delays a handover. Moreover, the self-stabilizing properties of the population dynamics support the fast internal simulation of acquired task knowledge allowing the robot to anticipate serial order errors.

Multi-Level Attention-Based Categorical Emotion Recognition Using Modulation-Filtered Cochleagram

Speech emotion recognition is a critical component for achieving natural human–robot interaction. The modulation-filtered cochleagram is a feature based on auditory modulation perception, which contains multi-dimensional spectral–temporal modulation representation. In this study, we propose an emotion recognition framework that utilizes a multi-level attention network to extract high-level emotional feature representations from the modulation-filtered cochleagram. Our approach utilizes channel-level attention and spatial-level attention modules to generate emotional saliency maps of channel and spatial feature representations, capturing significant emotional channel and feature space from the 3D convolution feature maps, respectively. Furthermore, we employ a temporal-level attention module to capture significant emotional regions from the concatenated feature sequence of the emotional saliency maps. Our experiments on the Interactive Emotional Dyadic Motion Capture (IEMOCAP) dataset demonstrate that the modulation-filtered cochleagram significantly improves the prediction performance of categorical emotion compared to other evaluated features. Moreover, our emotion recognition framework achieves comparable unweighted accuracy of 71% in categorical emotion recognition by comparing with several existing approaches. In summary, our study demonstrates the effectiveness of the modulation-filtered cochleagram in speech emotion recognition, and our proposed multi-level attention framework provides a promising direction for future research in this field.

MMFN: Emotion recognition by fusing touch gesture and facial expression information

Human social interaction is a multimodal process that integrates several communication channels, such as speech, facial expressions, body gestures, and touch. For intelligent robots, the ability to recognize human emotions based on these channels helps in realizing natural human-robot interaction. The existing works on multimodal emotion recognition (MER) mainly focus on facial expression, speech, text, and electrophysiological signals. However, there are few works on the implementation of MER based on touch. In this work, we established a facial expression and touch gesture emotion (FETE) dataset comprising six basic discrete emotions, which was made publicly available for the first time. In addition, a multi-eigenspace based multimodal fusion network (MMFN) was proposed for tactile-visual bimodal emotion recognition. The proposed MMFN projects the tactile and visual modalities in multi-eigenspace to learn the specific and shared representations and employs multiple classifiers to decode these features. We demonstrated the effectiveness of our network through experiments on the FETE dataset. The tactile-visual dual-modal emotion recognition using MMFN achieved an accuracy of 80.18% and an improvement of approximately 15% over the single-modality results.

Touchless and nonverbal human-robot interfaces: An overview of the state-of-the-art

Human-robot interfaces encompass developing technologies that determine how a user interacts with robot systems. Touchless and nonverbal interfaces are a subset of these in which the user can interact with robots without the use of buttons, speech, or other physical methods. The two driving forces behind the development of these methods are the desire for more natural Human-Robot Interaction (HRI) and the need for more efficacious assistive technologies. Many interface methods inspired by natural human interaction can improve HRIs. These include gaze/eye contact, emotion detection, and gesture recognition. Assistive robot systems can efficiently utilize these methods, as well as other interfaces that are uncommon in human interactions. These includes several unexplored methods such as biopotential signals, the electroencephalogram (EEG) being one example. Research is also underway in exploring the use of involuntary or subconscious signals in human interaction, such as lie detection or behavior analysis. This paper compiles research findings in several of these areas and discusses the benefits, limitations, and potential for future development in several essential touchless and nonverbal human-robot interface methods. This includes a discussion of Internet-of-Things (IoT) devices as they relate to how robot systems can gather information through the use of sensors. The paper also examines cognitive robots and explores possible advancements that can be explored in this area using next-generation mobile networking (5G and 6G) to allow for high data transfer rates, highlighting important information gathered by robots through different interfacing methods. This leads to several new avenues of future research.

Real-Time Robotic Mirrored Behavior of Facial Expressions and Head Motions Based on Lightweight Networks

The ability of a humanoid robot to imitate facial expressions with simultaneous head motions is crucial to natural human–robot interaction. This mirrored behavior from human beings to humanoid robots has high demands of similarity and real-time performance. To fulfill these needs, this article proposes a real-time robotic mirrored behavior of facial expressions and head motions based on lightweight networks. First, a humanoid robot that can change the state of its facial organs and neck through servo displacement is developed to achieve the mirrored behavior of facial expressions and head motions. Second, to overcome the high latency caused by deep learning models running in embedded devices, a lightweight deep learning network is constructed for detecting facial feature points, which can reduce model size and improve running speed without affecting the performance of the model. Finally, a mapping relationship of 68 facial feature points to optimal servo displacements is established to realize the mirrored behavior from human beings to humanoid robots. The experimental results show that the facial feature point recognition method based on the lightweight model performs better than other state-of-the-art methods, and our head motion tracking method can maintain high accuracy compared with the gold standard optical motion capture system NOKOV. Overall, our method ensures the accurate and real-time generation of robot mirrored behavior and has a certain reference value for the efficient and natural interaction between humans and robots.

IRWoZ: Constructing an Industrial Robot Wizard-of-OZ Dialoguing Dataset

Enabling a flexible and natural human-robot interaction (HRI) for industrial robots is a critical yet challenging task that can be facilitated by the use of conversational artificial intelligence (AI). Prior research has concentrated on strengthening interactions through the deployment of social robots, while disregarding the capabilities required to boost the flexibility and user experience associated with human-robot collaboration (HRC) on manufacturing tasks. One of the main challenges is the lack of publicly available industrial-oriented dialogue datasets for the training of conversational AI. In this work, we present an industrial robot wizard-of-Oz dialoguing dataste (IRWoZ) focused on enabling HRC in manufacturing tasks. The dataset covers four domains: assembly, transportation, position, and relocation. It is created using the Wizard-of-Oz technique to be less noisy. We manually constructed, annotated and validated dialogue segments (e.g., intentions, slots, annotations), as well as the responses. Building upon the proposed dataset, we benchmark it on the state-of-the-art (SoTA) language models, generative pre-trained (GPT-2) models, on dialogue state tracking and response generation tasks. We expect that the IRWoZ dataset will facilitate exciting ongoing dialogue research and we provide it freely accessible at https://github.com/lcroy/ToD4IR/tree/main/dataset.

A Mobile Natural Human–Robot Interaction Method for Virtual Chinese Acupuncture

Most of the existing virtual acupuncture methods cannot allow operators to interact naturally in a large area. To address this problem, this article proposes a mobile natural human–robot interaction interface for virtual Chinese acupuncture teaching, including an automatic hand tracking method and an acupuncture interaction method combining vision and force feedback. The automatic hand tracking method can automatically track and capture the operator’s hand movements, allowing the operator to perform acupuncture in a large area. The acupuncture interaction method combining vision and force reproduces the real acupuncture operation mode. The proposed interaction method provides the operator with visual feedback and force feedback without limitation, which improves the naturalness and authenticity of the operation. Through a series of experiments, the proposed method was proven to be more natural and efficient in the virtual acupuncture teaching scene from the point view of interaction. The operators achieve a stronger sense of immersion using this method.

Noise-tolerant zeroing neurodynamic algorithm for upper limb motion intention-based human–robot interaction control in non-ideal conditions

Human–robot interaction control, a crucial technology in industrial manufacturing and rehabilitation robot, has been widely explored in recent years. However, there are still some problems in the existing strategy need to be solved for the rehabilitation robot: First, the human–robot confrontation (HRC) in motion control affects the information interaction. Second, in non-ideal conditions, the safety and accuracy of the control system will be disturbed by external unknown factors. Third, the existing continuous-time controllers are hard to be applied to hardware platforms. In this paper, an upper limb-exoskeleton coupling dynamics model based on human active motion intention is established to overcome HRC, and promote active and natural human–robot interaction control. Specifically, the Elman neural network (ENN) model is devised and utilized to estimate human active motion intention using the surface electromyography (sEMG) signals. On the basis of human active motion intention, the discrete-time controller predicated on noise-tolerant zeroing neurodynamic model (termed as NTZND controller) is proposed for constructing the discrete-time human–robot interaction control system (DTHRICS) under non-ideal conditions. In addition, the zeroing neurodynamic (ZND) controller, the gradient neural network (GNN) and proportional–derivative (PD) controllers are designed for comparisons. Theoretical analyses demonstrate that the 0-stability, consistency and convergence of the designed DTHRICS. The performance indicators root mean square errors (RMSEs) and coefficient of determination (R2) of the DTHRICS with different controllers are analyzed and compared under ideal and non-ideal conditions. Ultimately, numerical results illustrate that the NTZND-based DTHRICS (with the lowest performance RMSEs →0 and the highest performance R2→1, p<0.05) is significantly superior to the DTHRICS with the ZND, the GNN and the PD controllers in terms of control accuracy, noise-tolerant property and real time performance, while utilized for upper limb multi-joint motion control under non-ideal conditions.

Bilateral personalized dialogue generation with contrastive learning

Generating personalized responses is one of the major challenges in natural human-robot interaction. Current researches in this field mainly focus on generating responses consistent with the robot's pre-assigned persona, while ignoring the user's persona. Such responses may be inappropriate or even offensive, which may lead to the bad user experience. Therefore, we propose a Bilateral Personalized Dialogue Generation (BPDG) method for dyadic conversation, which integrates user and robot personas into dialogue generation via designing a dynamic persona-aware fusion method. To bridge the gap between the learning objective function and evaluation metrics, the Conditional Mutual Information Maximum (CMIM) criterion is adopted with contrastive learning to select the proper response from the generated candidates. Moreover, a bilateral persona accuracy metric is designed to measure the degree of bilateral personalization. Experimental results demonstrate that, compared with several state-of-the-art methods, the final results of the proposed method are more personalized and consistent with bilateral personas in terms of both automatic and manual evaluations.

A Bio-Inspired Endogenous Attention-Based Architecture for a Social Robot.

A robust perception system is crucial for natural human–robot interaction. An essential capability of these systems is to provide a rich representation of the robot’s environment, typically using multiple sensory sources. Moreover, this information allows the robot to react to both external stimuli and user responses. The novel contribution of this paper is the development of a perception architecture, which was based on the bio-inspired concept of endogenous attention being integrated into a real social robot. In this paper, the architecture is defined at a theoretical level to provide insights into the underlying bio-inspired mechanisms and at a practical level to integrate and test the architecture within the complete architecture of a robot. We also defined mechanisms to establish the most salient stimulus for the detection or task in question. Furthermore, the attention-based architecture uses information from the robot’s decision-making system to produce user responses and robot decisions. Finally, this paper also presents the preliminary test results from the integration of this architecture into a real social robot.

Generalized Behavior Framework for Mobile Robots Teaming With Humans in Harsh Environments.

Industrial contexts, typically characterized by highly unstructured environments, where task sequences are difficult to hard-code and unforeseen events occur daily (e.g., oil and gas, energy generation, aeronautics) cannot completely rely upon automation to substitute the human dexterity and judgment skills. Robots operating in these conditions have the common requirement of being able to deploy appropriate behaviours in highly dynamic and unpredictable environments, while aiming to achieve a more natural human-robot interaction and a broad range of acceptability in providing useful and efficient services. The goal of this paper is to introduce a deliberative framework able to acquire, reuse and instantiate a collection of behaviours that promote an extension of the autonomy periods of mobile robotic platforms, with a focus on maintenance, repairing and overhaul applications. Behavior trees are employed to design the robotic system’s high-level deliberative intelligence, which integrates: social behaviors, aiming to capture the human’s emotional state and intention; the ability to either perform or support various process tasks; seamless planning and execution of human-robot shared work plans. In particular, the modularity, reactiveness and deliberation capacity that characterize the behaviour tree formalism are leveraged to interpret the human’s health and cognitive load for supporting her/him, and to complete a shared mission by collaboration or complete take-over. By enabling mobile robotic platforms to take-over risky jobs which the human cannot, should not or do not want to perform the proposed framework bears high potential to significantly improve the safety, productivity and efficiency in harsh working environments.

Living skin on a robot

•The method for generating seamless coverage of 3D objects with skin equivalent •Three-joint robotic finger covered with living skin equivalent •Wound repair of a robotic finger covered with dermis equivalent Humanoids are robots created with human forms or characteristics; these robots also have the potential to seamlessly interact with human beings. By replicating the appearances and functions (e.g., self-healing) of human beings, humanoids have the potential to establish more harmonic and natural human-robot interactions. Here, we propose the use of skin equivalent, a living skin model consisting of cells and extracellular matrix, as a human-like and self-healing coverage material for robots. We fabricated a three-joint robotic finger covered with skin equivalent by developing a method to cover three-dimensional objects with skin equivalent. Furthermore, inspired by the medical treatment of deeply burned skin using grafted hydrogels, we demonstrated wound repair of a dermis equivalent covering a robotic finger by culturing the wounded tissue grafted with a collagen sheet. With the above results, this research shows the potential of using skin equivalent as human-like and self-healing coverage material for robots. Humanoids are robots created with human forms or characteristics; these robots also have the potential to seamlessly interact with human beings. By replicating the appearances and functions (e.g., self-healing) of human beings, humanoids have the potential to establish more harmonic and natural human-robot interactions. Here, we propose the use of skin equivalent, a living skin model consisting of cells and extracellular matrix, as a human-like and self-healing coverage material for robots. We fabricated a three-joint robotic finger covered with skin equivalent by developing a method to cover three-dimensional objects with skin equivalent. Furthermore, inspired by the medical treatment of deeply burned skin using grafted hydrogels, we demonstrated wound repair of a dermis equivalent covering a robotic finger by culturing the wounded tissue grafted with a collagen sheet. With the above results, this research shows the potential of using skin equivalent as human-like and self-healing coverage material for robots.

Touch Gesture Recognition Using Spatiotemporal Fusion Features

The touch gesture is one of the most essential and effective means to transfer affective feelings and intents in humans’ communication. For an intelligent agent or a robot, the ability to automatically detect and recognize human touch can realize efficient and natural human–robot interaction. To this end, a novel spatiotemporal fusion feature extraction method is proposed for touch gesture classification tasks. The proposed method extracts time-frequency features from wavelet coefficients based on discrete wavelet transforms. Then, the feature array of space and frequency bands is constructed to extract the spatiotemporal fusion features. A publicly available touch gesture dataset called CoST is used to perform the touch gesture recognition. The recognition result of 14 gesture classes using a user-independent model yields an accuracy of up to 64.17%. Experimental results show that this method outperforms the state-of-the-art ones and that the spatiotemporal fusion features effectively boost the performance of touch gesture recognition.