Human-robot Interaction System Research Articles

Live human-robot interactive public demonstrations with automatic emotion and personality prediction.

Communication with humans is a multi-faceted phenomenon where the emotions, personality and non-verbal behaviours, as well as the verbal behaviours, play a significant role, and human-robot interaction (HRI) technologies should respect this complexity to achieve efficient and seamless communication. In this paper, we describe the design and execution of five public demonstrations made with two HRI systems that aimed at automatically sensing and analysing human participants' non-verbal behaviour and predicting their facial action units, facial expressions and personality in real time while they interacted with a small humanoid robot. We describe an overview of the challenges faced together with the lessons learned from those demonstrations in order to better inform the science and engineering fields to design and build better robots with more purposeful interaction capabilities. This article is part of the theme issue 'From social brains to social robots: applying neurocognitive insights to human-robot interaction'.

Multi-task emotion communication system with dynamic resource allocations

The research of long-distance emotion communication and interaction without time and space constraints is an important area in human-robot interaction (HRI) systems. Although many methods of emotion recognition have been studied for analyzing various emotion signals, the resource allocation of transmission for emotion communication signals of many pairs of users has not been fully considered nor solved at the same time. This paper proposes a new multi-task emotion communication system (MEmSys), where the transmission resources allocation issue is considered. Specifically, we firstly establish the architecture of MEmSys, and the entire emotion interaction process of the proposed system is introduced. By analyzing fairness and urgency of different tasks, the mathematical expressions of the minimum task transmission rates for all user pairs are derived. Then, a dynamic optimal resource allocation scheme is presented to maximize the sum of the task transmission rates in the proposed system. Moreover, simulation experiment results and performance analyses show that the resource utilization ratio of the proposed allocation scheme for multiple user pairs is significantly improved compared to the single user pair system. Finally, future works are discussed to provide insights for our next research.

Dual-Hand Detection for Human–Robot Interaction by a Parallel Network Based on Hand Detection and Body Pose Estimation

In this paper, a parallel network based on hand detection and body pose estimation is proposed to detect and distinguish human's right and left hands. The network is employed for human–robot interaction (HRI) based on hand gestures. This method fully uses hand feature information and hand information in the human body structure. One channel in the network uses a ResNet-Inception-Single Shot MultiBox Detector to extract hand feature information for human's hand detection. The other channel estimates human body pose first and then estimates the positions of the left and right hands using the forward kinematic tree of the human skeleton structure. Thereafter, the results of the two channels are fused. In the fusion module, the human body structure can be utilized to correct hand detection results and distinguish between the right and left hands. Experimental results verify that the parallel deep neural network can effectively improve the accuracy of hand detection and distinguish between the right and left hands effectively. This method is also used for the hand-gesture-based interaction between astronauts and an astronaut assistant robot. Our method can be suitably used in this HRI system.

Task-based Programming and Sequence Planning for Human-Robot Collaborative Assembly

The paper proposes a task-based programming method to support assembly processes in a human-robot collaborative workcell. The method for assembly sequence generation is implemented as a robot program homebuilt by composing pre-programmed elementary working steps. Developed system can support human operator as an interactive tool for collaboration with robot. Robot program and developed software are used to interface the robot with the human during the execution of the assembly tasks starting from single mechanical parts. The proposed Human Robot Interaction System (HRIS) performs an assembly job with the contemporary collaborative work of robots and humans that share tasks in the same workspace. The system is tested on a case study made on purpose and constituted by several different flanges mounted on a support in a variety of configurations. Thus, the method is general and can be easily adapted to a multiplicity of assembly tasks.

Development of a sociotechnical planning system for human-robot interaction in assembly systems focusing on small and medium-sized enterprises

As part of a research project a holistic planning system for human-robot interaction (HRI) is developed, focusing on the applicability for small and medium-sized enterprises (SMEs). This paper details several important aspects of the planning system, including an easy identification method of suitable processes for HRI. A key element is the work allocation between human workers and robots. The allocation considers e.g. technical feasibility, cycle time as well as personnel deployment and can easily be carried out by SMEs. Based on a simulation, a comprehensible scenario evaluation is developed, focusing primarily on economic aspects but also including ergonomic and safety considerations.

Enhanced Transparency for Physical Human-Robot Interaction Using Human Hand Impedance Compensation

In a physical human-robot interaction (pHRi) system, improving transparency that allows humans to move as if there is no robot is a challenging topic. In general pHRi, usage of the multiaxial force sensor for robot control is the norm. However, the signal measured from the force sensor contains not only the force applied to the robot by the human motion intention but also the influence of the natural force feedback between the robot and the human hand produced by the robot motion. Therefore, in order to improve the transparency, it is necessary to characterize the dynamics of the human hand as well as the dynamics of the robot. In this paper, an algorithm to improve the transparency is proposed. The proposed algorithm uses only a multiaxial force sensor and compensates the natural force feedback by using the human hand impedance. And it further improves the transparency of the pHRi system, which has a limitation adjusting admittance parameters. In order to verify the algorithm, a device that can measure the impedance of the human hand is introduced, and a system model analysis and experiments using the hydraulic upper limb exoskeleton are carried out.

Synthesizing Robot Programs with Interactive Tutor Mode

With the rapid development of the robotic industry, domestic robots have become increasingly popular. As domestic robots are expected to be personal assistants, it is important to develop a natural language-based human-robot interactive system for end-users who do not necessarily have much programming knowledge. To build such a system, we developed an interactive tutoring framework, named “Holert”, which can translate task descriptions in natural language to machine-interpretable logical forms automatically. Compared to previous works, Holert allows users to teach the robot by further explaining their intentions in an interactive tutor mode. Furthermore, Holert introduces a semantic dependency model to enable the robot to “understand” similar task descriptions. We have deployed Holert on an open-source robot platform, Turtlebot 2. Experimental results show that the system accuracy could be significantly improved by 163.9% with the support of the tutor mode. This system is also efficient. Even the longest task session with 10 sentences can be handled within 0.7 s.

Development, Dynamic Modeling, and Multi-Modal Control of a Therapeutic Exoskeleton for Upper Limb Rehabilitation Training.

Robot-assisted training is a promising technology in clinical rehabilitation providing effective treatment to the patients with motor disability. In this paper, a multi-modal control strategy for a therapeutic upper limb exoskeleton is proposed to assist the disabled persons perform patient-passive training and patient-cooperative training. A comprehensive overview of the exoskeleton with seven actuated degrees of freedom is introduced. The dynamic modeling and parameters identification strategies of the human-robot interaction system are analyzed. Moreover, an adaptive sliding mode controller with disturbance observer (ASMCDO) is developed to ensure the position control accuracy in patient-passive training. A cascade-proportional-integral-derivative (CPID)-based impedance controller with graphical game-like interface is designed to improve interaction compliance and motivate the active participation of patients in patient-cooperative training. Three typical experiments are conducted to verify the feasibility of the proposed control strategy, including the trajectory tracking experiments, the trajectory tracking experiments with impedance adjustment, and the intention-based training experiments. The experimental results suggest that the tracking error of ASMCDO controller is smaller than that of terminal sliding mode controller. By optimally changing the impedance parameters of CPID-based impedance controller, the training intensity can be adjusted to meet the requirement of different patients.

Design of an Interactive Control System for a Multisection Continuum Robot

Continuum robots exhibit unique advantages of flexibility and maneuverability in an unstructured environment. This paper presents the development of a human-robot interactive control system that includes two operational modes for controlling end-effector position and the entire robot shape with image feedback. To control the end-effector position, all the desired positions of the end effector fall within a reachable workspace constructed with an approximate boundary. To control the robot shape, a shape correspondence between the robot shape and the desired curve is determined to assist in examining whether the robot can follow the desired path with an acceptable collision with the environment. This correspondence can also assist users in robot motion planning. The proposed interactive control system ensures that the robot end effector follows the desired path, while the robot shape is regulated based on visual feedback. Experiments are performed to demonstrate the effectiveness of the proposed approach.

Three-Layer Weighted Fuzzy Support Vector Regression for Emotional Intention Understanding in Human–Robot Interaction

A three-layer weighted fuzzy support vector regression (TLWFSVR) model is proposed for understanding human intention, and it is based on the emotion-identification information in human–robot interaction. The TLWFSVR model consists of three layers, including adjusted weighted kernel fuzzy c -means for data clustering, fuzzy support vector regressions (FSVR) for information understanding, and weighted fusion for intention understanding. It aims to guarantee the quick convergence and satisfactory performance of the local FSVR via adjusting the weights of each feature in each cluster, in such a way that importance of different emotion-identification information is represented. Moreover, smooth human-oriented interaction can be obtained by endowing robot with human intention understanding capability. Experimental results show that the proposed TLWFSVR model obtains higher intention understanding accuracy and less computational time than that of two-layer fuzzy support vector regression, support vector regression, and back propagation neural network (BPNN), respectively. Additionally, the preliminary application experiments are performed in the developing human–robot interaction system, called emotional social robot system, where 12 volunteers and 2 mobile robots experience a scenario of “drinking at a bar.” Application results indicate that the bartender robot is able to understand customers’ order intentions.

A real-time and unsupervised face re-identification system for human-robot interaction

In the context of Human-Robot Interaction (HRI), face Re-Identification (face Re-ID) aims to verify if certain detected faces have already been observed by robots. The ability of distinguishing between different users is crucial in social robots as it will enable the robot to tailor the interaction strategy toward the users’ individual preferences. So far face recognition research has achieved great success, however little attention has been paid to the realistic applications of Face Re-ID in social robots. In this paper, we present an effective and unsupervised face Re-ID system which simultaneously re-identifies multiple faces for HRI. This Re-ID system employs Deep Convolutional Neural Networks to extract features, and an online clustering algorithm to determine the face's ID. Its performance is evaluated on two datasets: the TERESA video dataset collected by the TERESA robot, and the YouTube Face Dataset (YTF Dataset). We demonstrate that the optimised combination of techniques achieves an overall 93.55% accuracy on TERESA dataset and an overall 90.41% accuracy on YTF dataset. We have implemented the proposed method into a software module in the HCI^2 Framework [1] for it to be further integrated into the TERESA robot [2], and has achieved real-time performance at 10–26 Frames per second.

Interface Design of a Physical Human–Robot Interaction System for Human Impedance Adaptive Skill Transfer

It has been established that the transfer of human adaptive impedance is of great significance for physical human–robot interaction (pHRI). By processing the electromyography (EMG) signals collected from human muscles, the limb impedance could be extracted and transferred to robots. The existing impedance transfer interfaces rely only on visual feedback and, thus, may be insufficient for skill transfer in a sophisticated environment. In this paper, physical haptic feedback mechanism is introduced to result in muscle activity that would generate EMG signals in a natural manner, in order to achieve intuitive human impedance transfer through a designed coupling interface. Relevant processing methods are integrated into the system, including the spectral collaborative representation-based classifications method used for hand motion recognition; fast smooth envelop and dimensionality reduction algorithm for arm endpoint stiffness estimation. The tutor’s arm endpoint motion trajectory is directly transferred to the robot by the designed coupling module without the restriction of hands. Haptic feedback is provided to the human tutor according to skill learning performance to enhance the teaching experience. The interface has been experimentally tested by a plugging-in task and a cutting task. Compared with the existing interfaces, the developed one has shown a better performance. Note to Practitioners —This paper is motivated by the limited performance of skill transfer in the existing human–robot interfaces. Conventional robots perform tasks independently without interaction with humans. However, the new generation of robots with the characteristics, such as flexibility and compliance, become more involved in interacting with humans. Thus, advanced human robot interfaces are required to enable robots to learn human manipulation skills. In this paper, we propose a novel interface for human impedance adaptive skill transfer in a natural and intuitive manner. The developed interface has the following functionalities: 1) it transfers human arm impedance adaptive motion to the robot intuitively; 2) it senses human motion signals that are decoded into human hand gesture and arm endpoint stiffness that ia employed for natural human robot interaction; and 3) it provides human tutor haptic feedback for enhanced teaching experience. The interface can be potentially used in pHRI, teleoperation, human motor training systems, etc.

Precision Cascade Force Control of Multi-DOF Hydraulic Leg Exoskeleton

Hydraulic exoskeleton is a kind of the human robot interaction system, which can augment the human performance in the application of heavy load carrying. Due to the existence of complicated multi-joint nonlinear dynamics and various uncertainties, traditional robust control of these systems is hard to be realized in most of the practical research. In this paper, an adaptive robust cascade force controller is proposed for 3-DOF hydraulic leg exoskeleton to achieve accurate tracking of human motion. Specifically, the control strategy includes two levels. The desired joint positions, which can be assumed as the human motion intent as well, are generated in the high-level by attenuating the integral of human–machine interaction force. And in the low-level, an observer-based MIMO motion controller is developed for 3-DOF dynamics to track the generated human motion intent accurately. Adaptive robust control algorithms are developed in both control levels to address the strongly coupled high-order dynamics under parametric uncertainties and uncertain disturbances. Comparative simulations show that the human–machine interaction force can be attenuated exactly and robust performance to various uncertainties can be guaranteed, validating the effectiveness of the proposed scheme.

A novel human-robot controlling approach inspired by the processes of muscle memory and conditioned reflex

PurposeThe purpose of this paper is to provide a novel training-responding controlling approach for human–robot interaction. The approach is inspired by the processes of muscle memory and conditioned reflex. The approach is significant for dealing with the problems of robot’s redundant movements and operator’s fatigue in human–robot interaction system.Design/methodology/approachThis paper presented a directional double clustering algorithm (DDCA) to achieve the training process. The DDCA ensured that the initial clustering centers uniformly distributed in every desired cluster. A minimal resource allocation network was used to construct a memory responding algorithm (MRA). When the human–robot interaction system needed to carry out a task for more than one time, the desired movements of the robot were given by the MRA without repeated training. Experimentally demonstrated results showed the proposed training-responding controlling approach could successfully accomplish human–robot interaction tasks.FindingsThe training-responding controlling approach improved the robustness and reliability of the human–robot interaction system, which presented a novel controlling method for the operator.Practical implicationsThis approach has significant commercial applications, as a means of controlling for human–robot interaction could serve to point to the desired target and arrive at the appointed positions in industrial and household environment.Originality/valueThis work presented a novel training-responding human-robot controlling method. The human-robot controlling method dealt with the problems of robot’s redundant movements and operator’s fatigue. To the authors’ knowledge, the working processes of muscle memory and conditioned reflex have not been reported to apply to human-robot controlling.

Special Issue on Human-Robot Interaction Systems II

Six years have passed since the publication of our Special Issue on Human-Robot Interaction Systems in 2011. Since then, artificial intelligence and robotics have developed rapidly, and the opportunities for human beings and robots to work together have increased. The objective of this special issue’s twelve articles is to activate and expand high-quality research.In the first article, Y. Tamura, T. Akashi, and H. Osumi propose a computational model of robot’s gaze. In the second article, S. Hoshino and K. Uchida propose an interactive motion planner for robot navigation in dynamic environments. In the third article, T. Iio, Y. Yoshikawa, and H. Ishiguro develop a conversational robotic system based on human response. In the fourth article, K. Sakai, F. Dalla Libera, Y. Yoshikawa, and H. Ishiguro propose a method for generating bystander robots’ actions that is based on an analysis of the relative probabilities of human responses to robot actions. In the fifth article, T. Matsumaru and M. Narita present a newly developed support system for learning calligraphy strokes. In the sixth article, E. Tamura, Y. Yamashita, T. Yamashita, E. Sato-Shimokawara, and T. Yamaguchi present a method of driving a car simply by gesturing. In the seventh article, A. Kurosu and T. Hashimoto develop an eye robot with two degrees of freedom. It is intended for use as a communication robot. In the eighth article, T. Hashimoto, Y. Munakata, R. Yamanaka, and A. Kurosu report on a method for retrieving episodic memories. In the ninth article, Y. Nishikawa, Y. Kagawa, and A. Okazaki develop a spiral movement robot for inpatients. In the tenth article, Y. Umesawa, K. Doi, and H. Fujimoto develop an interface device that creates kinaesthetic illusions by inducing vibrations in muscle tendons, vibrations that coordinate with dual-joint movements. In the eleventh article, R. Horio, N. Uchiyama, and S. Sano propose a human-operated biped robot for transporting objects over rough terrain or up steps. In the closing contribution, T. Sakuraba, N. Uchiyama, S. Sano, and T. Sakaguchi present the design of a spring-based regenerative brake, and they verify its effectiveness by driving a system that uses it.We thank the referees for their comprehensive reviews and the staff members of Fuji Technology Press, Ltd. for their encouragement and advice.

Force-Mode Control of Rotary Series Elastic Actuators in a Lower Extremity Exoskeleton Using Model-Inverse Time Delay Control

For a physical human-robot interaction (pHRI) system such as an exoskeleton, it has been an important issue to control the interaction force between the user and the actuators because it directly determines the accuracy of the realized impedance. However, the accurate force control of the exoskeleton system has not been fully achieved due to the difficulties caused by uncertainties from pHRI. To overcome this problem, in this study, the series elastic actuator (SEA) was operated by a time delay control (TDC), which directly compensated the uncertainties without an accurate model of them, realizing the designed model dynamics in the actuator. However, in spite of the successful application of TDC, the delays between the desired and model dynamics severely degraded the force control performance of the SEA. Thus, a new strategy named model-inverse time delay control was proposed by introducing a virtual reference predicted by the inverse of model dynamics in TDC. The proposed method significantly improved performance of the force control, and it was verified by demonstrating zero impedance control with pHRI, free oscillation of a pendulum, and force control in the exoskeleton.

Toward Multimodal Human-Robot Interaction to Enhance Active Participation of Users in Gait Rehabilitation.

Robotic exoskeletons for physical rehabilitation have been utilized for retraining patients suffering from paraplegia and enhancing motor recovery in recent years. However, users are not voluntarily involved in most systems. This paper aims to develop a locomotion trainer with multiple gait patterns, which can be controlled by the active motion intention of users. A multimodal human-robot interaction (HRI) system is established to enhance subject's active participation during gait rehabilitation, which includes cognitive HRI (cHRI) and physical HRI (pHRI). The cHRI adopts brain-computer interface based on steady-state visual evoked potential. The pHRI is realized via admittance control based on electromyography. A central pattern generator is utilized to produce rhythmic and continuous lower joint trajectories, and its state variables are regulated by cHRI and pHRI. A custom-made leg exoskeleton prototype with the proposed multimodal HRI is tested on healthy subjects and stroke patients. The results show that voluntary and active participation can be effectively involved to achieve various assistive gait patterns.

Understanding human intention by connecting perception and action learning in artificial agents

To develop an advanced human–robot interaction system, it is important to first understand how human beings learn to perceive, think, and act in an ever-changing world. In this paper, we propose an intention understanding system that uses an Object Augmented-Supervised Multiple Timescale Recurrent Neural Network (OA-SMTRNN) and demonstrate the effects of perception–action connected learning in an artificial agent, which is inspired by psychological and neurological phenomena in humans. We believe that action and perception are not isolated processes in human mental development, and argue that these psychological and neurological interactions can be replicated in a human–machine scenario. The proposed OA-SMTRNN consists of perception and action modules and their connection, which are constructed of supervised multiple timescale recurrent neural networks and the deep auto-encoder, respectively, and connects their perception and action for understanding human intention. Our experimental results show the effects of perception–action connected learning, and demonstrate that robots can understand human intention with OA-SMTRNN through perception–action connected learning.

分人モデルの構築を目指した強化学習型ベイジアンネットワークの提案

A Japanese novelist Keiichiro Hirano proposes a concept of dividual to explain interaction between humans. This paper models the dividual using three machine learning techniques and develops a new human-robot interaction system using the proposed dividual model. The system consists of two parts; dividual identification and action selection. In the dividual identification, the system recognizes the person who interacts with and learns about him/her through the interaction using self-organizing map. In the action selection, the system selects an action using a Bayesian network whose conditional probability table is updated by Q-learning. Through computer simulations, it is verified that the proposed dividual model can select an appropriate dividual for the specific person to be interacted with. It is also shown that the dividual model can suggest an appropriate topic according to the specific person.

Adaptive Admittance Control for Human–Robot Interaction Using Model Reference Design and Adaptive Inverse Filtering

Corobotics involves humans and robots working collaboratively as a team. This requires physical human-robot interaction (pHRI) systems that can adapt to the preferences of different humans and have good robustness and stability properties. In this brief, a new inner-loop/outer-loop robot controller formulation is developed that makes pHRI robust to changes in both corobot and human user. First, an inner-loop controller with guaranteed robustness and stability causes a robot to behave like a prescribed admittance model. Second, an outer-loop controller tunes the admittance model so that the robot system assists humans with varying levels of skill to achieve task-specific objectives. This design technique cleanly separates robot-specific control from task performance objectives and allows formal inclusion in an outer design of both an ideal task model and unknown human operator dynamics. Experimental results with the controllers running on a PR2 robot demonstrate the effectiveness of this approach.