Human-robot Interaction Experiments Research Articles

Task Space Compliant Control and Six-Dimensional Force Regulation Toward Automated Robotic Ultrasound Imaging

<italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Objective:</i> We propose a general control framework for task space compliant motion and six-dimensional (6-D) force regulation towards automated robotic ultrasound (US) imaging. The framework endows a position-controlled robotic manipulator with the capability of accurate compliant motion in free space and accurate force control in motion-constrained environment. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Methods:</i> An intuitive six degree-of-freedom (6-DoF) admittance control model expressed in an arbitrary Cartesian body frame is mathematically derived with closed-form task space error mapping. Its practical implementation on widely-used collaborative manipulators is proposed to achieve full task space compliant behaviors and accurate 6-D force control. A hybrid control law is presented to achieve good motion accuracy in free space and improved coupled stability in motion-constrained environment. The coupled model of physical human-robot interaction is established and the reason for the improved coupled stability is analyzed through simulation. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Results:</i> Evaluation experiments on the proposed control framework were performed to show the effectiveness. The mean error of compliant trajectory following was less than 0.30 mm in free space. The mean relative force and moment control accuracy in three orthogonal directions was better than 0.5% and 0.8%, respectively. The improved coupled stability under the same model parameters was also confirmed by human-robot interaction experiments. Finally, an automated robotic US imaging experiment on a human volunteer in a real clinical scenario was carried out to show the potential application of our proposed framework. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Conclusion:</i> Experimental results have shown the advantages of the control framework, including satisfied force control accuracy, high accuracy of compliant motion, improved coupled stability, and system effectiveness on a human volunteer. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —This paper was motivated by the increasing needs of automated ultrasound (US) scanning for both diagnostic and interventional purpose. Clinical sonographers suffer from repeated workload when performing diagnostic US imaging, which could benefit from automated robotic scanning. Robotic US imaging involves physical interaction between the robot end-effector (i.e., US transducer) and the human body. The dynamics of the interaction is regulated by the control law to guarantee the contact of the US transducer and the safety of the procedure. Most existing works have focused on regulating in-plane contact force in terms of the position without considering the compliance in other dimensions. However, it is not a trivial work to extend the positional compliance to six degree-of-freedom (6-DoF) compliance. As the prevalence of low-cost collaborative robotic arms in medical scenarios, how to perform 6-DoF compliant trajectory following and accurate six-dimensional (6-D) force control on these robotic arms becomes increasingly important. This paper gives a complete general solution to achieve 6-DoF compliant control and 6-D force regulation with accurate kinematics on a position-controlled robotic arm. A hybrid control law is proposed to switch the government of “instantaneous model” and “theoretical model” to achieve compliant motion accuracy in free space and improved coupled stability in motion-constrained environment. No expensive torque sensors and torque control interface are required. And no prior geometric knowledge about the scanning object is needed. We have demonstrated the application for robotic US imaging in a real clinical scenario.

A Dual-Arm Robot Cooperation Framework Based on a Nonlinear Model Predictive Cooperative Control

Dual-arm robots show better task adaptability but face more constraints than single-arm robots. Inspired by human cooperation options, a nonlinear model predictive cooperative control (NMPCC) coupled with a cooperative index is proposed in this article. By adjusting the cooperative index, dual-arm cooperation can be classified into four modes: tracking, motion synchronization, impedance, and force priority. Thus, the cooperation operation problem is converted to a multiobjective optimization problem. Then, a nonlinear model predictive solver coupled with the ACADO toolkit is designed to solve the multiobjective optimization problem, where the robotic torque control input can be calculated in real time (less than 1 ms). The motion synchronization, cooperative transportation, and human dual-arm robot interaction experiments were conducted on dual Franka panda robots. Experiments reveal that the NMPCC control coupled with the cooperative index is easy to apply in dual-arm robots and can adapt to different complex manipulation tasks.

How do you do the things that you do? Ethological approach to the description of robot behaviour.

The detailed description of behaviour of the interacting parties is becoming more and more important in human-robot interaction (HRI), especially in social robotics (SR). With the rise in the number of publications, there is a substantial need for the objective and comprehensive description of implemented robot behaviours to ensure comparability and reproducibility of the studies. Ethograms and the meticulous analysis of behaviour was introduced long ago in animal behaviour research (cf. ethology). The adoption of this method in SR and HRI can ensure the desired clarity over robot behaviours, while also providing added benefits during robot development, behaviour modelling and analysis of HRI experiments. We provide an overview of the possible uses and advantages of ethograms in HRI, and propose a general framework for describing behaviour which can be adapted to the requirements of specific studies.

When Deep is not Enough: Towards Understanding Shallow and Continual Learning Models in Realistic Environmental Sound Classification for Robots

Although deep learning models are state-of-the-art models in audio classification, they fall short when applied in developmental robotic settings and human–robot interaction (HRI). The major drawback is that deep learning relies on supervised training with a large amount of data and annotations. In contrast, developmental learning strategies in human–robot interaction often deal with small-scale data acquired from HRI experiments and require the incremental addition of novel classes. Alternatively, shallow learning architectures that enable fast and yet robust learning are provided by simple distance metric-based learning and neural architectures implementing the reservoir computing paradigm. Similarly, continual learning algorithms receive more attention in the last years as they can integrate stable perceptual feature extraction using pre-trained deep learning models with open-set classification. As our research centers around reenacting the incremental learning of audio cues, we conducted a study on environmental sound classification using the iCaRL as well as the GDumb continual learning algorithms in comparison with a popular classifier in this domain, the kNN classifier, as well as employing an Echo State Network. We contrast our results with those obtained from a VGGish network that serves here as the performance upper bound that allows us to quantify the performance differences and to discuss current issues with continual learning in the audio domain. As only little is known about using shallow models or continual learning in the audio domain, we pass on additional techniques like data augmentation and create a simple experimental pipeline that is easy to reproduce. Although our selected algorithms are partially inferior in performance compared to the upper bound, our evaluation on three environmental sound datasets shows promising performance using continual learning for a subset of the DCASE2019 challenge dataset and the ESC10 dataset. As we do not address benchmarking in this paper, our study provides a good foundation for further research and computational improvements on shallow and continual learning models for robotic applications in the audio domain.

Designing Behaviors of Robots Based on the Artificial Emotion Expression Method in Human–Robot Interactions

How to express emotions through motion behaviors of robots (mainly for robotic arms) to achieve human–robot emotion interactions is the focus of this paper. An artificial emotion expression method that accords with human emotion that can deal with external stimuli and has the capability of emotion decision-making was proposed based on the motion behaviors of robot. Firstly, a three-dimensional emotion space was established based on the motion indexes (deviation coefficient, acceleration, and interval time). Then, an artificial emotion model, which was divided into three parts (the detection and processing of external events, the generation and modification of emotion response vectors, and the discretization of emotions) was established in the three-dimensional emotion space. Then emotion patterns (love, excited, happy, anxiety, hate) and emotion intensity were calculated based on the artificial emotion model in human–robot interaction experiments. Finally, the influence of motion behaviors of humanoid robot NAO on the emotion expression of experimenters was studied through human–robot emotion interaction experiments based on the emotion patterns and emotion intensity. The positive emotion patterns (love, excited, happy) and negative emotion patterns (anxiety, hate) of the experimenters were evaluated. The experimental results showed that the personalized emotion responses could be generated autonomously for external stimuli, and the change process of human emotions could be simulated effectively according to the established artificial emotion model. Furthermore, the experimenters could recognize the emotion patterns expressed by the robot according to the motion behaviors of the robot, and whether experimenters were familiar with robots did not influence the recognition of different emotion patterns.

A robot for overground physical human-robot interaction experiments.

Many anticipated physical human-robot interaction (pHRI) applications in the near future are overground tasks such as walking assistance. For investigating the biomechanics of human movement during pHRI, this work presents Ophrie, a novel interactive robot dedicated for physical interaction tasks with a human in overground settings. Unique design requirements for pHRI were considered in implementing the one-arm mobile robot, such as the low output impedance and the ability to apply small interaction forces. The robot can measure the human arm stiffness, an important physical quantity that can reveal human biomechanics during overground pHRI, while the human walks alongside the robot. This robot is anticipated to enable novel pHRI experiments and advance our understanding of intuitive and effective overground pHRI.

Gait rehabilitation training robot: A motion-intention recognition approach with safety and convenience

Motion-intention recognition is a vital prerequisite in active training when employing a gait rehabilitation training robot. To accurately recognize the motion intention of the elderly and the people with inconveniences in the human–robot interaction process, a novel approach of motion intention recognition with safety, accuracy, and convenience, including directional intention recognition (DIR) and speed intention recognition (SIR) is proposed in this paper. Firstly, the structures of the gait rehabilitation training robot and its motion-intention recognition system are illustrated. To ensure that the user walks in any desired direction safely and naturally, an improved distance-type fuzzy reasoning algorithm combined with a shake-intent filter and second-order optimization algorithm is proposed. It effectively eliminates the control error caused by body shaking and usage habits in human–robot interaction. Furthermore, by extracting from the pressure sensor data, a novel algorithm, taking advantage of the Gaussian probability density function (PDF)’s characteristics, is proposed for SIR, which does not increase the system complexity. Finally, a multi-directional fuzzy reasoning experiment and a human–robot interaction experiment are implemented. The results show that the algorithm can accurately recognize the motion direction intention and motion speed intention of people with weak motion capability, which also improves the safety and convenience of the interaction approach. The proposed method can be integrated into a walker with similar structures, and the whole system can be applied in hospitals, families, and other places for assisting the elderly and the disabled.

Coupled dynamic modeling and experimental validation of a collaborative industrial mobile manipulator with human-robot interaction

Dynamic modeling and analysis of a collaborative industrial mobile manipulator provide essential foundation and guidance for controlling it to achieve the desired physical interaction. However, it is a well-recognized challenge to precisely model the dynamics of a collaborative mobile manipulator due to its high-DOF (degree of freedom) structure, nonholonomic constraints, dynamic coupling between the manipulator and the mobile platform, and physical interaction with environments/humans. This paper presents a dynamic model of a new high-DOF nonholonomic collaborative mobile manipulator applied to physical human-robot collaboration scenarios, accounting for the dynamic coupling. A full Jacobian matrix introduces the dynamic coupling and is derived to solve the nonholonomic constraints on the velocity of the mobile platform. Besides, it facilitates the formulation of the dynamic model with independent generalized coordinates based on Lagrange equations. The comprehensive simulations with ADAMS are carried out along a pre-set trajectory to verify the coupled dynamic model. The dynamic modeling method is further validated by the human-robot interaction experiment. The simulation and testing results demonstrate the accuracy of the coupled dynamic model of the collaborative mobile manipulator.

An adaptive decision-making system supported on user preference predictions for human–robot interactive communication

Adapting to dynamic environments is essential for artificial agents, especially those aiming to communicate with people interactively. In this context, a social robot that adapts its behaviour to different users and proactively suggests their favourite activities may produce a more successful interaction. In this work, we describe how the autonomous decision-making system embedded in our social robot Mini can produce a personalised interactive communication experience by considering the preferences of the user the robot interacts with. We compared the performance of Top Label as Class and Ranking by Pairwise Comparison, two promising algorithms in the area, to find the one that best predicts the user preferences. Although both algorithms provide robust results in preference prediction, we decided to integrate Ranking by Pairwise Comparison since it provides better estimations. The method proposed in this contribution allows the autonomous decision-making system of the robot to work on different modes, balancing activity exploration with the selection of the favourite entertaining activities. The operation of the preference learning system is shown in three real case studies where the decision-making system works differently depending on the user the robot is facing. Then, we conducted a human–robot interaction experiment to investigate whether the robot users perceive the personalised selection of activities more appropriate than selecting the activities at random. The results show how the study participants found the personalised activity selection more appropriate, improving their likeability towards the robot and how intelligent they perceive the system. query Please check the edit made in the article title.

Caring for robots: How care comes to matter in human-machine interfacing.

Care robots promise to assist older people in an ageing society. This article investigates the socio-material conditions of care with robots by focusing on the usually invisible practices of human-machine interfacing. I define human-machine interfacing as the activities by roboticists and others to render interaction between robots and people possible in the first place. This includes, efforts to render prototypical arrangements of care 'robot-friendly'. In my video-assisted ethnography of human-robot interaction (HRI) experiments. I identify four types of interfacing practices, where care comes to matter: integrating the ephemeral entity that is 'a robot', helping it by way of mundane courtesies, making users 'fit' for interacting with it, and establishing corridors of interaction between the robot and people's bodies. I show that robots do not so much care for (older) people but rather, the other way around - people need to care for robots. Hence, care robots are not simply agents of care but also objects of care, rendering necessary a symmetrical analysis of human-machine interfacing. Furthermore, these practices do not merely reflect the prototypical state of the art in robotics. Rather, they indicate a more general mode of how robots and people interface. I argue that care with robots requires us to re-consider the exclusive focus on the human and at least complement it with care for the non-human and, incidentally, the robotic, too.

Human perception of intrinsically motivated autonomy in human-robot interaction

A challenge in using robots in human-inhabited environments is to design behavior that is engaging, yet robust to the perturbations induced by human interaction. Our idea is to imbue the robot with intrinsic motivation (IM) so that it can handle new situations and appears as a genuine social other to humans and thus be of more interest to a human interaction partner. Human-robot interaction (HRI) experiments mainly focus on scripted or teleoperated robots, that mimic characteristics such as IM to control isolated behavior factors. This article presents a “robotologist” study design that allows comparing autonomously generated behaviors with each other and, for the first time, evaluates the human perception of IM-based generated behavior in robots. We conducted a within-subjects user study ( N = 24) where participants interacted with a fully autonomous Sphero BB8 robot with different behavioral regimes: one realizing an adaptive, intrinsically motivated behavior and the other being reactive, but not adaptive. The robot and its behaviors are intentionally kept minimal to concentrate on the effect induced by IM. A quantitative analysis of post-interaction questionnaires showed a significantly higher perception of the dimension “Warmth” compared to the reactive baseline behavior. Warmth is considered a primary dimension for social attitude formation in human social cognition. A human perceived as warm (friendly, trustworthy) experiences more positive social interactions.

Study of Kinesthetic Negotiation Ability in Lightweight Comanipulative Decision-making Tasks: Design and Study of a Virtual Partner based on Human-human Interaction Observation

This article presents the results of an experiment on physical Human-Human Interaction (pHHI), where human dyads cooperate on a one-dimensional comanipulative task in a novel lightweight teleoperation setup. The results of this experiment show that humans are able to handle asymmetrical information about the task and solve conflicts using only the kinesthetic channel. Data from the pHHI experiment is used to design a virtual partner that can perform the task alongside a human. The virtual partner behavior is based on the observation that initiative is highly correlated to decision-making in our pHHI negotiation scenario. The virtual agent is then evaluated in a physical Human-Robot Interaction (pHRI) experiment. The results of the second experiment show that the virtual partner is able to perform the task without compromising the performances of the dyad and that a similar role distribution is observed in human-human and human-robot dyads. Moreover, the knowledge of the partner’s nature does not seem to influence the performances. The results obtained with the virtual partner are encouraging and could be used to design kinesthetic negotiation algorithms in pHRI settings. The main contributions of this article are: (1) supporting evidence of the possibility for humans to use the kinesthetic channel as mean of negotiation during physical Human-Human Interaction in the lightest known impedance negotiation tasks (with no virtual mass involved), (2) highlighting of the correlation between initiative and dyadic decision making, (3) design of a simple yet efficient virtual partner algorithm capable of realistic physical Human-Robot Interaction in the one-dimension tracking task used in the experimental setup. This design combines minimum jerk trajectory, switching role ability, decision criteria, and statistical parameters that are detailed in this article.

10 Years of Human-NAO Interaction Research: A Scoping Review.

The evolving field of human-robot interaction (HRI) necessitates that we better understand how social robots operate and interact with humans. This scoping review provides an overview of about 300 research works focusing on the use of the NAO robot from 2010 to 2020. This study presents one of the most extensive and inclusive pieces of evidence on the deployment of the humanoid NAO robot and its global reach. Unlike most reviews, we provide both qualitative and quantitative results regarding how NAO is being used and what has been achieved so far. We analyzed a wide range of theoretical, empirical, and technical contributions that provide multidimensional insights, such as general trends in terms of application, the robot capabilities, its input and output modalities of communication, and the human-robot interaction experiments that featured NAO (e.g. number and roles of participants, design, and the length of interaction). Lastly, we derive from the review some research gaps in current state-of-the-art and provide suggestions for the design of the next generation of social robots.

Artificial Eyes with Emotion and Light Responsive Pupils for Realistic Humanoid Robots

This study employs a novel 3D engineered robotic eye system with dielectric elastomer actuator (DEA) pupils and a 3D sculpted and colourised gelatin iris membrane to replicate the appearance and materiality of the human eye. A camera system for facial expression analysis (FEA) was installed in the left eye, and a photo-resistor for measuring light frequencies in the right. Unlike previous prototypes, this configuration permits the robotic eyes to respond to both light and emotion proximal to a human eye. A series of experiments were undertaken using a pupil tracking headset to monitor test subjects when observing positive and negative video stimuli. A second test measured pupil dilation ranges to high and low light frequencies using a high-powered artificial light. This data was converted into a series of algorithms for servomotor triangulation to control the photosensitive and emotive pupil dilation sequences. The robotic eyes were evaluated against the pupillometric data and video feeds of the human eyes to determine operational accuracy. Finally, the dilating robotic eye system was installed in a realistic humanoid robot (RHR) and comparatively evaluated in a human-robot interaction (HRI) experiment. The results of this study show that the robotic eyes can emulate the average pupil reflex of the human eye under typical light conditions and to positive and negative emotive stimuli. However, the results of the HRI experiment indicate that replicating natural eye contact behaviour was more significant than emulating pupil dilation.

Domain adversarial transfer for cross-domain and task-constrained grasp pose detection

Transferring the grasping skills learned from simulated environments to the real world is favorable for many robotic applications, in which the collecting and labeling processes of real-world visual grasping datasets are often expensive or even impractical. However, the models purely trained on simulated data are often difficult to generalize well to the unseen real world due to the domain gap between the training and testing data. In this paper, we propose a novel domain adversarial transfer network to narrow the domain gap for cross-domain and task-constrained grasp pose detection. Generative adversarial training is exploited to constrain the generator to produce simulation-like data for extracting the shared features with the joint distribution. We also propose to improve the backbone by extracting task-constrained grasp candidates and constructing the grasp candidate evaluator with a lightweight structure and an embedded recalibration technique. To validate the effectiveness and superiority of our proposed method, grasping performance evaluation and task-oriented human–robot interaction experiments were investigated. The experiment results indicate that the proposed method achieves state-of-the-art performance in these experimental settings. An average task-constrained grasping success rate of 83.3% without using any real-world labels for the task-oriented human–robot interaction experiment was achieved especially.

WoZ4U: An Open-Source Wizard-of-Oz Interface for Easy, Efficient and Robust HRI Experiments.

Wizard-of-Oz experiments play a vital role in Human-Robot Interaction (HRI), as they allow for quick and simple hypothesis testing. Still, a publicly available general tool to conduct such experiments is currently not available in the research community, and researchers often develop and implement their own tools, customized for each individual experiment. Besides being inefficient in terms of programming efforts, this also makes it harder for non-technical researchers to conduct Wizard-of-Oz experiments. In this paper, we present a general and easy-to-use tool for the Pepper robot, one of the most commonly used robots in this context. While we provide the concrete interface for Pepper robots only, the system architecture is independent of the type of robot and can be adapted for other robots. A configuration file, which saves experiment-specific parameters, enables a quick setup for reproducible and repeatable Wizard-of-Oz experiments. A central server provides a graphical interface via a browser while handling the mapping of user input to actions on the robot. In our interface, keyboard shortcuts may be assigned to phrases, gestures, and composite behaviors to simplify and speed up control of the robot. The interface is lightweight and independent of the operating system. Our initial tests confirm that the system is functional, flexible, and easy to use. The interface, including source code, is made commonly available, and we hope that it will be useful for researchers with any background who want to conduct HRI experiments.

Design of Hesitation Gestures for Nonverbal Human-Robot Negotiation of Conflicts

When the question of who should get access to a communal resource first is uncertain, people often negotiate via nonverbal communication to resolve the conflict. What should a robot be programmed to do when such conflicts arise in Human-Robot Interaction? The answer to this question varies depending on the context of the situation. Learning from how humans use hesitation gestures to negotiate a solution in such conflict situations, we present a human-inspired design of nonverbal hesitation gestures that can be used for Human-Robot Negotiation. We extracted characteristic features of such negotiative hesitations humans use, and subsequently designed a trajectory generator (Negotiative Hesitation Generator) that can re-create the features in robot responses to conflicts. Our human-subjects experiment demonstrates the efficacy of the designed robot behaviour against non-negotiative stopping behaviour of a robot. With positive results from our human-robot interaction experiment, we provide a validated trajectory generator with which one can explore the dynamics of human-robot nonverbal negotiation of resource conflicts.

Human Posture Prediction During Physical Human-Robot Interaction

When a human is interacting physically with a robot to accomplish a task, his/her posture is inevitably influenced by the robot movement. Since the human is not controllable, an active robot imposing a collaborative trajectory should predict the most likely human posture. This prediction should consider individual differences and preferences of movement execution, and it is necessary to evaluate the impact of the robot's action from an ergonomics standpoint. Here, we propose a method to predict, in probabilistic terms, the human postures of an individual for a given robot trajectory executed in a collaborative scenario. We formalize the problem as the prediction of the human joints velocity given the current posture and robot end-effector velocity. The key idea of our approach is to learn the distribution of the null space of the Jacobian and the weights of the weighted pseudo-inverse from demonstrated human movements: both carry information about human postural preferences, to leverage redundancy and ensure that the predicted posture will be coherent with the end-effector position. We validate our approach in a simulated toy problem and on two real human-robot interaction experiments where a human is physically interacting with a Franka robot.

Emergent Robotic Personality Traits via Agent-Based Simulation of Abstract Social Environments

This paper discusses the creation of an agent-based simulation model for interactive robotic faces, built based on data from physical human–robot interaction experiments, to explore hypotheses around how we might create emergent robotic personality traits, rather than pre-scripted ones based on programmatic rules. If an agent/robot can visually attend and behaviorally respond to social cues in its environment, and that environment varies, then idiosyncratic behavior that forms the basis of what we call a “personality” should theoretically be emergent. Here, we evaluate the stability of behavioral learning convergence in such social environments to test this idea. We conduct over 2000 separate simulations of an agent-based model in scaled-down, abstracted forms of the environment, each one representing an “experiment”, to see how different parameters interact to affect this process. Our findings suggest that there may be systematic dynamics in the learning patterns of an agent/robot in social environments, as well as significant interaction effects between the environmental setup and agent perceptual model. Furthermore, learning from deltas (Markovian approach) was more effective than only considering the current state space. We discuss the implications for HRI research, the design of interactive robotic faces, and the development of more robust theoretical frameworks of social interaction.

Experiments in Artificial Sociality

In this article, I explore how experiments with social robots enact and reconfigure more-than-human forms of sociality. I combine recent anthropological discussions of nonhuman sociality with Andy Pickering’s work on dances of agency (1993, 1995) and John Law’s method assemblages (2004) to show how human-robot interaction experiments enact open-ended and decentred configurations of entangling relations between humans and robots. I propose the concept of artificial sociality to capture both the ongoing enactments and multiple results of such experimental reconfigurations. Using these conceptual tools, I unpack the “curious robot experiment” from my ethnographic fieldwork in a Japanese robotics laboratory and compare the kinds of sociality produced in the two experimental conditions. I argue that the curious robot exemplifies what Pickering calls technologies of engagement (2018) by manifesting a form of artificial sociality that augments the unpredictability of dances of agency enacted in (re)configurations of entangling relations.