Human-robot Interaction Scenarios Research Articles

The Gaze Dialogue Model: Nonverbal Communication in HHI and HRI.

When humans interact with each other, eye gaze movements have to support motor control as well as communication. On the one hand, we need to fixate the task goal to retrieve visual information required for safe and precise action-execution. On the other hand, gaze movements fulfil the purpose of communication, both for reading the intention of our interaction partners, as well as to signal our action intentions to others. We study this Gaze Dialogue between two participants working on a collaborative task involving two types of actions: 1) individual action and 2) action-in-interaction. We recorded the eye-gaze data of both participants during the interaction sessions in order to build a computational model, the Gaze Dialogue, encoding the interplay of the eye movements during the dyadic interaction. The model also captures the correlation between the different gaze fixation points and the nature of the action. This knowledge is used to infer the type of action performed by an individual. We validated the model against the recorded eye-gaze behavior of one subject, taking the eye-gaze behavior of the other subject as the input. Finally, we used the model to design a humanoid robot controller that provides interpersonal gaze coordination in human-robot interaction scenarios. During the interaction, the robot is able to: 1) adequately infer the human action from gaze cues; 2) adjust its gaze fixation according to the human eye-gaze behavior; and 3) signal nonverbal cues that correlate with the robot's own action intentions.

Reliable and Accurate Implicit Neural Representation of Multiple Swept Volumes with Application to Safe Human–Robot Interaction

In automated production using collaborative robots in a manufacturing cell, a crucial aspect is to avoid collisions to ensure the safety of workers and robots in human–robot interaction. One approach for detecting collisions is using the swept volume (SV) to identify safe protective space for operation. We learn an accurate and reliable signed distance function (SDF) network from raw point clouds of a pre-computed SV to represent a class of linear joint motion trajectories. The network requires only a set of parameters and constant execution time, thus reducing the computational time and memory of collision checking due to the complexity of explicit geometry during task execution. The distance of collision danger foresaw by the learned SDF is exploited to reduce the frequency of collision detection calls in the dynamic environment and reduce the computational cost further. We assess the relative merits of the implicit neural representation of multiple SVs in terms of F1-score, error distance from the surface of the truth geometry, and 3D visualization by comparing favorably with a binary voxel network for learning a single SV with similar inference time. All the predicted errors of the geometry lie within a distance of 4 voxels from the surface of the truth geometry, and most reconstruction errors are within 3 voxels. A simulation of pick-and-place task execution in the human–robot interaction scenarios by leveraging the learned SDF as an efficient continuous collision detector is performed. The improvement in execution time and collision detection number is validated in the simulation.

RoSI: A Model for Predicting Robot Social Influence

A wide range of studies in Human-Robot Interaction (HRI) has shown that robots can influence the social behavior of humans. This phenomenon is commonly explained by the Media Equation. Fundamental to this theory is the idea that when faced with technology (like robots), people perceive it as a social agent with thoughts and intentions similar to those of humans. This perception guides the interaction with the technology and its predicted impact. However, HRI studies have also reported examples in which the Media Equation has been violated, that is when people treat the influence of robots differently from the influence of humans. To address this gap, we propose a model of Robot Social Influence (RoSI) with two contributing factors. The first factor is a robot’s violation of a person’s expectations, whether the robot exceeds expectations or fails to meet expectations. The second factor is a person’s social belonging with the robot, whether the person belongs to the same group as the robot or a different group. These factors are primary predictors of robots’ social influence and commonly mediate the influence of other factors. We review HRI literature and show how RoSI can explain robots’ social influence in concrete HRI scenarios.

Adaptive control for shape memory alloy actuated systems with applications to human-robot interaction.

Shape memory alloy (SMA) actuators are attractive options for robotic applications due to their salient features. So far, achieving precise control of SMA actuators and applying them to human-robot interaction scenarios remains a challenge. This paper proposes a novel approach to deal with the control problem of a SMA actuator. Departing from conventional mechanism models, we attempt to describe this nonlinear plant using a gray-box model, in which only the input current and the output displacement are measured. The control scheme consists of the model parameters updating and the control law calculation. The adaptation algorithm is founded on the multi-innovation concept and incorporates a dead-zone weighted factor, aiming to concurrently reduce computational complexities and enhance robustness properties. The control law is based on a PI controller, the gains of which are designed by the pole assignment technique. Theoretical analysis proves that the closed-loop performance can be ensured under mild conditions. The experiments are first conducted through the Beckhoff controller. The comparative results suggest that the proposed adaptive PI control strategy exhibits broad applicability, particularly under load variations. Subsequently, the SMA actuator is designed and incorporated into the hand rehabilitation robot. System position tracking experiments and passive rehabilitation training experiments for various gestures are then conducted. The experimental outcomes demonstrate that the hand rehabilitation robot, utilizing the SMA actuator, achieves higher position tracking accuracy and a more stable system under the adaptive control strategy proposed in this paper. Simultaneously, it successfully accommodates hand rehabilitation movements for multiple gestures. The adaptive controller proposed in this paper takes into account both the computational complexity of the model and the accuracy of the control results, Experimental results not only demonstrate the practicality and reliability of the controller but also attest to its potential application in human-machine interaction within the field of neural rehabilitation.

Estimating speaker direction on a humanoid robot with binaural acoustic signals.

To achieve human-like behaviour during speech interactions, it is necessary for a humanoid robot to estimate the location of a human talker. Here, we present a method to optimize the parameters used for the direction of arrival (DOA) estimation, while also considering real-time applications for human-robot interaction scenarios. This method is applied to binaural sound source localization framework on a humanoid robotic head. Real data is collected and annotated for this work. Optimizations are performed via a brute force method and a Bayesian model based method, results are validated and discussed, and effects on latency for real-time use are also explored.

ENHANCING HUMAN-ROBOT INTERACTION THROUGH GROUP EMOTION RECOGNITION

Abstract - This article explores within the field of Human-Robot Interaction (HRI), focusing on the complicated relationship between emotions, decision-making, and robot behaviors. Emotions are essential to effective communication and interaction, requiring the development of emotion recognition systems in robots. The article explores both individual and group emotion recognition, including microexpressions and macroexpressions. Group emotion dynamics, encompassing phenomena like emotional contagion, convergence, and social influence, are separated to understand how emotions combine within collective settings. A concept, Group Emotion Recognition (GER), is introduced, providing a framework for recognizing emotions within groups. GER involves proximity metrics, emotion classification, and entropy-based analysis to quantify emotion diversity. The article also outlines how GER can enhance user engagement, personalize interactions, improve group dynamics, and foster social acceptance in various human-robot interaction scenarios. Decision-making based on GER, driven by positive or negative emotion labels, is discussed, highlighting the adaptability and sensitivity required for effective human-robot interactions. Ethical considerations regarding the use of emotion recognition technology are addressed throughout the article, emphasizing responsible implementation. Overall, this work lays a solid foundation for advancing the field of HRI by integrating emotion recognition and decision-making to create emotionally intelligent and socially aware robots.

Human Robot Interaction : A Psychotechnical Perspective

This research paper delves into the multifaceted realm of Human-Robot Interaction (HRI) by adopting a multidisciplinary approach, amalgamating insights from psychology, technology, and neuroscience. From a psychological perspective, the study examines human responses to robots, considering factors like trust, empathy, and acceptance. It investigates how individual differences and cultural influences shape attitudes and behaviors in HRI scenarios. This psychological exploration forms a basis for designing robots aligned with human cognitive and emotional processes.

Exploring the Just Noticeable Interaction Stiffness Differences of an Impedance-Controlled Series Elastic Actuator

The integration of a passive elastic element in series between a motor and its load is popular in many human–robot interaction scenarios. By virtually imposing elastic behavior on the motor, an impedance control can act as a second stiffness to such an actuator. In this study, we investigated how participants perceived the different stiffness settings in a series elastic actuator by measuring the Just Noticeable Difference (JND) of the real stiffness of the elastic element and the virtual stiffness introduced by impedance control. We conducted a user study during which participants interacted with an impedance-controlled Series Elastic Actuator through a lever. During the user study, we varied the real stiffness of the elastic element and the virtual stiffness. We found that participants seem to perceive both the virtual stiffness and the real stiffness in the same way and in accordance to Weber’s law, which states that the stiffness JND is always equal to a fraction of the initial stiffness. Following these findings, we concluded that the impedance controller can implement an effective virtual stiffness with a behavior comparable to a real torsional spring. Therefore, a system combining real and virtual stiffness can simulate a single combined stiffness for a user interacting with it.

Never Trust Anything That Can Think for Itself, if You Can’t Control Its Privacy Settings: The Influence of a Robot’s Privacy Settings on Users’ Attitudes and Willingness to Self-disclose

AbstractWhen encountering social robots, potential users are often facing a dilemma between privacy and utility. That is, high utility often comes at the cost of lenient privacy settings, allowing the robot to store personal data and to connect to the internet permanently, which brings in associated data security risks. However, to date, it still remains unclear how this dilemma affects attitudes and behavioral intentions towards the respective robot. To shed light on the influence of a social robot’s privacy settings on robot-related attitudes and behavioral intentions, we conducted two online experiments with a total sample of N = 320 German university students. We hypothesized that strict privacy settings compared to lenient privacy settings of a social robot would result in more favorable attitudes and behavioral intentions towards the robot in Experiment 1. For Experiment 2, we expected more favorable attitudes and behavioral intentions for choosing independently the robot’s privacy settings in comparison to evaluating preset privacy settings. However, those two manipulations seemed to influence attitudes towards the robot in diverging domains: While strict privacy settings increased trust, decreased subjective ambivalence and increased the willingness to self-disclose compared to lenient privacy settings, the choice of privacy settings seemed to primarily impact robot likeability, contact intentions and the depth of potential self-disclosure. Strict compared to lenient privacy settings might reduce the risk associated with robot contact and thereby also reduce risk-related attitudes and increase trust-dependent behavioral intentions. However, if allowed to choose, people make the robot ‘their own’, through making a privacy-utility tradeoff. This tradeoff is likely a compromise between full privacy and full utility and thus does not reduce risks of robot-contact as much as strict privacy settings do. Future experiments should replicate these results using real-life human robot interaction and different scenarios to further investigate the psychological mechanisms causing such divergences.

Automatic Detection of Dyspnea in Real Human-Robot Interaction Scenarios.

A respiratory distress estimation technique for telephony previously proposed by the authors is adapted and evaluated in real static and dynamic HRI scenarios. The system is evaluated with a telephone dataset re-recorded using the robotic platform designed and implemented for this study. In addition, the original telephone training data are modified using an environmental model that incorporates natural robot-generated and external noise sources and reverberant effects using room impulse responses (RIRs). The results indicate that the average accuracy and AUC are just 0.4% less than those obtained with matched training/testing conditions with simulated data. Quite surprisingly, there is not much difference in accuracy and AUC between static and dynamic HRI conditions. Moreover, the beamforming methods delay-and-sum and MVDR lead to average improvement in accuracy and AUC equal to 8% and 2%, respectively, when applied to training and testing data. Regarding the complementarity of time-dependent and time-independent features, the combination of both types of classifiers provides the best joint accuracy and AUC score.

Snapture—a Novel Neural Architecture for Combined Static and Dynamic Hand Gesture Recognition

As robots are expected to get more involved in people’s everyday lives, frameworks that enable intuitive user interfaces are in demand. Hand gesture recognition systems provide a natural way of communication and, thus, are an integral part of seamless human-robot interaction (HRI). Recent years have witnessed an immense evolution of computational models powered by deep learning. However, state-of-the-art models fall short of expanding across different gesture domains, such as emblems and co-speech. In this paper, we propose a novel hybrid hand gesture recognition system. Our Snapture architecture enables learning both static and dynamic gestures: by capturing a so-called snapshot of the gesture performance at its peak, we integrate the hand pose and the dynamic movement. Moreover, we present a method for analyzing the motion profile of a gesture to uncover its dynamic characteristics, which allows regulating a static channel based on the amount of motion. Our evaluation demonstrates the superiority of our approach on two gesture benchmarks compared to a state-of-the-art CNNLSTM baseline. Our analysis on a gesture class basis unveils the potential of our Snapture architecture for performance improvements using RGB data. Thanks to its modular implementation, our framework allows the integration of other multimodal data, like facial expressions and head tracking, which are essential cues in HRI scenarios, into one architecture. Thus, our work contributes both to integrative gesture recognition research and machine learning applications for non-verbal communication with robots.

Design and Validation of Symmetrical Elastic Elements in Series Elastic Actuator

Abstract In recent years, the demand for robot joint compliance increased with more complex human–robot interaction scenarios. Series elastic actuators (SEAs) are extensively utilized in multiple fields with abilities to provide accurate force control and energy storage. As the main flexible component in SEA, the accurate modeling of the stiffness of the elastic element is essential. However, the existing stiffness models based on classical Euler–Bernoulli beam theory contain large errors with the actual situation, which increases the difficulty of design. In this paper, a typical elastic element is analyzed by the finite element method to investigate its stiffness properties with different geometric parameters. A more accurate stiffness model is proposed for designing elastic elements. The stiffness model is validated by designed experiments, with a fitting accuracy of 98.27%, which significantly exceeds the stiffness model based on classical beam theory. The proposed stiffness model can be applied to design elastic elements that meet specific stiffness requirements.

A method based on interpretable machine learning for recognizing the intensity of human engagement intention

To interact with humans more precisely and naturally, social robots need to “perceive” human engagement intention, especially need to recognize the main interaction person in multi-person interaction scenarios. By analyzing the intensity of human engagement intention (IHEI), social robots can distinguish the intention of different persons. Most existing research in this field mainly focus on analyzing whether a person has the intention to interact with the robot while lack of analysis of IHEI. In this regard, this paper proposes an approach for recognizing the engagement intention intensity. Four categories of visual features, including line of sight, head pose, distance and expression of human, are captured, and a CatBoost-based machine learning model is applied to train an optimal classifier for predicting the IHEI on the dataset. The experimental results show that this classifier can effectively predict the IHEI that can be applied into real human–robot interaction scenarios. Moreover, the proposed model is an interpretable machine learning model, where interpretability analysis on the trained classifier has been done to explore the deep associations between input features and engagement intention, thereby providing robust and effective robot social decision-making.

Learning Ergonomic Control in Human–Robot Symbiotic Walking

This article presents an imitation learning strategy for extracting ergonomically safe control policies in physical human–robot interaction scenarios. The presented approach seeks to proactively reduce the risk of injuries and musculoskeletal disorders by anticipating the ergonomic effects of a robot's actions on a human partner, e.g., how the ankle angle of a prosthesis affects future knee torques of the user. To this end, we extend ensemble Bayesian interaction primitives to enable the prediction of latent biomechanical variables. This methodology yields a reactive control strategy, which we evaluate in an assisted walking task with a robotic lower limb prosthesis. Building upon the learned interaction primitives, we also present a model-predictive control (MPC) strategy that actively steers the human–robot interaction toward ergonomic and safe movement regimes. We compare the introduced control strategies and highlight the framework's ability to generate ergonomic, biomechanically safe assistive prosthetic control. A rich analysis of constrained MPC shows a 20× reduction in the effects of large perturbations on prosthetic control system. We empirically demonstrate a 16% reduction in vertical knee reaction forces in real-world jumping experiments utilizing our control methodology and examine other optimal control strategies in simulated walking experiments.

Imitation System of Humanoid Robots and Its Applications

In this paper, we propose an imitation system that imitates human motions in videos to plan robot actions that are similar to human motions, with the aim of the complicated whole-body action planning of humanoid robots. Additionally, we created an interaction system that will enable basic human-robot interaction for our humanoid robot. To obtain the 3D coordinates of the key points on the human body, we used the 3D pose estimation model. The key points were then transformed into various trajectory files needed by the robot to complete the motion, using the mapping method proposed in this research, which refers to the control strategy and stability of the robot. In addition, we proposed some post-processing methods to post-process the trajectories. In the interaction system, we created a speech and vision system so that the robot could detect human gestures or postures and converse with people. It also has a music rhythm recognition system developed by seniors that enables the robot to dance to the beats of the song. Finally, through this system, we completed several human-robot interaction scenarios, which proved the convenience, and effectiveness of motion planning with an imitation system, and the completeness of the interaction system.

2D Image head pose estimation via latent space regression under occlusion settings

Head orientation is a challenging Computer Vision problem that has been extensively researched having a wide variety of applications. However, current state-of-the-art systems still underperform in the presence of occlusions and are unreliable for many task applications in such scenarios. This work proposes a novel deep learning approach for the problem of head pose estimation under occlusions. The strategy is based on latent space regression as a fundamental key to better structure the problem for occluded scenarios. Our model surpasses several state-of-the-art methodologies for occluded HPE, and achieves similar accuracy for non-occluded scenarios. We demonstrate the usefulness of the proposed approach with: (i) two synthetically occluded versions of the BIWI and AFLW2000 datasets, (ii) real-life occlusions of the Pandora dataset, and (iii) a real-life application to human-robot interaction scenarios where face occlusions often occur. Specifically, the autonomous feeding from a robotic arm.

Collaborative Autonomy: Human–Robot Interaction to the Test of Intelligent Help

A big challenge in human–robot interaction (HRI) is the design of autonomous robots that collaborate effectively with humans, exposing behaviors similar to those exhibited by humans when they interact with each other. Indeed, robots are part of daily life in multiple environments (i.e., cultural heritage sites, hospitals, offices, touristic scenarios and so on). In these contexts, robots have to coexist and interact with a wide spectrum of users not necessarily able or willing to adapt their interaction level to the kind requested by a machine: the users need to deal with artificial systems whose behaviors must be adapted as much as possible to the goals/needs of the users themselves, or more in general, to their mental states (beliefs, goals, plans and so on). In this paper, we introduce a cognitive architecture for adaptive and transparent human–robot interaction. The architecture allows a social robot to dynamically adjust its level of collaborative autonomy by restricting or expanding a delegated task on the basis of several context factors such as the mental states attributed to the human users involved in the interaction. This collaboration has to be based on different cognitive capabilities of the robot, i.e., the ability to build a user’s profile, to have a Theory of Mind of the user in terms of mental states attribution, to build a complex model of the context, intended both as a set of physical constraints and constraints due to the presence of other agents, with their own mental states. Based on the defined cognitive architecture and on the model of task delegation theorized by Castelfranchi and Falcone, the robot’s behavior is explainable by considering the abilities to attribute specific mental states to the user, the context in which it operates and its attitudes in adapting the level of autonomy to the user’s mental states and the context itself. The architecture has been implemented by exploiting the well known agent-oriented programming framework Jason. We provide the results of an HRI pilot study in which we recruited 26 real participants that have interacted with the humanoid robot Nao, widely used in HRI scenarios. The robot played the role of a museum assistant with the main goal to provide the user the most suitable museum exhibition to visit.

Emotion and Mood Blending in Embodied Artificial Agents: Expressing Affective States in the Mini Social Robot

Robots that are devised for assisting and interacting with humans are becoming fundamental in many applications, including in healthcare, education, and entertainment. For these robots, the capacity to exhibit affective states plays a crucial role in creating emotional bonding with the user. In this work, we present an affective architecture that grounds biological foundations to shape the affective state of the Mini social robot in terms of mood and emotion blending. The affective state depends upon the perception of stimuli in the environment, which influence how the robot behaves and affectively communicates with other peers. According to research in neuroscience, mood typically rules our affective state in the long run, while emotions do it in the short term, although both processes can overlap. Consequently, the model that is presented in this manuscript deals with emotion and mood blending towards expressing the robot’s internal state to the users. Thus, the primary novelty of our affective model is the expression of: (i) mood, (ii) punctual emotional reactions to stimuli, and (iii) the decay that mood and emotion undergo with time. The system evaluation explored whether users can correctly perceive the mood and emotions that the robot is expressing. In an online survey, users evaluated the robot’s expressions showing different moods and emotions. The results reveal that users could correctly perceive the robot’s mood and emotion. However, emotions were more easily recognized, probably because they are more intense affective states and mainly arise as a stimuli reaction. To conclude the manuscript, a case study shows how our model modulates Mini’s expressiveness depending on its affective state during a human-robot interaction scenario.

What’s on Your Mind, NICO?

Explainable AI has become an important field of research on neural machine learning models. However, most existing methods are designed as tools that provide expert users with additional insights into their models. In contrast, in human-robot interaction scenarios, non-expert users are frequently confronted with complex, embodied AI systems whose inner workings are unknown. Therefore, eXplainable Human-Robot Interaction (XHRI) should leverage the user’s intuitive ability to collaborate and to use efficient communication. Using NICO, the Neuro-Inspired COmpanion, as a use-case study, we propose an XHRI framework and show how different types of explanations enhance the interaction experience. These explanations range from (a) non-verbal cues for simple and intuitive feedback of inner states via (b) comprehensive verbal explanations of the robot’s intentions, knowledge and reasoning to (c) multimodal explanations using visualizations, speech and text. We revisit past HRI-related studies conducted with NICO and analyze them with the proposed framework. Furthermore, we present two novel XHRI approaches to extract suitable verbal and multimodal explanations from neural network modules in an HRI scenario.

Autonomous Critical Help by a Robotic Assistant in the Field of Cultural Heritage: A New Challenge for Evolving Human-Robot Interaction

Over the years, the purpose of cultural heritage (CH) sites (e.g., museums) has focused on providing personalized services to different users, with the main goal of adapting those services to the visitors’ personal traits, goals, and interests. In this work, we propose a computational cognitive model that provides an artificial agent (e.g., robot, virtual assistant) with the capability to personalize a museum visit to the goals and interests of the user that intends to visit the museum by taking into account the goals and interests of the museum curators that have designed the exhibition. In particular, we introduce and analyze a special type of help (critical help) that leads to a substantial change in the user’s request, with the objective of taking into account the needs that the same user cannot or has not been able to assess. The computational model has been implemented by exploiting the multi-agent oriented programming (MAOP) framework JaCaMo, which integrates three different multi-agent programming levels. We provide the results of a pilot study that we conducted in order to test the potential of the computational model. The experiment was conducted with 26 real participants that have interacted with the humanoid robot Nao, widely used in Human-Robot interaction (HRI) scenarios.