Robot Interaction Research Articles

Guarantees for Real Robotic Systems: Unifying Formal Controller Synthesis and Reachset-Conformant Identification

Robots are used increasingly often in safety-critical scenarios, such as robotic surgery or human–robot interaction. To ensure stringent performance criteria, formal controller synthesis is a promising direction to guarantee that robots behave as desired. However, formally ensured properties only transfer to the real robot when the model is appropriate. In this article, we address this problem by combining the identification of a reachset-conformant model with controller synthesis. Since the reachset-conformant model contains all the measured behaviors of the real robot, the safety properties of the model transfer to the real robot. The transferability is demonstrated by experiments on a real robot, for which we synthesize tracking controllers.

A Robust Region Control Approach for Simultaneous Trajectory Tracking and Compliant Physical Human–Robot Interaction

A Robust Region Control Approach for Simultaneous Trajectory Tracking and Compliant Physical Human–Robot Interaction

Stiffness-Switchable Hydrostatic Transmission Toward Safe Physical Human–Robot Interaction

Stiffness-Switchable Hydrostatic Transmission Toward Safe Physical Human–Robot Interaction

An Integrated in Situ Image Acquisition and Annotation Scheme for Instance Segmentation Models in Open Scenes With a Human–Robot Interaction Approach

An Integrated in Situ Image Acquisition and Annotation Scheme for Instance Segmentation Models in Open Scenes With a Human–Robot Interaction Approach

Effects of Failure Types on Trust Repairs in Human–Robot Interactions

Effects of Failure Types on Trust Repairs in Human–Robot Interactions

Human–robot collaborative interaction with human perception and action recognition

This paper presents a human–robot interaction system (HRIS) that utilizes human perception and action recognition to enable the robot to understand human intentions and flexibly interact with humans. A monocular multi-person three-dimensional (3D) pose estimation method is first proposed to perceive multi-person two-dimensional (2D) and 3D poses in interaction scenarios. Furthermore, a 3D skeleton poses tracking approach is adopted to locate the identity of each person in consecutive frames and enhance interactive stability. Then, an action recognition model is developed, which exploits tracked pose features to recognize the intentions of humans. An action-controlled interaction system is built with a modular approach to ensure flexibility in meeting multiple task requirements and facilitating flexible interaction. In the system, a distance-based safety solution is designed to avoid collisions between humans and robots. Finally, experimental results are presented to demonstrate the feasibility and effectiveness of the proposed methods and system.

A BTO/PVDF/PDMS Piezoelectric Tangential and Normal Force Sensor Inspired by a Wind Chime.

There is a growing demand for flexible pressure sensors in environmental monitoring and human-robot interaction robotics. A flexible and susceptible sensor can discriminate multidirectional pressure, thus effectively detecting signals of small environmental changes and providing solutions for personalized medicine. This paper proposes a multidimensional force detection sensor inspired by a wind chime structure with a three-dimensional force structure to detect and analyze normal and shear forces in real time. The force-sensing structure of the sensor consists of an upper and lower membrane on a polydimethylsiloxane substrate and four surrounding cylinders. A piezoelectric hemisphere is made of BTO/PVDF/PDMS composite material. The sensor columns in the wind chime structure surround the piezoelectric layer in the middle. When pressure is applied externally, the sensor columns are connected to the piezoelectric layer with a light touch. The piezoelectric hemisphere generates a voltage signal. Due to the particular structure of the sensor, it can accurately capture multidimensional forces and identify the direction of the external force by analyzing the position of the sensor and the output voltage amplitude. The development of such sensors shows excellent potential for self-powered wearable sensors, human-computer interaction, electronic skin, and soft robotics applications.

A 3K Planetary Gear Train with a flexure-based anti-backlash carrier for collaborative robots

Backdrivable joint actuators are critical to enhancing the human–robot interaction performance of collaborative robots. A Planetary Gear Train (PGT) with a high reduction ratio and high bidirectional transmission efficiencies is a promising gear reducer candidate for joint actuators. However, a conventional PGT normally has a large backlash resulting in low transmission accuracy and an uncontrollable dead zone, which hinders the applications in joint actuators. Utilizing the unique structural characteristic of the 3K-type PGT that its carrier is not participated in the torque transmission, a novel flexure-based carrier with the adjustable preload and stiffness is proposed to significantly reduce backlash. To study the effects of the proposed carrier on the backlash of the 3K-type PGT, a kinetostatic model and a modified backlash model are formulated. Simulation results indicate that the carrier reduces backlash effectively, and the step dead zone caused by backlash is transformed into a low stiffness zone. The effectiveness of the proposed carrier is further verified by experimental results such that the backlash of the prototype is reduced from 10.97 arcmin to 0.47 arcmin significantly.

Skeleton-RGB integrated highly similar human action prediction in human–robot collaborative assembly

Human–robot collaborative assembly (HRCA) combines the flexibility and adaptability of humans with the efficiency and reliability of robots during collaborative assembly operations, which facilitates complex product assembly in the mass personalisation paradigm. The cognitive ability of robots to recognise and predict human actions and make responses accordingly is essential but currently still limited, especially when facing highly similar human actions. To improve the cognitive ability of robots in HRCA, firstly, a two-stage skeleton-RGB integrated model focusing on human-parts interaction is proposed to recognise highly similar human actions. Specifically, it consists of a feature guidance module and a feature fusion module, which can balance the accuracy and efficiency of human action recognition. Secondly, an online prediction approach is developed to predict human actions ahead of schedule, which includes a pre-trained skeleton-RGB integrated model and a preprocessing module. Thirdly, considering the positioning accuracy of the parts to be assembled and the continuous update of human actions, a dynamic response scheme of the robot is designed. Finally, the feasibility and effectiveness of the proposed model and approach are verified by a case study of a worm-gear decelerator assembly. The experimental results demonstrate that the proposed model achieves precise human action recognition with a high accuracy of 93.75% and a lower computational cost. Specifically, only 15 frames from a skeleton stream and 5 frames (less than 16 frames in general) from an RGB video stream are adopted. Moreover, it only takes 1.026 s to achieve online human action prediction based on the proposed prediction method. The dynamic response scheme of the robot is also proven to be feasible. It is expected that the efficiency of human–robot interaction in HRCA can be improved from a closed-loop view of perception, prediction, and response.

Types of city dwellers’attitudes towards service robots in the city of Perm

The effectiveness of robot implementation in various societal sectors depends not only on the robot’s technical characteristics but also on how different social groups perceive and interact with them. Social attitudes towards robots, in turn, are influenced by sociocultural and sociodemographic factors. In the context of the sociology of technology and actor-network theory, this research explores the social significance of technical devices, the processes of social adaptation, and communication with robots. However, empirical research on human-robot interaction (HRI), especially in the Russian context, remains limited. This article presents the results of a study that examines the perceptions of robots by actual and potential users in two aspects: the level of basic social attitudes towards new technologies and robots (social acceptability) and the evaluation of interactions with robots in specific situations (practical acceptability). The empirical basis of the study consists of survey data from customers at a multifunctional center providing state and municipal services in Perm. They received services from either a service anthropomorphic robot, Promobot V.4, or a human employee at the center. Using cluster analysis, three types (clusters) of attitudes towards innovations and robots were identified at the level of basic social attitudes: robo-skeptics, robo-phobes, and robo-philes. The social characteristics of these identified types were examined. The study introduces an original method for measuring satisfaction with actual interactions with service robots based on various parameters (evaluation of service convenience, communication quality, personal activity, willingness to recommend robot services, etc.). The study confirms the hypothesis of the connection between the type of attitude towards innovations and robots and the satisfaction with actual interactions with robots in specific situations. The lowest levels of satisfaction with robot interactions and the desire to avoid such interactions were observed among robo-phobes, who are more frequently represented by individuals of senior age group (age 55 and above). The research results allow determining the prospects for implementing service robots in the public space of a major Russian city based on understanding the social attitudes of users.

Spontaneous Facial Behavior Analysis Using Deep Transformer-based Framework for Child–computer Interaction

A fascinating challenge in robotics-human interaction is imitating the emotion recognition capability of humans to robots with the aim to make human-robotics interaction natural, genuine and intuitive. To achieve the natural interaction in affective robots, human-machine interfaces, and autonomous vehicles, understanding our attitudes and opinions is very important, and it provides a practical and feasible path to realize the connection between machine and human. Multimodal interface that includes voice along with facial expression can manifest a large range of nuanced emotions compared to purely textual interfaces and provide a great value to improve the intelligence level of effective communication. Interfaces that fail to manifest or ignore user emotions may significantly impact the performance and risk being perceived as cold, socially inept, untrustworthy, and incompetent. To equip a child well for life, we need to help our children identify their feelings, manage them well, and express their needs in healthy, respectful, and direct ways. Early identification of emotional deficits can help to prevent low social functioning in children. In this work, we analyzed the child’s spontaneous behavior using multimodal facial expression and voice signal presenting multimodal transformer-based last feature fusion for facial behavior analysis in children to extract contextualized representations from RGB video sequence and Hematoxylin and eosin video sequence and then using these representations followed by pairwise concatenations of contextualized representations using cross-feature fusion technique to predict users emotions. To validate the performance of the proposed framework, we have performed experiments with the different pairwise concatenations of contextualized representations that showed significantly better performance than state-of-the-art method. Besides, we perform t-distributed stochastic neighbor embedding visualization to visualize the discriminative feature in lower dimension space and probability density estimation to visualize the prediction capability of our proposed model.

Human Action Recognition Based on Hierarchical Multi-Scale Adaptive Conv-Long Short-Term Memory Network

Recently, human action recognition has gained widespread use in fields such as human–robot interaction, healthcare, and sports. With the popularity of wearable devices, we can easily access sensor data of human actions for human action recognition. However, extracting spatio-temporal motion patterns from sensor data and capturing fine-grained action processes remain a challenge. To address this problem, we proposed a novel hierarchical multi-scale adaptive Conv-LSTM network structure called HMA Conv-LSTM. The spatial information of sensor signals is extracted by hierarchical multi-scale convolution with finer-grained features, and the multi-channel features are fused by adaptive channel feature fusion to retain important information and improve the efficiency of the model. The dynamic channel-selection-LSTM based on the attention mechanism captures the temporal context information and long-term dependence of the sensor signals. Experimental results show that the proposed model achieves Macro F1-scores of 0.68, 0.91, 0.53, and 0.96 on four public datasets: Opportunity, PAMAP2, USC-HAD, and Skoda, respectively. Our model demonstrates competitive performance when compared to several state-of-the-art approaches.

Never Trust Anything That Can Think for Itself, if You Can’t Control Its Privacy Settings: The Influence of a Robot’s Privacy Settings on Users’ Attitudes and Willingness to Self-disclose

AbstractWhen encountering social robots, potential users are often facing a dilemma between privacy and utility. That is, high utility often comes at the cost of lenient privacy settings, allowing the robot to store personal data and to connect to the internet permanently, which brings in associated data security risks. However, to date, it still remains unclear how this dilemma affects attitudes and behavioral intentions towards the respective robot. To shed light on the influence of a social robot’s privacy settings on robot-related attitudes and behavioral intentions, we conducted two online experiments with a total sample of N = 320 German university students. We hypothesized that strict privacy settings compared to lenient privacy settings of a social robot would result in more favorable attitudes and behavioral intentions towards the robot in Experiment 1. For Experiment 2, we expected more favorable attitudes and behavioral intentions for choosing independently the robot’s privacy settings in comparison to evaluating preset privacy settings. However, those two manipulations seemed to influence attitudes towards the robot in diverging domains: While strict privacy settings increased trust, decreased subjective ambivalence and increased the willingness to self-disclose compared to lenient privacy settings, the choice of privacy settings seemed to primarily impact robot likeability, contact intentions and the depth of potential self-disclosure. Strict compared to lenient privacy settings might reduce the risk associated with robot contact and thereby also reduce risk-related attitudes and increase trust-dependent behavioral intentions. However, if allowed to choose, people make the robot ‘their own’, through making a privacy-utility tradeoff. This tradeoff is likely a compromise between full privacy and full utility and thus does not reduce risks of robot-contact as much as strict privacy settings do. Future experiments should replicate these results using real-life human robot interaction and different scenarios to further investigate the psychological mechanisms causing such divergences.

Adaptive technique for physical human–robot interaction handling using proprioceptive sensors

The work focuses on the development of an adaptive technique for the physical interaction handling between a human and a robot, as well as its experimental validation. The proposed technique is based on the deep residual neural network and dedicated finite state machine, where the states are the robot behavior modes and transitions are the switchings between the states that depend on the interaction parameters and characteristics. It ensures the human operator safety and improves the human–robot collaboration performance by implementing various scenarios. In the scope of this technique, the parameters of human–robot interaction are used to select an appropriate robot reaction strategy using data from internal robot sensors only, i.e. proprioceptive sensors. These parameters define the interaction force vector and its application point on the robot surface, which allow to classify the interaction within the set of predefined categories. This classification distinguishes interactions applied at the tool or intermediate link (Tool/Link), having soft or hard nature (Soft/Hard), as well as having different intention (Intl/Accd) or duration (Short/Long). Based on identified category and the current robot state, the algorithm chooses an appropriate robot reaction. To confirm the efficiency the developed technique, an experimental study was conducted, which involved the collaboration between the real industrial manipulator KUKA LBR iiwa and the human operator.

Acceptance of social robot and its challenges: From privacy calculus perspectives

The aim of this study is to examine human–robot interaction (HRI) from the perspective of security, privacy, as well as social and personal norms. This study also strives to investigate users' hedonic and utilitarian attitudes towards using social robots. A social robot is a physical humanoid robot that communicates and interacts with humans and other devices. Social robots follow social rules and behaviors which make them different from other kinds of robots. They often behave like social partners to humans and have features such as eye-gaze, speech recognition, auto-gestures, and so on. Social robots use different types of sensitive data from the users which could pose security challenges and privacy threats. Not many studies have dealt with human–robot interactions (HRI) from security, privacy, social norms, and personal norms perspectives. With the existing literature as well as privacy calculus, motivational theory, and theory of planned behavior (TPB), a model has been developed conceptually, which was then validated using a sample of 403 users contemplating using social robots in the coming days. The study found that both users' hedonic and utilitarian attitudes impact significantly and positively their usage of social robots.

Bed nucleus of the stria terminalis GABA neurons are necessary for changes in foraging behaviour following an innate threat.

Foraging is a universal behaviour that has co-evolved with predation pressure. We investigated the role of the bed nucleus of the stria terminalis (BNST) GABA neurons in robotic and live predator threat processing and their consequences in post-threat encounter foraging. Both robotic and live predator interactions increased BNST GABA neuron activity. Mice were trained to procure food in a laboratory-based foraging apparatus in which food pellets were placed at incrementally greater distances from a nest zone. After mice learned to forage, they were exposed to a robotic or live predator threat, while BNST GABA neurons were chemogenetically inhibited. Post-robotic threat encounter, mice spent more time in the nest zone, but other foraging parameters were unchanged compared with pre-encounter behaviour. Inhibition of BNST GABA neurons had no effect on foraging behaviour post-robotic threat encounter. Following live predator exposure, control mice spent significantly more time in the nest zone, increased their latency to successfully forage, and significantly altered their overall foraging performance. Inhibition of BNST GABA neurons during live predator exposure prevented changes in foraging behaviour from developing after a live predator threat. BNST GABA neuron inhibition did not alter foraging behaviour during robotic or live predator threats. We conclude that these results demonstrate that while both robotic and live predator encounters effectively intrude on foraging behaviour, the perceived risk and behavioural consequences of the threat are distinguishable. Additionally, BNST GABA neurons may play a role in the integration of prior innate predator threat experience that results in hypervigilance during post-encounter foraging behaviour.

The impact of service robots on customer satisfaction online ratings: The moderating effects of rapport and contextual review factors

AbstractRecent research has established a positive relationship between the use of service robots powered by artificial intelligence in hospitality firms and customer satisfaction online ratings, a particularly important form of electronic word of mouth. However, it is not clear if and how this relationship is augmented or diminished by moderating factors. In this study, we examined four potential moderators by using machine learning and natural language processing techniques to analyze 20,166 online reviews of hotels that had implemented service robots. We had four key findings. First, a positive service robot‐satisfaction rating relationship was further enhanced by improved customer‐service robot rapport during the service encounter. Second, higher customer effort focused on service robots in a review reduced the service robot‐satisfaction rating relationship. Third, posting reviews using a mobile device (vs. other devices) showed higher satisfaction ratings. Finally, customers' prior experience in writing online reviews was unrelated to the service robot‐satisfaction rating relationship. Taken together, these results suggest that service robots should be designed to be interactive and encourage customers to build rapport, for example, by service robots engaging in conversational flows. Moreover, customers should be nudged to use their mobile devices to post timely reviews on their positive human–robot interactions.

Emotion recognition and artificial intelligence: A systematic review (2014–2023) and research recommendations

Emotion recognition is the ability to precisely infer human emotions from numerous sources and modalities using questionnaires, physical signals, and physiological signals. Recently, emotion recognition has gained attention because of its diverse application areas, like affective computing, healthcare, human–robot interactions, and market research. This paper provides a comprehensive and systematic review of emotion recognition techniques of the current decade. The paper includes emotion recognition using physical and physiological signals. Physical signals involve speech and facial expression, while physiological signals include electroencephalogram, electrocardiogram, galvanic skin response, and eye tracking. The paper provides an introduction to various emotion models, stimuli used for emotion elicitation, and the background of existing automated emotion recognition systems. This paper covers comprehensive searching and scanning of well-known datasets followed by design criteria for review. After a thorough analysis and discussion, we selected 142 journal articles using PRISMA guidelines. The review provides a detailed analysis of existing studies and available datasets of emotion recognition. Our review analysis also presented potential challenges in the existing literature and directions for future research.

Facial Expression Recognition Based on Zero-Addition Pretext Training and Feature Conjunction-Selection Network in Human–Robot Interaction

Facial Expression Recognition Based on Zero-Addition Pretext Training and Feature Conjunction-Selection Network in Human–Robot Interaction

A human-like action learning process: Progressive pose generation for motion prediction

Human motion prediction is crucial for the human–robot inter-action and self-driving. We human beings learn an action with two stages, i.e., the approximation stage and the adjustment stage. While the first stage learns an approximate pose in general, the second stage learns more details. Based on this observation, this paper proposes a new two-stage framework to predict the upcoming human motion poses progressively based on a historical sequence. In the approximation stage, we adopt the motion attention network to derive an approximation of the upcoming poses. In the adjustment stage, we propose a novel fine-tuning network that can extract both spatial and temporal features to enhance the prediction accuracy. To imitate the action learning procedure of our humans, the adjustment stage in our progressive prediction framework continuously adjusts the pose many times via a series of fine-tuning networks. Extensive experiments on three benchmark datasets (i.e., Human3.6M, CMU-Mocap, and 3DPW) show that our proposed method can outperform state-of-the-art methods in both short-term and long-term predictions.