Emotional Robot Research Articles

Utilizing an Emotional Robot Capable of Lip-Syncing in Robot-Assisted Speech Therapy Sessions for Children with Language Disorders.

This study scrutinizes the impacts of utilizing a socially assistive robot, the RASA robot, during speech therapy sessions for children with language disorders. Two capabilities were developed for the robotic platform to enhance children-robot interactions during speech therapy interventions: facial expression communication (containing recognition and expression) and lip-syncing. Facial expression recognition was conducted by training several well-known CNN architectures on one of the most extensive facial expressions databases, the AffectNet database, and then modifying them using the transfer learning strategy performed on the CK+ dataset. The robot's lip-syncing capability was designed in two steps. The first step was concerned with designing precise schemes of the articulatory elements needed during the pronunciation of the Persian phonemes (i.e., consonants and vowels). The second step included developing an algorithm to pronounce words by disassembling them into their components (including consonants and vowels) and then morphing them into each other successively. To pursue the study's primary goal, two comparable groups of children with language disorders were considered, the intervention and control groups. The intervention group attended therapy sessions in which the robot acted as the therapist's assistant, while the control group only communicated with the human therapist. The study's first purpose was to compare the children's engagement while playing a mimic game with the affective robot and the therapist, conducted via video coding. The second objective was to assess the efficacy of the robot's presence in the speech therapy sessions alongside the therapist, accomplished by administering the Persian Test of Language Development, Persian TOLD. According to the first scenario, playing with the affective robot is more engaging than playing with the therapist. Furthermore, the statistical analysis of the study's results indicates that participating in robot-assisted speech therapy (RAST) sessions enhances children with language disorders' achievements in comparison with taking part in conventional speech therapy interventions.

Exploding the android: encounters with social robotics in a science centre

ABSTRACT This paper examines science centre displays incorporating robotic technologies to provoke reflection on social robotics. Drawing on research in post-phenomenology and science and technology studies, it considers how exhibits in Born or Built? about robotic emotion undermine its framing narrative of human-robot convergence. Comparing these science centre exhibits to social robotics demonstrations draws attention to how the exhibits move away from convincing audiences about the truth or merits of these technologies to instead convey their difference and open-endedness. While there are limited popular narratives and images to invoke in exploring these ideas, relationships can be drawn between the displays and historical automata and contemporary robotic artwork, practices that frequently seek to promote reflection on robotic otherness, appearance and relationality. Representing robots with robotics points towards the relevance of style to conceptualising robotic diversity and materiality. Style in this sense is not simply an embellishment or surface addition to content, but a material expression inseparable from what is communicated and relative to technological, cultural and social milieus.

Interdisciplinary IoT and Emotion Knowledge Graph-Based Recommendation System to Boost Mental Health

Humans are feeling emotions every day, but they can still encounter difficulties understanding them. To better understand emotions, we integrated interdisciplinary knowledge about emotions from various domains such as neurosciences (e.g., neurobiology), physiology, and psychology (affective sciences, positive psychology, cognitive psychology, psychophysiology, neuropsychology, etc.). To organize the knowledge, we employ technologies such as Artificial Intelligence with Knowledge Graphs and Semantic Reasoning. Furthermore, Internet of Things (IoT) technologies can help to acquire physiological data knowledge. The goal of this paper is to aggregate the interdisciplinary knowledge and implement it within the Emotional Knowledge Graph (EmoKG). The Emotional Knowledge Graph is used within our naturopathy recommender system that suggests food to boost emotion (e.g., chocolate contains magnesium that is recommended when we feel depressed). The recommender system also answers a set of competency questions to easily retrieve emotional related-knowledge from EmoKG, such as what are the basic emotions and the more sophisticated ones, what are the neurotransmitters and hormones related to emotions, etc. To follow FAIR principles, EmoKG is mapped to existing knowledge bases found on the BioPortal biomedical ontology catalog such as SNOMEDCT, FMA, RXNORM, MedDRA, and also from emotion ontologies (when available online). We design the LOV4IoT-Emotion ontology catalog that encourages researchers from heterogeneous communities to apply FAIR principles by releasing online their (emotion) ontologies, datasets, rules, etc. The set of ontology codes shared online can be semi-automatically processed; if not available, the scientific publications describing the emotion ontologies are semi-automatically processed with Natural Language Processing (NLP) technologies. This research is also relevant for other use cases such as European projects (ACCRA for emotional robots to reduce the social isolation of aging people, StandICT for standardization, and AI4EU for Artificial Intelligence) and alliances for IoT such as AIOTI. The recommender system can be extended to address other advice such as aromatherapy and take into consideration medical devices to monitor patients’ vital signals related to emotions and mental health.

Human-Robot Scaffolding: An Architecture to Foster Problem-solving Skills

In order to give assertive support, robots need to understand the cognitive and emotional characteristics of learners while in the learning process. Also, if the task involves handling capacity, robots must have similar skills. Three concepts were explored to control the cognitive and emotional robot’s behavior: the psychological flow theory, the scaffolding pedagogic strategy, and the multiagents-software paradigm. Based on these concepts, the Human-Robot Scaffolding architecture was designed. It is divided into five blocks. First, the sensory block recognizes body gestures, speech, and task state. Second, the beliefs block estimates the skills and emotional state of learners. Third, the desires block validates the goals the robot can reach; the goals are grouped in skills development, emotional control, cognitive control, challenge control, life signals, and immediate support. Fourth, the intentions block, based on the goals competition strategy, selects the goal that the robot will perform. Finally, the action-planner block regulates the robot’s movements according to the robot’s emotions. The validation procedure was done with 53 learners ranging between 10 and 13 years old who study in public and private schools. Based on the research achievements, the robot fosters learning the Mean-Ends Analysis strategy and the solution of a problem. A video fragment that summarizes the research process is available in https://youtu.be/qbohCjBIwYc .

Survey and perspective on social emotions in robotics

This paper presents a review of available research concerning social emotions in robotics. In robotics, the study of emotions has been pursued since a long time. The popular research endeavors in robotics concern the study of robotic recognition, expression, and computational modeling of the basic mechanisms that underlie them. The advancements in research relevant to this domain are in accordance with well-known psychological findings obtained using the category and dimension theories. Many studies are based on these basic theories, which exclusively address the basic emotions. However, social emotions, also referred to as high-level emotions, have been investigated in psychology. We believe that these high-level emotions are worth investigating for the development of next-generation robotic systems – socially aware robots. This paper summarizes the findings concerning social emotions reported through psychology and neuroscience research endeavors along with a survey of studies concerning social emotions in robotics conducted to date. Moreover, this paper discusses future research directions to facilitate the implementation of social emotions in robots.

Emotion and memory model for social robots: a reinforcement learning based behaviour selection

ABSTRACT In this paper, we propose a reinforcement learning (RL) mechanism for social robots to select an action based on users’ learning performance and social engagement. We applied this behavior selection mechanism to extend the emotion and memory model, which allows a robot to create a memory account of the user’s emotional events and adapt its behavior based on the developed memory. We evaluated the model in a vocabulary-learning task at a school during a children’s game involving robot interaction to see if the model results in maintaining engagement and improving vocabulary learning across the four different interaction sessions. Generally, we observed positive findings based on child vocabulary learning and sustaining social engagement during all sessions. Compared to the trends of a previous study, we observed a higher level of social engagement across sessions in terms of the duration of the user gaze toward the robot. For vocabulary retention, we saw similar trends in general but also showing high vocabulary retention across some sessions. The findings indicate the benefits of applying RL techniques that have a reward system based on multi-modal user signals or cues.

Artificial Eyes with Emotion and Light Responsive Pupils for Realistic Humanoid Robots

This study employs a novel 3D engineered robotic eye system with dielectric elastomer actuator (DEA) pupils and a 3D sculpted and colourised gelatin iris membrane to replicate the appearance and materiality of the human eye. A camera system for facial expression analysis (FEA) was installed in the left eye, and a photo-resistor for measuring light frequencies in the right. Unlike previous prototypes, this configuration permits the robotic eyes to respond to both light and emotion proximal to a human eye. A series of experiments were undertaken using a pupil tracking headset to monitor test subjects when observing positive and negative video stimuli. A second test measured pupil dilation ranges to high and low light frequencies using a high-powered artificial light. This data was converted into a series of algorithms for servomotor triangulation to control the photosensitive and emotive pupil dilation sequences. The robotic eyes were evaluated against the pupillometric data and video feeds of the human eyes to determine operational accuracy. Finally, the dilating robotic eye system was installed in a realistic humanoid robot (RHR) and comparatively evaluated in a human-robot interaction (HRI) experiment. The results of this study show that the robotic eyes can emulate the average pupil reflex of the human eye under typical light conditions and to positive and negative emotive stimuli. However, the results of the HRI experiment indicate that replicating natural eye contact behaviour was more significant than emulating pupil dilation.

Creating and Capturing Artificial Emotions in Autonomous Robots and Software Agents

This paper presents ARTEMIS, a control system for autonomous robots or software agents. ARTEMIS can create human-like artificial emotions during interactions with their environment. We describe the underlying mechanisms for this. The control system also captures its past artificial emotions. A specific interpretation of a knowledge graph, called an Agent Knowledge Graph, stores these artificial emotions. ARTEMIS then utilizes current and stored emotions to adapt decision making and planning processes. As proof of concept, we realize a concrete software agent based on the ARTEMIS control system. This software agent acts as a user assistant and executes their orders and instructions. The environment of this user assistant consists of several other autonomous agents that offer their services. The execution of a user’s orders requires interactions of the user assistant with these autonomous service agents. These interactions lead to the creation of artificial emotions within the user assistant. The first experiments show that it is possible to realize an autonomous user assistant with plausible artificial emotions with ARTEMIS and record these artificial emotions in its Agent Knowledge Graph. The results also show that captured emotions support successful planning and decision making in complex dynamic environments. The user assistant with emotions surpasses an emotionless version of the user assistant.

Cognitive Emotional Interaction Model of Robot Based on Reinforcement Learning

In order to enhance the cognitive emotional computing ability of robot, a cognitive emotional interaction model of robot based on reinforcement learning is proposed, which combines immediate feedback and long-term trend according to PAD(Pleasure-Arousal-Dominance) emotional space. Firstly, according to the psychology theory of interpersonal communication, the human emotion generation process is simulated to generate human-like emotions, and the three influencing factors of similarity, positivity and empathy are extracted. Secondly, the relationship between the response emotion+ state and the contexted long-term emotion state is established by using the global co-ordination feature of reinforcement learning, so as to model the robot emotion generation process. Then, three factors are incorporated into the model reward mechanism for the evaluate of the interactive emotion state, to update the model and get the optimal emotional strategy. Finally, the optimal emotional state corresponding to the obtained optimal emotional strategy is used to update the robot's emotional state transition probability, and based on the sentiment values of the six basic emotional states in space, them are mapped to continuous emotional space to get the optimal response emotional value of the robot. Subjective and objective comparison experiments show that the model in this paper can effectively increase the delicateness, continuity, positivity and empathy of the robot's emotional expression, and can effectively reduce the robot's dependence on external emotional stimuli, further improving the harmonious and friendly human-computer interaction.

A Survey on Emotion Recognition for Human Robot Interaction

With the recent developments of technology and the advances in artificial intelligent and machine learning techniques, it becomes possible for the robot to acquire and show the emotions as a part of Human-Robot Interaction (HRI). An emotional robot can recognize the emotional states of humans so that it will be able to interact more naturally with its human counterpart in different environments. In this article, a survey on emotion recognition for HRI systems has been presented. The survey aims to achieve two objectives. Firstly, it aims to discuss the main challenges that face researchers when building emotional HRI systems. Secondly, it seeks to identify sensing channels that can be used to detect emotions and provides a literature review about recent researches published within each channel, along with the used methodologies and achieved results. Finally, some of the existing emotion recognition issues and recommendations for future works have been outlined.

Survey of Emotions in Human\u2013Robot Interactions: Perspectives from Robotic Psychology on 20 Years of Research

Knowledge production within the interdisciplinary field of human–robot interaction (HRI) with social robots has accelerated, despite the continued fragmentation of the research domain. Together, these features make it hard to remain at the forefront of research or assess the collective evidence pertaining to specific areas, such as the role of emotions in HRI. This systematic review of state-of-the-art research into humans’ recognition and responses to artificial emotions of social robots during HRI encompasses the years 2000–2020. In accordance with a stimulus–organism–response framework, the review advances robotic psychology by revealing current knowledge about (1) the generation of artificial robotic emotions (stimulus), (2) human recognition of robotic artificial emotions (organism), and (3) human responses to robotic emotions (response), as well as (4) other contingencies that affect emotions as moderators.

The future of service: The power of emotion in human-robot interaction

Astoundingly, recent technological advancements have enabled robots to display emotions. Yet, while emotional expression is valued in the field of service, understanding emotions in human-robot interaction remains underexplored. Since emotions are contagious/transmittable, this study utilised Instagram data to uncover how emotional robots influence potential consumers’ affective feelings. By employing machine learning algorithms and sentiment analysis, the findings suggest that the expressions of surprise and happiness are key to creating positive impacts on potential consumers. The cross-disciplinary nature of this study lays the groundwork for next-level social, design, and creative experiences in artificial intelligence research regarding consumer service and experience contexts.

Facial Expression Recognition Algorithm Based on Convolution Neural Network and Multi-Feature Fusion

Facial expression recognition is a hot topic in the field of computer vision. The related research results show high application value in many fields, such as human-computer interaction, intelligent emotional robot, fatigue driving detection, medical health, safety prevention and control, teaching evaluation and so on. However, the huge difference within the expression class still has a prominent impact on the expression recognition, and it is difficult to solve this problem by single feature and traditional feature fusion methods. Therefore, this paper uses neural network and feature fusion strategy to further characterize and classify the constructed features. Experiments show that this method can effectively overcome the interference caused by the differences within the expression class, and achieve ideal expression recognition results.

It\u2019s a Match: Task Assignment in Human\u2013Robot Collaboration Depends on Mind Perception

Robots are becoming more available for workplace collaboration, but many questions remain. Are people actually willing to assign collaborative tasks to robots? And if so, exactly which tasks will they assign to what kinds of robots? Here we leverage psychological theories on person-job fit and mind perception to investigate task assignment in human–robot collaborative work. We propose that people will assign robots to jobs based on their “perceived mind,” and also that people will show predictable social biases in their collaboration decisions. In this study, participants performed an arithmetic (i.e., calculating differences) and a social (i.e., judging emotional states) task, either alone or by collaborating with one of two robots: an emotionally capable robot or an emotionally incapable robot. Decisions to collaborate (i.e., to assign the robots to generate the answer) rates were high across all trials, especially for tasks that participants found challenging (i.e., the arithmetic task). Collaboration was predicted by perceived robot-task fit, such that the emotional robot was assigned the social task. Interestingly, the arithmetic task was assigned more to the emotionally incapable robot, despite the emotionally capable robot being equally capable of computation. This is consistent with social biases (e.g., gender bias) in mind perception and person-job fit. The theoretical and practical implications of this work for HRI are being discussed.

Robotic Telemedicine for Mental Health: A Multimodal Approach to Improve Human-Robot Engagement.

COVID-19 has severely impacted mental health in vulnerable demographics, in particular older adults, who face unprecedented isolation. Consequences, while globally severe, are acutely pronounced in low- and middle-income countries (LMICs) confronting pronounced gaps in resources and clinician accessibility. Social robots are well-recognized for their potential to support mental health, yet user compliance (i.e., trust) demands seamless affective human-robot interactions; natural ‘human-like’ conversations are required in simple, inexpensive, deployable platforms. We present the design, development, and pilot testing of a multimodal robotic framework fusing verbal (contextual speech) and nonverbal (facial expressions) social cues, aimed to improve engagement in human-robot interaction and ultimately facilitate mental health telemedicine during and beyond the COVID-19 pandemic. We report the design optimization of a hybrid face robot, which combines digital facial expressions based on mathematical affect space mapping with static 3D facial features. We further introduce a contextual virtual assistant with integrated cloud-based AI coupled to the robot’s facial representation of emotions, such that the robot adapts its emotional response to users’ speech in real-time. Experiments with healthy participants demonstrate emotion recognition exceeding 90% for happy, tired, sad, angry, surprised and stern/disgusted robotic emotions. When separated, stern and disgusted are occasionally transposed (70%+ accuracy overall) but are easily distinguishable from other emotions. A qualitative user experience analysis indicates overall enthusiastic and engaging reception to human-robot multimodal interaction with the new framework. The robot has been modified to enable clinical telemedicine for cognitive engagement with older adults and people with dementia (PwD) in LMICs. The mechanically simple and low-cost social robot has been deployed in pilot tests to support older individuals and PwD at the Schizophrenia Research Foundation (SCARF) in Chennai, India. A procedure for deployment addressing challenges in cultural acceptance, end-user acclimatization and resource allocation is further introduced. Results indicate strong promise to stimulate human-robot psychosocial interaction through the hybrid-face robotic system. Future work is targeting deployment for telemedicine to mitigate the mental health impact of COVID-19 on older adults and PwD in both LMICs and higher income regions.

Robotic Emotion Recognition Using Two-Level Features Fusion in Audio Signals of Speech

Speech emotion recognition (SER) is a challenging task, since the definition of emotions in sentences is ambiguous. Previous research work mainly focuses on extracting hand-craft features from audio signals to feed into shallow models. Recently, Visual Geometry Group like(VGGish) has replaced traditional feature extractors, due to its effects. VGGish feature vectors were viewed as Deep Neural Network (DNN) selected from a number of features. Although the existing studies on SER have achieved promising results, they only use single-level features. This paper proposes an emotion recognition system, based on speech signals, using two-level features with position information, <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">L</b> ater <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">F</b> eature <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">F</b> usion with <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">V</b> GGish <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">O</b> verlap(LFFVO), to tackle the present limitations. First, the position information, from two-level features, is extracted by Bi-direction Long Short Time Memory (BiLSTM) neural network, followed by features fusion, to predict the emotion. The proposed method improved accuracy from 48.2% (baseline) to 69.5%, when trained, validated and evaluated using an Interactive emotional dyadic motion capture database (IEMOCAP).

Feeling for Our Robot Overlords: Perceptions of Emotionally Expressive Social Robots in Initial Interactions

ABSTRACT Human-to-human scripting accounts for a user’s tendency in human-robot interactions (HRI) to utilize scripts just as they would if talking to another person. Can a robot’s emotional verbal expression, however, impact perceptions and further elicit human-to-human scripting? The present study examines the effects of a robot’s emotional expression of joy and sorrow on participant perceptions of anthropomorphism, animacy, likability, perceived intelligence, credibility, social presence, and uncertainty in an initial interaction. Results indicated that robot expressions of joy were rated significantly more likable, intelligent, credible, and socially present than expressions of sorrow. Implications for future research are discussed in light of our findings related to participant impressions of emotional robots and human-to-human scripting.

Emotion space modelling for social robots

Improving the interaction between artificial systems and their users is an important issue in artificial intelligence. In current trends in social robotics, this is accomplished by making such systems not just intelligent but also emotionally sensitive. Therefore, artificial emotional intelligence (AEI) is focused on simulating and extending natural emotion (especially human emotion) to provide robots with the capability to recognise and express emotions in human–robot interaction (HRI). In this treatise, we present an overview of the advances made in AEI by highlighting the progress recorded in the areas of emotion classification, emotional robots, and emotion space modelling in HRI. This review is aimed at providing readers with a succinct, yet adequate compendium of the progresses made in the emerging AEI sub-discipline. It is hoped this effort will stimulate further interest aimed at the pursuit of more advanced algorithms and models for available technologies and possible applications to other domains.

Emotional Intelligence Robotics to Motivate Interaction in E-Learning: An Algorithm

The development of emotional intelligence robotics in the learning environment plays valuable support for social interaction among students. Emotional intelligence robots should be scalable to recognize emotions, appear empathetic in learning situations, and enrich the confidence with students for active interaction. This paper presents some related issues about integrating emotional intelligence robotics in E-learning such as its role and outcomes to motivate interaction during education and discover the main aspects of the emotional intelligence between humans and robots. This paper aims to determine the design requirements of emotional robots. Besides, this paper proposed a framework of educational Robotics with Emotional Intelligence in Learning(EREIL). EREIL consists of three main units; student emotions discovery, student emotions representation, and EREIL-Student Communication (RSC). In addition, it introduces a perception of EREIL working. In the future, this paper tries to merge more sensor devices and machine learning algorithms to integrate face analysis with speech recognition. Besides, it can add a persuasion unit in the EREIL robot to convince students with better learning choices to their abilities.

Modeling the Timing and Duration of Grip Behavior to Express Emotions for a Social Robot

This letter addresses the effects of grip timing to express a robot's emotions to people while they are watching a video stimulus together. Past studies on human-robot touch interaction focused on the types of touch behaviors for expressing a robot's emotion, but timing factors have received less attention. In this letter, we conducted data collection to investigate the appropriate grip timing to express heartwarming and horror emotions. Participants identified touch (grip) timing and durations by using a robot while they are watching video stimuli. Typically, they preferred a grip timing before a climax for horror videos but after a climax for heartwarming videos. We modeled the timing and durations by a fitting approach to probabilistic distribution to reproduce human-like touch behaviors, and also we implemented the models to an android robot.