Emotional Robot Research Articles

Evaluation of Robot Emotion Expressions for Human–Robot Interaction

Evaluation of Robot Emotion Expressions for Human–Robot Interaction

Crafting for Emotion Appropriateness in Affective Robotics: Examining the Practicality of the OCC Model

Research in affective robotics has been using emotions to improve human-robot interaction. One important aspect has been to design recognizable and believable emotions in robotics. Recent work argued that externally displayed emotions on robots may or may not be appropriate for a given situation. However, the selection of emotions as appropriate/inappropriate is not trivial. We here examine the practicality of an established model to craft for emotion appropriateness based on situations of interaction. To do so, we explored the use of the Ortony, Clore, and Collins (OCC) model, which provides a psychological framework of appraisal in which the characteristics of situations are defined and connected to emotions, to identify emotion categories and create contrasting perceptions of emotion appropriateness. We then mapped these categories to four recognizable emotions on aerial robots and designed two video clips (3min35s each) of respectively appropriate and inappropriate emotions. The clips were evaluated in an online study ( N =100) where significant differences were found in attitudes toward the robot's emotions. This paper contributes initial findings to designing for emotion appropriateness.

Coupled Multimodal Emotional Feature Analysis Based on Broad-Deep Fusion Networks in Human-Robot Interaction.

A coupled multimodal emotional feature analysis (CMEFA) method based on broad-deep fusion networks, which divide multimodal emotion recognition into two layers, is proposed. First, facial emotional features and gesture emotional features are extracted using the broad and deep learning fusion network (BDFN). Considering that the bi-modal emotion is not completely independent of each other, canonical correlation analysis (CCA) is used to analyze and extract the correlation between the emotion features, and a coupling network is established for emotion recognition of the extracted bi-modal features. Both simulation and application experiments are completed. According to the simulation experiments completed on the bimodal face and body gesture database (FABO), the recognition rate of the proposed method has increased by 1.15% compared to that of the support vector machine recursive feature elimination (SVMRFE) (without considering the unbalanced contribution of features). Moreover, by using the proposed method, the multimodal recognition rate is 21.22%, 2.65%, 1.61%, 1.54%, and 0.20% higher than those of the fuzzy deep neural network with sparse autoencoder (FDNNSA), ResNet-101 + GFK, C3D + MCB + DBN, the hierarchical classification fusion strategy (HCFS), and cross-channel convolutional neural network (CCCNN), respectively. In addition, preliminary application experiments are carried out on our developed emotional social robot system, where emotional robot recognizes the emotions of eight volunteers based on their facial expressions and body gestures.

Beyond Explicit Acknowledgment: Brain Response Evidence of Human Skepticism towards Robotic Emotions

Using the N400 component of event-related brain potentials, a neurophysiological marker associated with processing incongruity, we examined brain responses to sentences spoken by a robot that had no arms or legs. Statements concerning physically impossible actions (e.g., knitting) elicit significant N400 responses, reflecting that participants perceived these statements as incongruent with the robot’s physical condition. However, this effect was attenuated for participants who indicated that the robot could have hidden limbs, indicating that expectations modify the way an agent’s utterances are interpreted. When it came to statements relating to emotional capabilities a distinct pattern was found. Although participants acknowledged that the robot could have emotions, there were significant N400 responses to statements about the robot’s emotional experiences (e.g., feeling happy). This effect was not modified by participants’ beliefs, suggesting a cognitive challenge of accepting robots as capable of experiencing emotions. Our findings thus point to a boundary in human acceptance of artificial social agents: while physical attributes may be negotiable based on expectations, emotional expressions are more difficult to establish as credible. By elucidating the cognitive mechanisms at play, our study informs the design of social robots that are capable of more effective communication to better support social connectivity and human well-being.

“I See What You Feel”: An Exploratory Study to Investigate the Understanding of Robot Emotions in Deaf Children

Research in the field of human–robot interactions (HRIs) has advanced significantly in recent years. Social humanoid robots have undergone severe testing and have been implemented in a variety of settings, for example, in educational institutions, healthcare facilities, and senior care centers. Humanoid robots have also been assessed across different population groups. However, research on various children groups is still scarce, especially among deaf children. This feasibility study explores the ability of both hearing and deaf children to interact with and recognize emotions expressed by NAO, the humanoid robot, without relying on sounds or speech. Initially, the children watched three video clips portraying emotions of happiness, sadness, and anger. Depending on the experimental condition, the children observed the humanoid robot respond to the emotions in the video clips in a congruent or incongruent manner before they were asked to recall which emotion the robot exhibited. The influence of empathy on the ability to recognize emotions was also investigated. The results revealed that there was no difference in the ability to recognize emotions between the two conditions (i.e., congruent and incongruent). Indeed, NAO responding with congruent emotions to video clips did not contribute to the children recognizing the emotion in NAO. Specifically, the ability to predict emotions in the video clips and gender (females) were identified as significant predictors to identify emotions in NAO. While no significant difference was identified between hearing and deaf children, this feasibility study aims to establish a foundation for future research on this important topic.

Investigating Emotional Impressions in Robots Using Clothing Colors

Expressing emotions is essential for ensuring smooth communication between people. In the context of human-robot symbiosis, robots are also required to express emotions. Although one method of robot emotion expression involves using LEDs or other forms of light to display colors, considering the possibility of expressing emotions through clothing colors is also necessary. In this study, we developed a simple robot called the “Tilting Robot,” which only performs simple tilting motions to investigate whether changes in the robot’s clothing color would affect the expressed emotions. In the experiment, participants were divided into two groups: motion and posture groups. The motion group was shown videos of the robot’s motion whereas the posture group was shown still images of the robot’s posture. The results showed that the red clothing in the posture group significantly expressed anger, whereas the blue clothing in the motion group significantly expressed sadness. The rating for blue clothing was 4.04 ± 1.30, which was near “undecided.” This suggests that blue clothing does not necessarily intensify the emotion of sadness, but other clothing colors may weaken its expression. The rating for red clothing was 2.86 ± 1.06, which was lower than “undecided.” This suggests that red clothing may not express anger, but could give an impression of vitality.

Influence of Watching TV Programs on Human Recognition of a Robot's Emotions

Influence of Watching TV Programs on Human Recognition of a Robot's Emotions

Memristor-Based Neural Network Circuit of Associative Memory With Overshadowing and Emotion Congruent Effect.

Most memristor-based neural network circuits consider only a single pattern of overshadowing or emotion, but the relationship between overshadowing and emotion is ignored. In this article, a memristor-based neural network circuit of associative memory with overshadowing and emotion congruent effect is designed, and overshadowing under multiple emotions is taken into account. The designed circuit mainly consists of an emotion module, a memory module, an inhibition module, and a feedback module. The generation and recovery of different emotions are realized by the emotion module. The functions of overshadowing under different emotions and recovery from overshadowing are achieved by the inhibition module and the memory module. Finally, the blocking caused by long-term overshadowing is implemented by the feedback module. The proposed circuit can be applied to bionic emotional robots and offers some references for brain-like systems.

Real-time emotion generation in human-robot dialogue using large language models.

Affective behaviors enable social robots to not only establish better connections with humans but also serve as a tool for the robots to express their internal states. It has been well established that emotions are important to signal understanding in Human-Robot Interaction (HRI). This work aims to harness the power of Large Language Models (LLM) and proposes an approach to control the affective behavior of robots. By interpreting emotion appraisal as an Emotion Recognition in Conversation (ERC) tasks, we used GPT-3.5 to predict the emotion of a robot's turn in real-time, using the dialogue history of the ongoing conversation. The robot signaled the predicted emotion using facial expressions. The model was evaluated in a within-subjects user study (N = 47) where the model-driven emotion generation was compared against conditions where the robot did not display any emotions and where it displayed incongruent emotions. The participants interacted with the robot by playing a card sorting game that was specifically designed to evoke emotions. The results indicated that the emotions were reliably generated by the LLM and the participants were able to perceive the robot's emotions. It was found that the robot expressing congruent model-driven facial emotion expressions were perceived to be significantly more human-like, emotionally appropriate, and elicit a more positive impression. Participants also scored significantly better in the card sorting game when the robot displayed congruent facial expressions. From a technical perspective, the study shows that LLMs can be used to control the affective behavior of robots reliably in real-time. Additionally, our results could be used in devising novel human-robot interactions, making robots more effective in roles where emotional interaction is important, such as therapy, companionship, or customer service.

Mathematical Models of Dynamics of Emotional Robots’ Interest in Media Projects

Mathematical Models of Dynamics of Emotional Robots’ Interest in Media Projects

Do Emotional Robots Get More Help? How a Robots Emotions Affect Collaborators Willingness to Help

Do Emotional Robots Get More Help? How a Robots Emotions Affect Collaborators Willingness to Help

Socially Assistive Robots: Considerations for Negative Effects on Empathy and Emotion Recognition in Human Interaction

Socially assistive robots (SARs) are a recent development in the field of artificial intelligence designed to assist human users through social interactions in vulnerable settings. SARs design has led to significant emotion recognition and empathy research in order to create sufficient and effective userrobot bonds. However, much of this research has focused on the successful emulation of emotion and empathy in robots, rather than the effects of prolonged exposure to such artificial emotion and empathy on subsequent human-to-human empathetic interaction and emotion recognition. In this paper, we present a review of interpersonal empathy in the context of human-robot relationships and discuss the potential consequences of these relationships on human-human social interactions. We also present recommendations for research into this field, drawing from real-life examples of humans bonding to inanimate artificial beings.

Children’s Digital and Non-Digital Play Practices with Cozmo, the Toy Robot

Children’s Digital and Non-Digital Play Practices with Cozmo, the Toy Robot

Designing Behaviors of Robots Based on the Artificial Emotion Expression Method in Human–Robot Interactions

How to express emotions through motion behaviors of robots (mainly for robotic arms) to achieve human–robot emotion interactions is the focus of this paper. An artificial emotion expression method that accords with human emotion that can deal with external stimuli and has the capability of emotion decision-making was proposed based on the motion behaviors of robot. Firstly, a three-dimensional emotion space was established based on the motion indexes (deviation coefficient, acceleration, and interval time). Then, an artificial emotion model, which was divided into three parts (the detection and processing of external events, the generation and modification of emotion response vectors, and the discretization of emotions) was established in the three-dimensional emotion space. Then emotion patterns (love, excited, happy, anxiety, hate) and emotion intensity were calculated based on the artificial emotion model in human–robot interaction experiments. Finally, the influence of motion behaviors of humanoid robot NAO on the emotion expression of experimenters was studied through human–robot emotion interaction experiments based on the emotion patterns and emotion intensity. The positive emotion patterns (love, excited, happy) and negative emotion patterns (anxiety, hate) of the experimenters were evaluated. The experimental results showed that the personalized emotion responses could be generated autonomously for external stimuli, and the change process of human emotions could be simulated effectively according to the established artificial emotion model. Furthermore, the experimenters could recognize the emotion patterns expressed by the robot according to the motion behaviors of the robot, and whether experimenters were familiar with robots did not influence the recognition of different emotion patterns.

Research on the influence of the baby schema effect on the cuteness and trustworthiness of social robot faces

Exploring people’s attitudes toward the appearance design of social robots in a low-cost and efficient way, and enhancing the experience of human–robot interaction have always been topics of concern for robot developers and interaction designers. This study aimed to explore the influence of the baby schema effect on users’ perceptions of cuteness and trustworthiness pertinent to social robot faces through two experiments. Experiment 1 used 100 uniformly processed pictures of robot faces in the real world to help explore the linear relationship among the degree of baby face, cuteness, and trustworthiness, and received a total of 98 valid questionnaires via the Internet. Experiment 2 was a 5 × 3 within-subjects factorial design. The research variables were robot type (i.e. MAKI, RoboThespian, Flobi, Pepper, and iCat) and baby schema (low schema, uncontrolled, and high schema); their impact on users’ perceptions of cuteness and trustworthiness was investigated. A total of 175 valid questionnaires were collected via the Internet. The generated results are as follows: (1) The degree of baby face and perceived emotion of social robot faces had a positive impact on trustworthiness for most real-world robots. (2) This study obtained the correlation formula of baby face, cuteness, and trustworthiness from a quantitative point of view, thus providing a reference for research on the related credibility of communication robots. (3) In general, baby schema effect also existed in the cuteness evaluation of most real-world robots. Faces with high schema were considered cuter and more trustworthy than uncontrolled or low schema faces. (4) Robot type and baby schema had a significant interaction with cuteness and trustworthiness. (5) However, for certain types of robots, baby schema effect may also have a counter-effect, that is, an overly high baby schema may reduce users’ perceptions of the cuteness and trustworthiness of social robots.

Correction to: Utilizing an Emotional Robot Capable of Lip-Syncing in Robot-Assisted Speech Therapy Sessions for Children with Language Disorders

Correction to: Utilizing an Emotional Robot Capable of Lip-Syncing in Robot-Assisted Speech Therapy Sessions for Children with Language Disorders

Older adult perspectives on emotion and stigma in social robots.

Social robot adoption by older adults and people living with dementia is limited by several barriers, including a lack of emotional alignment with social robots and perceptions of stigma around social robot use. The goal of this work was to address these barriers by engaging end-users in discussions on factors that could affect emotional connection to a social robot and considerations around using a social robot in public. We conducted seven 90-min online workshops with older adults with (n = 2) and without dementia (n = 25) and care partners (n = 17). Older adults with and without dementia were in their 50s - 80s, and care partners were in their 30s - 70s. Seven older adults, seven care partners, and one older adult with dementia indicated that they had used a robot before. Online workshops had 4-8 participants each, and began with video demonstrations of pet-like robot MiRo and tabletop robot T-Top, as well as a live demonstration of MiRo. Participants completed the Multi-Dimensional Robot Attitude Scale before the workshops and the Psychosocial Impact of Assistive Devices Scale and two Zoom polls during the workshops. The audio and chat responses from the workshops were transcribed verbatim and content analysis was performed on the transcripts. Our analysis revealed three broad themes and 10 subthemes. In their discussions on features and applications, participants highlighted preferred forms of communication with a robot and ways in which a robot could support connection between people. For example, robots could improve the quality of communication between care partners and the person for whom they care. While many agreed that a social robot should match their level of emotion and interactivity, participants had different preferences for social robot emotional range and display features. Finally, participants discussed considerations around showing a robot to other people; several participants suggested that a robot could help raise awareness of ageing and dementia while others shared concerns about stigma and attracting negative attention from an audience. Incorporating these findings into the design and implementation of social robots will result in devices that are better-suited to the needs of older adults, people living with dementia, and care partners.

コミュニケーションロボットの着衣色による感情表出手法の提案

In recent years, communication robots are commonly used in various purposes and situations such as guidance services at hotels and stations, elderly support, etc. in Japan. However, it is also shown that people still have a sense of anxiety toward robots. For this reason, there are few people who have motivations to communicate with robots. On the other hand, it has been suggested that the emotional expression of robots contributes to create a sense of familiarity and smooth communication. Therefore, we propose a method of emotional expression and enhancement by changing the color of the clothes the robot is wearing. In this research, we have matched the clothing color with speech emotion for communication robot. As a result of the experiment, we have found the matching between the clothing color and 10 emotions (joy, anger, sorrow, fear, shame, like, dislike, excite, relief and surprise).

Assessing the Applicability of Machine Learning Models for Robotic Emotion Monitoring: A Survey

Emotion monitoring can play a vital role in investigating mental health disorders that contribute to 14% of global diseases. Currently, the mental healthcare system is struggling to cope with the increasing demand. Robot-assisted mental health monitoring tools can take the enormous strain off the system. The current study explored existing state-of-art machine learning (ML) models and signal data from different bio-sensors assessed the suitability of robotic devices for surveilling different physiological and physical traits related to human emotions and discussed their potential applicability for mental health monitoring. Among the selected 80 articles, we subdivided our findings in terms of two different emotional categories, namely—discrete and valence-arousal (VA). By examining two different types of signals (physical and physiological) from 10 different signal sources, we found that RGB images and CNN models outperformed all other data sources and models, respectively, in both categories. Out of the 27 investigated discrete imaging signals, 25 reached higher than 80% accuracy, while the highest accuracy was observed from facial imaging signals (99.90%). Besides imaging signals, brain signals showed better potentiality than other data sources in both emotional categories, with accuracies of 99.40% and 96.88%. For both discrete and valence-arousal categories, neural network-based models illustrated superior performances. The majority of the neural network models achieved accuracies of over 80%, ranging from 80.14% to 99.90% in discrete, 83.79% to 96.88% in arousal, and 83.79% to 99.40% in valence. We also found that the performances of fusion signals (a combination of two or more signals) surpassed that of the individual ones in most cases, showing the importance of combining different signals for future model development. Overall, the potential implications of the survey are discussed, considering both human computing and mental health monitoring. The current study will definitely serve as the base for research in the field of human emotion recognition, with a particular focus on developing different robotic tools for mental health monitoring.

Nursing and human-computer interaction in healthcare robots for older people: An integrative review.

This study examined the published works related to healthcare robotics for older people using the attributes of health, nursing, and the human-computer interaction framework. An integrative literature review. A search strategy captured 55 eligible articles from databases (CINAHL, Embase, IEEE Xplore, and PubMed) and hand-searching approaches. Bibliometric and content analyses grounded on the health and nursing attributes and human-computer interaction framework were performed using MAXQDA. Finally, results were verified using critical friend feedback by a second reviewer. Most articles were from multiple authorship, published in non-nursing journals, and originating from developed economies. They primarily focused on applying healthcare robots in practice settings, physical health, and communication tasks. Using the human-computer interaction framework, it was found that older adults frequently served as the primary users while nurses, healthcare providers, and researchers functioned as secondary users and operators. Research articles focused on the usability, functionality, and acceptability of robotic systems. At the same time, theoretical papers explored the frameworks and the value of empathy and emotion in robots, human-computer interaction and nursing models and theories supporting healthcare practice, and gerontechnology. Current robotic systems are less anthropomorphic, operated through real-time direct and supervisory inputs, and mainly equipped with visual and auditory sensors and actuators with limited capability in performing health assessments. Results communicate the need for technological competency among nurses, advancements in increasing healthcare robot humanness, and the importance of conscientious efforts from an interdisciplinary research team in improving robotic system usability and utility for the care of older adults.