Nonverbal Emotion Recognition Research Articles

Impact of social anxiety on communication skills in face-to-face vs. online contexts

Individuals with social anxiety have been shown to prefer communicating through online platforms. Part of this preference may be accounted for by their self-perceived level of social skill ability in online modalities. However, it is unclear whether perceived social skill abilities change across in-person and online contexts. Therefore, this study investigated whether specific social skills, like sociability, assertiveness, self-disclosure, and non-verbal emotion recognition vary across online and face-to-face settings. We first validated the Real and Electronic Communications Skills (RECS) questionnaire using a confirmatory factor analysis in a sample of 780 participants, who completed the survey through Qualtrics. We then conducted a series of correlations and a doubly multivariate GLM to evaluate whether perceived social skills abilities are influenced by communication medium and social anxiety. Our results indicated that we were able to validate the RECS, and that specific social skills like sociability and assertiveness are influenced by communication medium. Specifically, higher social anxiety scores were associated with decreased in-person sociability, emotion recognition, and assertiveness. The results highlight a potential perceived barrier to approaching in-person interactions, which may exacerbate avoidance of social interaction, thus resulting in social isolation and loneliness.

Speech Emotion Recognition Applied to Real-World Medical Consultation.

Since 2020, the COVID-19 epidemic has changed our lives in healthcare behaviors. Forced to wear masks influenced doctor-patient interaction perceptions truly, thus, to build a satisfying relationship is not just empathize with facial expressions. The voice becomes more important for the sake of conquering the burden of masks. Hence, verbal and non-verbal communication will be crucial criteria for doctor-patient interaction during medical consultations and other conversations. In these years, speech emotion recognition has been a popular research domain. In spite of abundant work conducted, nonverbal emotion recognition in medical scenarios is still required to reveal. In this study, we investigate YAMNet transfer learning on Chinese Mandarin speech corpus NTHU-NTUA Chinese Interactive Emotion Corpus (NNIME) and use real-world dermatology clinic recording to test the generalization capability. The results showed that the accuracy validated on NNIME data was 0.59 for activation prediction and 0.57 for valence. Furthermore, the validation accuracy on the doctor-patient dataset was 0.24 for activation and 0.58 for valence, respectively.

Trainee psychotherapists’ emotion recognition accuracy during 1.5 years of psychotherapy education compared to a control group: no improvement after psychotherapy training

The ability to recognize and work with patients’ emotions is considered an important part of most psychotherapy approaches. Surprisingly, there is little systematic research on psychotherapists’ ability to recognize other people’s emotional expressions. In this study, we compared trainee psychotherapists’ nonverbal emotion recognition accuracy to a control group of undergraduate students at two time points: at the beginning and at the end of one and a half years of theoretical and practical psychotherapy training. Emotion recognition accuracy (ERA) was assessed using two standardized computer tasks, one for recognition of dynamic multimodal (facial, bodily, vocal) expressions and one for recognition of facial micro expressions. Initially, 154 participants enrolled in the study, 72 also took part in the follow-up. The trainee psychotherapists were moderately better at recognizing multimodal expressions, and slightly better at recognizing facial micro expressions, than the control group at the first test occasion. However, mixed multilevel modeling indicated that the ERA change trajectories for the two groups differed significantly. While the control group improved in their ability to recognize multimodal emotional expressions from pretest to follow-up, the trainee psychotherapists did not. Both groups improved their micro expression recognition accuracy, but the slope for the control group was significantly steeper than the trainee psychotherapists’. These results suggest that psychotherapy education and clinical training do not always contribute to improved emotion recognition accuracy beyond what could be expected due to time or other factors. Possible reasons for that finding as well as implications for the psychotherapy education are discussed.

Social cognitive deficit is associated with visuomotor coordination impairment and dopamine transporter availability in euthymic bipolar disorder

BackgroundExtensive evidence has suggested functional connections between co-occurring visuomotor and social cognitive deficits in neuropsychiatric disorders; however, such association has not been studied in bipolar disorder (BD). We aimed to investigate the relationship between visuomotor coordination and social cognition in the euthymic stage of BD (euBD). Given the shared neurobiological underpinnings involving the dopaminergic system and corticostriatal circuitry, we hypothesized a positive correlation between social cognition and visuomotor coordination in euBD patients. Methods40 euBD patients and 59 healthy control (HC) participants underwent evaluation of social (Diagnostic Analysis of Nonverbal Accuracy 2-Taiwan version (DANVA-2-TW)), non-social cognitive function and visuomotor coordination. A subgroup of participants completed single-photon emission computed tomography for striatal dopamine transporter (DAT) availability assessment. ResultsEuBD patients showed impaired nonverbal emotion recognition (ps ≤ 0.033) and poorer visuomotor coordination (ps < 0.003) compared to HC, with a positive correlation between these two abilities (r = 0.55, p < 0.01). However, after considering potential confounding factors, instead of visuomotor coordination, striatal DAT availability was a unique predictor of emotion recognition accuracy in euBD (beta = 0.33, p = 0.001). ConclusionOur study result supported a functional association between social cognition and visuomotor coordination in euBD, with striatal dopaminergic dysfunction emerged as a crucial contributing factor in their interrelation.

个体非言语情绪识别研究综述

个体非言语情绪识别研究综述

Recognition of Emotion from Verbal and Nonverbal Expressions and Its Relation to Effective Communication: A Preliminary Evidence of a Positive Link.

Previous work has shown that emotion recognition is positively related to effective social interactions, but the mechanism underlying this relationship has remained largely unclear. Here, we examined the possibility that people who understand others' emotions also talk to them using similar language. In the current study participants (N = 106) listened to emotional stories people shared from their own lives. They were later asked to recognize the storytellers' feelings and finally provide written support messages. Perceivers' ability to accurately recognize others' feelings was assessed using the Emotional Accuracy Test (EAT), which uses naturalistic verbal and nonverbal emotional cues, and using two standard tests of nonverbal emotion recognition (GERT, RMET). The language of the expressor (target) was compared to the language of the supporter (participant) to quantify Language Style Matching, a proxy for effective communication. People who perform better in emotion recognition with verbal cues (EAT) also communicate their understanding and support using language similar to the expresser (r = .22, p = .02). This relation was insignificant for tests without verbal information (RMET, GERT). The result provides additional construct validation for the EAT and supports the view that understanding the emotions of others and communicating with them are two manifestations of a broader interpersonal skill.

Facial emotion processing and language during early-to-middle childhood development: An event related potential study

Facial emotion processing (FEP) is critical to social cognitive ability. Developmentally, FEP rapidly improves in early childhood and continues to be fine-tuned throughout middle childhood and into adolescence. Previous research has suggested that language plays a role in the development of social cognitive skills, including non-verbal emotion recognition tasks. Here we investigated whether language is associated with specific neurophysiological indicators of FEP. One hundred and fourteen children (4–12 years) completed a language assessment and a FEP task including stimuli depicting anger, happiness, fear, and neutrality. EEG was used to record key event related potentials (ERPs; P100, N170, LPP at occipital and parietal sites separately) previously shown to be sensitive to faces and facial emotion. While there were no main effects of language, the P100 latency to negative expressions appeared to increase with language, while LPP amplitude increased with language for negative and neutral expressions. These findings suggest that language is linked to some early physiological indicators of FEP, but this is dependent on the facial expression. Future studies should explore the role of language in later stages of neural processing, with a focus on processes localised to ventromedial prefrontal regions.

Training Emotion Recognition Accuracy: Results for Multimodal Expressions and Facial Micro Expressions.

Nonverbal emotion recognition accuracy (ERA) is a central feature of successful communication and interaction, and is of importance for many professions. We developed and evaluated two ERA training programs—one focusing on dynamic multimodal expressions (audio, video, audio-video) and one focusing on facial micro expressions. Sixty-seven subjects were randomized to one of two experimental groups (multimodal, micro expression) or an active control group (emotional working memory task). Participants trained once weekly with a brief computerized training program for three consecutive weeks. Pre-post outcome measures consisted of a multimodal ERA task, a micro expression recognition task, and a task about patients' emotional cues. Post measurement took place approximately a week after the last training session. Non-parametric mixed analyses of variance using the Aligned Rank Transform were used to evaluate the effectiveness of the training programs. Results showed that multimodal training was significantly more effective in improving multimodal ERA compared to micro expression training or the control training; and the micro expression training was significantly more effective in improving micro expression ERA compared to the other two training conditions. Both pre-post effects can be interpreted as large. No group differences were found for the outcome measure about recognizing patients' emotion cues. There were no transfer effects of the training programs, meaning that participants only improved significantly for the specific facet of ERA that they had trained on. Further, low baseline ERA was associated with larger ERA improvements. Results are discussed with regard to methodological and conceptual aspects, and practical implications and future directions are explored.

Association of visual motor processing and social cognition in schizophrenia

Patients with schizophrenia have difficulties in social cognitive domains including emotion recognition and mentalization, and in sensorimotor processing and learning. The relationship between social cognitive deficits and sensorimotor function in patients with schizophrenia remains largely unexplored. With the hypothesis that impaired visual motor processing may decelerate information processing and subsequently affects various domains of social cognition, we examined the association of nonverbal emotion recognition, mentalization, and visual motor processing in schizophrenia. The study examined mentalization using the verbal subset of the Chinese version of Theory of Mind (CToM) Task, an equivalent task of the Faux Pas Test; emotion recognition using the Diagnostic Analysis of Nonverbal Accuracy 2-Taiwan version (DANVA-2-TW), and visual motor processing using a joystick tracking task controlled for basic motor function in 34 individuals with chronic schizophrenia in the community and 42 healthy controls. Patients with schizophrenia had significantly worse performance than healthy controls in social cognition, including facial, prosodic emotion recognition, and mentalization. Visual motor processing was also significantly worse in patients with schizophrenia. Only in patients with schizophrenia, both emotion recognition (mainly in prosodic modality, happy, and sad emotions) and mentalization were positively associated with their learning capacity of visual motor processing. These findings suggest a prospective role of sensorimotor function in their social cognitive deficits. Despite that the underlying neural mechanism needs further research, our findings may provide a new direction for restoration of social cognitive function in schizophrenia by enhancing visual motor processing ability.

Errors in identifying emotion in body postures and facial expressions among pediatric patients with bipolar disorder

ABSTRACT Introduction Facial emotion processing impairments have been consistently demonstrated among patients with pediatric bipolar disorder (PBD). However, less is known about other domains of nonverbal emotion recognition in this group. Therefore, the aim of the current study was to investigate emotion identification for body postures in addition to facial expressions among PBD patients and matched healthy comparison (HC) subjects. Method Youth with PBD (n = 25) and HC subjects (n = 25), equal on intellectual and demographic variables, completed the body postures and faces subtests of the Diagnostic Analysis of Nonverbal Accuracy-2 scale (DANVA-2) along with diagnostic and clinical assessments. Results Compared to HC, PBD patients made significantly more errors identifying emotion in both body postures and faces. Poorer performance on the faces subtest was associated with more problematic peer functioning. Greater manic symptoms were associated with poorer performance on negative emotional stimuli but not positive. Conclusions In addition to impairments in facial emotion processing, youth with PBD also show impairments in the ability to correctly identify emotion from body postures. In this group, greater manic symptomatology appears to impair processing for negative facial expressions and body postures, but not positive ones. Interestingly, facial emotion processing impairments appear to be a better indicator of real-world psychosocial difficulties among PBD patients compared to body postures. Psychosocial treatments for youth with PBD should focus on improving the ability to accurately identify and respond to various facial and body postural cues, as well as symptom management and emotion regulation strategies.

ОСОБЛИВОСТІ ЕКСПРЕСІЇ ОСІБ З РІЗНОЮ СХИЛЬНІСТЮ ДО АЛЕКСИТИМІЇ

Статтю присвячено проблемі розпізнавання власних емоційних переживань та їх зовнішніх виразів. Схильність особистості до розпізнавання емоцій через сприйняття експресії розглядається як усталена, багатокомпонентна, інтегральна властивість, що забезпечує точність, адекватність розуміння своїх емоційних переживань і переживань іншого. Актуальність розробки проблеми пов’язана з дефіцитом соціальних зв’язків, відсутністю досвіду в розпізнаванні емоцій, особливо в ситуації розвитку Інтернет-Комунікацій, що призводить до редукції емоційної лексики сучасної людини. Процес соціальної адаптації неможливий без здатності особистості розуміти та диференціювати емоції. Це спричинює необхідність вивчення здатності особистості до адекватного визначення власних емоційних станів і розуміння власного експресивного репертуару. Дослідження ґрунтується на припущенні, що труднощі розпізнавання власних емоцій та їх експресивних виявів можуть бути пов’язані з різною схильністю особистості до алекситимії. Алекситимія розглядається як нездатність сприймати, відчувати і називати словом (lexis) свої почуття, свої емоції (thymos).Мета роботи полягає у дослідження здатності до розуміння та розпізнавання власних експресивних проявів емоцій в осіб, котрі різняться схильністю до алекситимії. Розкрито поняття алекситимії як чинника розуміння емоцій; вивчено співвідношення між показниками означених змінних; виявлено осіб, схильних і не схильних до алекситимії, та визначено й порівняно особливості їх експресії. Емпірично встановлена наявність значимого від’ємного зв’язку між показниками виразної експресії та алекситимії.Показано, що тенденція до збільшення рівня алекситимії гальмує здатність особистості до експресивних проявів власних емоційних переживань, а прояви успішного розпізнавання власної експресії демонструють особи, котрі не виявили схильності до алекситимії.Доведено, що схильність до алекситимії є одним із психологічних чинників здатності особистості не тільки розуміти власні емоції, а й здатності ідентифікувати і використовувати власний експресивний репертуар, його можливості та ресурси.

Two facets of affective empathy: concern and distress have opposite relationships to emotion recognition

ABSTRACT Theories on empathy have argued that feeling empathy for others is related to accurate recognition of their emotions. Previous research that tested this assumption, however, has reported inconsistent findings. We suggest that this inconsistency may be due to a lack of consideration of the fact that empathy has two facets: empathic concern, namely the compassion for unfortunate others, and personal distress, the experience of discomfort in response to others’ distress. We test the hypothesis that empathic concern is positively related to emotion recognition, whereas personal distress is negatively related to emotion recognition. Individual tendencies to respond with concern or distress were measured with the standard IRI (Interpersonal Reactivity Index) self-report questionnaire. Emotion recognition performance was assessed with three standard tests of nonverbal emotion recognition. Across two studies (total N = 431) anddifferent emotion recognition tests, we found that these two facets of affective empathy have opposite relations to recognition of facial expressions of emotions: empathic concern was positively related, while personal distress was negatively related, to accurate emotion recognition. These findings fit with existing motivational models of empathy, suggesting that empathic concern and personal distress have opposing impacts on the likelihood that empathy makes one a better emotion observer.

How Well Can We Assess Our Ability to Understand Others' Feelings? Beliefs About Taking Others' Perspectives and Actual Understanding of Others' Emotions.

People vary in their beliefs about their tendency to engage in perspective taking and to understand other’s feelings. Often, however, those beliefs are suggested to be poor indicators of actual skills and thus provide an inaccurate reflection of performance. Few studies, however, have examined whether people’s beliefs accurately predict their performance on emotion recognition tasks using dynamic or spontaneous emotional expressions. We report six studies (N ranges from 186 to 315; Ntotal = 1,347) testing whether individuals’ report of their engagement in perspective taking, as measured by the Interpersonal Reactivity Index (IRI; Davis, 1983), is associated with accurate emotion recognition. In Studies 1–3, emotion recognition performance was assessed using three standard tests of nonverbal emotion recognition. To provide a more naturalistic test, we then assessed performance with a new emotion recognition test in Studies 4–6, using videos of real targets that share their emotional experiences. Participants’ multi-scalar ratings of the targets’ emotions were compared with the targets’ own emotion ratings. Across all studies, we found a modest, yet significant positive relationship: people who believe that they take the other’s perspective also perform better in tests of emotion recognition (r = 0.20, p < 0.001). Beliefs about taking others’ perspective thus reflect interpersonal reality, but only partially.

Facial and prosodic emotion recognition in social anxiety disorder

ABSTRACTIntroduction: Patients with social anxiety disorder (SAD) have a cognitive preference to negatively evaluate emotional information. In particular, the preferential biases in prosodic emotion recognition in SAD have been much less explored. The present study aims to investigate whether SAD patients retain negative evaluation biases across visual and auditory modalities when given sufficient response time to recognise emotions.Methods: Thirty-one SAD patients and 31 age- and gender-matched healthy participants completed a culturally suitable non-verbal emotion recognition task and received clinical assessments for social anxiety and depressive symptoms. A repeated measures analysis of variance was conducted to examine group differences in emotion recognition.Results: Compared to healthy participants, SAD patients were significantly less accurate at recognising facial and prosodic emotions, and spent more time on emotion recognition. The differences were mainly driven by the lower accuracy and longer reaction times for recognising fearful emotions in SAD patients. Within the SAD patients, lower accuracy of sad face recognition was associated with higher severity of depressive and social anxiety symptoms, particularly with avoidance symptoms.Conclusion: These findings may represent a cross-modality pattern of avoidance in the later stage of identifying negative emotions in SAD. This pattern may be linked to clinical symptom severity.

Multimodal emotional state recognition using sequence-dependent deep hierarchical features

Emotional state recognition has become an important topic for human–robot interaction in the past years. By determining emotion expressions, robots can identify important variables of human behavior and use these to communicate in a more human-like fashion and thereby extend the interaction possibilities. Human emotions are multimodal and spontaneous, which makes them hard to be recognized by robots. Each modality has its own restrictions and constraints which, together with the non-structured behavior of spontaneous expressions, create several difficulties for the approaches present in the literature, which are based on several explicit feature extraction techniques and manual modality fusion. Our model uses a hierarchical feature representation to deal with spontaneous emotions, and learns how to integrate multiple modalities for non-verbal emotion recognition, making it suitable to be used in an HRI scenario. Our experiments show that a significant improvement of recognition accuracy is achieved when we use hierarchical features and multimodal information, and our model improves the accuracy of state-of-the-art approaches from 82.5% reported in the literature to 91.3% for a benchmark dataset on spontaneous emotion expressions.

Adaptación española de la versión reducida multicanal del Perfil de Sensibilidad No Verbal (MiniPONS)

The Spanish version of the MiniPONS (Bänziger, Scherer, Hall y Rosenthal, 2011), the short version multichannel of the Profile of Nonverbal Sensitivity (PONS), is presented. This scale was designed to measure individual differences in the ability to recognize emotions, interpersonal attitudes, and communicative intentions in nonverbal channels. This short multichannel version is composed of 64 items nonverbal audio and/or visual scenes portrayed by a female in black and white color. The main aim of this study was to analyze the psychometric properties of MiniPONS using a large sample of university students (N = 1154) aged 18-60, this study also examined gender and age differences in nonverbal sensitivity. Normative data, reliability and validity were examined in three studies with 168, 254, and 1154 university students. The Spanish version of the MiniPONS has psychometric properties similar to those of the original version, the MiniPONS had moderate internal consistency and test-retest reliability, and shows reasonable convergent validity through significant correlations with other tests of non-verbal emotion recognition ability. Results revealed an advantage of women in all the conditions of stimulus presentation; women not only process more efficiently unisensory nonverbal information (face, body and voice) but may also be better at integrating emotional expressions. We conclude that the MiniPONS Spanish version is an adequate instrument to be used in measure of socio-nonverbal sensitivity. Limitations and potential applications for research are described. : E�F' p�Pr�span&gt;have not been studied for current literature. The aim of this study was to analyse changes in cortical arousal, blood lactate, muscle strength, autonomic modulation and rate of perceived exertion in a simulated urban combat. We analyzed 20 soldiers before and after an urban combat simulation. The results showed how urban combat produced high sympathetic nervous system activation, increasing the muscle strength, heart rate and blood lactate concentration of the soldiers. Despite this effort, rate of perceived exertion were not consistent with the physiological response that soldiers presented, the rate of perceived exertion was lower than the physiological response evaluated. Furthermore, the information processing and cortical arousal decreased after the urban combat simulation. These results have showed the psycho-physiological response of soldiers in combat, helping to better understanding and enabling an improvement of current training methods of soldiers.

Preschoolers’ recognition of emotional expressions: Relationships with other neurocognitive capacities

We cross-sectionally examined the development of the ability to recognize facial expressions of emotions in preschool-aged children and the relationship between this ability and other neurocognitive capacities, that is, attention/executive functions, language, memory/learning, sensorimotor functions, theory of mind, and visuospatial processing. Children aged 3 to 6 years with no significant developmental deficits (N = 370) were assessed with a nonverbal matching task of emotion recognition ability: The Affect Recognition subtest from the NEPSY-II. The relationship between emotion recognition ability and other neurocognitive capacities was analyzed using correlation, regression, and commonality analyses. The results showed that (a) emotion recognition ability improved with age—this development decelerating mildly between ages 5 and 6—(b) emotion recognition ability correlated with all other neurocognitive capacities, and (c) language, attention/executive functions, and theory of mind were significant predictors of emotion recognition ability in the regression analysis. As revealed by the commonality analysis, and in contrast to most previous studies, language was the most important predictor of nonverbal emotion recognition ability. These results suggest that nonverbal emotion matching is an early maturing skill that develops in relation to other neurocognitive capacities, especially linguistic ability.

Nonverbal emotion recognition and performance: differences matter differently

Purpose – This paper aims to explore and test the relationship between emotion recognition skill and assessment center performance after controlling for both general mental ability (GMA) and conscientiousness. It also seeks to test whether participant sex or race moderated these relationships.Design/methodology/approach – Using independent observers as raters, the paper tested 528 business students participating in a managerial assessment center, while they performed four distinct activities of: an in‐basket task; a team meeting for an executive hiring decision; a team meeting to discuss customer service initiatives; and an individual speech.Findings – Emotion recognition predicted assessment center performance uniquely over both GMA and conscientiousness, but results varied by race. Females were better at emotion recognition overall, but sex neither was related to assessment center performance nor moderated the relationship between it and emotion recognition. The paper also found that GMA moderated the e...

Measuring providers’ verbal and nonverbal emotion recognition ability: Reliability and validity of the Patient Emotion Cue Test (PECT)

Objective To describe the development and validation of the Patient Emotion Cue Test (PECT) as a tool to measure providers’ emotion cue recognition ability. Methods The PECT consists of 47 video clips depicting emotion cues that systematically vary in intensity of both verbal and nonverbal contents. The PECT assesses the provider's ability to detect and identify patients’ emotion cues accurately. A multi-stage development process produced the PECT. Reliability and validity were assessed in three studies. Results Scores on the PECT are normally distributed with significantly above chance responding. Across three studies, the PECT demonstrates convergent validity through significant correlations with standardized tests representing multiple channels of emotion recognition, including the face, body, and voice. The PECT shows adequate inter-item and split-half reliability. Conclusion The PECT is an easily administered, reliable, and valid test of emotion cue recognition. Practice implications The PECT can be used in future research on providers’ emotion recognition ability, for evaluating self-assessment of ability, and as a teaching tool in medical schools.

New technologies for information retrieval to achieve situational awareness and higher patient safety in the surgical operating room: the MRI institutional approach and review of the literature

Technical progress in the operating room (OR) increases constantly, but advanced techniques for error prevention are lacking. It has been the vision to create intelligent OR systems ("autopilot") that not only collect intraoperative data but also interpret whether the course of the operation is normal or deviating from the schedule ("situation awareness"), to recommend the adequate next steps of the intervention, and to identify imminent risky situations. Recently introduced technologies in health care for real-time data acquisition (bar code, radiofrequency identification [RFID], voice and emotion recognition) may have the potential to meet these demands. This report aims to identify, based on the authors' institutional experience and a review of the literature (MEDLINE search 2000-2010), which technologies are currently most promising for providing the required data and to describe their fields of application and potential limitations. Retrieval of information on the functional state of the peripheral devices in the OR is technically feasible by continuous sensor-based data acquisition and online analysis. Using bar code technologies, automatic instrument identification seems conceivable, with information given about the actual part of the procedure and indication of any change in the routine workflow. The dynamics of human activities also comprise key information. A promising technology for continuous personnel tracking is data acquisition with RFID. Emotional data capture and analysis in the OR are difficult. Although technically feasible, nonverbal emotion recognition is difficult to assess. In contrast, emotion recognition by speech seems to be a promising technology for further workflow prediction. The presented technologies are a first step to achieving an increased situational awareness in the OR. However, workflow definition in surgery is feasible only if the procedure is standardized, the peculiarities of the individual patient are taken into account, the level of the surgeon's expertise is regarded, and a comprehensive data capture can be obtained.