Affect Recognition Research Articles

Distinct Differences in Emotional Recognition According to Severity of Psychotic Symptoms in Early-Stage Schizophrenia.

Patients with schizophrenia are characterized by deficits in their ability to identify facial expressions of emotion, which are associated with impaired social and occupational function. An understanding of the deficits of facial affect recognition (FAR) early in the course of the illness can improve early intervention efforts to ameliorate potential functional deterioration. This study aimed to investigate the characteristics and correlations between psychotic symptoms and FAR deficits in patients with early-stage schizophrenia using data from the Korean Early Psychosis Cohort Study. Patients with schizophrenia were divided into three groups: 1) severely and markedly ill (n = 112), 2) moderately ill (n = 96), and 3) mildly ill (n = 115). These groups were compared with age- and sex-matched healthy controls. The FAR test was developed using Korean emotional faces from the Korean Facial Expressions of Emotion database. Error rates, correct response times, and nonresponse rates of each subset were calculated. Several psychopathology assessments were also performed. There were significantly more deficits associated with the recognition of anger (p < 0.01), fear (p < 0.01), and contempt (p < 0.01) in the three patient groups than in the healthy control group. In the severely and markedly ill states, all emotions apart from surprise had impaired error rates (p < 0.01 for all analyses). The error rates for happiness, sadness, disgust, surprise, and neutral faces were not significantly different between mildly ill patients and healthy controls. All emotions, except for sadness, had significantly more delayed correct response times in all patient groups than in the healthy controls (p < 0.01 for all analyses). The severity of psychotic symptoms was positively correlated with the happiness and neutral error rates, and depression was positively correlated with the happiness error rates. General social function was negatively correlated with the error rates for happiness, sadness, fear, disgust, and surprise. Overall, our results show that the severity of psychosis and clinical symptoms leads to distinct differences in certain emotions of patients with early-stage schizophrenia. It is considered that these specific emotional characteristics will help deepen our understanding of schizophrenia and contribute to early intervention and recovery of social function in patients with schizophrenia.

Affective computing in education: A systematic review and future research

It is becoming a trend to apply an emotional lens and to position emotions as central to educational interactions. Recently, affective computing has been one of the most actively research topics in education, attracting much attention from both academics and practitioners. However, despite the increasing number of papers published, there still are deficiencies and gaps in the comprehensive literature review in the specific area of affective computing in education. Therefore, this study presents a review of the literature on affective computing in education by selecting articles published from 2010 to 2017. A review protocol consisting of both automatic and manual searches is used to ensure the retrieval of all relevant studies. The final 94 selected papers are reviewed and relevant information extracted based on a set of research questions. This study classifies selected articles according to the research purposes, learning domains, channels and methods of affective recognition and expression, and emotion theories/models as well as the emotional states. The findings show the increased number and importance of affective computing studies in education domain in recent years. The research purposes of most affective computing studies are found to be designing emotion recognition and expression systems/methods/instruments as well as examining the relationships among emotion, motivation, learning style, and cognition. Affective measurement channels are classified into textual, visual, vocal, physiological, and multimodal channels, while the textual channel is recognized as the most widely-used affective measurement channel. Meanwhile, integration of textual and visual channels is the most widely-used multimodal channel in affective computing studies. Dimensional theories/models are the most preferred models for description of emotional states. Boredom, anger, anxiety, enjoyment, surprise, sadness, frustration, pride, hopefulness, hopelessness, shame, confusion, happiness, natural emotion, fear, joy, disgust, interest, relief, and excitement are reported as the top 20 emotional states in education domain. Finally, this study provides recommendations for future research directions to help researchers, policymakers and practitioners in the education sector to apply affective computing technology more effectively and to expand educational practices.

Acceptance and use of a multi-modal avatar-based tool for remediation of social cognition deficits

This paper focuses on the validation of a tool designed to improve affect recognition, a fundamental aspect of social cognition as it greatly affects the functionality and quality of life of patients with mental disorders. The presented tool facilitates the generation of multi-modal avatar-based therapies by mental health professionals in this important clinical domain. Moreover, the tool for remediation of social cognitive deficits may be customised to each patient’s impairment. This paper describes how the tool was assessed by therapists after viewing a video explaining its most relevant aspects. The participants were asked to fill in a questionnaire based on UTAUT2 for the study of the acceptance and use of this technology. In light of the results obtained from 41 therapists about their intention of use, the most important statement is that their interest for this kind of tools is high. Nonetheless, there are some factors that negatively affect their behavioural intention.

Victimization, violence and facial affect recognition in a community sample of first-episode psychosis patients.

This exploratory study is the first to examine previously identified variables of increased vulnerability to victimization, the prevalence of aggression in a cohort of patients with first-episode psychosis and the potential impact of impairments in facial affect recognition (FAR) on victimization. Sixty-nine male participants completed assessments of IQ, substance use, psychopathy, childhood trauma, aggressive behaviour and psychopathology. Participants were asked about violent victimization in the past year and charges for violent offences. FAR was assessed using the Animated Full Facial Comprehension Test. The victimized group (n = 25) had significantly higher psychopathy traits (P = .042) and lower recognition of sadness (P < .01) compared to the non-victimized group (n = 44). Participants who reported charges for violent offences (n = 15) had significantly higher scores on measures of physical abuse in childhood (P < .01), substance use (P < .05 for cannabis, P < .001 for cocaine and amphetamine use), psychopathy traits (P < .01), psychopathology (P = .031) and lifetime aggressive behaviour (P < .01). The logistic regression models using FAR sadness and Psychopathic Personality Inventory-Revised (PPI-R) total score as predictors for victimization (χ 2 [2] = 13.81, P = .001; Nagelkerke R 2 = .30) and using physical abuse in childhood, PPI-R total score, and psychopathology as predictors for violent offences (χ 2 [3] = 14.89, P = .002; Nagelkerke R 2 = .36) were significant. These findings highlight the value of considering victimization in psychotic illness from a social information processing perspective in addition to known clinical and lifestyle factors. Based on these results, future studies could explore the use of affect recognition training early in the course of the illness as a possible intervention to reduce victimization.

The Role of Haptic Stimuli on Affective Reading: a Pilot Study.

The affective role of touch has opened new perspectives in human-machine interaction. This paper presents an emotion recognition algorithm to investigate the role of tactile stimuli conveyed through a wearable haptic system during affective reading. To this end, a group of 32 healthy volunteers underwent an emotional stimulation by reading affective texts, with and without the concurrent presence of pleasant haptic stimuli. Throughout the experiment, autonomic nervous system dynamics was quantified through heart rate variability (HRV) and electrodermal activity (EDA) analyses. EDA and HRV features were then used as input of a SVM-RFE learning algorithm for an automatic recognition of neutral and arousing texts. The affective recognition of the reading was performed in the presence or absence of the haptic stimulation. Results show that the affective perception induced by the neutral and arousing reading were discriminated with a significantly improved accuracy (+14.5%) when a caress-like haptic stimulus was conveyed to the user.

From basic perception deficits to facial affect recognition impairments in schizophrenia

While impaired facial emotion recognition and magnocellular deficits in visual perception are core features of schizophrenia, their relationship is still unclear. Our aim was to analyze the oscillatory background of these processes and to investigate the connection between the magnocellular pathway deficit and the abnormal facial affect processing. Thirty-nine subjects with schizophrenia and forty socially matched healthy controls subjects were enrolled. A 128 channel EEG was recorded in three experimental tasks: first, participants viewed magnocellular biased low-spatial frequency (LSF) and parvocellular biased high-spatial frequency (HSF) Gabor-patches, then faces and houses were presented and in the third task a facial affect recognition task was presented with happy, sad and neutral faces. Event-related theta (4–7 Hz) synchronization (ERS) (i.e. an increase in theta power) by magnocellular biased stimuli was decreased in patients relative to controls, while no similar differences were found between groups in the parvocellular biased condition. ERS was significantly lower in patients compared to healthy controls both in the face and in the emotion recognition task. Theta ERS to magnocellular biased stimuli, but not to parvocellular biased stimuli, were correlated with emotion recognition performance. These findings indicate a bottom up disruption of face perception and emotion recognition in schizophrenia.

Community Integration in Traumatic Brain Injury: The Contributing Factor of Affect Recognition Deficits.

Individuals with traumatic brain injury (TBI) can experience social isolation, which is damaging to well-being and counterproductive to successful rehabilitation. It has been proposed that social cognitive deficits that commonly result from TBI may contribute to weakened social integration. However, the consequences of specific social cognitive deficits in TBI are still being delineated. The current work sought to better characterize the relationship between community integration and facial affect recognition (FAR) in TBI. A total of 27 participants with moderate to severe TBI and 30 healthy controls (HCs) completed two tests of FAR, which employed either static photographic stimuli or dynamic video stimuli (The Awareness of Social Inference Test). The Community Integration Questionnaire was also administered to participants. Participants with TBI were significantly impaired on both the static and dynamic FAR measures, yet the deficits were most pronounced within the dynamic task. Furthermore, participants with TBI reported lower community integration compared with HCs. FAR was positively associated with community integration in both groups, such that participants with proficient affect recognition skills were better integrated into their communities. FAR deficits may contribute to the lack of community integration often observed in TBI; thus, interventions designed to improve FAR may be beneficial to this population's ability to successfully reintegrate into society.

Neuroimaging Markers of Risk, Disease Expression, and Resilience to Bipolar Disorder.

Familial predisposition to bipolar disorder is associated with increased risk of affective morbidity in the first-degree relatives of patients. Nevertheless, a substantial proportion of relatives remain free of psychopathology throughout their lifetime. A series of studies reviewed here were designed to test whether resilience in these high-risk individuals is associated with adaptive brain plasticity. The findings presented here derive from structural and functional magnetic resonance imaging data obtained from patients, their resilient first-degree relatives, and healthy individuals. Patients and relatives showed similar abnormalities in activation and connectivity while performing tasks of interference control and facial affect recognition and in the resting-state connectivity of sensory and motor regions. Resilient relatives manifested unique neuroimaging features that differentiated them from patients and healthy individuals. Specifically, they had larger cerebellar vermis volume, enhanced prefrontal connectivity during task performance, and enhanced functional integration of the default mode network in task-free conditions. Resilience to bipolar disorder is not the reverse of risk but is associated with adaptive brain changes indicative of increased neural reserve. This line of research may open new avenues in preventing and treating bipolar disorder.

Relationship between social cognition and fatigue, depressive symptoms, and anxiety in multiple sclerosis

Emerging research indicates that in addition to physical and cognitive deficits, individuals with multiple sclerosis (MS) may also have impairments in social cognition, such as facial affect recognition and Theory of Mind (ToM). However, there is little research into how social cognition impairments relate to other domains in MS, such as mood and fatigue levels. The current study investigated whether social cognitive ability is associated with fatigue, depressive symptoms and anxiety in MS. Twenty-eight individuals with MS completed questionnaires assessing fatigue (Modified Fatigue Impact Scale), depression (Beck Depression Inventory) and anxiety (State-Trait Anxiety Inventory), as well as tasks of facial affect recognition and ToM (Reading the Mind in the Eyes; Strange Stories). Bivariate correlations were run to examine relationships between variables; partial correlations were subsequently used to ascertain whether these relationships persisted after controlling for cognitive ability (measured with the Symbol Digit Modalities Test). The results indicated that worse performance in both facial affect recognition and ToM were associated with higher rates of psychosocial fatigue, depressive symptoms, and anxiety levels; furthermore, these relationships were not explained by participants' cognitive ability. These preliminary results help us better understand the association between social cognitive abilities and other symptoms in MS, including depressive symptomatology, anxiety and fatigue.

Alone versus In-a-group

Recognition and analysis of human affect has been researched extensively within the field of computer science in the past two decades. However, most of the past research in automatic analysis of human affect has focused on the recognition of affect displayed by people in individual settings and little attention has been paid to the analysis of the affect expressed in group settings. In this article, we first analyze the affect expressed by each individual in terms of arousal and valence dimensions in both individual and group videos and then propose methods to recognize the contextual information, i.e., whether a person is alone or in-a-group by analyzing their face and body behavioral cues. For affect analysis, we first devise affect recognition models separately in individual and group videos and then introduce a cross-condition affect recognition model that is trained by combining the two different types of data. We conduct a set of experiments on two datasets that contain both individual and group videos. Our experiments show that (1) the proposed Volume Quantized Local Zernike Moments Fisher Vector outperforms other unimodal features in affect analysis; (2) the temporal learning model, Long-Short Term Memory Networks, works better than the static learning model, Support Vector Machine; (3) decision fusion helps to improve affect recognition, indicating that body behaviors carry emotional information that is complementary rather than redundant to the emotion content in facial behaviors; and (4) it is possible to predict the context, i.e., whether a person is alone or in-a-group, using their non-verbal behavioral cues.

Affective prosody and facial emotion recognition in first-episode schizophrenia: Associations with functioning & symptoms

Studies indicate that people with schizophrenia experience deficits in their ability to accurately detect emotions, both through facial expressions and voice intonation (i.e., prosody), and that functioning and symptoms are associated with these deficits. This study aimed to examine how facial emotion and affective prosody recognition are related to functioning and symptoms in a first-episode schizophrenia sample. Further, in light of research suggesting variable emotion-specific performance in people with schizophrenia, this study explored emotion-specific performance. Participants were 49 people with a recent first episode of schizophrenia taking part in a larger RCT. Results revealed that affective prosody recognition was significantly correlated with both role and social functioning. Regarding associations with psychiatric symptoms, facial emotion recognition was significantly, negatively associated with all three positive symptom scales, whereas affective prosody recognition was significantly, negatively associated with disorganization only. Emotion-specific analyses revealed that for affective prosody, participants were most accurate in recognizing anger and least accurate for disgust. For facial emotion recognition, participants were most accurate in recognizing happiness and least accurate for fear. Taken together, results suggest that affective prosody recognition is important for social and role functioning in people with first-episode schizophrenia. Results also suggest that this group may struggle more to identify negative emotions, though additional work is needed to clarify this pattern in affective prosody and determine real-world impact on social interactions.

The Direct and Indirect Relationship Between Social Cognition and Psychosocial Dysfunction in Major Depressive Disorder.

Background: Recent evidence suggests that depressed patients experience social cognitive deficits (e.g., poor affect recognition). However, very little is known regarding the contribution of social cognitive deficits to psychosocial dysfunction (e.g., occupational functioning). In particular, the mechanistic roles of depression severity and cognitive deficits (e.g., memory) in this domain have not been explored. The current study evaluated the extent to which mood symptoms and cognitive deficits provide a mechanistic explanation for the relationship between social cognitive and psychosocial deficits in major depressive disorder (MDD). Methods: Data were obtained from 111 participants with MDD (75 Female, mean age = 35, 84% Caucasian, 12% Asian, 4% Other) in the Cognitive Function and Mood Study (CoFaM-S), a cross-sectional study of mood, social cognition, cognition, and psychosocial functioning in mood disorders. Social cognitive abilities were assessed using the Social Perception subtest of the Wechsler Adult Intelligence Scale, and psychosocial dysfunction was clinically evaluated with the Functioning Assessment Short Test (FAST). Results: Cognitive deficits and mood symptoms did not significantly mediate relationships between social cognitive ability and psychosocial dysfunction. The exception was executive function, which mediated an indirect relationship between meaning interpretation (i.e., theory of mind) and self-perceived cognitive dysfunction. Conclusion: The results suggest that the relationship between social cognitive deficits and psychosocial dysfunction is not mechanistically explained by mood symptoms or nonsocial cognition. Development of treatment strategies targeting social cognitive deficits in patients with MDD is warranted.

AFAR: A Deep Learning Based Tool for Automated Facial Affect Recognition.

Automated facial affect recognition is crucial to multiple domains (e.g., health, education, entertainment). Commercial tools are available but costly and of unknown validity. Open-source ones [1] lack user-friendly GUI for use by non-programmers. For both types, evidence of domain transfer and options for retraining for use in new domains typically are lacking.

Emotion-specific performance across empathy tasks in schizophrenia: Influence of metacognitive capacity

People with schizophrenia exhibit deficits in emotion recognition that are associated with community and social functioning. Emotion-specific performance within emotion recognition tasks has been investigated, suggesting differential patterns of recognition for positive and negative emotions. However, no study has yet examined emotion-specific performance for a higher-order social cognitive construct such as empathy. This study aimed to: 1) examine emotion-specific performance on an empathy task, and 2) elucidate associations with four metacognitive domains: self-reflectivity, understanding of others' minds, decentration, and mastery. Fifty-seven people with schizophrenia or schizoaffective disorder participated. All were administered a computerized, performance-based measure of empathy and an interview-based measure of metacognitive capacity. Results revealed that, consistent with research on facial affect recognition, participants performed significantly better when recognizing happiness in empathic stimuli than all other emotions. Results also revealed positive associations between empathic performance and metacognitive self-reflectivity, across types of emotions. Other metacognitive domains were also associated with performance, but in a less consistent manner. Together, results indicate that not all emotions are created equal – happiness is easier to recognize for those with schizophrenia, suggesting that social cognitive interventions may be more helpful if focused on recognizing negative emotions. Results also emphasize the importance of metacognitive capacity for basic and higher-order social cognitive skills.

A survey on big data-driven digital phenotyping of mental health

The landscape of mental health has undergone tremendous changes within the last two decades, but the research on mental health is still at the initial stage with substantial knowledge gaps and the lack of precise diagnosis. Nowadays, big data and artificial intelligence offer new opportunities for the screening and prediction of mental problems. In this review paper, we outline the vision of digital phenotyping of mental health (DPMH) by fusing the enriched data from ubiquitous sensors, social media and healthcare systems, and present a broad overview of DPMH from sensing and computing perspectives. We first conduct a systematical literature review and propose the research framework, which highlights the key aspects related with mental health, and discuss the challenges elicited by the enriched data for digital phenotyping. Next, five key research strands including affect recognition, cognitive analytics, behavioral anomaly detection, social analytics, and biomarker analytics are unfolded in the psychiatric context. Finally, we discuss various open issues and the corresponding solutions to underpin the digital phenotyping of mental health.

Multivariate pattern analysis of the neural correlates of smoking cue attentional bias

The automatic capture of attention by drug cues, or attentional bias, is associated with craving and predicts future drug use. Despite its clinical significance, the neural bases of attentional bias to drug cues is not well understood. To address this gap, we undertook a neuroimaging investigation of the neural correlates of attentional bias towards smoking cues. Twenty-nine adults, including 14 active smokers and 15 non-smokers, completed a spatial cuing task during fMRI. A multivariate pattern analysis (MVPA) decoded the neural responses to the brief presentation of smoking versus neutral images. These data were correlated with behavioral measures of attentional bias, which included analyses targeting the neural correlates of response facilitation and cue-related task interference. We detected a set of brain-behavioral correlates that were similar across both smokers and non-smokers, indicating a role for stimuli salience in the absence of nicotine conditioning in smoking cue attentional bias. However, multiple smoking-related modifications to the neural correlates of attentional bias and its components were also identified. For example, regions demonstrating smoking-related differences in the neural correlates of attentional bias included the rostral anterior cingulate cortex and inferior frontal gyrus. Response facilitation effects of smoking were observed in the right orbitofrontal gyrus and bilateral middle temporal gyrus. Smoking-cue related task interference was related to smoking-related effects in the frontal eye fields. Our findings suggest that multiple cognitive, affective, and visual object recognition processes contribute to attentional bias towards smoking cues, and suggest multiple circuit modifications that may contribute to perpetuation of addiction.

A randomized controlled trial of training of affect recognition (TAR) in schizophrenia shows lasting effects for theory of mind.

Schizophrenia is characterized by social cognitive impairments that predict functioning. Social cognitive training aims to target these impairments. Although it can improve the targeted social cognitive domain, it is unclear if the training generalizes to non-targeted domains and to functioning, with lasting effects. This randomized controlled trial examined the effect of a targeted facial affect recognition training program, Training of Affect Recognition (TAR), in persons with schizophrenia. Individuals with schizophrenia were randomized to receive treatment as usual and TAR (n = 24) or treatment as usual (n = 24) after assessments with a comprehensive protocol at baseline (T1). Participants were reassessed immediately after the intervention period (T2: after 8weeks) and at 3-month follow-up (T3). The protocol included tests of social cognition (facial or body affect recognition, theory of mind), nonsocial cognition (Matrics Consensus Cognitive Battery), clinical symptoms (Positive and Negative Syndrome Scale, Calgary Depression Scale for Schizophrenia), functioning (self-reported, social or nonsocial functional capacity), self-esteem, self-efficacy and insight. Linear mixed models yielded a significant group × time interaction effect for a non-targeted social cognitive domain (theory of mind) and a trend-level effect for social functional capacity with the intervention group performing better over time. No beneficial effects on nonsocial cognition, other measures of functioning, clinical symptoms, or self-esteem/self-efficacy appeared for the TAR program. This study provides evidence for transfer and durability effects of facial affect recognition training to theory of mind, but also highlights the need for additional treatments to achieve functional benefits.

How surprising: Mild TBI impacts scan path during facial affect recognition.

How surprising: Mild TBI impacts scan path during facial affect recognition.

Emotional disturbances in multiple sclerosis: A neuropsychological and fMRI study

BackgroundEmotional disturbances in multiple sclerosis (MS) are often explored in terms of affect recognition, with controversial results that likely reflect the high lesional heterogeneity. Patients' emotional experience, however, has seldom been studied and has never been explored using fMRI. ObjectivesTo explore the emotional experience in MS and compare these data with fMRI measurements using for the first time real-life emotional scenes differing in valence and arousal. MethodsTwenty-five right-handed women with relapsing-remitting MS and 27 right-handed age-, sex-, and education-matched healthy controls visualized during an fMRI session, emotional scenes taken from the international affect picture system (IAPS) and differing in valence (positive, negative, neutral) and arousal (ranging from calm to excited). During a post-scanning debriefing, participants were asked to look again at each image and score it in terms of valence and arousal sensation on a scale of 1–9. ResultsCognitively well-preserved MS subjects presented a significantly more scattered emotional experience compared to controls in response to positive and negative pictures. In fMRI, MS patients also presented a higher variability of response when compared to controls in left inferior orbitofrontal cortex for positive stimulations. For negative condition, no significant results were observed between the two groups. However, a trend was detected in left amygdala, right fusiform gyrus, right caudate nucleus and right pallidum for negative stimulations. ConclusionIn response to emotional stimuli, MS subjects presented a scattered emotional experience subtended by a greater variability of brain response, highlighting an emotional pattern not previously reported in MS patients.

“See Me, Feel Me”: Two Modes of Affect Recognition for Real and Fictional Targets

It is commonly presupposed that there are no decisive differences between empathy with fictional characters on one hand and empathy with real persons on the other. I distinguish two types of processes of affect recognition "Perceptual Affect Recognition" and "Affective Affect Recognition". The consensus view about empathy with fictional characters has to be challenged if "empathy" refers to the former or the latter process because of the significant differences between the fictional and the non-fictional scenario: firstly, readers as "empathizers" cannot perceive the fictional target person directly but only the literary text. This is especially problematic for Affective State Recognition. Secondly, fictional characters do not exist, at least not in the sense that they are real people or entities with mental states which leads to relevant differences of the accuracy conditions of Affective Affect Recognition in the fictional and the nonfictional scenario.