Emotional Voice Research Articles

A Multimodal Emotion Detection System during Human–Robot Interaction

In this paper, a multimodal user-emotion detection system for social robots is presented. This system is intended to be used during human–robot interaction, and it is integrated as part of the overall interaction system of the robot: the Robotics Dialog System (RDS). Two modes are used to detect emotions: the voice and face expression analysis. In order to analyze the voice of the user, a new component has been developed: Gender and Emotion Voice Analysis (GEVA), which is written using the Chuck language. For emotion detection in facial expressions, the system, Gender and Emotion Facial Analysis (GEFA), has been also developed. This last system integrates two third-party solutions: Sophisticated High-speed Object Recognition Engine (SHORE) and Computer Expression Recognition Toolbox (CERT). Once these new components (GEVA and GEFA) give their results, a decision rule is applied in order to combine the information given by both of them. The result of this rule, the detected emotion, is integrated into the dialog system through communicative acts. Hence, each communicative act gives, among other things, the detected emotion of the user to the RDS so it can adapt its strategy in order to get a greater satisfaction degree during the human–robot dialog. Each of the new components, GEVA and GEFA, can also be used individually. Moreover, they are integrated with the robotic control platform ROS (Robot Operating System). Several experiments with real users were performed to determine the accuracy of each component and to set the final decision rule. The results obtained from applying this decision rule in these experiments show a high success rate in automatic user emotion recognition, improving the results given by the two information channels (audio and visual) separately.

Adaptation aftereffects in vocal emotion perception elicited by expressive faces and voices.

The perception of emotions is often suggested to be multimodal in nature, and bimodal as compared to unimodal (auditory or visual) presentation of emotional stimuli can lead to superior emotion recognition. In previous studies, contrastive aftereffects in emotion perception caused by perceptual adaptation have been shown for faces and for auditory affective vocalization, when adaptors were of the same modality. By contrast, crossmodal aftereffects in the perception of emotional vocalizations have not been demonstrated yet. In three experiments we investigated the influence of emotional voice as well as dynamic facial video adaptors on the perception of emotion-ambiguous voices morphed on an angry-to-happy continuum. Contrastive aftereffects were found for unimodal (voice) adaptation conditions, in that test voices were perceived as happier after adaptation to angry voices, and vice versa. Bimodal (voice + dynamic face) adaptors tended to elicit larger contrastive aftereffects. Importantly, crossmodal (dynamic face) adaptors also elicited substantial aftereffects in male, but not in female participants. Our results (1) support the idea of contrastive processing of emotions (2), show for the first time crossmodal adaptation effects under certain conditions, consistent with the idea that emotion processing is multimodal in nature, and (3) suggest gender differences in the sensory integration of facial and vocal emotional stimuli.

English L2 personas and the imagined global community of English users

I have Danish parents, grown up in France, lived in the UK for 10 years and now living in Holland for the past 9 years. I am a different person in each language, adapting myself to the culture of the people who speak it. I have always wondered how the language could affect the message so much. It also affects my tone of voice and my emotions. [Christina, on July 27th, 2011]I find myself being more self-depreciating and less likely to accept praise when speaking in Japanese than when speaking English. Furthermore, a colleague once told me that even if he can't hear, he can tell which language I am using from 10-15 metres away, by looking at my posture, gestures and general body language. [Tim, on July 27th, 2011]My friends once told me that when I switch to Russian even my facial features change, becoming colder and harsher – set jaw, narrow eyes, speaking in a low voice, but with an intensity that makes everyone else listen. [Julia, on July 30th, 2011]The above quotations represent just a small portion of blogosphere postings from bilingual and multilingual individuals commenting on how they perceive and appear to have very distinct and separate personalities when speaking the different languages in their linguistic repertoires. Many such postings, like the first one above by Christina, explicitly attribute this phenomenon to attempts on the part of speakers to assimilate to the cultural norms of the countries where the languages are traditionally spoken. Scholarly treatments of the same phenomenon (e.g., Bryant, 1984; Hu & Reiterer, 2009; Zukowski/Faust, 1997) generally do likewise, often citing Schumann's (1978, 1986) Acculturation Model, which equates L2 proficiency with the extent to which a learner is able to adopt the culture of a target language group, and Guiora's (1967, 1979) concept of Language Ego, in which the permeability of one's L1 identity determines receptiveness to taking on new linguistic identities. According to these theories, a learner of Korean, for example, would be likely to develop a distinctly Korean L2 persona (as well as linguistic proficiency) if he or she has both a high level of affinity for Korean culture and a very permeable L1 language ego. Such arguments still, no doubt, apply to languages such as Korean or Japanese that are intrinsically associated with specific countries and cultures. Given the status of English as an international lingua franca in today's world, however, it can no longer be assumed that learners of English have any motive or desire to acculturate into traditionally English-speaking cultures, such as those of the US, England, or Australia. If learners/users of English associate the language not with such traditionally English-speaking cultures, but instead with an imagined global community of English users, do they still develop English L2 personas that are distinct from their L1 personas and feel ‘like a different person’ when speaking English?

A CASE OF ORGANIC AMNESIC DISORDER SYNDROME DIAGNOSED WITH FMRI

ObjectivePost-concussion syndrome after traffic accident is often clinically associated with prolonged cognitive impairment due to brain lesions, but sometimes a symptom like amnesia may occur without structural abnormality in neuroimaging....

Emotional Voice Processing: Investigating the Role of Genetic Variation in the Serotonin Transporter across Development

The ability to effectively respond to emotional information carried in the human voice plays a pivotal role for social interactions. We examined how genetic factors, especially the serotonin transporter genetic variation (5-HTTLPR), affect the neurodynamics of emotional voice processing in infants and adults by measuring event-related brain potentials (ERPs). The results revealed that infants distinguish between emotions during an early perceptual processing stage, whereas adults recognize and evaluate the meaning of emotions during later semantic processing stages. While infants do discriminate between emotions, only in adults was genetic variation associated with neurophysiological differences in how positive and negative emotions are processed in the brain. This suggests that genetic association with neurocognitive functions emerges during development, emphasizing the role that variation in serotonin plays in the maturation of brain systems involved in emotion recognition.

Sex matters

This work investigates the impact of task load on irrelevant emotional voice processing between the sexes. The working memory task was manipulated by asking participants to evaluate the position of a visual stimulus (0-back) and compare the position of the current stimulus with the one or two trials before (1-back and 2-back). We recorded auditory event-related potentials while presenting the emotionally spoken syllables 'dada' and acoustically matched nonvocal sounds to healthy adults. Women, not men, showed larger amplitudes of emotional mismatch negativity (MMN), presumed to reflect preattentive auditory change detection, during the 1-back task than the 0-back and 2-back tasks. Among women only weaker emotional MMN amplitudes were associated with a higher 2-back accuracy. Neither task difficulty nor sex had any effect on MMN when responding to nonvocal sounds. Our findings suggest that there are sex differences in preattentive emotional voice processing under varying levels of task load.

Early development of the social and emotional brain in infancy

An important function of the brain is to scan one’s surroundings for the presence of biologically relevant features and grant them priority for attention and action. The significance of this function for an individual’s well-being and survival has given rise to the argument that specialized neural mechanisms for processing biologically salient stimuli may exist and that these mechanisms may require little, if any, experience for normal development. As evidenced by behavioral studies, the ontogeny of this network and associated cognitive capacities may emerge before the age of one year. With technological advances in recent years, researchers have been able to utilize electroencephalography, functional near-infrared spectroscopy, and functional magnetic resonance imaging to directly explore the early development of social-emotional brain mechanisms. In this paper, we systematically review three major aspects of early social and emotional development in infancy—facial expression, emotional voice, and early socialization. Based on empirical evidence, we describe the developmental process of the corresponding social and emotional neural network. We then discuss the impact of early social experience on the development of the social and emotional brain in terms of experience-expectant and experience-dependent mechanisms. We argue that the social-emotional neural network is not a congenital processing module that is independent of social experience. The core structure of the human social and emotional brain seems to reach anatomical maturity relatively early in development (zero to three months); however, the basic organization of the social-emotional network may be specified by an experience-expectant neural circuitry that emerges at five to seven months of age and is rapidly refined by exposure to universal features of social experience during a sensitive period of development. The network retains some plasticity throughout a person’s lifetime and can be fine-tuned by individual-specific experiences, i.e., experience-dependent development. Functional connectivity between emotion-processing networks and other prefrontal regulatory systems continues to develop into adolescence. Finally, based on the present study of the early development of the social and emotional brain, we indicate three promising directions for future research.

Does parental anxiety cause biases in the processing of child‐relevant threat material?

Anxiety leads to biases in processing personally relevant information. This study set out to examine whether anxious parents also experience biases in processing child-relevant material. Ninety parents acted as a control condition, or received a social anxiety or child-related anxiety induction. They completed a task examining attentional biases in relation to child-threat words and social-threat words, and a task examining ability to categorize emotion in children's faces and voices. There was a trend indicating group differences in attentional bias towards social-threat words, and this appears to have been only in the social anxiety condition, but not the child anxiety or control conditions. For child-threat words, attentional bias was present in the child anxiety condition, but not the social anxiety or control conditions. In the emotion recognition task, there was no difference between the control and child anxiety conditions, but the social anxiety condition were more likely to erroneously label children's faces and voices as sad. Parents' anxious biases may spill over into their child's world. Parents' anxious biases may spill over into their child's world. Anxious parents may have attentional biases towards threats in their children's environment. Anxious parents may over-attribute negative emotion to children.

Is the processing of affective prosody influenced by spatial attention? an ERP study

BackgroundThe present study asked whether the processing of affective prosody is modulated by spatial attention. Pseudo-words with a neutral, happy, threatening, and fearful prosody were presented at two spatial positions. Participants attended to one position in order to detect infrequent targets. Emotional prosody was task irrelevant. The electro-encephalogram (EEG) was recorded to assess processing differences as a function of spatial attention and emotional valence.ResultsEvent-related potentials (ERPs) differed as a function of emotional prosody both when attended and when unattended. While emotional prosody effects interacted with effects of spatial attention at early processing levels (< 200 ms), these effects were additive at later processing stages (> 200 ms).ConclusionsEmotional prosody, therefore, seems to be partially processed outside the focus of spatial attention. Whereas at early sensory processing stages spatial attention modulates the degree of emotional voice processing as a function of emotional valence, emotional prosody is processed outside of the focus of spatial attention at later processing stages.

Why and When Leader’s Positive Emotion Promotes Employee Voice Behavior

Although researchers have argued that employees often carefully examine social contexts before speaking up, the role of leaders’ emotion has received little attention. Our goal of this research is to address this important issue from an Emotion-as-Social-Information perspective by exploring the effects of leaders’ positive emotion on employees’ voice behavior. By collecting data of 641 daily interactions from 81 leader-employee dyads using the experience sampling method (ESM), we found that leaders’ positive emotion was positively related with employee voice. Further, such a relationship is through employees’ psychological safety via both emotional contagion and signaling mechanisms. Moreover, the effects of both emotional contagion and signaling mechanisms were stronger when leader-member exchange relationship was weak. These findings highlight the important role of leaders’ emotion in voice and also provide insights for when employees choose to speak up to leaders.

Detecting the sound of feelings: An ERP investigation of vocal emotion perception

Event Abstract Back to Event Detecting the sound of feelings: An ERP investigation of vocal emotion perception Christopher Sufani1*, Jacqueline A. Rushby1 and Skye McDonald1 1 University of New South Wales, School of Psychology, Australia Aims: In speech, an individual’s tone of voice modulates systematically to signal their emotional state and intentions. Hence the accurate perception of tone in voice plays a critical role in everyday psychosocial functioning. Using an oddball paradigm, the current study investigated event-related potential (ERP) correlates of acoustic analysis (100ms), emotional salience detection (200 ms) and cognitive evaluation (300-500 ms) of emotional voices, as proposed by Schirmer & Kotz (2006). Method: In the oddball task, 34 undergraduate psychology students (18 females, 20.0 years old) were fitted with EEG caps and told to mouse-click in response to rare auditory presentation of emotionally vocalised words. Participants completed two blocks of the oddball task, each consisting of words spoken in neutral (300), happy (25), angry (25) and disgusted (25) tones. Behavioural, responses and peak-amplitudes differences to emotional tones at 100, 200, and 300-500 ms were analysed. Results: The task elicited P1 (50-150 ms), N2 (150-250 ms), P3 (250-350 ms) and N3 (300-45 0ms) components. Rare emotional tones in comparison to frequent neutral tones elicited greater P1, N2 and N3 peak-amplitudes. Moreover, both N2 and N3 components elicited greater peak-amplitudes at right central regions in response to negative (angry/disgust) compared to positive tones. P3 however, elicited greater P3 peak-amplitudes to neutral tones. Behaviourally, participants were significantly better at detecting negative tones with significantly faster response times to angry tones. Conclusions: Given the early (100-200 ms) peak-amplitude differences between neutral and emotional tones, the P1 and N2 components are likely reflect early acoustic analysis and emotional salience detection processes. Differences in the topographical distribution of N3 may reflect a distinction in the neural mechanisms underlying the cognitive evaluation of positively and negatively valence tones. P3 reflected inhibitory processes associated with the nature of the task. Keywords: vocal emotion perception, Event-related potentials, oddball paradigm, P1, N2, P3, N3 Conference: ASP2013 - 23rd Annual meeting of the Australasian Society for Psychophysiology, Wollongong, Australia, 20 Nov - 22 Nov, 2013. Presentation Type: Oral Presentation Topic: Perception Citation: Sufani C, Rushby JA and McDonald S (2013). Detecting the sound of feelings: An ERP investigation of vocal emotion perception. Conference Abstract: ASP2013 - 23rd Annual meeting of the Australasian Society for Psychophysiology. doi: 10.3389/conf.fnhum.2013.213.00013 Copyright: The abstracts in this collection have not been subject to any Frontiers peer review or checks, and are not endorsed by Frontiers. They are made available through the Frontiers publishing platform as a service to conference organizers and presenters. The copyright in the individual abstracts is owned by the author of each abstract or his/her employer unless otherwise stated. Each abstract, as well as the collection of abstracts, are published under a Creative Commons CC-BY 4.0 (attribution) licence (https://creativecommons.org/licenses/by/4.0/) and may thus be reproduced, translated, adapted and be the subject of derivative works provided the authors and Frontiers are attributed. For Frontiers’ terms and conditions please see https://www.frontiersin.org/legal/terms-and-conditions. Received: 25 Oct 2013; Published Online: 05 Nov 2013. * Correspondence: Mr. Christopher Sufani, University of New South Wales, School of Psychology, Sydney, Australia, christopher.sufani@gmail.com Login Required This action requires you to be registered with Frontiers and logged in. To register or login click here. Abstract Info Abstract The Authors in Frontiers Christopher Sufani Jacqueline A Rushby Skye McDonald Google Christopher Sufani Jacqueline A Rushby Skye McDonald Google Scholar Christopher Sufani Jacqueline A Rushby Skye McDonald PubMed Christopher Sufani Jacqueline A Rushby Skye McDonald Related Article in Frontiers Google Scholar PubMed Abstract Close Back to top Javascript is disabled. Please enable Javascript in your browser settings in order to see all the content on this page.

Inferring the Human Emotional State of Mind using Assymetric Distrubution

This present paper highlights a methodology for Emotion Recognition based on Skew Symmetric Gaussian Mixture Model classifier and MFCC-SDC ceptral coefficients as the features for the recognition of various emotions from the generated data-set of emotional voices belonging to students of both genders in GITAM University. For training and testing of the developed methodology, the data collection is carried out from the students of GITAM University of Visakhapatnam campus using acting sequence consisting of five different emotions namely Happy, Sad, Angry, Neutral, Boredom; each uttering one short emotional base sentence. For training the data we have considered fifty speakers from different regions (30 male & 20 female) and one long sentence containing an emotional speech from each speaker. The experimentation is conducted on text dependent speech emotion recognition and results obtained are tabulated by constructing a Confusion Matrix and comparing with existing methodology like Gaussian mixture model.

Neurofunctional Underpinnings of Audiovisual Emotion Processing in Teens with Autism Spectrum Disorders

Despite successful performance on some audiovisual emotion tasks, hypoactivity has been observed in frontal and temporal integration cortices in individuals with autism spectrum disorders (ASD). Little is understood about the neurofunctional network underlying this ability in individuals with ASD. Research suggests that there may be processing biases in individuals with ASD, based on their ability to obtain meaningful information from the face and/or the voice. This functional magnetic resonance imaging study examined brain activity in teens with ASD (n = 18) and typically developing controls (n = 16) during audiovisual and unimodal emotion processing. Teens with ASD had a significantly lower accuracy when matching an emotional face to an emotion label. However, no differences in accuracy were observed between groups when matching an emotional voice or face-voice pair to an emotion label. In both groups brain activity during audiovisual emotion matching differed significantly from activity during unimodal emotion matching. Between-group analyses of audiovisual processing revealed significantly greater activation in teens with ASD in a parietofrontal network believed to be implicated in attention, goal-directed behaviors, and semantic processing. In contrast, controls showed greater activity in frontal and temporal association cortices during this task. These results suggest that in the absence of engaging integrative emotional networks during audiovisual emotion matching, teens with ASD may have recruited the parietofrontal network as an alternate compensatory system.

A randomised controlled trial evaluating a brief intervention for deficits in recognising emotional prosody following severe ABI

Many adults with acquired brain injuries, including traumatic brain injuries (TBI) have impaired emotion perception. Impaired perception of emotion in voice can occur independently to facial expression and represents a specific target for remediation. No research to date has addressed this. The current study used a randomised controlled trial to examine the efficacy of a short treatment (three x two-hour sessions) for improving the ability to recognise emotional prosody for people with acquired brain injury, mostly TBI. Ten participants were allocated to treatment and 10 to waitlist. All participants remained involved for the duration of the study in the groups to which they were allocated. There were no significant treatment effects for group, but analyses of individual performances indicated that six of the treated participants made demonstrable improvements on objective measures of prosody recognition. The reasons why some participants showed improvements while others did not, was not obvious. Improvements on objective lab-based measures did not generalise to relative reports of improvements in everyday communicative ability. Nor was there clear evidence of long-term effects. In conclusion, treatment of emotional prosody was effective in the short-term for half of the participants. Further research is required to determine what conditions are required to optimise generalisability and longer-term gains.

GMM-Based Emotional Voice Conversion Using Spectrum and Prosody Features

We propose Gaussian Mixture Model (GMM)-based emotional voice conversion using spectrum and prosody features. In recent years, speech recognition and synthesis techniques have been developed, and an emotional voice conversion technique is required for synthesizing more expressive voices. The common emotional conversion was based on transformation of neutral prosody to emotional prosody by using huge speech corpus. In this paper, we convert a neutral voice to an emotional voice using GMMs. GMM-based spectrum conversion is widely used to modify non linguistic information such as voice characteristics while keeping linguistic information unchanged. Because the conventional method converts either prosody or voice quality (spectrum), some emotions are not converted well. In our method, both prosody and voice quality are used for converting a neutral voice to an emotional voice, and it is able to obtain more expressive voices in comparison with conventional methods, such as prosody or spectrum conversion.

Feasibility and Initial Efficacy of a Comprehensive School‐Based Intervention for High‐Functioning Autism Spectrum Disorders

This study examined the feasibility and initial efficacy of a comprehensive school‐based intervention (CSBI) for 12 children with high‐functioning autism spectrum disorders, aged 6 to 9 years. Treatment included a 3‐week summer preparation program followed by a 10‐month CSBI, comprising social skills groups, therapeutic activities, face and voice emotion recognition instruction, an individual daily note, and parent training. Feasibility was supported in high levels of treatment fidelity and teacher‐ and parent‐reported acceptability and satisfaction. Pre–post comparisons suggested that children significantly improved their knowledge of target social skills, ability to identify emotions in facial and vocal expressions taught in the program, and broader emotion recognition skills in child faces. According to parent and teacher ratings, children displayed gains in their use of target social skills and broader social performance. Parents also reported a significant reduction in autism spectrum disorder‐related features.

Open-Trial Pilot Study of a Comprehensive School-Based Intervention for High-Functioning Autism Spectrum Disorders

There is a notable lack of manualized comprehensive school-based interventions (CSBIs) for children with high-functioning autism spectrum disorders (HFASDs). This pilot study examined the feasibility and initial efficacy of a CSBI for 12 children with HFASDs, aged 6 to 9 years. Treatment included a 3-week summer preparation program followed by a 10-month CSBI, composed of social skills groups, therapeutic activities, face and voice emotion recognition instruction, an individual daily note, and parent training. Feasibility was supported in high levels of treatment fidelity and teacher- and parent-reported acceptability and satisfaction. Pre–post comparisons suggested that children significantly improved their knowledge of target social skills and ability to identify emotions in facial and vocal expressions. According to parent and teacher ratings, children displayed gains in their use of target social skills and broader social performance, as well as reductions in ASD-related features.

Impaired processing of emotion in music, faces and voices supports a generalized emotional decoding deficit in alcoholism

To test the generalized emotional decoding impairment hypothesis in alcoholism. Cross-sectional behavioural study comparing emotion recognition conveyed by faces, voices and musical excerpts. Alcohol detoxification unit of Brugmann University Hospital. Twenty-five recently detoxified alcohol-dependent patients were compared to 25 normal controls matched for sex, age and educational level. From faces, voices and musical excerpts, participants were instructed to rate the intensity of several emotions on a scale from 0 for 'absent' to 9 for 'highly present'. Depression, anxiety and sustained/selective attention capacities were controlled for. Alcohol-dependent patients were less accurate than controls in identifying the target emotion in faces (P < 0.001), voices (P < 0.001) and musical excerpts (P < 0.001). Alcohol-dependent patients who are completing detoxification are impaired in recognizing emotions conveyed by faces, voices and music; these results suggest a generalized emotional decoding impairment. Hypothetically, deficits in the fronto-parietal mirror neurone system could link all these disturbances together.

Amygdala subregions differentially respond and rapidly adapt to threatening voices

Emotional states can influence the human voice during speech utterances. Here, we tested the sensitivity and signal adaptation of functional activity located in amygdala subregions to threatening voices during high-resolution functional magnetic resonance imaging. Bilateral superficial (SF) complex and the right laterobasal (LB) complex of the amygdala were generally sensitive to emotional cues from speech prosody. Activity was stronger, however, when listeners directly focused on the emotional prosody of the voice instead of attending to a nonemotional feature. Explicit attention to prosody especially elicited activity in the right LB complex. Furthermore, the right SF specifically showed an effect of sensitization indicated by a significant signal increase in response to emotional voices which were preceded by neutral events. The bilateral SF showed signal habituation to repeated emotional voices indicated by a significant signal decrease for an emotional event preceded by another emotional event. The right SF and LB finally showed an effect of desensitization after the processing of emotional voices indicated by a signal decrease for neutral events that followed emotional events. Thus, different amygdala subregions are sensitive to threatening emotional voices, and their activity depends on the attentional focus as well as on the proximal temporal context of other neutral and emotional events.

Processing of angry voices is modulated by visual load

Visual perceptual load has been shown to modulate brain activation to emotional facial expressions. However, it is unknown whether cross-modal effects of visual perceptual load on brain activation to threat-related auditory stimuli also exist. The current fMRI study investigated brain responses to angry and neutral voices while subjects had to solve an easy or a demanding visual task. Although the easy visual condition was associated with increased activation in the right superior temporal region to angry vs. neutral prosody, this effect was absent during the demanding task. Thus, our results show that cross-modal perceptual load modulates the activation to emotional voices in the auditory cortex and that high visual load prevents the increased processing of emotional prosody.