Emotional Speech Prosody Research Articles

Age-related decline in hearing and emotional prosody processing: A multi-feature oddball study

This study investigated how age and hearing loss contributed to the challenges in perceiving emotional prosody in speech. A multi-feature oddball paradigm was employed to assess mismatch negativity (MMN) and P3a responses to three deviant prosodies (happy, angry and sad) against the standard neutral prosody in spoken words. The participants were 22 adults in the age range of 18 to 70 with various degrees of hearing loss as assessed with pure tone audiometry. Linear mixed effects models revealed heightened MMN response to angry voice and strongest P3a response to happy voice among the three deviants. There was a significant positive correlation between age and MMN amplitude for the angry prosody. Significant correlations were also observed between hearing loss and MMN amplitude in response to both angry and happy stimuli. In addition, negative correlations were found between P3a amplitude and hearing loss as well as aging for happy stimuli. However, aging and hearing loss did not significantly affect the processing of sadness in this paradigm. These findings showed category-sensitive age-related decline in emotional prosody processing. The intricate correlations patterns among age, hearing loss, MMN, and P3a responses called for further exploration of broader implications in cognitive aging.

Second formant transitions for acoustic analysis to differentiate among dementia types

Recent evidence indicates that acoustic features associated with emotional prosody in speech may be an inexpensive, non-invasive method for differentiating among dementia types. In particular, frequency measures in the speech of people with dementia have been associated with listeners’ perception of emotional prosody. The purpose of this study was to determine if second formant (F2) transition information would enhance these differentiations. Prerecorded speech samples of Cookie Theft picture descriptions from 10 individuals with dementia of the Alzheimer’s type (DAT), 5 with vascular dementia (VaD), 9 with MCI, and 10 neurologically healthy controls (NHC) were obtained from the DementiaBank. Nine words that had initial obstruent consonants that occurred at least two times in each of the participant groups were selected for measurement. The F2 durations, extents, and slope were measured and analyzed. Across group comparisons revealed no pattern for F2 durations or extents. The fricatives had longer F2 durations than the plosives. Across word comparisons revealed significant differences across the consonant vowel combinations for all three measures. A group by word interaction occurred for the F2 slope with the VaD group exhibiting larger slopes than the other two groups.

Establishing Relationship between Emotional Prosody and Affective Pragmatics

This work aims to provide theoretical grounds to establish a relationship between emotional prosody and affective pragmatics. Affective pragmatics is a significant yet least studied area of research. This theoretical framework focuses how emotional expressions are constructive in channelizing the pragmatic meaning under the umbrella of affective pragmatics and also encompasses the speech engineering that conveys unabridged abstract emotions in the phenomenal process of emotion recognition. Since speech act theory focuses meaning at utterance level and not at emotional level, thus, there is a need to reflect on emotional expressions that function as paralinguistic features. There are several studies carried out on identification of emotions using prosodic modeling; however, there is no meticulous study that shows relationship between emotions and their pragmatic meanings. The current study will be an application of Theory of Affective Pragmatics (TAP) proposed by Andrea Scarantino in 2017, a theory analogous to speech act theory. The objectives of the study can be achieved through analysis of quantitative data procured taking advantage of the emotional recordings from Emotional Prosody Speech and Transcripts (EPST), a worldwide database used in emotion recognition processes, along with employment of appropriate prosodic features using a Hidden Markov Model (HMM), a popular emotion recognition statistical model. The outcomes of this study will contribute to benefit new researchers in the field of linguistics to understand affective pragmatics at a profound level as a novel area of research.

The disruptive effects of changing-state sound and emotional prosody on verbal short-term memory in blind, visually impaired, and sighted listeners

ABSTRACT Previous findings suggest that blind listeners are less susceptible to auditory distraction in a verbal serial recall task, compared to sighted individuals. However, it is unclear whether this is due to more selective encoding / filtering of auditory information (e.g. efficient perceptual streaming) or to enhanced attentional control. To test these alternatives, the interference induced by changing-state sound was compared with the disruptive effect of emotional speech prosody (happy, angry, fearful intonations) in blind (n = 17), visually impaired (n = 23), and sighted (n = 89) individuals. While a reduced changing-state effect suggests more efficient perceptual filtering, a reduced emotional prosody effect indicates enhanced attentional control. Blind participants were able to recall more items compared to sighted and visually impaired individuals. Moreover, in sighted and visually impaired participants, the changing-state effect was found to be enhanced with angry prosody, whereas blind individuals were less susceptible to changing-state sound regardless of the prosody. The results also suggest group differences in prosody processing, as both visually impaired and blind participants were better able to ignore fearful speech, compared to other prosodies. The findings suggest that profound visual sensory deprivation leads not only to a capacity increment in verbal short-term memory, but enables listeners to shield memory against auditory distracters, suggesting more efficient perceptual streaming.

Exploring Prosodic Pragmatics at Affective Level

This research work presents a conceptual framework to explore the interplay between intonation and emotional expressions that conveys pragmatic meaning of linguistic utterances. Theory of Affective Pragmatics (TAP), an innovative theory that functions parallel to speech act theory, is applied to introduce a conceptual framework to study prosodic pragmatics at a profound level. The affective component of attitude is introduced in the framework using Ostrom’s ABC model to comprehend affective pragmatics. For further study on this framework, quantitative research design is desirable to analyze data and the data is suggested to be obtained through emotional recordings from emotional prosody speech and transcripts (EPST). Praat as an apt tool recommended for the prosodic analysis of the data. This framework explains how intonation patterns signal unabridged emotions and feelings to understand speaker’s implied meaning. It elaborates the emotional expressions that convey pragmatic meaning to intellectualize affective pragmatics. It highlights some other facets that render affective pragmatics a new-fangled field of research. It has also been envisioned to broaden the scope of prosodic pragmatics as a distinctive research domain in linguistics. The framework is an application of TAP and provides grounds to researchers to understand prosodic pragmatics at a deeper level and to perform future researches in more effective ways. The research is unique in its essence as it offers a conceptual framework to explore affective component of attitude in prosodic pragmatics that has not yet been explored.

The perception of intonational and emotional speech prosody produced with and without a face mask: an exploratory individual differences study

Face masks affect the transmission of speech and obscure facial cues. Here, we examine how this reduction in acoustic and facial information affects a listener’s understanding of speech prosody. English sentence pairs that differed in their intonational (statement/question) and emotional (happy/sad) prosody were created. These pairs were recorded by a masked and unmasked speaker and manipulated to contain audio or not. This resulted in a continuum from typical unmasked speech with audio (easiest) to masked speech without audio (hardest). English listeners (N = 129) were tested on their discrimination of these statement/question and happy/sad pairs. We also collected six individual difference measures previously reported to affect various linguistic processes: Autism Spectrum Quotient, musical background, phonological short-term memory (digit span, 2-back), and congruence task (flanker, Simon) behavior. The results indicated that masked statement/question and happy/sad prosodies were harder to discriminate than unmasked prosodies. Masks can therefore make it more difficult to understand a speaker’s intended intonation or emotion. Importantly, listeners differed considerably in their ability to understand prosody. When wearing a mask, speakers should try to speak clearer and louder, if possible, and make intentions and emotions explicit to the listener.

Children's decoding of emotional prosody in four languages.

It is well established that adults can interpret emotional speech prosody independent of word meaning comprehension, even for emotional speech prosody in an unfamiliar language. However, the acquisition of this ability remains unclear. This study examined the decoding of four emotions (happy, sad, surprise, angry) conveyed with speech prosody in four languages (English, Chinese, French, Spanish) by American and Chinese children at 3 to 5 years of age-an age range when the ability to decode emotional prosody in one's native language emerges but remains fragile. Chinese and American children could decode the emotional meaning of speech prosody in both familiar and unfamiliar languages as young as 3 years old. Performance did not differ across the four languages used-a finding observed in both American and Chinese children. Thus, the in-group advantage of emotional prosody decoding reported for adults may not be evident by 5 years of age. Furthermore, emotional prosody decoding skills improved with age. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

Age-Related Changes in Voice Emotion Recognition by Postlingually Deafened Listeners With Cochlear Implants.

Identification of emotional prosody in speech declines with age in normally hearing (NH) adults. Cochlear implant (CI) users have deficits in the perception of prosody, but the effects of age on vocal emotion recognition by adult postlingually deaf CI users are not known. The objective of the present study was to examine age-related changes in CI users' and NH listeners' emotion recognition. Participants included 18 CI users (29.6 to 74.5 years) and 43 NH adults (25.8 to 74.8 years). Participants listened to emotion-neutral sentences spoken by a male and female talker in five emotions (happy, sad, scared, angry, neutral). NH adults heard them in four conditions: unprocessed (full spectrum) speech, 16-channel, 8-channel, and 4-channel noise-band vocoded speech. The adult CI users only listened to unprocessed (full spectrum) speech. Sensitivity (d') to emotions and Reaction Times were obtained using a single-interval, five-alternative, forced-choice paradigm. For NH participants, results indicated age-related declines in Accuracy and d', and age-related increases in Reaction Time in all conditions. Results indicated an overall deficit, as well as age-related declines in overall d' for CI users, but Reaction Times were elevated compared with NH listeners and did not show age-related changes. Analysis of Accuracy scores (hit rates) were generally consistent with d' data. Both CI users and NH listeners showed age-related deficits in emotion identification. The CI users' overall deficit in emotion perception, and their slower response times, suggest impaired social communication which may in turn impact overall well-being, particularly so for older CI users, as lower vocal emotion recognition scores have been associated with poorer subjective quality of life in CI patients.

What Do We Hear in the Voice? An Open-Ended Judgment Study of Emotional Speech Prosody.

The current study investigated what can be understood from another person's tone of voice. Participants from five English-speaking nations (Australia, India, Kenya, Singapore, and the United States) listened to vocal expressions of nine positive and nine negative affective states recorded by actors from their own nation. In response, they wrote open-ended judgments of what they believed the actor was trying to express. Responses cut across the chronological emotion process and included descriptions of situations, cognitive appraisals, feeling states, physiological arousal, expressive behaviors, emotion regulation, and attempts at social influence. Accuracy in terms of emotion categories was overall modest, whereas accuracy in terms of valence and arousal was more substantial. Coding participants' 57,380 responses yielded a taxonomy of 56 categories, which included affective states as well as person descriptors, communication behaviors, and abnormal states. Open-ended responses thus reveal a wide range of ways in which people spontaneously perceive the intent behind emotional speech prosody.

King Saud University Emotions Corpus: Construction, Analysis, Evaluation, and Comparison

Emotional speech recognition for the Arabic language is insufficiently tackled in the literature compared to other languages. In this paper, we present the work of creating and verifying the King Saud University Emotions (KSUEmotions) corpus, which was released by the Linguistic Data Consortium (LDC) in 2017 as the first public Arabic emotional speech corpus. KSUEmotions contains an emotional speech of twenty-three speakers from Saudi Arabia, Syria, and Yemen, and includes the emotions: neutral, happiness, sadness, surprise, and anger. The corpus content is verified in two different ways: a human perceptual test by nine listeners who rate emotional performance in audio files, and automatic emotion recognition. Two automatic emotion recognition systems are experimented with: Residual Neural Network and Convolutional Neural Network. This work also experiments with emotion recognition for the English language using the Emotional Prosody Speech and Transcripts Corpus (EPST). The current experimental work is conducted in three tracks: (i) monolingual, where independent experiments for Arabic and English are carried out, (ii) multilingual, where the Arabic and English corpora are merged in as mixed corpus, and (iii) cross-lingual, where models are trained using one language and tested using the other. A challenge encountered in this work is that the two corpora do not contain the same emotions. That problem is tackled by mapping the emotions to the arousal-valance space.

Sex differences in behavioral and brain responses to incongruity in emotional speech controlling for autistic traits

Accurate interpretation of speech requires the integration of verbal and nonverbal signals. This study investigated sex differences in behavior and neural activities associated with the integration of semantic content and emotional speech prosody, while the level of autistic traits was controlled for. Adults listened to Cantonese words spoken with happy and sad prosody, and made judgments on semantic valence while event-related potentials (ERP) were recorded. Behaviorally, men were slower than women in making semantic valence judgments. At the neural level, men had a greater congruity effect in the N400 component, whereas women had a greater congruity effect in the 1150–1300 ms time window for happy prosodies. There was no effect of sex in case of sad prosodies. Our study reveals novel findings on sex differences in the timing of the integration between verbal and non-verbal signals that cannot be explained by differences in autistic traits.

Emotional speech prosody perception in the first year of life

Prosodic aspects of speech are considered more salient than linguistic features in prenatal listening environment. The literature on early prosodic speech perception has been largely focused on the role of infant-directed speech in language development. While emotional prosody is also ubiquitous and contains diverse social cues, its perceptual development trajectory remains unclear. The current study adopted the central fixation paradigm to investigate 3- to 12-month-old infants' preferential listening attention to English monosyllabic words spoken in neutral, happy, angry, and sad tones. Forty-six infants completed the experiment and were assigned to younger and older groups with the median cut-off age of 8 months. Linear mixed-effect model results revealed a prevailing trend of shorter listening time for the sad tone. Furthermore, this reduced level of attention was only statistically significant in the younger infants but not the older group, suggesting that the lower-pitched and longer sad prosody was least attractive to the younger infants. With more exposure to emotional speech and the ongoing neural commitment towards language-specific speech perception in the older infants, the early differential interests in voice emotions showed signs of mitigation in the context of spoken words, leading to a relative increase in their listening time to the sad tone.

Speaker Identification in Different Emotional States in Arabic and English

Speaker recognition is an important application of digital speech processing. However, a major challenge degrading the robustness of speaker-recognition systems is variation in the emotional states of speakers, such as happiness, anger, sadness, or surprise. In this paper, we propose a speaker recognition system corresponding to three states, namely emotional, neutral, and with no consideration for a speaker's state (i.e., the speaker can be in an emotional state or neutral state), for two languages: Arabic and English. Additionally, cross-language speaker recognition was applied in emotional, neutral, and (emotional + neutral) states. Convolutional neural network and long short-term memory models were used to design a convolutional recurrent neural network (CRNN) main system. We also investigated the use of linearly spaced spectrograms as speech-feature inputs. The proposed system utilizes the KSUEmotions, emotional prosody speech and transcripts, WEST POINT, and TIMIT corpora. The CRNN system exhibited accuracies as high as 97.4% and 97.18% for Arabic and English emotional speech inputs, respectively, and 99.89% and 99.4% for Arabic and English neutral speech inputs, respectively. For the cross-language program, the overall CRNN system accuracy was as high as 91.83%, 99.88%, and 95.36% for emotional, neutral, and (emotional + neutral) states, respectively.

Emotional prosody in congenital amusia: Impaired and spared processes

Congenital amusia is a lifelong deficit of music processing, in particular of pitch processing. Most research investigating this neurodevelopmental disorder has focused on music perception, but pitch also has a critical role for intentional and emotional prosody in speech. Two previous studies investigating amusics’ emotional prosody recognition have shown either some deficit or no deficit (compared to controls). However, these previous studies have used only long sentence stimuli, which allow for limited control over acoustic content. Here, we tested amusic individuals for emotional prosody perception in sentences and vowels. For each type of material, participants performed an emotion categorization task, followed by intensity ratings of the recognized emotion. Compared to controls, amusic individuals had similar recognition of emotion in sentences, but poorer performance in vowels, especially when distinguishing sad and neutral stimuli. These lower performances in amusics were linked with difficulties in processing pitch and spectro-temporal parameters of the vowel stimuli. For emotion intensity, neither sentence nor vowel ratings differed between participant groups, suggesting preserved implicit processing of emotional prosody in amusia. These findings can be integrated into previous data showing preserved implicit processing of pitch and emotion in amusia alongside deficits in explicit recognition tasks. They are thus further supporting the hypothesis of impaired conscious analysis of pitch and timbre in this neurodevelopmental disorder.

The development of emotional speech prosody perception in 3- to 14-month infants: a preferential listening study

Developmental studies have shown strong evidence that socially enriched speech signal (including prosodic modifications) attracts infants’ attention and facilitates language development. While emotion understanding is evident at 9 months of age (Otte et al., 2015), the developmental trajectory of the emotional speech prosody perception is still unclear. The present study adopted a widely used preferential looking paradigm to measure 3- to 14-month-old infants’ listening preference to English words spoken in neutral, happy, angry, and sad tones. Analysis using a linear mixed model showed that infants’ preference of emotional prosody changed as a function of age. On average, the three-month olds listened longer to all emotional prosodies over the neutral one whereas older infants showed significantly diminished interests in the sad prosody, followed by the happy and angry voices. Around 12 months, infants appeared to listen to emotional prosodies equally with the exception of reduced interest in the angry prosody. These preferential listening measures were not correlated with the varying durations or fundamental frequencies of the spoken words for the different emotional categories, indicating that the development of emotional speech prosody is not purely driven by the acoustical properties but rather involves higher-order social cognition.

Cognitive and neural mechanisms underlying the mnemonic effect of songs after stroke.

Sung melody provides a mnemonic cue that can enhance the acquisition of novel verbal material in healthy subjects. Recent evidence suggests that also stroke patients, especially those with mild aphasia, can learn and recall novel narrative stories better when they are presented in sung than spoken format. Extending this finding, the present study explored the cognitive mechanisms underlying this effect by determining whether learning and recall of novel sung vs. spoken stories show a differential pattern of serial position effects (SPEs) and chunking effects in non-aphasic and aphasic stroke patients (N = 31) studied 6 months post-stroke. The structural neural correlates of these effects were also explored using voxel-based morphometry (VBM) and deterministic tractography (DT) analyses of structural MRI data. Non-aphasic patients showed more stable recall with reduced SPEs in the sung than spoken task, which was coupled with greater volume and integrity (indicated by fractional anisotropy, FA) of the left arcuate fasciculus. In contrast, compared to non-aphasic patients, the aphasic patients showed a larger recency effect (better recall of the last vs. middle part of the story) and enhanced chunking (larger units of correctly recalled consecutive items) in the sung than spoken task. In aphasics, the enhanced chunking and better recall on the middle verse in the sung vs. spoken task correlated also with better ability to perceive emotional prosody in speech. Neurally, the sung > spoken recency effect in aphasic patients was coupled with greater grey matter volume in a bilateral network of temporal, frontal, and parietal regions and also greater volume of the right inferior fronto-occipital fasciculus (IFOF). These results provide novel cognitive and neurobiological insight on how a repetitive sung melody can function as a verbal mnemonic aid after stroke.

Improving the performance of the speaker emotion recognition based on low dimension prosody features vector

Speaker emotion recognition is an important research issue as it finds lots of applications in human–robot interaction, computer–human interaction, etc. This work deals with the recognition of emotion of the speaker from speech utterance. For that features like pitch, log energy, zero crossing rate, and first three formant frequencies are used. Feature vectors are constructed using the 11 statistical parameters of each feature. The Artificial Neural Network (ANN) is chosen as a classifier owing to its universal function approximation capabilities. In ANN based classifier, the time required for training the network as well as for classification depends upon the dimension of feature vector. This work focused on development of a speaker emotion recognition system using prosody features as well as reduction of dimensionality of feature vectors. Here, principle component analysis (PCA) is used for feature vector dimensionality reduction. Emotional prosody speech and transcription from Linguistic Data Consortium (LDC) and Berlin emotional databases are considered for evaluating the performance of proposed approach for seven types of emotion recognition. The performance of the proposed method is compared with existing approaches and better performance is obtained with proposed method. From experimental results it is observed that 75.32% and 84.5% recognition rate is obtained for Berlin emotional database and LDC emotional speech database respectively.

Atypical perceptual and neural processing of emotional prosodic changes in children with autism spectrum disorders

ObjectiveThe present study explored the processing of emotional speech prosody in school-aged children with autism spectrum disorders (ASD) but without marked language impairments (children with ASD [no LI]). MethodsThe mismatch negativity (MMN)/the late discriminative negativity (LDN), reflecting pre-attentive auditory discrimination processes, and the P3a, indexing involuntary orienting to attention-catching changes, were recorded to natural word stimuli uttered with different emotional connotations (neutral, sad, scornful and commanding). Perceptual prosody discrimination was addressed with a behavioral sound-discrimination test. ResultsOverall, children with ASD (no LI) were slower in behaviorally discriminating prosodic features of speech stimuli than typically developed control children. Further, smaller standard-stimulus event related potentials (ERPs) and MMN/LDNs were found in children with ASD (no LI) than in controls. In addition, the amplitude of the P3a was diminished and differentially distributed on the scalp in children with ASD (no LI) than in control children. ConclusionsProcessing of words and changes in emotional speech prosody is impaired at various levels of information processing in school-aged children with ASD (no LI). SignificanceThe results suggest that low-level speech sound discrimination and orienting deficits might contribute to emotional speech prosody processing impairments observed in ASD.

Three-stage speaker verification architecture in emotional talking environments

Speaker verification performance in neutral talking environment is usually high, while it is sharply decreased in emotional talking environments. This performance degradation in emotional environments is due to the problem of mismatch between training in neutral environment while testing in emotional environments. In this work, a three-stage speaker verification architecture has been proposed to enhance speaker verification performance in emotional environments. This architecture is comprised of three cascaded stages: gender identification stage followed by an emotion identification stage followed by a speaker verification stage. The proposed framework has been evaluated on two distinct and independent emotional speech datasets: in-house dataset and Emotional Prosody Speech and Transcripts dataset. Our results show that speaker verification based on both gender information and emotion information is superior to each of speaker verification based on gender information only, emotion information only, and neither gender information nor emotion information. The attained average speaker verification performance based on the proposed framework is very alike to that attained in subjective assessment by human listeners.

Emotional Connotations of Musical Instrument Timbre in Comparison With Emotional Speech Prosody: Evidence From Acoustics and Event-Related Potentials.

Music and speech both communicate emotional meanings in addition to their domain-specific contents. But it is not clear whether and how the two kinds of emotional meanings are linked. The present study is focused on exploring the emotional connotations of musical timbre of isolated instrument sounds through the perspective of emotional speech prosody. The stimuli were isolated instrument sounds and emotional speech prosody categorized by listeners into anger, happiness and sadness, respectively. We first analyzed the timbral features of the stimuli, which showed that relations between the three emotions were relatively consistent in those features for speech and music. The results further echo the size-code hypothesis in which different sound timbre indicates different body size projections. Then we conducted an ERP experiment using a priming paradigm with isolated instrument sounds as primes and emotional speech prosody as targets. The results showed that emotionally incongruent instrument-speech pairs triggered a larger N400 response than emotionally congruent pairs. Taken together, this is the first study to provide evidence that the timbre of simple and isolated musical instrument sounds can convey emotion in a way similar to emotional speech prosody.