Speech Prosody Research Articles

The specificity of cerebral lateralization in perception of emotional intonation of speech among children with special abilities

Much attention has been recently given to examination of developmental peculiarities of cognitive processes of children and adolescents with special abilities. The goal lies in comparative study of the ontogenetic peculiarities of cerebral mechanisms underlying the processing of emotional information in children who are engaged in extracurricular musical and mathematical activities. The sampling included the children engaged in additional activities and the control group, and consisted of the three age groups: 7-10 year, 10-13 years, and 14-17 years. In the conducted experimental studies, the respondents had to identify the emotional intonation of speech in the given phrase. The study revealed the significant ontogenetic differences in perception of emotional prosody of speech between the control group and the children engaged in additional activities. In the group of children of 7-13 years, who are engaged in additional classes, the predominant activity of left hemisphere testifies to the specificity of inter-hemispheric connections in the perception of emotional information. The dependence of the effects of cerebral lateralization on age was analogous in the groups of children engaged in mathematical and musical activities; and differed from the results of the control group, which allows claiming that the interinfluence of mathematical and musical skills is to a certain extent explained by the peculiarities of developmental age of cerebral lateralization in childhood and adolescence.

Crucial Music Components Needed for Speech Perception Enhancement of Pediatric Cochlear Implant Users: A Systematic Review and Meta-Analysis

Background: Although many clinicians have attempted music training for the hearing-impaired children, no specific effects have yet been reported for individual music components. This paper seeks to discover specific music components that help in improving speech perception of children with cochlear implants (CI) and to identify the effective training periods and methods needed for each component. Method: While assessing 5 electronic databases, that is, ScienceDirect, Scopus, PubMed, CINAHL, and Web of Science, 1,638 articles were found initially. After the screening and eligibility assessment stage based on the Participants, Intervention, Comparisons, Outcome, and Study Design (PICOS) inclusion criteria, 18 of 1,449 articles were chosen. Results: A total of 18 studies and 14 studies (209 participants) were analyzed using a systematic review and meta-analysis, respectively. No publication bias was detected based on an Egger’s regression result even though the funnel plot was asymmetrical. The results of the meta-analysis revealed that the largest improvement was seen for rhythm perception, followed by the perception of pitch and harmony and smallest for timbre perception after the music training. The duration of training affected the rhythm, pitch, and harmony perception but not the timbre. Interestingly, musical activities, such as singing, produced the biggest effect size, implying that children with CI obtained the greatest benefits of music training by singing, followed by playing an instrument and achieved the smallest effect by only listening to musical stimuli. Significant improvement in pitch perception helped with the enhancement of prosody perception. Conclusion: Music training can improve the music perception of children with CI and enhance their speech prosody. Long training duration was shown to provide the largest training effect of the children’s perception improvement. The children with CI learned rhythm and pitch better than they did with harmony and timbre. These results support the finding of past studies that with music training, both rhythm and pitch perception can be improved, and it also helps in the development of prosody perception.

Prosodic Interactions in Anishinaabemowin Verbs

This research aims to better understand the link between prosody and verbs in Anishinaabemowin by investigating pitch placement in relation to verb placement in Anishinaabemowin utterances. The data is from a story by Ogimaawigwaebiik archived in Dibaajimowinaan; Anishinaabe Stories of Culture and Respect. 
 Anishinaabemowin, also known as Ojibwe, is a member of the Algonquin language family and is spoken throughout Southern Ontario and the Northern United States (Fairbanks, 2017). It is a polysynthetic language meaning it primarily uses affixes to convey meaning, particularly on the verbs. Prosody is the organization of various linguistics units (words, pitch, tone) into an utterance in the process of speech production. It conveys not only linguistics information but also contextual cues, intentions and attitudes (Fujisaki, 1997). 
 This research utilized two audio softwares, Audacity and Praat, to clean and segment the audio into utterances and then token sentences were selected based on verb placement (verb initial, verb second and verb final). These token sentences will be analyzed for pitch placement and then compared to see if verb placement affects prosody, further expanding on the current literature which states that pitch defaults to the verb (Frazier, accepted). This research is particularly important because there is a gap in existing literature on prosody in Anishinaabemowin and there are no experimental studies such as this. 
 

The duration of intrauterine development influences discrimination of speech prosody in infants.

Auditory speech discrimination is essential for normal language development. Children born preterm are at greater risk of language developmental delays. Using functional near-infrared spectroscopy at term-equivalent age, the present study investigated early discrimination of speech prosody in 62 neonates born between week 23 and 41 of gestational age (GA). We found a significant positive correlation between GA at birth and neural discrimination of forward versus backward speech at term-equivalent age. Cluster analysis identified a critical threshold at around week 32 of GA, pointing out the existence of subgroups. Infants born before week 32 of GA exhibited a significantly different pattern of hemodynamic response to speech stimuli compared to infants born at or after week 32 of GA. Thus, children born before the GA of 32weeks are especially vulnerable to early speech discrimination deficits. To support their early language development, we therefore suggest a close follow-up and additional speech and language therapy especially in the group of children born before week 32 of GA.

Distinct signatures of subjective confidence and objective accuracy in speech prosody

Whether speech prosody truly and naturally reflects a speaker's subjective confidence, rather than other dimensions such as objective accuracy, is unclear. Here, using a new approach combing psychophysics with acoustic analysis and automatic classification of verbal reports, we tease apart the contributions of sensory evidence, accuracy, and subjective confidence to speech prosody. We find that subjective confidence and objective accuracy are distinctly reflected in the loudness, duration and intonation of verbal reports. Strikingly, we show that a speaker's accuracy is encoded in speech prosody beyond their own metacognitive awareness, and that it can be automatically decoded from this information alone with performances up to 60%. These findings demonstrate that confidence and accuracy have separable prosodic signatures that are manifested with different timings, and on different acoustic dimensions. Thus, both subjective mental states of confidence, and objective states related to competence, can be directly inferred from this natural behavior.

Encoding and decoding of meaning through structured variability in intonational speech prosody

Speech prosody plays an important role in communication of meaning. The cognitive and computational mechanisms supporting this communication remain to be understood, however. Prosodic cues vary across talkers and speaking conditions, creating ambiguity in the sound-to-meaning mapping. We hypothesize that listeners ameliorate this ambiguity in part by learning talker-specific statistics of prosodic cues. To test this hypothesis, we investigate the production and recognition of question vs. statement prosody in American English. Experiment 1 elicits productions of questions and statements from 65 talkers to examine the distributional statistics characterizing within- and cross-talker variability in these productions. We use Bayesian ideal observer models to assess the predicted consequences of cross-talker variability on listeners' recognition of prosody. We find that learning of talker-specific distributional statistics is predicted to facilitate recognition, above and beyond what can be achieved via commonly assumed normalizations of prosodic cues. Experiment 2 tests this prediction in a comprehension experiment. We expose different groups of listeners to different prosodic input statistics and assess listeners' recognition of questions and statements both prior to, and following, exposure. Prior to exposure, ideal observer-derived predictions based on Experiment 1 provide a good qualitative fit against listeners' recognition of prosodic contours in Experiment 2. Following exposure, listeners shift the categorization boundary between questions and statements in ways consistent with learning of talker-specific statistics.

From image to text to speech: the effects of speech prosody on information sequencing in audio description

Abstract Given the extensive body of research in audio description – the verbal-vocal description of visual or audiovisual content for visually impaired audiences – it is striking how little attention has been paid thus far to the spoken dimension of audio description and its para-linguistic, prosodic aspects. This article complements the previous research into how audio description speech is received by the partially sighted audiences by analyzing how it is performed vocally. We study the audio description of pictorial art, and one aspect of prosody is examined in detail: pitch, and the segmentation of information in relation to it. We analyze this relation in a corpus of audio described pictorial art in Finnish by combining phonetic measurements of the pitch with discourse analysis of the information segmentation. Previous studies have already shown that a sentence-initial high pitch acts as a discourse-structuring device in interpreting. Our study shows that the same applies to audio description. In addition, our study suggests that there is a relationship between the scale in the rise of pitch and the scale of the topical transition. That is, when the topical transition is clear, the rise of pitch level between the beginnings of two consecutive spoken sentences is large. Analogically, when the topical transition is small, the change of the sentence-initial pitch level is also rather small.

Irish English PEPS-C (2015 edition) and Learners of ESL

Introduction: Profiling Elements of Prosody in Speech-Communication (PEPS-C) is a computerised test for assessing the abilities to understand and use speech prosody in communication. It has been used to obtain a profile of strengths and weaknesses in different prosodic forms and functions for different clinical populations (e.g., autism spectrum disorders, Down syndrome) and second language learners. The 2015 edition of PEPS-C incorporates four new subtests addressing the understanding and production of lexical stress and phrasal stress, and collapses four form subtests (Intonation/Short-Item Input and Output, Prosody/Long-Item Input and Output) into two (Discrimination, Imitation). However, the suitability of these new tasks has not been reported in any published studies, although they are likely to be relevant for learners of English. Moreover, the present authors update the Irish English (IE) version of PEPS-C to the 2015 edition for another research project on prosodic skills in children with spina bifida. Hence, this paper reports the making of PEPS-C 2015 (IE) and examines the usefulness of this test in assessing a group of non-native speakers of English, as compared to a group of native speakers of IE. Methods: PEPS-C 2015 for Irish English was developed by adapting the test items of the UK English version of the test. The PEPS-C 2015 (IE) was then trialled on 25 native speakers of Irish English and 10 Spanish speakers who speak English as a second language (ESL). Results: All native speakers of Irish English showed competence (scored ≥75% correct) in the comprehension and expression of prosodic form and functions assessed. For the ESL speakers, the test identified two areas of possible difficulty for this group: Phrase Stress Comprehension and Expression, and Contrastive Stress Comprehension. Conclusion: The PEPS-C 2015, with its extra stress tasks, might therefore be useful as a prosody assessment tool for ESL speakers, particularly those with a Romance first language or at an early stage of learning, but further research is needed.

Emotivity in the Voice: Prosodic, Lexical, and Cultural Appraisal of Complaining Speech.

Emotive speech is a social act in which a speaker displays emotional signals with a specific intention; in the case of third-party complaints, this intention is to elicit empathy in the listener. The present study assessed how the emotivity of complaints was perceived in various conditions. Participants listened to short statements describing painful or neutral situations, spoken with a complaining or neutral prosody, and evaluated how complaining the speaker sounded. In addition to manipulating features of the message, social-affiliative factors which could influence complaint perception were varied by adopting a cross-cultural design: participants were either Québécois (French Canadian) or French and listened to utterances expressed by both cultural groups. The presence of a complaining tone of voice had the largest effect on participant evaluations, while the nature of statements had a significant, but smaller influence. Marginal effects of culture on explicit evaluation of complaints were found. A multiple mediation analysis suggested that mean fundamental frequency was the main prosodic signal that participants relied on to detect complaints, though most of the prosody effect could not be linearly explained by acoustic parameters. These results highlight a tacit agreement between speaker and listener: what characterizes a complaint is how it is said (i.e., the tone of voice), more than what it is about or who produces it. More generally, the study emphasizes the central importance of prosody in expressive speech acts such as complaints, which are designed to strengthen social bonds and supportive responses in interactive behavior. This intentional and interpersonal aspect in the communication of emotions needs to be further considered in research on affect and communication.

Mental Simulations of Phonological Representations Are Causally Linked to Silent Reading of Direct Versus Indirect Speech.

Embodied theories propose that language is understood via mental simulations of sensory states related to perception and action. Given that direct speech (e.g., She says, “It’s a lovely day!”) is perceived to be more vivid than indirect speech (e.g., She says (that) it’s a lovely day) in perception, recent research shows in silent reading that more vivid speech representations are mentally simulated for direct speech than for indirect speech. This ‘simulated’ speech is found to contain suprasegmental prosodic representations (e.g., speech prosody) but its phonological detail and its causal role in silent reading of direct speech remain unclear. Here in three experiments, I explored the phonological aspect and the causal role of speech simulations in silent reading of tongue twisters in direct speech, indirect speech and non-speech sentences. The results demonstrated greater visual tongue-twister effects (phonemic interference) during silent reading (Experiment 1) but not oral reading (Experiment 2) of direct speech as compared to indirect speech and non-speech. The tongue-twister effects in silent reading of direct speech were selectively disrupted by phonological interference (concurrent articulation) as compared to manual interference (finger tapping) (Experiment 3). The results replicated more vivid speech simulations in silent reading of direct speech, and additionally extended them to the phonological dimension. Crucially, they demonstrated a causal role of phonological simulations in silent reading of direct speech, at least in tongue-twister reading. The findings are discussed in relation to multidimensionality and task dependence of mental simulation and its mechanisms.

L'importance de la prosodie dans les interactions orthophoniste-enfant

Most of the work that has been d edicated to the role of prosody in interactions between speech and language therapists and children has focused on patients' prosodic disorders and their treatment, rarely on the prosody of speech and language therapists. This study aims to partly fill this gap by qualitatively examining a corpus of video recordings of the interactions that occur in two speech therapist-child dyads in a therapeutic situation. The examination of the data revealed that the speech and language therapist spontaneously exploited her prosody to indicate to the children the nature of the activity in progress (e.g., informal exchange, planned language task), to point out to the children the erroneous nature of some of their productions and to help correct them. The review of the data also highlighted occasions when prosody was under-exploited. When prosody was used, its use was carried out unconsciously by the speech and language therapist, and appeared to be the result of her professional experience rather than a prior sensitization to the im portance of this tool. The results of this exploratory study tend to support a developed integration of prosody as a therapeutic goal and device during the initial and continuing education of speech and language therapists.

Chapter 9 - Mood and emotional disorders associated with parkinsonism, Huntington disease, and other movement disorders

This chapter provides a review of mood, emotional disorders, and emotion processing deficits associated with diseases that cause movement disorders, including Parkinson's disease, Lewy body dementia, multiple system atrophy, progressive supranuclear palsy, corticobasal degeneration, frontotemporal dementia with parkinsonism, Huntington's disease, essential tremor, dystonia, and tardive dyskinesia. For each disorder, a clinical description of the common signs and symptoms, disease progression, and epidemiology is provided. Then the mood and emotional disorders associated with each of these diseases are described and discussed in terms of clinical presentation, incidence, prevalence, and alterations in quality of life. Alterations of emotion communication, such as affective speech prosody and facial emotional expression, associated with these disorders are also discussed. In addition, if applicable, deficits in gestural and lexical/verbal emotion are reviewed. Throughout the chapter, the relationships among mood and emotional disorders, alterations of emotional experiences, social communication, and quality of life, as well as treatment, are emphasized.

Pitch Contour Stylization by Marking Voice Intonation

The stylization of pitch contour is a primary task in the speech prosody for the development of a linguistic model. The stylization of pitch contour is performed either by statistical learning or statistical analysis. The recent statistical learning models require a large amount of data for training purposes and rely on complex machine learning algorithms. Whereas, the statistical analysis methods perform stylization based on the shape of the contour and require further processing to capture the voice intonations of the speaker. The objective of this paper is to devise a low-complexity transcription algorithm for the stylization of pitch contour based on the voice intonation of a speaker. For this, we propose to use of pitch marks as a subset of points for the stylization of the pitch contour. The pitch marks are the instance of glottal closure in a speech waveform that captures characteristics of speech uttered by a speaker. The selected subset can interpolate the shape of the pitch contour and acts as a template to capture the intonation of a speaker’s voice, which can be used for designing applications in speech synthesis and speech morphing. The algorithm balances the quality of the stylized curve and its cost in terms of the number of data points used. We evaluate the performance of the proposed algorithm using the mean square error and the number of lines used for fitting the pitch contour. Furthermore, we perform a comparison with other existing stylization algorithms using the LibriSpeech ASR corpus.

Deep bidirectional neural networks for robust speech recognition under heavy background noise

The Deep Bidirectional LSTM (DBLSTM) with recurrent capability using the old conventional HMM modals performing very well and responding and providing a state-of-the-art technology with very good performance on the on various languages and speech database. Even with the current technologies, the outputs in these works relying on recurrent base neural network schemes. On account of explicit target strategies, it turns out to be extremely hard to coordinate the current day accessible discourse acknowledgment techniques due to the current huge jargon discourse acknowledgment frameworks. This research investigates the possibility of using the DBLSTM as speech acoustic speech miniature in which a calibre neural network scheme with HMM that is a hybrid structure. With few experimental findings we observed that the DBLSTM-HMM hybrid gives better and higher performance and provides better results on various NN schemes and all other previous works the previous works. It is also observed that it outclasses both conventional and deep network benchmarks on a most of the known existing algorithms. The improvements which are occurring in word error rate in the deep network algorithms is modest, considering the fact that a significant elevate in slice level precision. We propose an hybrid technique in which DBLSTM and HMM emerge to be exactly matched for tasks where acoustic modeling and speech prosodies predominates. To add more investigation, need to be conducted on large database and also need to see whether this is working on some specific languages.

King Saud University Emotions Corpus: Construction, Analysis, Evaluation, and Comparison

Emotional speech recognition for the Arabic language is insufficiently tackled in the literature compared to other languages. In this paper, we present the work of creating and verifying the King Saud University Emotions (KSUEmotions) corpus, which was released by the Linguistic Data Consortium (LDC) in 2017 as the first public Arabic emotional speech corpus. KSUEmotions contains an emotional speech of twenty-three speakers from Saudi Arabia, Syria, and Yemen, and includes the emotions: neutral, happiness, sadness, surprise, and anger. The corpus content is verified in two different ways: a human perceptual test by nine listeners who rate emotional performance in audio files, and automatic emotion recognition. Two automatic emotion recognition systems are experimented with: Residual Neural Network and Convolutional Neural Network. This work also experiments with emotion recognition for the English language using the Emotional Prosody Speech and Transcripts Corpus (EPST). The current experimental work is conducted in three tracks: (i) monolingual, where independent experiments for Arabic and English are carried out, (ii) multilingual, where the Arabic and English corpora are merged in as mixed corpus, and (iii) cross-lingual, where models are trained using one language and tested using the other. A challenge encountered in this work is that the two corpora do not contain the same emotions. That problem is tackled by mapping the emotions to the arousal-valance space.

Speech-Brain Frequency Entrainment of Dyslexia with and without Phonological Deficits.

Developmental dyslexia is a cognitive disorder characterized by difficulties in linguistic processing. Our purpose is to distinguish subtypes of developmental dyslexia by the level of speech–EEG frequency entrainment (δ: 1–4; β: 12.5–22.5; γ1: 25–35; and γ2: 35–80 Hz) in word/pseudoword auditory discrimination. Depending on the type of disabilities, dyslexics can divide into two subtypes—with less pronounced phonological deficits (NoPhoDys—visual dyslexia) and with more pronounced ones (PhoDys—phonological dyslexia). For correctly recognized stimuli, the δ-entrainment is significantly worse in dyslexic children compared to controls at a level of speech prosody and syllabic analysis. Controls and NoPhoDys show a stronger δ-entrainment in the left-hemispheric auditory cortex (AC), anterior temporal lobe (ATL), frontal, and motor cortices than PhoDys. Dyslexic subgroups concerning normolexics have a deficit of δ-entrainment in the left ATL, inferior frontal gyrus (IFG), and the right AC. PhoDys has higher δ-entrainment in the posterior part of adjacent STS regions than NoPhoDys. Insufficient low-frequency β changes over the IFG, the inferior parietal lobe of PhoDys compared to NoPhoDys correspond to their worse phonological short-term memory. Left-dominant 30 Hz-entrainment for normolexics to phonemic frequencies characterizes the right AC, adjacent regions to superior temporal sulcus of dyslexics. The pronounced 40 Hz-entrainment in PhoDys than the other groups suggest a hearing “reassembly” and a poor phonological working memory. Shifting up to higher-frequency γ-entrainment in the AC of NoPhoDys can lead to verbal memory deficits. Different patterns of cortical reorganization based on the left or right hemisphere lead to differential dyslexic profiles.

A meta-analysis of prosody in autism, Williams syndrome, and Down syndrome

Prosody, the rhythm and melody of speech, is an important component of effective communication, and it is an area of difficulty for many populations who struggle with communication. This paper is a meta-analysis of nine studies (and two sets of unpublished data) that assessed prosody using the Profiling Elements of Prosody in Speech-Communication (PEPS-C; Peppé & McCann, 2003) in autism spectrum disorder, Williams syndrome, and Down syndrome. Our original goal was to include studies involving any neurodevelopmental disorder that is commonly associated with intellectual disability, yet our systematic search, which included three databases (i.e., PsychInfo, ERIC, and PubMed), only identified studies involving these three groups. To be included in the meta-analysis, studies had to include a group (n ≥ 3 participants) with a neurodevelopmental disorder commonly associated with intellectual disability and a typically developing comparison group matched on chronological age, nonverbal abilities, or verbal abilities. Studies also needed to report original data using the PEPS-C and be available in English. Study quality was assessed using a checklist adapted from Downes et al. (2016). Results revealed that prosodic form was a weakness for each etiology, while unique patterns of strengths and weaknesses were evident for prosodic functions. Groups with autism (n = 5), all classified as high-functioning or with Asperger’s syndrome, exhibited weakness in emotional affect but some relative strengths with turn-end and focus tasks. Groups with Williams syndrome (n = 4) demonstrated weaknesses on phrase/sentence-level tasks and relative strengths on word-level tasks. Groups with Down syndrome (n = 2) had the greatest difficulty overall, though performance was better on receptive (vs. expressive) function tasks. By combining studies and related subtasks of the PEPS-C, we are able to more confidently generalize findings for each population and identify targets for intervention. However, given the limited number of studies identified, this paper also highlights the need for more research on prosody in intellectual disability.

Interpretation of Depression Detection Models via Feature Selection Methods.

Given the prevalence of depression worldwide and its major impact on society, several studies employed artificial intelligence modelling to automatically detect and assess depression. However, interpretation of these models and cues are rarely discussed in detail in the AI community, but have received increased attention lately. In this study, we aim to analyse the commonly selected features using a proposed framework of several feature selection methods and their effect on the classification results, which will provide an interpretation of the depression detection model. The developed framework aggregates and selects the most promising features for modelling depression detection from 38 feature selection algorithms of different categories. Using three real-world depression datasets, 902 behavioural cues were extracted from speech behaviour, speech prosody, eye movement and head pose. To verify the generalisability of the proposed framework, we applied the entire process to depression datasets individually and when combined. The results from the proposed framework showed that speech behaviour features (e.g. pauses) are the most distinctive features of the depression detection model. From the speech prosody modality, the strongest feature groups were F0, HNR, formants, and MFCC, while for the eye activity modality they were left-right eye movement and gaze direction, and for the head modality it was yaw head movement. Modelling depression detection using the selected features (even though there are only 9 features) outperformed using all features in all the individual and combined datasets. Our feature selection framework did not only provide an interpretation of the model, but was also able to produce a higher accuracy of depression detection with a small number of features in varied datasets. This could help to reduce the processing time needed to extract features and creating the model.

Temporal Coordination and Prosodic Structure in Autism Spectrum Disorder: Timing Across Speech and Non-speech Motor Domains.

Individuals with autism spectrum disorder (ASD) often exhibit disordered speech prosody, but sources of disordered prosody remain poorly understood. We explored patterns of temporal alignment and prosodic grouping in a speech-based metronome repetition task as well as manual coordination in a drum tapping task among Cantonese speakers with ASD and normal nonverbal IQ and matched controls. Results indicate similar group results for prosodic grouping patterns, but significant differences in relative timing and longer syllable durations at phrase ends for the ASD group. Variability on the speech task was significantly correlated with variability on the drumming task, consistent with the view that impairment in both speech and non-speech motor domains can be linked with deficits in temporal processing.

Sex differences in behavioral and brain responses to incongruity in emotional speech controlling for autistic traits

Accurate interpretation of speech requires the integration of verbal and nonverbal signals. This study investigated sex differences in behavior and neural activities associated with the integration of semantic content and emotional speech prosody, while the level of autistic traits was controlled for. Adults listened to Cantonese words spoken with happy and sad prosody, and made judgments on semantic valence while event-related potentials (ERP) were recorded. Behaviorally, men were slower than women in making semantic valence judgments. At the neural level, men had a greater congruity effect in the N400 component, whereas women had a greater congruity effect in the 1150–1300 ms time window for happy prosodies. There was no effect of sex in case of sad prosodies. Our study reveals novel findings on sex differences in the timing of the integration between verbal and non-verbal signals that cannot be explained by differences in autistic traits.