Prosodic Cues Research Articles

On the relation between lexical and prosodic cues of emotion in English: a preliminary corpus-based analysis

De nombreuses etudes ont montre que la parole emotionnelle est caracterisee par des indices sur plusieurs niveaux linguistiques. Dans cet article, nous analysons la relation entre les indices lexicaux et prosodiques des emotions en anglais dans les enregistrements de livres audios du corpus LibriSpeech. Afin de quantifier les indices lexicaux des emotions, nous avons utilise les outils d’analyse des sentiments disponibles dans la librairie Python NLTK (notamment Vader et SentiWordNet). En revanche, pour la quantification des indices prosodiques des emotions, nous avons extrait plusieurs parametres acoustiques indiquant les variations de la frequence fondamentale, de la qualite de la voix et du rythme. La relation entre indices lexicaux et prosodiques a ete ensuite analysee a travers des modeles lineaires a effets mixtes, qui ont revele des effets faibles mais significatifs. En particulier, seules les variations de F0 sembleraient liees a la valence des phrases et des mots, alors que les variations de F0, du rythme et de la qualite de la voix sembleraient liees au degre de positivite / negativite des phrases et des mots. Cependant, nos modeles n'expliquent qu'une faible part de variance des donnees. Nous estimons donc que la relation entre les indices lexicaux et prosodiques dans les enregistrements de livres audios existe mais est plutot faible. Il est possible que cette observation puisse s’appliquer de maniere plus generale aux cas de conceptualisation ou description des emotions pour lesquelles le locuteur ne se sent pas directement implique. En revanche, il est possible que la relation entre les indices lexicaux et prosodiques s’avere plus etroite dans les cas d’expression des emotions ou le locuteur se sentirait plus directement implique. De toute maniere, nos resultats ne doivent etre consideres que comme preliminaires, etant donne les limites d'une analyse entierement automatique.

The Language-specific Use of Fundamental Frequency Rise in Segmentation of an Artificial Language: Evidence from Listeners of Taiwanese Southern Min.

Experience with native-language prosody encourages language-specific strategies for speech segmentation. Conflicting findings from previous research suggest that these strategies may not be abstracted away from the acoustic manifestation of prosodic features in the native speech. Using the artificial language learning paradigm, the current study explores this possibility in connection with listeners of a lexical tone language called Taiwanese Southern Min (TSM). In TSM, the only rising lexical tone occurs almost only on the final syllable of the language's tone sandhi domain and is phonetically associated with final lengthening. Based on these observations, Experiment I examined what constituted a sufficient finality cue for use by TSM listeners to support segmentation: (a) final fundamental frequency (F0) rise only; or (b) final F0 rise conjoined with final lengthening. The results showed that segmentation was inhibited by the former cue but facilitated by the latter. Experiment II showed that the facilitation cannot be attributed entirely to final lengthening, as a null effect was found when final lengthening was the sole prosodic cue to segmentation. It is thus assumed that acoustic details as fine-grained as the lengthening of the rising tone are involved in the modulation of the segmentation strategy whereby TSM listeners perceive F0 rise as signaling finality. The inhibitory effect of final F0 rise alone found in Experiment I motivated Experiment III, which revealed that initial F0 rise in the absence of lengthening cues improved TSM listeners' segmentation. It is speculated that such use of initial F0 rise might reflect a cross-linguistic segmentation solution.

Finding phrases: On the role of co-verbal facial information in learning word order in infancy.

The input contains perceptually available cues, which might allow young infants to discover abstract properties of the target language. Thus, word frequency and prosodic prominence correlate systematically with basic word order in natural languages. Prelexical infants are sensitive to these frequency-based and prosodic cues, and use them to parse new input into phrases that follow the order characteristic of their native languages. Importantly, young infants readily integrate auditory and visual facial information while processing language. Here, we ask whether co-verbal visual information provided by talking faces also helps prelexical infants learn the word order of their native language in addition to word frequency and prosodic prominence. We created two structurally ambiguous artificial languages containing head nods produced by an animated avatar, aligned or misaligned with the frequency-based and prosodic information. During 4 minutes, two groups of 4- and 8-month-old infants were familiarized with the artificial language containing aligned auditory and visual cues, while two further groups were exposed to the misaligned language. Using a modified Headturn Preference Procedure, we tested infants’ preference for test items exhibiting the word order of the native language, French, vs. the opposite word order. At 4 months, infants had no preference, suggesting that 4-month-olds were not able to integrate the three available cues, or had not yet built a representation of word order. By contrast, 8-month-olds showed no preference when auditory and visual cues were aligned and a preference for the native word order when visual cues were misaligned. These results imply that infants at this age start to integrate the co-verbal visual and auditory cues.

AGE-RELATED DECLINES IN SOCIAL COGNITIVE PROCESSES OF OLDER ADULTS

Despite current literature suggesting that various social cognitive processes seem to be impaired in late adulthood, e.g., processing of social gaze cues, the trajectory decline of social cognition in late adulthood is not well understood (e.g., Grainger et al., 2018; Paal & Bereczkei, 2007). As part of a multi-institutional research project, we began to systematically investigate whether there is age-related decline in older adults’ ability to infer others’ mental states, integrate multiple referential cues, and identify emotional states of others using prosodic cues. Sixteen older adults aged 71-85, of which 9 were cognitively healthy and 7 with mild-to-moderate dementia, and 7 younger adults aged 19-37 underwent three tasks. In a theory-of-mind story task, participants answered true/false questions about the beliefs of the protagonists in the stories. A cue integration task assessed participants’ ability to integrate the experimenter’s gaze and semantic cues to identify a referent object. In an emotion-prosody task, participants judged whether the speaker sounded happy or sad in low-pass filtered audio. Non-parametric tests revealed that younger adults outperformed both groups of older adults (both ps=.001) in inferring the protagonists’ beliefs in the stories. Younger adults were also better and more accurate than both groups of older adults in integrating cues to identify the referent object and in using prosodic cues to identify emotional states respectively (ps<.001). Both groups of older adults did not differ significantly from each other in the tasks. These findings provide emerging and important insights into the decline of social cognitive processes in late adulthood.

A deficit in using prosodic cues to understand communicative intentions by children with autism spectrum disorders: An eye-tracking study

The present study investigated whether Mandarin-speaking preschool children with autism spectrum disorders (ASD) were able to use prosodic cues to understand others’ communicative intentions. Using the visual world eye-tracking paradigm, the study found that unlike typically developing (TD) 4-year-olds, both 4-year-olds with ASD and 5-year-olds with ASD exhibited an eye gaze pattern that reflected their inability to use prosodic cues to infer the intended meaning of the speaker. Their performance was relatively independent of their verbal IQ and mean length of utterance. In addition, the findings also show that there was no development in this ability from 4 years of age to 5 years of age. The findings indicate that Mandarin-speaking preschool children with ASD exhibit a deficit in using prosodic cues to understand the communicative intentions of the speaker, and this ability might be inherently impaired in ASD.

Tinnitus affects the relative roles of semantics and prosody in the perception of emotions in spoken language

Objective: Understanding communication difficulties related to tinnitus, by identifying tinnitus-related differences in the perception of spoken emotions, focussing on the roles of semantics (words), prosody (tone of speech) and their interaction.Study sample and design: Twenty-two people-with-tinnitus (PwT) and 24 people-without-tinnitus (PnT) listened to spoken sentences made of different combinations of four discrete emotions (anger, happiness, sadness, neutral) presented in the prosody and semantics (Test for Rating Emotions in Speech). In separate blocks, listeners were asked to attend to the sentence as a whole, integrating both speech channels (gauging integration), or to focus on one channel only (gauging identification and selective attention). Their task was to rate how much they agree the sentence conveys each of the predefined emotions.Results: Both groups identified emotions similarly, and performed with similar failures of selective attention. Group differences were found in the integration of channels. PnT showed a bias towards prosody, whereas PwT weighed both channels equally.Conclusions: Tinnitus appears to impact the integration of the prosodic and semantic channels. Three possible sources are suggested: (a) sensory: tinnitus may reduce prosodic cues. (b) Cognitive: tinnitus-related reduction in cognitive processing. (c) Affective: group differences were related to the existence of tinnitus, but not to the extent of tinnitus complaints and/or affective symptoms.

Myth busted? Challenging what we think we know about charismatic speech

Charisma is a complex phenomenon. This fact manifests itself not least in an abundance of myths, half-truths, and unanswered research questions. Most charisma myths have not been uncontroversial, and since empirical investigations have advanced quickly over the past years, we take the opportunity in this paper to revisit ten of the most important myths that relate primarily, but not exclusively, to the linguistic and phonetic aspects of charisma, such as the interactions between verbal and nonverbal and between segmental and prosodic cues, as well as the roles of breathing and fundamental frequency in charisma perception. The result is a very diverse picture. Some myths, including very old ones, can be accepted. Others must be rejected in the light of contradicting empirical results. The status of some myths remains unsettled. Furthermore, in discussing that diverse picture, our paper points towards knowledge gaps in research and practice and gives concrete directions as to where to go from here.

Prosodic cues in second-language speech processing: A visual world eye-tracking study

Listeners interpret cues in speech processing immediately rather than waiting until the end of a sentence. In particular, prosodic cues in auditory speech processing can aid listeners in building information structure and contrast sets. Native speakers even use this information in combination with syntactic and semantic information to build mental representations predictively. Research on second-language (L2) learners suggests that learners have difficulty integrating linguistic information across various domains, likely subject to L2 proficiency levels. The current study investigated eye-movement behavior of native speakers of English and Chinese learners of English in their use of contrastive intonational cues to restrict the set of upcoming referents in a visual world paradigm. Both native speakers and learners used contrastive pitch accent to restrict the set of referents. Whereas native speakers anticipated the upcoming set of referents, this was less clear in the L2 learners. This suggests that learners are able to integrate information across multiple domains to build information structure in the L2 but may not do so predictively. Prosodic processing was not affected by proficiency or working memory in the L2 speakers.

How Vocal Emotions Produced by Children With Cochlear Implants Are Perceived by Their Hearing Peers.

Purpose Cochlear implants (CIs) transmit a degraded version of the acoustic input to the listener. This impacts the perception of harmonic pitch, resulting in deficits in the perception of voice features critical to speech prosody. Such deficits may relate to changes in how children with CIs (CCIs) learn to produce vocal emotions. The purpose of this study was to investigate happy and sad emotional speech productions by school-age CCIs, compared to productions by children with normal hearing (NH), postlingually deaf adults with CIs, and adults with NH. Method All individuals recorded the same emotion-neutral sentences in a happy manner and a sad manner. These recordings were then used as stimuli in an emotion recognition task performed by child and adult listeners with NH. Their performance was taken as a measure of how well the 4 groups of talkers communicated the 2 emotions. Results Results showed high variability in the identifiability of emotions produced by CCIs, relative to other groups. Some CCIs produced highly identifiable emotions, while others showed deficits. The postlingually deaf adults with CIs produced highly identifiable emotions and relatively small intersubject variability. Age at implantation was found to be a significant predictor of performance by CCIs. In addition, the NH listeners' age predicted how well they could identify the emotions produced by CCIs. Thus, older NH child listeners were better able to identify the CCIs' intended emotions than younger NH child listeners. In contrast to the deficits in their emotion productions, CCIs produced highly intelligible words in the sentences carrying the emotions. Conclusions These results confirm previous findings showing deficits in CCIs' productions of prosodic cues and indicate that early auditory experience plays an important role in vocal emotion productions by individuals with CIs.

Disambiguating recasts with prosodic and extra-linguistic cues in task-based interactions among young learners

Abstract This study investigates the relative effects of enhanced and unenhanced recasts on young learners’ question development in L2 English, examining whether enhanced recasts can contribute to disambiguation. Forty-six Secondary One students in Hong Kong were evenly divided into an enhanced recast group and an unenhanced recast group. Each participant participated in a pretest, followed by three treatment sessions in three consecutive weeks. Participants in the enhanced recast group received enhanced recasts with prosodic (tonal stress) and extra-linguistic (gestures and facial expressions) cues from the teacher, whereas those in the unenhanced recast group received normal recasts when they made mistakes when asking questions. After that, an immediate post-test and a two-week delayed post-test were carried out to examine participants’ progress in L2 question formation. The results of the immediate post-test showed that both types of recasts contributed to L2 advancement, but the effects of enhanced recasts were relatively strong as compared with unenhanced recasts. Both groups performed significantly better in the delayed post-test than in the pretest, indicating the robust effects of recasts (enhanced or unenhanced) on learning L2 question formation. However, the differences between the two groups were significantly reduced in the delayed post-test. In addition, the enhanced group scored significantly lower in the delayed post-test than in the immediate post-test, suggesting a greater degenerating effect of enhanced recasts than unenhanced recasts.

Rhythmic structure facilitates learning from auditory input in newborn infants

Rhythm and metrical regularities are fundamental properties of music and poetry - and all of those are used in the interaction between infants and their parents. Music and rhythm perception have been shown to support auditory and language skills. Here we compare newborn infants’ learning from a song, a nursery rhyme, and normal speech for the first time in the same study. Infants’ electrophysiological brain responses revealed that the nursery rhyme condition facilitated learning from auditory input, and thus led to successful detection of deviations. These findings suggest that coincidence of prosodic cue patterns and to-be-learned items is more important than the format of the input. Overall, the present results support the view that rhythm is likely to create a template for future events, which allows auditory system to predict prospective input and thus facilitates language development.

PROSODY FACILITATES MEMORY RECALL IN L1 BUT NOT IN L2 IN HIGHLY PROFICIENT LISTENERS

AbstractProsody is crucial for language comprehension because it highlights underlying structures. This study explores whether prosody facilitates memory recall to the same extent in L1 and L2, and whether memory recall is poorer in L2 or whether language-specific differences can mitigate L2 processing difficulties. Nineteen Greek learners of English, and a monolingual English baseline, repeated three-digit chunks with and without prosodic cues in L1 and L2. Prosody was a major facilitator of memory recall only in L1 despite the high proficiency of learners. This indicates that L2 mastery of prosody perception is hard to attain, mirroring production studies. However, when prosodic boundary cues were absent, memory recall in L2 was comparable to L1. This demonstrates that language-specific differences can attenuate more general processing difficulties in L2. This study is the first to demonstrate differences in prosodic processing in L1 and L2 resulting in poorer memory recall in L2.

Acting in action: Prosodic analysis of character portrayal during acting.

During the process of acting, actors have to embody the characters that they are portraying by changing their vocal and gestural features to match standard conceptions of the characters. In this experimental study of acting, we had professional actors portray a series of stock characters (e.g., king, bully, lover), which were organized according to a predictive scheme based on the 2 orthogonal personality dimensions of assertiveness and cooperativeness. We measured 12 prosodic features of the actors' vocal productions, as related to pitch, loudness, timbre, and duration/timing. The results showed a significant effect of character assertiveness on all 12 vocal parameters, but a weaker effect of cooperativeness on fewer vocal parameters. These findings comprise the first experimental analysis of vocal gesturing during character portrayal in actors and demonstrate that actors reliably manipulate prosodic cues in a contrastive manner to differentiate characters based on their personality traits. (PsycINFO Database Record (c) 2019 APA, all rights reserved).

Multiple accent in alternative questions

Alternative Questions (AltQs) are typically characterized by two prosodic cues: a final falling boundary tone and a pitch accent on each disjunct. Recent accounts in the literature have taken the final fall as the central surface cue for AltQ interpretation or have assigned a vacuous semantic contribution to the multiple accent on the disjuncts. Based on data from English and Turkish, we argue that both cues are equally important and require modelling in a unified account of AltQs. Combining ingredients from the literature (Roberts, 1996; Biezma, 2009; Westera, 2017), we propose that, essentially, the multiple accent shapes the Question under Discussion (QUD) and that the final fall, or the lack thereof, indicates restrictions on the content of the QUD via (un)satisfaction of Attention Maxims.

Parallels in Processing Boundary Cues in Speech and Action.

Speech and action sequences are continuous streams of information that can be segmented into sub-units. In both domains, this segmentation can be facilitated by perceptual cues contained within the information stream. In speech, prosodic cues (e.g., a pause, pre-boundary lengthening, and pitch rise) mark boundaries between words and phrases, while boundaries between actions of an action sequence can be marked by kinematic cues (e.g., a pause, pre-boundary deceleration). The processing of prosodic boundary cues evokes an Event-related Potentials (ERP) component known as the Closure Positive Shift (CPS), and it is possible that the CPS reflects domain-general cognitive processes involved in segmentation, given that the CPS is also evoked by boundaries between subunits of non-speech auditory stimuli. This study further probed the domain-generality of the CPS and its underlying processes by investigating electrophysiological correlates of the processing of boundary cues in sequences of spoken verbs (auditory stimuli; Experiment 1; N = 23 adults) and actions (visual stimuli; Experiment 2; N = 23 adults). The EEG data from both experiments revealed a CPS-like broadly distributed positivity during the 250 ms prior to the onset of the post-boundary word or action, indicating similar electrophysiological correlates of boundary processing across domains, suggesting that the cognitive processes underlying speech and action segmentation might also be shared.

Sequential Interpretation of Pitch Prominence as Contrastive and Syntactic Information: Contrast Comes First, but Syntax Takes Over.

Pitch accent serves multiple duties (encoding lexical accent, syntactic structure, and focus) in spoken Japanese. This study investigates how listeners interpret a role-ambiguous pitch prominence surfacing as F0 rise, which could be a cue to the resolution of a syntactic ambiguity between two possible branching structures, or a signal of contrastive focus on the constituent accompanied by the rise. Two visual world paradigm experiments tested the same Japanese linguistic stimuli with and without pitch emphasis on the second word of structures of the following form: modifier + N1 + N2. In Experiment 1, the visual context suppressed the availability of the contrastive interpretation; in Experiment 2, the visual context made the contrastive interpretation available. We found that the same pitch event can be interpreted as both syntax-encoding and contrast-encoding information within the course of processing the same sentence, as long as contextual information is made visually available. When contrastive focus is pragmatically felicitous, it is computed immediately, as soon as the incoming input is accompanied by a notable pitch prominence (Experiment 2). The same prosodic cue can then be re-interpreted as a signal to syntax after the branching ambiguity is recognized due to subsequent input (Experiments 1 and 2). This is most consistent with the view that an initially assigned prosodic boundary is exploited for re-interpretation.

The glottal stop between segmental and suprasegmental processing: The case of Maltese

• Maltese uses the glottal stop as both an underlying and epenthetic segment. • Words with and without an initial underlying glottal stop can become homophonous. • Epenthetic glottal stops are partly prosodically conditioned. • Listeners take prosodic information into account in the late stages of speech perception. • The interaction between prosodic and segmental processing needs to be considered. Many languages mark vowel-initial words with a glottal stop. We show that this occurs in Maltese, even though the glottal stop also occurs as a phoneme in Maltese. As a consequence, words with and without an underlying (phonemic) glottal stop (e.g., a glottal stop-zero minimal pair qal /Ɂɑ:l/ vs. ghal /ɑ:l/ Engl., ‘he said’-‘because’) can become homophonous in connected speech. We first tested the extent of this phonetic marking of vowel-initial words in a production experiment and found that even in fluent productions, about half of the vowel-initial words are marked with an epenthetic glottal stop. The epenthetic glottal stop is more likely to occur when the preceding word is longer, showing a kind of preboundary lengthening at a phrase-level prosodic boundary. A subsequent perception study (Experiment 2) using a two-alternative forced-choice task with a minimal pair of a glottal stop-initial and a vowel-initial word indicated that listeners are sensitive to the durationally conditioned prosodic context before the test word, and they are more likely to perceive a vowel-initial word when the preceding word is lengthened. An additional eye-tracking study (Experiment 3) using onset-overlap pairs (e.g., qafla /Ɂɑflɑ/ - afda, /ɑfdɑ/ → [Ɂɑfda], Engl., ‘to trust’ - ‘chord’) showed no early influence of prosodic cues on segmental processing. But a gating experiment (Experiment 4) replicated the prosodic effect observed in Experiment 2. Taken together, our results indicate an interaction between prosodic processing and segmental processing that comes into effect relatively late in speech processing.

Danish dialogue particles in an interactional perspective

In our paper, we give an overview over what is known about some of the most frequent interjections in Danish talk-in-interaction: ja (‘yes’), nej (‘no’), mm (‘mm’), nå (approximately ‘oh’), and okay (‘okay’). We review the CA/IL literature on these words, and we present our own exemplary analyses of single instances of these words in extracts from our corpus of recorded, naturally occurring Danish interactions. Based on this, we argue that sequential position, epistemics, and affiliation and alignment should be taken into account when describing and categorizing dialogue particles in talk-in-interaction. Prosody and other phonetic cues are important for the realization of the above dimensions and functions and we review what is known about prosodic and phonetic cues plus add some of our own observations, without launching a full phonetic and prosodic analysis.

Bridging the brain structure—brain function gap in prosodic speech processing in older adults

Age-related decline in speech perception may result in difficulties partaking in spoken conversation and potentially lead to social isolation and cognitive decline in older adults. It is therefore important to better understand how age-related differences in neurostructural factors such as cortical thickness (CT) and cortical surface area (CSA) are related to neurophysiological sensitivity to speech cues in younger and older adults. Age-related differences in CT and CSA of bilateral auditory-related areas were extracted using FreeSurfer in younger and older adults with normal peripheral hearing. Behavioral and neurophysiological sensitivity to prosodic speech cues (word stress and fundamental frequency of oscillation) was evaluated using discrimination tasks and a passive oddball paradigm, while EEG was recorded, to quantify mismatch negativity responses. Results revealed (a) higher neural sensitivity (i.e., larger mismatch negativity responses) to word stress in older adults compared to younger adults, suggesting a higher importance of prosodic speech cues in the speech processing of older adults, and (b) lower CT in auditory-related regions in older compared to younger individuals, suggesting neuronal loss associated with aging. Within the older age group, less neuronal loss (i.e., higher CT) in a right auditory-related area (i.e., the supratemporal sulcus) was related to better performance in fundamental frequency discrimination, while higher CSA in left auditory-related areas was associated with higher neural sensitivity toward prosodic speech cues as evident in the mismatch negativity patterns. Overall, our results offer evidence for neurostructural changes in aging that are associated with differences in the extent to which left and right auditory-related areas are involved in speech processing in older adults. We argue that exploring age-related differences in brain structure and function associated with decline in speech perception in older adults may help develop much needed rehabilitation strategies for older adults with central age-related hearing loss.

Voice Emotion Recognition by Children With Mild-to-Moderate Hearing Loss.

Emotional communication is important in children's social development. Previous studies have shown deficits in voice emotion recognition by children with moderate-to-severe hearing loss or with cochlear implants. Little, however, is known about emotion recognition in children with mild-to-moderate hearing loss. The objective of this study was to compare voice emotion recognition by children with mild-to-moderate hearing loss relative to their peers with normal hearing, under conditions in which the emotional prosody was either more or less exaggerated (child-directed or adult-directed speech, respectively). We hypothesized that the performance of children with mild-to-moderate hearing loss would be comparable to their normally hearing peers when tested with child-directed materials but would show significant deficits in emotion recognition when tested with adult-directed materials, which have reduced prosodic cues. Nineteen school-aged children (8 to 14 years of age) with mild-to-moderate hearing loss and 20 children with normal hearing aged 6 to 17 years participated in the study. A group of 11 young, normally hearing adults was also tested. Stimuli comprised sentences spoken in one of five emotions (angry, happy, sad, neutral, and scared), either in a child-directed or in an adult-directed manner. The task was a single-interval, five-alternative forced-choice paradigm, in which the participants heard each sentence in turn and indicated which of the five emotions was associated with that sentence. Reaction time was also recorded as a measure of cognitive load. Acoustic analyses confirmed the exaggerated prosodic cues in the child-directed materials relative to the adult-directed materials. Results showed significant effects of age, specific emotion (happy, sad, etc.), and test materials (better performance with child-directed materials) in both groups of children, as well as susceptibility to talker variability. Contrary to our hypothesis, no significant differences were observed between the 2 groups of children in either emotion recognition (percent correct or d' values) or in reaction time, with either child- or adult-directed materials. Among children with hearing loss, degree of hearing loss (mild or moderate) did not predict performance. In children with hearing loss, interactions between vocabulary, materials, and age were observed, such that older children with stronger vocabulary showed better performance with child-directed speech. Such interactions were not observed in children with normal hearing. The pattern of results was broadly consistent across the different measures of accuracy, d', and reaction time. Children with mild-to-moderate hearing loss do not have significant deficits in overall voice emotion recognition compared with their normally hearing peers, but mechanisms involved may be different between the 2 groups. The results suggest a stronger role for linguistic ability in emotion recognition by children with normal hearing than by children with hearing loss.