F0 Contour Research Articles

What’s so complex about conversational speech? A comparison of HMM-based and transformer-based ASR architectures

Highly performing speech recognition is important for more fluent human-machine interaction (e.g., dialogue systems). Modern ASR architectures achieve human-level recognition performance on read speech but still perform sub-par on conversational speech, which arguably is or, at least, will be instrumental for human-machine interaction. Understanding the factors behind this shortcoming of modern ASR systems may suggest directions for improving them. In this work, we compare the performances of HMM- vs. transformer-based ASR architectures on a corpus of Austrian German conversational speech. Specifically, we investigate how strongly utterance length, prosody, pronunciation, and utterance complexity as measured by perplexity affect different ASR architectures. Among other findings, we observe that single-word utterances – which are characteristic of conversational speech and constitute roughly 30% of the corpus – are recognized more accurately if their F0 contour is flat; for longer utterances, the effects of the F0 contour tend to be weaker. We further find that zero-shot systems require longer utterance lengths and are less robust to pronunciation variation, which indicates that pronunciation lexicons and fine-tuning on the respective corpus are essential ingredients for the successful recognition of conversational speech.

Acoustics of the alignment of narrow focus prosody and lexical stress in native and nonnative English productions.

This study examined the acoustic correlates of the alignment between lexical stress and narrow focus prosody in English and investigated how this alignment is learned by second language learners whose native language is prosodically distinct from English. A production experiment was conducted where native English speakers and Korean learners of English were asked to read aloud sentences with a narrow focus on trisyllabic words with various stress patterns (e.g., ánimal, Novémber, submaríne). In the native English productions, the stressed syllables were characterized by either a rising F0 contour (for word-initial stress) or increased periodic energy (for word-medial and -final stress). The Korean speakers (KSs) produced stressed syllables with a rising F0 contour regardless of stress position, differing from the native speakers, and they exhibited greater periodic energy for word-medial and -final stressed syllables, similar to the native speakers. In addition, the KSs with higher English proficiency showed more native-like patterns by exhibiting a post-focus F0 drop immediately after word-initial stressed syllables. These findings are discussed in the context of existing literature and potential factors contributing to the challenges observed in L2 prosody learning.

Pupillometry reveals effects of pitch manipulation within and across words on listening effort and short-term memory.

For individuals with hearing loss, even successful speech communication comes at a cost. Cochlear implants transmit degraded information, specifically for voice pitch, which demands extra and sustained listening effort. The current study hypothesized that abnormal pitch patterns contribute to the additional listening effort, even in non-tonal language native speaking normally hearing listeners. We manipulated the fundamental frequency (F0)within and across words, while participants listen and repeat (simple intelligibility task), or listen, repeat, and later recall (concurrent encoding task) the words. In both experiments, the F0 manipulations resulted in small changes in intelligibility but no difference in free recall or subjective effort ratings. Pupillary metrics were yet sensitive to these manipulations: pupil dilations were larger when words were monotonized (flat contour) or inverted (the natural contour flipped upside-down), and larger when successive words were organized into a melodic pattern. The most likely interpretation is that the natural or expected F0 contour of a word contributes to its identity and facilitate its matching and retrieval from the phonological representation stored in long-term memory. Consequently, degrading words' F0 contour can result in extra listening effort. Our results call for solutions to improve pitch saliency and naturalness in future development of cochlear implants' signal processing strategies, even for non-tonal languages.

Individual variability in the use of tonal and non-tonal cues in intonationa).

Greek uses H*, L + H*, and H* + L, all followed by L-L% edge tones, as nuclear pitch accents in statements. A previous analysis demonstrated that these accents are distinguished by F0 scaling and contour shape. This study expands the earlier investigation by exploring additional cues, namely, voice quality, amplitude, and duration, in distinguishing the pitch accents, and investigating individual variability in the selection of both F0 and non-F0 cues. Bayesian multivariate analysis and hierarchical clustering demonstrate that the accents are distinguished not only by F0 but also by additional cues at the group level, with individual variability in cue selection.

Segmental Influences on the Perception of High Pitch Accent Scaling in American English.

Researchers investigating a broad array of questions in spoken language prosody routinely base their arguments on measurements taken from the F0 contours of representative speech samples. These analyses, however, frequently involve abstracting F0 contours away from the segmental strings that bear them, potentially overlooking in the process the role played by segmental qualities such as sonority or periodicity in the realization of F0 patterns by speakers and their interpretation by listeners. This paper reports the results of two experiments investigating how perception of F0 contours is affected by the segmental string over which those contours are realized. The first focuses on gaps in F0 contours created by voiceless obstruents such as stops and fricatives, while the second investigates F0 intervals spanning lower-sonority voiced segments, such as nasals and voiced fricatives. While these two scenarios might at first seem unrelated, we argue that listeners treat both with a single mechanism in perception, namely, by reducing (potentially to zero) the amount of weight accorded to those portions of the contour for determination of the speaker's intended F0 scaling level. We present an account of both effects within a unified model of F0 scaling perception called TCoG-F, with discussion of its implications for phonetic and phonological intonation research going forward.

A Cognitive View on Prosodic Relations

The paper presents a new F0 contour partitioning approach by using the category of prosodic relations also used by Ladd (2008) for improving the F0 contour descriptions based on phonological categories. The new approach involves a cognitive view on the low-high and high-low ’metrical’ structures of prosodic relations, by relating them to the structures of cognitive relations generated during speech object representations at the cortical level. In section 2, the paper presents the information structure model by defining the cognitive categories that describes prosodic relations of F0 contours involving their two overlapped structures and nuclear positions. CU_predicate-CU_argument and CU_theme-CU_rheme are the two structural levels of prosodic relations. The model proposes a binary-tree hierarchy to describe the articulation of prosodic relations within utterances. Two rules are formulated for the identification of nuclear constituents of prosodic relations. The utterances analysed in section 3 illustrate how prosodic and prominence relations can be identified by analysing acoustic cues of their F0 contours. Utterances correspond to English borad focus and narrow focus statements. Focus positions are deduced by only using the rule of the cognitive model.

Mandarin-speaking Children With Cochlear Implants Face Challenges in Using F0 Expansion to Express Contrastive Focus.

Children with cochlear implants (CIs) face challenges in perceiving fundamental frequency (F0) information because CIs do not transmit F0 effectively. In Mandarin, F0 can contrast meanings at the word level, that is, via lexical tones with distinct F0 contours, and signal contrastive relations between words at the utterance-level, that is, via contrastive focus with expanded F0 range and longer duration. Mandarin-speaking children with CIs have been reported to face challenges in producing distinct F0 contours across tones, but early implantation facilitates tonal acquisition. However, it is still unclear if utterance-level prosody, such as contrastive focus, is also challenging for these children, and if early implantation also offers benefits for focus production. Therefore, this study asked how accurately children with CIs can produce contrastive focus, and if early implantation leads to more accurate focus production, with acoustic patterns approaching that of children with typical hearing (TH). Participants included 55 Mandarin-speaking children (3 to 7 years) with CIs and 55 age-matched children with TH. Children produced noun phrases with and without contrastive focus, such as RED-COLORED cat versus red-colored cat. Three adult native listeners perceptually scored the productions as correct or incorrect. The "correct" productions were then acoustically analyzed in terms of F0 range and duration. Based on the perceptual scores, children with CIs produced focus with significantly lower accuracy (38%) than their TH peers (84%). The acoustic analysis on their "correct" productions showed that children with TH used both F0 and duration to mark focus, producing focal syllables with an expanded F0 range and long duration, and postfocal syllables with a reduced F0 range and short duration. However, children with CIs differed from children with TH in that they produced focal syllables with long duration but not an expanded F0 range, although they produced postfocal syllables with a reduced F0 range and short duration like their TH peers. In addition, early implantation correlated with the percept of more accurate focus productions and better use of F0 range in focal marking. This study finds that Mandarin-speaking children with CIs are still learning to apply appropriate acoustic cues to contrastive focus. The challenge appears to lie in the use of an expanded F0 range to mark focus, probably related to the limited transmission of F0 information through the CI devices. These findings thus have implications for parents and those working with children with CIs, showing that utterance-level prosody also requires speech remediation, and underscores the critical role of identifying problems early in the acquisition of F0 functions in Mandarin, not only at the word level but also at the utterance-level.

A meta-analysis of expressive prosody in cochlear implant users

IntroductionProsody is used to express indexical (identifying the talker), linguistic (e.g., question intonation, lexical stress), pragmatic (e.g., contrastive stress, sarcasm), and emotional/affective functions. It is manifested through changes in fundamental frequency (f0), intensity, and duration. F0 and intensity are degraded when perceived through a cochlear implant (CI). The purpose of this meta-analysis is to compare expressive prosody in speech produced by CI users versus normal hearing peers. MethodsA systematic search of the literature found 25 articles that met all inclusion criteria. These articles were assessed for quality, and data pertaining to the expression of f0, intensity, and duration, as well as classification accuracy and appropriateness ratings from normal hearing listeners, were extracted and meta-analyzed using random effects models. ResultsThe articles included in the meta-analysis were generally of acceptable or high quality. Meta-analyses revealed significant differences between individuals with CIs vs. normal hearing on all measures except mean f0, mean intensity, and rhythm. Effect sizes were generally medium to large. There was significant heterogeneity across studies, but little evidence of publication bias. ConclusionsCI users speak with less variable f0, smaller f0 contours, more variable intensity, a slower speech rate, and reduced final lengthening at syntactic boundaries. These acoustic differences are reflected in significantly poorer ratings of speech produced by CI users compared to their normal hearing peers, as assessed by groups of normal hearing listeners. Because atypical expressive prosody is associated with negative outcomes, clinicians should consider targeting prosody when working with individuals who use CIs.

Revisiting the nature and the context of T4 alternations: insights from disyllabic, trisyllabic and quadrisyllabic word/digit productions in Taiwan Mandarin.

The tone values of a Tone 4 (T4) syllable are conventionally assumed to change from '51' to '53' when the syllable is followed by another T4 syllable in Mandarin Chinese. Literature focusing on T4 alternation is still inconclusive regarding the contexts for the alternations and whether the phenomenon should be better categorized as tone sandhi (i.e.,represented as an abstract phonological rule in mental grammar) or tonal coarticulation (i.e.,a natural articulation phenomenon at the phonetic level). The current study probes into these issues by focusing on disyllabic pseudowords, right-branching trisyllabic words as well as unstructured trisyllabic and quadrisyllabic digits. Productions from a total of 148 participants were collected and fundamental frequency (f0) contours, vowel lengths and f0 slopes were included in the analysis. The results from the experiments supported the tonal coarticulation view and showed that the trigger for the alternations was the high-onset tones following T4. Implications to the phonological analysis on tonal alternations in Mandarin Chinese are discussed.

The prosody of theme, rheme and focus in Egyptian Arabic: A quantitative investigation of tunes, configurations and speaker variability

This paper investigates the prosody of sentences elicited in three Information Structure (IS) conditions: all-new, theme-rheme and rhematic focus-background. The sentences were produced by 18 speakers of Egyptian Arabic (EA). This is the first quantitative study to provide a comprehensive analysis of holistic f0 contours (by means of GAMM) and configurations of f0, duration and intensity (by means of FPCA) associated with the three IS conditions, both across and within speakers. A significant difference between focus-background and the other information structure conditions was found, but also strong inter-speaker variation in terms of strategies and the degree to which these strategies were applied. The results suggest that post-focus register lowering and the duration of the stressed syllables of the focused and the utterance-final word are more consistent cues to focus than a higher peak of the focus accent. In addition, some independence of duration and intensity from f0 could be identified. These results thus support the assumption that, when focus is marked prosodically in EA, it is marked by prominence. Nevertheless, the fact that a considerable number of EA speakers did not apply prosodic marking and the fact that prosodic focus marking was gradient rather than categorical suggest that EA does not have a fully conventionalized prosodic focus construction.

Paradigmatic and syntagmatic effects of information status on prosodic prominence - evidence from an interactive web-based production experiment in German.

In this paper, we investigate how information status is encoded paradigmatically and syntagmatically via prosodic prominence in German. In addition, we consider individual variability in the production of prominence. To answer our research questions, we collected controlled yet ecologically valid speech by applying an innovative recording paradigm. Participants were asked to perform an interactive reading task in collaboration with an interlocutor remotely via video calls. Results indicate that information status is encoded paradigmatically via the F0 contour, while syntagmatic effects are subtle and depend on the acoustic parameter used. Individual speakers differ primarily in their strength of encoding and secondarily in the type of parameters employed. While the paradigmatic effects we observe are in line with previous findings, our syntagmatic findings support two contradictory ideas, a balancing effect and a radiating effect. Along with the findings at the individual level, this study thus allows for new insights regarding the redundant and relational nature of prosodic prominence.

Language-Specific Prosody in Statements of Palenquero/Spanish Bilinguals

This study explores the extent to which Palenquero/Spanish bilinguals, a population that is said to have a residual high tone of African origin, keep their two languages temporally and intonationally distinct across statements. While creole languages that emerged from the contact of African and European languages, such as Palenquero, may develop hybrid prosodic systems with tones from substrate languages, and stress from the majority language, language-specific prosody might be expected to converge or simplify over the course of time. As prosodic convergence seems to be inescapable under Palenquero’s circumstances, which factors could support language-specific prosody in this population, if there are any? Two-hundred and thirty-four five-syllable statements were elicited through a discourse completion task, with the participation of ten Palenquero/Spanish bilinguals, in two unilingual sessions. Both phrase-final lengthening and F0 contours were assessed using linear mixed-effects models testing their association with final stress, language, and generation. F0 contours were dimensionally reduced using functional principal component analysis. Despite the strong similarities between the two languages, results indicate that both groups keep their two languages intonationally distinct using plateau-shaped contours in Palenquero initial rises followed by steeper declinations in Spanish. However, elderly bilinguals implement penultimate lengthening language-specifically, being more pronounced in Palenquero. Adults, in contrast, do not show this distinction. In addition to this, elderly speakers show hyperarticulation in Spanish intonation, increasing the difference between their languages. This leads us to believe that adults exhibit a more simplified prosodic system between their languages, relative to elderly bilinguals. In spite of such differences, both generations seem to have the same underlying process (perhaps a substrate effect) driving plateau-shaped intonation in Palenquero, which enhances language differentiation.

Exploring and explaining variation in phrase-final f0 movements in spontaneous Papuan Malay.

This study investigates the variation in phrase-final f0 movements found in dyadic unscripted conversations in Papuan Malay, an Eastern Indonesian language. This is done by a novel combination of exploratory and confirmatory classification techniques. In particular, this study investigates the linguistic factors that potentially drive f0 contour variation in phrase-final words produced in a naturalistic interactive dialogue task. To this end, a cluster analysis, manual labelling and random forest analysis are carried out to reveal the main sources of contour variation. These are: taking conversational interaction into account; turn transition, topic continuation, information structure (givenness and contrast), and context-independent properties of words such as word class, syllable structure, voicing and intrinsic f0. Results indicate that contour variation in Papuan Malay, in particular f0 direction and target level, is best explained by turn transitions between speakers, corroborating similar findings for related languages. The applied methods provide opportunities to further lower the threshold of incorporating intonation and prosody in the early stages of language documentation.

Examining the time-varying measures of F0 in sarcasm: Wigglines, spaciousness, and contour clustering

The present project examines the phonetic correlates, specifically those related to F0, of sarcastic speech. In a production study, American English speakers (N = 12) produced identically worded utterance pairs presented in contexts conducive to sarcasm and sincerity. Measures of F0 variability are contrasted; these are wiggliness and spaciousness [Wehrle, Cangemi, Krüger, & Grice (2018) Proceedings of AISV], F0 range and F0 mean SD. Raw values were entered into by-speaker logistic regression models. Wiggliness and spaciousness together were found to be comparable to F0 mean SD and F0 range in distinguishing sarcasm and sincerity for eight of the speakers. Intonational characteristics were further examined via by-speaker F0 contour clustering [Kaland, 2021]. These by-speaker analyses showed that many speakers produce contours characteristic of sarcasm or sincerity, but that these contours differ by speaker. Further exploration of a subset of nine speakers’ data showed that wiggliness and spaciousness alone can capture some of the differences between sincere and sarcastic contour clusters for some speakers. Speaker strategies vary in terms of F0, and sarcastic speech is characterized by reduced wiggliness and spaciousness for some of the speakers.

Regional differences in the production of tones in standard mandarin

This study investigates the intricate patterns of tonal variations in monosyllabic words across distinct regional accents of Standard Mandarin, with a focus on Shanghai, Guangzhou, and Beijing. Due to the differing tonal systems of the local Chinese dialects in the three cities, speakers from these areas may exhibit distinct patterns in their production of tones in Standard Mandarin, influenced by their native Chinese dialects. Therefore, to capture the nuances of tonal differences, the study extracts the time-normalized f0 values of four tones (i.e., level, rising, dipping, and falling tones) from monosyllabic words in a Mandarin Chinese speech corpus. The overall f0 contours of the four tones within and across the three regional accents are modeled using growth curve analysis. We expect to find that regional accents of Standard Mandarin exhibit significant tonal variations such that the tone shapes of speakers from Shanghai and Guangzhou may significantly deviate from those of Beijing speakers. Specifically, there are variations in contour shape and temporal dynamics, particularly in the quadratic term for dipping tones and the slopes of rising and falling tones. These findings can provide valuable insights into the intricate characteristics of regional tonal variations in Standard Mandarin.

Visual cues influence prosody perception among individuals with cochlear implants

Understanding a talker’s intention conveyed through prosody is essential for successful speech communication. Although visual speech cues are typically characterized by lip movements, there are also visual gestures including head tilts and eyebrow raises that signal prosody. Listeners with cochlear implants (CIs) may rely on these visual cues to solidify their prosody perception and guard against misinterpretations, particularly because of their poor perception of voice pitch. This study used audio-visual recordings of sentences in which a talker sometimes focused one word to signify a change in meaning. After characterizing acoustic and visual prosody cues in this stimulus set, we made prosodically mismatched variants of the stimuli by transplanting prosodically focused audio onto broad focus (i.e., prosodically unfocused) video, and vice versa. Visual influences on prosody perception were measured using acoustic analysis of participant reproductions in a vocal mimicry task. Results show reproduction of F0 contour—a key acoustic marker of prosodic focus—is influenced by perception of visual prosody cues. Specifically, visual cues indicating a focused word led to higher peak F0 in vocal reproductions of that word in the sentence. These results underscore the importance of multisensory information in supporting prosody perception for listeners with hearing loss.

Lexical Stress Identification in Cochlear Implant-Simulated Speech by Non-Native Listeners.

This study investigates whether a presumed difference in the perceptibility of cues to lexical stress in spectro-temporally degraded simulated cochlear implant (CI) speech affects how listeners weight these cues during a lexical stress identification task, specifically in their non-native language. Previous research suggests that in English, listeners predominantly rely on a reduction in vowel quality as a cue to lexical stress. In Dutch, changes in the fundamental frequency (F0) contour seem to have a greater functional weight than the vowel quality contrast. Generally, non-native listeners use the cue-weighting strategies from their native language in the non-native language. Moreover, few studies have suggested that these cues to lexical stress are differently perceptible in spectro-temporally degraded electric hearing, as CI users appear to make more effective use of changes in vowel quality than of changes in the F0 contour as cues to linguistic phenomena. In this study, native Dutch learners of English identified stressed syllables in CI-simulated and non-CI-simulated Dutch and English words that contained changes in the F0 contour and vowel quality as cues to lexical stress. The results indicate that neither the cue-weighting strategies in the native language nor in the non-native language are influenced by the perceptibility of cues in the spectro-temporally degraded speech signal. These results are in contrast to our expectations based on previous research and support the idea that cue weighting is a flexible and transferable process.

An exploratory investigation of interactions between syllabic prominence, initial geminates, and phrasal boundaries in Pattani Malay

AbstractThis study investigates interactions among relative syllabic prominence, initial geminates (IGs), and prosodic boundaries in Pattani Malay (PM) against a background of previous analyses claiming that IGs are moraic and trigger a ‘stress shift’ or the linking of a pitch accent to the initial syllable. We conducted an acoustic study with fourteen PM speakers, producing singleton–IG minimal pairs in naturalistic sentences. Our results show that the presence of IGs is not associated with the hypothesized phonological changes. Instead, it is associated with moderate increases in the duration of initial syllables, the intensity of the initial syllable vowels, and the f0 of the initial and final syllable vowels. On the other hand, the presence of a phrase-final prosodic boundary correlates with more drastic changes: in phrase-final position, final syllables exhibit final lengthening and falling contours of f0 and intensity, while, in the phrase-medial position, no lengthening is observed and f0 contours are rising. Furthermore, the effects of IGs are strongest in the phrase-final position, suggesting interactions between IGs and prosodic boundaries. Taken together, results cast doubts on the claim that IGs are moraic and associated with categorical differences in syllabic prominence profiles in PM and show that IG effects are modulated by prosodic boundaries.

Speaker embedding space cosine similarity comparisons of singing voice conversion models and voice morphing

We explore the use of cosine similarity between x-vector speaker embeddings as an objective metric to evaluate the effectiveness of singing voice conversion. Our system preprocesses a source singer’s audio to obtain melody features via the F0 contour, loudness curve, and phonetic posteriorgram. These are input to a denoising diffusion probabilistic acoustic model conditioned with another target voice’s speaker embedding to generate a mel spectrogram, which is passed through a HiFi-GAN vocoder to synthesize audio of the source song in the target timbre. We use cosine similarity between the converted audio’s speaker embedding and that of the target voice as an objective metric in two tasks. First, we show that we can morph between two voices: a smooth transition between two speaker embeddings in latent space results in a smooth transition of timbre in generated audio. This shows potential for creativity in the speaker embedding latent space to represent new voices. Second, we use cosine similarity to compare our diffusion acoustic model with a model based on DurIAN. We find that the latter has better conversion results, fewer parameters, and less training time. Overall, we conclude that cosine similarity is a helpful objective metric for voice morphing and conversion.

On the nature of Tone 4 alternation in Taiwan Mandarin: Insights from acoustic analysis of trisyllabic right-branching word productions

Tone 4 (T4) alternations refer to the phenomenon where the tone values of a T4 syllable reduces from ‘51' to ‘53' when it is followed by another T4 syllable in Mandarin Chinese. It is inconclusive if the nature of T4 alternations is tone sandhi (an abstract phonological rule) (Chao, 1948; Jang, 2021) or tonal coarticulation (an articulatory phenomenon in speech production) (Lin, 20007; Shen, 1990). The current study invited 30 Taiwan Mandarin speakers to produce 10 pairs of right-branching trisyllabic words, including sequences of [T4[T4-T4]] (e.g., dà diàn shì 'big television') and [T4[T4-Tone1]] syllables (e.g., dà diàn jiā ‘big store'). The T4 sandhi view predicted that the surface tone values of the first syllables in these two sequences would differ (“51i” vs. “53”), while the T4 coarticulation view predicted that the tone values would be the same (“53”). Results from the acoustic analysis on fundamental frequency (f0) contour, f0 slope and vowel length of the first T4 syllables indicated that the differences were not statistically significant. These results supported the tonal coarticulation view. Discussions pertaining to the mental grammar for the tone value distances of two adjacent T4 syllables and the typology of tonal coarticulation in Mandarin Chinese are formulated.