Pitch Contour Research Articles

Modification of Prosody for Emotion Conversion using Gaussian Regression Model

Emotion conversion is one of the most inspiring forefronts of research in the arena of emotional speech synthesis. The main focus of the work is to convert a neutral speech sentence to the target emotional speech sentence using signal processing techniques. The parameters used for emotion conversion are pitch contour and intensity along with the duration of the sentence. Kannada Emotional Speech (KES) Database is created and used for analysis. The database consists of 4 (sadness, happy, anger, and fear) emotions with neutral. The pitch contour of different emotional sentences are analyzed and Gaussian Regression Model (GRM) is proposed for predicting the target pitch contour. The evaluation of the proposed method is done using Objective test & Subjective test. For objective test, mean pitch, the standard deviation of pitch, mean intensity and duration of the sentences are used. Evaluation using a subjective test is performed by calculating Emotion Recognition Rate (ERR) with the help of confusion matrix and also by taking the Mean Opinion Score (MOS) rating of the conversion system on the scale of 1-5. The result of Subjective test indicates that the effectiveness and discernment of emotion are improved when GRM is used for pitch contour modification with intensity and duration. The most recognized emotion was sadness with MOS of 3.52 and ERR of 83% and the least recognized emotion was anger with MOS of 1.74 and ERR of 66%. The results of the subjective and objective test show that the converted sadness, happy and fear speech is seeming very close to usual sadness, anger and fear emotion.

Protracted Development on Native Tone Interpretation: Evidence From Mandarin-Learning Infants' Novel Word Learning.

Studies have shown that infants from cultures with tone languages develop categorical perception of their native lexical tone before their first birthday, but few studies have explored whether, and when, they interpret the phonemic function of lexical tone in word learning. Two habituation-switch experiments were conducted to explore whether Mandarin-learning infants could exploit tonal cues during their word learning, and detect a change when the association of two word-object pairs was switched. In Experiment 1, two words were solely differentiated by their lexical tones (/fāi/ vs. /făi/), and Mandarin-learning infants failed to detect the switch of tones at 14 months, but succeeded at 18 months. In Experiment 2, two words were markedly distinct (/fāi/ vs. /bǒu/), and infants could detect the change of words as early as 14 months. The results indicate that infants may not refer to the lexical function of tone during their novel word learning until 18 months, even though infants from birth are able to distinguish the Tone 1 vs. Tone 3 contrast. Given that lexical tone is expressed by variations of the pitch contours, which are also related to intonation, infants’ increasing knowledge of both tone and intonation may contribute to their misinterpretation of pitch contours in word learning at 14 months and, further, to their development of a sophisticated use of the phonemic function of lexical tone at 18 months of age.

The influence of tone language experience and speech style on the use of intonation in language discrimination.

This work tests whether listeners' use of suprasegmental information in speech perception is modulated by language background and speech style. Native Mandarin (tone language) and Malay (non-tone language) listeners completed an AX language discrimination task with four levels of signal degradation and two speech styles. Listeners in both groups showed more benefit from pitch information in read than in spontaneous speech. Mandarin listeners showed a greater benefit than Malay listeners from the inclusion of f0 information in a segmentally degraded signal, suggesting that experience with lexical tone may extend to increased attention and/or sensitivity to phrase-level pitch contours.

Probing the Underlying Principles of Perceived Immanent Accents Using a Modeling Approach.

This article deals with the question of how the perception of the “immanent accents” can be predicted and modeled. By immanent accent we mean any musical event in the score that is related to important points in the musical structure (e.g., tactus positions, melodic peaks) and is therefore able to capture the attention of a listener. Our aim was to investigate the underlying principles of these accented notes by combining quantitative modeling, music analysis and experimental methods. A listening experiment was conducted where 30 participants indicated perceived accented notes for 60 melodies, vocal and instrumental, selected from Baroque, Romantic and Post-tonal styles. This produced a large and unique collection of perceptual data about the perceived immanent accents, organized by styles consisting of vocal and instrumental melodies within Western art music. The music analysis of the indicated accents provided a preliminary list of musical features that could be identified as possible reasons for the raters’ perception of the immanent accents. These features related to the score in different ways, e.g., repeated fragments, single notes, or overall structure. A modeling approach was used to quantify the influence of feature groups related to pitch contour, tempo, timing, simple phrasing, and meter. A set of 43 computational features was defined from the music analysis and previous studies and extracted from the score representation. The mean ratings of the participants were predicted using multiple linear regression and support vector regression. The latter method (using cross-validation) obtained the best result of about 66% explained variance (r = 0.81) across all melodies and for a selected group of raters. The independent contribution of each feature group was relatively high for pitch contour and timing (9.6 and 7.0%). There were also significant contributions from tempo (4.5%), simple phrasing (4.4%), and meter (3.9%). Interestingly, the independent contribution varied greatly across participants, implying different listener strategies, and also some variability across different styles. The large differences among listeners emphasize the importance of considering the individual listener’s perception in future research in music perception.

The Role of Low-frequency Neural Oscillations in Speech Processing: Revisiting Delta Entrainment.

Studies that use measures of cerebro-acoustic coherence have shown that theta oscillations (3-10 Hz) entrain to syllable-size modulations in the energy envelope of speech. This entrainment creates sensory windows in processing acoustic cues. Recent reports submit that delta oscillations (<3 Hz) can be entrained by nonsensory content units like phrases and serve to process meaning-though such views face fundamental problems. Other studies suggest that delta underlies a sensory chunking linked to the processing of sequential attributes of speech sounds. This chunking associated with the "focus of attention" is commonly manifested by the temporal grouping of items in sequence recall. Similar grouping in speech may entrain delta. We investigate this view by examining how low-frequency oscillations entrain to three types of stimuli (tones, nonsense syllables, and utterances) having similar timing, pitch, and energy contours. Entrainment was indexed by "intertrial phase coherence" in the EEGs of 18 listeners. The results show that theta oscillations at central sites entrain to syllable-size elements in speech and tones. However, delta oscillations at frontotemporal sites specifically entrain to temporal groups in both meaningful utterances and meaningless syllables, which indicates that delta may support but does not directly bear on a processing of content. The findings overall suggest that, although theta entrainment relates to a processing of acoustic attributes, delta entrainment links to a sensory chunking that relates to a processing of properties of articulated sounds. The results also show that measures of intertrial phase coherence can be better suited than cerebro-acoustic coherence in revealing delta entrainment.

STYLISTIC ANALYSIS OF ENGLISH IMPERATIVE COMMAND MOOD TRANSFER TO JAVANESE

Imperative Mood (IM) for Javanese is strongly related to social status determining language choice. Triglossia of Ngoko, Madya and Krama are pictured in language use as well as prosodic features of pitch contour. To look at how English IM transfers to Javanese is the purpose of this research as well as how this process influences variant of pitch contours produced by Javanese speaker. This is a comparative study highlighting difference and similarity visualized by Javanese and English pitch contours with similar semantic meaning to be analyzed in pragmatics stylistic framework. Imperative to Command (IC) cannot take place in Bottom-Up situation in Javanese for social reason which puts Imperative to Request (IR) and Imperative to Invite (II) as substitutions. Using sound production of Javanese from Jogja and Solo on English and Javanese imperative, intonation pattern from both languages is analyzed using PRAAT and described in pitch range. Findings of this research display cultural background of Javanese in IM which may be useful for English pronunciation class.DOI:doi.org/10.24071/llt.2019.220102

On the perception of raga motifs by trained musicians.

A prominent aspect of the notion of musical similarity across the music of various cultures is related to the local matching of melodic motifs. This holds for Indian art music, a highly structured form with raga playing a critical role in the melodic organization. Apart from the tonal material, a raga is characterized by a set of melodic phrases that serve as important points of reference in a music performance. Musicians acquire in their training a knowledge of the melodic phrase shapes or motifs particular to a raga and the proficiency to render these correctly in performance. This phenomenon of learned schema might be expected to influence the musicians' perception of variations of the melodic motif in terms of pitch contour shape. Motivated by the parallels between the musical structure and prosodic structure in speech, identification and discrimination experiments are presented, which explore the differences between trained musicians' (TMs) and non-musicians' perception of ecologically valid synthesized variants of a raga-characteristic motif, presented both in and out of context. It is found that trained musicians are relatively insensitive to acoustic differences associated with note duration in the vicinity of a prototypical phrase shape while also clearly demonstrating the heightened sensitivity associated with categorical perception in the context of the boundary between ragas.

Joint Detection and Classification of Singing Voice Melody Using Convolutional Recurrent Neural Networks

Singing melody extraction essentially involves two tasks: one is detecting the activity of a singing voice in polyphonic music, and the other is estimating the pitch of a singing voice in the detected voiced segments. In this paper, we present a joint detection and classification (JDC) network that conducts the singing voice detection and the pitch estimation simultaneously. The JDC network is composed of the main network that predicts the pitch contours of the singing melody and an auxiliary network that facilitates the detection of the singing voice. The main network is built with a convolutional recurrent neural network with residual connections and predicts pitch labels that cover the vocal range with a high resolution, as well as non-voice status. The auxiliary network is trained to detect the singing voice using multi-level features shared from the main network. The two optimization processes are tied with a joint melody loss function. We evaluate the proposed model on multiple melody extraction and vocal detection datasets, including cross-dataset evaluation. The experiments demonstrate how the auxiliary network and the joint melody loss function improve the melody extraction performance. Furthermore, the results show that our method outperforms state-of-the-art algorithms on the datasets.

Non-speech and speech pitch perception among Cantonese-speaking children with autism spectrum disorder: An ERP study

The present study compared how Cantonese-speaking children with Autism Spectrum Disorder (ASD) and their typically developing counterparts perceived speech pitch and non-speech pitch information using ERP measurements. Sixteen children with ASD (mean age = 10.42 years, SD = 2.12 years) and sixteen normal controls (mean age = 9.48 years, SD =.86 years) participated in two experiments, in which Cantonese lexical tone contrasts and non-speech pitch variations were presented to children following an oddball paradigm when they watched a silent movie. The results showed that: 1) When processing speech pitch contour, the two groups did not differ in the amplitude of mismatch response (p-MMR), while typically developing controls showed larger mismatch negativity (MMN) responses than children with ASD. In the processing of speech pitch height, more positive p-MMR was observed among children with ASD than among normal controls and stronger MMN was found for typically developing children than for children with ASD. 2) For the processing of non-speech pitch, MMN rather than p-MMR was observed and the two groups did not differ significantly with each other in the amplitudes of MMN. These results indicated that Cantonese-speaking children with ASD manifested impaired ability when processing speech pitch information (i.e., lexical tone), which was in line with previous research. However, they did not show the advantage in processing non-speech or auditory pitch information, which was not in agreement with the previous studies. Results were discussed from the perspective of how language background (i.e., Cantonese) might shape the perceptive abilities of children.

Characterising maternal pitch contours used during interactions with infants at high and low risk for autism spectrum disorder

Infant-directed speech (IDS) is the speech register used when interacting with infants. Pitch contours are a salient aspect of IDS and facilitate infant language and socio-communicative development. Little research investigates pitch contours within the context of socio-communication or language deficits, such as infants at high-risk (HR) for Autism Spectrum Disorder (autism). The aim of this study was to characterise pitch contours used by mothers when interacting with HR infants compared to mothers interacting with low-risk (LR) infants. 18 mothers and their 12-month-old infant (12m, 6f) participated in 15-minute recorded interactions. Autism risk was assessed via parent and observer-report. Pearson product-moment correlations were performed to determine relationships between maternal pitch contours and autism risk. Increased risk for autism was associated with fewer utterances (r = -0.576, N = 18, p = 0.01) and fewer rising (r = -0.586, N = 18, p = 0.01), sinusoidal (r = -0.636, N = 18, p = 0.005), and flat contours (r = -0.679, N = 18, p = 0.01), and more complex (r = 0.584, N = 18, p = 0.01), rapid (r = 0.526, N = 18, p = 0.03), and u-shaped contours (r = 0.619, N = 18, p = 0.02). These preliminary data suggest that mothers of HR infants use different patterns of pitch contours than LR mothers. Further assessment of IDS used with HR infants is warranted to identify at what stage IDS patterns deviate between groups.

Automated Mandarin tone classification using deep neural networks trained on a large speech dataset

A tonal language is one in which the speaker’s intonation modifies the meaning of a word. In this work, we perform a rigorous analysis of intonation changes, or pitch contours, produced by native Mandarin speakers to predict the tone-contour type. Pitch contours are estimated using a number of different methods, also measuring each contour’s Mel-Frequency Cepstral Coefficients (MFCCs). The dataset used was autonomously generated from the Aishell open-source Mandarin speech corpus. Each sample was aligned with its transcript using Montreal Forced Alignment and segmented into individual words. The resulting corpus covers 11 topic domains, spoken by 400 individuals. Separate development, training, and testing datasets are created to ensure the integrity of our results. Pitch contours and their MFCCs are exposed to a number of machine learning techniques including clustered, regression, and traditional Deep Neural Network (DNN) approaches. MFCCs are additionally processed using convolutional neural networks. The models are used to predict the corresponding tone for a contour. Our work seeks to determine which intonation representations perform optimally for machine learning tasks. The tool is used to provide audio and visual feedback to learners of tonal languages. [Work supported by RPI Seed Grant and CISL].

The real-time dynamics of child-directed speech: Using pupillometry to evaluate children’s processing of natural pitch contours

Young children prefer child-directed speech (CDS) to adult-directed speech (ADS) (Cooper and Aslin 1990), and its structural and prosodic features are known to facilitate learning (Thiessen et al., 2005; Graf Estes and Hurley, 2013). However, little is known about how the real-time dynamics of CDS prosody affect young children’s engagement and learning. In Experiment 1, we evaluated moment-to-moment processing of pitch variation using measures of pupil size synchrony across children (Kang and Wheatley, 2017). 24-to-30-month-old children listened to a story in CDS and ADS, and we found that pupil synchrony was higher for CDS than ADS. Next, using hierarchical clustering, we uncovered 4 main word-level pitch contours from a natural CDS corpus and identified contours (specifically, U- and inverted-U-shaped contours), which elicited lower vs. higher synchrony, respectively. In Experiment 2, we found that children learned novel words better when they were presented in higher- vs. lower-synchrony contours. Importantly, synchrony for a novel word during training significantly predicted learning for the same word at test. By revealing a physiological response that is sensitive to the real-time dynamics of prosody, this investigation yields a new subsecond framework for understanding children’s engagement with a signal known to support early language learning.

A deep neural network approach to investigate tone space in languages

Phonological contrasts are usually signaled by multiple cues, and tonal languages typically involve multiple dimensions to distinguish between tones (e.g., duration, pitch contour, and voice quality, etc.). While the topic has been extensively studied, research has mostly used small datasets. This study employs a deep neural network (DNN) based speech recognizer trained on the AISHELL-1 (Bu et al., 2017) speech corpus (178 hours of read speech) to explore the tone space in Mandarin Chinese. A recent study shows that DNN models learn linguistically-interpretable information to distinguish between vowels (Weber et al., 2016). Specifically, from a low-dimensional Bottleneck layer, the model learns features comparable to F1 and F2. In the current study, we propose a more complicated Long Short-Term Memory (LSTM) model—with a Bottleneck layer implemented in the hidden layers—to account for variable duration, an important cue for tone discrimination. By interpreting the features learned in the Bottleneck layer, we explore what acoustic dimensions are involved in distinguishing tones. The large amount of data from the speech corpus also renders the results more convincing and provides additional insights not possible from studies with more limited data sets.

Effects of pitch contour and speaking rate on perception of foreign-accented speech

Perception of foreign accent is typically studied using an accentedness rating task. For example, native English listeners rate the degree of accentedness in sentences produced by non-native English speakers. However, in the past studies, it has been unclear that on what criteria participants used to judge accentedness. Here, native English speakers rated the accentedness of Korean-accented English sentences on a scale from 1 (strong accent) to 9 (little to no accent). Participants rated sentences that were unmodified or had one acoustic property removed. In one block, pitch contours of sentences were flattened and set to their mean values. In another block, speaking rates were set to the grand mean of all speaking rates (3.8 syllables/second). This way, changes in accentedness ratings across unmodified and modified sentences were attributable to the acoustic property that was removed. Accentedness ratings were not systematically influenced by manipulations of pitch contours but were influenced by speaking rate manipulations. Increasing the speaking rate (to 3.8 syllables/second) made sentences sound less accented than their unmodified versions; decreasing the speaking rate made sentences sound more accented than their unmodified versions. Results suggest that the speaking rate directly contributes to ratings of foreign accentedness.

The Perception of Tone and Focus in Mandarin by Indonesian Learners: A Case Study

In a tone language, the interface between tone, intonation, and focus will affect the pitch height and contour of tones. Previous perceptual studies revealed the potential conflicts in perceiving pitch variations at lexical and post-lexical levels that were experienced by either native listeners or listeners who speak Mandarin language as a second or foreign language. Rarely we find research in Indonesia that provides evidence for Mandarin language learners’ perceptual ability at a post-lexical level. This paper investigated how well learners with distinct first language (L1) background identify tones that are affected by the realization of focus and the presence and location of focus in distinct intonation types. Perceptual experiments were conducted towards two groups of listeners: Mandarin learners with Indonesian L1 and learners with a tone language L1 background (Hakka or Hokkien). Their identification accuracy (IA) rate in recognizing the tone type for the last syllable with a narrow focus was compared with their IA in identifying the location of the focus. In general, identifying tone type was easier than identifying focus position for both groups. However, the Mean from each group showed that learners with a tone language L1 were slightly better than the other group. Results exhibited more similarities between the two groups of the listener, which indicates that L1 background only has a mild effect on the perceptual ability of Indonesian learners of Mandarin as a foreign language.

Development of tonal discrimination in young heritage speakers of Cantonese

This study uses the Perceptual Assimilation Model for Suprasegmentals (PAM-S) (So & Best, 2008, 2010), supported by the assumptions of the L2 Intonation Learning theory (LILt, Mennen, 2015), to investigate how young heritage speakers of Cantonese living in the United States acquired Cantonese tones. Sixty-seven heritage speakers, aged 5–11, were tested on their perception of Cantonese tonal contrasts using an ABX discrimination task. They were compared to 64 peers aged 5–12 in Hong Kong, where Cantonese is spoken as the majority language but English is also acquired from a young age. Two pairs of tones were tested: Tones 2 (mid rising) and 5 (low rising), which have similar pitch heights and contours, and Tones 1 (high level) and 4 (low falling), which have a larger phonetic contrast. As predicted, the heritage speakers were more accurate in discriminating between the more distinct pair of tones than between the more similar pair. They also scored lower than their peers from Hong Kong in both contrast conditions. Age of testing predicted accuracy for both groups, and Chinese literacy also had a significant effect for the heritage speakers. The potential lack of the Tone 2–5 contrast in the heritage speakers’ input is discussed as an explanation for these findings. This study illustrates the divergence in heritage speakers’ phonological development compared to majority language speakers, and shows the relevance of the PAM-S and LILt to the heritage language context.

Mapping non-native pitch contours to meaning: Perceptual and experiential factors.

Infants show interesting patterns of flexibility and constraint early in word learning. Here, we explore perceptual and experiential factors that drive associative learning of labels that differ in pitch contour. Contrary to the salience hypothesis proposed in Experiment 1, English-learning 14-month-olds failed to map acoustically distinctive level and dipping labels to novel referents, even though they discriminated the labels when no potential referents were present. Conversely, infants readily mapped the less distinctive rising and dipping labels. In Experiment 2, we found that the degree of pitch variation in labels also does not account for learning. Instead, English-learning infants only learned if one of the labels had a rising pitch contour. We argue that experience with hearing and/or producing native language prosody may lead infants to initially over-interpret the role rising pitch plays in differentiating words. Together, our findings suggest that multiple factors contribute to whether specific acoustic forms will function as candidate object labels.

A tonal-language benefit for pitch in normally-hearing and cochlear-implanted children

In tonal languages, voice pitch inflections change the meaning of words, such that the brain processes pitch not merely as an acoustic characterization of sound but as semantic information. In normally-hearing (NH) adults, this linguistic pressure on pitch appears to sharpen its neural encoding and can lead to perceptual benefits, depending on the task relevance, potentially generalizing outside of the speech domain. In children, however, linguistic systems are still malleable, meaning that their encoding of voice pitch information might not receive as much neural specialization but might generalize more easily to ecologically irrelevant pitch contours. This would seem particularly true for early-deafened children wearing a cochlear implant (CI), who must exhibit great adaptability to unfamiliar sounds as their sense of pitch is severely degraded. Here, we provide the first demonstration of a tonal language benefit in dynamic pitch sensitivity among NH children (using both a sweep discrimination and labelling task) which extends partially to children with CI (i.e., in the labelling task only). Strong age effects suggest that sensitivity to pitch contours reaches adult-like levels early in tonal language speakers (possibly before 6 years of age) but continues to develop in non-tonal language speakers well into the teenage years. Overall, we conclude that language-dependent neuroplasticity can enhance behavioral sensitivity to dynamic pitch, even in extreme cases of auditory degradation, but it is most easily observable early in life.

The phonological status of Low tones in Shanghai tone sandhi

Abstract In Shanghai tone sandhi, with the exception of T5 (yangru) sandhi, a pitch-fall occurs at the second or third syllable of a phonological word (or a sandhi domain). Previous analyses argue that this is invoked by the insertion of a default Low tone to satisfy the Well-formedness Condition of the autosegmental theory. However, in the framework of the present autosegmental theory, that condition is no longer necessarily satisfied, and an alternative interpretation, adopting a boundary Low tone, has been suggested. To evaluate the appropriateness of the default and boundary interpretations, we compared pitch contours among di- to tetrasyllabic words in greater detail. The results show that, in T1 to T4 sandhi, disyllabic words tend to have lower pitch contours than tri- and tetrasyllabic words at the first and second syllables, and that, in tetrasyllables, minimum pitch values were constantly attested at the third syllable. These results indicate that in Shanghai tone sandhi, a boundary Low tone is assigned at the right edge of a phonological word, and it is further associated with the third syllable in tetrasyllables. This boundary interpretation further gives an appropriate explanation of the difference of the pitch-fall between Middle and New Shanghai.

Інтонація чоловічих негативних оцінних висловлень у сучасному англійському діалогічному мовленні

The article deals with a gender-oriented experimental phonetic study of explicit male negative evaluative utterances in modern English everyday dialogical speech. The realization of the category of evaluation in speech is in the limelight of modern sociolinguistic research aimed at establishing the linguistic behaviour peculiarities and verbalization of roles, norms and values attributed to men and women by society. Evaluation is objectified in language units of lexical, phonetic, syntactic and discourse levels. Negative evaluative utterances verbalise the negative evaluation of the object that doesn’t meet the subject’s expectations according to certain criteria. On the basis of auditory and acoustic analyses prosodic markers of male negative rational and emotional evaluative utterances in communication between men and in communication with women of the same social status have been outlined: pitch pattern, loudness, tempo and pausation. Invariant prosodic features of male negative evaluative utterances in communication between representatives of the same and different sex but of the same social status have been established: mid level pre-head; descending stepping and mid level scales; mid, narrowed and wide ranges of intonation group; high and low falling terminal tones; falling and rising-falling pitch contour; moderate tempo; moderate loudness; sad and dramatic timbre; voice pitch frequency maximum on head and nucleus of the first or second intonation group; peak intensity on nucleus and the first rhythm group; minimal average syllable duration; small and average unfilled tentative reflective pauses. Variant prosodic features of the following male negative evaluative utterances have been distinguished: 1. Rational evaluative utterances in communication between men: level pitch contour; accelerated tempo; reduced loudness; peak intensity on another rhythm group than pre-head, head and nucleus; small unfilled and average filled tentative choice pauses; 2. Rational evaluative utterances in communication between men and women: ascending stepping scale; the Low Low-Rise and the Rise; falling-rising pitch contour; increased loudness; small unfilled tentative psychological pauses; short filled tentative reflective pauses; intemporal psychological pauses; 3. Emotional evaluative utterances in communication between men: increased and reduced loudness; peak intensity on another rhythm group than pre-head, head and nucleus; small unfilled tentative psychological pauses; intemporal psychological, choice and reflective pauses; 4. Emotional evaluative utterances in communication between men and women: ascending stepping scale; widened and narrow ranges of intonation group; the Low Low-Rise and the High Rise-Fall; short average syllable duration; small filled tentative choice pauses; average filled tentative reflective pauses.