F0 Information Research Articles

Mandarin tone recognition based on unsupervised feature learning from spectrograms

In tone languages, such as Mandarin Chinese, a syllable with different tones conveys different meanings. Previously, Mandarin tone recognition based on Mel-frequency cepstral coefficients (MFCCs) and Convolutional Neural Networks (CNN) was examined and the results outperformed the model of conventional neural network using manually edited F0 data. In the present study, Mandarin tone recognition based on spectrograms, instead of MFCCs, was explored. Unsupervised feature learning was applied to the unlabeled spectrograms directly with a denoising autoencoder (dAE). Then, the model convolved the labeled spectrograms with the learnt “sound features” and produced a set of feature maps. A dataset that consisted of 4500 monosyllabic words collected from 125 children was used to evaluate the recognition performance. Compared with methods based on MFCCs, there are more parameters to train in the new approach based on spectrograms. As a result, the new model might better capture the statistical distribution in the original data. Therefore, the new approach, with unsupervised feature learning, could perform even better than previous methods based on MFCCs or those based on the extracted F0 information. The advantages and shortcomings of various approaches for lexical tone recognition will be discussed. [Work supported in part by the NIH NIDCD Grant No. R15-DC014587.]

Robustness against distortion of fundamental frequency cues in simulated electro-acoustic hearing.

Speech recognition by cochlear implant users can be improved by adding an audible low frequency acoustic signal to electrical hearing; the resulting improvement is deemed "electro-acoustic stimulation (EAS) benefit." However, a crucial low frequency cue, fundamental frequency (F0), can be distorted via the impaired auditory system. In order to understand how F0 distortions may affect EAS benefit, normal-hearing listeners were presented monaurally with vocoded speech (frequencies >250 Hz) and an acoustical signal (frequencies <250 Hz) with differing manipulations of the F0 signal, specifically: a pure tone with the correct mean F0 but with smaller variations around this mean, or a narrowband of white noise centered around F0, at varying bandwidths; a pure tone down-shifted in frequency by 50 Hz but keeping overall frequency modulations. Speech-recognition thresholds improved when tones with reduced frequency modulation were presented, and improved significantly for noise bands maintaining F0 information. A down-shifted tone, or only a tone to indicate voicing, showed no EAS benefit. These results confirm that the presence of the target's F0 is beneficial for EAS hearing in a noisy environment, and they indicate that the benefit is robust to F0 distortion, as long as the mean F0 and frequency modulations of F0 are preserved.

Effects of Semantic Context and Fundamental Frequency Contours on Mandarin Speech Recognition by Second Language Learners.

Speech recognition by second language (L2) learners in optimal and suboptimal conditions has been examined extensively with English as the target language in most previous studies. This study extended existing experimental protocols (Wang et al., 2013) to investigate Mandarin speech recognition by Japanese learners of Mandarin at two different levels (elementary vs. intermediate) of proficiency. The overall results showed that in addition to L2 proficiency, semantic context, F0 contours, and listening condition all affected the recognition performance on the Mandarin sentences. However, the effects of semantic context and F0 contours on L2 speech recognition diverged to some extent. Specifically, there was significant modulation effect of listening condition on semantic context, indicating that L2 learners made use of semantic context less efficiently in the interfering background than in quiet. In contrast, no significant modulation effect of listening condition on F0 contours was found. Furthermore, there was significant interaction between semantic context and F0 contours, indicating that semantic context becomes more important for L2 speech recognition when F0 information is degraded. None of these effects were found to be modulated by L2 proficiency. The discrepancy in the effects of semantic context and F0 contours on L2 speech recognition in the interfering background might be related to differences in processing capacities required by the two types of information in adverse listening conditions.

Perception and production abilities of question vs. statement intonation patterns in young deaf children with early cochlear implantation

Little research exists studying production and perception of prosody in early-implanted children with cochlear implants (CIs). Because of limited F0 information conveyed by CIs, there is particular interest in the impact of CIs on intonation perception and production. The aim of the present study was to describe and compare the ability of hearing (HC) and implanted (IC) children to perceive and produce distinctions between question vs. statement intonation patterns. Nine HC and 9 IC between 38 and 58 months of age participated in the study. All IC received their first implant prior to 15 months and had no other identified medical or developmental problems besides hearing loss. To determine the children’s ability to perceive rising and falling intonation, an imitation task was used. Results showed no significant difference in perception between the groups (p = .09). Intonation production was examined by eliciting yes–no questions and statements during a play-based task. Production measures included mean, range, and slope of F0. While F0 range was significantly greater for IC than for HC (p = .015), no other measure was found to be significantly different between the two groups. Overall, early-implanted deaf children with CIs appear to develop intonation comparable to their hearing peers.

Speaker-individuality in Fujisaki model f0 features: implications for forensic voice comparison

Fundamental frequency (f0) is a highly speaker-specific feature. Consequently, practitioners often use f0 information in forensic casework. Current research principally examines the use of long-term f0 statistics such as f0 means and standard deviations for forensic voice comparison. The present study investigates how short-term f0 features such as measured by the Fujisaki intonation model capture speaker-individuality. Based on data of a homogeneous group of Zurich German speakers, we conducted an experiment on a large corpus of read speech and on a subset of sentences that included speaking style variability (spontaneous vs. read). The latter is characteristic of forensic casework. Speakers demonstrated high between-speaker variability and low within-speaker variability across the two speaking styles for a number of f0 features. Given this evidence of speaker-individuality, we discuss Fujisaki f0 features’ potential for forensic voice comparison.

Musical Source Clustering and Identification in Polyphonic Audio

For music transcription or musical source separation, apart from knowing the multi-F0 contours, it is also important to know which F0 has been played by which instrument. This paper focuses on this aspect, i.e. given the polyphonic audio along with its multiple F0 contours, the proposed system clusters them so as to decide ‘which instrument played when.’ For the task of identifying the instrument or singers in the polyphonic audio, there are many supervised methods available. But many times individual source audio is not available for training. To address this problem, this paper proposes novel schemes using semi-supervised as well as unsupervised approach to source clustering. The proposed theoretical framework is based on auditory perception theory and is implemented using various tools like probabilistic latent component analysis and graph clustering, while taking into account various perceptual cues for characterizing a source. Experiments have been carried out over a wide variety of datasets - ranging from vocal to instrumental as well as from synthetic to real world music. The proposed scheme significantly outperforms a state of the art unsupervised scheme, which does not make use of the given F0 contours. The proposed semi-supervised approach also performs better than another semi-supervised scheme, which makes use of the given F0 information, in terms of computations as well as accuracy.

Effects of various electrode configurations on music perception, intonation and speaker gender identification

Advances in speech coding strategies and electrode array designs for cochlear implants (CIs) predominantly aim at improving speech perception. Current efforts are also directed at transmitting appropriate cues of the fundamental frequency (F0) to the auditory nerve with respect to speech quality, prosody, and music perception. The aim of this study was to examine the effects of various electrode configurations and coding strategies on speech intonation identification, speaker gender identification, and music quality rating. In six MED-EL CI users electrodes were selectively deactivated in order to simulate different insertion depths and inter-electrode distances when using the high definition continuous interleaved sampling (HDCIS) and fine structure processing (FSP) speech coding strategies. Identification of intonation and speaker gender was determined and music quality rating was assessed. For intonation identification HDCIS was robust against the different electrode configurations, whereas fine structure processing showed significantly worse results when a short electrode depth was simulated. In contrast, speaker gender recognition was not affected by electrode configuration or speech coding strategy. Music quality rating was sensitive to electrode configuration. In conclusion, the three experiments revealed different outcomes, even though they all addressed the reception of F0 cues. Rapid changes in F0, as seen with intonation, were the most sensitive to electrode configurations and coding strategies. In contrast, electrode configurations and coding strategies did not show large effects when F0 information was available over a longer time period, as seen with speaker gender. Music quality relies on additional spectral cues other than F0, and was poorest when a shallow insertion was simulated.

Pitch and loudness matching of unmodulated and modulated stimuli in cochlear implantees

The pitch elicited by unmodulated and amplitude modulated electrical pulse trains was examined with six adult cochlear implantees. In addition, for three of those subjects who had some hearing in their contralateral ear, the pitch of unmodulated electrical pulse trains was compared to that of complex harmonic acoustic tones. In the first experiment, pulse rate discrimination and the effects of place and level differences on pitch were examined for unmodulated pulse trains. General results were consistent with previous studies showing that variations in pulse rate, while holding loudness fixed, elicit changes in pitch at low rates, but become progressively harder to discriminate as rates approach approximately 300 pulses-per-second. Variations in place or level of stimulation generally produced changes in pitch consistent with tonotopic place and spread of excitation. In the second experiment, pitch and loudness of unmodulated pulse trains were compared with those of amplitude modulated stimuli as a function of modulation depth, rate, and shape, and presentation level. The pitch elicited by an amplitude modulated pulse train was generally higher than that of an unmodulated pulse train with a pulse rate equal to the modulation rate, and generally decreased toward that of the unmodulated pulse train as modulation depth or rate increased, or as presentation level decreased. Sharper/narrower modulation produced lower pitch. In the final experiment, the pitch heights of acoustic complex harmonic tones and unmodulated pulse trains were compared. When electrical pulse rate was equal to the fundamental frequency of the acoustic tone, similar pitch heights were elicited. The results from these experiments indicate that F0 rate pitch derived from the temporal envelope in existing clinical cochlear implant strategies may often be higher than that of acoustic harmonic tones at the same F0 in normal hearing, and that pitch growth with increasing F0 may be shallower. The relationship between F0 and rate pitch is expected to be more similar to acoustic stimulation for low F0 rates when using new pitch coding strategies that code F0 information via deep (narrow) amplitude modulation of the stimulus envelope. Although that similarity reduces as F0 approaches the upper limit of rate-pitch discrimination, that limit is reached sooner for the shallow (or broad) modulators used in existing clinical strategies.

The relative contribution of rhythm, intonation, and lexical information to the perception of prosodic disorder

Acoustic studies suggest that children with autism spectrum disorder (ASD) produce atypical rhythm and intonation [Diehl and Paul (2011)]. Behavioral studies indicate that children with ASD combine prosody with lexical content in atypical ways [Peppé et al. (2007)]. The current study assessed the relative contribution of rhythm, intonation, and language context to perception of prosodic disorder. Short excerpts were taken from narratives produced by 18 children with ASD and 18 typically developing controls. Prior study indicated that listeners easily distinguished groups on the basis of these excerpts. Here, the excerpts were resynthesized to control for voice quality and to allow for selective inclusion of F0, duration, intensity, and lexical information. Experiment 1 investigated listeners’ ability to distinguish the groups based on delexicalized samples that preserved only rhythm (duration + intensity), only intonation, or a combination of both. Experiment 2 investigated the contribution of lexical information to the judgments, and the interaction of lexical information with intonation. Results indicated that (1) listeners were less able to distinguish between groups in the Intonation Only condition, and (2) intonation had a negligible effect on performance when lexical content was present. We conclude that rhythm cues and lexical information contribute more to perceived disorder than intonation.

Modulation Frequency Discrimination with Modulated and Unmodulated Interference in Normal Hearing and in Cochlear-Implant Users

Differences in fundamental frequency (F0) provide an important cue for segregating simultaneous sounds. Cochlear implants (CIs) transmit F0 information primarily through the periodicity of the temporal envelope of the electrical pulse trains. Successful segregation of sounds with different F0s requires the ability to process multiple F0s simultaneously, but it is unknown whether CI users have this ability. This study measured modulation frequency discrimination thresholds for half-wave rectified sinusoidal envelopes modulated at 115Hz in CI users and normal-hearing (NH) listeners. The target modulation was presented in isolation or in the presence of an interferer. Discrimination thresholds were strongly affected by the presence of an interferer, even when it was unmodulated and spectrally remote. Interferer modulation increased interference and often led to very high discrimination thresholds, especially when the interfering modulation frequency was lower than that of the target. Introducing a temporal offset between the interferer and the target led to at best modest improvements in performance in CI users and NH listeners. The results suggest no fundamental difference between acoustic and electric hearing in processing single or multiple envelope-based F0s, but confirm that differences in F0 are unlikely to provide a robust cue for perceptual segregation in CI users.

Acoustic cues for the recognition of self-voice and other-voice.

Self-recognition, being indispensable for successful social communication, has become a major focus in current social neuroscience. The physical aspects of the self are most typically manifested in the face and voice. Compared with the wealth of studies on self-face recognition, self-voice recognition (SVR) has not gained much attention. Converging evidence has suggested that the fundamental frequency (F0) and formant structures serve as the key acoustic cues for other-voice recognition (OVR). However, little is known about which, and how, acoustic cues are utilized for SVR as opposed to OVR. To address this question, we independently manipulated the F0 and formant information of recorded voices and investigated their contributions to SVR and OVR. Japanese participants were presented with recorded vocal stimuli and were asked to identify the speaker—either themselves or one of their peers. Six groups of 5 peers of the same sex participated in the study. Under conditions where the formant information was fully preserved and where only the frequencies lower than the third formant (F3) were retained, accuracies of SVR deteriorated significantly with the modulation of the F0, and the results were comparable for OVR. By contrast, under a condition where only the frequencies higher than F3 were retained, the accuracy of SVR was significantly higher than that of OVR throughout the range of F0 modulations, and the F0 scarcely affected the accuracies of SVR and OVR. Our results indicate that while both F0 and formant information are involved in SVR, as well as in OVR, the advantage of SVR is manifested only when major formant information for speech intelligibility is absent. These findings imply the robustness of self-voice representation, possibly by virtue of auditory familiarity and other factors such as its association with motor/articulatory representation.

Acoustic correlates of stress and pitch accent in Tashlhiyt Berber

This paper reports results of a phonetic study of stress and pitch accent in Tashlhiyt Berber, an Afro-Asiatic language that is famous for possessing words consisting of only obstruents. This study examines evidence for word-level stress and phrase-level accent in Tashlhiyt and addresses the question of how both prosodic properties are acoustically realized in segments, such as obstruents, that provide a relatively impoverished backdrop for the manifestation of prominence. Results indicate that F0 is raised in the phrase-final nucleus relative to both the penultimate nucleus of the phrase and also to word-final nuclei that are phrase-medial. This raising of F0 is consistently observed on sonorant nuclei, inconsistently associated with voiced obstruents, and absent from voiceless obstruents, which lack an F0 but often trigger insertion of an epenthetic vowel to aid in the realization of F0 information. A further result is that intensity is higher in the nucleus of a word-final syllable relative to one in the penult of a word. We interpret the greater intensity associated with word-final syllables as a marker of word-level stress and the raising of F0 in phrase-final syllables as a phrase-level pitch accent docking on the final stressed syllable of a phrase.

Beneficial acoustic speech cues for cochlear implant users with residual acoustic hearing

This study investigated which acoustic cues within the speech signal are responsible for bimodal speech perception benefit. Seven cochlear implant (CI) users with usable residual hearing at low frequencies in the non-implanted ear participated. Sentence tests were performed in near-quiet (some noise on the CI side to reduce scores from ceiling) and in a modulated noise background, with the implant alone and with the addition, in the hearing ear, of one of four types of acoustic signals derived from the same sentences: (1) a complex tone modulated by the fundamental frequency (F0) and amplitude envelope contours; (2) a pure tone modulated by the F0 and amplitude contours; (3) a noise-vocoded signal; (4) unprocessed speech. The modulated tones provided F0 information without spectral shape information, whilst the vocoded signal presented spectral shape information without F0 information. For the group as a whole, only the unprocessed speech condition provided significant benefit over implant-alone scores, in both near-quiet and noise. This suggests that, on average, F0 or spectral cues in isolation provided limited benefit for these subjects in the tested listening conditions, and that the significant benefit observed in the full-signal condition was derived from implantees' use of a combination of these cues.

Three- to five-year-old children in Taiwan show little development in their production of monosyllabic Mandarin lexical tones

While a couple of studies reported that 2-year-old children in Beijing have mastered the production of Mandarin tones in various contexts, several studies found that three-year-old children learning Mandarin as a first language in the U.S. have not produced adult-like tones in monosyllabic words. This study collected monosyllabic Mandarin tone productions from 33 three- to five-year-old children growing up in Taiwan. Five judges categorized the tones of the 734 child productions and 92 productions by 4 adults via low-pass filtered words in which the segmental information was degraded while F0 information was retained. Adult tones were categorized with 93%, 96%, 82%, and 94% accuracy. Children's tones were identified with significantly lower accuracy (p<.05) at 63%, 50%, 50% and 77%, respectively. Age accounted for 0.2%, 2.4%, 4.0% and 8.2% of the variance in children's accuracy of the four tones, respectively, suggesting little developmental change. Children produced T4 more accurately than T1. T2 and T3 were significantly more difficult. These results are in line with findings in previous studies with children growing up in the U.S. using the same methodology and seem to support that tone development is related to maturation of speech motor control. [Work supported by NSF EAPSI]

Perception of Mandarin Chinese with cochlear implants using enhanced temporal pitch cues

A cochlear implant (CI) signal processing strategy named F0 modulation (F0mod) was compared with the advanced combination encoder (ACE) strategy in a group of four post-lingually deafened Mandarin Chinese speaking CI listeners. F0 provides an enhanced temporal pitch cue by amplitude modulating the multichannel electrical stimulation pattern at the fundamental frequency (F0) of the incoming speech signal. Word and sentence recognition tests were carried out in quiet and in noise. The responses for the word-recognition test were further segmented into phoneme and tone scores. Off-line implementations of ACE and F0mod were used, and electrical stimulation patterns were directly streamed to the CI subject's implant. To focus on the feasibility of enhanced temporal cues for tonal language perception, idealized F0 information that was extracted from speech tokens in quiet was used in the F0mod processing of speech-in-noise mixtures. The results indicated significantly better lexical tone perception with the F0mod strategy than with ACE for the male voice (p<0.05). No significant differences in sentence recognition were found between F0mod and ACE.

Sine-wave speech recognition in a tonal language

It is hypothesized that in sine-wave replicas of natural speech, lexical tone recognition would be severely impaired due to the loss of F0 information, but the linguistic information at the sentence level could be retrieved even with limited tone information. Forty-one native Mandarin-Chinese-speaking listeners participated in the experiments. Results showed that sine-wave tone-recognition performance was on average only 32.7% correct. However, sine-wave sentence-recognition performance was very accurate, approximately 92% correct on average. Therefore the functional load of lexical tones on sentence recognition is limited, and the high-level recognition of sine-wave sentences is likely attributed to the perceptual organization that is influenced by top-down processes.

Pitch Contour Matching and Interactional Alignment across Turns: An Acoustic Investigation

In order to explore the influence of context on the phonetic design of talk-in-interaction, we investigated the pitch characteristics of short turns (insertions) that are produced by one speaker between turns from another speaker. We investigated the hypothesis that the speaker of the insertion designs her turn as a pitch match to the prior turn in order to align with the previous speaker's agenda, whereas non-matching displays that the speaker of the insertion is non-aligning, for example to initiate a new action. Data were taken from the AMI meeting corpus, focusing on the spontaneous talk of first-language English participants. Using sequential analysis, 177 insertions were classified as either aligning or non-aligning in accordance with definitions of these terms in the Conversation Analysis literature. The degree of similarity between the pitch contour of the insertion and that of the prior speaker's turn was measured, using a new technique that integrates normalized F0 and intensity information. The results showed that aligning insertions were significantly more similar to the immediately preceding turn, in terms of pitch contour, than were non-aligning insertions. This supports the view that choice of pitch contour is managed locally, rather than by reference to an intonational lexicon.

Predicting the intelligibility of vocoded and wideband Mandarin Chinese

Due to the limited number of cochlear implantees speaking Mandarin Chinese, it is extremely difficult to evaluate new speech coding algorithms designed for tonal languages. Access to an intelligibility index that could reliably predict the intelligibility of vocoded (and non-vocoded) Mandarin Chinese is a viable solution to address this challenge. The speech-transmission index (STI) and coherence-based intelligibility measures, among others, have been examined extensively for predicting the intelligibility of English speech but have not been evaluated for vocoded or wideband (non-vocoded) Mandarin speech despite the perceptual differences between the two languages. The results indicated that the coherence-based measures seem to be influenced by the characteristics of the spoken language. The highest correlation (r = 0.91-0.97) was obtained in Mandarin Chinese with a weighted coherence measure that included primarily information from high-intensity voiced segments (e.g., vowels) containing F0 information, known to be important for lexical tone recognition. In contrast, in English, highest correlation was obtained with a coherence measure that included information from weak consonants and vowel/consonant transitions. A band-importance function was proposed that captured information about the amplitude envelope contour. A higher modulation rate (100 Hz) was found necessary for the STI-based measures for maximum correlation (r = 0.94-0.96) with vocoded Mandarin and English recognition.

Training listeners to report fundamental frequency and formant range information independently.

The vowels of speakers of different sizes vary in terms of their average f0 and formant frequencies. In general, larger speakers produce vowels with lower formant frequencies and lower f0s. Listeners have demonstrated the ability to estimate the approximate size of a speaker and previous experiments have shown that these judgments are based on the joint consideration of f0 and formant range information. Thus both lower f0s and lower formant frequency ranges are associated with larger speakers by listeners. Studies which have asked listeners to evaluate voices have focused on the extraction of apparent speaker characteristics (which are informed by f0 and formant frequencies) rather than asking speakers to report f0 and formant range information directly. The current study consists of a training procedure by which participants will learn to report the f0 and formant range of voices independently. The training consists of a voice matching game in which participants hear a pair of vowels produced by a voice and are asked to indicate which of the candidate voices they just heard. Results will be analyzed in light of current theories of vowel perception and normalization.

Combining information across frequency regions in fundamental frequency discrimination.

Sensitivity to fundamental frequency (F0) differences was measured for two complex tones, A and B, which had the same F0 but were filtered into two different frequency regions. Tones were presented either alone or together. A signal-detection analysis was used to predict effects of combining F0 information across frequency regions. For 400-ms tones containing only unresolved harmonics, the first experiment showed that performance (in terms of d(')) for the combined presentation was better than for the isolated tones but was not optimal (assuming independent channels and noises) and was independent of the relative timing of pulses in the envelopes of tones A and B (varied by changing the starting phase of components of tone B relative to those of tone A). The nonoptimal performance was shown not to be due to peripheral masking (experiment II), or to listeners paying attention mainly to one frequency region (experiment III), nor was it specific to conditions where all harmonics were unresolved (experiment IV). In contrast, optimal performance in F0 discrimination for combined presentation was observed for 50-ms tones (experiment V). The results may reflect the limited ability of the human auditory system to integrate information simultaneously in the time and the frequency domains.