Formant Frequency Information Research Articles

Dialect Classification From a Single Sonorant Sound Using Deep Neural Networks

During spoken communication, the fine acoustic properties of human speech can reveal vital sociolinguistic and linguistic information about speakers and thus, these properties can function as reliable identification markers of speakers’ identity. One key information speech reveals is speakers' dialect. The first aim of this study is to provide a machine learning method that can distinguish the dialect from acoustic productions of sonorant sounds. The second aim is to determine the classification accuracy of dialects from the temporal and spectral information of a single sonorant sound and the classification accuracy of dialects using additional co-articulatory information from the adjacent vowel. To this end, this paper provides two classification approaches. The first classification approach aims to distinguish two Greek dialects, namely Athenian Greek, the prototypical form of Standard Modern Greek and Cypriot Greek using measures of temporal and spectral information (i.e., spectral moments) from four sonorant consonants /m n l r/. The second classification study aims to distinguish the dialects using coarticulatory information (e.g., formants frequencies F1-F5, F0, etc.) from the adjacent vowel in addition to spectral and temporal information from sonorants. In both classification approaches, we have employed Deep Neural Networks, which we compared with Support Vector Machines, Random Forests, and Decision Trees. The findings show that neural networks distinguish the two dialects using a combination of spectral moments, temporal information, and formant frequency information with 81\% classification accuracy, which is a 14% accuracy gain over employing temporal properties and spectral moments alone. In conclusion, Deep Neural Networks can classify the dialect from single consonant productions making them capable of identifying sociophonetic shibboleths.

Smartphone based real-time super Gaussian single microphone Speech Enhancement to improve intelligibility for hearing aid users using formant information.

In this paper, we present a Speech Enhancement (SE) technique to improve intelligibility of speech perceived by Hearing Aid users using smartphone as an assistive device. We use the formant frequency information to improve the overall quality and intelligibility of the speech. The proposed SE method is based on new super Gaussian joint maximum a Posteriori (SGJMAP) estimator. Using the priori information of formant frequency locations, the derived gain function has " tradeoff" factors that allows the smartphone user to customize perceptual preference, by controlling the amount of noise suppression and speech distortion in real-time. The formant frequency information helps the hearing aid user to control the gains over the non-formant frequency band, allowing the HA users to attain more noise suppression while maintaining the speech intelligibility using a smartphone application. Objective intelligibility measures and subjective results reflect the usability of the developed SE application in noisy real world acoustic environment.

Formant Frequency-based Speech Enhancement Technique to improve Intelligibility for hearing aid users with smartphone as an assistive device.

In this paper, we present a Speech Enhancement (SE) method implemented on a smartphone, and this arrangement functions as an assistive device to hearing aids (HA). Many benchmark single channel SE algorithms implemented on HAs provide considerable improvement in speech quality, while speech intelligibility improvement still remains a prime challenge. The proposed SE method based on Log spectral amplitude estimator improves speech intelligibility in the noisy real world acoustic environment using the priori information of formant frequency locations. The formant frequency information avails us to control the amount of speech distortion in these frequency bands, thereby controlling speech distortion. We introduce a 'scaling' parameter for the SE gain function, which controls the gains over the non-formant frequency band, allowing the HA users to customize the playback speech using a smartphone application to their listening preference. Objective intelligibility measures show the effectiveness of the proposed SE method. Subjective results reflect the suitability of the developed Speech Enhancement application in real-world noisy conditions at SNR levels of -5 dB, 0 dB and 5 dB.

Contribution of formant frequency information to vowel perception in steady-state noise by cochlear implant users

Cochlear implant (CI) recipients have difficulty understanding speech in noise even at moderate signal-to-noise ratios. Knowing the mechanisms they use to understand speech in noise may facilitate the search for better speech processing algorithms. In the present study, a computational model is used to assess whether CI users' vowel identification in noise can be explained by formant frequency cues (F1 and F2). Vowel identification was tested with 12 unilateral CI users in quiet and in noise. Formant cues were measured from vowels in each condition, specific to each subject's speech processor. Noise distorted the location of vowels in the F2 vs F1 plane in comparison to quiet. The best fit model to subjects' data in quiet produced model predictions in noise that were within 8% of actual scores on average. Predictions in noise were much better when assuming that subjects used a priori knowledge regarding how formant information is degraded in noise (experiment 1). However, the model's best fit to subjects' confusion matrices in noise was worse than in quiet, suggesting that CI users utilize formant cues to identify vowels in noise, but to a different extent than how they identify vowels in quiet (experiment 2).

Does knowing speaker sex facilitate vowel recognition at short durations?

A man, woman or child saying the same vowel do so with very different voices. The auditory system solves the complex problem of extracting what the man, woman or child has said despite substantial differences in the acoustic properties of their voices. Much of the acoustic variation between the voices of men and woman is due to changes in the underlying anatomical mechanisms for producing speech. If the auditory system knew the sex of the speaker then it could potentially correct for speaker sex related acoustic variation thus facilitating vowel recognition. This study measured the minimum stimulus duration necessary to accurately discriminate whether a brief vowel segment was spoken by a man or woman, and the minimum stimulus duration necessary to accuately recognise what vowel was spoken. Results showed that reliable vowel recognition precedesreliable speaker sex discrimination, thus questioning the use of speaker sex information in compensating for speaker sex related acoustic variation in the voice. Furthermore, the pattern of performance across experiments where the fundamental frequency and formant frequency information of speaker's voices were systematically varied, was markedly different depending on whether the task was speaker-sex discrimination or vowel recognition. This argues for there being little relationship between perception of speaker sex (indexical information) and perception of what has been said (linguistic information) at short durations.

Effect of Speaking Rate on Recognition of Synthetic and Natural Speech by Normal-Hearing and Cochlear Implant Listeners

Most studies have evaluated cochlear implant (CI) performance using "clear" speech materials, which are highly intelligible and well articulated. CI users may encounter much greater variability in speech patterns in the "real world," including synthetic speech. In this study, the authors measured sentence recognition with multiple talkers and speaking rates, and with naturally produced and synthetic speech in listeners with normal hearing (NH) and CIs. NH and CI subjects were asked to recognize naturally produced or synthetic sentences, presented at a slow, normal, or fast speaking rate. Natural speech was produced by one male and one female talker; synthetic speech was generated to simulate a male and female talker. For natural speech, the speaking rate was time-scaled while preserving voice pitch and formant frequency information. For synthetic speech, the speaking rate was adjusted within the speech synthesis engine. NH subjects were tested while listening to unprocessed speech or to an eight-channel acoustic CI simulation. CI subjects were tested while listening with their clinical processors and the recommended microphone sensitivity and volume settings. The NH group performed significantly better than did the CI-simulation group, and the CI-simulation group performed significantly better than did the CI group. For all subject groups, sentence recognition was significantly better with natural speech than with synthetic speech. The performance deficit with synthetic speech was relatively small for NH subjects listening to unprocessed speech. However, the performance deficit with synthetic speech was much greater for CI subjects and for CI-simulation subjects. There was significant effect of talker gender, with slightly better performance with the female talker for CI subjects and slightly better performance with the male talker for the CI simulations. For all subject groups, sentence recognition was significantly poorer only at the fast rate. CI performance was very poor (approximately 10% correct) at the fast rate. CI listeners are susceptible to variability in speech patterns caused by speaking rate and production style (natural versus synthetic). CI performance with clear speech materials may overestimate performance in real-world listening conditions. The poorer CI performance may be because of other factors besides reduced spectro-temporal resolution, such the quality of electric stimulation, duration of deafness, or cortical processing. Optimizing the input or training may improve CI users' tolerance for variability in speech patterns.

Auditory Spectral Integration in the Perception of Static Vowels

To evaluate potential contributions of broadband spectral integration in the perception of static vowels. Specifically, can the auditory system infer formant frequency information from changes in the intensity weighting across harmonics when the formant itself is missing? Does this type of integration produce the same results in the lower (first formant [F1]) and higher (second formant [F2]) regions? Does the spacing between the spectral components affect a listener's ability to integrate the acoustic cues? Twenty young listeners with normal hearing identified synthesized vowel-like stimuli created for adjustments in the F1 region (/Λ/-/α/, /i/-/ε/) and in the F2 region (/Λ/-/æ/). There were 2 types of stimuli: (a) 2-formant tokens and (b) tokens in which 1 formant was removed and 2 pairs of sine waves were inserted below and above the missing formant; the intensities of these harmonics were modified to cause variations in their spectral center of gravity (COG). The COG effects were tested over a wide range of frequencies. Obtained patterns were consistent with calculated changes to the spectral COG, in both the F1 and F2 regions. The spacing of the sine waves did not affect listeners' responses. The auditory system may perform broadband integration as a type of auditory wideband spectral analysis.

Acoustic correlates of devoiced Japanese vowels.

The analysis of acoustic correlates of devoiced vowels produced by young native Japanese speakers is presented. In addition to gradient amounts of voicing seen in vowels undergoing devoicing, gradient amounts of fricativization can be observed, supporting an analysis of devoicing in Japanese as gestural overlap of both glottal gesture and oral stricture. Specifically, the surfacing of voiceless consonant‐devoiced vowel sites as voiceless fricatives, evidenced by high‐frequency frication in spectrograms and a significantly higher number of zero crossings, will be discussed, as will the interplay between devoicing and fricativization in the data set. Analysis of the formant frequency information at the vowel sites, preserved in the frication and allowing the perception of the original vowels even in the face of heavy loss of vocality, is also presented.

The Relative Contributions of Speaking Fundamental Frequency and Formant Frequencies to Gender Identification Based on Isolated Vowels

The purpose of this study was to determine the accuracy with which listeners could identify the gender of a speaker from a synthesized isolated vowel based on the natural production of that speaker when (1) the fundamental frequency was consistent with the speaker's gender, (2) the fundamental frequency was inconsistent with the the speaker's gender, and (3) the speaker was transgendered. Ten male-to-female transgendered persons, 10 men and 10 women, served as subjects. Each speaker produced the vowels /i/, /u/, and //. These vowels were analyzed for fundamental frequency and the first three formant frequencies and bandwidths. Formant frequency and bandwidth information was used to synthesize two vowel tokens for each speaker, one at a fundamental frequency of 120 Hz and one at 240 Hz. Listeners were asked to listen to these tokens and determine whether the original speaker was male or female. Listeners were not aware of the use of transgendered speakers. Results showed that, in all cases, gender identifications were based on fundamental frequency, even when fundamental frequency and formant frequency information was contradictory.

Acoustic correlates of ‘‘devoiced’’ vowels in standard modern Japanese

This paper documents the acoustic characteristics of what have been called devoiced vowels in Tokyo Japanese, traditionally attributed to the phonological rule of high vowel devoicing [T. J. Vance, Introduction to Japanese Phonology (1987)]. More recently they have been attributed to articulatory gesture overlap [S-A. Jun and M. E. Beckman, Paper, Annual Meeting of the LSA (1993)]. Current research by the author indicates both processes are consistent with the data. The vowels which have undergone phonological processing, however, are not simply devoiced; they have undergone a process of obstruentization similar to that found in the Turkish language of Uyghur discussed both pedologically [R. F. Hahn, Spoken Uyghur (1991)] and phonologically [E. M. Kaisse, Language 68, 313–332 (1992)]. Specifically, the vowel sites surface as voiceless fricatives at the place of articulation of the preceding voiceless consonant. This is evidenced acoustically by high-frequency frication in spectrograms and a high number of zero crossings. Additionally, formant frequency information is preserved, as evidenced by formant bars in spectrograms and spectral sections, allowing the perception of the original vowels in the face of heavy loss of vocality.

Vowel perception and neural plasticity in cochlear implant users: The effect of more-basal-than-normal stimulation

Because formant frequencies are a crucial cue in identifying English vowels, some pulsatile multichannel cochlear implants attempt to deliver formant frequency information by stimulating different electrodes depending on instantaneous formant values. For users of these cochlear implants, it may be possible to predict performance in vowel confusion studies based on the subject’s ability to discriminate different electrodes. One possible confounding factor is that cochlear implants stimulate cochlear locations that are more basal than normal (i.e., associated with higher frequencies). To what extent is this more basal-than-normal stimulation a problem? Is it important to stimulate the ‘‘physiologically correct’’ places in the cochlea, or (to state a diametrically opposed hypothesis) is the place code infinitely plastic? A mathematical model of vowel perception with cochlear implants is proposed that explicitly tests both hypotheses. The model attempts to predict a postlingually deaf subject’s vowel confusions based on his electrode discrimination and on his ‘‘perceptual bias’’ (meaning, in this case, the extent to which he has adapted to more-basal-than-normal stimulation). The conclusion is that more-basal-than-normal stimulation of the cochlea degrades speech perception shortly after implantation, but becomes less critical as subjects adapt to the new location.

Comparison of Two Multichannel Tactile Devices as Supplements to Speechreading in a Postlingually Deafened Adult

The purpose of this study was to conduct a single-subject comparison of the effectiveness of two multichannel vibrotactile devices that encode different classes of speech information. One device, the Portapitch, is designed to convey fundamental frequency (F0) and its variation over time. The other, the TACTAID 7, is designed to convey the first two formant frequencies (F1 and F2) and their variation over time. The subject, a postlingually deafened adult, underwent an intensive 17-wk training and testing protocol with the Portapitch and then completed a similar 17-wk protocol with the TACTAID 7. Performance measures were obtained on phonetic-contrast perception by speechreading alone, tactile device alone, and speechreading plus tactile device, and on open-set word and sentence recognition by speechreading alone and speechreading plus tactile device. On phonetic-contrast testing, the subject demonstrated some ability to perceive voicing, stress, and intonation contrasts using the Portapitch, but gave little evidence of phonetic-contrast perception with the TACTAID 7. On open-set word recognition testing, no significant improvements were seen with either device. On open-set sentence recognition testing, the subject showed a significant 9 percentage point enhancement effect using the Portapitch; the mean 5 percentage point enhancement effect provided by the TACTAID 7 was not statistically significant. A small advantage was seen in favor of the tactile display of F0 relative to the tactile display of formant frequency information on both phonetic-contrast testing and open-set sentence recognition. The difference, however, was of questionable significance and could have been confounded with an order effect. Nevertheless, the subject's preference was for the tactile formant frequency display.

Natural Vowel Perception by Patients with the Ineraid Cochlear Implant

Vowel recognition was tested in 10 patients using the Ineraid cochlear implant. The vowels were produced by a male speaker in the context 'heed, hid, head, had, hawed, hood, who'd, hud' and 'heard'. Performance varied from 34 to 93% correct. A descriptive feature system for the vowels was determined from an acoustic analysis. An information transfer analysis of these features suggested that information about the first formant frequency, vowel duration and fundamental frequency was transmitted. Information about the second and third formant frequency was transmitted less well. A sequential information transmission analysis suggested that the features of the first formant and duration accounted for nearly 80% of the information transmitted. The fundamental frequency and second formant frequency information accounted for an additional 8%. Information provided by the third formant frequency was largely redundant.

Use of spectral and temporal cues in vowel identification by normal‐hearing and hearing‐impaired listeners

Listeners with sensorineural hearing impairments generally show deficits in frequency resolution accompanying their decreased auditory sensitivity. Poor frequency resolution may make it difficult for these listeners to distinguish among vowels with similar formant frequencies. In English, these vowels tend to form tense/lax pairs (e.g., i/ɪ, æ/ɛ) which differ in duration. The present study examined hearing‐impaired and normal‐hearing listeners' use of duration and formant frequency information in the labeling of synthetic CVC stimuli forming a /bit‐bit/continuum. Durational and F2 frequency cues to vowel identity varied systematically across stimuli. Frequency resolution at 2000 Hz (the F2 region for i/ɪ was measured using a notched‐noise masking paradigm. A temporal difference limen for a narrow‐band noise centered at 2000 Hz was also measured. Subjects with normal frequency and temporal resolution tended to rely primarily on formant frequency information in the vowel labeling task. However, subjects with abnormal frequency resolution but near‐normal temporal difference limens made greater use of vowel duration in vowel identification. [Work supported by NIH.]

Articulatory and acoustic signatures of selected monosyllabic words

At previous meetings of this Society, acoustic signatures for selected words from an intelligibility test have been described. The signatures consist minimally of the formant trajectories of the syllable nucleus of each word. The formant frequency information can be augmented by other acoustic information, such as information on nasal murmurs, stop bursts, silences, and frication intervals. A major goal of this work has been to determine the interspeaker variance of the signatures and to evaluate their usefulness in characterizing speech disorders. This report extends the research to include articulatory data obtained by x-ray microbeam. The articulatory signatures take the form of planar motion paths of radiopaque markers attached to the articulators. Articulatory data were obtained from five talkers. Selected results will be presented to illustrate the articulatory signatures and to show their relationships to acoustic signatures of the same words. The primary goal of this report is to describe intraspeaker and interspeaker variances as they relate to the possibility of developing nonnative standards for articulatory-acoustic signatures. [Work supported in part by NIH.]

Vowel and consonant recognition of cochlear implant patients using formant-estimating speech processors.

Vowel and consonant confusion matrices were collected in the hearing alone (H), lipreading alone (L), and hearing plus lipreading (HL) conditions for 28 patients participating in the clinical trial of the multiple-channel cochlear implant. All patients were profound-to-totally deaf and "hearing" refers to the presentation of auditory information via the implant. The average scores were 49% for vowels and 37% for consonants in the H condition and the HL scores were significantly higher than the L scores. Information transmission and multidimensional scaling analyses showed that different speech features were conveyed at different levels in the H and L conditions. In the HL condition, the visual and auditory signals provided independent information sources for each feature. For vowels, the auditory signal was the major source of duration information, while the visual signal was the major source of first and second formant frequency information. The implant provided information about the amplitude envelope of the speech and the estimated frequency of the main spectral peak between 800 and 4000 Hz, which was useful for consonant recognition. A speech processor that coded the estimated frequency and amplitude of an additional peak between 300 and 1000 Hz was shown to increase the vowel and consonant recognition in the H condition by improving the transmission of first formant and voicing information.

Speechreading supplemented with auditorily presented speech parameters.

Results are reported from two experiments in which the benefit of supplementing speechreading with auditorily presented information about the speech signal was investigated. In experiment I, speechreading was supplemented with information about the prosody of the speech signal. For ten normal-hearing subjects with no experience in speechreading, the intelligibility score for sentences increased significantly when speechreading was supplemented with information about the overall amplitude of the speech signal, information about the fundamental frequency, or both. Binary information about voicing appeared not to be a significant supplement. In experiment II, the best-scoring supplements of experiment I were compared with two supplementary signals from our previous studies, i.e., information about the sound-pressure levels in two 1-oct filter bands centered at 500 and 3160 Hz, or information about the frequencies of the first and second formants from voiced speech segments. Sentence-intelligibility scores were measured for 24 normal-hearing subjects with no experience in speechreading, and for 12 normal-hearing experienced speechreaders. For the inexperienced speechreaders, the sound-pressure levels appeared to be the best supplement (87.1% correct syllables). For the experienced speechreaders, the formant-frequency information (88.6% correct), and the fundamental-frequency plus amplitude information (86.0% correct), were equally efficient supplements as the sound-pressure information (86.1% correct). Discrimination of phonemes (both consonants and vowels) was measured for the group of 24 inexperienced speechreaders. Percentage correct responses, confusion among phonemes, and the percentage of transmitted information about different types of manner and place of articulation and about the feature voicing are presented.

Evidence of talker-independent information for vowels.

The vowel information present in initial and final regions of /b/—vowel—/b/ syllables was examined in this study. Vowels were identified for unedited syllables spoken by a man and a woman, for the initial 20% of those syllables, for the final 20% of the syllables, for the initial and final 20% of the syllables combined and separated by a 60% silent gap, and for the initial and final 20% of the syllables interchanged across talkers and separated by a 60% silent gap. Results indicate: (1) that there is considerable vowel information present in the dynamic regions at the beginnings and endings of syllables; (2) that the information is, to a large extent, carried relationally by those regions; (3) that the information is talker-independent in form; and (4) that the information is complementary to, and distinct from, formant frequency information present in a syllable's center. An experiment assessing the perceived source(s) of these stimuli suggests that source perception is influenced by as yet unspecified acoustic modulations defined at the syllable level.

Speechreading supplemented with formant-frequency information from voiced speech.

The benefit of supplementing speechreading with information about the frequencies of the first and second formants from the voiced sections of the speech signal was studied by presenting short sentences to 18 normal-hearing listeners under the following three conditions: (a) speechreading combined with listening to the formant-frequency information, (b) speechreading only, and (c) formant-frequency information only. The formant frequencies were presented either as pure tones or as a complex speechlike signal, obtained by filtering a periodic pulse sequence of 250 Hz by a cascade of four second-order bandpass filters (with constant bandwidth); the center frequencies of two of these filters followed the frequencies of the first and second formants, whereas the frequencies of the others remained constant. The percentage of correctly identified syllables increased from 22.8 in the case of speechreading only to 82.0 in the case of speechreading while listening to the complex speechlike signal. Listening to the formant information only scored 33.2% correct. However, comparison with the best-scoring condition of our previous study [Breeuwer and Plomp, J. Acoust. Soc. Am. 76, 686-691 (1984)] indicates that information about the sound-pressure levels in two one-octave filter bands with center frequencies of 500 and 3160 Hz is a more effective supplement to speechreading than the formant-frequency information.

Speechreading supplemented with frequency-selective sound-pressure information and formant-frequency information

The benefit of supplementing speechreading with frequency-selective sound-pressure information or formant-frequency information was studied by auditorily presenting this information to normal-hearing listeners without experience in speechreading. In a first experiment the sound-pressure levels in one or two, octave or 1/3-octave bands were used to amplitude-modulate pure-tone carriers with frequencies equal to the center frequencies of the filterbands (500, 1600, or 3160 Hz). Eighteen listeners scored 65.7% correct syllables (from totally unknown sentences) for speechreading with sound-pressure information in a single one-octave band at 500 Hz, and 86.7% correct syllables for speechreading with two one-octave bands at 500 and 3160 Hz, respectively. Speechreading-only scored 22.8%. In a second experiment the frequencies of the first and second formant from voiced speech were presented either as pure tones, or as a complex speechlike signal. In this experiment 18 (other) listeners scored 82.0% correct syllables on speechreading with formant-frequency information. No significant difference was found between presenting this information as pure tones or as a complex speechlike signal. Speechreading-only scored again 22.8% correct syllables.