Segregation Of Concurrent Vowels Research Articles

Modeling Concurrent Vowel Scores Using the Time Delay Neural Network and Multitask Learning

The difference in fundamental frequency (F0) between vowels is a segregation cue for identifying concurrent vowels. Younger listeners with normal hearing (YNH) showed that the percent identification score of both vowels improved with increasing F0 difference and asymptoted at higher F0 difference. The current study developed a deep-neural-network model consisting of a time-delay neural network and multitask learning (TDNN-MTL) to predict the concurrent vowel scores. The input to the TDNN-MTL were temporal responses of concurrent vowels obtained using the auditory-nerve model. The TDNN-MTL was trained at a 3-Hz F0 difference until the score (80%) became closer to the YNH score (85%). The TDNN-MTL learned the formant coding and aided in the segregation of concurrent vowels into dominant and recessive vowels. The total weighted loss was tailored to mimic the dominant-recessive relationship of the YNH scores. The TDNN-MTL was tested against the other five F0 differences. The TDNN-MTL model scores of both vowels were successful in predicting the YNH scores. The chi-square tests revealed that the TDNN-MTL model scores correlated well with the YNH scores compared with other F0-segregation and multi-layer perceptron models. The TDNN-MTL also correctly predicted the one vowel identification scores and F0 benefit. These findings suggest that the trained TDNN-MTL accurately predicts the formant coding of the concurrent vowels across F0 differences, which aided in capturing the YNH scores. The TDNN-MTL can be extended to validate the behavioral studies on concurrent vowels obtained across acoustic (e.g., vowel duration and level) and auditory changes (e.g., aging and hearing loss).

Infants' use of isolated and combined temporal cues in speech sound segregation.

This paper investigates infants' and adults' use of envelope cues and combined onset asynchrony and envelope cues in the segregation of concurrent vowels. Listeners heard superimposed vowel pairs consisting of two different vowels spoken by a male and a female talker and were trained to respond to one specific target vowel, either the male /u:/ or male /i:/. Vowel detection was measured in three conditions. In the baseline condition the two superimposed vowels had similar amplitude envelopes and synchronous onset. In the envelope cue condition, the amplitude envelopes of the two vowels differed. In the combined cue condition, both the onset time and amplitude envelopes of the two vowels differed. Seven-month-old infants' concurrent vowel segregation improved both with envelope and with combined onset asynchrony and envelope cues to the same extent as adults'. A preliminary investigation with 3-month-old infants suggested that neither envelope cues nor combined asynchrony and envelope cues improved their ability to detect the target vowel. Taken together, these results suggest that envelope and combined onset-asynchrony cues are available to infants as they attempt to process competing speech sounds, at least after 7 months of age.

Infants use onset asynchrony cues in auditory scene analysis.

This experiment investigated the effect of onset asynchrony on the segregation of concurrent vowels in infants and adults. Two vowels, randomly chosen from seven American-English vowels, were superimposed. Each vowel pair contained one vowel by a male and one by a female talker. A train of such vowel pairs was presented to listeners, who were trained to respond to the male target vowel /i:/ or /u:/. The ability to identify the target vowel was compared among three conditions: synchronous onset, 100-, and 200-ms onset asynchrony. Experiment 1 measured performance, in d', in 7-month-old infants and adults. Infants and adults performed better with asynchronous than synchronous vowel onset, regardless of asynchrony duration. Experiment 2 compared the proportion of 3-month-old infants achieving an 80% correct criterion with and without onset asynchrony. Significantly more infants reached criterion with asynchronous than with synchronous vowel onset. Asynchrony duration did not influence performance. These experiments show that infants, as young as 3 months old, benefit from onset asynchrony.

Infants' use of amplitude modulation cues in the segregation of concurrent vowels

Infants have difficulties separating speech from competing sounds. One explanation is that infants do not use acoustic cues to sound source segregation as adults do. This study investigated 3- and 7-month-old infants’ and adults’ ability to use AM cues in concurrent vowel segregation. Seven American-English vowels were produced by a male and by a female talker. One male and one female vowel were randomly chosen and superimposed. A train of such vowel pairs was presented to listeners, who were trained to respond to the male target vowel /i:/ or /u:/. Target-to-nontarget ratios were chosen to equate performance across age groups. In the baseline condition, all vowels had flat envelopes. In the cue condition, male vowels were modulated with three envelopes from infant-directed vowels. The proportion of 3-month-old infants achieving an 80% correct criterion at a ratio of + 15dB was higher in the cue than in the baseline condition. For 7-month olds and adults d' could be estimated based on 15 target and 15 nontarget trials in each condition. Preliminary results indicate that both 7-month-old infants and adults have a higher d' in the cue than in the baseline condition. Thus, infants appear to use AM to segregate concurrent vowels.

Infants' use of onset asynchrony in the segregation of concurrent vowels

Separating speech from competing speech is a difficult task for infants even though their ability to encode sounds appears to be mature. One possible explanation is that infants have greater difficulties extracting speech from the complex sound mixture arriving at their ears. Adults use acoustic cues such as differences in onset or harmonicity to group components that belong to the same sound source and separate those that do not. The sparse evidence on how infants segregate sounds indicates that they use some of these acoustic cues. However, temporal cues, which are strong cues for adults, have not been investigated in infants. This study examined infants’ ability to use onset asynchrony cues to separate simultaneous vowels using a single-interval observer-based procedure. Three- and seven-month-old infants and young adults were trained to identify one target vowel in a modified double-vowel paradigm. Listeners were tested in a simultaneous onset condition and either a 100 or 200 ms asynchrony condition. Cue benefit was defined as the improvement in sensitivity resulting from the addition of the asynchrony cue. Pilot data indicates that an onset asynchrony of 100ms improves performance for all age groups and that the cue benefit increases with age.

Neural Representation of Concurrent Vowels in Macaque Primary Auditory Cortex.

Successful speech perception in real-world environments requires that the auditory system segregate competing voices that overlap in frequency and time into separate streams. Vowels are major constituents of speech and are comprised of frequencies (harmonics) that are integer multiples of a common fundamental frequency (F0). The pitch and identity of a vowel are determined by its F0 and spectral envelope (formant structure), respectively. When two spectrally overlapping vowels differing in F0 are presented concurrently, they can be readily perceived as two separate “auditory objects” with pitches at their respective F0s. A difference in pitch between two simultaneous vowels provides a powerful cue for their segregation, which in turn, facilitates their individual identification. The neural mechanisms underlying the segregation of concurrent vowels based on pitch differences are poorly understood. Here, we examine neural population responses in macaque primary auditory cortex (A1) to single and double concurrent vowels (/a/ and /i/) that differ in F0 such that they are heard as two separate auditory objects with distinct pitches. We find that neural population responses in A1 can resolve, via a rate-place code, lower harmonics of both single and double concurrent vowels. Furthermore, we show that the formant structures, and hence the identities, of single vowels can be reliably recovered from the neural representation of double concurrent vowels. We conclude that A1 contains sufficient spectral information to enable concurrent vowel segregation and identification by downstream cortical areas.

Neural Segregation of Concurrent Speech: Effects of Background Noise and Reverberation on Auditory Scene Analysis in the Ventral Cochlear Nucleus.

Concurrent complex sounds (e.g., two voices speaking at once) are perceptually disentangled into separate "auditory objects". This neural processing often occurs in the presence of acoustic-signal distortions from noise and reverberation (e.g., in a busy restaurant). A difference in periodicity between sounds is a strong segregation cue under quiet, anechoic conditions. However, noise and reverberation exert differential effects on speech intelligibility under "cocktail-party" listening conditions. Previous neurophysiological studies have concentrated on understanding auditory scene analysis under ideal listening conditions. Here, we examine the effects of noise and reverberation on periodicity-based neural segregation of concurrent vowels /a/ and /i/, in the responses of single units in the guinea-pig ventral cochlear nucleus (VCN): the first processing station of the auditory brain stem. In line with human psychoacoustic data, we find reverberation significantly impairs segregation when vowels have an intonated pitch contour, but not when they are spoken on a monotone. In contrast, noise impairs segregation independent of intonation pattern. These results are informative for models of speech processing under ecologically valid listening conditions, where noise and reverberation abound.

Computational Auditory Scene Analysis: Principles, Algorithms and Applications

First Page

Age-related changes in neural activity associated with concurrent vowel segregation

Older adults exhibit degraded speech comprehension in complex sound environments, which may be related to overall age-related declines in low-level sound segregation. This hypothesis was tested by measuring event-related potentials (ERPs) while listeners identified two different vowels presented simultaneously. Older adults were less accurate than young adults at identifying the two vowels, although both groups improved similarly with increasing fundamental frequency differences (Δ ƒ 0) between vowels. Reaction time data showed that older adults took more time to process stimuli, especially those with smaller Δ ƒ 0. A negative ERP wave indexing the automatic registration of Δ ƒ 0 (the object-related negativity) was reduced in older adults. In contrast, young and older adults showed a similar pattern of neural activity indexing attentive processing of Δ ƒ 0. The results suggest that aging affects the ability to automatically segregate speech sounds.

Waveform interactions and the segregation of concurrent vowels.

Two experiments investigated the effects of small values of fundamental frequency difference (delta F0) on the identification of concurrent vowels. As delta F0's get smaller, mechanisms that exploit them must necessarily fail, and the pattern of breakdown may tell which mechanisms are used by the auditory system. Small delta F0's also present a methodological difficulty. If the stimulus is shorter than the beat period, its spectrum depends on which part of the beat pattern is sampled. A different starting phase might produce a different experimental outcome, and the experiment may lack generality. The first experiment explored the effects of delta F0's as small as 0.4%. The smallest delta F0 conditions were synthesized with several starting phases obtained by gating successive segments of the beat pattern. An improvement in identification was demonstrated for delta F0's as small as 0.4% for all segments. Differences between segments (or starting phase) were also observed, but when averaged over vowel pairs they were of small magnitude compared to delta F0 effects. The nature of delta F0-induced waveform interactions and the factors that affect them are discussed in detail in a tutorial section, and the hypothesis that the improvement in identification is the result of such interactions (beat hypothesis) is examined. It is unlikely that this hypothesis can account for the effects observed. The reduced benefit of delta F0 for identification at smaller delta F0's more likely reflects the breakdown of the same F0-guided segregation mechanism that operates at larger delta F0's.

Concurrent vowel identification. III. A neural model of harmonic interference cancellation

This paper presents a “neural cancellation filter” capable of segregating weak targets from competing harmonic backgrounds, and a model of concurrent vowel segregation based upon it. The elementary cancellation filter comprises a delay line and an inhibitory synapse. Filters within each peripheral channel are tuned to the period of the competing sound to suppress its correlates within the neural discharge pattern. In combination with a pattern matching model based on autocorrelation functions summed over channels, the cancellation filter forms a model of concurrent vowel identification. The model predicts the number of vowels reported for each stimulus (when subjects are allowed to report one or two) and identification rates. It belongs to the class of “harmonic cancellation” models that are supported by experimental evidence that vowel identification is better when competing sounds are harmonic than inharmonic. Two alternative schemes using the same filter are also considered. One derives a “place” representation from the magnitude of the filter output. The other uses the ratio of filter input/output to select channels.

The role of frequency modulation in the perceptual segregation of concurrent vowels.

Two experiments investigated the effect of frequency modulation on the identification of vowel sounds presented concurrently with interfering vowels. In experiment 1, identification thresholds were measured for each of five target vowels, masked, in each trial, by one of ten masking vowels. Both target and masking vowels were synthesized using harmonically spaced frequency components. Inharmonic spacing was used in order to prevent powerful grouping processes which exploit fundamental frequency from dominating the results. The target vowels were synthesized with sinusoidal frequency modulation on each frequency component which was either coherent (same phase) or incoherent (random phases). The masking vowels were synthesized with components which were either modulated in the same way as the target vowel or were unmodulated. Identification thresholds were lower when the masking vowel had no modulation. The effect occurred for both coherent and incoherent frequency modulation, indicating that it is mediated by the movement of each component independently, rather than by grouping of coherently modulated components. This result is consistent in some respects with judgments of the prominence of competing vowels [S. E. McAdams, J. Acoust. Soc. Am. 85, 2148-2159 (1989)], which show that modulated vowels are more prominent than unmodulated vowels regardless of the type of modulation applied to the competing vowels. Experiment 2 used a paradigm similar to that developed by McAdams, in order to compare more directly the effect of FM on vowel identification and vowel prominence. On each trial, three vowels were presented concurrently. Either none, one, or two of the vowels were modulated throughout, while modulation was applied to another vowel (the target) halfway through the stimulus. The vowels were either harmonic (with different fundamental frequencies) and coherently modulated or inharmonic and incoherently modulated. Accuracy of identification of the target vowel was not significantly different in the harmonic/coherent and inharmonic/incoherent conditions and declined, in each case, as the number of modulated background vowels increased. Overall, the results of experiments 1 and 2, and of McAdams' prominence judgment data, suggest that there is an auditory mechanism for detecting frequency modulation which can alert the listener to the presence of frequency modulated sounds, but which is insensitive to across-frequency differences in the pattern of that modulation.

Waveform dynamics and the perceptual segregation of concurrent vowels

When two synthetic vowels are presented concurrently, listeners identify the vowels more accurately if they differ in fundamental frequency (F0) or if one of them is preceded/followed by a gliding (versus static) formant pattern. Previous experiments have shown that gliding formants generally do not help listeners identify the vowel to which they are linked; instead, they make the vowel without transitions easier to identify. One explanation is that the formant transition region provides a brief interval during which the competing steady-state vowel is perceptually more prominent. This interpretation is supported by two computational models that perform a filter bank analysis, process the waveform in each filter channel using a sliding temporal window, and determine which region of the signal provides the strongest evidence of each vowel. Model A computed the energy in each channel at successive time intervals to generate running excitation patterns. Model B used a temporal analysis to generate running autocorrelation functions, and included a further stage to partition the channels based on periodicity cues. Both models predicted effects of F0 and gliding formants, but model B provided better predictions of the pattern of listeners’ identification responses. Identification of concurrent vowels appears to benefit from an analysis of the composite waveform using a sliding temporal window, combined with a form of F0-guided source segregation.

A model of concurrent vowel segregation using an implementation of the subtraction strategy

The perceptual experiments performed by Lea [‘‘Auditory Modeling of Vowel Perception,’’ unpublished doctoral thesis, University of Nottingham (1992)] required listeners to identify the members of pairs of steady-state synthetic vowels presented monaurally, called concurrent vowels. Accuracy of identification was higher when one vowel was voiced and the other whispered compared to control conditions in which both were whispered or both were voiced with the same fundamental frequency (f0). Surprisingly, the improvement in accuracy was restricted to the whispered member of the voiced/whispered pair. This outcome is compatible with the idea that one strategy used by listeners to recover a target voice from a mixture of voices is to cancel an interfering voice by a process of spectrotemporal subtraction. Here, a computational model is described that implements this strategy. The model is based on a bank of bandpass filters and an array of autocorrelators. The period of the dominant pitch in a stimulus is estimated from the largest peak in the normalized sum of the autocorrelation functions. This estimate guides the synthesis of an array of autocorrelation functions that are subtracted from the array generated by the stimulus, hence, canceling an interfering voice. Following Meddis and Hewitt [J. Acoust. Soc. Am. 91, 233–245 (1992)], vowel identification is performed by matching the short-time part of the resulting summed autocorrelation functions to a set of stored templates. The model makes good qualitative predictions of the accuracy of listeners’ identification responses to the stimuli used in the perceptual experiments. In addition, it makes good quantitative predictions of the effects of introducing a difference in f0 between concurrent vowels reported previously by Assmann and Summerfield [J. Acoust. Soc. Am. 88, 680–697 (1990)]. [Work supported by the MRC of the UK.]

Perceptual segregation of concurrent vowels

The ability to group together the acoustic components of a voice and segregate them from competing voices is an important skill in speech communication. To study the auditory processes involved, pairs of vowels were presented simultaneously to listeners with normal hearing. Identification performance improved as the fundamental-frequency difference between the members of the pair increased. Two auditory models were compared to predict this result. The composite waveform was passed through an array of bandpass filters simulating the cat's limited frequency resolution. The “place” model estimated the two fundamentals from peaks in the profile of the rms outputs of the filters. It then sampled the profile at integer multiples of each fundamental to reconstruct the spectra of the constituent vowels. The “place-time” model estimated the fundamentals from dominant periodicities in the waveforms emerging from the filters. The amount of synchronization to each fundamental was plotted as a function of filter center frequency to reconstruct the spectra. The place-time model estimated the two fundamentals more accurately than the place model, and predicted identification responses more precisely. This outcome is compatible with the idea that a temporal analysis is involved in the segregation of concurrent sounds.