Simultaneous Pitches Research Articles

Synthesis and Analysis-By-Synthesis of Modulated Diplophonic Glottal Area Waveforms

Diplophonia is a type of disordered voice in which two simultaneous pitches are perceived. Most commonly in diplophonic voices, the vocal folds are divided into two parts that vibrate at different frequencies. The glottal area is the projected area of the space between the vocal folds. The glottal area in time is referred to as the glottal area waveform (GAW). The GAW is modeled for diplophonic voice by superimposing two partial GAWs (pGAWs) that are trains of single-peak pulses with different pulse frequencies, i.e., fundamental frequencies ( $f_o$ s). In current kinematic models of diplophonic vocal fold vibration, the pGAWs are assumed to be quasiperiodic. This assumption is mitigated here by modulating pulse-to-pulse cycle length and amplitude. Both random and deterministic modulations are considered. Deterministic modulations depend on the difference of the pGAWs’ instantaneous phases. Model GAWs are fitted to input GAWs using an analysis-by-synthesis approach which we refer to as ‘modulated pulse trains decomposition’ (MPD). MPD is shown to be applicable to diplophonic as well as to nondiplophonic types of dysphonia, which include multi-pulse patterns, random timing behaviours, and chaos. It is mostly robust against modulations but degraded by large random modulations. MPD is compared to a deep autoencoder neural network, and the WaveGlow neural network. In terms of time-domain fitting errors, MPD outperforms the other two approaches unless random modulations are large. MPD outperforms the best of the other two approaches by up to approximately 5 dB. For large random modulations, the deep autoencoder network achieves the smallest fitting errors. In terms of magnitude spectrum fitting errors, WaveGlow is superior except for natural input GAWs containing only nondiplophonic types of dysphonia. Also pulse timing errors are shown to be advantageous for MPD.

The Perception of Multiple Simultaneous Pitches as a Function of Number of Spectral Channels and Spectral Spread in a Noise-Excited Envelope Vocoder.

Cochlear implant (CI) listeners typically perform poorly on tasks involving the pitch of complex tones. This limitation in performance is thought to be mainly due to the restricted number of active channels and the broad current spread that leads to channel interactions and subsequent loss of precise spectral information, with temporal information limited primarily to temporal-envelope cues. Little is known about the degree of spectral resolution required to perceive combinations of multiple pitches, or a single pitch in the presence of other interfering tones in the same spectral region. This study used noise-excited envelope vocoders that simulate the limited resolution of CIs to explore the perception of multiple pitches presented simultaneously. The results show that the resolution required for perceiving multiple complex pitches is comparable to that found in a previous study using single complex tones. Although relatively high performance can be achieved with 48 channels, performance remained near chance when even limited spectral spread (with filter slopes as steep as 144 dB/octave) was introduced to the simulations. Overall, these tight constraints suggest that current CI technology will not be able to convey the pitches of combinations of spectrally overlapping complex tones.

Pitch discrimination with mixtures of three concurrent harmonic complexes.

In natural listening contexts, especially in music, it is common to hear three or more simultaneous pitches, but few empirical or theoretical studies have addressed how this is achieved. Place and pattern-recognition theories of pitch require at least some harmonics to be spectrally resolved for pitch to be extracted, but it is unclear how often such conditions exist when multiple complex tones are presented together. In three behavioral experiments, mixtures of three concurrent complexes were filtered into a single bandpass spectral region, and the relationship between the fundamental frequencies and spectral region was varied in order to manipulate the extent to which harmonics were resolved either before or after mixing. In experiment 1, listeners discriminated major from minor triads (a difference of 1 semitone in one note of the triad). In experiments 2 and 3, listeners compared the pitch of a probe tone with that of a subsequent target, embedded within two other tones. All three experiments demonstrated above-chance performance, even in conditions where the combinations of harmonic components were unlikely to be resolved after mixing, suggesting that fully resolved harmonics may not be necessary to extract the pitch from multiple simultaneous complexes.

Multiple f0 pitch estimation for musical applications using dynamic Bayesian networks and learned priors

The identification of multiple simultaneous pitches is a challenging signal processing task and cannot at present be performed as well as trained human subjects. Moreover, it appears that successful human performance depends on skill acquisition and knowledge of musical conventions. Even human capabilities are likely fairly poor in the absence of training and musical context. We present a framework, using Dynamic Bayesian Networks, that permits the principled incorporation of models of music theory, musical instruments, and human pitch perception. A particular advantage of this approach is the ability to develop each of these models independently, relying either on expert knowledge or machine learning as necessary. Models of appropriate complexity can then be selected for a specific application. In the present work, we focus on the use of learned models of musical context, specifically Deep Markov Models, and use these to improve inferences about simultaneous pitches. The main drawback of this framework is the intractability of the inference computations and the computational expense of approximation methods. We explore particle filtering as an approach to addressing these problems with the ultimate aim of making a system useable in a musical performance system. [Work supported by NSF BCS-1539276 and IBM AIRC grant.]

Spectral and temporal models of human pitch perception with mixtures of three concurrent harmonic complexes

In music and other everyday situations, humans are often presented with three or more simultaneous pitches, each carried by a harmonic complex tone, but few empirical or theoretical studies have addressed how pitch is perceived in this context. In three behavioral experiments, mixtures of three concurrent complexes were filtered into a single bandpass spectral region, and the relationship between the fundamental frequencies and spectral region was varied in order to manipulate the extent to which harmonics were resolved either before or after mixing. Listeners were asked to discriminate major from minor chords (Experiment 1) or to compare the pitch of a probe tone to that of a target embedded in the mixture (Experiments 2 and 3). In all three experiments, listeners performed above chance even under conditions where traditional rate-place models would not predict individually resolved components. Human behavioral results were compared to predictions from two classes of pitch model: a rate-place model using harmonic template matching and a temporal model using summary autocorrelation. Predictions from a combined model, using both rate-place and temporal information, were more accurate than predictions from either model alone, suggesting that humans may integrate these two kinds of information. [Work supported by NIH grant R01DC005216.]

Pitch discrimination for multiple simultaneous complexes: Effects of harmonic resolvability

In natural listening contexts, especially in music, it is common to hear three or more simultaneous pitches, each defined by a harmonic complex tone. In order to extract the pitches of these tones using the rate-place code, peripherally resolved components are required, but the peripheral resolvability of a complex tone is reduced when multiple pitches are presented. In this experiment, we investigated the effect of resolvability on pitch discrimination in the context of multiple pitches. Listeners were asked to discriminate the direction of a 0.5-semitone pitch change at the end of a four-tone sequence, where the last tone in the sequence was embedded in a mixture of two other simultaneous tones. Tones were either pure tones, or complex tones filtered into one of two bandpass regions, in order to manipulate the extent to which harmonics were resolved both before and after mixing. Discrimination performance was significantly above chance, even in the high-frequency region where the combination of harmonic components should have been unresolved after mixing, suggesting that resolved harmonics may not be necessary to extract the pitch from multiple complex tones. Predictions from spectral and temporal models of pitch were compared with the results. [Work supported by NIH grant R01DC005216.]

Neural representation of a melodic motif: Effects of polyphonic contexts

In music, a melodic motif is often played repeatedly in different pitch ranges and at different times. Event-related potential (ERP) studies have shown that the mismatch negativity (MMN) reflects memory trace processing that encodes two separate melodic lines (“voices”) with different motifs. Here we investigated whether a single motif presented in two voices is encoded as a single entity or two separate entities, and whether motifs overlapping in time impede or enhance encoding strength. Electroencephalogram (EEG) from 11 musically-trained participants was recorded while they passively listened to sequences of 5-note motifs where the 5th note either descended (standard) or ascended (deviant) relative to the previous note (20% deviant rate). Motifs were presented either in one pitch range, or alternated between two pitch ranges, creating an “upper” and a “lower” voice. Further, motifs were either temporally isolated (silence in between), or temporally concurrent with two tones overlapping. When motifs were temporally isolated, MMN amplitude in the one-pitch-range condition was similar to that in the two-pitch-range upper voice. In contrast, no MMN, but P3a, was observed in the two-pitch-range lower voice. When motifs were temporally concurrent and presented in two pitch ranges, MMN exhibited a more posterior distribution in the upper voice, but again, was absent in the lower voice. These results suggest that motifs presented in two separate voices are not encoded entirely independently, but hierarchically, causing asymmetry between the upper and lower voice encoding even when no simultaneous pitches are presented.

Combining Spectral and Temporal Representations for Multipitch Estimation of Polyphonic Music

Due to the difficulty of creating pitch-labeled training data that cover the rich diversity found in music signals, unsupervised feature-based approaches derived from signal processing and feature design remain critical for multipitch estimation (MPE) of polyphonic music. While a large number of feature representations have been proposed in the literature, an effective means of combining different domains of features for MPE is still needed. In this paper, we propose a novel approach, referred to as combined frequency and periodicity (CFP), that detects pitches according to the agreement of a harmonic series in the frequency domain and a subharmonic series in the lag (quefrency) domain. This approach nicely aggregates the complementary advantages of the two feature domains in different frequency ranges, and improves the robustness of the pitch detection function to the interference of the overtones of simultaneous pitches. We report a comprehensive evaluation that compares CFP against three state-of-the-art approaches using three MPE datasets and four symphonies. The evaluation is characteristic of the coverage and complexity of music (in terms of instrument type and degree of polyphony). In addition, we also evaluate the performance of the MPE approaches when a number of audio degradations are applied. Results show that the proposed unsupervised method performs consistently well across the types of Western polyphonic music considered, and is robust to audio degradations such as high-pass filtering and MP3 compression.

Effects of resolved and unresolved harmonics in the perception of multiple pitches

The mechanisms underlying pitch perception have been studied for several decades. Most studies have concentrated on the pitch of single sounds, some have investigated the pitch of two simultaneously presented sounds, but few if any have investigated pitch perception with three or more simultaneous pitches. In contrast, in music it is rule, rather than the exception, that we are presented with multiple pitches. Here, pitch perception was studied with three simultaneously presented harmonic complexes, filtered into a single bandpass region to control the extent to which harmonics from each of the three tones were spectrally resolved. The tones comprised the three possible inversions of the major or minor triad, and the listener’s task was to discriminate between major and minor. Preliminary data suggest that subjects who could reliably discriminate major and minor triads in a pure tone training task were also capable of this discrimination when tone complexes were filtered such that the combination of harmonic components should have been unresolved on the cochlea. This finding suggests resolved harmonics may not be necessary to extract the pitch from multiple complex tones. Predictions from various spectral and temporal models of pitch were compared with the results. [Work supported by NIH grant R01DC05216.]

Perception of Polyphony With Cochlear Implants for 2 and 3 Simultaneous Pitches

It was hypothesized that cochlear implant (CI) subjects would be able to correctly identify 1, 2, and 3 simultaneous pitches through direct electrical stimulation. We further hypothesized that the location on the implant array and the fundamental frequency of the pitches would have an impact on the performance. "They gave me back speech but not music" is a sentence commonly heard by CI subjects. One of the reasons is that in music, multiple streams are frequently played at the same time, which is an essential feature of harmony. Current CI speech processors do not allow CI users to perceive such complex polyphonic sounds. In the present study, the authors assessed the ability of CI subjects to perceive simultaneous modulation frequencies based on direct electrical stimulation. Ten CI subjects were asked to identify 1, 2, and 3 simultaneous pitches applied on different electrodes using sinusoidal amplitude modulation. All stimuli were loudness balanced before the actual identification task. Subjects were able to identify 1, 2, and 3 simultaneous pitches. The further the distance between the 2 electrodes, the better was the performance in the 2-pitch condition. The distance between the modulation frequencies had a significant effect on the performance in the 2-and 3-pitch condition. Subjects are able to identify complex polyphonic stimuli based on the number of active electrodes. The additional polyphonic rate pitch cue improves performance in some conditions.

Development of Simultaneous Pitch Encoding: Infants Show a High Voice Superiority Effect

Infants must learn to make sense of real-world auditory environments containing simultaneous and overlapping sounds. In adults, event-related potential studies have demonstrated the existence of separate preattentive memory traces for concurrent note sequences and revealed perceptual dominance for encoding of the voice with higher fundamental frequency of 2 simultaneous tones or melodies. Here, we presented 2 simultaneous streams of notes (15 semitones apart) to 7-month-old infants. On 50% of trials, either the higher or the lower note was modified by one semitone, up or down, leaving 50% standard trials. Infants showed mismatch negativity (MMN) to changes in both voices, indicating separate memory traces for each voice. Furthermore, MMN was earlier and larger for the higher voice as in adults. When in the context of a second voice, representation of the lower voice was decreased and that of the higher voice increased compared with when each voice was presented alone. Additionally, correlations between MMN amplitude and amount of weekly music listening suggest that experience affects the development of auditory memory. In sum, the ability to process simultaneous pitches and the dominance of the highest voice emerge early during infancy and are likely important for the perceptual organization of sound in realistic environments.

Frequency Discrimination in a Simultaneous Two-Tone Signal

Discrimination of just noticeable change in pitch has been widely investigated. In contrast, works on discrimination of two simultaneous pitches produced by two tones have been scarce. In this work two methods were used to nd the minimum frequency interval in a two-tone at which listeners perceive two distinct pitches. In the rst, listeners tuned manually the frequency interval between components in a two-tone. In the second, the frequency interval between tones in a two-tone was switched between ve di erent xed values, and a pitch of a probe tone was compared with pitches in a two-tone. Both methods were used in two di erent groups of listeners, musicians and non-musicians. The rst method yielded substantially narrower intervals. Both methods revealed that musicians were able to discriminate pitches at narrower intervals than non-musicians. The raw results of tests were evaluated by two approaches, one was based on tting of the psychometric function and the other on statistical tests.

Music on more than one note: Pitch perception and neural coding of concurrent harmonic tones.

The basis of Western music lies in the combination of simultaneous (harmony) and sequential (melody) harmonic complex tones or notes. Listeners must “hear out” simultaneous pitches and “track” pitch sequences over time. Surprisingly few psychoacoustic studies have studied pitch perception under such “natural” conditions of more than one note at a time. Similarly, the neural mechanisms that support these abilities remain poorly understood. Here, we will review psychoacoustical and neurophysiological findings, which concur to suggest an important role for frequency selectivity in the ability of the auditory system to segregate concurrent notes. In particular, the ability to accurately discriminate changes in the fundamental frequency or, subjectively, pitch of a “target” harmonic complex tone in the presence of a “masker” tone occupying the same spectral region seems to covary with the limits peripheral frequency selectivity. Given that harmonic complex tones are an important class of sound for both music and speech and that frequency selectivity is usually adversely affected by cochlear damage, such results may have important implications for the current understanding of auditory scene analysis of concurrent music and speech sounds and of the listening difficulties experienced by hearing‐impaired individuals. [Work supported by NIH Grant R01 DC05216.]

Simultaneous pitches are encoded separately in auditory cortex: an MMNm study

This study examined whether two simultaneous pitches have separate memory representations or an integrated representation in preattentive auditory memory. Mismatch negativity fields were examined when a pitch change occurred in either the higher-pitched or the lower-pitched tone at 25% probability each, thus making the total deviation rate of the two-tone dyad 50%. Clear MMNm was obtained for deviants in both tones confirming separate memory traces for concurrent tones. At the same time, deviants to the lower-pitched, but not higher-pitched, tone within the two-tone dyad elicited a reduced MMNm compared to when each tone was presented alone, indicating that the representations of two pitches are not completely independent.

How we hear music: the relationship between music and the hearing mechanism

Part 1 Preliminaries. Part 2 Aural archaeology. Part 3 Hearing selects intervals. Part 4 The beguiling harmonic theory. Part 5 The imitating voice. Part 6 Hearing simultaneous pitches. Part 7 Patterns in harmony. Part 8 Loudness: the basic dynamic scale. Part 9 Music through the hearing machine. Part 10 A sense of direction. Part 11 Time and rhythm. Part 12 Conclusions.

Pitch identification of simultaneous diotic and dichotic two-tone complexes.

This study examines subjects' ability to recognize the pitches of two missing fundamentals in two simultaneous two-tone complexes whose partials are distributed in various ways between subjects' ears. The data show that identification performance is affected on different levels. Limited frequency resolution in the peripheral auditory system can degrade performance, but only if none of the four stimulus partials is aurally resolved. Identification performance is only weakly dependent on the manner of distributing partials between the ears. In some cases it was found that, probably at a very central level (e.g., attention), the identification processes of both simultaneous pitches interfere with one another. Some subjects are more likely to identify the pitch of one two-tone complex when the harmonic order of the other complex is higher than when this harmonic order is lower. Finally, some subjects tend to hear the complex tones analytically, i.e., perceive pitches of single partials instead of the missing fundamentals for some distribution of partials between the ears.

Pitch of narrow-band signals.

This study investigated the pitch elicited by complex narrow bandwidth signals. These signals ranged from two-component tones to multiple-component approximations of narrow-band noise. All were contained within 10-, 20-, or 50-Hz bandwidths. Listeners were asked to adjust the frequency of a pure tone to match the pitch they heard in a given complex signal. A simple model suggests that the pitch of these complex signals should match that of a pure tone set to the center frequency. For a majority of the signals, the performance of three of our four listeners was not different from the model predictions. However, the two-component signals were apparently resolved and two simultaneous pitches were heard by some listeners. Our fourth listener heard two sequential pitches in many of the complex signals. We are unable to account for that performance although the pitch matches were very repeatable. These results have implications for an understanding of pitch coding and auditory spectral resolution.