Models Of Pitch Perception Research Articles

Pitch shifts and peak selection in autocorrelation models of pitch perception

The standard autocorrelation (AC) model of pitch perception is capable of predicting a wide range of pitch phenomena, but there are a number of pitch-shift effects that it fails to predict. One of these is the pitch shift associated with a single mistuned harmonic, another noise edge pitch (NEP). A recent extension of the model by Hartmann et al. [J. Acoust. Soc. Am. 145, 1993–2008 (2019)] employs a method that considers multiple peaks in the AC function, enabling the prediction of NEP. This extension, however, reduces the ability of the model to predict mistuned harmonic pitch. It can be shown that an additional extension to the AC model, which modifies the peak selection process, is capable of predicting both types of pitch shift. Specifically, a delay time is assigned to a peak of the AC function in a manner that takes into account the entire shape of the bump surrounding a local maximum. The end result is closely related to a template fitting procedure in the frequency domain, but it is suggested that such a procedure could correspond to processes in the time domain. The neuronal dynamics that support synchrony enhancement are suggested as a possible mechanism.

How to pick a peak: Pitch and peak shifting in temporal models of pitch perception.

The standard autocorrelation model of pitch perception posits that the pitch of a stimulus can be predicted from the first major peak of a summary autocorrelation function (SACF) after the zero-delay peak. Models based on this theory are capable of predicting a wide range of pitch phenomena. There are, however, a number of cases where the approach fails. Two examples are noise edge pitch (NEP) and the pitch induced by the mistuning of a single component of an otherwise harmonic stimulus. Hartmann, Cariani, and Colburn [(2019). J. Acoust. Soc. Am. 145, 1993-2008] recently proposed the use of multiple SACF peaks in the estimation process. This enables prediction of the NEP but suppresses the shift associated with a mistuned harmonic. A functional model is developed that can predict both of these pitch phenomena. The multiple-peak framework is extended with a non-standard peak-selection method that associates a delay time to a given peak in a manner that takes into account the entire shape of the bump surrounding the peak. This effectively shifts the peak location slightly for non-harmonic stimuli. A possible physiological mechanism that could induce such peak shifting is discussed, and the model is tested against existing psychophysical data.

Place and Temporal Cues in Cochlear Implant Pitch and Melody Perception.

The present study compared pitch and melody perception using cochlear place of excitation and temporal cues in six adult nucleus cochlear implant (CI) recipients. The stimuli were synthesized tones presented through a loudspeaker, and recipients used the Advanced Combinational Encoder (ACE) sound coding strategy on their own sound processors. Three types of tones were used, denoted H3, H4, and P5. H3 tones were harmonic tones with fundamental frequencies in the range C3–C4 (131–262 Hz), providing temporal pitch cues alone. H4 tones were harmonic tones with fundamental frequencies in the range C4–C5 (262–523 Hz), providing a mixture of temporal and place cues. P5 tones were pure tones with fundamental frequencies in the range C5–C6 (523–1046 Hz), providing place pitch cues alone. Four experimental procedures were used: pitch discrimination, pitch ranking, backward modified melodies, and warped modified melodies. In each trial of the modified melodies tests, subjects heard a familiar melody and a version with modified pitch (in randomized order), and had to select the unmodified melody. In all four procedures, many scores were much lower than would be expected for normal hearing listeners, implying that the strength of the perceived pitch was weak. Discrimination and ranking with H3 and P5 tones was poor for two-semitone intervals, but near perfect for intervals of five semitones and larger. H4 tones provided the lowest group mean scores in all four procedures, with some pitch reversals observed in pitch ranking. Group mean scores for P5 tones (place cues alone) were at least as high as those for H3 tones (temporal cues alone). The relatively good scores on the melody tasks with P5 tones were surprising, given the lack of temporal cues, raising the possibility of musical pitch using place cues alone. However, the alternative possibility that the CI recipients perceived the place cues as brightness, rather than musical pitch per se, cannot be excluded. These findings show that pitch perception models need to incorporate neural place representations alongside temporal cues if they are to predict pitch and melody perception in the absence of temporal cues.

Noise edge pitch and models of pitch perception.

Monaural noise edge pitch (NEP) is evoked by a broadband noise with a sharp falling edge in the power spectrum. The pitch is heard near the spectral edge frequency but shifted slightly into the frequency region of the noise. Thus, the pitch of a lowpass (LP) noise is matched by a pure tone typically 2%–5% below the edge, whereas the pitch of highpass (HP) noise is matched a comparable amount above the edge. Musically trained listeners can recognize musical intervals between NEPs. The pitches can be understood from a temporal pattern-matching model of pitch perception based on the peaks of a simplified autocorrelation function. The pitch shifts arise from limits on the autocorrelation window duration. An alternative place-theory approach explains the pitch shifts as the result of lateral inhibition. Psychophysical experiments using edge frequencies of 100 Hz and below find that LP-noise pitches exist but HP-noise pitches do not. The result is consistent with a temporal analysis in tonotopic regions outside the noise band. LP and HP experiments with high-frequency edges find that pitch tends to disappear as the edge frequency approaches 5000 Hz, as expected from a timing theory, though exceptional listeners can go an octave higher.

Monaural edge pitch and models of pitch perception

Noise with a sharp spectral edge produces a definite, perceived pitch. The pitch of lowpass noise lies slightly below the edge and the pitch of highpass noise slightly above it. The pitch shifts away from the edge, expressed in semitones, increase with the decreasing edge frequency. A neural timing model based on positive peaks of the autocorrelation function accounts for the broad features of the shifts. So, does a place model based on the peak of an excitation pattern as enhanced by lateral inhibition. As the edge frequency decreases below 150 Hz, the pitch persists for a lowpass noise but disappears for highpass noise. This result is consistent with the timing model but not with the place model. For high edge frequencies, the pitch is stronger for highpass noise than for lowpass noise—consistent with both timing and place models. As the edge frequency approaches 5000 Hz, the pitch disappears for most listeners, but for some listeners, a pitch persists for edge frequencies beyond 8000 Hz. The latter res...

Autocorrelation models of pitch perception: In memory of ray meddis

About 60 years ago, Licklider (1951, Experientia) described a qualitative model of autocorrelation that appeared to explain complex pitch perception. However, it was almost 40 years later before, Slaney and Lyon (1990, ICAASP) and Meddis and Hewitt (1991, JASA) incorporated autocorrelation into computational auditory models to test Licklider’s model of pitch perception. The Meddis and Hewitt frontend combined a level-dependent gammatone filterbank with Meddis’ (1986, JASA) inner hair-cell model. The “autocorrelograms” it produced revealed the temporal regularity of simulated auditory-nerve interspike intervals that occur in response to complex stimuli. The model was/is remarkably successful in accounting for a large set of complex pitch perception data. The model emphasizes the temporal structure of the neural information produced by a complex sound, in contrast to models that use a simple spectral representation of the resolved harmonics of a complex sound. Today, those who model complex pitch perception either use an autocorrelation-like approach or argue why such an approach does not work. The Meddis-Hewitt model is almost always considered in these modeling efforts. This presentation will describe the Meddis-Hewitt model and discuss its lasting effect on the study of complex pitch perception. [Work supported by NIDCD and Facebook Reality Labs grants to WAY.]

Multiple f0 pitch estimation for musical applications using dynamic Bayesian networks and learned priors

The identification of multiple simultaneous pitches is a challenging signal processing task and cannot at present be performed as well as trained human subjects. Moreover, it appears that successful human performance depends on skill acquisition and knowledge of musical conventions. Even human capabilities are likely fairly poor in the absence of training and musical context. We present a framework, using Dynamic Bayesian Networks, that permits the principled incorporation of models of music theory, musical instruments, and human pitch perception. A particular advantage of this approach is the ability to develop each of these models independently, relying either on expert knowledge or machine learning as necessary. Models of appropriate complexity can then be selected for a specific application. In the present work, we focus on the use of learned models of musical context, specifically Deep Markov Models, and use these to improve inferences about simultaneous pitches. The main drawback of this framework is the intractability of the inference computations and the computational expense of approximation methods. We explore particle filtering as an approach to addressing these problems with the ultimate aim of making a system useable in a musical performance system. [Work supported by NSF BCS-1539276 and IBM AIRC grant.]

Spectral and temporal models of human pitch perception with mixtures of three concurrent harmonic complexes

In music and other everyday situations, humans are often presented with three or more simultaneous pitches, each carried by a harmonic complex tone, but few empirical or theoretical studies have addressed how pitch is perceived in this context. In three behavioral experiments, mixtures of three concurrent complexes were filtered into a single bandpass spectral region, and the relationship between the fundamental frequencies and spectral region was varied in order to manipulate the extent to which harmonics were resolved either before or after mixing. Listeners were asked to discriminate major from minor chords (Experiment 1) or to compare the pitch of a probe tone to that of a target embedded in the mixture (Experiments 2 and 3). In all three experiments, listeners performed above chance even under conditions where traditional rate-place models would not predict individually resolved components. Human behavioral results were compared to predictions from two classes of pitch model: a rate-place model using harmonic template matching and a temporal model using summary autocorrelation. Predictions from a combined model, using both rate-place and temporal information, were more accurate than predictions from either model alone, suggesting that humans may integrate these two kinds of information. [Work supported by NIH grant R01DC005216.]

Modeling Pitch Perception With an Active Auditory Model Extended by Octopus Cells.

Pitch is an essential category for musical sensations. Models of pitch perception are vividly discussed up to date. Most of them rely on definitions of mathematical methods in the spectral or temporal domain. Our proposed pitch perception model is composed of an active auditory model extended by octopus cells. The active auditory model is the same as used in the Stimulation based on Auditory Modeling (SAM), a successful cochlear implant sound processing strategy extended here by modeling the functional behavior of the octopus cells in the ventral cochlear nucleus and by modeling their connections to the auditory nerve fibers (ANFs). The neurophysiological parameterization of the extended model is fully described in the time domain. The model is based on latency-phase en- and decoding as octopus cells are latency-phase rectifiers in their local receptive fields. Pitch is ubiquitously represented by cascaded firing sweeps of octopus cells. Based on the firing patterns of octopus cells, inter-spike interval histograms can be aggregated, in which the place of the global maximum is assumed to encode the pitch.

An integrated model of pitch perception incorporating place and temporal pitch codes with application to cochlear implant research

Although the neural mechanisms underlying pitch perception are not yet fully understood, there is general agreement that place and temporal representations of pitch are both used by the auditory system. This paper describes a neural network model of pitch perception that integrates both codes of pitch and explores the contributions of, and the interactions between, the two representations in simulated pitch ranking trials in normal and cochlear implant hearing. The model can replicate various psychophysical observations including the perception of the missing fundamental pitch and sensitivity to pitch interval sizes. As a case study, the model was used to investigate the efficiency of pitch perception cues in a novel sound processing scheme, Stimulation based on Auditory Modelling (SAM), that aims to improve pitch perception in cochlear implant hearing. Results showed that enhancement of the pitch perception cues would lead to better pitch ranking scores in the integrated model only if the place and temporal pitch cues were consistent.

Binaural Diplacusis and Its Relationship with Hearing-Threshold Asymmetry.

Binaural pitch diplacusis refers to a perceptual anomaly whereby the same sound is perceived as having a different pitch depending on whether it is presented in the left or the right ear. Results in the literature suggest that this phenomenon is more prevalent, and larger, in individuals with asymmetric hearing loss than in individuals with symmetric hearing. However, because studies devoted to this effect have thus far involved small samples, the prevalence of the effect, and its relationship with interaural asymmetries in hearing thresholds, remain unclear. In this study, psychometric functions for interaural pitch comparisons were measured in 55 subjects, including 12 normal-hearing and 43 hearing-impaired participants. Statistically significant pitch differences between the left and right ears were observed in normal-hearing participants, but the effect was usually small (less than 1.5/16 octave, or about 7%). For the hearing-impaired participants, statistically significant interaural pitch differences were found in about three-quarters of the cases. Moreover, for about half of these participants, the difference exceeded 1.5/16 octaves and, in some participants, was as large as or larger than 1/4 octave. This was the case even for the lowest frequency tested, 500 Hz. The pitch differences were weakly, but significantly, correlated with the difference in hearing thresholds between the two ears, such that larger threshold asymmetries were statistically associated with larger pitch differences. For the vast majority of the hearing-impaired participants, the direction of the pitch differences was such that pitch was perceived as higher on the side with the higher (i.e., ‘worse’) hearing thresholds than on the opposite side. These findings are difficult to reconcile with purely temporal models of pitch perception, but may be accounted for by place-based or spectrotemporal models.

Insights on the Neuromagnetic Representation of Temporal Asymmetry in Human Auditory Cortex.

Communication sounds are typically asymmetric in time and human listeners are highly sensitive to this short-term temporal asymmetry. Nevertheless, causal neurophysiological correlates of auditory perceptual asymmetry remain largely elusive to our current analyses and models. Auditory modelling and animal electrophysiological recordings suggest that perceptual asymmetry results from the presence of multiple time scales of temporal integration, central to the auditory periphery. To test this hypothesis we recorded auditory evoked fields (AEF) elicited by asymmetric sounds in humans. We found a strong correlation between perceived tonal salience of ramped and damped sinusoids and the AEFs, as quantified by the amplitude of the N100m dynamics. The N100m amplitude increased with stimulus half-life time, showing a maximum difference between the ramped and damped stimulus for a modulation half-life time of 4 ms which is greatly reduced at 0.5 ms and 32 ms. This behaviour of the N100m closely parallels psychophysical data in a manner that: i) longer half-life times are associated with a stronger tonal percept, and ii) perceptual differences between damped and ramped are maximal at 4 ms half-life time. Interestingly, differences in evoked fields were significantly stronger in the right hemisphere, indicating some degree of hemispheric specialisation. Furthermore, the N100m magnitude was successfully explained by a pitch perception model using multiple scales of temporal integration of auditory nerve activity patterns. This striking correlation between AEFs, perception, and model predictions suggests that the physiological mechanisms involved in the processing of pitch evoked by temporal asymmetric sounds are reflected in the N100m.

Learning Pitch with STDP: A Computational Model of Place and Temporal Pitch Perception Using Spiking Neural Networks.

Pitch perception is important for understanding speech prosody, music perception, recognizing tones in tonal languages, and perceiving speech in noisy environments. The two principal pitch perception theories consider the place of maximum neural excitation along the auditory nerve and the temporal pattern of the auditory neurons’ action potentials (spikes) as pitch cues. This paper describes a biophysical mechanism by which fine-structure temporal information can be extracted from the spikes generated at the auditory periphery. Deriving meaningful pitch-related information from spike times requires neural structures specialized in capturing synchronous or correlated activity from amongst neural events. The emergence of such pitch-processing neural mechanisms is described through a computational model of auditory processing. Simulation results show that a correlation-based, unsupervised, spike-based form of Hebbian learning can explain the development of neural structures required for recognizing the pitch of simple and complex tones, with or without the fundamental frequency. The temporal code is robust to variations in the spectral shape of the signal and thus can explain the phenomenon of pitch constancy.

The influence of sung vowels on pitch perception

Phonologists have demonstrated that in speech there is an intrinsic pitch of vowels, i.e., that different pitch heights are perceived for different spoken vowels with the same mean fundamental frequency. Only limited work has been done on the impact of sung vowels on perceived pitch and previous studies, using forced choice paradigms, have found conflicting results comparing Front/Close to Back/Open vowels: Ternstrom, Sundberg, and Collden (1988), using manipulated sung tones, found that Front/Close vowels were perceived as higher than other vowels while Fowler and Brown (1997), using synthesized tones, found that Front/Close was perceived lower than Back/Open. The current study uses a method-of-adjustment paradigm to examine pitch perception of all four extremal vowels: Front/Close (/i/), Front/Open (/ae/), Back/Close (/u/), and Back/Open (/ɑ/), using completely controlled, but realistic synthesis. It also assesses whether different models of pitch perception are necessary for the various vowel types. In the experiment, the subjects are asked to match a reference tone that has no formants against a stimulus tone with first and second formants corresponding to a particular vowel. Both the reference and stimulus tones are identical in their synthesis except for the formants and are 300 ms in length.

Effects of harmonic roving on pitch discrimination

Performance in pitch discrimination tasks is limited by variability intrinsic to listeners which may arise from peripheral auditory coding limitations or more central noise sources. The present study aimed at quantifying such “internal noise” by estimating the amount of harmonic roving required to impair pitch discrimination performance. Fundamental-frequency difference limens (F0DLs) were obtained in normal-hearing listeners with and without musical training for complex tones filtered between 1.5 and 3.5 kHz with F0s of 300 Hz (resolved harmonics) and 75 Hz (unresolved harmonics). The harmonicity of the tone complexes was varied by systematically roving the frequency of individual harmonics, which was taken from a Gaussian distribution centered on the nominal frequency in every stimulus presentation. The amount of roving was determined by the standard deviation of this distribution, which varied between 0% and 16% of the tested F0. F0DLs for resolved harmonics remained unaffected for up to 6% roving, and increased thereafter. For unresolved harmonics, performance remained stable up to larger roving values. The results demonstrate a systematic relationship between F0DLs and stimulus variability that could be used to quantify the internal noise and provide strong constraints for physiologically inspired models of pitch perception.

Application of a pitch perception model to investigate the effect of stimulation field spread on the pitch ranking abilities of cochlear implant recipients

Although many cochlear implant (CI) recipients perceive speech very well in favorable conditions, they still have difficulty with music, speech in noisy environments, and tonal languages. Studies show that CI users’ performance in these tasks are correlated with their ability to perceive pitch. The spread of stimulation field from the electrodes to the auditory nerve is one of the factors affecting performance. This study proposes a model of auditory perception to predict the performance of CI users in pitch ranking tasks using an existing sound processing scheme. The model is then used as a platform to investigate the effect of stimulation field spread on performance.

The emotional connotations of major versus minor tonality: One or more origins?

The association between major/minor tonality and positive/negative emotional valence is psychologically robust, but without a single accepted explanation. I compare six partially related theories. Dissonance: On average, passages in minor keys are more dissonant because, on average, the minor triad is more dissonant (rougher, less harmonic) or because tonal structure is more complex. Alterity and markedness: Major triads and scales are more common than minor, and positive valence is more common than negative. Major and positive valence are the norm; minor and negative are marked Others. Uncertainty: The minor triad has a more ambiguous (less salient) root than the major, and the minor scale has more variable form and a more ambiguous (less stable) tonic; uncertainty is associated with anger, sadness, distress, and grief. Speech: By comparison to major triads and scales, minor contain pitch(es) that are lower than expected – just as sad speech is lower than expected. Salience: In diatonic chord progressions, flattened diatonic scale degrees are more salient than sharpened because their harmonics better match the prevailing scale. Scale degrees 3 and 6 are more likely to destabilize tonality in minor than major tonalities. Familiarity: Arbitrary emotional differences between major and minor were reinforced in a historical process of cultural differentiation. For each theory, there are credible arguments and evidence for and against. All theories are broadly consistent with Terhardt’s pattern-recognition model of pitch perception (non-musical perceptual familiarity with the harmonic series), Schenker’s concept of prolongation (specifically, tonal voice leading as a prolongation of the tonic triad), evolutionary explanations of the emotional connotations of alterity, and a psychohistory of tonality in which melody, polyphony, leading tones, and the major–minor system emerged at different times, explicable by different psychological principles.

Real-time implementation of a polyphonic pitch perception model

This algorithm is a real-time implementation of a polyphonic pitch perception model previously described in Braasch et al. [POMA 19, 015027 (2013)]. The model simulates the rate code for pitch detection by taking advantage of phase locking techniques. Complex input tones are processed through a filter bank, and the output of each filter is run through its own separate autocorrelation following the Licklider model. After conversion to polar form, analysis is done on the phase output of the autocorrelation, where the algorithm computes the time delay between localized peaks. This gives the fundamental period of the tone within a given filter; these values are then normalized by the magnitude output of the autocorrelation and combined with output data from the other filters to give full spectral information. The algorithm uses an adjustable running frame window to trade off between frequency resolution and rapid changes in pitch. The model can accurately extract missing or implied fundamental frequencies.

Principal pitch of frequency-modulated tones with asymmetrical modulation waveform: A comparison of models

In this work, the overall perceived pitch (principal pitch) of pure tones modulated in frequency with an asymmetric waveform is studied. The dependence of the principal pitch on the degree of asymmetric modulation was obtained from a psychophysical experiment. The modulation waveform consisted of a flat portion of constant frequency and two linear segments forming a peak. Consistent with previous results, significant pitch shifts with respect to the time-averaged geometric mean were observed. The direction of the shifts was always toward the flat portion of the modulation. The results from the psychophysical experiment, along with those obtained from previously reported studies, were compared with the predictions of six models of pitch perception proposed in the literature. Even though no single model was able to predict accurately the perceived pitch for all experiments, there were two models that give robust predictions that are within the range of acceptable tuning of modulated tones for almost all the cases. Both models point to the existence of an underlying "stability sensitive" mechanism for the computation of pitch that gives more weight to the portion of the stimuli where the frequency is changing more slowly.

Pitch and plasticity: insights from the pitch matching of chords by musicians with absolute and relative pitch.

Absolute pitch (AP) is a form of sound recognition in which musical note names are associated with discrete musical pitch categories. The accuracy of pitch matching by non-AP musicians for chords has recently been shown to depend on stimulus familiarity, pointing to a role of spectral recognition mechanisms in the early stages of pitch processing. Here we show that pitch matching accuracy by AP musicians was also dependent on their familiarity with the chord stimulus. This suggests that the pitch matching abilities of both AP and non-AP musicians for concurrently presented pitches are dependent on initial recognition of the chord. The dual mechanism model of pitch perception previously proposed by the authors suggests that spectral processing associated with sound recognition primes waveform processing to extract stimulus periodicity and refine pitch perception. The findings presented in this paper are consistent with the dual mechanism model of pitch, and in the case of AP musicians, the formation of nominal pitch categories based on both spectral and periodicity information.