Inharmonic Sounds Research Articles

A unitary model of auditory frequency change perception.

Changes in the frequency content of sounds over time are arguably the most basic form of information about the behavior of sound-emitting objects. In perceptual studies, such changes have mostly been investigated separately, as aspects of either pitch or timbre. Here, we propose a unitary account of "up" and "down" subjective judgments of frequency change, based on a model combining auditory correlates of acoustic cues in a sound-specific and listener-specific manner. To do so, we introduce a generalized version of so-called Shepard tones, allowing symmetric manipulations of spectral information on a fine scale, usually associated to pitch (spectral fine structure, SFS), and on a coarse scale, usually associated timbre (spectral envelope, SE). In a series of behavioral experiments, listeners reported "up" or "down" shifts across pairs of generalized Shepard tones that differed in SFS, in SE, or in both. We observed the classic properties of Shepard tones for either SFS or SE shifts: subjective judgements followed the smallest log-frequency change direction, with cases of ambiguity and circularity. Interestingly, when both SFS and SE changes were applied concurrently (synergistically or antagonistically), we observed a trade-off between cues. Listeners were encouraged to report when they perceived "both" directions of change concurrently, but this rarely happened, suggesting a unitary percept. A computational model could accurately fit the behavioral data by combining different cues reflecting frequency changes after auditory filtering. The model revealed that cue weighting depended on the nature of the sound. When presented with harmonic sounds, listeners put more weight on SFS-related cues, whereas inharmonic sounds led to more weight on SE-related cues. Moreover, these stimulus-based factors were modulated by inter-individual differences, revealing variability across listeners in the detailed recipe for "up" and "down" judgments. We argue that frequency changes are tracked perceptually via the adaptive combination of a diverse set of cues, in a manner that is in fact similar to the derivation of other basic auditory dimensions such as spatial location.

Relative pitch representations and invariance to timbre

Information in speech and music is often conveyed through changes in fundamental frequency (f0), perceived by humans as “relative pitch”. Relative pitch judgments are complicated by two facts. First, sounds can simultaneously vary in timbre due to filtering imposed by a vocal tract or instrument body. Second, relative pitch can be extracted in two ways: by measuring changes in constituent frequency components from one sound to another, or by estimating the f0 of each sound and comparing the estimates. We examined the effects of timbral differences on relative pitch judgments, and whether any invariance to timbre depends on whether judgments are based on constituent frequencies or their f0. Listeners performed up/down and interval discrimination tasks with pairs of spoken vowels, instrument notes, or synthetic tones, synthesized to be either harmonic or inharmonic. Inharmonic sounds lack a well-defined f0, such that relative pitch must be extracted from changes in individual frequencies. Pitch judgments were less accurate when vowels/instruments were different compared to when they were the same, and were biased by the associated timbre differences. However, this bias was similar for harmonic and inharmonic sounds, and was observed even in conditions where judgments of harmonic sounds were based on f0 representations. Relative pitch judgments are thus not invariant to timbre, even when timbral variation is naturalistic, and when such judgments are based on representations of f0.

Enhanced mismatch negativity in harmonic compared with inharmonic sounds.

Many natural sounds have frequency spectra composed of integer multiples of a fundamental frequency. This property, known as harmonicity, plays an important role in auditory information processing. However, the extent to which harmonicity influences the processing of sound features beyond pitch is still unclear. This is interesting because harmonic sounds have lower information entropy than inharmonic sounds. According to predictive processing accounts of perception, this property could produce more salient neural responses due to the brain's weighting of sensory signals according to their uncertainty. In the present study, we used electroencephalography to investigate brain responses to harmonic and inharmonic sounds commonly occurring in music: Piano tones and hi‐hat cymbal sounds. In a multifeature oddball paradigm, we measured mismatch negativity (MMN) and P3a responses to timbre, intensity, and location deviants in listeners with and without congenital amusia—an impairment of pitch processing. As hypothesized, we observed larger amplitudes and earlier latencies (for both MMN and P3a) in harmonic compared with inharmonic sounds. These harmonicity effects were modulated by sound feature. Moreover, the difference in P3a latency between harmonic and inharmonic sounds was larger for controls than amusics. We propose an explanation of these results based on predictive coding and discuss the relationship between harmonicity, information entropy, and precision weighting of prediction errors.

Can Haptic Stimulation Enhance Music Perception in Hearing-Impaired Listeners?

Cochlear implants (CIs) have been remarkably successful at restoring hearing in severely-to-profoundly hearing-impaired individuals. However, users often struggle to deconstruct complex auditory scenes with multiple simultaneous sounds, which can result in reduced music enjoyment and impaired speech understanding in background noise. Hearing aid users often have similar issues, though these are typically less acute. Several recent studies have shown that haptic stimulation can enhance CI listening by giving access to sound features that are poorly transmitted through the electrical CI signal. This “electro-haptic stimulation” improves melody recognition and pitch discrimination, as well as speech-in-noise performance and sound localization. The success of this approach suggests it could also enhance auditory perception in hearing-aid users and other hearing-impaired listeners. This review focuses on the use of haptic stimulation to enhance music perception in hearing-impaired listeners. Music is prevalent throughout everyday life, being critical to media such as film and video games, and often being central to events such as weddings and funerals. It represents the biggest challenge for signal processing, as it is typically an extremely complex acoustic signal, containing multiple simultaneous harmonic and inharmonic sounds. Signal-processing approaches developed for enhancing music perception could therefore have significant utility for other key issues faced by hearing-impaired listeners, such as understanding speech in noisy environments. This review first discusses the limits of music perception in hearing-impaired listeners and the limits of the tactile system. It then discusses the evidence around integration of audio and haptic stimulation in the brain. Next, the features, suitability, and success of current haptic devices for enhancing music perception are reviewed, as well as the signal-processing approaches that could be deployed in future haptic devices. Finally, the cutting-edge technologies that could be exploited for enhancing music perception with haptics are discussed. These include the latest micro motor and driver technology, low-power wireless technology, machine learning, big data, and cloud computing. New approaches for enhancing music perception in hearing-impaired listeners could substantially improve quality of life. Furthermore, effective haptic techniques for providing complex sound information could offer a non-invasive, affordable means for enhancing listening more broadly in hearing-impaired individuals.

Inhibitory gating of coincidence-dependent sensory binding in secondary auditory cortex

Integration of multi-frequency sounds into a unified perceptual object is critical for recognizing syllables in speech. This “feature binding” relies on the precise synchrony of each component’s onset timing, but little is known regarding its neural correlates. We find that multi-frequency sounds prevalent in vocalizations, specifically harmonics, preferentially activate the mouse secondary auditory cortex (A2), whose response deteriorates with shifts in component onset timings. The temporal window for harmonics integration in A2 was broadened by inactivation of somatostatin-expressing interneurons (SOM cells), but not parvalbumin-expressing interneurons (PV cells). Importantly, A2 has functionally connected subnetworks of neurons preferentially encoding harmonic over inharmonic sounds. These subnetworks are stable across days and exist prior to experimental harmonics exposure, suggesting their formation during development. Furthermore, A2 inactivation impairs performance in a discrimination task for coincident harmonics. Together, we propose A2 as a locus for multi-frequency integration, which may form the circuit basis for vocal processing.

Time-dependent discrimination advantages for harmonic sounds suggest efficient coding for memory

Perceptual systems have finite memory resources and must store incoming signals in compressed formats. To explore whether representations of a sound's pitch might derive from this need for compression, we compared discrimination of harmonic and inharmonic sounds across delays. In contrast to inharmonic spectra, harmonic spectra can be summarized, and thus compressed, using their fundamental frequency (f0). Participants heard two sounds and judged which was higher. Despite being comparable for sounds presented back-to-back, discrimination was better for harmonic than inharmonic stimuli when sounds were separated in time, implicating memory representations unique to harmonic sounds. Patterns of individual differences (correlations between thresholds in different conditions) indicated that listeners use different representations depending on the time delay between sounds, directly comparing the spectra of temporally adjacent sounds, but transitioning to comparing f0s across delays. The need to store sound in memory appears to determine reliance on f0-based pitch and may explain its importance in music, in which listeners must extract relationships between notes separated in time.

Spectral Pitch Similarity is a Predictor of Perceived Change in Sound- as Well as Note-Based Music

Spectral pitch similarity (SPS) is a measure of the similarity between spectra of any pair of sounds. It has proved powerful in predicting perceived stability and fit of notes and chords in various tonal and microtonal instrumental contexts, that is, with discrete tones whose spectra are harmonic or close to harmonic. Here we assess the possible contribution of SPS to listeners’ continuous perceptions of change in music with fewer discrete events and with noisy or profoundly inharmonic sounds, such as electroacoustic music. Previous studies have shown that time series of perception of change in a range of music can be reasonably represented by time series models, whose predictors comprise autoregression together with series representing acoustic intensity and, usually, the timbral parameter spectral flatness. Here, we study possible roles for SPS in such models of continuous perceptions of change in a range of both instrumental (note-based) and sound-based music (generally containing more noise and fewer discrete events). In the first analysis, perceived change in three pieces of electroacoustic and one of piano music is modeled, to assess the possible contribution of (de-noised) SPS in cooperation with acoustic intensity and spectral flatness series. In the second analysis, a broad range of nine pieces is studied in relation to the wider range of distinctive spectral predictors useful in previous perceptual work, together with intensity and SPS. The second analysis uses cross-sectional (mixed-effects) time series analysis to take advantage of all the individual response series in the dataset, and to assess the possible generality of a predictive role for SPS. SPS proves to be a useful feature, making a predictive contribution distinct from other spectral parameters. Because SPS is a psychoacoustic “bottom up” feature, it may have wide applicability across both the familiar and the unfamiliar in the music to which we are exposed.

Electric and acoustic harmonic integration predicts speech-in-noise performance in hybrid cochlear implant users

BackgroundPitch perception of complex tones relies on place or temporal fine structure-based mechanisms from resolved harmonics and the temporal envelope of unresolved harmonics. Combining this information is essential for speech-in-noise performance, as it allows segregation of a target speaker from background noise. In hybrid cochlear implant (H-CI) users, low frequency acoustic hearing should provide pitch from resolved harmonics while high frequency electric hearing should provide temporal envelope pitch from unresolved harmonics. How the acoustic and electric auditory inputs interact for H-CI users is largely unknown. Harmonicity and inharmonicity are emergent features of sound in which overtones are concordant or discordant with the fundamental frequency. We hypothesized that some H-CI users would be able to integrate acoustic and electric information for complex tone pitch perception, and that this ability would be correlated with speech-in-noise performance. In this study, we used perception of inharmonicity to demonstrate this integration. MethodsFifteen H-CI users with only acoustic hearing below 500 Hz, only electric hearing above 2 kHz, and more than 6 months CI experience, along with eighteen normal hearing (NH) controls, were presented with harmonic and inharmonic sounds. The stimulus was created with a low frequency component, corresponding with the H-CI user's acoustic hearing (fundamental frequency between 125 and 174 Hz), and a high frequency component, corresponding with electric hearing. Subjects were asked to identify the more inharmonic sound, which requires the perceptual integration of the low and high components. Speech-in-noise performance was tested in both groups using the California Consonant Test (CCT), and perception of Consonant-Nucleus-Consonant (CNC) words in quiet and AzBio sentences in noise were tested for the H-CI users. ResultsEight of the H-CI subjects (53%), and all of the NH subjects, scored significantly above chance level for at least one subset of the inharmonicity detection task. Inharmonicity detection ability, but not age or pure tone average, predicted speech scores in a linear model. These results were significantly correlated with speech scores in both quiet and noise for H-CI users, but not with speech in noise performance for NH listeners. Musical experience predicted inharmonicity detection ability, but did not predict speech performance. ConclusionsWe demonstrate integration of acoustic and electric information in H-CI users for complex pitch sensation. The correlation with speech scores in H-CI users might be associated with the ability to segregate a target speaker from background noise using the speaker's fundamental frequency.

Multiple mechanisms in pitch perception revealed by individual differences

Pitch perception is traditionally assumed to involve the estimation of fundamental frequency (F0) from sound. Some pitch-related tasks, such as those involving musical intervals, are more difficult with inharmonic sounds (lacking a clear F0). Other tasks, such as up-down pitch discrimination, are unaffected by inharmonicity, suggesting that listeners track the spectrum, not the F0. To test whether these effects reflect distinct mechanisms, we measured individual differences among large sets of participants on a battery of tasks. Performance on tasks using a common mechanism should be correlated across participants. Two tone up-down discrimination for wideband harmonic tones was highly correlated with performance for wideband inharmonic tones. To isolate F0-based pitch, we also measured discrimination for pairs of harmonic tones with non-overlapping sets of harmonics. Harmonic and inharmonic discrimination did not strongly correlate with this latter task. These results suggest that the same mechanism subse...

Diversity in pitch perception revealed by task dependence.

Pitch conveys critical information in speech, music, and other natural sounds, and is conventionally defined as the perceptual correlate of a sound's fundamental frequency (F0). Although pitch is widely assumed to be subserved by a single F0 estimation process, real-world pitch tasks vary enormously, raising the possibility of underlying mechanistic diversity. To probe pitch mechanisms we conducted a battery of pitch-related music and speech tasks using conventional harmonic sounds and inharmonic sounds whose frequencies lack a common F0. Some pitch-related abilities – those relying on musical interval or voice recognition – were strongly impaired by inharmonicity, suggesting a reliance on F0. However, other tasks, including those dependent on pitch contours in speech and music, were unaffected by inharmonicity, suggesting a mechanism that tracks the frequency spectrum rather than the F0. The results suggest that pitch perception is mediated by several different mechanisms, only some of which conform to traditional notions of pitch.

Harmonic template neurons in primate auditory cortex underlying complex sound processing

Harmonicity is a fundamental element of music, speech, and animal vocalizations. How the auditory system extracts harmonic structures embedded in complex sounds and uses them to form a coherent unitary entity is not fully understood. Despite the prevalence of sounds rich in harmonic structures in our everyday hearing environment, it has remained largely unknown what neural mechanisms are used by the primate auditory cortex to extract these biologically important acoustic structures. In this study, we discovered a unique class of harmonic template neurons in the core region of auditory cortex of a highly vocal New World primate, the common marmoset (Callithrix jacchus), across the entire hearing frequency range. Marmosets have a rich vocal repertoire and a similar hearing range to that of humans. Responses of these neurons show nonlinear facilitation to harmonic complex sounds over inharmonic sounds, selectivity for particular harmonic structures beyond two-tone combinations, and sensitivity to harmonic number and spectral regularity. Our findings suggest that the harmonic template neurons in auditory cortex may play an important role in processing sounds with harmonic structures, such as animal vocalizations, human speech, and music.

The role of physics in inner-ear physiology and auditory perception

Auditory analysis of acoustical stimuli has mainly been connected with Fourier analysis. This touches the basic link between auditory perception and the operation of the hearing organs, namely, physics and mathematics. This relation has amply been demonstrated by the work and ideas of Georg von Békésy. Later, the healthy cochlea (inner ear) was found to contain amplifying elements that boost frequency selectivity. Associated with this there is a pronounced nonlinearity. Furthermore, the ear does not only absorb and process sounds, it also emits sound waves. Mathematical models of all these processes must contain subsets serving them all, a goal that has not yet been reached. On the contrary, recent findings on the movements of the various membranes and cells in the organ of Corti have increased the difficulties of mathematical modeling. Additional subjects to be covered in the lecture are (1) in psychophysics of hearing: pitch perception, inharmonic sounds, critical bands, hearing of patients with hearing loss, and auditory revalidation, and (2) in auditory neuroscience: recording of single fibers of the auditory nerve, reverse correlation, inverse solutions of cochlear mechanics, nonlinear analysis, and optical coherence tomography (OCT).

Enhancing a tone by shifting its frequency or intensity

When a test sound consisting of pure tones with equal intensities is preceded by a precursor sound identical to the test sound except for a reduction in the intensity of one tone, an auditory "enhancement" phenomenon occurs: In the test sound, the tone which was previously softer stands out perceptually. Here, enhancement was investigated using inharmonic sounds made up of five pure tones well resolved in the auditory periphery. It was found that enhancement can be elicited not only by increases in intensity but also by shifts in frequency. In both cases, when the precursor and test sounds are separated by a 500-ms delay, inserting a burst of pink noise during the delay has little effect on enhancement. Presenting the precursor and test sounds to opposite ears rather than to the same ear significantly reduces the enhancement resulting from increases in intensity, but not the enhancement resulting from shifts in frequency. This difference suggests that the mechanisms of enhancement are not identical for the two types of change. For frequency shifts, enhancement may be partly based on the existence of automatic "frequency-shift detectors" [Demany and Ramos, J. Acoust. Soc. Am. 117, 833-841 (2005)].

Khalid Naji

<h3>Abstract</h3> Many natural sounds have frequency spectra composed of integer multiples of a fundamental frequency. This property, known as harmonicity, plays an important role in auditory information processing. However, the...

Ambiguous Pitch and the Temporal Representation of Inharmonic Iterated Rippled Noise in the Ventral Cochlear Nucleus

Neural coding of the pitch of complex sounds is vital for animals' ability to communicate and to perceptually organize natural acoustic scenes. Harmonic complex sounds typically have a well defined pitch corresponding to their fundamental frequency, whereas inharmonic sounds can exhibit pitch ambiguity: their pitch can have more than one value. Iterated rippled noise (IRN), a common "pitch stimulus," is generated from broadband noise by a cascade of delay-and-add steps, with the delayed noise phase-shifted by varphi degrees. By varying varphi, the (in)harmonicity, and therefore the pitch ambiguity, of IRN can be manipulated. Recordings were made from single-units in the ventral cochlear nucleus of anesthetized guinea pigs in response to IRN and complex tones, systematically varying the inharmonicity. In their all-order interspike interval distributions, primary-like and chopper units tuned within the phase-locking range of best frequencies represent the waveform temporal fine structure (which varies with varphi). In contrast, those units tuned to higher frequencies represent the temporal-envelope modulation (independent of varphi). We show a temporal representation of ambiguous pitch for IRN and complex tones based on responses to the stimulus fine structure. Within the dominance region for pitch this representation follows the predictions of classic human behavioral experiments and provides a unifying contribution to possible neuro-temporal explanations for the pitch shift and pitch ambiguity associated with many inharmonic sounds.

Broadband Auditory Stream Segregation by Hearing-Impaired and Normal-Hearing Listeners

To investigate the effects of hearing loss on auditory stream segregation of broadband inharmonic sounds. Auditory stream segregation by listeners with normal and impaired hearing was measured for 6-component inharmonic sounds ("A" and "B") using objective and subjective methods. Components in the A stimuli ranged between 1000 and 4000 Hz, whereas B stimuli were generated at the same frequency ratio but scaled upward in frequency relative to the A stimuli. In Experiment 1, streaming was measured by having listeners detect a delay inserted into a sequence of A and B stimuli (A_B_A_B_...) for B stimuli with different frequencies. In Experiment 2, streaming was measured using an ABA_ABA_... sequence, and the frequency of the B stimulus decreased until listeners reported that they could "no longer hear two separate streams." Experiment 1 indicated no significant differences between groups in the size of the just detectable delay and no significant interactions between group and the scaling factor between the B and A stimuli. Experiment 2 revealed no significant differences in streaming abilities between normal-hearing and hearing-impaired groups. Overall, results indicate that listeners with normal and impaired hearing have similar auditory streaming abilities for broadband inharmonic complex stimuli.

Inharmonic sounds of bowed strings in Western music and Beijing opera

Due to the nonlinear bow‐string interaction, inharmonic sounds can be produced by bowed strings. Such sounds are characterized by subharmonics, sidebands, and noises. Bow location, bow speed, and bow pressure on the string have been found critical to producing inharmonic sounds on the violin or the cello. In the accompaniment music of Beijing opera singing, inharmonic sounds produced by the major bowed‐string instrument (the jinghu) are sometimes called ‘‘blossoming tones.’’ They are played with a special touch technique of the left hand, and a chaotic behavior of string vibration arises from a nonlinear boundary condition with unilateral constraints. The spectra of blossoming tones can be viewed as high‐pass noises, with the cutoff frequency of about 1.6 kHz. Blossoming tones are sophisticatedly used for emotional expressions in Beijing opera. Their harsh quality recalls some pathological human voices.

Wavelets in Recognition of Bird Sounds

This paper presents a novel method to recognize inharmonic and transient bird sounds efficiently. The recognition algorithm consists of feature extraction using wavelet decomposition and recognition using either supervised or unsupervised classifier. The proposed method was tested on sounds of eight bird species of which five species have inharmonic sounds and three reference species have harmonic sounds. Inharmonic sounds are not well matched to the conventional spectral analysis methods, because the spectral domain does not include any visible trajectories that computer can track and identify. Thus, the wavelet analysis was selected due to its ability to preserve both frequency and temporal information, and its ability to analyze signals which contain discontinuities and sharp spikes. The shift invariant feature vectors calculated from the wavelet coefficients were used as inputs of two neural networks: the unsupervised self-organizing map (SOM) and the supervised multilayer perceptron (MLP). The results were encouraging: the SOM network recognized 78% and the MLP network 96% of the test sounds correctly.

Tunable dispersion filter design for piano synthesis

The tunable dispersion filter is a new design approach presented in this letter to provide dispersion modeling for digital waveguide synthesis of musical instruments, which do not produce extremely inharmonic sounds, such as the piano. We propose to use a cascade of second-order allpass filters for modeling dispersion. The filter coefficients can be calculated by using simple formulae based on the Thiran allpass filter design, which is usually used for fractional delay approximation. Unlike the previous allpass filter approximations, this filter design is easily scalable to produce various inharmonicity values for a wide range of fundamental frequencies.

Inharmonic Sounds: Implications as to «Pitch», «Timbre» and «Consonance»

This paper relates the structure of certain types of inharmonic complex sounds such as produced by idiophones (gongs, bells, etc.) to perceptual observations as well as to models of pitch and timbre perception. Since most of the current models operate in the time domain focussing on the concept of periodicity, it is argued that perception of pitch of inharmonic sounds, in general, is more ambiguous because (quasi-)periodicity in many such signals is more difficult to detect. Periodicity typically deteriorates with increasing inharmonicity of spectral components while it is maximum in perfectly harmonic spectra. Thereby a clear relationship between the temporal structure of a signal as reflected by its time function and wave-shape, and its spectral composition can be established that is of relevance also as to perception of timbre and consonance.