Harmonic Cues Research Articles

Long latency auditory evoked potentials and object-related negativity based on harmonicity in hearing-impaired children

Hearing-impaired children (HIC) have difficulty understanding speech in noise, which may be due to difficulty parsing concurrent sound object based on harmonicity cues. Using long latency auditory evoked potentials (LLAEPs) and object-related negativity (ORN), a neural metric of concurrent sound segregation, this study investigated the sensitivity of HIC in processing harmonic relation. The participants were 14 normal-hearing children (NHC) with an average age of 7.82 ± 1.31 years and 17 HIC with an average age of 7.98 ± 1.25 years. They were presented with a sequence of 200 Hz harmonic complex tones that had either all harmonic in tune or the third harmonic mistuned by 2%, 4%, 8%, and 16% of its original value while neuroelectric brain activity was recorded. The analysis of scalp-recorded LLAEPs revealed lower N2 amplitudes elicited by the tuned stimuli in HIC than control. The ORN, isolated in difference wave between LLAEP elicited by tuned and mistuned stimuli, was delayed and smaller in HIC than NHC. This study showed that deficits in processing harmonic relation in HIC, which may contribute to their difficulty in understanding speech in noise. As a result, top-down and bottom-up rehabilitations aiming to improve processing of basic acoustic characteristics, including harmonics are recommended for children with hearing loss.

Young children pause on phrase boundaries in self-paced music listening: The role of harmonic cues.

Proper segmentation of auditory streams is essential for understanding music. Many cues, including meter, melodic contour, and harmony, influence adults' perception of musical phrase boundaries. To date, no studies have examined young children's musical grouping in a production task. We used a musical self-pacing method to investigate (1) whether dwell times index young children's musical phrase grouping and, if so, (2) whether children dwell longer on phrase boundaries defined by harmonic cues specifically. In Experiment 1, we asked 3-year-old children to self-pace through chord progressions from Bach chorales (sequences in which metrical, harmonic, and melodic contour grouping cues aligned) by pressing a computer key to present each chord in the sequence. Participants dwelled longer on chords in the 8th position, which corresponded to phrase endings. In Experiment 2, we tested 3-, 4-, and 7-year-old children's sensitivity to harmonic cues to phrase grouping when metrical regularity cues and melodic contour cues were misaligned with the harmonic phrase boundaries. In this case, 7 and 4 year olds but not 3 year olds dwelled longer on harmonic phrase boundaries, suggesting that the influence of harmonic cues on phrase boundary perception develops substantially between 3 and 4 years of age in Western children. Overall, we show that the musical dwell time method is child-friendly and can be used to investigate various aspects of young children's musical understanding, including phrase grouping and harmonic knowledge. (PsycINFO Database Record

Supplemental Material for Young Children Pause on Phrase Boundaries in Self-Paced Music Listening: The Role of Harmonic Cues

Supplemental Material for Young Children Pause on Phrase Boundaries in Self-Paced Music Listening: The Role of Harmonic Cues

Multisensory object perception in infancy: 4-month-olds perceive a mistuned harmonic as a separate auditory and visual object

Infants learn to use auditory and visual information to organize the sensory world into identifiable objects with particular locations. Here we use a behavioural method to examine infants’ use of harmonicity cues to auditory object perception in a multisensory context. Sounds emitted by different objects sum in the air and the auditory system must figure out which parts of the complex waveform belong to different sources (auditory objects). One important cue to this source separation is that complex tones with pitch typically contain a fundamental frequency and harmonics at integer multiples of the fundamental. Consequently, adults hear a mistuned harmonic in a complex sound as a distinct auditory object (Alain, Theunissen, Chevalier, Batty, & Taylor, 2003). Previous work by our group demonstrated that 4-month-old infants are also sensitive to this cue. They behaviourally discriminate a complex tone with a mistuned harmonic from the same complex with in-tune harmonics, and show an object-related event-related potential (ERP) electrophysiological (EEG) response to the stimulus with mistuned harmonics. In the present study we use an audiovisual procedure to investigate whether infants perceive a complex tone with an 8% mistuned harmonic as emanating from two objects, rather than merely detecting the mistuned cue. We paired in-tune and mistuned complex tones with visual displays that contained either one or two bouncing balls. Four-month-old infants showed surprise at the incongruous pairings, looking longer at the display of two balls when paired with the in-tune complex and at the display of one ball when paired with the mistuned harmonic complex. We conclude that infants use harmonicity as a cue for source separation when integrating auditory and visual information in object perception.

The role of continuous low-frequency harmonicity cues for interrupted speech perception in bimodal hearing.

Low-frequency acoustic cues have been shown to enhance speech perception by cochlear-implant users, particularly when target speech occurs in a competing background. The present study examined the extent to which a continuous representation of low-frequency harmonicity cues contributes to bimodal benefit in simulated bimodal listeners. Experiment 1 examined the benefit of restoring a continuous temporal envelope to the low-frequency ear while the vocoder ear received a temporally interrupted stimulus. Experiment 2 examined the effect of providing continuous harmonicity cues in the low-frequency ear as compared to restoring a continuous temporal envelope in the vocoder ear. Findings indicate that bimodal benefit for temporally interrupted speech increases when continuity is restored to either or both ears. The primary benefit appears to stem from the continuous temporal envelope in the low-frequency region providing additional phonetic cues related to manner and F1 frequency; a secondary contribution is provided by low-frequency harmonicity cues when a continuous representation of the temporal envelope is present in the low-frequency, or both ears. The continuous temporal envelope and harmonicity cues of low-frequency speech are thought to support bimodal benefit by facilitating identification of word and syllable boundaries, and by restoring partial phonetic cues that occur during gaps in the temporally interrupted stimulus.

Listeners lengthen phrase boundaries in self-paced music.

Previous work has shown that musicians tend to slow down as they approach phrase boundaries (phrase-final lengthening). In the present experiments, we used a paradigm from the action perception literature, the dwell time paradigm (Hard, Recchia, & Tversky, 2011), to investigate whether participants engage in phrase boundary lengthening when self-pacing through musical sequences. When participants used a key press to produce each successive chord of Bach chorales, they dwelled longer on boundary chords than nonboundary chords in both the original chorales and atonal manipulations of the chorales. When a novel musical sequence was composed that controlled for metrical and melodic contour cues to boundaries, the dwell time difference between boundaries and nonboundaries was greater in the tonal condition than in the atonal condition. Furthermore, similar results were found for a group of nonmusicians, suggesting that phrase-final lengthening in musical production is not dependent on musical training and can be evoked by harmonic cues.

A function for binaural integration in auditory grouping and segregation in the inferior colliculus.

Responses of neurons to binaural, harmonic complex stimuli in urethane-anesthetized guinea pig inferior colliculus (IC) are reported. To assess the binaural integration of harmonicity cues for sound segregation and grouping, responses were measured to harmonic complexes with different fundamental frequencies presented to each ear. Simultaneously gated harmonic stimuli with fundamental frequencies of 125 Hz and 145 Hz were presented to the left and right ears, respectively, and recordings made from 96 neurons with characteristic frequencies >2 kHz in the central nucleus of the IC. Of these units, 70 responded continuously throughout the stimulus and were excited by the stimulus at the contralateral ear. The stimulus at the ipsilateral ear excited (EE: 14%; 10/70), inhibited (EI: 33%; 23/70), or had no significant effect (EO: 53%; 37/70), defined by the effect on firing rate. The neurons phase locked to the temporal envelope at each ear to varying degrees depending on signal level. Many of the cells (predominantly EO) were dominated by the response to the contralateral stimulus. Another group (predominantly EI) synchronized to the contralateral stimulus and were suppressed by the ipsilateral stimulus in a phasic manner. A third group synchronized to the stimuli at both ears (predominantly EE). Finally, a group only responded when the waveform peaks from each ear coincided. We conclude that these groups of neurons represent different "streams" of information but exhibit modifications of the response rather than encoding a feature of the stimulus, like pitch.

The Effects of Cochlear Implant Signal Processing on Discrimination of Musical Instrument Harmonic Changes

Objectives: Compare MED-EL fine structure processing (FSP) and high-definition continuous interleaved sampling (HDCIS) in discrimination of musical instrument harmonic changes. Methods: Twelve MED-EL users were programmed with novel FSP and HDCIS maps. With each map they underwent discrimination testing of paired sound stimuli. The stimuli were native and harmonically-manipulated sound clips of a single note played by an instrument. The instruments tested were the trumpet, alto saxophone, and tuba. The harmonic manipulations consisted of removal of either the first or second harmonic. Discrimination was quantified by calculating d′; higher d′ values indicated better performance. Results: There were no significant differences in discrimination performance between FSP and HDCIS across conditions. However, for both strategies, significant differences in discrimination were observed between discrimination of the first and the second harmonic manipulation conditions for the trumpet and the tuba. For trumpet, significantly better performance was observed with removal of the second harmonic (d′ = 1.57) compared with removal of the first harmonic (d′ = 0.58; P < .05). For tuba, the opposite was observed (first d′ = 1.12, second d′ = 0; P < .05). Discrimination of the saxophone stimuli was significantly more difficult compared to the other instruments, which was likely related to the saxophone harmonics’ close frequency/channel proximity. Conclusions: Results from this novel-map study did not show a difference in harmonic cue discrimination between FSP and HDCIS. Evidence was found regarding important harmonic information for individual instruments tested. Perception of harmonic cues could be affected by channel discrimination. These results can be used to improve cochlear implant signal processing strategy.

Harmonic cues for speech segmentation: a cross-linguistic corpus study on child-directed speech

Previous studies on the role of vowel harmony in word segmentation are based on artificial languages where harmonic cues reliably signal word boundaries. In this corpus study run on the data available at CHILDES, we investigated whether natural languages provide a learner with reliable segmentation cues similar to the ones created artificially. We observed that in harmonic languages (child-directed speech to thirty-five Turkish and three Hungarian children), but not in non-harmonic ones (child-directed speech to one Farsi and four Polish children), harmonic vowel sequences are more likely to appear within words, and non-harmonic ones mostly appear across word boundaries, suggesting that natural harmonic languages provide a learner with regular cues that could potentially be used for word segmentation along with other cues.

Breaking News

Breaking News

Effect of Harmonicity on the Detection of a Signal in a Complex Masker and on Spatial Release from Masking

The amount of masking of sounds from one source (signals) by sounds from a competing source (maskers) heavily depends on the sound characteristics of the masker and the signal and on their relative spatial location. Numerous studies investigated the ability to detect a signal in a speech or a noise masker or the effect of spatial separation of signal and masker on the amount of masking, but there is a lack of studies investigating the combined effects of many cues on the masking as is typical for natural listening situations. The current study using free-field listening systematically evaluates the combined effects of harmonicity and inharmonicity cues in multi-tone maskers and cues resulting from spatial separation of target signal and masker on the detection of a pure tone in a multi-tone or a noise masker. A linear binaural processing model was implemented to predict the masked thresholds in order to estimate whether the observed thresholds can be accounted for by energetic masking in the auditory periphery or whether other effects are involved. Thresholds were determined for combinations of two target frequencies (1 and 8 kHz), two spatial configurations (masker and target either co-located or spatially separated by 90 degrees azimuth), and five different masker types (four complex multi-tone stimuli, one noise masker). A spatial separation of target and masker resulted in a release from masking for all masker types. The amount of masking significantly depended on the masker type and frequency range. The various harmonic and inharmonic relations between target and masker or between components of the masker resulted in a complex pattern of increased or decreased masked thresholds in comparison to the predicted energetic masking. The results indicate that harmonicity cues affect the detectability of a tonal target in a complex masker.

A grouping approach to harmonic complexes

Humans seem to perform sound-source separation for quasi-periodic sounds, such as speech, mostly on harmonicity cues. To model this function, most machine algorithms use a pitch-based approach to group the speech parts of the spectrum. In these methods the pitch is obtained either explicitly, in autocorrelation methods, or implicitly, as in harmonic sieves. If the estimation of pitch is wrong, the grouping will fail as well. In this paper we show a method that performs harmonic grouping without first calculating the pitch. Instead a pitch estimate is associated with each grouping hypothesis. Making the grouping independent of the pitch estimate makes it more robust in noisy settings. The algorithm obtains possible harmonics by tracking energy peaks in a cochleogram. Co-occuring harmonics are compared in terms of frequency difference. Grouping hypotheses are formed by combining harmonics with similar frequency differences. Consistency checks are performed on these hypotheses and hypotheses with compatible properties are combined into harmonic complexes. Every harmonic complex is evaluated on the number of the harmonics, the number of subsequent harmonics and the presence of a harmonic at the pitch position. By using the number of subsequent harmonics octave errors are prevented. Multiple concurrent harmonic complexes can be found as long as the spectral overlap is small.

Infants Perception of Rhythmic Patterns

We explored 9-month-old infants perception of auditory temporal sequences in a series of three experiments. In Experiment 1, we presented some infants with tone sequences that were expected to induce a strongly metric framework and others with a sequence that was expected to induce a weakly metric framework or no such framework. Infants detected a change in the context of the former sequences but not in the latter sequence. In Experiment 2, infants listened to a tone sequence with temporal cues to duple or triple meter. Infants detected a change in the pattern with duple meter but not in the pattern with triple meter. In Experiment 3, infants listened to a tone sequence with harmonic cues to duple or triple meter. As in Experiment 2, infants detected a change in the context of the duple meter pattern but not in the context of triple meter. These findings are consistent with processing predispositions for auditory temporal sequences that induce a metric framework, particularly those in duple meter.

Contribution of harmonicity and location to auditory object formation in free field: Evidence from event-related brain potentials

The contribution of location and harmonicity cues in sound segregation was investigated using behavioral reports and source waveforms derived from the scalp-recorded evoked potentials. Participants were presented with sounds composed of multiple harmonics in a free-field environment. The third harmonic was either tuned or mistuned and could be presented from the same or different location from the remaining harmonics. Presenting the third harmonic at a different location than the remaining harmonics increased the likelihood of hearing the tuned or slightly (i.e., 2%) mistuned harmonic as a separate object. Partials mistuned by 16% of their original value "pop out" of the complex and were paralleled by an object-related negativity (ORN) that superimposed the N1 and P2 components. For the 2% mistuned stimuli, the ORN was present only when the mistuned harmonic was presented at a different location than the remaining harmonics. Presenting the tuned harmonic at a different location also yielded changes in neural activity between 150 and 250 ms after sound onset. The behavioral and electrophysiological results indicate that listeners can segregate sounds based on harmonicity or location alone. The results also indicate that a conjunction of harmonicity and location cues contribute to sound segregation primarily when harmonicity is ambiguous.

Tapping to Ragtime: Cues to Pulse Finding

Two experiments investigated cues to pulse finding using a relatively unconstrained, naturalistic paradigm. Participants tapped what they felt was a comfortable pulse on a keyboard playing a percussive sound. The stimulus materials were based on ragtime excerpts, played metronomically (i.e., without expressive timing or tempo variation). The first experiment, with 8 musically experienced and 8 musically inexperienced subjects, played each excerpt in two versions: a pitch-varied version (the original excerpt) and a monotonic version (with all tones changed to middle C) that was designed to remove all melodic and harmonic cues to pulse. Neither the absence of pitch information nor musical experience significantly affected performance. The second experiment tested 12 musically experienced subjects on shorter excerpts from the same ragtime pieces. Full (right-hand and left-hand parts together) and right-hand-only versions of the excerpts were each played in pitch-varied and monotonic versions. Removing the left-hand part significantly affected tapping performance on a number of measures, causing a lower percentage of tapping on the downbeat, more off-beat taps, more aperiodic taps, more switches between tapping modes, a higher variability of the intertap interval, and larger deviations from the beat. As a whole, these indicate a negative effect of removing the left-hand part. Again, differences between pitch-varied and monotonic versions were generally small. Analysis of the music revealed the following cues to pulse finding: a predictable alternating bass pattern in the left-hand part and a majority of notes on metrically strong positions in both the right-hand and left-hand parts. These results suggest that, for piano ragtime music, temporal cues are prominently available for finding and following the pulse and that pulse finding is largely independent of pitch information. Implications of the experimental measures and music-analytic techniques for models of pulse perception are considered.

A sound segregation algorithm for reverberant conditions

A system has been developed that enables a wanted speech signal to be extracted from a background of unwanted speech and other interference under real-life conditions. Using only two microphones 25 cm apart it exploits directional and harmonicity cues in a hybrid algorithm which takes advantages of both. The output of the algorithm has been tested subjectively by human listeners, and objectively by a speech recognition system. These tests show improved intelligibility of the wanted speech signal. In one set of tests, for example, the performance of a speech recogniser after segregation was similar at a signal to noise ratio of 6 dB as without segregation at a signal to noise ratio of 20 dB.

Detection of a tonal stimulus masked by a complex with coherent or incoherent frequency modulation

The purpose of this study is to investigate if coherent frequency modulation by itself can induce auditory streaming when there are no harmonicity cues. A signal tone was presented with a complex of tones whose frequency fluctuated. To separate the effects of frequency modulation from those of harmonicity, components of the masker were not spaced by a fixed frequency distance but by a critical bandwidth. The detection threshold of the signal was tested in two conditions. In the first, called the coherent condition, all components of the masker were modulated coherently. In the second, incoherent condition, the masker was divided into two parts, the on-signal part and the flanking part. The on-signal part was the bandpass filter of the complex that was centered on the signal frequency and the flanking part was the rest of it. To make the modulation incoherent, the flanking part was moved forward or backward by half a cycle of modulation. The signal was detected with less difficulty in the coherent condition, indicating that the components with coherent frequency modulation were integrated even if they were inharmonic.

Rhythm perception and differences in accent weights for musicians and nonmusicians.

In order to investigate the contribution of harmonic-temporal and structural features to the perception of musical rhythm, three experiments were conducted in which a harmonic and a temporal accent were pitted against each other in such a way as to form five possible patterns. In three experiments, the temporal structure of various chord progressions was manipulated in an effort to determine the harmonic contributions to the inference of meter. The final experiment differed from the first two in the use of nondiatonic progressions that implied an unlikely key modulation. In all experiments, musicians and nonmusicians were requested to report perceived rhythm patterns in an attempt to determine the relative salience of various accents. Results indicated that changes in the temporal structure led to predictable change in an inferred meter, and that all diatonic chord progressions led to similar patterns of responses in which coincidences of harmonic, temporal, and metrical accents were perceptually salient events. Unusual progressions implying key modulations resulted in a qualitatively distinct pattern of results, and, in all experiments, amount of formal musical training was found to be a good predictor of the use of harmonic cues.

Identification de voyelles simultanées harmoniques et inharmoniques

It is known that the auditory system uses harmonicity cues to separate concurrent voiced sounds that differ in fundamental frequency (FO). What is less clear is whether the harmonicity of the target sound is used to enhance it within the mixture, or whether that of the background is used to cancel it and thus allow the target to emerge. An experiment was designed to answer this question. Subjects were presented with pairs of concurrent synthetic vowels, each of which was either harmonic or inharmonic. Results were scored according to the harmonicity of the vowel identified (the target), and that of the second vowel (the ground). For a given target, identification was best for harmonic grounds, except when target and ground were both harmonic and had the same FO. This is compatible with the cancellation hypothesis. On the other hand, identification for a given ground was worse when the target was harmonic. This is the opposite of the effect predicted by the enhancement hypothesis

Discriminating between coherent and incoherent frequency modulation of complex tones

A series of experiments measured the discrimination by human listeners of frequency-modulated complex tones which differed only in the coherence of frequency modulation (FM). For the coherently modulated tones all components were modulated by the same 5-Hz sinusoid, and by the same percentage of their starting frequencies, whereas for the incoherently modulated tones the modulation of one (target) component differed from that of the rest. When the 400-ms complex was composed of consecutive harmonics of a common fundamental, performance improved monotonically with increases in modulator delay, and was nearly perfect at the longest delays. When the complex was inharmonic, performance was near chance at all modular delays, both for component frequencies between 1500 and 2500 Hz, and for component frequencies between 400 and 800 Hz. It is argued that listeners detected incoherence in harmonic complexes by detecting the resulting mistuning of the target component. This conclusion was supported by the finding that listeners were usually at least as good at detecting a fixed mistuning of the center component of a harmonic complex as they were at detecting a modulator phase delay imposed on it. A final experiment, with a stimulus duration of 1 s and slower modulation rates, showed that listeners could detect incoherence for some inharmonic complexes. However, detection was worse than for harmonic complexes and was, it is argued, based on weak harmonicity cues. The results of all experiments point to the absence of an across-frequency mechanism specific to the detection of FM incoherence.