Encoding Of Pitch Research Articles

Decoding linguistically-relevant pitch patterns from frequency-following responses using hidden Markov models

Pitch encoding is often studied with frequency-following response (FFR), a scalp-recorded potential reflecting phase-locked activity from auditory subcortical ensembles. Prior work using FFR have shown that long-term language experience modulates subcortical encoding of linguistically-relevant pitch patterns. These studies typically rely on FFRs averaging across thousands of repetitions, due to low signal-to-noise ratio of single-trial FFRs. Here, we evaluated the extent to which hidden Markov models (HMMs), with fewer numbers of trials, can be used to quantify pitch encoding as well as capture language experience-dependent plasticity in pitch encoding. FFRs were recorded from fourteen Mandarin Chinese and fourteen American English passively listening to four Mandarin tones (1000 trials per tone). HMMs were used to recognize FFRs to each tone in individual participants. Specifically, HMMs were trained and tested across FFR sets of different sizes, ranging from 50 to 500 trials. Results showed that HMMs were able to recognize tones from FFRs, above chance, across all training sizes and languages. Interestingly, HMMs picked up language differences (Chinese > English) at very small sizes for training (e.g., 200) and testing (e.g., 100). These findings highlight the potential benefits of using HMMs to reduce experimental time and efforts in FFR data collection. [Project funded by NIH NoR01DC013315 834 (B.C.).]

Stability and plasticity in neural encoding of linguistically relevant pitch patterns.

While lifelong language experience modulates subcortical encoding of pitch patterns, there is emerging evidence that short-term training introduced in adulthood also shapes subcortical pitch encoding. Here we use a cross-language design to examine the stability of language experience-dependent subcortical plasticity over multiple days. We then examine the extent to which behavioral relevance induced by sound-to-category training leads to plastic changes in subcortical pitch encoding in adulthood relative to adolescence, a period of ongoing maturation of subcortical and cortical auditory processing. Frequency-following responses (FFRs), which reflect phase-locked activity from subcortical neural ensembles, were elicited while participants passively listened to pitch patterns reflective of Mandarin tones. In experiment 1, FFRs were recorded across three consecutive days from native Chinese-speaking (n = 10) and English-speaking (n = 10) adults. In experiment 2, FFRs were recorded from native English-speaking adolescents (n = 20) and adults (n = 15) before, during, and immediately after a session of sound-to-category training, as well as a day after training ceased. Experiment 1 demonstrated the stability of language experience-dependent subcortical plasticity in pitch encoding across multiple days of passive exposure to linguistic pitch patterns. In contrast, experiment 2 revealed an enhancement in subcortical pitch encoding that emerged a day after the sound-to-category training, with some developmental differences observed. Taken together, these findings suggest that behavioral relevance is a critical component for the observation of plasticity in the subcortical encoding of pitch.NEW & NOTEWORTHY We examine the timescale of experience-dependent auditory plasticity to linguistically relevant pitch patterns. We find extreme stability in lifelong experience-dependent plasticity. We further demonstrate that subcortical function in adolescents and adults is modulated by a single session of sound-to-category training. Our results suggest that behavioral relevance is a necessary ingredient for neural changes in pitch encoding to be observed throughout human development. These findings contribute to the neurophysiological understanding of long- and short-term experience-dependent modulation of pitch.

Loudness and pitch perception using Dynamically Compensated Virtual Channels

Reducing power consumption is important for the development of smaller cochlear implant (CI) speech processors. Simultaneous electrode stimulation may improve power efficiency by minimizing the required current applied to a given electrode. Simultaneous in-phase stimulation on adjacent electrodes (i.e. virtual channels) can be used to elicit pitch percepts intermediate to the ones provided by each of the physical electrodes in isolation. Virtual channels are typically implemented in monopolar stimulation mode, producing broad excitation patterns. Focused stimulation may reduce the excitation patterns, but is inefficient in terms of power consumption. To create a more power efficient virtual channel, we developed the Dynamically Compensated Virtual Channel (DC-VC) using four adjacent electrodes. The two central electrodes are current steered using the coefficient α (0<α<1 ) whereas the two flanking electrodes are used to focus/unfocus the stimulation with the coefficient σ (−1<σ<1). With increasing values of σ, power can be saved at the potential expense of generating broader electric fields. Additionally, reshaping the electric fields might also alter place pitch coding.The goal of the present study is to investigate the tradeoff between place pitch encoding and power savings using simultaneous electrode stimulation in the DC-VC configuration. A computational model and psychophysical experiments in CI users have been used for that purpose.Results from 10 adult Advanced Bionics CI users have been collected. Results show that the required current to produce comfortable levels is significantly reduced with increasing σ as predicted by the computational model. Moreover, no significant differences in the estimated number of discriminable steps were detected for the different values of σ. From these results, we conclude that DC-VCs can reduce power consumption without decreasing the number of discriminable place pitch steps.

Imperfect pitch: Gabor's uncertainty principle and the pitch of extremely brief sounds.

How brief must a sound be before its pitch is no longer perceived? The uncertainty tradeoff between temporal and spectral resolution (Gabor's principle) limits the minimum duration required for accurate pitch identification or discrimination. Prior studies have reported that pitch can be extracted from sinusoidal pulses as brief as half a cycle. This finding has been used in a number of classic papers to develop models of pitch encoding. We have found that phase randomization, which eliminates timbre confounds, degrades this ability to chance, raising serious concerns over the foundation on which classic pitch models have been built. The current study investigated whether subthreshold pitch cues may still exist in partial-cycle pulses revealed through statistical integration in a time series containing multiple pulses. To this end, we measured frequency-discrimination thresholds in a two-interval forced-choice task for trains of partial-cycle random-phase tone pulses. We found that residual pitch cues exist in these pulses but discriminating them requires an order of magnitude (ten times) larger frequency difference than that reported previously, necessitating a re-evaluation of pitch models built on earlier findings. We also found that as pulse duration is decreased to less than two cycles its pitch becomes biased toward higher frequencies, consistent with predictions of an auto-correlation model of pitch extraction.

The Effects of Absolute Pitch and Tone Language on Pitch Processing and Encoding in Musicians

Absolute pitch (AP) is the rare ability to identify or produce a specific pitch without a reference pitch, which appears to be more prevalent in tone-language speakers than non-tone-language speakers. Numerous studies support a close relationship between AP, music, and language. Despite this relationship, the extent to which these factors contribute to the processing and encoding of pitch has not yet been investigated. Addressing this research question would provide insights into the relationship between music and language, as well as the mechanisms of AP. To this aim, we recruited AP musicians and non-AP musicians who were either tone-language (Mandarin and Cantonese) or non-tone language speakers. Participants completed a zero- and one-back working memory task using music and non-music (control) stimuli. In general, AP participants had better accuracy and faster reaction times than participants without AP. This effect remained even after controlling for the age at which participants began formal music lessons. We did not observe a performance advantage afforded by speaking a tone language, nor a cumulative advantage afforded by having AP and being a tone-language speaker.

Brainstem encoding of speech and musical stimuli in congenital amusia: evidence from Cantonese speakers.

Congenital amusia is a neurodevelopmental disorder of musical processing that also impacts subtle aspects of speech processing. It remains debated at what stage(s) of auditory processing deficits in amusia arise. In this study, we investigated whether amusia originates from impaired subcortical encoding of speech (in quiet and noise) and musical sounds in the brainstem. Fourteen Cantonese-speaking amusics and 14 matched controls passively listened to six Cantonese lexical tones in quiet, two Cantonese tones in noise (signal-to-noise ratios at 0 and 20 dB), and two cello tones in quiet while their frequency-following responses (FFRs) to these tones were recorded. All participants also completed a behavioral lexical tone identification task. The results indicated normal brainstem encoding of pitch in speech (in quiet and noise) and musical stimuli in amusics relative to controls, as measured by FFR pitch strength, pitch error, and stimulus-to-response correlation. There was also no group difference in neural conduction time or FFR amplitudes. Both groups demonstrated better FFRs to speech (in quiet and noise) than to musical stimuli. However, a significant group difference was observed for tone identification, with amusics showing significantly lower accuracy than controls. Analysis of the tone confusion matrices suggested that amusics were more likely than controls to confuse between tones that shared similar acoustic features. Interestingly, this deficit in lexical tone identification was not coupled with brainstem abnormality for either speech or musical stimuli. Together, our results suggest that the amusic brainstem is not functioning abnormally, although higher-order linguistic pitch processing is impaired in amusia. This finding has significant implications for theories of central auditory processing, requiring further investigations into how different stages of auditory processing interact in the human brain.

The temporal coding of pitch: Insights from human electrophysiology

Brian Moore and Roy Patterson have made seminal contributions to our understanding of pitch perception, and in particular the use of temporal pitch information by the auditory brain. The pitch of sounds may be encoded, at least in part, by the tendency of neurons to phase lock to the temporal fine structure and/or envelope of basilar membrane vibration. Direct physiological measures of this mechanism are difficult in humans. However, the gross activity of neurons in the brainstem can be measured using electrophysiological techniques. The frequency-following response (FFR) is an electrophysiological measure of phase locking in the rostral brainstem. The FFR differs between musicians and non-musicians, and across linguistic groups, and is sensitive to short-term pitch discrimination training. These findings suggest that the FFR may reflect neural activity relevant to the encoding of pitch, although other results suggest that it may reflect basic peripheral encoding, rather than the output of a pitch extraction process. Recent results from our laboratory show that combining behavioral and FFR measures can provide insights into the coding of the pitch of pure tones and the coding of musical consonance. The FFR may be a blunt tool, but it provides information that cannot be obtained using other techniques, and this may be particularly useful in investigations of the effects of age and hearing loss on the neural coding of pitch.

Neural Representation of Harmonic Complex Tones in Primary Auditory Cortex of the Awake Monkey

Many natural sounds are periodic and consist of frequencies (harmonics) that are integer multiples of a common fundamental frequency (F0). Such harmonic complex tones (HCTs) evoke a pitch corresponding to their F0, which plays a key role in the perception of speech and music. "Pitch-selective" neurons have been identified in non-primary auditory cortex of marmoset monkeys. Noninvasive studies point to a putative "pitch center" located in a homologous cortical region in humans. It remains unclear whether there is sufficient spectral and temporal information available at the level of primary auditory cortex (A1) to enable reliable pitch extraction in non-primary auditory cortex. Here we evaluated multiunit responses to HCTs in A1 of awake macaques using a stimulus design employed in auditory nerve studies of pitch encoding. The F0 of the HCTs was varied in small increments, such that harmonics of the HCTs fell either on the peak or on the sides of the neuronal pure tone tuning functions. Resultant response-amplitude-versus-harmonic-number functions ("rate-place profiles") displayed a periodic pattern reflecting the neuronal representation of individual HCT harmonics. Consistent with psychoacoustic findings in humans, lower harmonics were better resolved in rate-place profiles than higher harmonics. Lower F0s were also temporally represented by neuronal phase-locking to the periodic waveform of the HCTs. Findings indicate that population responses in A1 contain sufficient spectral and temporal information for extracting the pitch of HCTs by neurons in downstream cortical areas that receive their input from A1.

Auditory Training: Evidence for Neural Plasticity in Older Adults.

Improvements in digital amplification, cochlear implants, and other innovations have extended the potential for improving hearing function; yet, there remains a need for further hearing improvement in challenging listening situations, such as when trying to understand speech in noise or when listening to music. Here, we review evidence from animal and human models of plasticity in the brain's ability to process speech and other meaningful stimuli. We considered studies targeting populations of younger through older adults, emphasizing studies that have employed randomized controlled designs and have made connections between neural and behavioral changes. Overall results indicate that the brain remains malleable through older adulthood, provided that treatment algorithms have been modified to allow for changes in learning with age. Improvements in speech-in-noise perception and cognition function accompany neural changes in auditory processing. The training-related improvements noted across studies support the need to consider auditory training strategies in the management of individuals who express concerns about hearing in difficult listening situations. Given evidence from studies engaging the brain's reward centers, future research should consider how these centers can be naturally activated during training.

Breaking News

Breaking News

Cognition and the Evolution of Music: Pitfalls and Prospects

What was the role of music in the evolutionary history of human beings? We address this question from the point of view that musicality can be defined as a cognitive trait. Although it has been argued that we will never know how cognitive traits evolved (Lewontin, 1998), we argue that we may know the evolution of music by investigating the fundamental cognitive mechanisms of musicality, for example, relative pitch, tonal encoding of pitch, and beat induction. In addition, we show that a nomological network of evidence (Schmitt & Pilcher, 2004) can be built around the hypothesis that musicality is a cognitive adaptation. Within this network, different modes of evidence are gathered to support a specific evolutionary hypothesis. We show that the combination of psychological, medical, physiological, genetic, phylogenetic, hunter-gatherer, and cross-cultural evidence indicates that musicality is a cognitive adaptation.

Experience-dependent plasticity in pitch encoding

Linguistic and musical pitch provide an analytic window to evaluate how neural representations of important pitch attributes of a sound undergo transformation from early sensory to later cognitive stages of processing in the human brain, and how pitch-relevant experience shapes these representations. These pitch attributes are shaped differentially depending on their functional relevance to a listener. Neural encoding of pitch-relevant information is shaped by the perceptual salience of domain-specific features at subcortical (auditory brainstem) and cortical stages of processing. The emergence of a functional ear asymmetry in the neural encoding of pitch-relevant information at a lower sensory processing level supports the view that local and feedforward and feedback mechanisms are involved in pitch-relevant processing. A theoretical framework for a neural network is proposed involving coordination between local, feedforward, and feedback components that can account for experience-induced enhancement of pitch representations at multiple levels of the auditory pathway.

Development of Simultaneous Pitch Encoding: Infants Show a High Voice Superiority Effect

Infants must learn to make sense of real-world auditory environments containing simultaneous and overlapping sounds. In adults, event-related potential studies have demonstrated the existence of separate preattentive memory traces for concurrent note sequences and revealed perceptual dominance for encoding of the voice with higher fundamental frequency of 2 simultaneous tones or melodies. Here, we presented 2 simultaneous streams of notes (15 semitones apart) to 7-month-old infants. On 50% of trials, either the higher or the lower note was modified by one semitone, up or down, leaving 50% standard trials. Infants showed mismatch negativity (MMN) to changes in both voices, indicating separate memory traces for each voice. Furthermore, MMN was earlier and larger for the higher voice as in adults. When in the context of a second voice, representation of the lower voice was decreased and that of the higher voice increased compared with when each voice was presented alone. Additionally, correlations between MMN amplitude and amount of weekly music listening suggest that experience affects the development of auditory memory. In sum, the ability to process simultaneous pitches and the dominance of the highest voice emerge early during infancy and are likely important for the perceptual organization of sound in realistic environments.

ERP correlates of pitch error detection in complex Tone and Voice auditory feedback with missing fundamental

Previous studies have shown that the pitch of a sound is perceived in the absence of its fundamental frequency (F0), suggesting that a distinct mechanism may resolve pitch based on a pattern that exists between harmonic frequencies. The present study investigated whether such a mechanism is active during voice pitch control. ERPs were recorded in response to +200 cents pitch shifts in the auditory feedback of self-vocalizations and complex tones with and without the F0. The absence of the fundamental induced no difference in ERP latencies. However, a right-hemisphere difference was found in the N1 amplitudes with larger responses to complex tones that included the fundamental compared to when it was missing. The P1 and N1 latencies were shorter in the left hemisphere, and the N1 and P2 amplitudes were larger bilaterally for pitch shifts in voice and complex tones compared with pure tones. These findings suggest hemispheric differences in neural encoding of pitch in sounds with missing fundamental. Data from the present study suggest that the right cortical auditory areas, thought to be specialized for spectral processing, may utilize different mechanisms to resolve pitch in sounds with missing fundamental. The left hemisphere seems to perform faster processing to resolve pitch based on the rate of temporal variations in complex sounds compared with pure tones. These effects indicate that the differential neural processing of pitch in the left and right hemispheres may enable the audio-vocal system to detect temporal and spectral variations in the auditory feedback for vocal pitch control.

Linguistic status of timbre influences pitch encoding in the brainstem

The aim of this experiment is to assess the effects of the linguistic status of timbre on pitch processing in the brainstem. Brainstem frequency following responses were evoked by the Mandarin high-rising lexical tone superimposed on a native vowel quality ([i]), nonnative vowel quality ([œ]), and iterated rippled noise (nonspeech). Results revealed that voice fundamental frequency magnitudes were larger when concomitant with a native vowel quality compared with either nonnative vowel quality or nonspeech timbre. Such experience-dependent effects suggest that subcortical sensory encoding of pitch interacts with timbre in the human brainstem. As a consequence, responses of the perceptual system can be differentially shaped to pitch patterns in relation to the linguistic status of their concomitant timbre.

Cross-Linguistic Comparison of Frequency-Following Responses to Voice Pitch in American and Chinese Neonates and Adults

Cross-language studies, as reflected by the scalp-recorded frequency-following response (FFR) to voice pitch, have shown the influence of dominant linguistic environments on the encoding of voice pitch at the brainstem level in normal-hearing adults. Research questions that remained unanswered included the characteristics of the FFR to voice pitch in neonates during their immediate postnatal period and the relative contributions of the biological capacities present at birth versus the influence of the listener's postnatal linguistic experience. The purpose of this study was to investigate the characteristics of FFR to voice pitch in neonates during their first few days of life and to examine the relative contributions of the "biological capacity" versus "linguistic experience" influences on pitch processing in the human brainstem. Twelve American neonates (five males, 1-3 days old) and 12 Chinese neonates (seven males, 1-3 days old) were recruited to examine the characteristics of the FFRs during their immediate postnatal days of life. Twelve American adults (three males; age: mean ± SD = 24.6 ± 3.0 yr) and 12 Chinese adults (six males; age: mean ± SD = 25.3 ± 2.6 yr) were also recruited to determine the relative contributions of biological and linguistic influences. A Chinese monosyllable that mimics the English vowel /i/ with a rising pitch (117-166 Hz) was used to elicit the FFR to voice pitch in all participants. Two-way analysis of variance (i.e., the language [English versus Chinese] and age [neonate versus adult] factors) showed a significant difference in Pitch Strength for language (p = 0.035, F = 4.716). A post hoc Tukey-Kramer analysis further demonstrated that Chinese adults had significantly larger Pitch Strength values than Chinese neonates (p = 0.024). This finding, coupled with the fact that American neonates and American adults had comparable Pitch Strength values, supported the linguistic experience model. On the other hand, Pitch Strength obtained from the American neonates, American adults, and Chinese neonates were not significantly different from each other, supporting the biological capacity model. This study demonstrated an early maturation of voice-pitch processing in neonates starting from 1 to 3 days after birth and a significant effect of linguistic experience on the neural processing of voice pitch at the brainstem level. These findings provide a significant conceptual advancement and a basis for further examination of developmental maturation of subcortical representation of speech features, such as pitch, timing, and harmonics. These findings can also be used to help identify neonates at risk for delays in voice-pitch perception and provide new directions for preventive and therapeutic interventions for patients with central auditory processing deficits, hearing loss, and other types of communication disorders.

Online Recognition of Music Is Influenced by Relative and Absolute Pitch Information

Three experiments explored online recognition in a nonspeech domain, using a novel experimental paradigm. Adults learned to associate abstract shapes with particular melodies, and at test they identified a played melody's associated shape. To implicitly measure recognition, visual fixations to the associated shape versus a distractor shape were measured as the melody played. Degree of similarity between associated melodies was varied to assess what types of pitch information adults use in recognition. Fixation and error data suggest that adults naturally recognize music, like language, incrementally, computing matches to representations before melody offset, despite the fact that music, unlike language, provides no pressure to execute recognition rapidly. Further, adults use both absolute and relative pitch information in recognition. The implicit nature of the dependent measure should permit use with a range of populations to evaluate postulated developmental and evolutionary changes in pitch encoding.

Functional ear (a)symmetry in brainstem neural activity relevant to encoding of voice pitch: A precursor for hemispheric specialization?

Pitch processing is lateralized to the right hemisphere; linguistic pitch is further mediated by left cortical areas. This experiment investigates whether ear asymmetries vary in brainstem representation of pitch depending on linguistic status. Brainstem frequency-following responses (FFRs) were elicited by monaural stimulation of the left and right ear of 15 native speakers of Mandarin Chinese using two synthetic speech stimuli that differ in linguistic status of tone. One represented a native lexical tone (Tone 2: T2); the other, T2′, a nonnative variant in which the pitch contour was a mirror image of T2 with the same starting and ending frequencies. Two 40-ms portions of f 0 contours were selected in order to compare two regions (R1, early; R2 late) differing in pitch acceleration rate and perceptual saliency. In R2, linguistic status effects revealed that T2 exhibited a larger degree of FFR rightward ear asymmetry as reflected in f 0 amplitude relative to T2′. Relative to midline (ear asymmetry = 0), the only ear asymmetry reaching significance was that favoring left ear stimulation elicited by T2′. By left- and right-ear stimulation separately, FFRs elicited by T2 were larger than T2′ in the right ear only. Within T2′, FFRs elicited by the earlier region were larger than the later in both ears. Within T2, no significant differences in FFRS were observed between regions in either ear. Collectively, these findings support the idea that origins of cortical processing preferences for perceptually-salient portions of pitch are rooted in early, preattentive stages of processing in the brainstem.

Neural Encoding of Speech and Music: Implications for Hearing Speech in Noise.

Understanding speech in a background of competing noise is challenging, especially for individuals with hearing loss or deficits in auditory processing ability. The ability to hear in background noise cannot be predicted from the audiogram, an assessment of peripheral hearing ability; therefore, it is important to consider the impact of central and cognitive factors on speech-in-noise perception. Auditory processing in complex environments is reflected in neural encoding of pitch, timing, and timbre, the crucial elements of speech and music. Musical expertise in processing pitch, timing, and timbre may transfer to enhancements in speech-in-noise perception due to shared neural pathways for speech and music. Through cognitive-sensory interactions, musicians develop skills enabling them to selectively listen to relevant signals embedded in a network of melodies and harmonies, and this experience leads in turn to enhanced ability to focus on one voice in a background of other voices. Here we review recent work examining the biological mechanisms of speech and music perception and the potential for musical experience to ameliorate speech-in-noise listening difficulties.

Relationship between selective auditory attention and brainstem encoding in musicians and non‐musicians.

This study explores whether differences in performance in a challenging selective auditory attention task are related to the strength of the sub‐cortical encoding of stimulus pitch. In each behavioral trial, two simultaneous digits, differing in pitch by 6 semitones, were presented. A preceding visual cue instructed listeners to report the target digit with either the higher or lower pitch. Subjects both were more accurate at reporting the target digit and had a shorter reaction time when the visual‐cue‐to‐auditory‐target interval was long compared to trials with a short preparatory interval. Moreover, behavioral performance was generally better for subjects with musical training than for non‐musicians. The brainstem frequency‐following responses (FFRs) were recorded in response to a complex tone in both quiet and in noise for the same listeners. Consistent with past reports, musicians demonstrated a stronger FFR than non‐musicians; in addition, the musicians’ FFR response was less affected by noise compared to non‐musicians. The strength of the FFR correlated with individual performance on the pitch‐based selective auditory attention task. Results suggest that the robustness of the brainstem encoding of acoustic inputs directly determines the ability to selectively attend to an auditory target using pitch differences.