Spectro-temporal Envelope Research Articles

Object-specific adaptation in the auditory cortex of bats.

Identifying behaviorally relevant sounds is a vital function of the auditory system. Echolocating bats must negotiate a wealth of sounds during navigation and foraging. They must detect relatively rare but behaviorally relevant echoes and segregate them from many unimportant background echoes. For this, the bat's auditory system might rely on neural deviance detection-a process influencing the excitability of a neuron depending on the frequency of occurrence of a stimulus. To investigate neural deviance detection in the auditory cortex (AC) of anesthetized bats (Phyllostomus discolor), we designed sequences of repetitive virtual echoes differing in the spectrotemporal envelope, resembling those that bats might perceive in their natural environment. A standard echo was repeatedly played 10 times in these sequences, followed by a deviant echo at the end. Time intervals between echoes within the sequences varied. Our results show that neurons in the AC of the bat P. discolor are sensitive to novel virtual echoes presented at the end of these repetitive sequences: In 49% (62/126) of cortical neurons, extracellularly recorded responses adapted to the standard echo but showed a strong response to the deviant echo presented at the end. This effect depended strongly on the time intervals between echoes, with stronger adaptation at shorter intervals. This type of response might indicate a form of neuronal deviance detection mechanism in the AC that could help the bats to detect echoes of novel and potentially important objects within a stream of homogeneous background echoes.NEW & NOTEWORTHY In this study, we show that neurons in the auditory cortex of the bat Phyllostomus discolor are sensitive to novel acoustic stimuli in the context of repetitive virtual echo sequences differing in spectrotemporal envelope. This represents a form of neuronal deviance detection that might help the bats to detect echoes of rare but relevant objects among the clutter.

A Speech Preprocessing Method Based on Perceptually Optimized Envelope Processing to Increase Intelligibility in Reverberant Environments

Speech intelligibility in public places can be degraded by the environmental noise and reverberation. In this study, a new near-end listening enhancement (NELE) approach is proposed in which using a time varying filter jointly enhances the onsets and reduces the overlap masking. For optimization, some look-ahead in clean speech and prior knowledge of room impulse response (RIR) are required. In this method, by optimizing a defined cost function, the Spectro-Temporal Envelope of reverb speech is optimized to be as close as possible to that of clean speech. In this cost function, onsets of speech are optimized with increased weight. This approach is different from overlap-masking ratio (OMR) and speech enhancement (OE) approaches (Grosse, van de Par, 2017, J. Audio Eng. Soc., Vol. 65 (1/2), pp. 31–41) that only consider previous frames in each time slot for determining the time variant filtering. The SRT measurements show that the new optimization framework enhances the speech intelligibility up to 2 dB more that OE.

Communication breakdown: Limits of spectro-temporal resolution for the perception of bat communication calls

During vocal communication, the spectro-temporal structure of vocalizations conveys important contextual information. Bats excel in the use of sounds for echolocation by meticulous encoding of signals in the temporal domain. We therefore hypothesized that for social communication as well, bats would excel at detecting minute distortions in the spectro-temporal structure of calls. To test this hypothesis, we systematically introduced spectro-temporal distortion to communication calls of Phyllostomus discolor bats. We broke down each call into windows of the same length and randomized the phase spectrum inside each window. The overall degree of spectro-temporal distortion in communication calls increased with window length. Modelling the bat auditory periphery revealed that cochlear mechanisms allow discrimination of fast spectro-temporal envelopes. We evaluated model predictions with experimental psychophysical and neurophysiological data. We first assessed bats’ performance in discriminating original versions of calls from increasingly distorted versions of the same calls. We further examined cortical responses to determine additional specializations for call discrimination at the cortical level. Psychophysical and cortical responses concurred with model predictions, revealing discrimination thresholds in the range of 8–15 ms randomization-window length. Our data suggest that specialized cortical areas are not necessary to impart psychophysical resilience to temporal distortion in communication calls.

Effects of Training on Lateralization for Simulations of Cochlear Implants and Single-Sided Deafness.

While cochlear implantation has benefitted many patients with single-sided deafness (SSD), there is great variability in cochlear implant (CI) outcomes and binaural performance remains poorer than that of normal-hearing (NH) listeners. Differences in sound quality across ears—temporal fine structure (TFS) information with acoustic hearing vs. coarse spectro-temporal envelope information with electric hearing—may limit integration of acoustic and electric patterns. Binaural performance may also be limited by inter-aural mismatch between the acoustic input frequency and the place of stimulation in the cochlea. SSD CI patients must learn to accommodate these differences between acoustic and electric stimulation to maximize binaural performance. It is possible that training may increase and/or accelerate accommodation and further improve binaural performance. In this study, we evaluated lateralization training in NH subjects listening to broad simulations of SSD CI signal processing. A 16-channel vocoder was used to simulate the coarse spectro-temporal cues available with electric hearing; the degree of inter-aural mismatch was varied by adjusting the simulated insertion depth (SID) to be 25 mm (SID25), 22 mm (SID22) and 19 mm (SID19) from the base of the cochlea. Lateralization was measured using headphones and head-related transfer functions (HRTFs). Baseline lateralization was measured for unprocessed speech (UN) delivered to the left ear to simulate SSD and for binaural performance with the acoustic ear combined with the 16-channel vocoders (UN+SID25, UN+SID22 and UN+SID19). After completing baseline measurements, subjects completed six lateralization training exercises with the UN+SID22 condition, after which performance was re-measured for all baseline conditions. Post-training performance was significantly better than baseline for all conditions (p < 0.05 in all cases), with no significant difference in training benefits among conditions. Given that there was no significant difference between the SSD and the SSD CI conditions before or after training, the results suggest that NH listeners were unable to integrate TFS and coarse spectro-temporal cues across ears for lateralization, and that inter-aural mismatch played a secondary role at best. While lateralization training may benefit SSD CI patients, the training may largely improve spectral analysis with the acoustic ear alone, rather than improve integration of acoustic and electric hearing.

Inharmonic speech reveals the role of harmonicity in the cocktail party problem

The “cocktail party problem” requires us to discern individual sound sources from mixtures of sources. The brain must use knowledge of natural sound regularities for this purpose. One much-discussed regularity is the tendency for frequencies to be harmonically related (integer multiples of a fundamental frequency). To test the role of harmonicity in real-world sound segregation, we developed speech analysis/synthesis tools to perturb the carrier frequencies of speech, disrupting harmonic frequency relations while maintaining the spectrotemporal envelope that determines phonemic content. We find that violations of harmonicity cause individual frequencies of speech to segregate from each other, impair the intelligibility of concurrent utterances despite leaving intelligibility of single utterances intact, and cause listeners to lose track of target talkers. However, additional segregation deficits result from replacing harmonic frequencies with noise (simulating whispering), suggesting additional grouping cues enabled by voiced speech excitation. Our results demonstrate acoustic grouping cues in real-world sound segregation.

Identification of the spectrotemporal modulations that support speech intelligibility in hearing-impaired and normal-hearing listeners

Accurate encoding of the spectrotemporal envelope of speech is essential for intelligible perception. We have recently developed a psychophysical procedure that identifies the relative contributions of particular spectrotemporal modulations (STMs) to intelligibility. Here, we use the procedure to test the hypothesis that intelligibility is supported by different patterns of STMs in hearing-impaired versus normal-hearing listeners. A group of 20 hearing-impaired listeners and an age-matched group of 13 normal-hearing (≤ 25 dB HL from 0.25-4 kHz) listeners performed a speech recognition task in which acoustically-degraded sentences presented over headphones were repeated back verbally and scored for keywords correctly identified. Different patterns of STMs were removed on each trial by applying a randomly-shaped filter to the 2-D modulation power spectrum. Reverse correlation was used to identify STMs that predicted performance (i.e., intelligibility) across trials. Three main findings describe the results: (1) the group-average patterns of STMs supporting intelligibility did not differ between hearing-impaired and normal-hearing listeners; (2) greater individual variability in STM patterns was observed within the hearing-impaired group; and (3) hearing-impaired listeners required more overall STM information to perform the task. The results suggest hearing-impaired listeners rely on the same STM information as normal-hearing listeners but encode this information less efficiently.

Classification of place of articulation in unvoiced stops with spectro-temporal surface modeling

Unvoiced stops are rapidly varying sounds with acoustic cues to place identity linked to the temporal dynamics. Neurophysiological studies have indicated the importance of joint spectro-temporal processing in the human perception of stops. In this study, two distinct approaches to modeling the spectro-temporal envelope of unvoiced stop phone segments are investigated with a view to obtaining a low-dimensional feature vector for automatic place classification. Classification accuracies on the TIMIT database and a Marathi words dataset show the overall superiority of classifier combination of polynomial surface coefficients and 2D-DCT. A comparison of performance with published results on the place classification of stops revealed that the proposed spectro-temporal feature systems improve upon the best previous systems’ performances. The results indicate that joint spectro-temporal features may be usefully incorporated in hierarchical phone classifiers based on diverse class-specific features.

Dynamics of phase-independent spectro-temporal tuning in primary auditory cortex of the awake ferret

Tuning of cortical neurons is often measured as a static property, or during a steady-state regime, despite a number of studies suggesting that tuning depends on when it is measured during a neuron’s response (e.g., onset vs. sustained vs. offset). We have previously shown that phase-locked tuning to feature transients evolves as a dynamic quantity from the onset of the sound. In this follow-up study, we examined the phase-independent tuning during feature transients. Based on previous results, we hypothesized phase-independent tuning should evolve on the same timescale as phase-locked tuning. We used stimuli of constant level, but alternating between flat spectro-temporal envelope and a modulated envelope with well-defined spectral density and temporal periodicity. This allowed the measure of changes in tuning to novel spectro-temporal content, as happens during running speech and other sounds with rapid transitions without a confounding change in sound level. For 95% of neurons, tuning changed significantly from the onset, over the course of the response. For a majority of these cells, the change occurred within the first 40ms following a feature onset, often even around 10–20ms. This solidifies the idea that tuning can change rapidly from onset tuning to the sustained, steady-state tuning.

Contrast tuned responses in primary auditory cortex of the awake ferret

Auditory neurons are often characterized by their spectro-temporal receptive field (STRF), a linear measure that captures overall trends of neural responses to modulations of the spectro-temporal envelopes of sounds. We have previously shown that primary auditory cortex neurons of the awake ferret are better characterized by STRFs followed by a non-trivial non-linearity. This non-linearity is a half-wave rectification followed by a squaring function, indicating that cortical neurons probably encode higher-order statistics of the spectrum of sounds. In this article, we introduce the concept of a contrast receptive field (CRF) and show that neurons in the auditory cortex encode quadratic statistics of the spectro-temporal envelope of sounds, which we call auditory contrast. We reveal phase-dependent contrast tuning in single units. Most units with a reliable STRF also possess a reliable CRF, such that the response to stimulus contrast complements the linear response described by the STRF. The relationship between the STRF and the CRF is analyzed in terms of orthogonality, co-localization in time-frequency, feature orientation selectivity, and output non-linearity interdependence. Our study shows that contrast can be used by auditory cortex neurons to sharpen, in a noise-resistant fashion, their responses to dynamic spectral profiles.

Spectro-temporal envelope changes caused by temporal fine structure modification

The study of speech from which the temporal fine structure (TFS) has been removed has become an important research area. Common procedures for removing TFS include noise and tone vocoders. In the noise vocoder, bands of noise are modulated by the envelope of the speech within each band, and in the tone vocoder the carrier is a sinusoid at the center of each frequency band. Five different procedures for removing TFS are evaluated in this paper: the noise vocoder, a low-noise noise approach in which the noise envelope is replaced by the speech envelope in each frequency band, phase randomization within each band, the tone vocoder, and sinusoidal modeling with random phase. The effects of TFS modification on the speech envelope are evaluated using an index based on the envelope time-frequency modulation. The results show that for all of the TFS techniques implemented in this study, there is a substantial loss in the accuracy of reproduction of the envelope time-frequency modulation. The tone vocoder gives the best accuracy, followed by the procedure that replaces the noise envelope with the speech envelope in each band.

Phonetic encoding by intracranial signals in human auditory cortex

Event Abstract Back to Event Phonetic encoding by intracranial signals in human auditory cortex Brian N. Pasley1*, Nathan E. Crone2, Robert T. Knight1 and Edward F. Chang3 1 University of California at Berkeley, United States 2 The Johns Hopkins University, United States 3 University of California at San Francisco, United States Speech comprehension requires mapping time-varying frequency representations onto categorical phonetic representations. How the auditory system decomposes complex acoustic signals into elementary features and then combines these features to form invariant representations of auditory objects is unknown. To study auditory and phonetic representation, we recorded intracranial signals from epileptic patients, using subdural electrode grids placed over left or right superior temporal gyrus (STG). Patients listened passively to phonetically-transcribed English sentences from a variety of speakers. We evaluated the ability of a phonetic-based neural encoding model to account for speech-induced high gamma (70-150 Hz) responses. We compared the predictive power of the phonetic model to that of linear and nonlinear spectro-temporal models. The phonetic model is based on a set of categorical predictors, one for each of 58 distinct consonant and vowel phones from the TIMIT phonetic alphabet. The linear auditory model is based on the spectro-temporal envelope of the stimulus, while the nonlinear model is based on the modulation content. The models are fitted to neural responses at each electrode site and predictive power is evaluated as the correlation between actual and predicted responses from a validation data set. Across most STG sites, the phonetic model provided significantly better predictions compared to the auditory models, with correlations up to r = .6. Phonetic tuning in the fitted models exhibited selectivity for consonant-vowel sequences at specific sites. The results indicate that higher order auditory areas in human STG encode both auditory and acoustically invariant phonetic information. The pattern of phonetic tuning is consistent with a role in lexical recognition. Keywords: Intracranial signals, Language Conference: XI International Conference on Cognitive Neuroscience (ICON XI), Palma, Mallorca, Spain, 25 Sep - 29 Sep, 2011. Presentation Type: Poster Presentation Topic: Poster Sessions: Neural Bases of Language Citation: Pasley BN, Crone NE, Knight RT and Chang EF (2011). Phonetic encoding by intracranial signals in human auditory cortex. Conference Abstract: XI International Conference on Cognitive Neuroscience (ICON XI). doi: 10.3389/conf.fnhum.2011.207.00287 Copyright: The abstracts in this collection have not been subject to any Frontiers peer review or checks, and are not endorsed by Frontiers. They are made available through the Frontiers publishing platform as a service to conference organizers and presenters. The copyright in the individual abstracts is owned by the author of each abstract or his/her employer unless otherwise stated. Each abstract, as well as the collection of abstracts, are published under a Creative Commons CC-BY 4.0 (attribution) licence (https://creativecommons.org/licenses/by/4.0/) and may thus be reproduced, translated, adapted and be the subject of derivative works provided the authors and Frontiers are attributed. For Frontiers’ terms and conditions please see https://www.frontiersin.org/legal/terms-and-conditions. Received: 22 Nov 2011; Published Online: 28 Nov 2011. * Correspondence: Dr. Brian N Pasley, University of California at Berkeley, Berkeley, United States, bpasley@berkeley.edu Login Required This action requires you to be registered with Frontiers and logged in. To register or login click here. Abstract Info Abstract The Authors in Frontiers Brian N Pasley Nathan E Crone Robert T Knight Edward F Chang Google Brian N Pasley Nathan E Crone Robert T Knight Edward F Chang Google Scholar Brian N Pasley Nathan E Crone Robert T Knight Edward F Chang PubMed Brian N Pasley Nathan E Crone Robert T Knight Edward F Chang Related Article in Frontiers Google Scholar PubMed Abstract Close Back to top Javascript is disabled. Please enable Javascript in your browser settings in order to see all the content on this page.

Dynamic Spectral Envelope Modeling for Timbre Analysis of Musical Instrument Sounds

We present a computational model of musical instrument sounds that focuses on capturing the dynamic behavior of the spectral envelope. A set of spectro-temporal envelopes belonging to different notes of each instrument are extracted by means of sinusoidal modeling and subsequent frequency interpolation, before being subjected to principal component analysis. The prototypical evolution of the envelopes in the obtained reduced-dimensional space is modeled as a nonstationary Gaussian Process. This results in a compact representation in the form of a set of prototype curves in feature space, or equivalently of prototype spectro-temporal envelopes in the time-frequency domain. Finally, the obtained models are successfully evaluated in the context of two music content analysis tasks: classification of instrument samples and detection of instruments in monaural polyphonic mixtures.

Dynamics of spectro-temporal tuning in primary auditory cortex of the awake ferret

We previously characterized the steady-state spectro-temporal tuning properties of cortical cells with respect to broadband sounds by using sounds with sinusoidal spectro-temporal modulation envelope where spectral density and temporal periodicity were constant over several seconds. However, since speech and other natural sounds have spectro-temporal features that change substantially over milliseconds, we study the dynamics of tuning by using stimuli of constant overall intensity, but alternating between a flat spectro-temporal envelope and a modulated envelope with well defined spectral density and temporal periodicity. This allows us to define the tuning of cortical cells to speech-like and other rapid transitions, on the order of milliseconds, as well as the time evolution of this tuning in response to the appearance of new features in a sound. Responses of 92 cells in AI were analyzed based on the temporal evolution of the following measures of tuning after a rapid transition in the stimulus: center of mass and breadth of tuning; separability and direction selectivity; temporal and spectral asymmetry. We find that tuning center of mass increased in 70% of cells for spectral density and in 68% of cells for temporal periodicity, while roughly half of cells (47%) broadened their tuning, with the other half (53%) sharpening tuning. The majority of cells (73%) were initially not direction selective, as measured by an inseparability index, which had an initial low value that then increased to a higher steady state value. Most cells were characterized by temporal symmetry, while spectral symmetry was initially high and then progressed to low steady-state values (61%). We demonstrate that cortical neurons can be characterized by a lag-dependent modulation transfer function. This characterization, when measured through to steady-state, becomes equivalent to the classical spectro-temporal receptive field.

Response adaptation to broadband sounds in primary auditory cortex of the awake ferret

Driven by previous reports of adaptation to persistent stimuli in other brain regions, we investigated adaptive effects in the Primary Auditory Cortex of awake non-behaving ferrets ( Mustela putorius furo). Electrophysiological data was obtained in response to the presentation of auditory gratings with a structured spectro-temporal envelope of varying bandwidth which had repeated transitions between low and high modulation depths. The responses were analyzed in terms of the evoked spike rates and in terms of the degree of phase locking to the modulation. We found two populations of cells, both of which showed adaptation in the traditional sense. For one population, we also found a second order of adaptation – i.e., adaptation of the adaptation. This suggests the existence of at least two coding strategies which differ in the weight placed on sensory context.

Dynamics of precise spike timing in primary auditory cortex.

Although single units in primary auditory cortex (A1) exhibit accurate timing in their phasic response to the onset of sound (precision of a few milliseconds), paradoxically, they are unable to sustain synchronized responses to repeated stimuli at rates much beyond 20 Hz. To explore the relationship between these two aspects of cortical response, we designed a broadband stimulus with a slowly modulated spectrotemporal envelope riding on top of a rapidly modulated waveform (or fine structure). Using this stimulus, we quantified the ability of cortical cells to encode independently and simultaneously the stimulus envelope and fine structure. Specifically, by reverse-correlating unit responses with these two stimulus dimensions, we measured the spectrotemporal response fields (STRFs) associated with the processing of the envelope, the fine structure, and the complete stimulus. A1 cells respond well to the slow spectrotemporal envelopes and produce a wide variety of STRFs. In over 70% of cases, A1 units also track the fine-structure modulations precisely, throughout the stimulus, and for frequencies up to several hundred Hertz. Such a dual response, however, is contingent on the cell being driven by both fast and slow modulations, in that the response to the slowly modulated envelope gates the expression of the fine structure. We also demonstrate that either a simplified model of synaptic depression and facilitation, and/or a cortical network of thalamic excitation and cortical inhibition can account for major trends in the observed findings. Finally, we discuss the potential functional significance and perceptual relevance of these coexistent, complementary dynamic response modes.

Spectro-temporal response field characterization with dynamic ripples in ferret primary auditory cortex.

To understand the neural representation of broadband, dynamic sounds in primary auditory cortex (AI), we characterize responses using the spectro-temporal response field (STRF). The STRF describes, predicts, and fully characterizes the linear dynamics of neurons in response to sounds with rich spectro-temporal envelopes. It is computed from the responses to elementary "ripples," a family of sounds with drifting sinusoidal spectral envelopes. The collection of responses to all elementary ripples is the spectro-temporal transfer function. The complex spectro-temporal envelope of any broadband, dynamic sound can expressed as the linear sum of individual ripples. Previous experiments using ripples with downward drifting spectra suggested that the transfer function is separable, i.e., it is reducible into a product of purely temporal and purely spectral functions. Here we measure the responses to upward and downward drifting ripples, assuming reparability within each direction, to determine if the total bidirectional transfer function is fully separable. In general, the combined transfer function for two directions is not symmetric, and hence units in AI are not, in general, fully separable. Consequently, many AI units have complex response properties such as sensitivity to direction of motion, though most inseparable units are not strongly directionally selective. We show that for most neurons, the lack of full separability stems from differences between the upward and downward spectral cross-sections but not from the temporal cross-sections; this places strong constraints on the neural inputs of these AI units.