Gammatone Filterbank Research Articles

Analysis of acoustic features for speech intelligibility prediction models analysis of acoustic features for speech intelligibility prediction models

We have developed a new model to predict speech intelligibility of synthetic sounds processed by nonlinear speech enhancement algorithms. The model involves two recent auditory models: the dynamic compressive gammachirp (dcGC) auditory filterbank and the speech envelope power spectrum model (sEPSM). The dcGC-sEPSM was compared with commonly used prediction models based on perceptual intelligibility scores of speech sounds enhanced by classic spectral subtraction and state-of-the-art Wiener filtering. As a result, the dcGC-sEPSM predicted the scores better than the coherence SII (CSII), the short-time objective intelligibility (STOI), and the original sEPSM using the gammatone filterbank. There was, however, still inconsistency between the prediction and data. In this work, we show the analysis of acoustic features used in the prediction models. The CSII calculates the magnitude-squared coherence between clean and processed spectra to derive a signal-to-distortion ratio. The STOI calculates the correlation coefficients between the short-time frame vectors of clean and degraded sound at the output of the one-third octave filterbank. The sEPSM calculates the signal-to-noise ratio in the envelope modulation domain at the output of the auditory filterbank. We summarize the methods and discuss desirable features that improve speech intelligibility predictions.

Human-inspired modulation frequency features for noise-robust ASR

This paper investigates a computational model that combines a frontend based on an auditory model with an exemplar-based sparse coding procedure for estimating the posterior probabilities of sub-word units when processing noisified speech. Envelope modulation spectrogram (EMS) features are extracted using an auditory model which decomposes the envelopes of the outputs of a bank of gammatone filters into one lowpass and multiple bandpass components. Through a systematic analysis of the configuration of the modulation filterbank, we investigate how and why different configurations affect the posterior probabilities of sub-word units by measuring the recognition accuracy on a semantics-free speech recognition task. Our main finding is that representing speech signal dynamics by means of multiple bandpass filters typically improves recognition accuracy. This effect is particularly noticeable in very noisy conditions. In addition we find that to have maximum noise robustness, the bandpass filters should focus on low modulation frequencies. This reenforces our intuition that noise robustness can be increased by exploiting redundancy in those frequency channels which have long enough integration time not to suffer from envelope modulations that are solely due to noise. The ASR system we design based on these findings behaves more similar to human recognition of noisified digit strings than conventional ASR systems. Thanks to the relation between the modulation filterbank and procedures for computing dynamic acoustic features in conventional ASR systems, the finding can be used for improving the frontends in those systems.

A Study of Musical Pitch Distance Using a Self-Organized Hierarchical Linear Dynamical System on Acoustic Signals

The hierarchical linear dynamical system (HLDS) is a self-organizing architecture to cluster acoustic time series. The HLDS architecture is equivalent to a Kalman filter whose top-layer state learns to create subspaces that tessellate the acoustic signal in regions that correspond to different musical pitches. The observation layer of the HLDS is built from a biologically plausible gammatone filter bank that provides the representation space for the state assignments. An important characteristic of the methodology is that it is adaptive and self-organizing, i.e., previous exposure to the acoustic input is the only requirement for learning and recognition. In this article we show that the representation space that the algorithm learns from acoustic signals preserves the organization found in monophonic notes, and exhibits (for isolated pitches and triads) properties suggested in the theory of efficient chromatic voice leading and neo-Riemannian theories.

Speech enhancement of instantaneous amplitude and phase for applications in noisy reverberant environments

We previously proved that restoring the instantaneous amplitude as well as instantaneous phase on the output from Gammatone filterbank plays a significant role in speech enhancement. However, dereverberation is still a challenge since the previously proposed scheme can only work in noisy environments. In this paper, we extend our previously proposed scheme to general speech enhancement for removing both the effects of noise and reverberation by restoring instantaneous amplitude and phase simultaneously. Objective and subjective experiments were conducted under various noisy reverberant conditions to evaluate the effectiveness of the extension of the proposed scheme. The signal to error ratio (SER), correlation, PESQ, and SNR loss were used in objective evaluations. The normalized mean preference score and correctness in modified rhyme test (MRT) were used in subjective evaluations. We also tested how effective our proposed scheme is as a front-end for an automatic speech recognition (ASR) system in realistic noisy reverberant environments. The results of all evaluations revealed that the proposed scheme could effectively improve quality and intelligibility of speech signals under noisy reverberant conditions.

A compact digital gamma-tone filter processor

Area consumption is one of the most important design constrains in the development of compact digital systems. Several authors have proposed making compact Cochlear Implant processors using Gamma-tone filter banks. These model aspects of the cochlea spectral filtering. A good area-efficient design of the Gamma-tone Filter Bank could reduce the amount of circuitry allowing patients to wear these cochlear implants more easily. In consequence, many authors have reduced the area by using the minimum number of registers when implementing this type of filter. However, critical paths limit their performance. Here a compact Gamma-tone Filter processor, formulated using the impulse invariant transformation together with a normalization method, is presented. The normalization method in the model guarantees the same precision for any filter order. In addition, area resources are kept low due to the implementation of a single Second Order Section (SOS) IIR stage for processing several SOS IIR stages and several channels at different times. Results show that the combination of the properties of the model and the implementation techniques generate a processor with high processing speed, expending less resources than reported in the literature.

Attention selectively modulates cortical entrainment in different regions of the speech spectrum

Recent studies have uncovered a neural response that appears to track the envelope of speech, and have shown that this tracking process is mediated by attention. It has been argued that this tracking reflects a process of phase-locking to the fluctuations of stimulus energy, ensuring that this energy arrives during periods of high neuronal excitability. Because all acoustic stimuli are decomposed into spectral channels at the cochlea, and this spectral decomposition is maintained along the ascending auditory pathway and into auditory cortex, we hypothesized that the overall stimulus envelope is not as relevant to cortical processing as the individual frequency channels; attention may be mediating envelope tracking differentially across these spectral channels. To test this we reanalyzed data reported by Horton et al. (2013), where high-density EEG was recorded while adults attended to one of two competing naturalistic speech streams. In order to simulate cochlear filtering, the stimuli were passed through a gammatone filterbank, and temporal envelopes were extracted at each filter output. Following Horton et al. (2013), the attended and unattended envelopes were cross-correlated with the EEG, and local maxima were extracted at three different latency ranges corresponding to distinct peaks in the cross-correlation function (N1, P2, and N2). We found that the ratio between the attended and unattended cross-correlation functions varied across frequency channels in the N1 latency range, consistent with the hypothesis that attention differentially modulates envelope-tracking activity across spectral channels.

Neuro-steered noise suppression for auditory prostheses

In a multi-speaker scenario, a major challenge for noise suppression systems in hearing instruments is to determine which sound source the listener is attending to. It has been shown that a linear decoder can extract a neural signal from EEG recordings that is better correlated with the envelope of the attended speech signal than with the envelopes of the other signals. This can be exploited to perform auditory attention detection (AAD), which can then steer a noise suppression algorithm. The speech signal is passed through a model of the auditory periphery before extracting its envelope. We compared 7 different periphery models and found that best AAD performance was obtained with a gamma-tone filter bank followed by power-law compression. Most AAD studies so far have employed a dichotic paradigm, wherein each ear receives a separate speech stream. We compared this to a more realistic setup where speech was simulated to originate form two different spatial locations, and found that although listening conditions were harder, AAD performance was better than for the dichotic setup. Finally, we designed a neuro-steered denoising algorithm that informs the voice activity detection stage of a multi-channel Wiener filter based on AAD, and found a large signal-to-noise-ratio improvement at the output.

Improving neural decoding in the central auditory system using bio-inspired spectro-temporal representations and a generalized bilinear model.

We study the impact of different encoding models and spectro-temporal representations on the accuracy of Bayesian decoding of neural activity recorded from the central auditory system. Two encoding models, a generalized linear model (GLM) and a generalized bilinear model (GBM), are compared along with three different spectro-temporal representations of the input stimuli: a spectrogram and two bio-inspired representations, i.e. a gammatone filter bank (GFB) and a spikegram. Signal to noise ratios between the reconstructed and original representations are used to evaluate the decoding, or reconstruction accuracy. We experimentally show that the reconstruction accuracy is best with the spikegram representation and worst with the spectrogram representation and, furthermore, that using a GBM instead of a GLM significantly increases the reconstruction accuracy. In fact, our results show that the spikegram reconstruction accuracy with a GBM fitting yields an SNR that is 3.3 dB better than when using the standard decoding approach of reconstructing a spectrogram with GLM fitting.

Perceptually Accurate Reproduction of Recorded Sound Fields in a Reverberant Room Using Spatially Distributed Loudspeakers

In sound reproduction, it can be desirable to reproduce a recording made, e.g., in a concert hall, in a playback room, such that the listener has a reasonable accurate impression of the original sound source, including the room acoustics of the recording room. A perceptually motivated approach is developed which aims to accurately reproduce the perceptual spatial and monaural cues of the direct and reverberant sound fields of the recording room for playback in a reverberant room. For this approach, the direct sound, recorded close to the sound source, is optimized and processed with an auditory motivated gammatone filterbank such that spectral cues (the coloration) are faithfully reproduced in the playback room. In addition, the reverberant sound is recorded at two distant positions from the sound source in the recording room. These signals are rendered over two dipole loudspeakers. Due to the arrangement of the dipoles, the listener receives no direct sound field from the dipoles and, thus, only the diffuse field in the playback room is excited. In this way the reverberant field can be adjusted independent of the direct sound. Therefore, the spectral cues and the interaural cross correlation of the reverberant sound field can be optimized separately to make sure they match the cues which were present in the recording room.

Mean Hilbert envelope coefficients (MHEC) for robust speaker and language identification

Adverse noisy conditions pose great challenges to automatic speech applications including speaker and language identification (SID and LID), where mel-frequency cepstral coefficients (MFCC) are the most commonly adopted acoustic features. Although systems trained using MFCCs provide competitive performance under matched conditions, it is well-known that such systems are susceptible to acoustic mismatch between training and test conditions due to noise and channel degradations. Motivated by this fact, this study proposes an alternative noise-robust acoustic feature front-end that is capable of capturing speaker identity as well as language structure/content conveyed in the speech signal. Specifically, a feature extraction procedure inspired by the human auditory processing is proposed. The proposed feature is based on the Hilbert envelope of Gammatone filterbank outputs that represent the envelope of the auditory nerve response. The subband amplitude modulations, which are captured through smoothed Hilbert envelopes (a.k.a. temporal envelopes), carry useful acoustic information and have been shown to be robust to signal degradations. Effectiveness of the proposed front-end, which is entitled mean Hilbert envelope coefficients (MHEC), is evaluated in the context of SID and LID tasks using degraded speech material from the DARPA Robust Automatic Transcription of Speech (RATS) program. In addition, we investigate the impact of the dynamic range compression stage in the MHEC feature extraction process on performance using logarithmic and power-law nonlinearities. Experimental results indicate that: (i) the MHEC feature is highly effective and performs favorably compared to other conventional and state-of-the-art front-ends, and (ii) the power-law non-linearity consistently yields the best performance across different conditions for both SID and LID tasks.

Acoustic feature extraction method for robust speaker identification

When there is a mismatch between the acoustic training environment and the testing environment, the performance of automatic speaker identification systems degrades significantly. A robust feature extraction method for speaker recognition based on the gammatone filter is therefore proposed in this paper. By employing the working mechanism of the human auditory model instead of the traditional triangular filter banks, gammatone filter banks are used to simulate the auditory model of the human ear cochlea. The cube root compression method, equal loudness technology, and relative spectral (RASTA) filtering technology are incorporated into the robust feature extraction process. A simulation experiment is conducted based on the Gaussian mixture model (GMM) recognition algorithm. The experimental results indicate that the proposed feature parameters could show superior robustness and represent the characteristics of the speaker better than the conventional mel-frequency cepstrum coefficient (MFCC), cochlear cepstrum coefficient (CFCC) and relative spectra-perceptual linear predictive (RASTA-PLP) parameters.

Vowel context effects on the spectral dynamics of English and Japanese sibilant fricatives

Previous analyses of vowel context effects on the sibilant fricatives of English (/s, ʃ/) and Japanese (/s, ɕ/) have focused on spectral properties computed from a limited number of time points, e.g., frication midpoint or vowel onset. However, the spectra of sibilants vary temporally; thus, it is worth considering how their spectral dynamics vary across vocalic contexts. Vowel context effects were investigated with respect to the trajectories of three psychoacoustic measures computed across the timecourse of native, adult speakers' productions of word-initial, pre-vocalic sibilants. Psychoacoustic spectra were computed from 10 ms windows, spaced evenly across the frication, by passing each window through a bank of gammatone filters, which modeled the auditory system's differential frequency selectivity. From these psychoacoustic spectra, the peak frequency, excitation-drop (difference between maximum high-band and minimum low-band excitation), and the half-power bandwidth of the peak were computed. In Japanese, the peak frequency and the excitation-drop trajectories showed effects of vowel height—the trajectories for high (vs. mid and low) vowel contexts diverging from 50–75% of the fricative duration onward. In English, the excitation-drop trajectories showed similar effects of vowel height; however, the trajectories for high vowels diverged later than in Japanese. Peak bandwidth exhibited context effects only for Japanese /ɕ/, where it was lower in back vowel contexts across the first 75% of the fricative duration.

Online Monaural Speech Enhancement Based on Periodicity Analysis and A Priori SNR Estimation

This paper describes an online algorithm for enhancing monaural noisy speech. First, a novel phase-corrected low-delay gammatone filterbank is derived for signal subband decomposition and resynthesis; the subband signals are then analyzed frame by frame. Second, a novel feature named periodicity degree (PD) is proposed to be used for detecting and estimating the fundamental period ( ${{\rm P}_0}$ ) in each frame and for estimating the signal-to-noise ratio (SNR) in each frame-subband signal unit. The PD is calculated in each unit as the multiplication of the normalized autocorrelation and the comb filter ratio, and shown to be robust in various low-SNR conditions. Third, the noise energy level in each signal unit is estimated recursively based on the estimated SNR for units with high PD and based on the noisy signal energy level for units with low PD. Then the a priori SNR is estimated using a decision-directed approach with the estimated noise level. Finally, a revised Wiener gain is calculated, smoothed, and applied to each unit; the processed units are summed across subbands and frames to form the enhanced signal. The ${{\rm P}_{ 0}}$ detection accuracy of the algorithm was evaluated on two corpora and showed comparable performance on one corpus and better performance on the other corpus when compared to a recently published pitch detection algorithm. The speech enhancement effect of the algorithm was evaluated on one corpus with two objective criteria and showed better performance in one highly non-stationary noise and comparable performance in two other noises when compared to a state-of-the-art statistical-model based algorithm.

Optimizing pulse-spreading harmonic complexes to minimize intrinsic modulations after auditory filtering.

All signals, except sine waves, exhibit intrinsic modulations that affect perceptual masking. Reducing the physical intrinsic modulations of a broadband signal does not necessarily have a perceptual impact: auditory filtering can reintroduce modulations. Broadband signals with low intrinsic modulations after auditory filtering have proved difficult to design. To that end, this paper introduces a class of signals termed pulse-spreading harmonic complexes (PSHCs). PSHCs are generated by summing harmonically related components with such a phase that the resulting waveform exhibits pulses equally-spaced within a repetition period. The order of a PSHC determines its pulse rate. Simulations with a gamma-tone filterbank suggest an optimal pulse rate at which, after auditory filtering, the PSHC's intrinsic modulations are lowest. These intrinsic modulations appear to be less than those for broadband pseudo-random (PR) or low-noise (LN) noise. This hypothesis was tested in a modulation-detection experiment involving five modulation rates ranging from 8 to 128 Hz and both broadband and narrowband carriers using PSHCs, PR, and LN noise. PSHC showed the lowest thresholds of all broadband signals. Results imply that optimized PSHCs exhibit less intrinsic modulations after auditory filtering than any other broadband signal previously considered.

Spiking models of interaural level difference encoding - beyond the rate subtraction code

Interaural Level Difference (ILD) provides an important cue for the location of a sound source in the azimuthal plane. Typically, ILD decoding in the brainstem is modeled as a subtraction of spike rates, with inhibitory inputs from one ear subtracted from the excitatory inputs from the other [1-3]. The inferior colliculus (IC) is known to receive input from this circuit, and to encode the spatial location of sounds. Recent experimental evidence suggests that inhibitory input for ILD doesn’t provide subtraction, but instead provides a gain adjustment [4]. In addition, the exact mechanism of the creation of spatial receptive fields in the inferior colliculus remains unclear, and may also be a gain mechanism [5]. The excitatory input to the IC from neurons that decode ILD may contain spike timing cues for location. These spike-timing cues may be initiated in the ILD encoding cells even if the cues are absent from the inputs from the cochlear nucleus. In this study we used a spiking neuron model to recreate and model the full circuit of ILD sensitivity, and explore both the issue of ILD decoding, and the representation of sound source location in the IC. The auditory periphery was modeled as a gammatone filterbank which provided inputs directly to a leaky integrate-and-fire model representing the cells of the cochlear nucleus. These cells are known to lock to the envelope of the sound stimulus, and this behavior was recreated by low-pass filtering of the gammatone filterbank inputs to the cells, and use of a dynamic spike threshold mechanism [6]. The ILD sensitive cells and IC cells were both modeled as simple leaky integrate-and-fire neurons. The model was able to recreate important experimental results regarding ILD encoding cells, particularly the variation of sensitivity with source intensity, and successfully created spatial receptive fields like those found in the IC. The results will be helpful in the future understanding of the binaural mechanisms of the auditory brainstem.

The auditory image model and me

As a Ph.D. student in Roy Patterson's group at the MRC Applied Psychology Unit in Cambridge, UK, in the early 1990s, I was introduced to computer models of hearing. Much of my Ph.D. was devoted to exploring Roy’s Auditory Image Model, a time-domain model for representing regularities in hearing sensations that we hear. It was built on a gammatone filterbank, a hair-cell model, and strobed temporal integration, and was programmed with a speed that was remarkable for the age. The pictures and movies that it made—and the insights into hearing that it gave—were exciting and inspiring; my resulting enthusiasm for what good models can do has remained with me throughout my scientific career. This talk will describe some of Roy’s contributions to modelling and his influence on the field, as ever-improving computational models are crucial to making progress in understanding how hearing works. [Work supported by the Medical Research Council and the Chief Scientist Office, Scotland.]

Cochleagram-based audio pattern separation using two-dimensional non-negative matrix factorization with automatic sparsity adaptation

An unsupervised single channel audio separation method from pattern recognition viewpoint is presented. The proposed method does not require training knowledge and the separation system is based on non-uniform time-frequency (TF) analysis and feature extraction. Unlike conventional research that concentrates on the use of spectrogram or its variants, the proposed separation algorithm uses an alternative TF representation based on the gammatone filterbank. In particular, the monaural mixed audio signal is shown to be considerably more separable in this non-uniform TF domain. The analysis of signal separability to verify this finding is provided. In addition, a variational Bayesian approach is derived to learn the sparsity parameters for optimizing the matrix factorization. Experimental tests have been conducted, which show that the extraction of the spectral dictionary and temporal codes is more efficient using sparsity learning and subsequently leads to better separation performance.

Speech enhancement based on perceptual filter bank improvement

In this paper, a new denoising approach is presented based on perceptual analysis. The noisy signal is split by a gammatone filterbank with nonlinear frequency distributions according to ERB scale. The frequency masking threshold is calculated in each sub-band according to the Johnston model in the output of the Wiener filter. This threshold is then used in the gain function given by the perceptual filter. The evaluation tests are performed by using objective criterion including perceptual evaluation of speech quality as well as subjective criterion including mean opinion score. Obtained results show that the proposed method achieves best results in terms of quality and intelligibility of enhanced signal.

A Comparison of Spectro-Temporal Representations of Audio Signals

This article compares methods for the conversion of timeseries into a spectro-temporal representation. These methods are designed based on a resemblance with the auditory processing of sound in the mammalian inner ear, or on mathematical principles related to, for example, Fourier analysis. This study provides a comparison between several of these methods. Two tests were devised for this comparison: one based on susceptibility to noise and one on the expression of spectro-temporal detail. These two aspects were considered of importance for real world applications. While some methods produced good results on one of the two tests, others produced good results on both. Overall the transmission line model using an impedance function suggested by Zweig (“Finding the impedance of the organ of Corti,” J. Acoust. Soc. Amer., vol. 89, no. 3, pp. 1229-1254, 1991) provided the best results, though not significantly. Also a larger computational load may hinder application in some domains. The gammatone filterbank and straightforward spectrogram provide good alternatives with less computational load. The introduction of nonlinearity was shown to deteriorate performance on both tests, in both the filterbank and in the transmission line model.

Underwater acoustic target classification and auditory feature identification based on dissimilarity evaluation

The purpose of this study is to explore perceptual classification of underwater acoustic targets and auditory features used by human being. First, we design a paired comparison experiment. Then we use the CLASCAL algorithm to model the dissimilarity ratings as a perceptual space, and analyze the properties in three common dimensions, specialties, 3 subjects' latent classes and their roles in target perceptual classification. Finally, based on the gammatone filterbank, we find some auditory features that can effectively underlie 3 common dimensions and beat properties, so as to use them to construct a binary decision tree to classify some new samples; thus we can provide some guidance about how to use these features in practical applications.