Computational Auditory Scene Analysis Research Articles

Robust Pitch Estimation and Tracking For Speakers Based on Subband Encoding and The Generalized Labeled Multi-Bernoulli Filter

This paper proposes a new pitch estimator and a novel pitch tracker for speakers. We first decompose the sound signal into subbands using an auditory filterbank, assuming time–frequency sparsity of human speech. Instead of directly selecting the number of subbands according to experience, we propose a novel frequency coverage metric to derive the number of subbands and the center frequencies of the filterbank. The subband signals are then encoded inspired by the computational auditory scene analysis approach, and the normalized autocorrelations are calculated for pitch estimation. To suppress spurious errors and track the speaker identity, the temporal continuity constraint is exploited and a generalized labeled multi-Bernoulli filter is adapted for pitch tracking, where we use a novel pitch state transition model based on the Ornstein–Uhlenbeck process, and the measurement-driven birth model for adaptive new births of pitch targets. Experimental evaluations with various additive noises demonstrate that the proposed methods have achieved better accuracy compared with several state-of-the-art pitch estimation methods in most studied scenarios. Tests using real recordings in a reverberant room also show that the proposed method is robust against reverberation.

Environmental sound processing and its applications

As part of the effort to develop techniques for understanding environments using sound, many studies in the field of computational auditory scene analysis have focused on using computers to perform functions carried out naturally by the human auditory system. Thanks to recent progress in machine‐learning techniques, these environmental sound‐processing techniques have significantly improved and a widening variety of applications has resulted in considerable interest in this field. In this review, we introduce the fundamental techniques of environmental sound processing, as well as recent advances in front‐end and back‐end processing and potential applications for these techniques. Prospects for further progress in the field of environmental sound processing and the challenges still to be overcome are also discussed. © 2019 Institute of Electrical Engineers of Japan. Published by John Wiley & Sons, Inc.

A New Ratio Mask Representation for CASA-Based Speech Enhancement

In the computational auditory scene analysis method, the ideal ratio mask or alternatively the ideal binary mask is the key point to reconstruct the enhanced signal. The ratio mask in its Wiener filtering or its square root form is currently considered. However, this kind of ratio mask overlooked one important issue. It does not exploit the inter-channel correlation (ICC) in the noisy speech, noise, and clean speech spectra. Thus, in this paper, we first propose a novel ratio mask representation by utilizing the ICC. In this way, we adaptively reallocate the power ratio of the speech and noise during the construction of ratio mask, thus, more speech and noise components are retained and masked at the same time, respectively. Second, the channel-weight contour based on the equal loudness hearing attribute is adopted to revise this new ratio mask in each Gammatone filterbank channel. Finally, the revised ratio mask is effectively used to train a five-layer structured deep neural network. Experiments show that the proposed ratio mask performs better than the conventional ratio mask representation and other series of enhancement algorithms in terms of speech quality, intelligibility, and spectral distortion under different signal to noise ratio conditions using six types of noises.

Separation of Reverberant Speech Based on Computational Auditory Scene Analysis

This paper proposes a computational auditory scene analysis approach to separation of room reverberant speech, which performs multi-pitch tracking and supervised classification. The algorithm trains speech and non-speech model separately, which learns to map from harmonic features to grouping cue encoding the posterior probability of time-frequency unit being dominated by the target and periodic interference. Then, a likelihood ratio test selects the correct model for labeling time-frequency unit. Experimental results show that the proposed approach produces strong pitch tracking results and leads to significant improvements of predicted speech intelligibility and quality. Compared with the classical Jin-Wang algorithm, the average SNR of this algorithm is improved by 1.22 dB.

Using an Adjustment Training and a Smoothing Mask for Speech Segregation

This paper focuses on the improvement of speech intelligibility and nature auditory perception. A dual microphone computational auditory scene analysis (CASA) based speech segregation system is proposed. A deep neural network (DNN) is equipped to estimate the parameter mask, which is used to train a smoothing mask to segregate the target speech from the mixture. A mask smoothing method is proposed to reduce the musical noise, which is caused by estimation errors. The performance of the proposed method is systematic evaluated with the simulated and recording data. The tests show that the proposed method improves the signal to noise ratio (SNR), suppress the musical noise, and has good performance on untrained locations and reverberant test conditions too.

An Unsupervised Two-Talker Speech Separation System Based on CASA

On the basis of the theory about blind separation of monaural speech based on computational auditory scene analysis (CASA), a two-talker speech separation system combining CASA and speaker recognition was proposed to separate speech from other speech interferences in this paper. First, a tandem algorithm is used to organize voiced speech, then based on the clustering of gammatone frequency cepstral coefficients (GFCCs), an object function is established to recognize the speaker, and the best group is achieved through exhaustive search or beam search, so that voiced speech is organized sequentially. Second, unvoiced segments are generated by estimating onset/offset, and then unvoiced–voiced (U–V) segments and unvoiced–unvoiced (U–U) segments are separated respectively. The U–V segments are managed via the binary mask of the separated voiced speech, while the U–V segments are separated evenly. So far the unvoiced segments are separated. The simulation and performance evaluation verify the feasibility and effectiveness of the proposed algorithm.

A new time-frequency binary mask estimation method based on convex optimization of speech power

In conventional computational auditory scene analysis, the segment segregation and pitch estimation are necessary. However, it is hard to obtain an accurate pitch contour of clean speech for segregating the segments in low signal-to-noise ratio. This often leads to an inaccurate estimation of binary mask and produces artifacts and temporal discontinuity in the enhanced speech. To overcome this problem, we propose in this paper a new estimation method for binary mask based on convex optimization of speech power. In this proposed method, the segment segregation and pitch estimation are excluded and only the speech power of each Gammatone channel is considered as a key cue used for labeling the binary masks. Considering the cross-correlation between the power spectra of noisy speech and noise in each channel, the objective function of speech power is built, and the speech power is solved by the gradient descent method. Accordingly, the time-frequency units of speech and noise are labeled by computing a decision factor derived from the powers of noisy speech, estimated speech and the pre-estimated noise. The erroneous local masks are refined by time-frequency unit smoothing. The objective measurements including segmental signal-to-noise ratio, HIT-False Alarm rate, the percentage of energy loss and the percentage of noise residue and the additional subjective listening test demonstrate the effectiveness of the proposed method.

IRM estimation based on data field of cochleagram for speech enhancement

When computational auditory scene analysis (CASA) is used for the speech enhancement, it can mask noise effectively by an accurate mask estimation approach. In this paper, we attempt to apply the ideal ratio mask (IRM) estimation based on the spectral dependency into the speech cochleagram for enhancing speech. To achieve the spectral dependency, the concept of data field (DF) is introduced to model the time-frequency (T-F) relationship of the cochleagram so that the obtained results (termed as the potentials) with the adjacent spectral information are used eventually to estimate the IRM. In the estimation framework, we firstly use a pre-processed module to obtain initial T-F values of noise and speech. Then, given initial estimations of noise and speech, we can employ DF model to obtain the forms of speech and noise potentials, which are viewed as the energy with the information of its neighbors. Subsequently, based on the forms of speech and noise potentials, their optimal potentials that reflect their respective optimal distribution are obtained by the optimal influence factors. Finally, we attempt to obtain the masking value using the potentials of speech and noise for restoring clean target speech signal. Our algorithm is evaluated and compared with the reference methods, and it can yield an effective improvement in speech quality.

Post-Processing for the Mask of Computational Auditory Scene Analysis in Monaural Speech Segregation

Speech segregation is one of the most difficult tasks in speech processing. This paper uses computational auditory scene analysis, support vector machine classifier, and post-processing on binary mask to separate speech from background noise. Melfrequency cepstral coefficients and pitch are the two features used for support vector machine classification. Connected Component Labeling, Hole Filling, and Morphology are applied on the resulting binary mask as post-processing. Experimental results show that our method separates speech from background noise effectively.

Past review, current progress, and challenges ahead on the cocktail party problem

The cocktail party problem, i.e., tracing and recognizing the speech of a specific speaker when multiple speakers talk simultaneously, is one of the critical problems yet to be solved to enable the wide application of automatic speech recognition (ASR) systems. In this overview paper, we review the techniques proposed in the last two decades in attacking this problem. We focus our discussions on the speech separation problem given its central role in the cocktail party environment, and describe the conventional single-channel techniques such as computational auditory scene analysis (CASA), non-negative matrix factorization (NMF) and generative models, the conventional multi-channel techniques such as beamforming and multi-channel blind source separation, and the newly developed deep learning-based techniques, such as deep clustering (DPCL), the deep attractor network (DANet), and permutation invariant training (PIT). We also present techniques developed to improve ASR accuracy and speaker identification in the cocktail party environment. We argue effectively exploiting information in the microphone array, the acoustic training set, and the language itself using a more powerful model. Better optimization objective and techniques will be the approach to solving the cocktail party problem.

Image Processing Techniques for Segments Grouping in Monaural Speech Separation

Monaural speech separation is the process of separating the target speech from the noisy speech mixture recorded using single microphone. It is a challenging problem in speech signal processing, and recently, computational auditory scene analysis (CASA) finds a reasonable solution to solve this problem. This research work proposes an image analysis-based algorithm to enhance the binary T–F mask obtained in the initial segmentation stage of CASA-based monaural speech separation systems to improve the speech quality. The proposed algorithm consists of labeling the initial segmentation mask, boundary extraction, active pixel detection and finally eliminating the noisy non-active pixels. In labeling, the T–F mask obtained from the initial segmentation is labeled as periodicity pixel matrix and non-periodicity pixel matrix. Next boundaries are created by connecting all the possible nearby periodicity pixel matrix and non-periodicity pixel matrix as speech boundary. Some speech boundary may include noisy T–F units as holes, and these holes are treated using the proposed algorithm to properly classify them as the speech-dominant or noise-dominant T–F units in the active pixel detection process. Finally, the noisy T–F units are eliminated. The performance of the proposed algorithm is evaluated using TIMIT speech database. The experimental results show that the proposed algorithm improves the quality of the separated speech by increasing the signal-to-noise ratio by an average value of 9.64 dB and reduces the noise residue by 25.55% as compared to the noisy speech mixture.

A Bayesian computational basis for auditory selective attention using head rotation and the interaural time-difference cue.

The process of resolving mixtures of several sounds into their separate individual streams is known as auditory scene analysis and it remains a challenging task for computational systems. It is well-known that animals use binaural differences in arrival time and intensity at the two ears to find the arrival angle of sounds in the azimuthal plane, and this localization function has sometimes been considered sufficient to enable the un-mixing of complex scenes. However, the ability of such systems to resolve distinct sound sources in both space and frequency remains limited. The neural computations for detecting interaural time difference (ITD) have been well studied and have served as the inspiration for computational auditory scene analysis systems, however a crucial limitation of ITD models is that they produce ambiguous or “phantom” images in the scene. This has been thought to limit their usefulness at frequencies above about 1khz in humans. We present a simple Bayesian model and an implementation on a robot that uses ITD information recursively. The model makes use of head rotations to show that ITD information is sufficient to unambiguously resolve sound sources in both space and frequency. Contrary to commonly held assumptions about sound localization, we show that the ITD cue used with high-frequency sound can provide accurate and unambiguous localization and resolution of competing sounds. Our findings suggest that an “active hearing” approach could be useful in robotic systems that operate in natural, noisy settings. We also suggest that neurophysiological models of sound localization in animals could benefit from revision to include the influence of top-down memory and sensorimotor integration across head rotations.

Multitalker Speech Separation With Utterance-Level Permutation Invariant Training of Deep Recurrent Neural Networks

In this paper, we propose the utterance-level permutation invariant training (uPIT) technique. uPIT is a practically applicable, end-to-end, deep-learning-based solution for speaker independent multitalker speech separation. Specifically, uPIT extends the recently proposed permutation invariant training (PIT) technique with an utterance-level cost function, hence eliminating the need for solving an additional permutation problem during inference, which is otherwise required by frame-level PIT. We achieve this using recurrent neural networks (RNNs) that, during training, minimize the utterance-level separation error, hence forcing separated frames belonging to the same speaker to be aligned to the same output stream. In practice, this allows RNNs, trained with uPIT, to separate multitalker mixed speech without any prior knowledge of signal duration, number of speakers, speaker identity, or gender. We evaluated uPIT on the WSJ0 and Danish two- and three-talker mixed-speech separation tasks and found that uPIT outperforms techniques based on nonnegative matrix factorization and computational auditory scene analysis, and compares favorably with deep clustering, and the deep attractor network. Furthermore, we found that models trained with uPIT generalize well to unseen speakers and languages. Finally, we found that a single model, trained with uPIT, can handle both two-speaker, and three-speaker speech mixtures.

Flight of the bumble bee: Buzzes predict pollination services.

Multiple interacting factors drive recent declines in wild and managed bees, threatening their pollination services. Widespread and intensive monitoring could lead to more effective management of wild and managed bees. However, tracking their dynamic populations is costly. We tested the effectiveness of an inexpensive, noninvasive and passive acoustic survey technique for monitoring bumble bee behavior and pollination services. First, we assessed the relationship between the first harmonic of the flight buzz (characteristic frequency) and pollinator functional traits that influence pollination success using flight cage experiments and a literature search. We analyzed passive acoustic survey data from three locations on Pennsylvania Mountain, Colorado to estimate bumble bee activity. We developed an algorithm based on Computational Auditory Scene Analysis that identified and quantified the number of buzzes recorded in each location. We then compared visual and acoustic estimates of bumble bee activity. Using pollinator exclusion experiments, we tested the power of buzz density to predict pollination services at the landscape scale for two bumble bee pollinated alpine forbs (Trifolium dasyphyllum and T. parryi). We found that the characteristic frequency was correlated with traits known to affect pollination efficacy, explaining 30–52% of variation in body size and tongue length. Buzz density was highly correlated with visual estimates of bumble bee density (r = 0.97), indicating that acoustic signals are predictive of bumble bee activity. Buzz density predicted seed set in two alpine forbs when bumble bees were permitted access to the flowers, but not when they were excluded from visiting. Our results indicate that acoustic signatures of flight can be deciphered to monitor bee activity and pollination services to bumble bee pollinated plants. We propose that applications of this technique could assist scientists and farmers in rapidly detecting and responding to bee population declines.

Evaluation of scene analysis using real and simulated acoustic mixtures: Lessons learnt from the CHiME speech recognition challenges

Computational auditory scene analysis is increasingly presented in the literature as a set of auditory-inspired techniques for estimating “Ideal Binary Masks” (IBM), i.e., time-frequency domain segregations of the attended source and the acoustic background based on a local signal-to-noise ratio objective (Wang and Brown, 2006). This talk argues that although IBMs may be a useful stand-in when evaluating signal-processing systems, they can provide a misleading perspective when considering models of auditory cognition. First, there is no evidence that human cognition computes or requires an explicit binary mask representation (ideal or otherwise). Second, evaluation of an IBM requires artificially-mixed acoustic scenes in order to provide access to the ground truth mask. It is possible that systems that work well on artificially mixed acoustic scenes will fail to generalize to real data. The danger in predicting real performance from results obtained on artificial mixtures is seen in an analysis of systems submitted to the recent CHiME distant microphone speech recognition challenges which evaluates on both types of data (http://spandh.dcs.shef.ac.uk/chime). It is argued that rather than presume specific internal representations, auditory scene analysis systems can be best evaluated by direct comparison of human and machine percepts, e.g., in the case of a speech recognition task, comparison of human and machine transcriptions at a phonetic level.

Sound-Environment Monitoring Method Based on Computational Auditory Scene Analysis

Monitoring techniques are a key technology for examining the conditions in various scenarios, e.g., structural conditions, weather conditions, and disasters. In order to understand such scenarios, the appropriate extraction of their features from observation data is important. This paper proposes a monitoring method that allows sound environments to be expressed as a sound pattern. To this end, the concept of synesthesia is exploited. That is, the keys, tones, and pitches of the monitored sound are expressed using the three elements of color, that is, the hue, saturation, and brightness, respectively. In this paper, it is assumed that the hue, saturation, and brightness can be detected from the chromagram, sonogram, and sound spectrogram, respectively, based on a previous synesthesia experiment. Then, the sound pattern can be drawn using color, yielding a “painted sound map.” The usefulness of the proposed monitoring technique is verified using environmental sound data observed at a galleria.

Progress in Understanding Auditory Scene Analysis

In this paper, I make the following claims: (1) Subjective experience is tremendously useful in guiding productive research. (2) Studies of auditory scene analysis (ASA) in adults, newborn infants, and non-human animals (e.g., in goldfish or pigeons) establish the generality of ASA and suggest that it has an innate foundation. (3) ASA theory does not favor one musical style over another. (4) The principles used in the composition of polyphony (slightly modified) apply not only to one particular musical style or culture but to any form of layered music. (5) Neural explanations of ASA do not supersede explanations in terms of capacities; the two are complementary. (6) In computational auditory scene analysis (CASA) – ASA by computer systems – or any adequate theory of ASA, the most difficult challenge will be to discover how the contributions of a very large number of types of acoustical evidence and top-down schemas (acquired knowledge about the sound sources in our environments), can be coordinated without producing conflict that disables the system. (7) Finally I argue that the movement of a listener within the auditory scene provides him/her/it with rich information that should not be ignored by ASA theorists and researchers.

Classification of Overlapped Audio Events Based on AT, PLSA, and the Combination of Them

Audio event classification, as an important part of Computational Auditory Scene Analysis, has attracted much attention. Currently, the classification technology is mature enough to classify isolated audio events accurately, but for overlapped audio events, it performs much worse. While in real life, most audio documents would have cer- tain percentage of overlaps, and so the overlap classifica- tion problem is an important part of audio classification. Nowadays, the work on overlapped audio event classifica- tion is still scarce, and most existing overlap classification systems can only recognize one audio event for an overlap. In this paper, in order to deal with overlaps, we innova- tively introduce the author-topic (AT) model which was first proposed for text analysis into audio classification, and innovatively combine it with PLSA (Probabilistic La- tent Semantic Analysis). We propose 4 systems, i.e. AT, PLSA, AT-PLSA and PLSA-AT, to classify overlaps. The 4 proposed systems have the ability to recognize two or more audio events for an overlap. The experimental results show that the 4 systems perform well in classifying overlapped audio events, whether it is the overlap in training set or the overlap out of training set. Also they perform well in clas- sifying isolated audio events.

Advanced methods of signal processing in acoustics

Signal processing is applied in virtually all areas of modern acoustics to extract, classify, and/or quantify relevant information from acoustic measurements. Methods range from classical approaches based on Fourier and time-frequency analysis, to array signal processing, feature extraction, computational auditory scene analysis, and Bayesian inference, which incorporates physical models of the acoustic system under investigation together with advanced sampling techniques. This talk highlights new approaches to signal processing recently applied in a broad variety of acoustical problems.

An Unsupervised Approach to Close-Talk Speech Enhancement

A K-means based unsupervised approach to close-talk speech enhancement is proposed in this paper. With the frame work of computational auditory scene analysis (CASA), the dual-microphone energy difference (DMED) is used as the cue to classify the noise domain time-frequency (T-F) units and target speech domain units. A ratio mask is used to separate the target speech and noise. Experiment results show the robust performance of the proposed algorithm than the Wiener filtering algorithm.