Auditory Scene Analysis Research Articles

Harmonic Cancellation-A Fundamental of Auditory Scene Analysis.

This paper reviews the hypothesis of harmonic cancellation according to which an interfering sound is suppressed or canceled on the basis of its harmonicity (or periodicity in the time domain) for the purpose of Auditory Scene Analysis. It defines the concept, discusses theoretical arguments in its favor, and reviews experimental results that support it, or not. If correct, the hypothesis may draw on time-domain processing of temporally accurate neural representations within the brainstem, as required also by the classic equalization-cancellation model of binaural unmasking. The hypothesis predicts that a target sound corrupted by interference will be easier to hear if the interference is harmonic than inharmonic, all else being equal. This prediction is borne out in a number of behavioral studies, but not all. The paper reviews those results, with the aim to understand the inconsistencies and come up with a reliable conclusion for, or against, the hypothesis of harmonic cancellation within the auditory system.

Indoor Multi-Speaker Localization Based on Bayesian Nonparametrics in the Circular Harmonic Domain

Circular microphone arrays have been used for multi-speaker localization in computational auditory scene analysis, for their high flexibility in sound field analysis, including the generation of frequency-invariant eigenbeams for wideband acoustic sources. However, the localization performance of existing circular harmonic approaches, such as circular harmonics beamformer (CHB) depends strongly on the physical characteristics (such as shape) of sensor arrays, and the level of uncertainties presented in acoustic environments (such as background noise, room reverberation, and the number of sources). These uncertainties may limit the performance or practical application of the speaker localization algorithms. To address these issues, in this paper, we present a new indoor multi-speaker localization method in the circular harmonic domain based on the acoustic holography beamforming (AHB) technique and the Bayesian nonparametrics (BNP) method. More specifically, we use the AHB technique, which combines the delay-and-sum beamforming with acoustic-holography-based virtual sensing, to generate direction of arrival (DOA) measurements in the time-frequency (TF) domain, and then design a BNP algorithm based on the infinite Gaussian mixture model (IGMM) to estimate the DOAs of the individual sources without the prior knowledge about the number of sources. These estimates may degrade in the presence of room reverberation and background noise. To address this issue, we develop a robust TF bin selection and permutation method on the basis of mixture weights, using power, power ratio and local variance estimated at each TF bin. Experiments performed on both simulated and real-data show that our method gives significantly better performance, than four recent baseline methods, in a variety of noise and reverberation levels, in terms of the root-mean-square error (RMSE) of the DOA estimation and the source detecting success rate.

Towards More Efficient DNN-Based Speech Enhancement Using Quantized Correlation Mask

Many studies on deep learning-based speech enhancement (SE) utilizing the computational auditory scene analysis method typically employs the ideal binary mask or the ideal ratio mask to reconstruct the enhanced speech signal. However, many SE applications in real scenarios demand a desirable balance between denoising capability and computational cost. In this study, first, an improvement over the ideal ratio mask to attain more superior SE performance is proposed through introducing an efficient adaptive correlation-based factor for adjusting the ratio mask. The proposed method exploits the correlation coefficients among the noisy speech, noise and clean speech to effectively re-distribute the power ratio of the speech and noise during the ratio mask construction phase. Second, to make the supervised SE system more computationally-efficient, quantization techniques are considered to reduce the number of bits needed to represent floating numbers, leading to a more compact SE model. The proposed quantized correlation mask is utilized in conjunction with a 4-layer deep neural network (DNN-QCM) comprising dropout regulation, pre-training and noise-aware training to derive a robust and high-order mapping in enhancement, and to improve generalization capability in unseen conditions. Results show that the quantized correlation mask outperforms the conventional ratio mask representation and the other SE algorithms used for comparison. When compared to a DNN with ideal ratio mask as its learning targets, the DNN-QCM provided an improvement of approximately 6.5% in the short-time objective intelligibility score and 11.0% in the perceptual evaluation of speech quality score. The introduction of the quantization method can reduce the neural network weights to a 5-bit representation from a 32-bit, while effectively suppressing stationary and non-stationary noise. Timing analyses also show that with the techniques incorporated in the proposed DNN-QCM system to increase its compactness, the training and inference time can be reduced by 15.7% and 10.5%, respectively.

Training enhances the ability of listeners to exploit visual information for auditory scene analysis

The ability to use temporal relationships between cross-modal cues facilitates perception and behavior. Previously we observed that temporally correlated changes in the size of a visual stimulus and the intensity in an auditory stimulus influenced the ability of listeners to perform an auditory selective attention task (Maddox, Atilgan, Bizley, & Lee, 2015). Participants detected timbral changes in a target sound while ignoring those in a simultaneously presented masker. When the visual stimulus was temporally coherent with the target sound, performance was significantly better than when the visual stimulus was temporally coherent with the masker, despite the visual stimulus conveying no task-relevant information. Here, we trained observers to detect audiovisual temporal coherence and asked whether this changed the way in which they were able to exploit visual information in the auditory selective attention task. We observed that after training, participants were able to benefit from temporal coherence between the visual stimulus and both the target and masker streams, relative to the condition in which the visual stimulus was coherent with neither sound. However, we did not observe such changes in a second group that were trained to discriminate modulation rate differences between temporally coherent audiovisual streams, although they did show an improvement in their overall performance. A control group did not change their performance between pretest and post-test and did not change how they exploited visual information. These results provide insights into how crossmodal experience may optimize multisensory integration.

Audiovisual integration improves task performance in AD and bvFTD

AbstractBackgroundComplex analysis of sensory inputs, such as auditory scene analysis and spatial localisation, is impaired in Alzheimer’s disease (AD) and behavioural variant frontotemporal dementia (bvFTD). Multisensory integration research demonstrates that synchronous stimulation in one sense (e.g. vision) can modify processing of suboptimal sensory inputs from another sense (e.g. hearing) in the healthy brain. However, the potential benefit of multisensory stimulation to reduce common symptoms in dementia has not been fully examined.MethodPatients living with AD (n = 20) or bvFTD (n = 12), and healthy age‐matched controls (n = 18) were recruited to perform three brief cognitive tasks: 1. Cocktail party effect, identify a name spoken in a noisy environment; 2. Spatial ventriloquism, discriminate the location of an auditory stimulus (left/right of centre); and 3. Detect a suboptimal visual stimulus. Task performance was compared between unisensory, audiovisual congruent, and audiovisual incongruent conditions, and between groups.Result1. Identification of spoken names in a noisy auditory environment was significantly worse for both patient groups than controls. All groups performed significantly better when viewing an accompanying video of congruent lip movements. The AD group showed some improvement for incongruent videos that cued the onset of the spoken name. 2. Auditory spatial discrimination was significantly worse in the AD group than controls, but both groups were significantly biased by the locations of the accompanying visual stimulus. The bvFTD group was only influenced by spatially incongruent visual stimulation. 3. Detection of a near‐threshold peripheral visual stimulus was improved by a synchronous auditory stimulus, irrespective of spatial congruency, in the bvFTD and control groups. In the AD group, only spatially congruent or neutral auditory stimuli benefitted visual stimulus detection.ConclusionCongruent audiovisual stimulation improves the ability of people living with AD or bvFTD to understand voices in a noisy environment, locate auditory sounds, and detect unreliable visual events. Congruency of audiovisual stimulation in different domains (e.g. semantic, spatial) provides evidence for disease‐specific stratification of multisensory integration profiles. As such, multisensory integration provides a mechanism to improve reliability of sensory inputs and the potential to alleviate symptoms in daily life if tailored according to disease.

Sensitivity to temporal structure facilitates perceptual analysis of complex auditory scenes

The notion that sensitivity to the statistical structure of the environment is pivotal to perception has recently garnered considerable attention. Here we investigated this issue in the context of hearing. Building on previous work (Sohoglu and Chait, 2016a; elife), stimuli were artificial ‘soundscapes’ populated by multiple (up to 14) simultaneous streams (‘auditory objects’) comprised of tone-pip sequences, each with a distinct frequency and pattern of amplitude modulation. Sequences were either temporally regular or random.We show that listeners’ ability to detect abrupt appearance or disappearance of a stream is facilitated when scene streams were characterized by a temporally regular fluctuation pattern. The regularity of the changing stream as well as that of the background (non-changing) streams contribute independently to this effect. Remarkably, listeners benefit from regularity even when they are not consciously aware of it. These findings establish that perception of complex acoustic scenes relies on the availability of detailed representations of the regularities automatically extracted from multiple concurrent streams.

Harmonic-Temporal Factor Decomposition for Unsupervised Monaural Separation of Harmonic Sounds

We address the problem of separating a monaural mixture of harmonic sounds into the audio signals of individual semitones in an unsupervised manner. Unsupervised monaural audio source separation has thus far been mainly addressed by two approaches: one rooted in computational auditory scene analysis (CASA) and the other based on non-negative matrix factorization (NMF). These approaches focus on different clues for making source separation possible. A CASA-based method called harmonic-temporal clustering (HTC) focuses on a local time-frequency structure of individual sources, whereas NMF focuses on a global time-frequency structure of music spectrograms. These clues do not conflict with each other and can be used to achieve a more reliable audio source separation algorithm. Hence, we propose a monaural audio source separation framework, harmonic-temporal factor decomposition (HTFD), by developing a spectrogram model that encompasses the features of the models used in the NMF and HTC approaches. We further incorporate a source-filter model to build an extension of HTFD, source-filter HTFD (SF-HTFD). We derive efficient parameter estimation algorithms of HTFD and SF-HTFD based on the auxiliary function principle. We show, through music source separation experiments, the efficacy of HTFD and SF-HTFD compared with conventional methods. Furthermore, we demonstrate the effectiveness of HTFD and SF-HTFD for automatic musical key transposition.

Sound event aware environmental sound segmentation with Mask U-Net

This paper proposes an environmental sound segmentation method using Mask U-Net. In recent years, human–robot interactions, especially speech dialogue, have been assessed by auditory scene analysis. Methods, such as noise reduction, section detection, and sound source separation have been proposed for robot audition, acoustic signal processing, and machine learning. However, such conventional approaches have three drawbacks: (1) Many studies have analyzed individual functions, which are regarded as being a cascade. Cascade systems can, however, result in the accumulation of errors generated at each functional block. (2) Unlike conventional cascade systems, deep learning-based methods that simultaneously detect sections, separate sound sources, and identify classes have also been proposed for speech separation. These techniques can be extended for multiple classes of environmental sounds, but their performance becomes degraded with large variations in sound event lengths among classes. (3) In addition, these methods have recurrent neural network layers making it difficult to process calculations in parallel. This paper proposes an environmental sound segmentation method called Mask U-Net, which robustly differentiates sound event lengths among classes. Simulation experiments using a developed 75-class environmental sound data set showed that the proposed method was faster than conventional methods and showed high segmentation performance.

Modeling the contribution of auditory scene analysis principles to perceptual effects of orchestration

Auditory Scene Analysis (ASA) research has provided knowledge about the principles underlying auditory organization processes and has been successfully applied in computational ASA. Based on these findings, we applied these principles within a musical context. The aim is to understand and computationally model the perception of effects, such as blend and segregation, created by the combining and contrasting of properties of traditional Western instruments which result from three auditory processes: concurrent, sequential, and segmental grouping. The initial aim was to evaluate the extent to which the symbolic data provided in a musical score provide sufficient data to model the perception of these orchestral effects. Preliminary implementations have achieved an average accuracy score of 81%, suggesting that perceptual effects of orchestration can be partially retrieved by calculations based on ASA principles using symbolic data. However, many cases indicate that including properties of the acoustic signal would enhance the predictive power. This approach also provides us with the means to investigate the relative weights of the different principles involved in these grouping processes in order to understand their relative importance in musical contexts. These findings contribute to the creation of a framework for studying and understanding the perceptual characteristics of orchestration practice.

Lightweight Online Separation of the Sound Source of Interest through BLSTM-Based Binary Masking

Online audio source separation has beenan important part of auditory scene analysis and robotaudition. The main type of technique to carry thisout, because of its online capabilities, has been spatialfiltering (or beamforming), where it is assumed that thelocation (mainly, the direction of arrival; DOA) of thesource of interest (SOI) is known. However, thesetechniques suffer from considerable interference leakagein the final result. In this paper, we propose a two steptechnique: 1) a phase-based beamformer that provides,in addition to the estimation of the SOI, an estimationof the cumulative environmental interference; and 2) aBLSTM-based TF binary masking stage that calculatesa binary mask that aims to separate the SOI from thecumulative environmental interference. In our tests, thistechnique provides a signal-to-interference ratio (SIR)above 20 dB with simulated data. Because of thenature of the beamformer outputs, the label permutationproblem is handled from the beginning. This makes theproposed solution a lightweight alternative that requiresconsiderably less computational resources (almost anorder of magnitude) compared to current deep-learningbased techniques, while providing a comparable SIRperformance.

Impairments of auditory scene analysis in posterior cortical atrophy.

Although posterior cortical atrophy is often regarded as the canonical ‘visual dementia’, auditory symptoms may also be salient in this disorder. Patients often report particular difficulty hearing in busy environments; however, the core cognitive process—parsing of the auditory environment (‘auditory scene analysis’)—has been poorly characterized. In this cross-sectional study, we used customized perceptual tasks to assess two generic cognitive operations underpinning auditory scene analysis—sound source segregation and sound event grouping—in a cohort of 21 patients with posterior cortical atrophy, referenced to 15 healthy age-matched individuals and 21 patients with typical Alzheimer’s disease. After adjusting for peripheral hearing function and performance on control tasks assessing perceptual and executive response demands, patients with posterior cortical atrophy performed significantly worse on both auditory scene analysis tasks relative to healthy controls and patients with typical Alzheimer’s disease (all P < 0.05). Our findings provide further evidence of central auditory dysfunction in posterior cortical atrophy, with implications for our pathophysiological understanding of Alzheimer syndromes as well as clinical diagnosis and management.

Treefrogs exploit temporal coherence to form perceptual objects of communication signals.

For many animals, navigating their environment requires an ability to organize continuous streams of sensory input into discrete 'perceptual objects' that correspond to physical entities in visual and auditory scenes. The human visual and auditory systems follow several Gestalt laws of perceptual organization to bind constituent features into coherent perceptual objects. A largely unexplored question is whether nonhuman animals follow similar Gestalt laws in perceiving behaviourally relevant stimuli, such as communication signals. We used females of Cope's grey treefrog (Hyla chrysoscelis) to test the hypothesis that temporal coherence-a powerful Gestalt principle in human auditory scene analysis-promotes perceptual binding in forming auditory objects of species-typical vocalizations. According to the principle of temporal coherence, sound elements that start and stop at the same time or that modulate coherently over time are likely to become bound together into the same auditory object. We found that the natural temporal coherence between two spectral components of advertisement calls promotes their perceptual binding into auditory objects of advertisement calls. Our findings confirm the broad ecological validity of temporal coherence as a Gestalt law of auditory perceptual organization guiding the formation of biologically relevant perceptual objects in animal behaviour.

A two-stage deep learning algorithm for talker-independent speaker separation in reverberant conditions.

Speaker separation is a special case of speech separation, in which the mixture signal comprises two or more speakers. Many talker-independent speaker separation methods have been introduced in recent years to address this problem in anechoic conditions. To consider more realistic environments, this paper investigates talker-independent speaker separation in reverberant conditions. To effectively deal with speaker separation and speech dereverberation, extending the deep computational auditory scene analysis (CASA) approach to a two-stage system is proposed. In this method, reverberant utterances are first separated and separated utterances are then dereverberated. The proposed two-stage deep CASA system significantly outperforms a baseline one-stage deep CASA method in real reverberant conditions. The proposed system has superior separation performance at the frame level and higher accuracy in assigning separated frames to individual speakers. The proposed system successfully generalizes to an unseen speech corpus and exhibits similar performance to a talker-dependent system.

Origins of music in credible signaling.

Music comprises a diverse category of cognitive phenomena that likely represent both the effects of psychological adaptations that are specific to music (e.g., rhythmic entrainment) and the effects of adaptations for non-musical functions (e.g., auditory scene analysis). How did music evolve? Here, we show that prevailing views on the evolution of music - that music is a byproduct of other evolved faculties, evolved for social bonding, or evolved to signal mate quality - are incomplete or wrong. We argue instead that music evolved as a credible signal in at least two contexts: coalitional interactions and infant care. Specifically, we propose that (1) the production and reception of coordinated, entrained rhythmic displays is a co-evolved system for credibly signaling coalition strength, size, and coordination ability; and (2) the production and reception of infant-directed song is a co-evolved system for credibly signaling parental attention to secondarily altricial infants. These proposals, supported by interdisciplinary evidence, suggest that basic features of music, such as melody and rhythm, result from adaptations in the proper domain of human music. The adaptations provide a foundation for the cultural evolution of music in its actual domain, yielding the diversity of musical forms and musical behaviors found worldwide.

Gerçek Ortamlarda Artımlı Öğrenme ile Gerçek Zamanlı İşitsel Sahne Analizi

Continual learning for scene analysis is a continuous process to incrementally learn distinct events, actions, and even noise models from past experiences using different sensory modalities. In this paper, an Auditory Scene Analysis (ASA) approach based on a continual learning system is developed to incrementally learn the acoustic events in a dynamically-changing domestic environment. The events being salient sound sources are localized by a Sound Source Localization (SSL) method to robustly process the signals of the localized sound source in the domestic scene where multiple sources can co-exist. For real-time ASA, audio patterns are segmented from the acoustic signal stream of the localized source for extraction of the audio features, and construction of a feature set for each pattern. The continual learning is employed via a time-series algorithm, Hidden Markov Model (HMM), on these feature sets from acoustic signals stemming from the sources. The learning process is investigated by conducting a variety of experiments to evaluate the performance of Unknown Event Detection (UED), Acoustic Event Recognition (AER), and continual learning using a Hierarchical HMM algorithm. The Hierarchical HMM consists of two layers: 1) a lower layer in which AER is performed using an HMM for each event and the event-wise likelihood thresholds; and 2) an upper layer in which UED is achieved by one HMM with a suspicion threshold through the audio features with their proto symbols stemming from the lower layer HMMs. We verified the effectiveness of the proposed system capable of continual learning, AER and UED in terms of False-Positive Rates, True-Positive Rates, recognition accuracy and computational time to meet the demands in a learning task of multiple events in real-time. The effectiveness of the AER system has been verified with high accuracy, and a short retraining time in real-time ASA having nine different sounds.

An overview on informational masking

Background and Aim: In noisy environments, two types of masking including energetic mask­ing (EM) and informational masking (IM) occur. EM results from the spectral overlap of the target and maskers on the basilar membrane, while IM occurs at higher level. This paper aimed to rev­iew the concept of IM in terms of historical perspective and definitions, the important cues for releasing from it, age-related effects and its neural basis. Recent Findings: The data from psychoacoustic, behavioral, and neuro-imaging studies were revi­ewed and discussed in order to provide an overall image of IM. According to these studies, it seems that perceptual segregation between the target and maskers is the most important cues for relea­sing from IM. This process takes place simply and without any effort in adults with normal hearing; however, it does not occur easily in chil­dren, elderly people and those with impaired hearing. Moreover, it seems that both top-down and bottom-up processing are involved in IM for­mation. Conclusion: Since IM leads to failure in selec­tion of auditory objects and prevents the indivi­dual from auditory scene analysis, understanding the IM concept leads to a better knowledge of speech perception in noise. Keywords: Informational masking; perceptual masking; energetic masking

Robot Audition and Computational Auditory Scene Analysis

Robot audition aims at developing robot's ears that work in the real world, that is, machine listening of multiple sound sources. Its critical problem is noise. Speech interfaces have become more familiar and more indispensable as smartphones and artificial intelligence (AI) speakers spread. Their critical problems are noise and multiple simultaneous speakers. Recently two technological advances have contributed to significantly improve the performance of speech interfaces and robot audition. Emerging deep learning technology has improved noise robustness of automatic speech recognition, whereas microphone array processing has improved the performance of preprocessing such as noise reduction. Herein, an overview and history of robot audition are provided together with introduction of an open‐source software for robot audition and its wide applications in the real world. Also, it is discussed how robot audition contributes to the development of computational auditory scene analysis, that is, understanding of real‐world auditory environments.

A talker-independent deep learning algorithm to increase intelligibility for hearing-impaired listeners in reverberant competing talker conditions.

Deep learning based speech separation or noise reduction needs to generalize to voices not encountered during training and to operate under multiple corruptions. The current study provides such a demonstration for hearing-impaired (HI) listeners. Sentence intelligibility was assessed under conditions of a single interfering talker and substantial amounts of room reverberation. A talker-independent deep computational auditory scene analysis (CASA) algorithm was employed, in which talkers were separated and dereverberated in each time frame (simultaneous grouping stage), then the separated frames were organized to form two streams (sequential grouping stage). The deep neural networks consisted of specialized convolutional neural networks, one based on U-Net and the other a temporal convolutional network. It was found that every HI (and normal-hearing, NH) listener received algorithm benefit in every condition. Benefit averaged across all conditions ranged from 52 to 76 percentage points for individual HI listeners and averaged 65 points. Further, processed HI intelligibility significantly exceeded unprocessed NH intelligibility. Although the current utterance-based model was not implemented as a real-time system, a perspective on this important issue is provided. It is concluded that deep CASA represents a powerful framework capable of producing large increases in HI intelligibility for potentially any two voices.

Neural dynamics supporting auditory long-term memory effects on target detection

Auditory long-term memory has been shown to facilitate signal detection. However, the nature and timing of the cognitive processes supporting such benefits remain equivocal. We measured neuroelectric brain activity while young adults were presented with a contextual memory cue designed to assist with the detection of a faint pure tone target embedded in an audio clip of an everyday environmental scene (e.g., the soundtrack of a restaurant). During an initial familiarization task, participants heard such audio clips, half of which included a target sound (memory cue trials) at a specific time and location (left or right ear), as well as audio clips without a target (neutral trials). Following a 1-h or 24-h retention interval, the same audio clips were presented, but now all included a target. Participants were asked to press a button as soon as they heard the pure tone target. Overall, participants were faster and more accurate during memory than neutral cue trials. The auditory contextual memory effects on performance coincided with three temporally and spatially distinct neural modulations, which encompassed changes in the amplitude of event-related potential as well as changes in theta, alpha, beta and gamma power. Brain electrical source analyses revealed greater source activity in memory than neutral cue trials in the right superior temporal gyrus and left parietal cortex. Conversely, neutral trials were associated with greater source activity than memory cue trials in the left posterior medial temporal lobe. Target detection was associated with increased negativity (N2), and a late positive (P3b) wave at frontal and parietal sites, respectively. The effect of auditory contextual memory on brain activity preceding target onset showed little lateralization. Together, these results are consistent with contextual memory facilitating retrieval of target-context associations and deployment and management of auditory attentional resources to when the target occurred. The results also suggest that the auditory cortices, parietal cortex, and medial temporal lobe may be parts of a neural network enabling memory-guided attention during auditory scene analysis.

小特集「音環境理解の近年の動向」にあたって

小特集「音環境理解の近年の動向」にあたって