Adverse Acoustic Environments Research Articles

ANSD-MA-MSE: Adaptive Neural Speaker Diarization Using Memory-Aware Multi-Speaker Embedding

In this paper, we propose a neural speaker diarization (NSD) network architecture consisting of three key components. First, a memory-aware multi-speaker embedding (MA-MSE) mechanism is proposed to facilitate a dynamical refinement of speaker embedding to reduce a potential data mismatch between the speaker embedding extraction and the NSD network. Next, a speaker selection procedure is introduced to handle situations where the detected number of speakers is different from the assumed speaker size in the NSD network. Finally, an adaptive procedure is proposed to improve the required prior information for the nonoverlap speech segments in a given utterance during each iteration. We call our proposed framework adaptive neural speaker diarization with memory-aware multi-speaker embedding (ANSD-MA-MSE). Our method improves diarization performance in realistic operating scenarios, such as adverse acoustic environments, domain mismatches, and a varying, rather than fixed, number of speakers. Having been tested on both the AMI corpus and the DIHARD-III evaluation sets, our proposed approach consistently outperforms other state-of-the-art techniques in diarization error rates, including the results reported by the best single-model system in the DIHARD-III challenge. Our code is publicly available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/Maokui-He/NSD-MA-MSE</uri> .

AUC optimization for deep learning-based voice activity detection

Voice activity detection (VAD) based on deep neural networks (DNN) have demonstrated good performance in adverse acoustic environments. Current DNN-based VAD optimizes a surrogate function, e.g., minimum cross-entropy or minimum squared error, at a given decision threshold. However, VAD usually works on-the-fly with a dynamic decision threshold, and the receiver operating characteristic (ROC) curve is a global evaluation metric for VAD at all possible decision thresholds. In this paper, we propose to maximize the area under the ROC curve (MaxAUC) by DNN, which can maximize the performance of VAD in terms of the entire ROC curve. However, the objective of the AUC maximization is nondifferentiable. To overcome this difficulty, we relax the nondifferentiable loss function to two differentiable approximation functions—sigmoid loss and hinge loss. To study the effectiveness of the proposed MaxAUC-DNN VAD, we take either a standard feedforward neural network or a bidirectional long short-term memory network as the DNN model with either the state-of-the-art multi-resolution cochleagram or short-term Fourier transform as the acoustic feature. We conducted noise-independent training to all comparison methods. Experimental results show that taking AUC as the optimization objective results in higher performance than the common objectives of the minimum squared error and minimum cross-entropy. The experimental conclusion is consistent across different DNN structures, acoustic features, noise scenarios, training sets, and languages.

Microphone Array Beamforming With High Flexible Interference Attenuation and Noise Reduction

This paper studies the problem of microphone array beamforming to enhance a speech signal of interest in adverse acoustic environments, where interference and additive background noise coexist. The problem is formulated as one of convex optimization whose solution under a specified level of interference attenuation leads to an interference controlled maximum noise reduction (ICMR) beamformer, which can be expressed as a linear combination of two MVDR beamformers: one attempts to extract the desired source signal while the other attempts to extract the interference. The combination coefficients are functions of the array manifold vectors, noise coherence matrix, and the specified interference attenuation factor. By tuning the interference attenuation factor, the ICMR beamformer can be implemented to achieve aggressive interference attenuation or even eliminate interference completely; but this may lead to less additive noise suppression or even noise amplification. To control the maximum sacrifice in gain (SG) of the signal-to-noise ratio (SNR) that is acceptable for additive reduction, a variant of ICMR is derived, which is named as the ICMR-SG beamformer. Simulations are performed and the results show that the ICMR beamformer is able to control the amount of interference attenuation. In comparison, ICMR-SG controls the maximum SG of SNR while achieving the optimal possible level of interference attenuation.

Postural Control While Listening in Younger and Middle-Aged Adults.

The motivation for this research is to determine whether a listening-while-balancing task would be sensitive to quantifying listening effort in middle age. The premise behind this exploratory work is that a decrease in postural control would be demonstrated in challenging acoustic conditions, more so in middle-aged than in younger adults. A dual-task paradigm was employed with speech understanding as one task and postural control as the other. For the speech perception task, participants listened to and repeated back sentences in the presence of other sentences or steady-state noise. Targets and maskers were presented in both spatially-coincident and spatially-separated conditions. The postural control task required participants to stand on a force platform either in normal stance (with feet approximately shoulder-width apart) or in tandem stance (with one foot behind the other). Participants also rated their subjective listening effort at the end of each block of trials. Postural control was poorer for both groups of participants when the listening task was completed at a more adverse (vs. less adverse) signal-to-noise ratio. When participants were standing normally, postural control in dual-task conditions was negatively associated with degree of high-frequency hearing loss, with individuals who had higher pure-tone thresholds exhibiting poorer balance. Correlation analyses also indicated that reduced speech recognition ability was associated with poorer postural control in both single- and dual-task conditions. Middle-aged participants exhibited larger dual-task costs when the masker was speech, as compared to when it was noise. Individuals who reported expending greater effort on the listening task exhibited larger dual-task costs when in normal stance. Listening under challenging acoustic conditions can have a negative impact on postural control, more so in middle-aged than in younger adults. One explanation for this finding is that the increased effort required to successfully listen in adverse environments leaves fewer resources for maintaining balance, particularly as people age. These results provide preliminary support for using this type of ecologically-valid dual-task paradigm to quantify the costs associated with understanding speech in adverse acoustic environments.

Cross-modal plasticity in adult single-sided deafness revealed by alpha band resting-state functional connectivity

Single-sided deafness (SSD) or profound unilateral hearing loss is the condition where the transfer of acoustic information to the brain is restricted to one ear. SSD impairment is most evident under adverse acoustic environments with overlapping interference, which burdens cognitive resources. It is known that bilateral deafness induces cross-modal brain plasticity within visual cortical areas. Here we investigate whether similar cross-modal plasticity is observed in adult-onset SSD. In SSD patients (n ​= ​29) and matched controls (n ​= ​29) we estimated voxel level resting-state power and functional connectivity in the alpha band (8–12 ​Hz) from magnetoencephalography (MEG) data. We examined both global functional connectivity (mean functional connectivity of each voxel with the rest of the brain), and seeded functional connectivity of primary auditory cortices (A1), primary visual cortices (V1) and posterior cingulate cortex (PCC) of the default mode network (DMN). Power reduction was observed in left auditory cortex. Global functional connectivity showed reduction in frontal cortices and enhancement in visual cortex. Seeded functional connectivity of auditory cortices showed reduction in temporal, frontal and occipital regions, and enhancement in parietal cortex. Interestingly, seeded functional connectivity of visual cortices showed enhancement in visual cortices, inferior parietal lobe, post-central gyrus, and the precuneus, and reduction in auditory cortex. Seeded functional connectivity of PCC showed reduction in frontal cortical regions that are part of the DMN, attention, and working memory networks. Adult-onset SSD exhibited widespread cross-modal brain plasticity involving alterations in auditory, visual, attention, working memory and default mode networks.

Multiple Sound Source Counting and Localization Based on TF-Wise Spatial Spectrum Clustering

This paper addresses the problem of multiple sound source counting and localization in adverse acoustic environments, using microphone array recordings. The proposed time-frequency TF wise spatial spectrum clustering based method contains two stages. First, given the received sensor signals, the spatial correlation matrix is computed and denoised in the TF domain. The TF-wise spatial spectrum is estimated based on the signal subspace information, and further enhanced by an exponential transform, which can increase the reliability of the source presence possibility reflected by spatial spectrum. Second, to jointly count and localize sound sources, the enhanced TF-wise spatial spectra are divided into several clusters with each cluster corresponding to one source. Sources are successively detected by searching the significant peaks of the remaining global spatial spectrum, which is formed using unassigned spatial spectra. After each new source detection, spatial spectra are reassigned to detected sources according to the dominance association between them. The interaction between sources is reduced by iteratively performing new source detection and spatial spectrum assignment. Experiments on both simulated data and real-world data demonstrate the superiority of the proposed method for multiple sound source counting and localization in the environment with different levels of noise and reverberation.

Multimodal Voice Conversion Under Adverse Environment Using a Deep Convolutional Neural Network

This paper presents a voice conversion (VC) technique under noisy environments. Typically, VC methods use only audio information for conversion in a noiseless environment. However, existing conversion methods do not always achieve satisfactory results in an adverse acoustic environment. To solve this problem, we propose a multimodal voice conversion model based on a deep convolutional neural network (MDCNN) built by combining two convolutional neural networks (CNN) and a deep neural network (DNN) for VC under noisy environments. In the MDCNN, both the acoustic and visual information are incorporated into the voice conversion to improve its robustness in adverse acoustic conditions. The two CNNs are designed to extract acoustic and visual features, and the DNN is designed to capture the nonlinear mapping relation of source speech and target speech. Experimental results indicate that the proposed MDCNN outperforms two existing approaches in noisy environments.

Multi-Speaker DOA Estimation Using Deep Convolutional Networks Trained With Noise Signals

Supervised learning-based methods for source localization, being data driven, can be adapted to different acoustic conditions via training and have been shown to be robust to adverse acoustic environments. In this paper, a convolutional neural network (CNN) based supervised learning method for estimating the direction of arrival (DOA) of multiple speakers is proposed. Multi-speaker DOA estimation is formulated as a multi-class multi-label classification problem, where the assignment of each DOA label to the input feature is treated as a separate binary classification problem. The phase component of the short-time Fourier transform (STFT) coefficients of the received microphone signals are directly fed into the CNN, and the features for DOA estimation are learnt during training. Utilizing the assumption of disjoint speaker activity in the STFT domain, a novel method is proposed to train the CNN with synthesized noise signals. Through experimental evaluation with both simulated and measured acoustic impulse responses, the ability of the proposed DOA estimation approach to adapt to unseen acoustic conditions and its robustness to unseen noise type is demonstrated. Through additional empirical investigation, it is also shown that with an array of M microphone our proposed framework yields the best localization performance with M-1 convolution layers. The ability of the proposed method to accurately localize speakers in a dynamic acoustic scenario with varying number of sources is also shown.

An Improved Bi-Level Thresholding Based Uncertainty Evaluation for Speech Enhancement in Non-Stationary Noises

This paper proposes a new speech enhancement framework to improve the quality of speeches recorded under adverse acoustic environments based on the speech presence uncertainty. Since the uncertainty evaluation gives a more and clear discrimination about the speech and noise, this paper proposes a new uncertainty evaluation mechanism as a preprocessing mechanism to the noise suppression methods. This mechanism relates with energies of a noisy speech signal and classifies the speech segments and noise segments more perfectly. In addition to the quality enhancement, this approach also reduces the unnecessary computational burden over the speech processing system. Extensive simulations are carried out over the speech signals with different types of non-stationary noises like babble noise, exhibition noise, restaurant noise and train station noises and the performance is measured with the performance metrics namely the Output SNR, AvgSegSNR, PESQ and COMP. The comparative analysis of proposed approach over the conventional approaches shows an outstanding performance in all environments.

Speech learning in noise or quiet

Noise made speech perception much more challenging for non-native listeners than for native listeners. We proposed that speech training in noise may better improve non-native listeners’speech perception. In the present study, three groups of native Chinese listeners were trained on English vowel perception under three conditions including quiet (TIQ), noise (TIN), and watching videos in English as an active control (C). Vowel identification in quiet and noise was assessed before, right after training, and three months later after training. Results showed that all listeners significantly improved their vowel identification after training, and the training effect was retained three months later. Compared with the control group, the TIQ group improved more for vowel identification in quiet condition, but not for that in noise conditions. The TIN group improved vowel identification in noise significantly more than the TIQ and control groups. Compared with the TIQ，less informational masking and energetic masking was also found in the post-training test for the TIN than that for the TIQ. These results suggested that vowel perceptual training in noise may better improve L2 vowel perception in adverse acoustic environments than training in quiet. Implications for second language speech perception training were discussed.

Perception of keywords elicited under various adverse acoustic environments

In a recent study, keywords were elicited in spontaneous speech samples from eight pairs of interlocutors who completed the “spot-the-difference” Diapix task in one of three acoustic environments: quiet, 2-talker babble, or 8-talker babble. In some conditions, talkers were in the same acoustic environment and in others they were in disparate environments. In all cases, the acoustic environments were presented over headphones in order to obtain clean recordings of the speech but maintain the influence of the environment on production. The current study investigated the perception of the keywords in each of the three acoustic environments at different signal-to-noise ratios, when they were presented in either the same or disparate adverse environment than ones in which they were elicited. Results from the current experiment will be discussed with regard to intelligibility obtained when words were presented in the same vs. different environments; the differences obtained in perception might indicate whether speech modifications in adverse environments are specific to a particular environment or more global in nature. The correlations between perception results to acoustic measures of these keywords in these various acoustic environment will also identify key factors that might improve communication effectiveness in adverse environments.

Cortical Measures of Binaural Processing Predict Spatial Release from Masking Performance.

Binaural sensitivity is an important contributor to the ability to understand speech in adverse acoustical environments such as restaurants and other social gatherings. The ability to accurately report on binaural percepts is not commonly measured, however, as extensive training is required before reliable measures can be obtained. Here, we investigated the use of auditory evoked potentials (AEPs) as a rapid physiological indicator of detection of interaural phase differences (IPDs) by assessing cortical responses to 180° IPDs embedded in amplitude-modulated carrier tones. We predicted that decrements in encoding of IPDs would be evident in middle age, with further declines found with advancing age and hearing loss. Thus, participants in experiment #1 were young to middle-aged adults with relatively good hearing thresholds while participants in experiment #2 were older individuals with typical age-related hearing loss. Results revealed that while many of the participants in experiment #1 could encode IPDs in stimuli up to 1,000 Hz, few of the participants in experiment #2 had discernable responses to stimuli above 750 Hz. These results are consistent with previous studies that have found that aging and hearing loss impose frequency limits on the ability to encode interaural phase information present in the fine structure of auditory stimuli. We further hypothesized that AEP measures of binaural sensitivity would be predictive of participants' ability to benefit from spatial separation between sound sources, a phenomenon known as spatial release from masking (SRM) which depends upon binaural cues. Results indicate that not only were objective IPD measures well correlated with and predictive of behavioral SRM measures in both experiments, but that they provided much stronger predictive value than age or hearing loss. Overall, the present work shows that objective measures of the encoding of interaural phase information can be readily obtained using commonly available AEP equipment, allowing accurate determination of the degree to which binaural sensitivity has been reduced in individual listeners due to aging and/or hearing loss. In fact, objective AEP measures of interaural phase encoding are actually better predictors of SRM in speech-in-speech conditions than are age, hearing loss, or the combination of age and hearing loss.

Speech Recognition Algorithm in Complex Noisy Environments Based on Multi-Space Compensation

Speech recognition rate drops significantly when interfered by noise in complex environment. In order to improve the accuracy and the robustness of the speech recognition in adverse acoustical environments, this paper reviewed the main problems of noise robust speech recognition, proposed a multi-space compensation algorithm which from signal-space, feature-space and model-space based on wiener filter, histogram equalization and vector Taylor series. Theory analyses and experiment results show that the proposed method can overcome the defect of the sharp descent of speech recognition rate of existing speech recognition algorithm interfered by environmental noise and improve the accuracy and the robustness in adverse acoustical environments. The algorithm provides the theoretical support for the speech recognition in airport, station, wharf and other complex noise environment.

Playback attack detection for text-dependent speaker verification over telephone channels

Playback attacks constitute one of the biggest threats in biometric speaker verification systems, in which a previously recorded passphrase is played back by an unprivileged person in order to gain access. This paper features a description of the playback attack detection (PAD) algorithm, designed to protect text-dependent speaker verification systems from the aforementioned spoofing attacks. The paper also describes the usage of spectral landmarks and score normalization methods in the playback detection algorithm. Different factors are discussed in terms of the performance of the algorithm. The authors investigate two issues: (1) extracting the PAD features which are robust against channel variations and (2) the robustness of the algorithm in adverse acoustical environments (e.g. office, street, cocktail party noise). The experiments are performed on a prepared speech corpus containing 4187 occurrences of a passphrase spoken by 175 speakers. The results of the experiment show the equal error rate (EER) to be as low as 1.0%. These findings demonstrate that such spoofing-oriented playback attacks can be effectively detected and should not be considered a significant argument against applications of text-dependent speaker verification.

Evaluations on underdetermined blind source separation in adverse environments using time-frequency masking

The successful implementation of speech processing systems in the real world depends on its ability to handle adverse acoustic conditions with undesirable factors such as room reverberation and background noise. In this study, an extension to the established multiple sensors degenerate unmixing estimation technique (MENUET) algorithm for blind source separation is proposed based on the fuzzy c-means clustering to yield improvements in separation ability for underdetermined situations using a nonlinear microphone array. However, rather than test the blind source separation ability solely on reverberant conditions, this paper extends this to include a variety of simulated and real-world noisy environments. Results reported encouraging separation ability and improved perceptual quality of the separated sources for such adverse conditions. Not only does this establish this proposed methodology as a credible improvement to the system, but also implies further applicability in areas such as noise suppression in adverse acoustic environments.

The MoveOn database: motorcycle environment speech and noise database for command and control applications

The MoveOn speech and noise database was purposely designed and implemented in support of research on spoken dialogue interaction in a motorcycle environment. The distinctiveness of the MoveOn database results from the requirements of the application domain—an information support and operational command and control system for the two-wheel police force—and also from the specifics of the adverse open-air acoustic environment. In this article, we first outline the target application, motivating the database design and purpose, and then report on the implementation details. The main challenges related to the choice of equipment, the organization of recording sessions, and some difficulties that were experienced during this effort, are discussed. We offer a detailed account of the database statistics, the suggested data splits in subsets, and discuss results from automatic speech recognition experiments which illustrate the degree of complexity of the operational environment.

Room acoustic effects on speech comprehension by English-as-a-second-language versus native English-speaking listeners

English-as-a-second-language (ESL) listeners are generally more impaired than native English-speaking listeners on speech intelligibility tasks under reverberation and noise; however, more study is needed to ascertain these groups’ performances on speech comprehension tasks, rather than intelligibility. A recent study (Valente et al, 2012) showed that speech comprehension by both native English-speaking adults and children was more negatively affected by adverse acoustic environments than sentence recognition. The current project investigates the speech comprehension of both ESL and native English-speaking adult listeners under combinations of reverberation and noise. Sets of 15-minute long listening comprehension tests based on the format of the Test of English for International Communication (TOEIC) were developed, anechoically recorded, convolved with binaural room impulse responses (BRIRs) to produce five mid-frequency reverberation times (0.4 to 1.1 seconds), and presented over loudspeakers in the Nebraska acoustics chamber. Background noise was also varied at three levels (RC-30, 40 and 50) through an Armstrong i-Ceiling system. In total then, subjects were individually exposed to 15 acoustic combinations of reverberation and background noise. Preliminary results will be presented and compared between ESL and native English listeners. [Work supported by a UNL Durham School Seed Grant and the Paul S. Veneklasen Research Foundation.]

A stereophonic acoustic signal extraction scheme for noisy and reverberant environments

In this contribution, a novel two-channel acoustic front-end for robust automatic speech recognition in adverse acoustic environments with nonstationary interference and reverberation is proposed. From a MISO system perspective, a statistically optimum source signal extraction scheme based on the multichannel Wiener filter (MWF) is discussed for application in noisy and underdetermined scenarios. For free-field and diffuse noise conditions, this optimum scheme reduces to a Delay & Sum beamformer followed by a single-channel Wiener postfilter. Scenarios with multiple simultaneously interfering sources and background noise are usually modeled by a diffuse noise field. However, in reality, the free-field assumption is very weak because of the reverberant nature of acoustic environments. Therefore, we propose to estimate this simplified MWF solution in each frequency bin separately to cope with reverberation. We show that this approach can very efficiently be realized by the combination of a blocking matrix based on semi-blind source separation (‘directional BSS’), which provides a continuously updated reference of all undesired noise and interference components separated from the desired source and its reflections, and a single-channel Wiener postfilter. Moreover, it is shown, how the obtained reference signal of all undesired components can efficiently be used to realize the Wiener postfilter, and at the same time, generalizes well-known postfilter realizations. The proposed front-end and its integration into an automatic speech recognition (ASR) system are analyzed and evaluated in noisy living-room-like environments according to the PASCAL CHiME challenge. A comparison to a simplified front-end based on a free-field assumption shows that the introduced system substantially improves the speech quality and the recognition performance under the considered adverse conditions.

Neurophysiological Influence of Musical Training on Speech Perception

Does musical training affect our perception of speech? For example, does learning to play a musical instrument modify the neural circuitry for auditory processing in a way that improves one's ability to perceive speech more clearly in noisy environments? If so, can speech perception in individuals with hearing loss (HL), who struggle in noisy situations, benefit from musical training? While music and speech exhibit some specialization in neural processing, there is evidence suggesting that skills acquired through musical training for specific acoustical processes may transfer to, and thereby improve, speech perception. The neurophysiological mechanisms underlying the influence of musical training on speech processing and the extent of this influence remains a rich area to be explored. A prerequisite for such transfer is the facilitation of greater neurophysiological overlap between speech and music processing following musical training. This review first establishes a neurophysiological link between musical training and speech perception, and subsequently provides further hypotheses on the neurophysiological implications of musical training on speech perception in adverse acoustical environments and in individuals with HL.

Factors affecting the intelligibility of recorded speech: Considerations for forensic audio “best evidence”.

Derived from a traditional common law rule of evidence, the “best evidence” standard as applied to recorded audio prescribes that an original recording, and not a duplicated or altered copy, will be presented in legal proceedings. The intent of this standard is to ensure that the integrity of the original evidence is preserved, such that a court is reasonably assured that it is being presented with the most complete and accurate record of the evidence. However, when considering forensic audio recordings of speech, which are frequently made in adverse acoustic environments, presentation of such recordings in their original form may not afford a court with the opportunity for a complete and accurate assessment of the evidence in question—namely, what words are being spoken on the recording? The current paper summarizes the technological and listener-based factors that should be considered when speech intelligibility is of prime importance in meeting the best evidence standard for presentation of forensic audio in court proceedings. Illustrative examples from recent court cases will be provided.