Target Speech Signal Research Articles

Sampling Rate Offset Estimation and Compensation for Distributed Adaptive Node-Specific Signal Estimation in Wireless Acoustic Sensor Networks

Sampling rate offsets (SROs) between devices in a heterogeneous wireless acoustic sensor network (WASN) can hinder the ability of distributed adaptive algorithms to perform as intended when they rely on coherent signal processing. In this paper, we present an SRO estimation and compensation method to allow the deployment of the distributed adaptive node-specific signal estimation (DANSE) algorithm in WASNs composed of asynchronous devices. The signals available at each node are first utilised in a coherence-drift-based method to blindly estimate SROs which are then compensated for via phase shifts in the frequency domain. A modification of the weighted overlap-add (WOLA) implementation of DANSE is introduced to account for SRO-induced full-sample drifts, permitting per-sample signal transmission via an approximation of the WOLA process as a time-domain convolution. The performance of the proposed algorithm is evaluated in the context of distributed noise reduction for the estimation of a target speech signal in an asynchronous WASN.

Ideal ratio mask estimation using supervised DNN approach for target speech signal enhancement

The most challenging process in recent Speech Enhancement (SE) systems is to exclude the non-stationary noises and additive white Gaussian noise in real-time applications. Several SE techniques suggested were not successful in real-time scenarios to eliminate noises in the speech signals due to the high utilization of resources. So, a Sliding Window Empirical Mode Decomposition including a Variant of Variational Model Decomposition and Hurst (SWEMD-VVMDH) technique was developed for minimizing the difficulty in real-time applications. But this is the statistical framework that takes a long time for computations. Hence in this article, this SWEMD-VVMDH technique is extended using Deep Neural Network (DNN) that learns the decomposed speech signals via SWEMD-VVMDH efficiently to achieve SE. At first, the noisy speech signals are decomposed into Intrinsic Mode Functions (IMFs) by the SWEMD Hurst (SWEMDH) technique. Then, the Time-Delay Estimation (TDE)-based VVMD was performed on the IMFs to elect the most relevant IMFs according to the Hurst exponent and lessen the low- as well as high-frequency noise elements in the speech signal. For each signal frame, the target features are chosen and fed to the DNN that learns these features to estimate the Ideal Ratio Mask (IRM) in a supervised manner. The abilities of DNN are enhanced for the categories of background noise, and the Signal-to-Noise Ratio (SNR) of the speech signals. Also, the noise category dimension and the SNR dimension are chosen for training and testing manifold DNNs since these are dimensions often taken into account for the SE systems. Further, the IRM in each frequency channel for all noisy signal samples is concatenated to reconstruct the noiseless speech signal. At last, the experimental outcomes exhibit considerable improvement in SE under different categories of noises.

Time-Domain Joint Training Strategies of Speech Enhancement and Intent Classification Neural Models.

Robustness against background noise and reverberation is essential for many real-world speech-based applications. One way to achieve this robustness is to employ a speech enhancement front-end that, independently of the back-end, removes the environmental perturbations from the target speech signal. However, although the enhancement front-end typically increases the speech quality from an intelligibility perspective, it tends to introduce distortions which deteriorate the performance of subsequent processing modules. In this paper, we investigate strategies for jointly training neural models for both speech enhancement and the back-end, which optimize a combined loss function. In this way, the enhancement front-end is guided by the back-end to provide more effective enhancement. Differently from typical state-of-the-art approaches employing on spectral features or neural embeddings, we operate in the time domain, processing raw waveforms in both components. As application scenario we consider intent classification in noisy environments. In particular, the front-end speech enhancement module is based on Wave-U-Net while the intent classifier is implemented as a temporal convolutional network. Exhaustive experiments are reported on versions of the Fluent Speech Commands corpus contaminated with noises from the Microsoft Scalable Noisy Speech Dataset, shedding light and providing insight about the most promising training approaches.

Neural Spectrospatial Filtering

As the most widely-used spatial filtering approach for multi-channel speech separation, beamforming extracts the target speech signal arriving from a specific direction. An emerging alternative approach is multi-channel complex spectral mapping, which trains a deep neural network (DNN) to directly estimate the real and imaginary spectrograms of the target speech signal from those of the multi-channel noisy mixture. In this all-neural approach, the trained DNN itself becomes a nonlinear, time-varying spectrospatial filter. However, it remains unclear how this approach performs relative to commonly-used beamforming techniques on different array configurations and acoustic environments. This paper is devoted to examining this issue in a systematic way. Comprehensive evaluations show that multi-channel complex spectral mapping achieves separation performance comparable to or better than beamforming for different array geometries and speech separation tasks and reduces to monaural complex spectral mapping in single-channel conditions, demonstrating the general utility of this approach on multi-channel and single-channel speech separation. In addition, such an approach is computationally more efficient than widely-used mask-based beamforming. We conclude that this neural spectrospatial filter provides a strong alternative to traditional and mask-based beamforming.

The Impact of Temporally Coherent Visual Cues on Speech Perception in Complex Auditory Environments.

Speech perception often takes place in noisy environments, where multiple auditory signals compete with one another. The addition of visual cues such as talkers’ faces or lip movements to an auditory signal can help improve the intelligibility of speech in those suboptimal listening environments. This is referred to as audiovisual benefits. The current study aimed to delineate the signal-to-noise ratio (SNR) conditions under which visual presentations of the acoustic amplitude envelopes have their most significant impact on speech perception. Seventeen adults with normal hearing were recruited. Participants were presented with spoken sentences in babble noise either in auditory-only or auditory-visual conditions with various SNRs at −7, −5, −3, −1, and 1 dB. The visual stimulus applied in this study was a sphere that varied in size syncing with the amplitude envelope of the target speech signals. Participants were asked to transcribe the sentences they heard. Results showed that a significant improvement in accuracy in the auditory-visual condition versus the audio-only condition was obtained at the SNRs of −3 and −1 dB, but no improvement was observed in other SNRs. These results showed that dynamic temporal visual information can benefit speech perception in noise, and the optimal facilitative effects of visual amplitude envelope can be observed under an intermediate SNR range.

Cortical Tracking of a Background Speaker Modulates the Comprehension of a Foreground Speech Signal.

Understanding speech in background noise is a difficult task. The tracking of speech rhythms such as the rate of syllables and words by cortical activity has emerged as a key neural mechanism for speech-in-noise comprehension. In particular, recent investigations have used transcranial alternating current stimulation (tACS) with the envelope of a speech signal to influence the cortical speech tracking, demonstrating that this type of stimulation modulates comprehension and therefore providing evidence of a functional role of the cortical tracking in speech processing. Cortical activity has been found to track the rhythms of a background speaker as well, but the functional significance of this neural response remains unclear. Here we use a speech-comprehension task with a target speaker in the presence of a distractor voice to show that tACS with the speech envelope of the target voice as well as tACS with the envelope of the distractor speaker both modulate the comprehension of the target speech. Because the envelope of the distractor speech does not carry information about the target speech stream, the modulation of speech comprehension through tACS with this envelope provides evidence that the cortical tracking of the background speaker affects the comprehension of the foreground speech signal. The phase dependency of the resulting modulation of speech comprehension is, however, opposite to that obtained from tACS with the envelope of the target speech signal. This suggests that the cortical tracking of the ignored speech stream and that of the attended speech stream may compete for neural resources.SIGNIFICANCE STATEMENT Loud environments such as busy pubs or restaurants can make conversation difficult. However, they also allow us to eavesdrop into other conversations that occur in the background. In particular, we often notice when somebody else mentions our name, even if we have not been listening to that person. However, the neural mechanisms by which background speech is processed remain poorly understood. Here we use transcranial alternating current stimulation, a technique through which neural activity in the cerebral cortex can be influenced, to show that cortical responses to rhythms in the distractor speech modulate the comprehension of the target speaker. Our results provide evidence that the cortical tracking of background speech rhythms plays a functional role in speech processing.

Transfer Subspace Learning for Unsupervised Cross-Corpus Speech Emotion Recognition

In many practical applications, a speech emotion recognition model learned on a source (training) domain but applied to a novel target (testing) domain degenerates even significantly due to the mismatch between the two domains. Aiming at learning a better speech emotion recognition model for the target domain, the paper investigates this interesting problem, i.e., unsupervised cross-corpus speech emotion recognition (SER), in which the training and testing speech signals come from two different speech emotion corpora. Meanwhile, the training speech signals are labeled, while the label information of the testing speech signals is entirely unknown. To deal with this problem, we propose a simple yet effective method called transfer subspace learning (TRaSL). TRaSL aims at learning a projection matrix with which we can transform the source and target speech signals from the original feature space to the label space. The transformed source and target speech signals in the label space would share similar feature distributions. Consequently, the classifier learned on the labeled source speech signals can effectively predict the emotional states of the unlabeled target speech signals. To evaluate the performance of the proposed TRaSL method, we carry out extensive cross-corpus SER experiments on four speech emotion corpora including IEMOCAP, EmoDB, eNTERFACE, and AFEW 4.0. Compared with recent state-of-the-art cross-corpus SER methods, the proposed TRaSL can achieve more satisfactory overall results.

An Overview of Deep-Learning-Based Audio-Visual Speech Enhancement and Separation

Speech enhancement and speech separation are two related tasks, whose purpose is to extract either one or more target speech signals, respectively, from a mixture of sounds generated by several sources. Traditionally, these tasks have been tackled using signal processing and machine learning techniques applied to the available acoustic signals. Since the visual aspect of speech is essentially unaffected by the acoustic environment, visual information from the target speakers, such as lip movements and facial expressions, has also been used for speech enhancement and speech separation systems. In order to efficiently fuse acoustic and visual information, researchers have exploited the flexibility of data-driven approaches, specifically deep learning, achieving strong performance. The ceaseless proposal of a large number of techniques to extract features and fuse multimodal information has highlighted the need for an overview that comprehensively describes and discusses audio-visual speech enhancement and separation based on deep learning. In this paper, we provide a systematic survey of this research topic, focusing on the main elements that characterise the systems in the literature: acoustic features; visual features; deep learning methods; fusion techniques; training targets and objective functions. In addition, we review deep-learning-based methods for speech reconstruction from silent videos and audio-visual sound source separation for non-speech signals, since these methods can be more or less directly applied to audio-visual speech enhancement and separation. Finally, we survey commonly employed audio-visual speech datasets, given their central role in the development of data-driven approaches, and evaluation methods, because they are generally used to compare different systems and determine their performance.

Blind Adaptive Mask to Improve Intelligibility of Non-Stationary Noisy Speech

This letter proposes a novel blind acoustic mask (BAM) designed to adaptively detect noise components and preserve target speech segments in time domain. A robust standard deviation estimator is applied to the non-stationary noisy speech to identify noise masking elements. The main contribution of the proposed solution is the use of this noise statistics to derive an adaptive information to define and select samples with lower noise proportion. Thus, preserving speech intelligibility. Additionally, no information of the target speech and noise signals statistics is previously required to this non-ideal mask. The BAM and three competitive methods, Ideal Binary Mask (IBM), Target Binary Mask (TBM), and Non-stationary Noise Estimation for Speech Enhancement (NNESE), are evaluated considering speech signals corrupted by three non-stationary acoustic noises and six values of signal-to-noise ratio (SNR). Results demonstrate that the BAM technique achieves intelligibility gains comparable to ideal masks while maintaining good speech quality.

Neural tracking of the speech envelope is differentially modulated by attention and language experience

The ability to selectively attend to a speech signal amid competing sounds is a significant challenge, especially for listeners trying to comprehend non-native speech. Attention is critical to direct neural processing resources to the most essential information. Here, neural tracking of the speech envelope of an English story narrative and cortical auditory evoked potentials (CAEPs) to non-speech stimuli were simultaneously assayed in native and non-native listeners of English. Although native listeners exhibited higher narrative comprehension accuracy, non-native listeners exhibited enhanced neural tracking of the speech envelope and heightened CAEP magnitudes. These results support an emerging view that although attention to a target speech signal enhances neural tracking of the speech envelope, this mechanism itself may not confer speech comprehension advantages. Our findings suggest that non-native listeners may engage neural attentional processes that enhance low-level acoustic features, regardless if the target signal contains speech or non-speech information.

Aligning Hearing Aid and Cochlear Implant Improves Hearing Outcome in Bimodal Cochlear Implant Users.

Patients with unilateral deafness and residual hearing on the contralateral ear can benefit from a cochlear implant (CI) on one side and a hearing aid (HA) on the other. However, hearing improvement among these patients is heterogenous. Interindividual differences in bimodal benefit may be caused by a mismatch of CI and HA. The aim of this study was to clinically apply a HA fitting strategy and to evaluate hearing outcome with and without a dedicated bimodal fitting formula. Prospective non-randomized study. Tertiary referral center. Twelve patients using a CI processor and a conventional HA were enrolled. Before and after the new HA had been adjusted to the patient and linked to the CI, pure-tone audiometry and localization tests were performed. Speech perception was determined in quiet and noise. Tests were repeated after 6 and 12 weeks. To evaluate the subjective listening comfort two questionnaires (Oldenburg Inventory and HISQUI19) were assessed. Therapeutic. Word recognition in quiet, sentence recognition in noise. Speech perception in noise improved significantly: directed suppression of noise helped to segregate the target speech signal from a mixture of sounds or competing speakers. Evaluation of the questionnaires revealed a positive subjective hearing experience compared with patients' initial settings of the devices. By linking CI and HA hearing and speech perception can be improved. However, good counselling at the outset is essential to obtain enhanced outcome.

Decoding the Attended Speaker From EEG Using Adaptive Evaluation Intervals Captures Fluctuations in Attentional Listening.

Listeners differ in their ability to attend to a speech stream in the presence of a competing sound. Differences in speech intelligibility in noise cannot be fully explained by the hearing ability which suggests the involvement of additional cognitive factors. A better understanding of the temporal fluctuations in the ability to pay selective auditory attention to a desired speech stream may help in explaining these variabilities. In order to better understand the temporal dynamics of selective auditory attention, we developed an online auditory attention decoding (AAD) processing pipeline based on speech envelope tracking in the electroencephalogram (EEG). Participants had to attend to one audiobook story while a second one had to be ignored. Online AAD was applied to track the attention toward the target speech signal. Individual temporal attention profiles were computed by combining an established AAD method with an adaptive staircase procedure. The individual decoding performance over time was analyzed and linked to behavioral performance as well as subjective ratings of listening effort, motivation, and fatigue. The grand average attended speaker decoding profile derived in the online experiment indicated performance above chance level. Parameters describing the individual AAD performance in each testing block indicated significant differences in decoding performance over time to be closely related to the behavioral performance in the selective listening task. Further, an exploratory analysis indicated that subjects with poor decoding performance reported higher listening effort and fatigue compared to good performers. Taken together our results show that online EEG based AAD in a complex listening situation is feasible. Adaptive attended speaker decoding profiles over time could be used as an objective measure of behavioral performance and listening effort. The developed online processing pipeline could also serve as a basis for future EEG based near real-time auditory neurofeedback systems.

Voice Conversion Using a Perceptual Criterion

In voice conversion (VC), it is highly desirable to obtain transformed speech signals that are perceptually close to a target speaker’s voice. To this end, a perceptually meaningful criterion where the human auditory system was taken into consideration in measuring the distances between the converted and the target voices was adopted in the proposed VC scheme. The conversion rules for the features associated with the spectral envelope and the pitch modification factor were jointly constructed so that perceptual distance measurement was minimized. This minimization problem was solved using a deep neural network (DNN) framework where input features and target features were derived from source speech signals and time-aligned version of target speech signals, respectively. The validation tests were carried out for the CMU ARCTIC database to evaluate the effectiveness of the proposed method, especially in terms of perceptual quality. The experimental results showed that the proposed method yielded perceptually preferred results compared with independent conversion using conventional mean-square error (MSE) criterion. The maximum improvement in perceptual evaluation of speech quality (PESQ) was 0.312, compared with the conventional VC method.

Design and evaluation of personal audio systems based on speech privacy constraints.

Personal audio refers to the generation of spatially distinct sound zones that allow individuals within a shared space to listen to their own audio material without affecting, or being affected, by others. Recent interest in such systems has focussed on their performance in public spaces where speech privacy is desirable. To achieve this goal, speech is focussed towards the target listener and a masking signal is focussed into the area where the target speech signal could otherwise be overheard. An effective masking signal must substantially reduce the intelligibility in this region without becoming an annoyance to those nearby. To assess these perceptual requirements, listening tests were carried out using two examples of loudspeaker arrays with different spatial aliasing characteristics, to determine the impacts of different masking signal spectra on speech intelligibility and subjective preference. The results of these tests were used, alongside objective and subjective metrics, to form a design specification for private personal audio systems.

Deep Learning for Talker-dependent Reverberant Speaker Separation: An Empirical Study.

Speaker separation refers to the problem of separating speech signals from a mixture of simultaneous speakers. Previous studies are limited to addressing the speaker separation problem in anechoic conditions. This paper addresses the problem of talker-dependent speaker separation in reverberant conditions, which are characteristic of real-world environments. We employ recurrent neural networks with bidirectional long short-term memory (BLSTM) to separate and dereverberate the target speech signal. We propose two-stage networks to effectively deal with both speaker separation and speech dereverberation. In the two-stage model, the first stage separates and dereverberates two-talker mixtures and the second stage further enhances the separated target signal. We have extensively evaluated the two-stage architecture, and our empirical results demonstrate large improvements over unprocessed mixtures and clear performance gain over single-stage networks in a wide range of target-to-interferer ratios and reverberation times in simulated as well as recorded rooms. Moreover, we show that time-frequency masking yields better performance than spectral mapping for reverberant speaker separation.

A new perceptually weighted cost function in deep neural network based speech enhancement systems

Speech intelligibility improvement is an important task to increase human perception in telecommunication systems and hearing aids when the speech is degraded by the background noises. Although, deep neural network (DNN) based learning architectures which use mean square error (MSE) as the cost function has been found to be very successful in speech enhancement areas, they typically attempt to enhance the speech quality by uniformly optimizing the separation of a target speech signal from a noisy observation over all frequency bands. In this work, we propose a new cost function which further focuses on speech intelligibility improvement based on a psychoacoustic model. The band-importance function, which is a principal component of speech intelligibility index (SII), has been used to determine the relative contribution to speech intelligibility provided by each frequency band in learning algorithm. In addition, we augment a signal to noise ratio (SNR) estimation to the network to improve the generalization of the method to unseen noisy conditions. The performance of the proposed MSE cost function is compared with the conventional MSE cost function in the same conditions. Our approach shows better performance in objective speech intelligibility measures such as coherence SII (CSII) and short-time objective intelligibility (STOI), while mitigating quality scores in perceptual evaluation of speech quality (PESQ) and speech distortion (SD) measure.

Consequences of interaural decorrelation of speech temporal fine structure

The ability to understand a target speech signal against a background of interfering speech signals is typically improved when the interfering signals are spatially separated (spatial release from masking; SRM). Swaminathan et al. (2016) found a significant reduction in SRM when the temporal fine structure (TFS) across the left and right ears was decorrelated, suggesting that binaural TFS provides important cues that support SRM. One interpretation is that degrading binaural TFS prevents the extraction of reliable interaural time differences (ITDs), which in turn leads to a reduction in SRM. We tested this hypothesis by systematically decorrelating the binaural TFS, and measuring the effects on both ITD discrimination for speech stimuli and SRM in normal hearing listeners. We show that decorrelation leads to a systematic increase in ITD discrimination thresholds, as well as a systematic decrease in SRM. This supports the idea that binaural TFS is needed to access ITD information in speech, which is in turn required for SRM. Additionally, the relationship between these two tasks provides a framework for determining the extent to which reduced SRM in listeners with hearing loss can be attributed to reduced sensitivity to binaural TFS.

Today's Hearing Technologies: Definitions, Regulations, Overlaps

Today's Hearing Technologies: Definitions, Regulations, Overlaps

Auditory distraction by speech: Can a babble masker restore working memory performance and subjective perception to baseline?

Background speech has a detrimental impact on employees’ privacy perception and cognitive performance in open-plan offices. Sound masking that covers speech sounds can help to improve the speech privacy and the ability to work undisturbed in open office environments. Recently, non-artificial sounds such as babble or water sounds have been suggested as masking sounds because they may be perceived as more natural and subsequently less annoying. This paper compares the working memory performance and annoyance perception during background speech that was masked by different babble sounds and a waterfall sound to speech that was masked by stationary noise with the frequency spectrum of the distracting background speech signal (target speech). In a first laboratory experiment the effects of different babble sounds were compared with the noise sound at a speech-to-noise ratio of −3 dB. All subjects had to complete a serial short-term memory task and a questionnaire that covered subjective ratings. All sound conditions with masked target speech produced similar error rates and annoyance ratings as unmasked target speech. In a second laboratory experiment, the speech-to-noise ratio of target speech that was masked by babble consisting of 48 target speech signals was varied between −6 and −12 dB in steps of 3 dB. Target based babble resulted in similar error rates as stationary noise at −6 dB speech-to-noise ratio but the findings suggest that target speech that is masked by babble might even be perceived as more annoying than speech that is masked by stationary target spectrum based noise.

Cortical entrainment to speech under competing-talker conditions: Effects of cognitive load due to presentation rate and task difficulty

Previous work has demonstrated that, when listeners attend to a target talker and ignore a distractor, their neural activity is more strongly entrained to the amplitude envelope of the target speech. Moreover, target-talker entrainment under these conditions can be higher for second-language listeners than native listeners, which might be due to the greater focused attention required to understand second-language speech. The present study manipulated the cognitive difficulty of the task in an attempt to make native-language listeners have entrainment results more like those of second-language listeners. Listeners heard simultaneous sentences spoken by male and female speakers, and had to detect semantically anomalous catch trials. The sentences had high or low semantic predictability (low-predictability sentences require more lexical processing to evaluate meaning) and were presented with short or long intervals between sentences (0.5 or 3.0 s, with shorter intervals forcing listeners to make a quicker evaluation of meaning). The preliminary results suggest that native-language listeners indeed have higher target-talker entrainment under more cognitively difficult conditions, suggesting that cognitive load may modulate focused attention to the target speech signal.