Speech Mixtures Research Articles

MixCycle: Unsupervised Speech Separation via Cyclic Mixture Permutation Invariant Training

We introduce two unsupervised source separation methods, which involve self-supervised training from single-channel two-source speech mixtures. Our first method, mixture permutation invariant training (MixPIT), enables learning a neural network model which separates the underlying sources via a challenging proxy task without supervision from the reference sources. Our second method, cyclic mixture permutation invariant training (MixCycle), uses MixPIT as a building block in a cyclic fashion for continuous learning. MixCycle gradually converts the problem from separating mixtures of mixtures into separating single mixtures. We compare our methods to common supervised and unsupervised baselines: permutation invariant training with dynamic mixing (PIT-DM) and mixture invariant training (MixIT). We show that MixCycle outperforms MixIT and reaches a performance level very close to the supervised baseline (PIT-DM) while circumventing the over-separation issue of MixIT. Also, we propose a self-evaluation technique inspired by MixCycle that estimates model performance without utilizing any reference sources. We show that it yields results consistent with an evaluation on reference sources (LibriMix) and also with an informal listening test conducted on a real-life mixtures dataset (REAL-M).

Heterogeneous Separation Consistency Training for Adaptation of Unsupervised Speech Separation

Recently, supervised speech separation has made great progress. However, limited by the nature of supervised training, most existing separation methods require ground-truth sources and are trained on synthetic datasets. This ground-truth reliance is problematic, because the ground-truth signals are usually unavailable in real conditions. Moreover, in many industry scenarios, the real acoustic characteristics deviate far from the ones in simulated datasets. Therefore, the performance usually degrades significantly when applying the supervised speech separation models to real applications. To address these problems, in this study, we propose a novel separation consistency training, termed SCT, to exploit the real-world unlabeled mixtures for improving cross-domain unsupervised speech separation in an iterative manner, by leveraging upon the complementary information obtained from heterogeneous (structurally distinct but behaviorally complementary) models. SCT follows a framework using two heterogeneous neural networks (HNNs) to produce high confidence pseudo labels of unlabeled real speech mixtures. These labels are then updated, and used to refine the HNNs to produce more reliable consistent separation results for real mixture pseudo-labeling. To maximally utilize the large complementary information between different separation networks, a cross-knowledge adaptation is further proposed. Together with simulated dataset, those real mixtures with high confidence pseudo labels are then used to update the HNN separation models iteratively. In addition, we find that combing the heterogeneous separation outputs by a simple linear fusion can further slightly improve the final system performance.

Enhancing the correlation between the quality and intelligibility objective metrics with the subjective scores by shallow feed forward neural network for time–frequency masking speech separation algorithms

Multiple objective metrics are in use by the researchers to evaluate the performance of separation systems. In this paper, we investigate the correlation of different state-of-the-art widely used objective evaluation metrics with the subjective evaluation for different time–frequency (TF) masking based binaural speech separation algorithms and find the metrics which correlate best with the human perception of quality and intelligibility for such algorithms. We separate a speech source from a speech mixture by a speech separation algorithm, and evaluate its quality and intelligibility by using the objective evaluation metrics. Then we carry out the subjective listening test of this estimated speech source on a large number of participants. For each algorithm, we repeat this process for 10 mixtures and then find the correlation between the objective and average subjective score of all the participants by Pearson’s correlation coefficient and calculate the statistical significance of these results. We also rank the separation algorithms both by the subjective and the objective testing. The results show that none of the existing objective metrics for judging the quality and intelligibility of the speech separated by TF masking based binaural source separation algorithms correlates well with the human perception. We then use shallow feed forward (FF) classification and regression neural networks (NNs) to enhance this correlation, enabling almost 87% and 88% classification accuracy for the quality and intelligibility objective metrics respectively by using the classification NN, and 98% correlation with the average subjective quality scores by using the regression NN.

Smartphone-based single-channel speech enhancement application for hearing aids.

This work presents a single-channel speech enhancement (SE) framework based on the super-Gaussian extension of the joint maximum a posteriori (SGJMAP) estimation rule. The developed SE algorithm is an open-source research smartphone-based application for hearing improvement studies. In this algorithm, the SGJMAP-based estimation for noisy speech mixture is smoothed along the frequency axis by a Mel filter-bank, resulting in a Mel-warped frequency-domain SGJMAP estimation. The impulse response of this Mel-warped estimation is obtained by applying a Mel-warped inverse discrete cosine transform (Mel-IDCT). This helps in filtering out the background noise and enhancing the speech signal. The proposed application is implemented on an iPhone (Apple, Cupertino, CA) to operate in real time and tested with normal-hearing (NH) and hearing-impaired (HI) listeners with different types of hearing aids through wireless connectivity. The objective speech quality and intelligibility test results are used to compare the performance of the proposed algorithm to existing conventional single-channel SE methods. Additionally, test results from NH and HI listeners show substantial improvement in speech recognition with the developed method in simulated real-world noisy conditions at different signal-to-noise ratio levels.

Underdetermined blind source separation of speech mixtures unifying dictionary learning and sparse representation

Underdetermined blind source separation of speech mixtures unifying dictionary learning and sparse representation

Speech signal enhancement in cocktail party scenarios by deep learning based virtual sensing of head-mounted microphones

The cocktail party effect refers to the human sense of hearing’s ability to pay attention to a single conversation while filtering out all other background noise. To mimic this human hearing ability for people with hearing loss, scientists integrate beamforming algorithms into the signal processing path of hearing aids or implants’ audio processors. Although these algorithms’ performance strongly depends on the number and spatial arrangement of the microphones, most devices are equipped with a small number of microphones mounted close to each other on the audio processor housing. We measured and evaluated the impact of the number and spatial arrangement of hearing aid or head-mounted microphones on the performance of the established Minimum Variance Distortionless Response beamformer in cocktail party scenarios. The measurements revealed that the optimal microphone placement exploits monaural cues (pinna-effect), is close to the target signal, and creates a large distance spread due to its spatial arrangement. However, this microphone placement is impractical for hearing aid or implant users, as it includes microphone positions such as on the forehead. To overcome microphones’ placement at impractical positions, we propose a deep virtual sensing estimation of the corresponding audio signals. The results of objective measures and a subjective listening test with 20 participants showed that the virtually sensed microphone signals significantly improved the speech quality, especially in cocktail party scenarios with low signal-to-noise ratios. Subjective speech quality was assessed using a 3-alternative forced choice procedure to determine which of the presented speech mixtures was most pleasant to understand. Hearing aid and cochlear implant (CI) users might benefit from the presented approach using virtually sensed microphone signals, especially in noisy environments.

Space-time characterization of community noise and sound sources in Accra, Ghana

Urban noise pollution is an emerging public health concern in growing cities in sub-Saharan Africa (SSA), but the sound environment in SSA cities is understudied. We leveraged a large-scale measurement campaign to characterize the spatial and temporal patterns of measured sound levels and sound sources in Accra, Ghana. We measured sound levels and recorded audio clips at 146 representative locations, involving 7-days (136 locations) and 1-year measurements between 2019 and 2020. We calculated metrics of noise levels and intermittency and analyzed audio recordings using a pre-trained neural network to identify sources. Commercial, business, and industrial areas and areas near major roads had the highest median daily sound levels (LAeq24hr: 69 dBA and 72 dBA) and the lowest percentage of intermittent sound; the vice-versa was found for peri urban areas. Road-transport sounds dominated the overall sound environment but mixtures of other sound sources, including animals, human speech, and outdoor music, dominated in various locations and at different times. Environmental noise levels in Accra exceeded both international and national health-based guidelines. Detailed information on the acoustical environmental quality (including sound levels and types) in Accra may guide environmental policy formulation and evaluation to improve the health of urban residents.

Effects of Age on Cortical Tracking of Word-Level Features of Continuous Competing Speech.

Speech-in-noise comprehension difficulties are common among the elderly population, yet traditional objective measures of speech perception are largely insensitive to this deficit, particularly in the absence of clinical hearing loss. In recent years, a growing body of research in young normal-hearing adults has demonstrated that high-level features related to speech semantics and lexical predictability elicit strong centro-parietal negativity in the EEG signal around 400 ms following the word onset. Here we investigate effects of age on cortical tracking of these word-level features within a two-talker speech mixture, and their relationship with self-reported difficulties with speech-in-noise understanding. While undergoing EEG recordings, younger and older adult participants listened to a continuous narrative story in the presence of a distractor story. We then utilized forward encoding models to estimate cortical tracking of four speech features: (1) word onsets, (2) “semantic” dissimilarity of each word relative to the preceding context, (3) lexical surprisal for each word, and (4) overall word audibility. Our results revealed robust tracking of all features for attended speech, with surprisal and word audibility showing significantly stronger contributions to neural activity than dissimilarity. Additionally, older adults exhibited significantly stronger tracking of word-level features than younger adults, especially over frontal electrode sites, potentially reflecting increased listening effort. Finally, neuro-behavioral analyses revealed trends of a negative relationship between subjective speech-in-noise perception difficulties and the model goodness-of-fit for attended speech, as well as a positive relationship between task performance and the goodness-of-fit, indicating behavioral relevance of these measures. Together, our results demonstrate the utility of modeling cortical responses to multi-talker speech using complex, word-level features and the potential for their use to study changes in speech processing due to aging and hearing loss.

A RobustICA-based algorithmic system for blind separation of convolutive mixtures

In this paper, we propose a frequency-domain method employing robust independent component analysis (RICA) to address the multichannel Blind Source Separation (BSS) problem of convolutive speech mixtures in highly reverberant environments. We impose regularization processes to tackle the ill-conditioning problem of the covariance matrix and to mitigate the performance degradation. We evaluate the impact of several parameters on the performance of separation, e.g., windowing type and overlapping ratio of the frequency domain method. We then assess and compare different techniques to solve the frequency-domain permutation ambiguity. Furthermore, we develop an algorithm to separate the source signals in adverse conditions, i.e. high reverberation conditions when short observation signals are available. Finally, through extensive simulations and real-world experiments, we evaluate and demonstrate the superiority of the presented convolutive algorithmic system in comparison to other BSS algorithms, including recursive regularized ICA (RR-ICA) and independent vector analysis (IVA).

Blind separation of underdetermined Convolutive speech mixtures by time–frequency masking with the reduction of musical noise of separated signals

The main focus of this paper is the separation of underdetermined convolutive blind speech in a multi-speaker environment. We present a method based on mask prediction in the time-frequency domain. Firstly, depending on the sparsity of signals in the time-frequency (TF) domain, we extimate speakers’ masks by clustering the relative absolute and Hermitian angle features extracted from the frequency components of the mixtures. Speech separation algorithms that are based on the sparsity and disjoint orthogonality of the speech signals in the time-frequency domain are not efficient when more than one source is active. Hence, in this paper, the cluster centers are estimated mostly based on the TF units that probably have only one active source. The correlations between the estimated masks, belonging to adjacent frequency bins, are leveraged to solve the permutation problem. To increase the accuracy, we have zeroed the value of masks at the TF unit without any active source. Moreover, in clustering, we employ a weighting function to consider the parts of masks that probably contains just one active source. Finally, in order to decrease the musical noise of the separated signals and improve their quality, sparse filters in the time-domain are utilized to re-estimate the separated signals. Performance of the proposed method is evaluated by a number of simulated and real speech signals. The simulated experiments were performed using a public dataset and Roomsim simulator. Compared the proposed method with some conventional algorithms, we observed that our separation method is more accurate than other approaches.

Using the phase inversion method and loudness comparisons for the evaluation of noise reduction algorithms in hearing aids

The phase inversion method, a technical measurement procedure, is often used to evaluate the performance of noise reduction algorithms in hearing aids. However, a detailed comparison of these technical measurements with the perceived loudness is missing. Therefore, commercially available hearing aids of six different manufacturers were evaluated technically and in a study with 18 normal-hearing listeners. First, the output signals of the hearing aids with and without activated noise reduction were recorded in a test box. Then, the test subjects evaluated the perceived loudness of these recordings within multiple two alternative forced choice (2-AFC) tasks. During one task, the test subjects had to focus either on the speech or noise signal and were asked to select the louder of two signals, which both contained a mixture of speech and noise. These results provide not only the perceived SNR but also the perceived speech and noise levels. Comparing the results of the 2-AFC tasks and the phase inversion method basically shows good agreement. Nevertheless, a simple computation of the sound pressure level can lead to significant deviations. Therefore, another possibility for the analysis of the results of the phase inversion method to better match the perceived loudness is presented.

Learning Optimum Number of Bases for Indian Languages in Non-negative Matrix Factorization based Multilingual Speech Separation

Non-negative matrix factorization-based audio source separation separating a target source has shown significant performance improvement when the spectral bases attained after factorization exhibits latent structures in the mixed audio signal comprising multiple speaker sources. If all the sources are known, the spectral bases may be inferred on priority by using a training process on the database of isolated sources. The number of bases inferred for a source should not include bases matching spectral patterns of the interfering sources in the audio mixture; otherwise, the estimated target source after separation will be incorporated with undesirable spectral patterns. It is difficult to distinguish and separate similar audio sources in an overlapped speech, leading to a complex speech processing task. Therefore, this research attempts to learn an optimum number of bases for Indian languages leading to successful separation of target source in multi-lingual multiple speaker speech mixtures using non-negative matrix factorization. The languages used for utterances are Hindi, Marathi, Gujarati, and Bengali. The speaker combinations used are female-female, male-male, and female-male. The optimum number of bases which was determined by evaluating improvement in the separation performance was found to be 40 for all the languages considered.

The effect of fundamental frequency contour similarity on multi-talker listening in older and younger adults.

Older adults with hearing loss have greater difficulty recognizing target speech in multi-talker environments than young adults with normal hearing, especially when target and masker speech streams are perceptually similar. A difference in fundamental frequency (f0) contour depth is an effective stream segregation cue for young adults with normal hearing. This study examined whether older adults with varying degrees of sensorineural hearing loss are able to utilize differences in target/masker f0 contour depth to improve speech recognition in multi-talker listening. Speech recognition thresholds (SRTs) were measured for speech mixtures composed of target/masker streams with flat, normal, and exaggerated speaking styles, in which f0 contour depth systematically varied. Computational modeling estimated differences in energetic masking across listening conditions. Young adults had lower SRTs than older adults; a result that was partially explained by differences in audibility predicted by the model. However, audibility differences did not explain why young adults experienced a benefit from mismatched target/masker f0 contour depth, while in most conditions, older adults did not. Reduced ability to use segregation cues (differences in target/masker f0 contour depth), and deficits grouping speech with variable f0 contours likely contribute to difficulties experienced by older adults in challenging acoustic environments.

Single‐channel dereverberation and denoising based on lower band trained SA‐LSTMs

The supervised single-channel speech enhancement presents one mixture recording at the input of the neural network and updates network parameters in order to generate an output as the reconstructed speech signal. However, current neural networks-based single-channel speech enhancement methods are not able to fully utilise pertinence with the specific frequency range of speech signals with limited computational complexity. In this study, the authors studied the power spectral density of mixtures with human speech and noise interferences. Based on the theory that the speech signal distributes at the lower band, they proposed a method to train signal approximation (SA) based neural networks with the lower frequency band of the speech mixture to improve the performance. To realise the lower band approach for single-channel speech enhancement, the method uses a long short-term memory (LSTM) block to exploit short-time Fourier transform of the desired frequency range. Furthermore, in order to improve the speech enhancement performance within reverberant room environments, the dereverberation mask and the enhanced ratio mask are exploited as the training targets of two LSTM blocks, respectively. The detailed evaluations confirm that the proposed method outperforms the state-of-the-art methods.

Brain-informed speech separation (BISS) for enhancement of target speaker in multitalker speech perception

Hearing-impaired people often struggle to follow the speech stream of an individual talker in noisy environments. Recent studies show that the brain tracks attended speech and that the attended talker can be decoded from neural data on a single-trial level. This raises the possibility of “neuro-steered” hearing devices in which the brain-decoded intention of a hearing-impaired listener is used to enhance the voice of the attended speaker from a speech separation front-end. So far, methods that use this paradigm have focused on optimizing the brain decoding and the acoustic speech separation independently. In this work, we propose a novel framework called brain-informed speech separation (BISS)11BISS: brain-informed speech separation. in which the information about the attended speech, as decoded from the subject’s brain, is directly used to perform speech separation in the front-end. We present a deep learning model that uses neural data to extract the clean audio signal that a listener is attending to from a multi-talker speech mixture. We show that the framework can be applied successfully to the decoded output from either invasive intracranial electroencephalography (iEEG) or non-invasive electroencephalography (EEG) recordings from hearing-impaired subjects. It also results in improved speech separation, even in scenes with background noise. The generalization capability of the system renders it a perfect candidate for neuro-steered hearing-assistive devices.

Single channel speech dereverberation and separation using RPCA and SNMF

Single-channel speech dereverberation and separation has been a challenging problem and is of high significance in speech processing applications. Many researchers have tried to address this problem separately, either investigating speech dereverberation or separation. Most of the available literature on joint dereverberation and separation research is for multichannel. Moreover, in joint dereverberation and separation research, mostly noise is considered as the other source. Therefore, there is an emergent need to provide an optimal and efficient solution to the single channel speech dereverberation and separation problem. In this paper, we work on speech dereverberation and separation for single-channel, and we do not consider any noise source except the reverberation effects. We combine the two dominant methods, namely robust principal component analysis (RPCA) and sparse non-negative matrix factorization (SNMF), for the dereverberation and separation of the underlying speeches from the speech mixture. Firstly, we use the RPCA algorithm to dereverberate the reverberant mixture, and then we use the SNMF technique to separate the speeches from the speech mixture. We consider unseen RIRs conditions and compare the results with the baseline. The experimental results show that the proposed algorithm improves the speech quality both in evaluation parameters as well as listening.

A study on unsupervised monaural reverberant speech separation

Separating individual source signals is a challenging task in musical and multitalker source separation. This work studies unsupervised monaural (co-channel) speech separation (UCSS) in reverberant environment. UCSS is the problem of separating the individual speakers from multispeaker speech without using any training data and with minimum information regarding mixing condition and sources. In this paper, state-of-art UCSS algorithms based on auditory and statistical approaches are evaluated for reverberant speech mixtures and results are discussed. This work also proposes to use multiresolution cochleagram and Constant Q Transform (CQT) spectrogram feature with two-dimensional Non-negative matrix factorization. Results show that proposed algorithm with CQT spectrogram feature gave an improvement of 1.986 and 1.262 in terms of speech intelligibility and 0.296 db and 0.561 db in terms of signal to interference ratio compared to state-of-art statistical and auditory approach respectively at T60 of 0.610s.

Probabilistic Stone’s Blind Source Separation with application to channel estimation and multi-node identification in MIMO IoT green communication and multimedia systems

By the increasing growth of the Internet of Things (IoT) which provides interconnection and communications between electronic devices and corresponding sensors, a large volume of data is exchanged by multi-input multi-output (MIMO) telecommunication systems. In the case of IoT, reducing the data volume by removing the data redundancy results in more green communication by less power consumption for data transmission and less required storage memory. An approach to avoid the pilot data thus having less redundant data is using blind techniques. This research work presents an improvement to Stone’s blind source separation (BSS) precision, robustness, and computation load and its application to blind MIMO IoT interference channel estimation, multi-nodes IoT data detection, separation and identification in a MIMO-OFDM IoT network. Stone’s BSS is based on complexity conjecture indicating the independent sources have higher predictability than the mixtures. The presented improvement to Stone’s BSS is by a probabilistic modification to the short-term predictability merit by acquiring the prediction coefficients proportional to probability weights which follow a super Gaussian distribution assumption for sources. The probabilistic modification to Stone’s BSS (P-Stone) makes it maximally compatible with a pre-specified probability distribution model, and thereof the signal recovery is not only due to predictability maximization, but it is also inherent to increasing non-gaussianity which results in more independent recoveries, and less dependent on serial dependency of sources. Despite the Stone’s BSS, the proposed merit function does not need any long-term predictor; thus, it achieves around fifty times lower complexity load using just short-term predictors. The superiority of P-Stone to Stone BSS, AMUSE, and SOBI as well-known second-order techniques has been statistically evaluated and clarified through the experiments over MIMO-IoT networks of different combinations. As well, the comparative analysis over multimedia mixtures of music, speech, and images demonstrates its efficiency dominance.

The importance of processing resolution in "ideal time-frequency segregation" of masked speech and the implications for predicting speech intelligibility.

Ideal time-frequency segregation (ITFS) is a signal processing technique that may be used to estimate the energetic and informational components of speech-on-speech masking. A core assumption of ITFS is that it roughly emulates the effects of energetic masking (EM) in a speech mixture. Thus, when speech identification thresholds are measured for ITFS-processed stimuli and compared to thresholds for unprocessed stimuli, the difference can be attributed to informational masking (IM). Interpreting this difference as a direct metric of IM, however, is complicated by the fine time-frequency (T-F) resolution typically used during ITFS, which may yield target "glimpses" that are too narrow/brief to be resolved by the ear in the mixture. Estimates of IM, therefore, may be inflated because the full effects of EM are not accounted for. Here, T-F resolution was varied during ITFS to determine if/how estimates of IM depend on processing resolution. Speech identification thresholds were measured for speech and noise maskers after ITFS. Reduced frequency resolution yielded poorer thresholds for both masker types. Reduced temporal resolution did so for noise maskers only. Results suggest that processing resolution strongly influences estimates of IM and implies that current approaches to predicting masked speech intelligibility should be modified to account for IM.

Binaural sensitivity and release from speech-on-speech masking in listeners with and without hearing loss.

Listeners with sensorineural hearing loss routinely experience less spatial release from masking (SRM) in speech mixtures than listeners with normal hearing. Hearing-impaired listeners have also been shown to have degraded temporal fine structure (TFS) sensitivity, a consequence of which is degraded access to interaural time differences (ITDs) contained in the TFS. Since these "binaural TFS" cues are critical for spatial hearing, it has been hypothesized that degraded binaural TFS sensitivity accounts for the limited SRM experienced by hearing-impaired listeners. In this study, speech stimuli were noise-vocoded using carriers that were systematically decorrelated across the left and right ears, thus simulating degraded binaural TFS sensitivity. Both (1) ITD sensitivity in quiet and (2) SRM in speech mixtures spatialized using ITDs (or binaural release from masking; BRM) were measured as a function of TFS interaural decorrelation in young normal-hearing and hearing-impaired listeners. This allowed for the examination of the relationship between ITD sensitivity and BRM over a wide range of ITD thresholds. This paper found that, for a given ITD sensitivity, hearing-impaired listeners experienced less BRM than normal-hearing listeners, suggesting that binaural TFS sensitivity can account for only a modest portion of the BRM deficit in hearing-impaired listeners. However, substantial individual variability was observed.