Noisy Acoustic Environments Research Articles

A GPU-accelerated real-time human voice separation framework for mobile phones

Mobile speech communication can experience significant degradation in quality when users are in a noisy acoustic environment. With the rapid development of artificial intelligence in recent years, deep learning based monaural speech separation methods have shown remarkable progress in boosting the performance of the separation accuracy. However, the latency and computational cost of these methods remain far insufficient for mobile devices. Performance and power constraints make it still challenging to deploy such methods on mobile devices due to their high computational complexity. In this paper, we present VoiceBit, an efficient and light-weight human voice separation framework for real-time speech separation on mobile devices. Specifically, we propose a light-weight speech separation network to segregate human voice and interfering noises directly from time-domain signals. We binarize the convolution blocks in down-sampling blocks to reduce computation complexity and memory footprint, and leverage scaler layers as well as learnable bias layers to enhance the representation ability of binary filters. In addition, we present a set of parallel optimizations to accelerate the operations in VoiceBit. Specifically, we adopt KKC-minor format for weight matrices of convolution layers to coalesce memory access from global memory. Then, we explore different methods to implement the transposed convolution operation under PhoneBit framework. Experimental results on the MUSDB18-HQ dataset and VCTK dataset show that VoiceBit achieves significant speedup and energy efficiency compared with state-of-the-art frameworks, while maintaining minimal compromise in accuracy.

MuteIt

In this paper, we present MuteIt, an ear-worn system for recognizing unvoiced human commands. MuteIt presents an intuitive alternative to voice-based interactions that can be unreliable in noisy environments, disruptive to those around us, and compromise our privacy. We propose a twin-IMU set up to track the user's jaw motion and cancel motion artifacts caused by head and body movements. MuteIt processes jaw motion during word articulation to break each word signal into its constituent syllables, and further each syllable into phonemes (vowels, visemes, and plosives). Recognizing unvoiced commands by only tracking jaw motion is challenging. As a secondary articulator, jaw motion is not distinctive enough for unvoiced speech recognition. MuteIt combines IMU data with the anatomy of jaw movement as well as principles from linguistics, to model the task of word recognition as an estimation problem. Rather than employing machine learning to train a word classifier, we reconstruct each word as a sequence of phonemes using a bi-directional particle filter, enabling the system to be easily scaled to a large set of words. We validate MuteIt for 20 subjects with diverse speech accents to recognize 100 common command words. MuteIt achieves a mean word recognition accuracy of 94.8% in noise-free conditions. When compared with common voice assistants, MuteIt outperforms them in noisy acoustic environments, achieving higher than 90% recognition accuracy. Even in the presence of motion artifacts, such as head movement, walking, and riding in a moving vehicle, MuteIt achieves mean word recognition accuracy of 91% over all scenarios.

Children's intentional switching of auditory selective attention in spatial and noisy acoustic environments in comparison to adults.

Children's development and education take place in educational buildings with highly complex acoustic scenes, including spatially distributed target speakers, many surrounding distracting sounds, and general background noises. Auditory selective attention, therefore, is a valuable tool to orient oneself, to focus on specific sound sources, and to extract relevant information. Until now, it is unknown to what extent children have developed the cognitive processes of intentional attention control in spatial situations and how they differ from adults. This work provides a paradigm to examine children's intentional switching of auditory selective attention that also allows to examine effects due to noisy and spatial sound environments presented virtually via headphones. A listening experiment was conducted in Germany with 24 children (6-10 years, 50% female) and 24 young adults (18-26 years, 50% female). First, results revealed higher error rates and lower reaction times in conditions with noise (relative to conditions without noise) for children, but not for adults. This assumes that children are more sensitive to noise and conclude faster with noise trials, taking errors into account. Second, although auditory attention flexibility reflected in attention switch and relevant information selection was comparable between children and adults, it was found that adults benefited from spatial cues when selecting the relevant information. This was not observed to the same extent in children. These results suggest that children's cognitive processes are affected at significantly lower noise levels than adults and that noise effect assessment methods should consider spatial aspects. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

An initiation to revive the unique sound of Indonesian cities

As a large country with thousands of ethnic groups and cultures, it is hoped that every city in Indonesia will have its uniqueness. However, preliminary data collected from 10 major cities in Indonesia shows no uniqueness. The most visited public places in these cities, i.e. parks and squares, which are generally associated with natural sounds, are dominated by human and traffic noise. Surprisingly, a noisy acoustic environment is not considered a nuisance. The study reported here looks for reasons why people ignore the noise. An online questionnaire developed using a 5-point Likert scale was distributed to collect data due to the Covid-19 pandemic. Five hundred and ninety-five respondents participated in the survey. ANOVA and Kruskal Wallis test were run to identify differences between soundscape dimensions and differences in soundscape attribute ratings, respectively. The data shows that Indonesians visit public places for communal or social activities, which are triggered by the attractiveness of the places and the types of activities they can participate in. It is the reason why noise is not considered a nuisance. Pleasantness and eventfulness are the two dominant soundscape dimensions found in this study. In the Indonesian context, pleasure correlates with events. Eventfulness is associated with the number of people and their activities in public places. However, in most of the cities surveyed, eventfulness scores were low when they were unable to engage in the events held in public places. They visit public places based on the attractiveness of the place and the activities, and they feel comfortable in noisy public places when they can be involved in the activity. Once people become attached to communal activities in public places, the pleasantness dimension also exists. Thus, two things need to be considered to improve the acoustic environment of cities in Indonesia. First is by reducing traffic noise to increase the dimensions of eventfulness by using attractive attractions in public places. Second is to investigate the types of attractions that are of interest, if possible, is to restore local culture with its unique sound to build a unique city soundscape. In this study, participants identified the uniqueness of sounds in public places by using sounds that could not be classified as unique such as the voice of and the music played by street vendors.

Bayesian signal processing: A practical guide to particle filtering design and performance

When uncertain acoustic processes can no longer be characterized by Gaussian or for that matter unimodal (single peak) distributions along with the fact that the underlying phenomenology is nonstationary (time-varying) and nonlinear, then more general Bayesian processors must be applied to solve the underlying signal enhancement/extraction problem. A particle filter provides a solution to this multimodal (multiple peaks) posterior distribution estimation problem in noisy acoustic environments. A particle filter is a sequential Markov chain Monte Carlo processor capable of providing reasonable performance for data evolving from a multimodal distribution by estimating a nonparametric representation of the posterior distribution from which a multitude of meaningful statistics can be retrieved. However, the question of evaluating its performance can be challenging even for the simplest of processes. For instance, it is well-known that Kalman filter optimality can be obtained only when the resulting error residuals (innovations) are zero-mean and white. It is not that simple for the particle filter. Once characterized, the performance of the particle filter must be analyzed for it to be of practical value. Here a set of design and analysis criteria is discussed and applied to demonstrate their ability to quantify particle filtering performance.

Deep Learning Based Real-time Speech Enhancement for Dual-microphone Mobile Phones.

In mobile speech communication, speech signals can be severely corrupted by background noise when the far-end talker is in a noisy acoustic environment. To suppress background noise, speech enhancement systems are typically integrated into mobile phones, in which one or more microphones are deployed. In this study, we propose a novel deep learning based approach to real-time speech enhancement for dual-microphone mobile phones. The proposed approach employs a new densely-connected convolutional recurrent network to perform dual-channel complex spectral mapping. We utilize a structured pruning technique to compress the model without significantly degrading the enhancement performance, which yields a low-latency and memory-efficient enhancement system for real-time processing. Experimental results suggest that the proposed approach consistently outperforms an earlier approach to dual-channel speech enhancement for mobile phone communication, as well as a deep learning based beamformer.

Visual Enhancement of Relevant Speech in a ‘Cocktail Party’

Lip-reading improves intelligibility in noisy acoustical environments. We hypothesized that watching mouth movements benefits speech comprehension in a 'cocktail party' by strengthening the encoding of the neural representations of the visually paired speech stream. In an audiovisual (AV) task, EEG was recorded as participants watched and listened to videos of a speaker uttering a sentence while also hearing a concurrent sentence by a speaker of the opposite gender. A key manipulation was that each audio sentence had a 200-ms segment replaced by white noise. To assess comprehension, subjects were tasked with transcribing the AV-attended sentence on randomly selected trials. In the auditory-only trials, subjects listened to the same sentences and completed the same task while watching a static picture of a speaker of either gender. Subjects directed their listening to the voice of the gender of the speaker in the video. We found that the N1 auditory-evoked potential (AEP) time-locked to white noise onsets was significantly more inhibited for the AV-attended sentences than for those of the auditorily-attended (A-attended) and AV-unattended sentences. N1 inhibition to noise onsets has been shown to index restoration of phonemic representations of degraded speech. These results underscore that attention and congruency in the AV setting help streamline the complex auditory scene, partly by reinforcing the neural representations of the visually attended stream, heightening the perception of continuity and comprehension.

Inverse Patch Transfer Function With Fast Iterative Shrinkage-Thresholding Algorithm as a Tool for Sparse Source Identification

Inverse patch transfer functions (iPTF) method is an efficient technique for sound field reconstruction and separation of arbitrarily shaped sound sources in noisy acoustic environments. By applying Neumann boundary condition to a closed virtual cavity surrounding the source, the Helmholtz integral equation is simplified and can be solved with a Green's function satisfying the Neumann boundary condition. However, in the identification of sparsely distributed sources, ghost sources appear in the result solved by using the iPTF method with classic Tikhonov regularization methods and influence the accuracy of identification. In the present work, an evanescent Green's function with fast convergence of calculation is utilized, and a technique that combines the iPTF method and Fast Iterative Shrinkage-Thresholding Algorithm aimed at improving the performance for the identification of sparsely distributed source is proposed. Then double layer measurements, instead of using expensive p-u probes, are employed to acquire the normal velocity of the hologram surface. In numerical simulations, the normal velocities of two anti-phased piston sources are well reconstructed with high sparsity while a disturbing source is radiating in the sound field within the frequency band of 50 to 1000 Hz. Finally, an experiment on two baffled loudspeakers has been carried out. The results of simulations and experiments indicate that the proposed technique has obviously improved the accuracy of the iPTF method in the identification of sparsely distributed sources for the frequency band from 50 to 1000 Hz.

ML Estimation and CRBs for Reverberation, Speech, and Noise PSDs in Rank-Deficient Noise Field

Speech communication systems are prone to performance degradation in reverberant and noisy acoustic environments. Dereverberation and noise reduction algorithms typically require several model parameters, e.g. the speech, reverberation and noise power spectral densities (PSDs). A commonly used assumption is that the noise PSD matrix is known. However, in practical acoustic scenarios, the noise PSD matrix is unknown and should be estimated along with the speech and reverberation PSDs. In this article, we consider the case of rank-deficient noise PSD matrix, which arises when the noise signal consists of multiple directional noise sources, whose number is less than the number of microphones. We derive two closed-form maximum likelihood estimators (MLEs). The first is a non-blocking-based estimator which jointly estimates the speech, reverberation and noise PSDs, and the second is a blocking-based estimator, which first blocks the speech signal and then jointly estimates the reverberation and noise PSDs. Both estimators are analytically compared and analyzed, and mean square errors (MSEs) expressions are derived. Furthermore, Cramer-Rao Bounds (CRBs) on the estimated PSDs are derived. The proposed estimators are examined using both simulation and real reverberant and noisy signals, demonstrating the advantage of the proposed method compared to competing estimators.

End-to-end audiovisual speech activity detection with bimodal recurrent neural models

Speech activity detection (SAD) plays an important role in current speech processing systems, including automatic speech recognition (ASR). SAD is particularly difficult in environments with acoustic noise. A practical solution is to incorporate visual information, increasing the robustness of the SAD approach. An audiovisual system has the advantage of being robust to different speech modes (e.g., whisper speech) or background noise. Recent advances in audiovisual speech processing using deep learning have opened opportunities to capture in a principled way the temporal relationships between acoustic and visual features. This study explores this idea, proposing the bimodal recurrent neural network (BRNN) framework for SAD. The approach models the temporal dynamic of the sequential audiovisual data, improving the accuracy and robustness of the proposed SAD system. Instead of estimating hand-crafted features, the study investigates an end-to-end training approach, where acoustic and visual features are directly learned from the raw data during training. The experimental evaluation considers a large audiovisual corpus with over 60.8 h of recordings, collected from 105 speakers. The results demonstrate that the proposed framework leads to absolute improvements up to 1.2% under practical scenarios over a VAD baseline using only audio implemented with deep neural network (DNN). The proposed approach achieves 92.7% F1-score when it is evaluated using the sensors from a portable tablet under noisy acoustic environment, which is only 1.0% lower than the performance obtained under ideal conditions (e.g., clean speech obtained with a high definition camera and a close-talking microphone).

Straightforward impedance eduction method for non-grazing incidence wave with multiple modes

The control of modern aeroengine noise poses great challenge to the design of acoustic liners, due to the existence of multiple higher-order modes over a very wide frequency range in acoustic nacelles. In order to meet the pressing demand, the straightforward impedance eduction method is further developed such that it can be applied to more general acoustic environment where there are both plane mode and higher-order modes incident upon a test liner. A novelty of the present method is using a diagonally mounted microphone array on the opposite wall of the test liner in a flow duct, thus the measured wall sound pressure simultaneously contains the information of the modes in both the axial and transverse directions. Then, the employment of Prony's method enables the realization of the full modal decomposition to the acoustic field in the flow duct. Numerical experiments simulating acoustic fields with a finite element model are conducted to investigate and validate the feasibility of the present method for both a small-scale and a large-scale flow ducts. Two ceramic tubular liners and a perforated liner are tested as specimens, whose impedance values to be educed are known a priori by means of existing impedance models. Random perturbation is added into the acoustic field data to simulate realistic noisy acoustic environments. The results show that the present method can educe the imposed impedance with a good accuracy when the liner specimens are subjected to the sound field composing of multiple incident modes including several transverse modes in both ducts. The frequency scope of the impedance eduction is considerably extended when compared with the methods restricted by the assumption of plane incident mode, with the upper frequency reaching up to 6.0 kHz and 5.0 kHz for the ducts, respectively. A parametric study indicates that the accuracy of impedance eduction can be promoted by increasing the signal-to-noise ratio and the number of microphones.

Nonuniform impacts of forward suppression on neural responses to preferred stimuli and nonpreferred stimuli in the rat auditory cortex.

In natural conditions, human and animals need to extract target sound information from noisy acoustic environments for communication and survival. However, how the contextual environmental sounds impact the tuning of central auditory neurons to target sound source azimuth over a wide range of sound levels is not fully understood. Here, we determined the azimuth-level response areas (ALRAs) of rat auditory cortex neurons by recording their responses to probe tones varying with levels and sound source azimuths under both quiet (probe alone) and forward masking conditions (preceding noise + probe). In quiet, cortical neurons responded stronger to their preferred stimuli than to their nonpreferred stimuli. In forward masking conditions, an effective preceding noise reduced the extents of the ALRAs and suppressed the neural responses across the ALRAs by decreasing the response strength and lengthening the first-spike latency. The forward suppressive effect on neural response strength was increased with increasing masker level and decreased with prolonging the time interval between masker and probe. For a portion of cortical neurons studied, the effects of forward suppression on the response strength to preferred stimuli was weaker than those to nonpreferred stimuli, and the recovery from forward suppression of the response strength to preferred stimuli was earlier than that to nonpreferred stimuli. We suggest that this nonuniform forward suppression of neural responses to preferred stimuli and to nonpreferred stimuli is important for cortical neurons to maintain their relative stable preferences for target sound source azimuth and level in noisy acoustic environments.

A Comparative Study of Blind Source Separation for Bioacoustics Sounds based on FastICA, PCA and NMF

Blind Source Separation (BSS) is a task of separating a set of source signals from mixed signal without (or very little information) of both the sources and the mixing process. This paper addresses the problem of BSS in bio-acoustic mixed signals. In a noisy acoustic environment, animal species recognition based on vocalization remains a challenging task. In order to robustly recognize the specific species, the source signals of interest need to be separated from the mixed signals. This separation process is a significant pre-processing step before the recognition process takes place. In this paper, three different source separation methods namely Fast Fixed-Point Independent Component Analysis algorithms (FastICA), Principal Component Analysis (PCA) and Non-Negative Matrix Factorization (NMF) are implemented. In this experiment, the mixtures of frog sound signals are used as input. The quality of separated source signals using FastICA, PCA and NMF algorithms are compared and evaluated according to BSS_EVAL toolbox metrics. These metrics consist of signal to distortion ratio (SDR), signal to interference ratio (SIR) and signal to artifacts ratio (SAR). The results show that FastICA with negentropy technique for finding a maximum non-gaussianity has the best performances in separating mixed signals.

Sensing in a noisy world: lessons from auditory specialists, echolocating bats.

All animals face the essential task of extracting biologically meaningful sensory information from the 'noisy' backdrop of their environments. Here, we examine mechanisms used by echolocating bats to localize objects, track small prey and communicate in complex and noisy acoustic environments. Bats actively control and coordinate both the emission and reception of sound stimuli through integrated sensory and motor mechanisms that have evolved together over tens of millions of years. We discuss how bats behave in different ecological scenarios, including detecting and discriminating target echoes from background objects, minimizing acoustic interference from competing conspecifics and overcoming insect noise. Bats tackle these problems by deploying a remarkable array of auditory behaviors, sometimes in combination with the use of other senses. Behavioral strategies such as ceasing sonar call production and active jamming of the signals of competitors provide further insight into the capabilities and limitations of echolocation. We relate these findings to the broader topic of how animals extract relevant sensory information in noisy environments. While bats have highly refined abilities for operating under noisy conditions, they face the same challenges encountered by many other species. We propose that the specialized sensory mechanisms identified in bats are likely to occur in analogous systems across the animal kingdom.

The cochlear implant and possibilities for narrowing the remaining gaps between prosthetic and normal hearing.

BackgroundThe cochlear implant has become the standard of care for severe or worse losses in hearing and indeed has produced the first substantial restoration of a lost or absent human sense using a medical intervention. However, the devices are not perfect and many efforts to narrow the remaining gaps between prosthetic and normal hearing are underway.ObjectiveTo assess the present status of cochlear implants and to describe possibilities for improving them.ResultsThe present-day devices work well in quiet conditions for the great majority of users. However, not all users have high levels of speech reception in quiet and nearly all users struggle with speech reception in typically noisy acoustic environments. In addition, perception of sounds more complex than speech, such as most music, is generally poor unless residual hearing at low frequencies can be stimulated acoustically in conjunction with the electrical stimuli provided by the implant. Possibilities for improving the present devices include increasing the spatial specificity of neural excitation by reducing masking effects or with new stimulus modes; prudent pruning of interfering or otherwise detrimental electrodes from the stimulation map; a further relaxation in the criteria for implant candidacy, based on recent evidence from persons with high levels of residual hearing and to allow many more people to benefit from cochlear implants; and “top down” or “brain centric” approaches to implant designs and applications.ConclusionsProgress in the development of the cochlear implant and related treatments has been remarkable but room remains for improvements. The future looks bright as there are multiple promising possibilities for improvements and many talented teams are pursuing them.

The impact of cochlear implantation on spatial hearing and listening effort

We studied the potential benefits of bilateral hearing in cochlear implants (CI) users, and in patients with single-sided deafness (SSD) who receive a CI in the deaf ear and have normal hearing in the other ear. We hypothesized that listening effort, measured with pupil dilation, can reveal benefits of bilateral hearing that may not be consistently observed when localization or spatial release from masking (SRM) are measured. Result from 12 bilateral CI users showed reduction in listening effort with bilateral hearing compared to the poor ear or better ear alone. In patients with SSD, benefits of adding a CI to a normal hearing ear can emerge over a protracted period of a year or longer. In addition, for at least some of the subjects bilateral hearing (adding a CI to the normal hearing ear) produced release from listening effort even in conditions where SRM was not observed. That is, speech intelligibility did not always improve with spatial separation of target and competing speech, but pupil dilation was reduced, suggesting that pupillometry can reveal subtle aspects of bilateral benefits. The benefits shown with pupillometry are consistent with the patients’ subjective reports of improved ability to function in complex, noisy acoustic environments.

EEG-Informed Attended Speaker Extraction From Recorded Speech Mixtures With Application in Neuro-Steered Hearing Prostheses

We aim to extract and denoise the attended speaker in a noisy two-speaker acoustic scenario, relying on microphone array recordings from a binaural hearing aid, which are complemented with electroencephalography (EEG) recordings to infer the speaker of interest. In this study, we propose a modular processing flow that first extracts the two speech envelopes from the microphone recordings, then selects the attended speech envelope based on the EEG, and finally uses this envelope to inform a multichannel speech separation and denoising algorithm. Strong suppression of interfering (unattended) speech and background noise is achieved, while the attended speech is preserved. Furthermore, EEG-based auditory attention detection (AAD) is shown to be robust to the use of noisy speech signals. Our results show that AAD-based speaker extraction from microphone array recordings is feasible and robust, even in noisy acoustic environments, and without access to the clean speech signals to perform EEG-based AAD. Current research on AAD always assumes the availability of the clean speech signals, which limits the applicability in real settings. We have extended this research to detect the attended speaker even when only microphone recordings with noisy speech mixtures are available. This is an enabling ingredient for new brain-computer interfaces and effective filtering schemes in neuro-steered hearing prostheses. Here, we provide a first proof of concept for EEG-informed attended speaker extraction and denoising.

Non‐negative matrix factorisation‐based subband decomposition for acoustic source localisation

A novel non-negative matrix factorization (NMF) based subband decomposition in frequency spatial domain for acoustic source localization using a microphone array is introduced. The proposed method decomposes source and noise subband and emphasises source dominant frequency bins for more accurate source representation. By employing NMF, delay basis vectors and their subband information in frequency spatial domain for each frame is extracted. The proposed algorithm is evaluated in both simulated noise and real noise with a speech corpus database. Experimental results clearly indicate that the algorithm performs more accurately than other conventional algorithms under both reverberant and noisy acoustic environments.

Patterns and dynamics of (bird) soundscapes: A biosemiotic interpretation

The soundscape, which is defined as the entire acoustic environment of an area, is a relevant biosemiotic ingredient of environmental complexity. It is composed of geophonies, anthrophonies, and biophonies where, in temperate biomes, birds are the major producers of the latter. The soundscape is heterogeneous in terms of space and time, and is affected by landscape features such as vegetation cover (patch size and shape, length of edges, etc.). It also operates as a communication network in which intra- and inter-specific interactions create a complex, eavesdropping-broadcasting network, namely, the soundtope. This acoustic context provides public information for tracking resources like territory, mates, food, safety, and roosting sites and it can be considered a multifunctional eco-field. The Acoustic Complexity Index (ACI) is a new metric that is used to efficiently process the sound files that, when collected by sampling matrixes, return by interpolation the spatial distribution of a soundscape. High quality soundscapes are sought to guarantee the biosemiotic mechanisms among species, but polluted conditions (noisy acoustic environment of anthropogenic origin) degrade and mask biophonies, thus reducing communication efficiency, with severe consequences for both individual and collective fitness. Preserving high quality soundscapes is therefore an important conservation issue and a target to aim for if we are to assure animal and human wellbeing.

A rain forest dusk chorus: cacophony or sounds of silence?

A rain forest dusk chorus consists of a large number of individuals of acoustically communicating species signaling at the same time. How different species achieve effective intra-specific communication in this complex and noisy acoustic environment is not well understood. In this study we examined acoustic masking interference in an assemblage of rain forest crickets and katydids. We used signal structures and spacing of signalers to estimate temporal, spectral and active space overlap between species. We then examined these overlaps for evidence of strategies of masking avoidance in the assemblage: we asked whether species whose signals have high temporal or spectral overlap avoid calling together. Whereas we found evidence that species with high temporal overlap may avoid calling together, there was no relation between spectral overlap and calling activity. There was also no correlation between the spectral and temporal overlaps of the signals of different species. In addition, we found little evidence that species calling in the understorey actively use spacing to minimize acoustic overlap. Increasing call intensity and tuning receivers however emerged as powerful strategies to minimize acoustic overlap. Effective acoustic overlaps were on average close to zero for most individuals in natural, multispecies choruses, even in the absence of behavioral avoidance mechanisms such as inhibition of calling or active spacing. Thus, call temporal structure, intensity and frequency together provide sufficient parameter space for several species to call together yet communicate effectively with little interference in the apparent cacophony of a rain forest dusk chorus.