Binaural Microphone Research Articles

Extraction of Sound Sources in Specified Area Using Binaural Microphones

Extraction of Sound Sources in Specified Area Using Binaural Microphones

Using deep learning for recreating binaural audio

In recent years, there has been an increase in research around generating spatialized audio using a mono audio signal. Methods like using neural networks which combine image segmentation with object location for adding back the spatial qualities are often developed. Instead, our project focuses on taking arbitrary mono input sound and an input angle, and outputting spatial stereo audio of the input sound with the directionality of the angle. This is different from current implementations as it is a simpler approach to what spatial audio generation is, and it allows for the use in the model in new areas. Using a binaural microphone and a custom-made anechoic chamber, 120 hours of labelled binaural audio was recorded for use in our model. The audio consists of frequency sweeps, pink noise, and phonetically balanced speech. Our method is to predict a complex short-time Fourier transform mask which will contain the phase and amplitude within it. The model is an autoencoder based on the U-NET model, which is applied to the mono input before being compared to the labelled data for training. With this simpler approach, we hope to make spatial audio more accessible for a variety of applications.

DeepEar: Sound Localization With Binaural Microphones

The binaural microphone, which refers to a pair of microphones with artificial human-shaped ears, is widely used in hearing aids and spatial audio recording to improve sound quality. It is crucial for such devices to find the voice direction in many applications such as binaural sound enhancement. However, sound localization with two microphones remains challenging, especially in multi-source scenarios. Most previous work utilized microphone arrays to deal with the multi-source localization problem. Extra microphones yet have space constraints for deployment in many scenarios ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">e</i> . <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">g</i> ., hearing aids). Inspired by the fact that humans have evolved to locate multiple sound sources with only two ears, we propose DeepEar, a binaural microphone-based sound localization system. To this end, we design a multisector-based neural network to locate multiple sound sources simultaneously, where each sector is a discretized region of the space for different angle of arrivals. DeepEar fuses explicit hand-crafted features and implicit latent sound representatives to facilitate sound localization. More importantly, the trained DeepEar model can adapt to new environments with a minimum amount of extra training data. The experiment results show that DeepEar substantially outperforms the state-of-the-art binaural deep learning approach by a large margin in terms of sound detection accuracy and azimuth estimation error.

Art Is Magic

Art Is Magic

Localizing concurrent sound sources with binaural microphones: A simulation study

The human auditory system can localize multiple sound sources using time, intensity, and frequency cues in the sound received by the two ears. Being able to spatially segregate the sources helps perception in a challenging condition when multiple sounds coexist. This study used model simulations to explore an algorithm for localizing multiple sources in azimuth with binaural (i.e., two) microphones. The algorithm relies on the “sparseness” property of daily signals in the time-frequency domain, and sound coming from different locations carrying unique spatial features will form clusters. Based on an interaural normalization procedure, the model generated spiral patterns for sound sources in the frontal hemifield. The model itself was created using broadband noise for better accuracy, because speech typically has sporadic energy at high frequencies. The model at an arbitrary frequency can be used to predict locations of speech and music that occurred alone or concurrently, and a classification algorithm was applied to measure the localization error. Under anechoic conditions, averaged errors in azimuth increased from 4.5° to 19° with RMS errors ranging from 6.4° to 26.7° as model frequency increased from 300 to 3000 Hz. The low-frequency model performance using short speech sound was notably better than the generalized cross-correlation model. Two types of room reverberations were then introduced to simulate difficult listening conditions. Model performance under reverberation was more resilient at low frequencies than at high frequencies. Overall, our study presented a spiral model for rapidly predicting horizontal locations of concurrent sound that is suitable for real-world scenarios.

Study of speaker localization with binaural microphone array incorporating auditory filters and lateral angle estimation

Speaker localization for binaural microphone arrays has been widely studied for applications such as speech communication, video conferencing, and robot audition. Many methods developed for this task, including the direct path dominance (DPD) test, share common stages in their processing, which include transformation using the short-time Fourier transform (STFT), and a direction of arrival (DOA) search that is based on the head related transfer function (HRTF) set. In this paper, alternatives to these processing stages, motivated by human hearing, are proposed. These include incorporating an auditory filter bank to replace the STFT, and a new DOA search based on transformed HRTF as steering vectors. A simulation study and an experimental study are conducted to validate the proposed alternatives, and both are applied to two binaural DOA estimation methods; the results show that the proposed method compares favorably with current methods.

Closed-loop sound source localization in neuromorphic systems

Sound source localization (SSL) is used in various applications such as industrial noise-control, speech detection in mobile phones, speech enhancement in hearing aids and many more. Newest video conferencing setups use SSL. The position of a speaker is detected from the difference in the audio waves received by a microphone array. After detection the camera focuses onto the location of the speaker. The human brain is also able to detect the location of a speaker from auditory signals. It uses, among other cues, the difference in amplitude and arrival time of the sound wave at the two ears, called interaural level and time difference. However, the substrate and computational primitives of our brain are different from classical digital computing. Due to its low power consumption of around 20 W and its performance in real time the human brain has become a great source of inspiration for emerging technologies. One of these technologies is neuromorphic hardware which implements the fundamental principles of brain computing identified until today using complementary metal-oxide-semiconductor technologies and new devices. In this work we propose the first neuromorphic closed-loop robotic system that uses the interaural time difference for SSL in real time. Our system can successfully locate sound sources such as human speech. In a closed-loop experiment, the robotic platform turned immediately into the direction of the sound source with a turning velocity linearly proportional to the angle difference between sound source and binaural microphones. After this initial turn, the robotic platform remains at the direction of the sound source. Even though the system only uses very few resources of the available hardware, consumes around 1 W, and was only tuned by hand, meaning it does not contain any learning at all, it already reaches performances comparable to other neuromorphic approaches. The SSL system presented in this article brings us one step closer towards neuromorphic event-based systems for robotics and embodied computing.

A corpus-based approach to define Turkish soundscapes attributes

Considering the challenges of translating soundscape attributes, this study aims to identify soundscape attributes in Turkish. To conduct this study, an online questionnaire with two parts was prepared and sent to 200 native Turkish bilingual speakers from all around Turkey. In the first part, the participants were asked to translate the eight soundscape attributes in ISO 12913-2:2018 from English to Turkish. In addition to direct translations, the participants were asked to describe the acoustic environment they were present in with five perceptual adjectives in their own words. The adjectives from direct translations and the open-ended question were combined to form a pool of 196 adjectives. Through a two-step elimination process by six soundscape experts, a total of 80 attributes were obtained. Twenty-four binaural sound recordings, using a 3Dio binaural microphone and Zoom H4N recorder, were collected from indoor and outdoor public spaces with different functions (e.g., concert hall, masjid, design studio, cafe). Each participant was asked to listen to six 30s-long sound recordings one at a time and rate the 80 attributes on a five-point scale for each recording individually. The results show that the opposite adjectives with similar perceptual meanings were placed together in two main clusters.

Analysing changes in physiological response to different soundscape scenarios.

Noise pollution is one of the key environmental stressors leading to multiple health impacts for people and communities. Exposure to high noise levels may also be responsible for sudden emotional and physiological changes in humans. The experiment examines the change in Galvanic Skin Response (GSR) signals during exposure to pre-recorded soundscape scenarios. These scenarios were recorded in New Delhi, India and London, UK, using a binaural microphone set. The listening experiment was conducted in a laboratory, where 24 healthy individuals without any hearing impairment or any psychological issues participated. A total of 30 soundscape scenarios were presented to each participant in a randomised order. The continuous decomposition analysis is conducted to decompose that data into tonic and phasic components. The phasic component of the signal is used for the analysis. It is observed that skin conductance response (SCR) increases with changes in the overall loudness of the signal. Psychoacoustic indicators are used to identify the relationship between different acoustical characters of noise and changes in SCR. The perception-based survey suggests that GSR response is highly dependent on the pleasantness level of the soundscape scenario.

Deep Learning-Based Speech Specific Source Localization by Using Binaural and Monaural Microphone Arrays in Hearing Aids

A deep learning-based method is proposed for jointly detecting and localizing speech sources in a complex acoustic scene by using microphones of a hearing aid. Motivated by the human auditory system, peripheral preprocessing is applied on the microphone signals to obtain auditory subband signals that serve as input to the proposed deep neural network for detecting and localizing speech sources. In the proposed neural network, a combination of residual and dense aggregation learning is utilized rather than the conventional residual learning to preserve and reuse the spatial representations at the output layers. This process is performed to improve the gradient flow in deeper layers, in the training stage. The learning curves show that the proposed residual-dense aggregation mapping do improve the speed and accuracy of the convergence. The proposed model shows good performance in joint speech source detection and localization using a binaural microphone array (i.e., three channels at each side) but also using a monaural microphone array (i.e., four channels at the right side) despite of the short distances between the microphones. The proposed methods also outperform neural networks that are directly using STFT components of the binaural or monaural microphone arrays. In addition, the proposed models extended with learnable peripheral processing show slightly improved in detection and localization scores compared to the proposed models using the plain auditory subband signals, in both the binaural and monaural microphone arrays but only so, when the learnable peripheral processing is initialized with parameters stemming from human peripheral processing.

Binaural SoundNet: Predicting Semantics, Depth and Motion With Binaural Sounds.

Humans can robustly recognize and localize objects by using visual and/or auditory cues. While machines are able to do the same with visual data already, less work has been done with sounds. This work develops an approach for scene understanding purely based on binaural sounds. The considered tasks include predicting the semantic masks of sound-making objects, the motion of sound-making objects, and the depth map of the scene. To this aim, we propose a novel sensor setup and record a new audio-visual dataset of street scenes with eight professional binaural microphones and a 360 °camera. The co-existence of visual and audio cues is leveraged for supervision transfer. In particular, we employ a cross-modal distillation framework that consists of multiple vision 'teacher' methods and a sound 'student' method - the student method is trained to generate the same results as the teacher methods do. This way, the auditory system can be trained without using human annotations. To further boost the performance, we propose another novel auxiliary task, coined Spatial Sound Super-Resolution, to increase the directional resolution of sounds. We then formulate the four tasks into one end-to-end trainable multi-tasking network aiming to boost the overall performance. Experimental results show that 1) our method achieves good results for all four tasks, 2) the four tasks are mutually beneficial - training them together achieves the best performance, 3) the number and orientation of microphones are both important, and 4) features learned from the standard spectrogram and features obtained by the classic signal processing pipeline are complementary for auditory perception tasks. The data and code are released on the project page: https://www.trace.ethz.ch/publications/2020/sound_perception/index.html.

Unsupervised Synthetic Acoustic Image Generation for Audio-Visual Scene Understanding.

Acoustic images are an emergent data modality for multimodal scene understanding. Such images have the peculiarity of distinguishing the spectral signature of the sound coming from different directions in space, thus providing a richer information as compared to that derived from single or binaural microphones. However, acoustic images are typically generated by cumbersome and costly microphone arrays which are not as widespread as ordinary microphones. This paper shows that it is still possible to generate acoustic images from off-the-shelf cameras equipped with only a single microphone and how they can be exploited for audio-visual scene understanding. We propose three architectures inspired by Variational Autoencoder, U-Net and adversarial models, and we assess their advantages and drawbacks. Such models are trained to generate spatialized audio by conditioning them to the associated video sequence and its corresponding monaural audio track. Our models are trained using the data collected by a microphone array as ground truth. Thus they learn to mimic the output of an array of microphones in the very same conditions. We assess the quality of the generated acoustic images considering standard generation metrics and different downstream tasks (classification, cross-modal retrieval and sound localization). We also evaluate our proposed models by considering multimodal datasets containing acoustic images, as well as datasets containing just monaural audio signals and RGB video frames. In all of the addressed downstream tasks we obtain notable performances using the generated acoustic data, when compared to the state of the art and to the results obtained using real acoustic images as input.

Audio-Visual Localization by Synthetic Acoustic Image Generation

Acoustic images constitute an emergent data modality for multimodal scene understanding. Such images have the peculiarity to distinguish the spectral signature of sounds coming from different directions in space, thus providing richer information than the one derived from mono and binaural microphones. However, acoustic images are typically generated by cumbersome microphone arrays, which are not as widespread as ordinary microphones mounted on optical cameras. To exploit this empowered modality while using standard microphones and cameras we propose to leverage the generation of synthetic acoustic images from common audio-video data for the task of audio-visual localization. The generation of synthetic acoustic images is obtained by a novel deep architecture, based on Variational Autoencoder and U-Net models, which is trained to reconstruct the ground truth spatialized audio data collected by a microphone array, from the associated video and its corresponding monaural audio signal. Namely, the model learns how to mimic what an array of microphones can produce in the same conditions. We assess the quality of the generated synthetic acoustic images on the task of unsupervised sound source localization in a qualitative and quantitative manner, while also considering standard generation metrics. Our model is evaluated by considering both multimodal datasets containing acoustic images, used for the training, and unseen datasets containing just monaural audio signals and RGB frames, showing to reach more accurate localization results as compared to the state of the art.

Towards a sonic ecology of urban life: ethnography of sound perception in Cairo

ABSTRACT This study on sound perception in Cairo uses a methodological procedure described in a previous issue of this journal [11(3)]. The procedure involves equipping inhabitants of Cairo, the Egyptian capital, with binaural microphones that record the surrounding urban sounds during one of their daily journeys (without the researcher). Participants later describe and comment on the sounds while listening to the recording. Analysis of this material allowed us first to establish an organized lexicon in categories. We identified a structured “natural language of sounds”. The data obtained reveal covert categories that describe three key domains of urban life: the active city, the city in movement, and the relational city. A principal finding is that sound perception systematically relates sounds to their origin, i.e. both the source and its social situation. This socialization of sound led us to the notion of “sound constructs” as products of an immediate socialization of the perception of sound. Experiment clarifies how perception operates in Cairo, notably through territory differentiation using sonic saliences and soundmarks. Finally, we propose a “sonic ecology” of the city: how residents collectively experience the sound dimensions of their urban territory, navigate between very different territories, recognize them and respond to them.

Evolutionary tuning of filters coefficients for binaural audio equalization

This paper presents a novel multichannel audio equalization technique based on evolutionary computation algorithms for tuning the filters coefficients. Specifically, two distinct evolutionary algorithms are used on purpose, i.e. the Particle Swarm Optimization (PSO) and Gravitational Search Algorithm (GSA). Two alternative solutions for the definition of evolutionary particles have been devised and tested with both techniques. Given the desired frequency response, the fitness function is formulated in terms of amplitude spectral distance. These techniques have been assessed by computer experiments, conducted on a in-car binaural equalization scenario considering 7 loudspeakers and a binaural microphone. The obtained results show that the proposed solutions achieve a remarkably superior performance compared to the baseline methods, with a 5 times reduction of the mean square error in the amplitude spectral domain.

ASMR: bodily pleasure, online performance, digital modality

A woman taps her fingernails against the bristles of a hairbrush. A young man snaps his fingers on either side of a visible binaural microphone. A girl speaks softly, almost whispering; each of her...

Re]Mixing Space: Charting Sonic Accessibility and Social Equity in Creative Urban Contexts

Sonic inclusion – the ways in which sound includes people in space – has received little attention in the design of the built environment. It is proposed that the design and management of contemporary urban space is grounded in an “auraltypical” perspective where the primary focus is those with “otologically normal” hearing. This position is driven by an idealized model of hearing that frames design through the binary positioning of “good/bad” ears and leads to spaces that are socially exclusive. In contrast, the emerging paradigm of “auraldiversity” acknowledges the diversity of human hearing and the multitude of elements that place the hearing modality within a state of constant flux. This article presents a case study of Remixing the Globe, a participatory sound mapping workshop and exhibition held at Shakespeare’s Globe, London in 2017. The workshop brought together a cohort of disabled adults, each with self-identified lived experience of sonic exclusion. The group utilized digital technologies, such as binaural microphones, pressure sensors and smartphone-activated sound stickers, to map sonic objects, environments and experiences throughout the building. Specific focus was given to mapping the elements of the environment that were understood as pertinent to the inclusion or exclusion of d/Deaf and disabled people. Findings emphasize the important role that sonic cartography can play in examining accessibility in urban space. The article calls for a new politics of sound mapping in which d/Deaf and disability embodiment is foregrounded and through which a critical examination of sound and social inclusion in urban spaces can begin to emerge.

Research and development of sound quality in portable testing and evaluation system based on self-adaptive neural network

Ordinary microphone cannot accurately reflect authentic human feelings in the sound quality testing and evaluation. By contrast, professional artificial head system can produce similar results with human feelings through the simulation of human auditory system, but it is expensive and unable to occupy the driver's seat to record the vehicle sound. Here we design a portable system based on self-adaptive neural network (NN) which can obtain the result of sound quality evaluation similar with the artificial head. The system consists of binaural microphone, signal conditioning units (SCU), and mobile phone. Firstly, sound is perceived by binaural microphone, processed by SCU, and collected by mobile phone. Initial sound quality objective parameters are calculated after noise reduction for sound signals and then the accurate objective parameters can be obtained from the initial parameters through the trained NN model. To train the model, the portable system and the HEAD system are simultaneously used to collect sound sample as the input and output sample sets under different conditions. After the input samples are preprocessed, wavelet entropy eigenvalues based on the best tree wavelet packet analysis are calculated along with psychoacoustic parameters of all the samples. Finally, we obtain the combined feature vectors and successfully train the model. Checked by the practical data, the relative error of sound quality objective parameters produced by the NN model is less than 5% compared with the HEAD system, which conforms to general engineering requirements. This inexpensive and user-friendly portable system paves a new way in the sound quality testing and evaluation that are highly desirable for vehicle acoustical designs and improvements.

A Robot for Test Bed Aimed at Improving Telepresence System and Evasion from Discomfort Stimuli by Online Learning

This study contributes to improving the comfort of telepresence communication by adaptations to people. We developed a handheld telepresence robot comprised of a binaural microphone and a head mounted display as a test bed and conducted surveys on unpleasantness caused by special devices in use. We found that numerous people experienced an increase in unpleasant sound when the binaural microphone was being used. It was also found that types of unpleasant stimuli differed from person to person. Furthermore, we propose an automatic unpleasant stimuli avoidance system using online machine learning architecture constructed from echo state networks (ESNs) and Accumulator Based Arbitration Models (ABAMs) that can also flexibly adapt to remote users. Because the handheld telepresence robot can avoid unpleasant stimuli before remote users experience these stimuli, it provides them with a more comfortable communication environment while notifying people around the robot that uncomfortable stimulation is being avoided.

Prediction of speech intelligibility using pseudo-binaural room impulse responses.

Head orientation (HO) affects better-ear-listening and spatial-release-from-masking, which are two key aspects in binaural speech intelligibility. To incorporate HO in speech intelligibility prediction, binaural room impulse responses (BRIRs) for every HO of interest could be used. Due to the limited spectral bandwidth of speech, however, approximate representations might be sufficient, which can be measured more quickly. A comparison was done between pseudo-BRIRs generated with a motion tracked binaural microphone array and a first order Ambisonics microphone using the spatial decomposition method (SDM). The accuracy of the Ambisonics/SDM approach was comparable to that of real BRIRs, indicating its suitability for speech intelligibility prediction.