Sound Event Localization Research Articles

A research for sound event localization and detection based on local–global adaptive fusion and temporal importance network

A research for sound event localization and detection based on local–global adaptive fusion and temporal importance network

CRATI: Contrastive representation-based multimodal sound event localization and detection

Sound event localization and detection (SELD) refers to classifying sound categories and locating their locations with acoustic models on the same multichannel audio. Recently, SELD has been rapidly evolving by leveraging advanced approaches from other research areas, and the benchmark SELD datasets have become increasingly realistic with simultaneously captured videos provided. Vibration produces sound, we usually associate visual objects with their sound, i.e., we hear footsteps from a walking person, and hear a jangle from one running bell. It comes naturally to think about using multimodal information (image–audio–text vs audio merely), to strengthen sound event detection (SED) accuracies and decrease sound source localization (SSL) errors. In this paper, we propose one contrastive representation-based multimodal acoustic model (CRATI) for SELD, which is designed to learn contrastive audio representations from audio, text, and image in an end-to-end manner. Experiments on the real dataset of STARSS23 and the synthesized dataset of TAU-NIGENS Spatial Sound Events 2021 both show that our CRATI model can learn more effective audio features with additional constraints to minimize the difference among audio and text (SED and SSL annotations in this work). Image input is not conducive to improving SELD performance, as only minor visual changes can be observed from consecutive frames. Compared to the baseline system, our model increases the SED F-score by 11% and decreases the SSL error by 31.02° on the STARSS23 dataset, respectively.

Designing a practical physical framework for AI-enhanced sound detection and localization in vehicles

This paper presents the practical framework of an AI-implemented system currently in development for the detection and localization of external sound events to the vehicle. We delve into the initial block of the automotive project, analyzing sounds of interest and the noise we encounter, addressing the strategic arrangement of the four microphones on the sides of the vehicle, conceived as the sound input system. This configuration not only optimizes signal capture but also acts as sensitive ears, allowing the system to effectively analyze the acoustic environment. To mitigate wind interference, we have designed a microphone placement system that considers the vehicle's aerodynamics. The strategic location and implementation of aerodynamic protectors significantly minimize the impact of wind on the microphones, ensuring a sharper capture of acoustic signals. Complementing this arrangement, we apply advanced noise elimination processes, further enhancing the quality of the captured signals. Although in its early stages, this project aspires to provide effective solutions for vehicular acoustic perception, marking the beginning of a journey towards innovation in the detection and localization of sounds in mobile environments.

Joint Spatio-Temporal-Frequency Representation Learning for Improved Sound Event Localization and Detection.

Sound event localization and detection (SELD) is a crucial component of machine listening that aims to simultaneously identify and localize sound events in multichannel audio recordings. This task demands an integrated analysis of spatial, temporal, and frequency domains to accurately characterize sound events. The spatial domain pertains to the varying acoustic signals captured by multichannel microphones, which are essential for determining the location of sound sources. However, the majority of recent studies have focused on time-frequency correlations and spatio-temporal correlations separately, leading to inadequate performance in real-life scenarios. In this paper, we propose a novel SELD method that utilizes the newly developed Spatio-Temporal-Frequency Fusion Network (STFF-Net) to jointly learn comprehensive features across spatial, temporal, and frequency domains of sound events. The backbone of our STFF-Net is the Enhanced-3D (E3D) residual block, which combines 3D convolutions with a parameter-free attention mechanism to capture and refine the intricate correlations among these domains. Furthermore, our method incorporates the multi-ACCDOA format to effectively handle homogeneous overlaps between sound events. During the evaluation, we conduct extensive experiments on three de facto benchmark datasets, and our results demonstrate that the proposed SELD method significantly outperforms current state-of-the-art approaches.

A hybrid offline-online method for sound event localization and detection

A hybrid offline-online method for sound event localization and detection

Polyphonic sound event localization and detection using channel-wise FusionNet

Polyphonic sound event localization and detection using channel-wise FusionNet

Static Sound Event Localization and Detection Using Bipartite Matching Loss for Emergency Monitoring

In this paper, we propose a method for estimating the classes and directions of static audio objects using stereo microphones in a drone environment. Drones are being increasingly used across various fields, with the integration of sensors such as cameras and microphones, broadening their scope of application. Therefore, we suggest a method that attaches stereo microphones to drones for the detection and direction estimation of specific emergency monitoring. Specifically, the proposed neural network is configured to estimate fixed-size audio predictions and employs bipartite matching loss for comparison with actual audio objects. To train the proposed network structure, we built an audio dataset related to speech and drones in an outdoor environment. The proposed technique for identifying and localizing sound events, based on the bipartite matching loss we proposed, works better than those of the other teams in our group.

Polyphonic sound event localization and detection based on Multiple Attention Fusion ResNet.

Sound event localization and detection have been applied in various fields. Due to the polyphony and noise interference, it becomes challenging to accurately predict the sound event and their occurrence locations. Aiming at this problem, we propose a Multiple Attention Fusion ResNet, which uses ResNet34 as the base network. Given the situation that the sound duration is not fixed, and there are multiple polyphonic and noise, we introduce the Gated Channel Transform to enhance the residual basic block. This enables the model to capture contextual information, evaluate channel weights, and reduce the interference caused by polyphony and noise. Furthermore, Split Attention is introduced to the model for capturing cross-channel information, which enhances the ability to distinguish the polyphony. Finally, Coordinate Attention is introduced to the model so that the model can focus on both the channel information and spatial location information of sound events. Experiments were conducted on two different datasets, TAU-NIGENS Spatial Sound Events 2020, and TAU-NIGENS Spatial Sound Events 2021. The results demonstrate that the proposed model significantly outperforms state-of-the-art methods under multiple polyphonic and noise-directional interference environments and it achieves competitive performance under a single polyphonic environment.

W2v-SELD: A Sound Event Localization and Detection Framework for Self-Supervised Spatial Audio Pre-Training

w2v-SELD: A Sound Event Localization and Detection Framework for Self-Supervised Spatial Audio Pre-Training

Selective-Memory Meta-Learning With Environment Representations for Sound Event Localization and Detection

Selective-Memory Meta-Learning With Environment Representations for Sound Event Localization and Detection

Automated Audio Data Augmentation Network Using Bi-Level Optimization for Sound Event Localization and Detection

Automated Audio Data Augmentation Network Using Bi-Level Optimization for Sound Event Localization and Detection

Sound event localization and detection using a spatial omni-dimensional dynamic interactions network

Sound event localization and detection using a spatial omni-dimensional dynamic interactions network

Divided spectro-temporal transformer for sound event localization and detection in real scenes

Sound event localization and detection (SELD) involves the detection of sound events (SED) and the estimation of their direction of arrival (DoA) by utilizing multichannel sound signals. Recent research in SELD has predominantly focused on deep neural network (DNN) based models, which specifically emphasize learning temporal context. Examples of these models include the convolutional recurrent neural network (CRNN) and the ResNet-conformer architecture, which handle spectral and channel information only as the embeddings of temporal features. To fully exploit spectral information providing a crucial cue for both SED and DoA, it is imperative to devise a network architecture that effectively learns both spectral and temporal contexts. In this regard, we propose a divided transformer architecture that separately identifies the spectral and temporal context to encourage the model to learn more spectral characteristics of signals while retaining the temporal context. The efficacy of the divided spectro-temporal transformer approach is validated using the DCASE 2022 and 2023 challenge task 3 datasets. Furthermore, a series of parameter studies carried out to optimize the performance of SELD demonstrates that the number of frequency bins for attention and the pooling location impact the performance, and the divided spectro-temporal transformer is beneficial for both SED and DoA.

Sound Event Localization and Detection Using Parallel Multi-attention Enhancement

Sound Event Localization and Detection Using Parallel Multi-attention Enhancement

HAAC: Hierarchical audio augmentation chain for ACCDOA described sound event localization and detection

The goal of sound event localization and detection (SELD) is to detect the temporal occurrence activity of a known set of sound events and locate them in the spatial space. We argue that acquiring a large audio dataset is essential for one deep neural network-based SELD system learned as one supervised task. Nonetheless, gathering and annotating such datasets is a costly and time-intensive process. Hence, various data augmentation methods have attracted attention as a solution to increase sample diversity from the limited collections. In this paper, we propose to augment the limited audio samples for the deep neural network-based SELD system in two ways. One is the hierarchical audio augmentation chain (HAAC) proposed for the activity-coupled Cartesian direction of arrival output representation (ACCDOA) described SELD task. It consists of three waveform and spectrogram augmentation techniques, which are exquisitely assembled from the feature map augmentation to audio channel swapping, and finally sample mixup. Second, we propose to augment the training samples by generating more simulated audio samples and making the selected sound events list publicly available to the community. Experiments on the STARSS22 dataset showed that our HAAC audio augmentation chain greatly improved the SELD performance, which increased the sound event detection score by 24% and decreased the localization error by 12.1°. We demonstrate it’s one simple yet effective approach, compared to other data augmentation methods. Moreover, with more simulated audio samples, generated by convolving selected sound events with SRIRs, used for training, the SELD performance was improved greatly.

Sound event localization and detection using element-wise attention gate and asymmetric convolutional recurrent neural networks

There are problems that standard square convolution kernel has insufficient representation ability and recurrent neural network usually ignores the importance of different elements within an input vector in sound event localization and detection. This paper proposes an element-wise attention gate-asymmetric convolutional recurrent neural network (EleAttG-ACRNN), to improve the performance of sound event localization and detection. First, a convolutional neural network with context gating and asymmetric squeeze excitation residual is constructed, where asymmetric convolution enhances the capability of the square convolution kernel; squeeze excitation can improve the interdependence between channels; context gating can weight the important features and suppress the irrelevant features. Next, in order to improve the expressiveness of the model, we integrate the element-wise attention gate into the bidirectional gated recurrent network, which is to highlight the importance of different elements within an input vector, and further learn the temporal context information. Evaluation results using the TAU Spatial Sound Events 2019-Ambisonic dataset show the effectiveness of the proposed method, and it improves SELD performance up to 0.05 in error rate, 1.7% in F-score, 0.7° in DOA error, and 4.5% in Frame recall compared to a CRNN method.

Sound Event Localization and Detection Using Imbalanced Real and Synthetic Data via Multi-Generator

This study proposes a sound event localization and detection (SELD) method using imbalanced real and synthetic data via a multi-generator. The proposed method is based on a residual convolutional neural network (RCNN) and a transformer encoder for real spatial sound scenes. SELD aims to classify the sound event, detect the onset and offset of the classified event, and estimate the direction of the sound event. In Detection and Classification of Acoustic Scenes and Events (DCASE) 2022 Task 3, SELD is performed with a few real spatial sound scene data and a relatively large number of synthetic data. When a model is trained using imbalanced data, it can proceed by focusing only on a larger number of data. Thus, a multi-generator that samples real and synthetic data at a specific rate in one batch is proposed to prevent this problem. We applied the data augmentation technique SpecAugment and used time-frequency masking to the dataset. Furthermore, we propose a neural network architecture to apply the RCNN and transformer encoder. Several models were trained with various structures and hyperparameters, and several ensemble models were obtained by "cherry-picking" specific models. Based on the experiment, the single model of the proposed method and the model applied with the ensemble exhibited improved performance compared with the baseline model.

Sound event localization and detection based on deep learning

Sound event localization and detection based on deep learning

A Four-Stage Data Augmentation Approach to ResNet-Conformer Based Acoustic Modeling for Sound Event Localization and Detection

In this paper, we propose a novel four-stage data augmentation approach to ResNet-Conformer based acoustic modeling for sound event localization and detection (SELD). First, we explore two spatial augmentation techniques, namely audio channel swapping (ACS) and multi-channel simulation (MCS), to deal with data sparsity in SELD. ACS and MDS focus on augmenting the limited training data with expanding direction of arrival (DOA) representations such that the acoustic models trained with the augmented data are robust to localization variations of acoustic sources. Next, time-domain mixing (TDM) and time-frequency masking (TFM) are also investigated to deal with overlapping sound events and data diversity. Finally, ACS, MCS, TDM and TFM are combined in a step-by-step manner to form an effective four-stage data augmentation scheme. Tested on the Detection and Classification of Acoustic Scenes and Events (DCASE) 2020 data set, our proposed augmentation approach greatly improves the system performance, ranking our submitted system in the first place in the SELD task of the DCASE 2020 Challenge. Furthermore, we employ a ResNet-Conformer architecture to model both global and local context dependencies of an audio sequence and win the first place in the DCASE 2022 SELD evaluations.

SELD U-Net: Joint Optimization of Sound Event Localization and Detection With Noise Reduction

SELD U-Net: Joint Optimization of Sound Event Localization and Detection With Noise Reduction