Sound Event Detection System Research Articles

A sound event detection support system for smart home based on “two-to-one” teacher–student learning

Sound event detection (SED) is a core technology in smart home projects that rely on detected sound events to trigger specific actions. SED systems face two major challenges: high labeling costs and complex acoustic environments. To reduce labeling costs, some semi-supervised systems extract both global and local features for classification. However, these methods treat global and local features equally, not accounting for their varying importance when recognizing different types of sound events. Furthermore, to address complex acoustic environments, some studies use multitask learning frameworks to introduce SED-related tasks as auxiliaries to improve detection performance. However, these methods fail to align tasks within the framework, leading to conflicting outputs that may limit system performance. To address these issues, in this paper we propose a “two-to-one” teacher-student learning based semi-supervised SED system. This system employs a gating mechanism to selectively enhance global and local features, improving adaptability to different types of sound events, and incorporates a cross-task alignment module to interact SED with related tasks, reducing the risk of performance degradation caused by conflicting outputs. Experimental results on two datasets demonstrate that our system achieves the best performance in all metrics, with EB-F1 scores of 48.1 % and 64.7 %, representing improvements of 15.3 % and 10.6 % over the baseline ConformerSED system, respectively. Our work offers an effective SED solution for smart home projects by providing a semi-supervised SED system that performs well while reducing labeling costs.

Multi-granularity acoustic information fusion for sound event detection

Most previous works on sound event detection (SED) are based on binary hard labels of sound events, leaving other scales of information underexplored. To address this problem, we introduce multiple granularities of knowledge into the system to perform hierarchical acoustic information fusion for SED. Specifically, we present an interactive dual-conformer (IDC) module to adaptively fuse the medium-grained and fine-grained acoustic information based on the hard and soft labels of sound events. In addition, we propose a scene-dependent mask estimator (SDME) module to extract the coarse-grained information from acoustic scenes, introducing the scene-event relationships into the SED system. Experimental results show that the proposed IDC and SDME modules efficiently fuse the acoustic information at different scales and therefore further improve the SED performance. The proposed system achieved Top 1 performance in DCASE 2023 Challenge Task 4B.

Improved method for drone sound event detection system aiming at the impact of background noise and angle deviation

Drones pose a hidden threat to public safety, and effective and accurate detection technology for the drone that exist in the environment is imminent, in which how to weaken the influence of target sound source localization deviation and strong background interference noise on the detection task is the key to improve the accuracy of drone sound event detection. In this paper, a method has been proposed to improve the accuracy of drone acoustic event detection, named as the linear shrinkage-subspace projection-power spectral density filter method (LSP). This method mainly including covariance matrix reconstruction, steering vector recalibration, and filter coefficient redesign method. Firstly, based on Minimum Variance Distortionless Response, the linear shrinkage method is used to suppress the interference and noise components in the signal plus interference covariance matrix, and the sample covariance matrix is reconstructed to eliminate background interference noise. Then, the correlation between the steering vector and the eigenvector is used to eliminate the angle correlation term, and the subspace projection method is combined to recalibrate the steering vector, so as to improve the ability of the beamforming method to resist the angle deviation and realize the correction of the target source positioning deviation. Next, a redesign method for wiener filter coefficients based on the estimated power spectral density is used to further weaken background interference noise. In order to verify the accuracy of the proposed method, a complete drone sound event detection system is constructed by combining the deep learning drone sound event detection classifier, and the evaluation is carried out according to different angular deviations and interference sound distances. In addition, a new evaluation criterion is proposed, named as the Machine-Human Extreme Hearing Distance Rate (MHDR), which analogizes the system's detection ability with the ear's auditory detection ability. The research results of this article indicate that the detection accuracy of the detection system shows satisfactory accuracy when the proposed method is applied to circular microphone array, that improved by 15.12 % compared to existing methods. The proposed method improves the detection accuracy of the drone acoustic event detection task under the influence of sound source position deviation and strong background speech interference, and provides a reference for the development of anti-drone technology.

Novel sound event and sound activity detection framework based on intrinsic mode functions and deep learning

AbstractThe detection of sound events has become increasingly important due to the development of signal processing methods, social media, and the need for automatic labeling methods in applications such as smart cities, navigation, and security systems. For example, in such applications, it is often important to detect sound events at different levels, such as the presence or absence of an event in the segment, or to specify the beginning and end of the sound event and its duration. This study proposes a method to reduce the feature dimensions of a Sound Event Detection (SED) system while maintaining the system’s efficiency. The proposed method, using Empirical Mode Decomposition (EMD), Intrinsic Mode Functions (IMFs), and extraction of locally regulated features from different IMFs of the signal, shows a promising performance relative to the conventional features of SED systems. In addition, the feature dimensions of the proposed method are much smaller than those of conventional methods. To prove the effectiveness of the proposed features in SED tasks, two segment-based approaches for event detection and sound activity detection were implemented using the suggested features, and their effectiveness was confirmed. Simulation results on the URBAN SED dataset showed that the proposed approach reduces the number of input features by more than 99% compared with state-of-the-art methods while maintaining accuracy. According to the obtained results, the proposed method is quite promising.

Teacher-Student Framework for Polyphonic Semi-supervised Sound Event Detection: Survey and Empirical Analysis

Polyphonic sound event detection refers to the task of automatically identifying sound events occurring simultaneously in an auditory scene. Due to the inherent complexity and variability of real-world auditory scenes, building robust detectors for polyphonic sound event detection poses a significant challenge. The task becomes further more challenging without sufficient annotated data to develop sound event detection systems under a supervised learning regime. In this paper, we explore the recent developments in polyphonic sound event detection, with a particular emphasis on the application of Teacher-Student techniques within the semi-supervised learning paradigm. Unlike previous works, we have consolidated and organized the fragmented literature on Teacher-Student techniques for polyphonic sound event detection. By examining the latest research, categorizing Teacher-Student approaches, and conducting an empirical study to assess the performance of each approach, this survey offers valuable insights and practical guidance for researchers and practitioners in the field. Our findings highlight the potential benefits of utilizing multiple learners, ensuring consistent predictions, and making thoughtful choices regarding perturbation strategies.

DEW: A wavelet approach of rare sound event detection.

This paper presents a novel sound event detection (SED) system for rare events occurring in an open environment. Wavelet multiresolution analysis (MRA) is used to decompose the input audio clip of 30 seconds into five levels. Wavelet denoising is then applied on the third and fifth levels of MRA to filter out the background. Significant transitions, which may represent the onset of a rare event, are then estimated in these two levels by combining the peak-finding algorithm with the K-medoids clustering algorithm. The small portions of one-second duration, called 'chunks' are cropped from the input audio signal corresponding to the estimated locations of the significant transitions. Features from these chunks are extracted by the wavelet scattering network (WSN) and are given as input to a support vector machine (SVM) classifier, which classifies them. The proposed SED framework produces an error rate comparable to the SED systems based on convolutional neural network (CNN) architecture. Also, the proposed algorithm is computationally efficient and lightweight as compared to deep learning models, as it has no learnable parameter. It requires only a single epoch of training, which is 5, 10, 200, and 600 times lesser than the models based on CNNs and deep neural networks (DNNs), CNN with long short-term memory (LSTM) network, convolutional recurrent neural network (CRNN), and CNN respectively. The proposed model neither requires concatenation with previous frames for anomaly detection nor any additional training data creation needed for other comparative deep learning models. It needs to check almost 360 times fewer chunks for the presence of rare events than the other baseline systems used for comparison in this paper. All these characteristics make the proposed system suitable for real-time applications on resource-limited devices.

Polyphonic Sound Event Detection Using Mel-Pseudo Constant Q-Transform and Deep Neural Network

The task of identification of sound events in a particular surrounding is known as Sound Event Detection (SED) or Acoustic Event Detection (AED). The occurrence of sound events is unstructured and also displays wide variations in both temporal structure and frequency content. Sound events may be non-overlapped (monophonic) or overlapped (polyphonic) in nature. In real-time scenarios, polyphonic SED is most commonly seen as compared to monophonic SED. In this paper, a Mel-Pseudo Constant Q-Transform (MP-CQT) technique is introduced to perform polyphonic SED to effectively learn both monophonic and polyphonic sound events. A pseudo CQT technique is adapted to extract features from the audio files and their Mel spectrograms. The Mel-scale is believed to broadly simulate human perception system. The classifier used is a Convolutional Recurrent Neural Network (CRNN). Comparison of the performance of the proposed MP-CQT technique along with CRNN is presented and a considerable performance improvement is observed. The proposed method achieved an average error rate of 0.684 and average F1 score of 52.3%. The proposed approach is also analyzed for the robustness by adding an additional noise at different Signal to Noise Ratios (SNRs) to the audio files. The proposed method for SED task has displayed improved performance as compared to state-of-the-art SED systems. The introduction of new feature extraction technique has shown promising improvement in the performance of the polyphonic SED system.

Cross-Referencing Self-Training Network for Sound Event Detection in Audio Mixtures.

Sound event detection is an important facet of audio tagging that aims to identify sounds of interest and define both the sound category and time boundaries for each sound event in a continuous recording. With advances in deep neural networks, there has been tremendous improvement in the performance of sound event detection systems, although at the expense of costly data collection and labeling efforts. In fact, current state-of-the-art methods employ supervised training methods that leverage large amounts of data samples and corresponding labels in order to facilitate identification of sound category and time stamps of events. As an alternative, the current study proposes a semi-supervised method for generating pseudo-labels from unsupervised data using a student-teacher scheme that balances self-training and cross-training. Additionally, this paper explores post-processing which extracts sound intervals from network prediction, for further improvement in sound event detection performance. The proposed approach is evaluated on sound event detection task for the DCASE2020 challenge. The results of these methods on both "validation" and "public evaluation" sets of DESED database show significant improvement compared to the state-of-the art systems in semi-supervised learning.

Sound Event Detection System Based on VGGSKCCT Model Architecture with Knowledge Distillation

ABSTRACT Sound event detection involves detecting acoustic events of multiple classes in audio recordings, along with the times of occurrence. Detection and Classification of Acoustic Scenes and Events (DCASE) Task 4 for sound event detection in domestic environments is a contest on this task. In this paper, we engineer sound event detection systems using the data provided and the performance metrics defined in this contest. Note the performance metrics of polyphonic sound detection scores (PSDS) in 2 scenarios are adopted recently to be practical and effective. Our system development started with a basic system through reference to various systems in the contests of previous years. We developed a system similar to that used by the winning team in DCASE Challenge 2021. A clip-level consistency branch is then added to the model architecture to increase the performance of the PSDS in scenario 2, which focuses on identifying different event classes. In addition, we use knowledge distillation with the mean teacher model to improve system performance. In this way, the model can learn from the pre-trained model without being fully restricted by its performance. Finally, we further enhance the system robustness through consistency criteria in the second stage of training. On the official validation set of Domestic Environment Sound Event Detection (DESED) dataset, our final system achieves 0.418 and 0.661 on the PSDS in the two scenarios. It outperforms the 2021 baseline system with 0.341 and 0.546 on both scores quite significantly.

Zero-Shot Audio Source Separation through Query-Based Learning from Weakly-Labeled Data

Deep learning techniques for separating audio into different sound sources face several challenges. Standard architectures require training separate models for different types of audio sources. Although some universal separators employ a single model to target multiple sources, they have difficulty generalizing to unseen sources. In this paper, we propose a three-component pipeline to train a universal audio source separator from a large, but weakly-labeled dataset: AudioSet. First, we propose a transformer-based sound event detection system for processing weakly-labeled training data. Second, we devise a query-based audio separation model that leverages this data for model training. Third, we design a latent embedding processor to encode queries that specify audio targets for separation, allowing for zero-shot generalization. Our approach uses a single model for source separation of multiple sound types, and relies solely on weakly-labeled data for training. In addition, the proposed audio separator can be used in a zero-shot setting, learning to separate types of audio sources that were never seen in training. To evaluate the separation performance, we test our model on MUSDB18, while training on the disjoint AudioSet. We further verify the zero-shot performance by conducting another experiment on audio source types that are held-out from training. The model achieves comparable Source-to-Distortion Ratio (SDR) performance to current supervised models in both cases.

ANALYSIS OF THE SOUND EVENT DETECTION METHODS AND SYSTEMS

Detection and recognition of loud sounds and characteristic noises can significantly increase the level of safety and ensure timely response to various emergency situations. Audio event detection is the first step in recognizing audio signals in a continuous audio input stream. This article presents a number of problems that are associated with the development of sound event detection systems, such as the deviation for each environment and each sound category, overlapping audio events, unreliable training data, etc. Both methods for detecting monophonic impulsive audio event and polyphonic sound event detection methods which are used in the state-of-the-art sound event detection systems are presented. Such systems are presented in Detection and Classification of Acoustic Scenes and Events (DCASE) challenges and workshops, which take place every year. Beside a majority of works focusing on the improving overall performance in terms of accuracy many other aspects have also been studied. Several systems presented at DCASE 2021 task 4 were considered, and based on their analysis, there was a conclusion about possible future for sound event detection systems. Also the actual directions in the development of modern audio analytics systems are presented, including the study and use of various architectures of neural networks, the use of several data augmentation techniques, such as universal sound separation, etc.

Exploring Regression-based Approach for Sound Event Detection in Noisy Environments

Sound-event detection enables machines to detect when a particular sound event has occurred in addition to classifying the type of event. Successful detection of various sound events is paramount in building secure surveillance systems and other smart home appliances. However, noisy events and environ-ments exacerbate the performance of many sound event detection models, rendering them ineffective in real-world scenarios. Hence, the need for robust sound event detection algorithms in noisy environments with low inference times arises. You Only Hear Once (YOHO) is a purely convolutional architecture that uses a regression-based approach for sound-event-detection instead of the more common, frame-wise classification-based approach. The YOHO architecture proved robust in noisy environments, outperforming convolutional recurrent neural networks popular in sound event detection systems. Additionally, different ways to enhance the performance of the YOHO architecture are explored, experimenting with different computer vision architectures, dy-namic convolutional layers, pretrained audio neural networks and data augmentation methods to help improve the performance of the models on noisy data. Amongst several modifications to the YOHO architecture, the Frequency Dynamic Convolution Layers helped improve the internal model data representations by enforcing frequency-dependent convolution operations, which helped improve YOHO performance on noisy audios in outdoor and vehicular environments. Similarly, the FilterAugment data augmentation method and Convolutional Block Attention Module helped improve YOHO’s performance on the VOICe dataset containing noisy audios by augmenting the data and improving internal model representations of the input audio data using attention, respectively.

An Analysis of Sound Event Detection under Acoustic Degradation Using Multi-Resolution Systems

The Sound Event Detection task aims to determine the temporal locations of acoustic events in audio clips. In recent years, the relevance of this field is rising due to the introduction of datasets such as Google AudioSet or DESED (Domestic Environment Sound Event Detection) and competitive evaluations like the DCASE Challenge (Detection and Classification of Acoustic Scenes and Events). In this paper, we analyze the performance of Sound Event Detection systems under diverse artificial acoustic conditions such as high- or low-pass filtering and clipping or dynamic range compression, as well as under an scenario of high overlap between events. For this purpose, the audio was obtained from the Evaluation subset of the DESED dataset, whereas the systems were trained in the context of the DCASE Challenge 2020 Task 4. Our systems are based upon the challenge baseline, which consists of a Convolutional-Recurrent Neural Network trained using the Mean Teacher method, and they employ a multiresolution approach which is able to improve the Sound Event Detection performance through the use of several resolutions during the extraction of Mel-spectrogram features. We provide insights on the benefits of this multiresolution approach in different acoustic settings, and compare the performance of the single-resolution systems in the aforementioned scenarios when using different resolutions. Furthermore, we complement the analysis of the performance in the high-overlap scenario by assessing the degree of overlap of each event category in sound event detection datasets.

Performance analysis of weakly-supervised sound event detection system based on the mean-teacher convolutional recurrent neural network model

Performance analysis of weakly-supervised sound event detection system based on the mean-teacher convolutional recurrent neural network model

A Multi-Resolution CRNN-Based Approach for Semi-Supervised Sound Event Detection in DCASE 2020 Challenge

Sound Event Detection is a task with a rising relevance over the recent years in the field of audio signal processing, due to the creation of specific datasets such as Google AudioSet or DESED (Domestic Environment Sound Event Detection) and the introduction of competitive evaluations like the DCASE Challenge (Detection and Classification of Acoustic Scenes and Events). The different categories of acoustic events can present diverse temporal and spectral characteristics. However, most approaches use a fixed time-frequency resolution to represent the audio segments. This work proposes a multi-resolution analysis for feature extraction in Sound Event Detection, hypothesizing that different resolutions can be more adequate for the detection of different sound event categories, and that combining the information provided by multiple resolutions could improve the performance of Sound Event Detection systems. Experiments are carried out over the DESED dataset in the context of the DCASE 2020 Challenge, concluding that the combination of up to 5 resolutions allows a neural network-based system to obtain better results than single-resolution models in terms of event-based F1-score in every event category and in terms of PSDS (Polyphonic Sound Detection Score). Furthermore, we analyze the impact of score thresholding in the computation of F1-score results, finding that the standard value of 0.5 is suboptimal and proposing an alternative strategy based in the use of a specific threshold for each event category, which obtains further improvements in performance.

Multiclass sound event detection for respiratory disease diagnosis

Recordings from daily life contain a mix of sound types such as speech, coughing sounds, breathing sounds, and environmental noise, which provide a wealth of information related to linguistic messages, acoustic scenes, and health. These sounds in their unprocessed form overlap in both time and frequency domains, posing challenges to current methods for information extraction and analysis. To address this challenge, we introduce a novel multiclass sound event detection system that discriminates between speech, coughs, breathing, and other miscellaneous sounds (e.g., dogs barking, toilets flushing, babies crying) in the Coswara database, a crowdsourced database using a website application launched in response to COVID-19. This method extracts a feature set that includes Zero Crossing Rates, Short-Term Energy, and Mel-Frequency Cepstral Coefficients and uses Random Forests for multiclass classification. Preliminary results of the system yield a balanced accuracy of 87.5% detecting between speech, coughs, breathing, and other miscellaneous sounds. Using the multiclass sound event detection system, a time and acoustic-mediated forced-alignment technique is employed to discriminate complex sounds in real-time. We envision that the system could be used on devices with existing information extraction methods formonitoring respiratory diseases, such as pneumonia, pertussis, and COVID-19.

Multi-Scale and Single-Scale Fully Convolutional Networks for Sound Event Detection

Among various Sound Event Detection (SED) systems, Recurrent Neural Networks (RNN), such as long short-term memory unit and gated recurrent unit, is used to capture temporal dependencies, but it is confined in its length of temporal dependencies, resulting in a failure to model sound events with long duration. What’s more, RNN is incapable to process datasets in parallel, leading to low efficiency and low industrial value. Given these shortcomings, we propose to use dilated convolution (and causal dilated convolution) to capture temporal dependencies, as its great ability to ensure high time resolution and obtain longer temporal dependencies under the filter size and the network depth unchanged. In addition, dilated convolution can be parallelized, so it has higher efficiency and industrial value. Based on this, we propose Single-Scale Fully Convolutional Networks (SS-FCN) composed of convolutional neural networks and dilated convolutional networks, with the former to provide frequency invariance and the later to capture temporal dependencies. With the help of dilated convolution to control the length of temporal dependencies, we observe SS-FCN modeling a single length of temporal dependencies achieves superior detection performance for finite kinds of events. For better performance, we propose Multi-Scale Fully Convolutional Networks (MS-FCN), in which the feature fusion module is introduced to capture long short-term dependencies by fusing features with different length of temporal dependencies. The proposed methods achieve competitive performance on three main datasets with higher efficiency. The results show that SED systems based on Fully Convolutional Networks have further research value and potential.

Mel-log energies analysis of authentic audible intrusion activities in a Malaysian forest

Wildlife has been endangered due to illegal activities. This requires more effective surveillance measures. Felling timber and poaching are regular illegal activities but challenging to detect. Hence authorities should resort to modern technologies such as employing autonoumous surveillance to stop them. The Malaysian forest audio data were recorded to lay a foundation in initiating a cheaper and practical approach. Hence this paper reports the collection, processing and analysis of audio data in preparation to develop an autonomous sound event detection system. The recording was an emulation of possible illegal activities in a reserved forest. Sounds of chainsaw and hand hatchet cutting tree trunks were taken. It was found that there was a distinct pattern in the Mel-log energies audio feature of the sound, which could be used to identify illegal activities. Thus, it is believed that a detection through audio is a possible approach to be employed as one of the methods to stop illegal activities in the tropical reserve forests like those in Malaysia.

Joint Optimization of Deep Neural Network-Based Dereverberation and Beamforming for Sound Event Detection in Multi-Channel Environments.

In this paper, we propose joint optimization of deep neural network (DNN)-supported dereverberation and beamforming for the convolutional recurrent neural network (CRNN)-based sound event detection (SED) in multi-channel environments. First, the short-time Fourier transform (STFT) coefficients are calculated from multi-channel audio signals under the noisy and reverberant environments, which are then enhanced by the DNN-supported weighted prediction error (WPE) dereverberation with the estimated masks. Next, the STFT coefficients of the dereverberated multi-channel audio signals are conveyed to the DNN-supported minimum variance distortionless response (MVDR) beamformer in which DNN-supported MVDR beamforming is carried out with the source and noise masks estimated by the DNN. As a result, the single-channel enhanced STFT coefficients are shown at the output and tossed to the CRNN-based SED system, and then, the three modules are jointly trained by the single loss function designed for SED. Furthermore, to ease the difficulty of training a deep learning model for SED caused by the imbalance in the amount of data for each class, the focal loss is used as a loss function. Experimental results show that joint training of DNN-supported dereverberation and beamforming with the SED model under the supervision of focal loss significantly improves the performance under the noisy and reverberant environments.

A Comprehensive Review of Polyphonic Sound Event Detection

One of the most amazing functions of the human auditory system is the ability to detect all kinds of sound events in the environment. With the technologies and hardware advances, polyphonic Sound Event Detection (SED) can be developed to mimic the ability of the human auditory system. However, the development of a SED system is no trivial task, and several different factors often hinder accuracy. Although there are several overview papers available, most of them only provide a theoretical overview of algorithms used with little discussion. Thus, to the best of the authors' knowledge, there is no comprehensive review that covers this particular domain. Therefore, this paper aims to provide an in-depth discussion of different methodologies proposed by various authors that include the features used, detection algorithms, and their corresponding accuracy and limitations. Additional information on possible trends is also discussed that can be useful for future development works.